Experienced Points: Why Do Games Take Up so Much Space?

By Shamus Posted Tuesday Sep 15, 2015

Filed under: Column 83 comments

My column this week talks about game sizes (in terms of disk space) but wait don’t hit the back button yet it’s more interesting than it sounds I promise. Also, it’s better punctuated than the previous sentence, so do give it a try.

 


From The Archives:
 

83 thoughts on “Experienced Points: Why Do Games Take Up so Much Space?

  1. Wide And Nerdy says:

    Just speaking for me, if you could have worked compression into the title, I wouldn’t have even needed you to say “its more interesting than it sounds.” But then that’s because I know you know the lower level stuff enough to give me something interesting there.

    1. MrGuy says:

      Exprncd Pnts: Why Do Gms Tk Up so Mch Spc?

      There ya go.

  2. Neko says:

    (Obviously it wouldn’t make any sense if you could put a zip file inside a zip file and continue to see the data get smaller. In fact, if you put a zip inside of a zip, the end result will often be a file that’s slightly larger, since the data inside is already as small as it can get.)

    I present to you Droste.zip =)

  3. Grudgeal says:

    I have absolutely no knowledge of system architecture or games development, so if it’s all the same to you I’m going to read this week’s column and imagine it’s about Space space. You know, the one with astronauts in it.

    1. MichaelGC says:

      I was trying to work out how many Blu-Rays you’d need to store all the information in the Universe (which is mostly space-space, of course, not counting a few small lumpy bits).

      It’s somewhere between 2 x 10^77 and 2 x 10^113.

      It’s possible I may not have chosen the most accurate method of calculation.

      1. 4th Dimension says:

        The follow up question of course would be would these fit within the confines of the known universe? And how many Blue ray DVDs it would take to record all the data of such an universe where somebody carelessly dumped the data of another universe in Blue Ray DVD form.

      2. Abnaxis says:

        Wouldn’t the uncertainty principle make it impossible to store all the information in the Universe, regardless of the number of Blue-Rays?

        1. 4th Dimension says:

          Maybe record all the data above quantum level, and quantum problems express statistically?

        2. Humanoid says:

          Well if the universe consisted purely of Blu-rays and nothing else, then it’d be fairly easy to describe.

          1. 4th Dimension says:

            Not excatly. Even if two blue ray DVDs have excatly the same data BURNED into them, they are not completly identicall physically, and will differ on the nano and atomic level. So you will still need to record that atomic data to record the universe.

  4. Da Mage says:

    You mention Wolfenstein as an example for file sizes, here is another one.

    Elder Scrolls IV: Oblivion
    Meshes: 659 MB
    Textures: 1.13 GB
    Audio: 2.05 GB

    Elder Scrolls V: Skyrim
    Meshes: 1.45 GB
    Textures: 1.29 GB
    Audio: 2.26 GB

    These have been recorded by the filesizes of the .bsa archives that bethesda used to compress their data. What’s interesting here here is that in the 5 years between games, the filesize of the audio and textures has hardly changed, however the models filesize has doubled.

    1. 4th Dimension says:

      To be fair Skyrim doesn’t really have diverse enviroments. That is it has couple of enviroment sets (Norse Dungeno, Dwarf Dungeon, Cave, Forest, Wooden setlement, Stone settlement, Rocks) which they then copy pase all over the place. The only thing that diferes one city from another is the layout thus the meshes, but the texture are excatly the same. Other games might have goone for a different feel in each city, but in ES games they are only different in layout and small details.

      1. Ilseroth says:

        Plenty of open world games do it this way. For a recent example, take a look at metal gear solid V. Very few of it’s buildings are unique to one location, and most of its outposts are copy paste of each other with slight rearranging of buildings and enemy spawn locations.

        It is only specific locations that have unique buildings, and not necessarily all of them just a few to give the location its own flair.

        1. 4th Dimension says:

          Sure, but this expected repetition does drive down the need for detailed unique textures. Also Raygerio makes a good point that since Skyrim and Oblivion were bound by the requirements of previous console generation and had to run on the same hardware the texture sizes and diversity had to have been similar so it could run on such old hardware.
          As such Oblivion vs Skyrim comparision isn’t as usefull.

    2. Raygereio says:

      Both Oblivion and Skyrim were designed with the previous console generation’s hardware limitations in mind. Texture budget among other things.

      I think a more interesting comparison will be FO3 and the upcoming FO4.

    3. Scourge says:

      No mentioning of Kkrieger ?

  5. Wide And Nerdy says:

    So are we forever past the point where CPUs would pose a practical bottleneck on file compression?

    1. Zak McKracken says:

      as far as I know, h265 (video codec) will not work on some older machines (without hardware support), as it is too CPU-intense. Of course most processors, especially for mobile devices, have dedicated circuitry at least for many of the operations required for video decoding, so it’ll still work on most things currently in use — it’s getting close, though.

    2. AileTheAlien says:

      Even if CPUs were no longer a bottleneck to decompressing thing on the fly, it still makes sense to not decompress everything on the CPU. Really, you want to do your decompression as late as you can – ideally on the lump of hardware that’s actually giving the end result to the user. This’ll save bandwidth on all of your wires/cables/insert-name-here.

      For example, I can watch a YouTube video at high-def on my Chromecast, because it’s grabbing the compressed data straight from YouTube. If I try to play a game on my computer at the same resolution, and watch the game on my TV with my Chromecast, it’s blurry, and about 5 frames per second. There’s just not enough bandwidth to handle what I’ve trying to do.

      1. Abnaxis says:

        I still want to know how Steam manages to stream a games with acceptable latency on my local LAN…

        Compression trickery is almost certainly involed

    3. MrGuy says:

      To some degree, that’s like asking if we’re past the point where CPUs are a bottleneck for performance in general.

      If the CPU is doing nothing else, it can comfortably decompress data fast enough to render huge amounts of data (think high quality video) without breaking a sweat.

      But now think if the CPU is busy doing other stuff. Let’s say you have a ton of physics objects flying around from an explosion, and the CPU’s keeping track of all of them – speed, collisions, bouncing off scenery and each other, etc. Suddenly the CPU is very, very busy. Now suddenly throw the load of decompressing large volumes of texture data. Example – if you spun around suddenly and all the scenery behind the camera (with all its accompanying texture data out to the view limit) suddenly needs to be on the screen. The decompression might not be a big load by itself, but given the CPU has a lot of other things on its mind, it might be too much right at that moment to do everything smoothly.

      Edit: That’s assuming the CPU is the one doing your texture decompression (others have pointed later in this discussion there are techniques for bypassing the CPU entirely to decompress texture data…)

  6. Nevermind says:

    A correction for your article:

    Game developers don’t want to have to compress the game data if they don’t have to. It makes loading times longer (this was more of a concern in the 1990s than today) because the computer has to spend time un-compressing the data before it can be put to use.

    In fact, at least with today’s technology, compressing stuff makes it load FASTER, not slower. The reason is that HDDs, and especially optical drives, are glacially slow compared to RAM. They’re so slow that it’s faster to read, say, 10 mb and decompress on the fly than read 20 mb directly. Of course, this does not work with the best compression algorithims (the best ratio-wise, I mean), but the general approach is used all over the place.

    1. Wide And Nerdy says:

      20 millibits?

      That’s twenty thousandths of a bit. How can you have any fraction of a single bit? Is compression that good?

      . . . I think I’ve learned the lessons of pedantic nitpicking well. I’ve had so many mentors here.

      1. Nevermind says:

        > How can you have any fraction of a single bit?
        You can’t if you’re speaking of “amount of storage”, but maybe we’re measuring read speed and getting like 20mb/ns or something (-8

      2. 4th Dimension says:

        Considering that it’s IMPOSSIBLE to have a memory that small since the data can not be simpler than yes or a no (other than none) sizes lower than a bit are impossible. So the differentiation between M and m is not necessary.

        Allthough the fact that he seem to be measuring memory in bits and not bytes does concern me.

        1. Ingvar M says:

          Technically, millibits per second MAY be a useful measure of speed. If it takes you 50 seconds to transmit one bit, you are transferring data at millibits per second.

          This may, say, be over a heavily encrypted channel, using something like wheat-and-chaff to disguise the actual interesting data.

          For storage sizes, millibits are probably not very useful (although again in a very technical sense, RAID encoding stores less than one bit per bit stored, since the error correction values aren’t strictly speaking data).

          1. 4th Dimension says:

            Well, we are not talking about data speed now are we ;)

            Doesn’t RAID use parity? As in it stores ADITIONAL bits so they can check if the data is faulty? Then to store M bits you would use up N bits on the drive, where M<N? In that case it doesn't store less than one bit per bit of actuall data. One bit of data takes up a fraction between 1 and 2, but on the actuall hard drive we don't store singular bits but sets of them so a non integer amount of bits is never stored.

            Or am I wrong. Possibly.

            1. Ingvar says:

              Bits are, well, tricky? They’re a measure of information. So if you store the answer to four yes/no questions, yo have stored four bits. They’re also colloquially used to name storage locations for yes/no answers, so if you use five locations to store your four answers, have you stored 4 or 5 bits?

              If you count the information you have stored (which, to some extent, I think is the sensible approach), you have stored 0.8 bits per location.

              So if we talk about storage locations, non-integral bits probably never make sense. If we’re talking information content, integral bits are the only thing. If we’re talking information-per-location (which we almost always do, I guess), fractional bits MAY make sense, in some types of storage schemes, but probably more frequently on more permanent storage (disks, tapes, punch-cards, mylar tape, …).

              I don’t honestly know how prevalent ECC memory is, in this day and age, but if it’s at all common, there’s also not-useful-for-storage bits being used for error correction.

      3. The Snide Sniper says:

        Fractions of a bit make sense in algebraic coding (related to Huffman coding).

        The basic idea of Huffman coding is to make common symbols (eg. letters) use fewer bits than the less common symbols.
        Algebraic coding takes it a step further, and lets symbols occupy non-integer numbers of bits.

      4. Zak McKracken says:

        Let’s say you’ve got some compression algorithm which will achieve compression ratio of 1:50 on a certain type of data.

        So 1 GB would be reduced to 20MB, 1 MB to 20kB and so on.

        Soooo, what would one bit be reduced to? Tell me that, Mr Smartypants!

        1. 4th Dimension says:

          Compression ratio gratly depends on the data being compressed, not only TYPE but the actuall DATA, since something highly noisy and variable is much harder to express through equations. Thus a data of one bit can not be compressed further since there is no complexity to simplify, the light is on or not.

          Well thinking about it, if we had a large anough a dataset of these 1 bit datas, and if we accept a certain level of lossiness, than we might be able to try predicting the value based on statistics and any errors sweep under the rug of inevitable errors caused by lossy compression. Then we could achieve infinite compression since we would compress 1b into 0b, but during “decompression” only a certain percentage of bits would be correct. But think of the savings on the disk.
          I’m off to write a paper about this marvelous discovery! To the academia mobile . . .

          1. Ingvar says:

            And if we sacrifice the ability to uncompress, I have an almost-perfect compression scheme that can be iterated until only four bytes remain (although there’s a limit to the size of the original data, I don’t think it can encode data sets larger than about 2 ** 2000).

            Count the number of 0-bits and 1-bits in the dataset.
            Write first the number of 0-bits, then the number of 1-bits, in the following format:

            You can then iterate on this until you reach a four-byte value (for extra bonus, iterate until you hit a fixpoint).

            Unfortunately, it’s a 100% lossy compression. But that’s OK, right?

            1. Zak McKracken says:

              There are already more elaborate schemes for this. They’re called hash functions. You may have seen SHA-2, SHA-3 and MD5 in some places. There are a few others, and you just explained how they work. In veeeery simple terms :)

              In only slightly less simple terms, they’re methods which condense any size of data down into a small “fingerprint” which is still mostly unique, from which you cannot reconstruct the original data, and for which you also cannot easily find a dataset which will produce the same hash. That latter bit is where your method would fail fairly quickly, I think…

              1. Ingvar M says:

                Yeah, it’s carefully designed to sound like it’s useful while having NO USE WHAT-SO-EVER. Actually, I tell a lie, it would possibly make for a good interview coding question, if I hadn’t published it in public right now. Dang!

                Now I wonder if it tends to hit a fix point (intuition says “yes” and, if so, if that fixpoint tends to be the same across files or differ depending on the file. That sounds l;ike approximately 2 ** 32 possible fixpoints, so is actually testable by exhaustive search. I shall refrain to shout “TO THE CODEMOBILE!” and file that for ‘if I get around to write code this weekend’.

      5. Wide And Nerdy says:

        I thought I’d attained the 8th plateau of nitpicking but I see that I still have much to learn.

        1. The Rocketeer says:

          This is programming; you literally need to nitpick electrons to stay competitive. A for effort, though.

          1. 4th Dimension says:

            You only nitpick electrons? What about the holes huh? Clearly you are a casual nitpicker ;)

      6. Volfram says:

        lazy capitalization standards and anal-retentive pedantry.

    2. Richard says:

      Indeed, it is more than an order of magnitude faster to store the game data in a single file compared to a pile of individual files, and compressed is often even faster, partly because of something I mentioned a few days ago.

      If you take a look at (say) Half-Life 2, nearly all the game content is in 25 “VPK” files, a total of 2.55GB.

      VPKs aren’t compressed as a whole, and the HL2 VPKs get a little better than 2:1 compression.

      However, decompression is now extremely cheap, so surprisingly compressing it often does make it faster, by burning some CPU to save disk access.

      I’ve done direct comparisons of ZIP against folder-structure-on-disk, and even on an SSD (zero seek time) the folder structure takes much more than 10 times as long to load.

      Aside from that textures are almost always compressed – complete with the obligatory flamewars about the right one to use in each situation – PNG v DXT1 v DXT3…

      Modern GPUs accept several compressed texture formats directly, which is even better as using these results in far less data to transfer across the GPU bus.
      – This does mean the game simply won’t run on GPUs that don’t support one of the compressed texture formats it has (or can decompress into)

      1. Wide And Nerdy says:

        Are we sacrificing anything else to have that direct file support on the GPU? I thought I’d read that part of what makes GPUs so fast is there simplicity (well, that and being able to throw tons of cores at the same task).

        Has anyone got low level graphics programming for complete idiots?

        1. 4th Dimension says:

          GPUs can do direct conversion of textures because they probably have on board a chip that is designed only to do such conversions. If you can build a hardware implementation of some process it will invariably beat the implementation of the same process on a similar processor. The tradeoff here is that you can use that chip ONLY to convert between formats that it was designed for while the CPU can do ANY computation. Also you can not implement everything this way simply because the hardware would gro WAAAAAYYYY too complicated and big.

    3. Volfram says:

      I’d wanted to mention exactly this. I did some work on a procedural image system and once I’d gotten it completed, I did some performance testing to see if any of my optimizations were worth it and if I should try for any further.

      TL/DR: compression is probably worth it on modern systems because computational power has been increasing at the rate of Moore’s Law, while disk access has been increasing at a much slower rate. It’s generally faster to pull as little information off the disk as possible and let the CPU sort out what to do with it.

      –Long, skippable, ego-stoking version–

      My files came in 3 formats: an HTML-like .txt file, a binary version that was not human-readable, and a version of the binary file that had been run through a standard library compression algorithm.

      For an example of how effective the system I put together was(someone mentioned Kkrieger above, I adopted the same strategy Farbrausch used and though I’ll never be as good at it as they are, the results are still impressive), one of the test files I put together that spews out(as I recall) sixteen 128x128x32-bit color images.

      The .txt file is about 16kb
      the .pri(PRocedural Image, uncompressed) is 12kb
      the .prc(PRocedural image, Compressed) is barely over 2kb

      I don’t remember the exact numbers, but the load and processing times were similarly faster for the smaller files than even a single uncompressed image, and the processing was deliberately about as complicated as I had the patience for at the time, leading me to conclude that for most files, the bottleneck would be disk read speed, not processor speed.

  7. kunedog says:

    I knew 500GB was too low when it was announced as the standard for both of the newest consoles. But it’s worth noting that at least the PS4’s hard drive is user-replaceable (for more space and/or SSD speed) while the X-Bone’s is not. It’s just one of the many things Sony did so much better this generation, even when it shouldn’t have cost Microsoft much to keep up.

    1. Thomas says:

      Both of them are going to release increasingly large harddrive sizes I imagine as the generation goes on.

      I don’t think 500GB is too much of a problem though. It means you can probably only have 10ish AAA games installed at a time, but that’s probably sufficient for most people. I’ve got 4 games installed on my PC right now (2 of those are indie too) and I rarely go above 5.

    2. Daniel says:

      XBox One allows you to plug in any USB 3.0 hard drive that is 256 GB or larger for use as external storage. Games and Apps can be saved and run from there.

      Even easier than replacing an internal drive. Especially when you want to take your installed games with you to play on another Xbox One.

      1. Wide And Nerdy says:

        Wii U does that too, although you need your external hard drive to be an externally powered one. I have 2TB drive for it which is the largest it supports but considering how much smaller most Wii U games are, 2TB is all I’ll ever need.

        1. Humanoid says:

          You can use a USB Y-cable to connect a USB-powered external HDD to a Wii U – i.e. it connects the HDD to two USB ports on the console. Probably worth checking if the particular model of HDD has been confirmed to work, since different makes and models may have subtly different power requirements. I wonder how much difference an external SSD would make.

          Aside, it’s disappointing to hear Microsoft kept their restrictions on the internal drive replacements. This means there’s been no change since the previous generation, where the PS3 was compatible with essentially any 2.5″ disk you liked, even SSDs, whereas the XB360 limited you to the proprietary 250GB drive they sold for an extortionate price. Mind you, users who did replace their PS3 drive with an SSD reported next to no improvement, so there’s some sort of bottleneck somewhere that’s independent of disk I/O. Wonder if the PS4 is similarly bottlenecked.

          1. Wide And Nerdy says:

            Well, considering games were already tweaked for the specific capabilities of the console, I could see how the processes wouldn’t be ready to account for a sudden increase in hard drive performance. Its kind of like Mass Effect 1. They set the elevator scenes based on old hard drive speeds, so it will take as long no matter how quickly you’re loading data behind the scenes.

            1. guy says:

              Also, it’s not like the processor signals the disk to send all the data it needs and then waits until it’s done arriving. It can and usually does do something else while waiting for the data to arrive, so it can read separate parts off the disk and do whatever is needed to get the stuff it’s already read set up for playing. The disk access takes so much longer than anything else that it’s usually the limiting factor, but it doesn’t have to be. Also, the PS3 memory architecture is strange and inconvenient.

  8. Septyn says:

    I don’t entirely agree that there hasn’t been meaningful improvements in general compression. I think people are the victim of inertia in adopting new compression technology, partially because OSes like Windows support ZIP as the lowest common denominator.

    Compression is bound by a triangle with time, ratio, and memory as the corners. You can have a fast, low memory footprint compressor but it’ll have a terrible ratio (new size to original size), or you can shoot for ultra-high ratios but you need 4GB of RAM and it’ll take 8 hours. The standard DEFLATE algorithm leans toward the fast-but-bad ratio side of the triangle, because it was developed decades ago when computers had megabytes of memory and clock speeds in kilohertz.

    For work I maintain 21 TB of text log files, and using some reasonable settings in 7zip (using the ppmd algorithm) I keep it stored on 1 TB of hard drive space. I could go smaller but the space gains aren’t worth the extra time.

    Textures and code isn’t as compressible as text, of course, but there are still people working on finding new sweet spots in that triangle. But it’s unlikely a game company is going to invest in research for that type of tech, or use cutting edge and unproven algorithms in their must-release-by-Christmas title.

    For further reading I suggest Googling things like Calgary Corpus, zpaq, and ppmd. There’s some interesting stuff going on out there.

    1. DrMcCoy says:

      There also already are a few texture compressions algorithms wildly supported by graphics hardware. So you can just push the compressed data to the GPU, and have all the savings advantages apply to GPU memory and transfer times as well; instead of just decompressing the textures on the CPU before using them.

      On desktop systems, that’s generally the S3TC DXTn formats. They’re block-based, compressing down by a factor of either 4 or 6 (depending on whether you want alpha or not).

      1. Peter H. Coffin says:

        Using those kind of tricks make graphics card (or console) manufacturers very happy. Because they’re almost never portable between (for example) AMD and nVidia, or often even between 7xxx cards and 9xxx cards. It’s awesome if you’re developing fluid dynamics software that’s got literally dozens of potential customers in the market, but not so great for games, where the range of hardware is huge.

    2. Scourge says:

      As you mention this am I reminded that not all games are equally good when it comes to being compressed.

      Romance of the three Kingdoms 7 for example was 2.4 Gb or so unpacked. But when compressed, it didn’t even have to be a particular good compression, it was reduced down to 268.88 MB.

      That is almost a 10 to 1 ration.

      1. Volfram says:

        Text-heavy games tend to compress well(without knowing anything about Romance of the Three Kingdoms). There’s also functionality used for the Playstation Portable where data on the outer edge of a disk tended to load faster than data on the inner edge, so many games were zero-padded to optimize loadtimes. When certain features came out that allowed owners to make copies of their disks(which may or may not have been legal), it wasn’t uncommon to see a 1.8GB UMD ISO for, say, Final Fantasy: War of the Lions, “compress” down to under 400MB because so much of it had been a giant buffer file.

  9. The Snide Sniper says:

    Even if data compression hasn’t really advanced much in the past decade, there still remain some things that can be done to make old games smaller:

    Text can be compressed, though text contributes little to total size.
    Older software may use uncompressed audio, which may now be encoded using MP3 or Vorbis formats.
    Newer image and video compression methods may yield gains of ~20% or so.
    Applying data compression methods to the 3D models used might yield some additional space, though re-formatting the models for better compression would require some extra work.
    If you are willing to devote all necessary time, you could difference-encode similar textures.

    When all is said and done, however, you’ll have spent days to save a few megabytes that no-one really cares about anymore.

    New games might be made smaller if more information was known about the target platform. Will the user want 4k x 4k textures? No? Okay, here are some 1k x 1k textures, for approximately 16x space savings. Different textures can be given different priorities (the love interest’s face, for instance, is a better use of pixels than generic rock number 573).

    I suspect that if space ever becomes an issue again, we’ll see it answered with conditional installations, rather than better compression.

    1. DrMcCoy says:

      Older software may use uncompressed audio

      Before MP3, old games frequently lowered the sampling frequency to 22050 Hz or even 11025 Hz, and/or went down to 8 bit per sample. And/or used something like (A)DPCM to compress audio (there’s literally hundreds of slightly different ADPCM algorithms out there).

      There’s not much to gain there for the most part.

      1. The Snide Sniper says:

        You have a good point there. Modern formats might not even support such low bitrates. File-size improvements might only be possible for relatively recent games.

        Music might also be in MIDI format, which would not benefit from MP3 or Vorbis re-encoding, but might benefit from more general data-compression (eg. zip).

        That said, better audio compression (such as using variable bitrate formats) would likely be more useful – for reducing file size – than better video or mesh compression. My examination of game files reveals that audio tends to comprise the lion’s share of the data.

        1. guy says:

          Huh, I would not have expected audio to play much of a role. I guess it’s because it tends not to be reused on the same scale as textures.

        2. Mephane says:

          If/when text-to-speech becomes 100% indistinguishable from actual voice actors, I can imagine a lot of space may be saved in dialog-heavy games by replacing the voice files with (annotated?) text.

          1. Wide And Nerdy says:

            I’m craving the day that happens. Just imagine what we could do with that. We could even have the same line emoted differently based on different conditions and do so dynamically. Writers could go back to having the amount of content of the old non voiced text heavy games and be able to revise up till launch.

          2. MrGuy says:

            I think some forward thinking developer should use a Mechanical Turk-like approach, and replace pre-recorded voice with cheap-to-hire voice actors overseas reading the lines in real time from a script…

            Yes, I know, Stephenson already thought of it…

  10. Ingvar M says:

    I am bound to point to one of the more delightful writings on Zip files that I have come across.

    Russ Cox, on self-referential Zip archives

  11. Draklaw says:

    What can we fit in 96k ? (in 2004). Wikipedia link for those who prefer.

    1. DrMcCoy says:

      Of course, that’s not a general purpose compression, but highly tuned generators for textures, geometry and the like. The Farbrausch people are really good in those. :)

      I still really love debris.

    2. MichaelGC says:

      Also, icosahedral link for those who prefer! :D

      http://www.shamusyoung.com/twentysidedtale/?p=537

    3. Volfram says:

      Early exposure to works of Farbrausch and the impression their skills and accomplishments made on me had a great impact in the strategies I adopted (and am still adopting)in my own designs later.

      I don’t like this world where a typical game download may take up more space than an entire season of anime.

      And yes, Debris is really good.

  12. Zak McKracken says:

    I think there was a LOT of pressure on people to keep size down before the advent of the CD-ROM, and what people invented to deal with it but still increase complexity of games, was procedural generation.
    Which, as Draklaw and Bones up there show, can be extremely effective.

    … I’m wondering why Shamus, of all people, did not mention this.

    Then, with the CD, suddenly we had more space than an average hard drive could hold, so people started adding video like there’s no tomorrow. Of course it was only lightly compressed compared to today’s methods, because CPU’s could not have handled the load. Then, when we had just gotten to 8CDs, came the DVD-ROM and things continued until the advent of web video brought better codecs, which brought game sizes down or at least slowed the growth a bit, depending on the amount of pre-rendered video sequences (and recorded sounds) in a game.

    These days it’s much less about available capacity and more about loading times etc.. Which means that now game sizes vary wildly, depending on the amount of procedural vs. “handmade” content, and any other number of developer choices. I suppose that as long as loading screens are not too long, it’s pretty much a result of the way the game content is created.

    1. MichaelGC says:

      Aye, well different audience over on The Escapist, o’ course, and the brevity of the article notes above suggest someone currently attempting temporal compression; i.e. doing x number of things in the time it takes to do y number of things, where y<<x.

      Genuine if utterly tangential question – are textures the kinds of things that lend themselves to procedural generation? Given those .kkreiger screenshots, it’s clearly do-able (they’re quite same-y in colour, thus distinguishing it from other shooters not-at-all…), but wouldn’t it be a lot harder to proc-generate interesting, varied textures than to “just” generate the shape of a level or planet or whatever?

      (I should note that I may be overestimating the difficulty due to excess quantities of not having a clue.)

      1. Xeorm says:

        They can be generated procedurally, but I wouldn’t say that they lend themselves well to doing them procedurally. Usually much easier to make it beforehand to check that it’s good, rather than doing it at runtime.

        1. MrGuy says:

          Check out Shamus’ coding article on Pixel City for some examples of creating textures procedurally (and associated challenges).

      2. Zak McKracken says:

        Yeah, you’re probably right about the y<<x thing …

        I think the answer to the procedural textures would be "it depends", again…

        Back in the day in Truespace I used loads of procedural textures: For wood, marble, orange-peel kinda bumps, waves… combined with a seeded quasi-random generator (you provide some number, it generates a random-looking noise from it, but with the same input number, it will always be exactly the same) — veery useful, especially for someone who can't draw and did not have access to good photographed textures.
        You could also easily generate procedural texture maps for text or regular manufactured stuff, then add some perturbation to make it look less clinical.

        Then again, if you use the same procedural maps over and over again, it will start to become boring at some point (unless you spend the effort to create a more complex procedural system), and of course it becomes much harder to put feature x at location y on a given texture on a specific object, so while this may work fine for some things, especially peripheral ones, it also complicates the creation process depending on how much control someone wants. And it can be a source for glitches.

        So I suppose it's a bit like other procedural content: It creates a completely different set of demands on the creators, so depending on what kind of team you have and what kind of result you want to achieve, different amounts of procedural content can be appropriate. I wouldn't be surprised if storage space was a distant third to fourth in the ranking of arguments.

        1. MichaelGC says:

          Thanks all! I think part of the reason I was thinking it would be so difficult was that I was sort-of imagining having to generate large, object-encompassing textures completely from scratch – rather than e.g. ‘take a tiny existing picture and do boatloads of cunning pseudo-random things to it and with it’. All part of not having a clue of course – anyway, I’m off to visit Pixel City! :D

  13. Adeon says:

    Huh, I’ll admit I had wondered why the Wolfenstien games were so large compared to the other games in my Steam library. I was unaware of the megatexture thing so that’s good to know.

  14. Hermocrates says:

    To my knowledge, there are only four games that use [Megatextures].

    Actually, there are five, but the one you’re likely forgetting is incredibly forgettable as a game. It’s Enemy Territory: QUAKE Wars, the id Tech 4 game that first introduced MegaTextures. But I’ll admit, it’s an outlier, and isn’t representative of MegaTextures as we see them in id Tech 5 games: the game is only about 5.10 GB, only 3.15 GB of which are the compressed MegaTexture files, and lacks the virtual texturing that allowed id Tech 5’s MegaTextures to be so large.

    It was also vastly more disappointing than RAGE ever was.

    1. sv_blond says:

      The Doom 3 source has a MegaTexture.cpp file, whatever that means in practice. While on the subject I think Rage is underrated graphically and I think they put the engine too good use. As for Wolfenstein I’m not very far into the game, but it doesn’t seem like they got much out of having virtual textures.

  15. Dude says:

    There are better compression methods out there. Pirates are quite well versed in them. When a new game comes out, within a couple of days, usually an unaltered disk image of the game is available, which for most current AAA games tends to be upwards of 15+ GB. Then a week or two later a “Repack” version comes out, which discards other (non-English) language audio files and localization, and some extras, but does not discard any of the core components (textures etc), and can easily reduce the size of the same installation setup to about 5GB through those compression methods. That’s a 1:3 reduction in download size, making for very happy pirates.

    Uncompressing/first install from such setups probably takes a while, but then, these days installing any big game takes a while.

    I sometimes think that the reason developers/publishers–especially on the PC–don’t bother using these compression tools is to discourage pirates from pirating, because 15GB downloads are still not a piece of cake to download for most of the world.

    1. MrGuy says:

      I sometimes think that the reason developers/publishers”“especially on the PC”“don't bother using these compression tools is to discourage pirates from pirating, because 15GB downloads are still not a piece of cake to download for most of the world.

      By your own argument, this seems unlikely to be the rationale. Because (as you point out) nothing stops the pirates from compressing the cracked version after-the-fact before distributing it.

      As an anti-piracy tool, this seems essentially useless, and just makes things annoying for paying customers (by eating their hard drive space) without actually making things appreciably harder for pirates.

      On second thought, the same could be said for DRM, so I guess maybe it IS why they do it…

      1. guy says:

        I suspect the real reason is that they want to only distribute one version and not deal with making separate versions for each language. That sounds very inconvenient for them and bilingual people.

      2. Dude says:

        Since when did publishing companies care for paying customers? Every form of DRM only troubles paying customers. Since the dawn of DRM.

    2. 4th Dimension says:

      On the other hand their bying costomers too will have to download those 15 GBs of data too, AND deal with all sorts of annoying DRM.
      So how is downloading 15GB+DRM easier than downloading 15GB (or less)?

  16. AdamS says:

    As someone who just replaced his aging laptop with a tablet with superior processing power (yes it’s been that long since my last upgrade.) but inferior memory, this is a topic near and dear to my heart.

  17. Jeff says:

    The most important thing to know about compressions, is that they should be done to the beat of Staying Alive.

Thanks for joining the discussion. Be nice, don't post angry, and enjoy yourself. This is supposed to be fun. Your email address will not be published. Required fields are marked*

You can enclose spoilers in <strike> tags like so:
<strike>Darth Vader is Luke's father!</strike>

You can make things italics like this:
Can you imagine having Darth Vader as your <i>father</i>?

You can make things bold like this:
I'm <b>very</b> glad Darth Vader isn't my father.

You can make links like this:
I'm reading about <a href="http://en.wikipedia.org/wiki/Darth_Vader">Darth Vader</a> on Wikipedia!

You can quote someone like this:
Darth Vader said <blockquote>Luke, I am your father.</blockquote>

Leave a Reply to guy Cancel reply

Your email address will not be published.