Game Sales vs Game Quality

By Shamus Posted Monday Sep 22, 2008

Filed under: Video Games 46 comments

Chris’ survival Horror Quest has a brilliant post that examines the sales performance of PS2 games against their metacritic scores. He’s looking to see how much quality affects sales. He charted 1,281 games and shows us the breakdown in a number of very interesting graphs.

The only nitpick I have is that I’ve never thought scores were all that useful for determining quality. The way the review system works, a critic usually sits down and pushes through a game in less than a week and then hammers out a review. (And the whole system is a sham in the PC realm, where the reviewer is likely using a top-end PC and a review copy that might not have the DRM found in the retail version.) The process suffers from the same problem that movie reviews do, which is that the reviewers are voracious consumers of games, to the point where they make “hardcore” gamers seem “casual”. Add in the marketing “tilt” effected by big name publishers (which we caught a glimpse of in the firing of Jeff Gerstman) and you have a system where scores don’t have a lot to do with quality. I trust scores to filter out the really horrible stuff, but beyond that I rely on demos and word of mouth. I’ve seen many big-name, top-rated games that turned out to be “meh”, and I’ve seen some real gems that were given modest scores by critics.

The disparity between scores and quality might account for some of the seeming randomness in Chris’ charts, but absent a way to quantify subjective things for that many games, there just isn’t any good way to sort that out.

I will add that I wish I had access to the data he’s using. There are two things I’d like to do with it:

  1. Color-code the dots by year, perhaps using red for the first year of the PS2 and going through the color spectrum from there. It would be interesting to see if patterns emerged from the noise, if certain years trended higher than others, and so on.
  2. Size the dots by the relative size of the publisher. EA would have great big dots, and little operations would be tiny specs. While big-name publishers don’t always throw heaps of money at a game, I’ll bet we’d see some patterns emerge that would tell us about the value of large marketing campaigns.

Only 8 diggs on that article? That’s a crime. It was a great read.

 


From The Archives:
 

46 thoughts on “Game Sales vs Game Quality

  1. Kel'Thuzad says:

    I usually read the reviews for laughs. I don’t trust Gamespot anymore. I guess I don’t trust any reviewers.

    1. Gilf says:

      Except Yahtzee. Yahtzee isn’t even really in it for consumer advice; he’s just entertaining. I at least trust him to do the job he says he will.

  2. Chilango2 says:

    Yeah, I definitely don’t trust GS anymore, since the Gertsman firing. I mean, it was pretty clear even before that (Oblivion) that they had become suckers for pretty graphics and didn’t take into account how games ran if your system was below the bleeding edge, but an actual Payola type scandal was the straw that broke the camels back.

    I wish, honestly, I knew of a review site I *could* trust. I trust Shamus, Zero Punctuation, and the Escapist (which isn’t to say I *agree* with them), but those set ups aren’t necessarily the best if I have random game X I am thinking of buying because it sounds interesting and I’d like to know what it’s like. GS is good for pointing out games that are massivily flawed so long a their not big budget titles, however, for what little that’s worth.

  3. Matt says:

    Out of interest, Shamus: which modestly-scored games would you consider gems?

    I’m currently replaying Arcanum, which is definitely one of the best CRPGs out there. If I remember correctly, it got okay but by no means great scores. Admittedly, it looks pretty bad (ugly rather than functional) for the most part and the controls were awkward at best, but then I rarely see reviewers go down to 70 scores because a game looks great, has smooth controls but is as generic as they come…

    P.S.: As far as trusting reviews is concerned – if I know a reviewer pretty well, I’ll also know whether my assessment would be similar to theirs or not. Just ignore the scores. And yes, mainstream reviews are biased towards mainstream gamers… which is something that makes sense, even though I’m not in their target audience.

  4. Gildan Bladeborn says:

    @Matt:
    My biggest gripe with Arcanum revolves around the combat system: It’s garbage. The Fallout team made Arcanum, so it’s not as if they haven’t designed a turn-based RPG before. Real-time combat is extremely annoying, especially when you’re playing a marksman, since everything moves so fast and it’s like a bad version of Diablo. The Turn-based system seems okay until you realize that you’d be getting wailed on less often if you were using the Real-time system, which inevitably leads me to what I do: have the game switch to Turn-based by default, fire just enough to not use up my turn, and then switch to Real-time and take my chances. Obviously this is not what the system is supposed to do.

    If the game had used a Baldur’s Gate style Real-time system (real time that you can pause so you can give orders/plan tactics) it would be practically perfect.

    Oh, if you haven’t tracked down the unofficial Arcanum patch, you really should. The community has fixed a ton of issues I didn’t even realize were issues until I read the patch notes, heh.

  5. The Lone Duck says:

    As far as gaming journalist (oxymoron) reviews go, I read 1up and Kotaku, but just for information. I pay more attention to sites like Shamus here, and Penny Arcade. I buy games so rarely, and have access to demos, that I’m usually pretty whether I’ll like a game before I buy it.
    I agree that many reviews are not constructed well. Often times, reviews will simply tally the features and modes, rather than critique the experience. (You know, the mindset that any game without multiplayer is made better with multiplayer.) Aside from making sure the controls work, they don’t examine the design and structure of the game.
    I think a lot of the problem lies in the intent of the review. Most reviewers are trying to tell us what we should and shouldn’t buy; they’re trying to help us get the most bang for the buck. That’s not a malicious goal by any means. But I think we’d be better served, by an examination of the game itself. Rather than jusging it in terms of money value (is it worth sixty dollars, forty, twenty? Tell us what the game tries to be, and how it does so. Don’t rag on Dynasty Warriors for being Dynasty Warriors, anymore than you criticize a clown for wearing the same big shoes.
    For that matter, how many reviews cover games that are moving? That are sublime? That are thought provoking? None I see, except maybe something in the nethers of the internet.
    To me, the biggest problem with reviews as they stand, is they don’t challenge the industry. They just accept and want the status quo. When will we get the Kubricks, Bergmans, and Kurosawas of videogames. When will we see videogames that actually mean something aside from a cheap thrill. There’s been some movement in that direction, but not much. And reviews don’t even look at that. Suikoden II and Xenogears are two of my favorite games. But reviews don’t look at characterization, or the themes of war’s tragedy. They just look at the mechanics. It’s as if a film critic discusses cinematography without looking at the themes of the movie. As I see it, gaming journalism holds back the medium with its limited expectations.
    I have hope for games in the future. Becoming more like movies is a good thing for games, if we ever want them to be more than toys for young adults.

  6. K says:

    It definitely was a great read. I don’t like survival horror games and I have that blog in my RSS now due to that exact article. I would like to see where your ideas go with company size and/or year.

  7. Steve C says:

    Did you tell Chris what graphs you would like to see? If you ask him directly, he might do it since it only adds to his article.

    I liked his article, but I thought that his implicit assumption that score = good indicator of quality is not true. A bump up in score can be purchased by marketing dollars, (but not a huge bump). Unit sales can also be an indicator of game quality. I don’t know how best to mine the data to screen out reviewer payola but Shamus your 2nd idea is a good place to start.

  8. Factoid says:

    I hope he posts his raw data. I want to do a regression analysis on it. Graphs are nice for seeing relationships between data, but there’s nothing like a good r^2 value to tell you just how much the relationship between X and Y are correlated.

  9. Thanakil says:

    What I think may pose a problem when comparing score and sales :
    Often reviewers will “increase” (or basically, be a LOT more lenient) toward games that they know will sells well/are popular.
    They don’t want to frustrate the fanboys, they don’t want to lose readership because they scored a game too low for the taste of the fans.

    Easiest examples would be Halo 3 or the GTA games in general. GTA4 has a (from reviewers) metacritic score of 98.

    The GTA games are great, but 98? There’s no way that you can get 84 reviewers (the number of scores counted) and all have them agree that “damn, this game is really REALLY close to perfection!”. (only 5 scores under 90)
    And the few reviews who actually give lower scores to the game (80 to 87) are from websites who know that they don’t have to be afraid of a huge backlash from the fans because of it. (the fans themselves rated the game 8.1, which I personally think is too low, but meh!)

    So for the higher end of the spectrum (in sales) it’s more like Sales affecting Score, and not Score affecting Sales. The “problem zone” was interesting to look at anyway tho, since those are games that “most likely” (not all of them, but vast majority) weren’t awaited that bad, and received scores that (most likely, again) were most likely deserved.

    What I would like to see : Fan/Reader scores (from Metacritic) and sales instead.
    But that gotta be a pain in the ass to do.
    Really interesting article nevertheless!

  10. “which modestly-scored games would you consider gems?” – I am not Shamus, but I will pipe in on this question to complain about the -10% to -20% “niche” penalty, which is the flip side of the 10% to 25% “hype and ads” bonus. The awesomest SRPG in the world can only score about 85% nowadays. This was less true four or five years ago.

    If you’re into niche games, there’s a lot of gems that rate relatively poorly, simply because they’re “niche”, but by the same token they tend to have the character that people around here are looking for.

  11. Primogenitor says:

    While this shows some correlation, that is not causation. My opinion: big corporations (e.g. M$, £A) can pay more marketing, so sell more because more people know about it, and can “pay-off” reviewers (either directly or indirectly by getting it onto the easier review sites).

    Plus, number of units sold will increase with time since release. Review score will not (probably). So newer games will have less linkage than older ones.

    Now, a graph of “average score vs publisher” or “average no units sold vs publisher” would be interesting. And some means with standard errors would be nice too, rather than just a scatter graph.

  12. Viktor says:

    @Chilango

    I’ve generally found the X-Play TV show on G4 to be accurate. Not perfect, but the at least are much more likely to say “It looks pretty, but…” than a lot of other reviewers.

  13. Factoid says:

    Re-running these numbers with aggregated user reviews instead of outlet reviews would be interesting also.

  14. The Lone Duck says:

    Doh! Just realized, in the fury of my rant, I strayed from the point of the article. Quality does not equate sales; at least not in the short term. To continue my movie analogy, Kagemusha, a Kurosawa film, is much better than say, Scary Movie. But Scary Movie sold more. Part of it is the LCD (Lowest Common Denominator), that the majority of people don’t have an eye for quality, or the desire to appreciate quality. I hate saying that, because it sounds so elitist, but that’s what the circumstances seem to indicate. Now that is short term. I’m no film historian, but I think it’s safe to say that a movie like The Seventh Seal has sold more over time than an equivalent crappy movie of that time. So quality does equate to sales, but only in the long term. Look at literature; the pulp fiction of Victorian England doesn’t sell now, unless you’re a scholar. (I already talked about quality via review score; now I’m talking about the objective quality that retrospect allows.)
    This metaphor fails for videogames in two ways. Number one, the media changes are more drastic for games than movies. Transfering film to DVD is easier than emulating, let’s say, a Sega Saturn game for the PC. Emulation, hardware, these things make it harder for clssics to avoid sinking into anonymity. Secondly, vidogame businesses are more fragile than movie businesses. Sony, Warner Brothers, MGM, they are large companies that still have access to the original Masters of many old films. Videogames are highly idependent, and it’s quite often that original code is lost.
    Perhaps with that in mind, it’s better for the long run to have large corporate videogame companies like EA. If a stable company can create a secure archive, than classic games can continue to be remade. The conception of “classic” is necessary, for games to transcend elaborate toy status.
    I’d like to see a quality versus sales chart for Playstation 1 games, a long-term analysis. By now, the “classics” for that system have been somewhat identified. I’m curious what the sales for that reflect now that time has passed.

  15. unitled says:

    Didn’t you post a link to this article ages ago? Or did you just post a link to the blog? Whichever, I read it a little while ago and it’s definitely a good read.

    I tend to use review sites like GS and the like to give me a rough idea of what the game is like; the main problem I have with them is that the reviewers tend to not like the kind of games I like… The very good at giving the main good and bad points about a game, then I can go look at gameplay videos, read forums (dodging soilers) and make a decision on whether to buy it. Then, when I buy it, I can feed back into that structure and help spread the word about particularly good (or bad) games.

    I found Arcanum to be okay… The setting had a lot more promise than actually came through in the final product. The combat was abysmal, and it just turned into a combat slog in many places. I much prefer Fallout/Fallout 2 any day of the week; I’m currently playing through FO2 again so I can work myself up to be disappointed by Fallout 3.

  16. Using semilog (log on the x-axis) would probably have made the plot much easier to see, considering that the differences in sales were orders of magnitude different rather than something linear.

    So yes, we need access to the data to correct this. :-P The comments indicate that he is not allowed to share the data, though. Maybe it can be scraped from various websites.

  17. MintSkittle says:

    In the super zoomed graph, there are about a dozen games that scored 90+, yet sold less than 300k units. I wanna know what games those were.

  18. LintMan says:

    With reviews, there is something I call the “Black & White Effect”, named after the fist game where I really noticed the phenomenon.

    Basically, Black & White was a heavily hyped and extremely anticipated game with all sorts of fancy and unique new features. But unfortunately it was lacking in the gameplay/fun dept. (sound like a familiar description?)

    When the game finally came out, reviewers and fans alike were saying “wow! you gotta check this out!”, and the 95% scores rolled in. Then like a week later, no one was playing it anymore: it was all flash and no substance.

    By the time B&W2 was being rolled out, I had already seen a few rueful editorial remarks in the gaming mags like “what were we thinking?” regarding their B&W ratings.

    I had fully expected this exact same thing to happen with Spore, but the DRM controversy has overshadowed any debate over actual gameplay there. But read PC Gamer’s review of Spore to see a reviewer desperately grasping for straws to justify her 91% “Editor’s Choice” score.

    So anyway, yeah, I don’t think reviews necessarily reflect quality either, at least not as far as the big budget/big publisher games go.

  19. I scraped the Metacritic PS2 scores into a nice CSV file (available at that link). Now I have to find and scrape some sales figures and we got it. However, sales figures seems to be the difficult-to-obtain information.

  20. And I scraped some sales figures for various platforms in this sales.csv file. Now they just need to be put together and we have something to go by.

  21. Carra says:

    # A huge number of PS2 games games (about 45%) failed to ship more than 100,000 units.

    Ugh, so half of all ps2 games released are loosing money.

    And it’s great to see that good games sell:
    http://www.dreamdawn.com/sh/features/img/sales_vs_score/average-units-by-score.gif

  22. Mari says:

    I guess you could say I rely on video game reviews when selecting titles, but not in the traditional way. Numeric scores mean nothing to me. I read things like your dissections and other “amateur” reviews that actually dig into the meat of what they liked and didn’t like about a game. From that, I can get a vaguely accurate idea of what the game is like and weigh that against my own desires in a game.

    Your “yay” or “nay” means nothing, it’s the depth to which you treat the subject that helps me. Let’s face it, what makes a game “good” or not is mostly subjective. There are some objective measures like “broken game/does not run/requires walk through to avoid breaking the game/comes loaded with DRM” but mostly it’s about a game meeting your own play style.

  23. Barron says:

    The problem is that there is no single “quality” metric. You can’t say “this game is 93.7% awesome, that game is 39.33% awesome” There are games, like The Sims, or Spore, or WoW, which are designed to appeal to a huge number of people. Other games cater to a specific group of people. Look at Eve Online. Obviously it has fewer players than WoW, but to its own playerbase, it is the better game. Similarly, Dwarf Fortress is one of my favorite games of all time, but I’m not about to go recommending it to everyone I meet because the vast majority of the people who play it will never have the patience to figure out what is going on.

    I think the best we can do is find reviewers we can trust (for example, reviewers who post a review based on the WHOLE game, not the first 20 minutes of the demo) and just be happy when the games we like sell well.

  24. Eric says:

    Of all the reviewers out there, I trust Electronic Gaming Monthly. They in my opinion are the most unbiased, and fair. They got Denis Dyack’s Too Human review first as an exclusive, and gave it a c-. They tell you the negatives first, then if there are any postive aspects they tell you.

    *End Transmission*

  25. Chris says:

    Hey Shamus, thanks for linking to me twice in the space of a week! I am glad that you enjoy my blog–seeing you and your readers comments is very motivating for me.

    About the data:
    – Yeah, I need to do a proper regression analysis on this data. When I wrote this article I was interested in settling a very basic debate that was going on in the game industry, namely “is it worth it to pay extra money to make a game good?” I did enough research to show that there was some sort of correlation, but once I had the data all excel’d out I realized that the more interesting conclusion is that most PS2 games failed (or at least, would have failed if they cost what games cost today for development).
    – This data comes from two sources: Metacritic (which I scraped and is basically publicly available) and NPD reports (which are not publicly available). I can’t release the raw data for the NPD stuff (and I was also careful not to put numbers next to the names of specific games), sorry. There is some publicly available info (like http://www.vgchartz.com and this article: http://www.edge-online.com/features/the-top-100-games-21st-century ), but it’s somewhat untrustworthy or limited (the top 100 games are, for the most part, all awesome, so you don’t learn much about the actual relationship of quality to sales).
    – It’s true that review scores are not a very good metric of quality. I ranted about this very topic on a post which you very kindly linked to (the one about games as consumer products rather than disposable media). I think the best way to assess the quality of a game is to find a reviewer with which your personal tastes are aligned and listen to what he or she has to say. That said, I still thought that there was value in this research because a) metacritic is an aggregate score of sites that have been vetted for “reliability” (not to say that it’s good, but hopefully aggregation removes some of the bias from individual reviews), and b) because publishers look at Metacritic and GameRankings.com as metrics of quality and make development decisions based on those numbers (at my last job, bonuses were based on getting particular metacritic scores). So even if the individual scores are totally garbage (I think that they need to be taken with a grain of salt but are, as a whole, not bad), I thought that the analysis was interesting because it gives us some insight into the ways that publishers make decisions.

    A lot of your readers have posted asking for further analysis. Maybe I can dredge up the old .xls and see if I can get some more information out of it. The market is changing rapidly and the conclusions may be getting weaker in their applicability to today’s games, but I’m very interested in this stuff and would like to work on it some more if I can find the time.

  26. Lanthanide says:

    Some people here have been asking for a good game review site. About the only site I trust any more is bit-tech.net. They’re about 1/2 hardware 1/2 gaming, and they don’t review every game under the sun, but they do generally review the big names. They also review quite a few indie games. They had an article about payola a few months ago where they explicitly said they would not take part in it (although they do have site-wide advertising on occassion, such as for Spore and Mass Effect as well as general nVidia advertising).

    They also have an interesting system of only giving full scores for games because they figure that half-scores don’t actually make sense. Their rating scale is such that a game rated 7 is recommended for everyone to play, and a 6 is worth it only for those interested. Their break-even point is 6.5, which is an impossible rating on their site because they don’t give half-scores. They recently gave the Stalker expansion (Clear Skies or something?) a 2 out of 10 because the game is so cripplingly buggy.

  27. KarmaDoor says:

    I think what would be truly interesting is to see the sales versus scores in individual categories (graphics, sound, gameplay, etc.) I’m a bit confused if those are already in the compiled data, and if not that would seem like a mind-numbing task.

    As to trusting reviews, I only read them and generally don’t use the score except to scrape off the lower end when searching through lists. (Anything below 50% is highly unlikely to be playable, nevermind enjoyable.) I’ve read reviews and then looked at scores for categories and at times have found massive discrepancies between the them. Metacritic has never jived with my preferences and I have yet to find a magazine worth subscribing to. (GMR was fairly good, though, as a “bonus” to EB’s Gamer’s Edge card back in the day.)

  28. Poet says:

    I just wanted to make Shamus aware: I’ve turned the internet into a drinking game involving tequila.
    Furry porn is worth 1 shot.
    Lolcats are worth 1 shot.
    People trying to explain things they obviously don’t understand are 2 shots.
    Bananaphone is 2 shots.
    Misuse or misunderstanding of internet shorthand (lol, roflcopters, ftw, etc.) is 3 shots.
    DRM complaints are 4 shots.

    I should also point out that, at the moment, I can’t feel my tongue. You may have given me alcohol poisoning.

  29. Patrick the Irate says:

    What I found interesting was that 9 of the 11 top selling games were all part of a series, and not a one time release title. Standout sequels like GTA succeed on there relative ingenuity and innovative gameplay concepts. Gran Turismo is a niche game for racing fans and rarely reaches the masses of gamers, but it is one of the most technically sound, and the real-life physics is amazing. Madden is, well it defines the phrase “easy to learn, impossible to master”. The game is as simple or complex as you want it to be, and has almost infinite replay-ability. Point is, none of the above games have cutting edge graphics that cost the most during development, and are is on the usual suspect list of any bug. They succeed with creativity and ATTENTION TO DETAIL. They succeed with good writing, dedication to fun and GAME creation by people other than programmers, marketers and suit-and-tie corporate ass-eaters. 9 out of the top 11 for 3 series titles, interesting…….

  30. Patrick the Irate says:

    When are you gonna do a strip of NetHack anyways? Don’t make me come over there….

  31. Arkmagius says:

    The way to find a great review site: look at the scores for a highly rated, average quality game at your favorite scores aggregate. Then, check the sites that gave it a ‘real’ score.

  32. David B says:

    My issue with game reviews is that they come out on day 1, or before. This may be ok for console games, but with any game that involves human interaction, it’s problematic. People play the game longer than 1 day. They play with other people. The game may involve content farther along than the review could get to.

    A day 1 review is like looking at the cover of a book and saying “This is going to be a best-seller.” Or judging a movie by it’s previews.

    The most recent example of this, of course, is Age of Conan. Everyone raved about Tortage – the first area, and then discovered that Tortage was essentially a demo, and that the rest of the game was unfinished. And nobody is going *back* to their reviews to say “Hey, sorry guys, we messed up. Here’s a revised score.”

    I would love to see a site or magazine that doesn’t review games until they’ve been out for at least 3 months. This has the benefit of being able to point out the failures, but also giving problematic games a chance to redeem themselves.

  33. Morzas says:

    reviewers are voracious consumers of games, to the point where they make “hardcore” gamers seem “casual”

    Oh, no they’re not. I remember the lead editor of PC Gamer writing an article on StarCraft 2 and claiming that he got owned by a “pro gamer” when in actuality it was just a guy who’d played a lot of StarCraft. He even went on the say that the game requires no thought, just good reflexes, which is complete nonsense.

  34. Kevin says:

    A reviewer you trust is worth more than all the pee in China.

  35. Matt says:

    What I honestly don’t get is why a fair number of people seem to want (or expect) one reviewer or games site to provide the perfect review for them. Admittedly, Gamespot, IGN, Gamespy and the like are all very mainstream and often swayed by big names, but read or at least browse all of them and you already have a fairly good idea of what a game is and what it does. And to be honest, I don’t think the reviews are that bad – it’s the overinflated scores that are the major problem.

    Add some user reviews and you’ve got a fairly well rounded picture of whether you’ll like a game or not. (My personal preference at the moment would have to be Eurogamer, though.)

    Going exclusively by user reviews and message board comments may work for some, but I have a massive dislike for how a game (or film/book/comic/artist) is either brilliant or utter crap for most users. Is it just me or are most people stuck in binary where something is either 1 or 0?

  36. Scourge says:

    Remonds me of the game Nightwatch, a fun game, quite hard Turn based strategy wih RPG elements based upon Russian Novels/Movie.

    I’d have given it a 7/10, for the interesting story, the character makup and the abilities your char has.

    The gameplay itself was slow though and the game lagged, although I met the suggested requirements.

    The second game, Daywatch, was worse and I would’ve given it only a 2/10. Why? I met the required system specs, even excelled in some, but it lagged still. The lower I put the settings the worse the lag was, seriously.

    The story also made no sense. it seems the maincharacters, after fighting for their love in the prequeal, seperated for whatever reason.

    The story made no sense at all to me, gameplay didn’t improve, and it crashed too.

    I wrote a review about that game on gamefaqs.com . My review wasn’t published, reason? None specific, i just didn’t ‘meet certain requirements’.

    Tsk..

  37. JKjoker says:

    i think numbers need to be divided by the console install base at the time of release, selling 400k units when there are 500k consoles available is a hell of a lot better than selling 5m copies with 20m consoles available even if the second one made more money

  38. Dix says:

    First, I have no use for reviews written by people who get money to write them. Reading user reviews means I have to sort through some idiots, but I prefer the honest idiot with a chip on his shoulder to the reviewer whose company has been paid off by the game publisher and whose job relies on positive prose.

    I look for long reviews with a ‘reasonable’ score attached. The ‘reasonable’ score tells me the reviewer is a person who considers lots of aspects of the game and found some stuff to like and stuff to dislike and therefore didn’t fall into the binary good/bad trap. The long review tells me they talk about what those things were. Did they hate the game for its graphics? Great: I couldn’t care less about graphics, so I’m golden. Was the gameplay kludgy? Not so great: my dexterity isn’t that good to begin with and I don’t need handicaps.

    This system breaks down when a game is just perfect all the way through (um… well, in theory it breaks down, but even an excellent game will have some reviewers giving it an 8/10, so those are the reviews I look for) or complete crap all the way through. In the latter case the 6/10 and up scores will usually be rare among their 5-and-below brethen – the first sign – and the corresponding reviews begin with excuses: “This isn’t as bad a game as all that. Why, it never even kicked me in the shins and stole my wallet…”

    All of this aside, if I could just watch a video of actual gameplay of every game I was ever to consider buying, it would help me a lot more than any review. It surprises me that more ‘user driven’ game sites aren’t posting these, and I therefore assume (heh) that there’s some pressing copyright reason not to.

  39. B.J. says:

    As much as people would love to claim otherwise, publishers do not dictate review scores, gamers do. Look at Halo 3. I doubt there is a single critic who will put that game on their top ten list of greatest games of all time. Yet it received nines and tens across the board. Did microsoft pay off all the reviewers in the world to give that average game godly scores? Nope, they didn’t have to. The pressure to give Halo 3 high marks came FROM THE FANS. Any website or magazine which failed to give Halo massive praise received a deluge of hate mail from outraged fans.

    This happens all the time. If a big name, super-hyped game gets even 1/10 less than a perfect score, howls of protest ring across the internet. People who scream and rage that Super Smash Brothers Brawl or GTA 4 got ‘only’ a 9.8 on some dumb review site. These are the people skewing scores.

  40. Hank Paulson says:

    My take on the whole Gerstman thing is that he was going to get fired for something else altogether, and used the really, really negative review as a way to cover his ass on the way out.

    I agree that game reviewers are the fans’ bitches. When your target demo are austic, borderline personalities, you can exploit their easily-excitable nature and sense of alienation really easily, but the downside is you’re effectively prevented from doing anything remotely meaningful owing to their inability to accept difference or disagreement.

  41. tussock says:

    That #sales axis really, desperately needs to be a log value, so we can see what’s going on. It’s like an infinite number of zooms all at once.

    Edit: Huzzah! 42.

  42. Tesh says:

    Interesting data. I agree that more people should take a look at it. I think his generalized conclusions are pretty spot-on.

    Since I’ve been blogging about MMOs lately, I wonder if a similar analysis might be useful to take a look at their data.

  43. I think your initial statistical nitpicks are in fact the whole point. Of COURSE GTA IV, Metal Gear Solid IV, etc. sold a lot of copies and were rated well. The two are correlated and would be if every game Metacriticed up to 90% was a flaming pile of garbage. Fans will buy the games, and fans will punish the slightest deviation from 9.0 for their sacred horse franchise. Meanwhile, advertising and marketing people will guarantee that both occur.

    The statistics show that popular franchises can be excellent. They also show that brilliant games can sell poorly. The latter is the ominous part.

  44. Blue_Pie_Ninja says:

    The review show I use that is really great is Good Game. You can tell they are good when they are government funded ;) They try to cover every area in a game they can and they talk about all areas as well, and regularly review indie games if they have the time in each episode. They have a guess the game segment, that’s great as well as a lot of humour. Generally their scores from 3-5 (they use a 5 star system) goes from average to the best, and below that it is subjective. (They used to use a 10 rubber chicken system but changed it)

Thanks for joining the discussion. Be nice, don't post angry, and enjoy yourself. This is supposed to be fun. Your email address will not be published. Required fields are marked*

You can enclose spoilers in <strike> tags like so:
<strike>Darth Vader is Luke's father!</strike>

You can make things italics like this:
Can you imagine having Darth Vader as your <i>father</i>?

You can make things bold like this:
I'm <b>very</b> glad Darth Vader isn't my father.

You can make links like this:
I'm reading about <a href="http://en.wikipedia.org/wiki/Darth_Vader">Darth Vader</a> on Wikipedia!

You can quote someone like this:
Darth Vader said <blockquote>Luke, I am your father.</blockquote>

Leave a Reply to Patrick the Irate Cancel reply

Your email address will not be published.