About the Author
Mass Effect
Final Fantasy X
Batman:Arkham City
Borderlands Series
Weekly Column
Champions Online
World of Warcraft
DM of the Rings
Good Robot
Project Frontier

Rating Movie Critics

By Shamus
on Monday Jan 29, 2007
Filed under:


Mark has an interesting post which looks at the various “Best Movies of 2006” lists from several critics, and then runs the numbers for each critic to see how much money “their” movies made or lost. It’s an interesting read.

I used to labor under the misconception that movie critics were somehow supposed to predict if I would like a movie or not. I guess the fact that such a task is obviously impossible should have tipped me off. It really confused me when they failed to point out that I would find Independance Day to be loud, stupid, and brimming with nonsense. I mean, if you can’t predict that I’ll hate ID4, then your mind-reading skills are highly suspect in my book. That one is just so easy.

Of course, I’ve since realized that I was missing the point. Movie reviews are entertainment. It doesn’t matter if you think Gili was a masterwork or that Citizen Kane is a hack job, as long as you keep the reader amused, you’re a good critic.

However, if we are going to judge critics on their ability to predict if a movie will be generally liked or not, then it would be interesting if you could harvest the data from something like Rotten Tomatoes and see who has the best track record for predicting “good” movies. That is, who has the fewest “bad” calls, with a bad call being defined as giving “thumbs up” on movies that scored low on the tomatometer or “thumbs down” for movies that scored well. (Middle-of-the-road movies in the 45% – 59% range would be ignored.) I wonder who would have the best hit percentage?

Note that this would be different from what Mark has done, since some movie budgets are so low that they will make money no matter how much people hate them. The inverse can be true as well.

Sadly, I don’t think there is any good (that is, easy and cheap) way to obtain that data.

Comments (18)

  1. Phlux says:

    I would love to do this same thing with game reviewers, except the data is often much more difficult to come by. Many game companies are privately owned, many are outside the US, etc… The sales figures and budgets are often not disclosed. I heard a rumor once that the budget for GTA: San Andreas was something like 55 million dollars. I’ve no doubt it more than made it’s money back, as it would only have to sell a little over a million units to break even, and it probably sold more than double that just domestically.

    Anyone know a good place to find any info like that on how much games cost, versus how many are sold, versus how well they are liked by reviewers? The last one is easily obtained at a place like metacritic, but the others are more elusive to me.

  2. Mark says:

    I’d love to see such an analysis for Games, but I can’t think of any sort of definitive resource for games’ meta data. For some reason, movies have very good informational resources on the web (IMDB and Box Office Mojo are invaluable, and Rotten Tomatoes and Metacritic are useful as well), so data crunching is a lot easier for movies. I’m no expert, but I’d like to try this again sometime in a more automated way.

  3. Mark says:

    Oh, and as far as the critics-as-entertainment viewpoint goes, it’s certainly an interesting way to look at it. I’ve always felt that reviews say a lot about the reviewer, and that if you find a consistent reviewer, you can use them as a semi-accurate guage for how much you’d like a movie. Note that this doesn’t mean the critic has to have the same taste as you, you just need to recognize where you differ from the critic and adjust your response appropriately. For instance, if someone wrote that “This was the best movie since Independance Day,” you’d know to avoid that movie, even though that clown liked it (this is, of course, an extreme example).

  4. Brett says:

    The problem with a lot of movie reviews is that they focus on telling you what the reviewer thought of the movie. If you happen to find a reviewer who thinks the same way, and likes the same things, as you, then you’re set. Otherwise, not so much.

    What they ought to do is:
    1) Identify the target audience of the movie.
    2) Tell you how well the movie succeeds at entertaining their target audience.

    Lord of the Rings, for example, was tailored to nerds. It was very well tailored, but if you didn’t like that sort of thing at all, you wouldn’t have liked the movie.

  5. Stiehle says:

    Ah man… I really LIKE Independence Day. Sure it’s chock full of holes in the plot and the leaps in logic are somewhat dizzying, but it was a pretty cool action movie, in my book. And could they have gotten anyone better than Judd Hirsch to play Jeff Goldblum’s dad? Good Lord the really DID look like father and son to me!

    And yeah… I’m a Wil Smith fan. There, I’ve gotten that off my chest. =)

  6. Lanthanide says:

    Given Hollywood Accounting (http://en.wikipedia.org/wiki/Hollywood_accounting), that exercise is basically completely pointless, as all the different movies could easily be employing different strategies of distributing their money for both costs and profits, so a film that appears to have made a loss may really have made a much bigger profit that another one that looked like it had a fair profit.

  7. Robert says:

    Like Spider Robinson, I’d draw a distinction between a critic and a reviewer. A critic analyzes the work (be it film, book, or whatever) for artistic themes and so on; a reviewer tells me enough so I know whether I will like it or not.

    The problem with our local paper is that the film reviewer seems to think she’s a film critic: too much analysis, not enough information to decide whether the movie will be one I like. Either that, or she just takes the capsule ‘reviews’ written by the studio…

  8. Richard Dragonbane says:

    Man, I LOVE reading twenty-sided tale! It’s the best entertainment since Independence Day! :)

  9. Deoxy says:

    I long ago found that critics were a less bad judge is I simply reverd what thy said (movie sucks=good movie, etc). Still not great, but vry noticably less bad.

    Then I moved on to what they say about the movies. While I haven’t gone to the movies (or read reviews) enough in the last 3-5 yaers to really have the exact verbiage on the tip of my rongu anymore, it was (then) quite easy to look at what they said was good or bad about the movie and judge for myself if that made it good or bad.

    Basically, I find most movie critics/reviewers to be snobbish morons, and when they say a movie sucks, it makes me want to go see it.

    Oh, and Independence Day was a rather enjoyable action movie, IMO. Of course, by “action movie” I mean “a movie with huge plot holes and leaps of logic that you watch because it’s fun to watch things get blown up, there will likely be some enjoyable lines, and there will be a happy ending.” When you ASSUME that they won’t be able to be consistent with themselves and you only watch for the above reasons, a lot of movies become “enjoyable” – not “good”, “deep”, “life-changing”, etc, but worth relaxing to for an hour or two upon occasion.

    And Will Smith is generally a moderately good action movie actor. His movies generally deliver just to the definition above.

  10. Shamus says:

    I do like Will Smith. I also like seeing stuff get blown up, and I like witty banter.

    What I can’t stand is just the sheer ineptness of many alleged plots. Every time they do something that doesn’t make sense, I am yanked out of the movie. If a “surprise” is obvious a mile off (oh man, this scientist is so dead in the next five minutes!) then I spend the next five minutes just waiting for them to get it out of their system and move on.

    What boggles my mind is when a movie has a budget near $100m, where they spend millions on special effects and actors and huge sets and explosions, and then they use a script that could have been much better with just a couple of hours of work. It just makes no sense to do things poorly when it would take just a tiny ammount of effort to do them right.

    I’ve noticed that most people who like movies I hate are ones that can surf over plot holes without getting all worked up about it. I guess this is something eccentric about me: I need movies to make some kind of sense.

  11. Shamus says:

    Getting back to ID4 – I liked the first half of the movie. I liked almost everything with Will “I’m carrying this movie myself” Smith. The parts with Pesident Wonder Boy made me want to gnaw my own leg off.

  12. Zack says:

    Shamus I am with you. ID4 was the most banal piece of crap I have ever seen. I laughed when I heard the studio couldn’t figure out why it did poorly overseas. Here is a hint, “Foreigners aren’t American and get offended when a movie says the world NEED Americans to lead them.”

    The “president” in the movie inspired an almost visceral knee-jerk reaction from me. I was hoping the aliens would vaporize the podium during his speech. Why spend ten minutes on that claptrap? It was nationalistic drivel from any empire and most of the fools watching thought was motivational. *sigh* Well they seemed to like it enough to enjoy it from the talking heads these days.

    Will Smith did a decent job, but the plot leaps were leaps through hyperspace. Who would have guessed that the alien spacefleet was susceptible to Apple share viruses? Or that an alien fleet had so little grasp of strategy and no reinforcements. *takes pill*

    Ok, I am calm now. Sorry about that. Lets just say I don’t like that movie and move on.

  13. Zack says:

    To come up with good reviews you really need a system that take into account both your high scores and low scores and maps correlations vs other users in the review database.

    Amazon com used to have a pretty good system. You put in high scores for China Melville, Steven Erikson, and Sean Russel but pan JK Rowlings and another poor or “young-teen” fantasy writer and you get very grown up fantasy authors rather than the latest Underdark book. If you only review books others like or that are very popular the system doesn’t optimize nearly as well, but once you get into some obscure reviews you get correlations to all sorts of new materials.

    The same thing works well for movies. You have to use less well known movies sometimes but then you find tons you never heard of before. Meanwhile netflix review system would see you liked Firefly and then recommend other westerns… And despite ranking hundreds of westerns poorly it would still recommends others because you had one movie in the Genre. (It may have improved recently because it finally stopped, I think)

    Generally recommendations based of other individuals reviews correlate MUCH better than professional reviewers. Also lists made with a few obscure movies/books you really liked sometimes are treasure troves for new authors or directors you never heard of.

  14. Shamus says:

    The same thing works well for movies. You have to use less well known movies sometimes but then you find tons you never heard of before. Meanwhile netflix review system would see you liked Firefly and then recommend other westerns… And despite ranking hundreds of westerns poorly it would still recommends others because you had one movie in the Genre. (It may have improved recently because it finally stopped, I think)

    Yeeesssss. Hates it forever. I once told the thing I liked Alien, so now it thinks I want to watch nothing but teen slasher flicks. No matter how many horror titles I reject, it is still happy to suggest that I might like to see “Fleshripper Bonesaw Sorority IX: The Spleen-suckers” or whatever vomit-inducing splatterfest they have gathering dust on the shelves right now.

    Now that I also said I liked Silent Hill, I’m sure I’m doomed.

  15. Zack says:

    You can do what I do and actually use amazon ratings to guide your choices for netflix. It may be a pain to set up enough ratings, but I have consistently found Amazon is a lot more clueful when it comes to statistical analysis.

  16. Deoxy says:

    “Here is a hint, “Foreigners aren't American and get offended when a movie says the world NEED Americans to lead them.””

    Here is a hint: it is essentially true. NATO: “Needs Americans To Operate” is a well known (and generally true) statement.

    One of these days, America is going to get tired of being both the whipping boy AND the heavy lifter in the world, and one of them will stop. Since we don’t generally like killing people just for insulting us, it will probably be the heavy lifting.

    Imagine if we stopped just ONE of these:
    -disaster relief (yeah, the UN helped SO MUCH with that whole tsunami thing)
    -foreign aid (there goes the entire national budget of most countries in Africa, among other things)
    -policing the seas (piracy is low in the world primarily because of American sea power)
    -military intervention (if America doesn’t do it, it almost never gets done – look at the Sudan right now for just one example among many)

    America is FAR FAR FAR from perfect, but that doesn’t mean the world isn’t a much better place for America being involved.

  17. Evil Otto says:

    “No matter how many horror titles I reject, it is still happy to suggest that I might like to see “Fleshripper Bonesaw Sorority IX: The Spleen-suckers””

    That one wasn’t as good as “Fleshripper Bonesaw Sorority VII: Panty Slaughter.”

  18. What I would like would be a site that asked you to rate movies and them matched you with reviewers whose ratings best matched your own. I’d pay for a site like that. It wouldn’t even have to be pro reviewers. The site could just match us up with other users like ourselves. Sometimes they’d see a movie first, and we’d benefit from their judgment, and sometimes we’d see it first, and they’d benefit from ours.

Leave a Reply

Comments are moderated and may not be posted immediately. Required fields are marked *


Thanks for joining the discussion. Be nice, don't post angry, and enjoy yourself. This is supposed to be fun.

You can enclose spoilers in <strike> tags like so:
<strike>Darth Vader is Luke's father!</strike>

You can make things italics like this:
Can you imagine having Darth Vader as your <i>father</i>?

You can make things bold like this:
I'm <b>very</b> glad Darth Vader isn't my father.

You can make links like this:
I'm reading about <a href="http://en.wikipedia.org/wiki/Darth_Vader">Darth Vader</a> on Wikipedia!

You can quote someone like this:
Darth Vader said <blockquote>Luke, I am your father.</blockquote>