Bob's Blog

Reviewer bias? When you average 20 different reviews you've got to figure that most of the bias is washed out (though you still have to watch out for systematic bias).

 

It does not wash out "continental" bias though, since most reviews tracked by gamerankings reflect the american point of view.

 

While this is a rather obvious aspect regarding the ratings of say an American Football simulation vs. a European Soccer manager, there also seem to be different tastes I have no explanation for. If you look up Midways "Rise & Fall" in gamerankings for example, you will see it scoring ~65%, when this game did not score less than 80% in any German magazine

 

No problem for you probably, since you actually are american, but for me it means that I can trust gamerankings only in the extremes, i.e. assuming that a score lower than 50% means it sucks and above 90% it can be considered really outstanding, but inbetween I have to consult other sources.

 

BTW: You can find many German Genesis ratings here: www.kultpower.de

Just type the game name in the box on the left frame, then try the results for "Power Play" or "Video Games" Reviews. They did really competent German p.o.v. reviews back in the day, so it might be interesting for you to compare with your averages, or add some European influence

 

BTW: Are you considering gamefaqs reviews?

Thanks for the link. And thanks for the perspective on Europe vs. American reviewer bias; that's what I meant by systematic bias, though I wasn't aware of any specific examples. I think you'll always run into that problem when a reviewer is reviewing a game that has a lot of outside context, like sports games (I would imagine that most baseball games, if they even get released (do they?) in Europe, review somewhat poorly over there.)

 

But I have seen the same issues in reviews of games that are aimed at niche audiences, like games based on relatively obscure cartoon series, etc. And it was interesting to read recently-written reviews of Phantasy Star III vs. an older review from a VG&CE mag from 1991. In the older review, the author didn't care, or probably didn't notice, all the storyline/gameplay/world differences between PSIII and the other Phantasy Star games, which have caused most modern reviewers to call that game the "black sheep" of the Phantasy Star series.

 

But, in any case, it probably is best to just use it for a broad overview. Gaiares, with 14 different reviews, has an average score of 85.00. Thunder Force III, also with 14 different reviews, has an average score of 82.79. Is that difference significant in any way? Absolutely not. But, on the other hand, Sub Terrania, with 13 reviews, has an average score of 67.69. Is the difference between it and Gaiares significant? Absolutely.

 

And you have to take your own bias into account - if you love great music then you'll want to bump up the games with great music, etc. And some reviews are just incomprehensible to me. Like I've mentioned before, Strider gets great marks, I have it in the list above of the top ten games for the Genesis; I wouldn't put it anywhere near that mark, and really, I don't understand how anyone could. But that's just me, I suppose.

 

EDIT: Just saw your question about gamefaqs.com

 

I think there is a single review from gamefaqs.com in there, which I randomly happened to read and thought was a good review, so I added it. But mostly...I dunno. I tried to pull reviews from sites where the reviewer or review team reviewed many games, or where there was some quality control on the reviews, so I didn't get reviews that were some guy, in his 5 minutes of free time spouting off about how teh awesom3!!11 some game from his youth was and giving it straight 10s. Some of those snuck in, I'm sure, but I tried to avoid them.

 

When I get a chance I'll probably go through gamefaqs.com, and mobygames.com, and other similar sites, and try to pull out well-written reviews, but right now I'm going to concentrate on getting SNES games in there.

 

EDIT II: Besides which, the reviews on those sites I don't really need, honestly. Do I really need another 20 reviews for Sonic The Hedgehog? It isn't really going to budge it's position at all. I'd like more reviews for the little-reviewed games, like Aerobiz, Volfied, Rampart, and others, and most of the sports games, and the Japanese/European games. But I haven't seen many of those on gamefaqs/mobygames/etc.

And you have to take your own bias into account - if you love great music then you'll want to bump up the games with great music, etc. And some reviews are just incomprehensible to me. Like I've mentioned before, Strider gets great marks, I have it in the list above of the top ten games for the Genesis; I wouldn't put it anywhere near that mark, and really, I don't understand how anyone could. But that's just me, I suppose.

 

In the end your own bias is what counts the most indeed. Personally I'm wondering for example that Warsong didn't make the Top 10 you posted above, it sure is ten times more fun than all those Sonics. Also shouldn't there rather be Streets of Rage and M.U.S.H.A.?

When I get a chance I'll probably go through gamefaqs.com, and mobygames.com, and other similar sites, and try to pull out well-written reviews, but right now I'm going to concentrate on getting SNES games in there.

 

Mobygames not only has "user" reviews, but very often also old magazine reviews/scores.

Here look, dozens of ratings for Paperboy versions for example:

http://www.mobygames.com/game/paperboy/mobyrank

 

Hm... and your Genesis Rankings say that Paperboy is better than its sequel... good to know

In the end your own bias is what counts the most indeed. Personally I'm wondering for example that Warsong didn't make the Top 10 you posted above, it sure is ten times more fun than all those Sonics. Also shouldn't there rather be Streets of Rage and M.U.S.H.A.?

Agreed; I was pretty surprised, since the Genesis is known these days for having great shooters and great beat-em-ups, that all of the top games were platformers. If you look at the most-reviewed games, like all games that have 10+ reviews, the vast, vast majority of the top games are platformers:

Sonic The Hedgehog 3

Sonic The Hedgehog 2

Shinobi III: Return of the Ninja Master

Sonic The Hedgehog

Strider

Sonic and Knuckles

Gunstar Heroes

Earthworm Jim

Ghouls 'N Ghosts

Super Monaco GP

Flashback

Castle of Illusion Starring Mickey Mouse

Road Rash

Phantasy Star IV

Vectorman

Castlevania Bloodlines

Gaiares

NHL 94

Phantasy Star II

Earthworm Jim 2

 

You've got 9 platformers, a racing game, two more platformers, another racing game, an RPG, two more platformers, a shooter, a hockey game, another RPG, and another platformer. (Streets of Rage just misses this cut; it's the next game on the list.)

 

I think that's just selective memory working against us here. We remember the Genesis for what made it special; all the great shooters, beat-em-ups, it's great RPGs. But, like the NES, the SNES, and other consoles of its time, something like 75% of its library was platformers. We just don't remember them because there were so damn many of them and you don't remember the Genesis for Flashback, which is a fantastic game, because there were a million other games out there like it, not only for the Genny but also for the NES, the SNES, the TG16, etc.

 

And I dunno. The games are grouped very close together here; like I mentioned earlier, there isn't much difference between a game rated 86 (Warsong, M.U.S.H.A.) and a game rated 89 (Strider, Sonic The Hedgehog). But there are 10 games in that range, so it's easy to get pushed off the bottom of top-X lists.

When I get a chance I'll probably go through gamefaqs.com, and mobygames.com, and other similar sites, and try to pull out well-written reviews, but right now I'm going to concentrate on getting SNES games in there.

 

Mobygames not only has "user" reviews, but very often also old magazine reviews/scores.

So does Digital Press. And I've got 'em, from both sources. Actually, I added some reviews from old VG&CE mags I've had laying around for 15 years just this morning.

 

And yes, almost every review of Paperboy 2 I've read has commented that it is not as good as Paperboy.

I think you should average the EGM votes into one, else a single magazine is getting 4 votes per game, no?

I think you should average the EGM votes into one, else a single magazine is getting 4 votes per game, no?

I went back and forth on this and, without being able to see the magazines I just couldn't decide. I eventually went with allowing each vote to stand on its own since I assume that each score is from a different person - and I ended up loosening my standards such that anyone that scores a game (that is not obviously a joke) gets counted.

 

So I started adding gamefaqs user scores.

 

I don't know that it makes much difference in the end, anyway. There were remarkably few outliers.

There were remarkably few outliers.

 

What with the guy that rates Starflight with 7% on his page?

I didn't say there were *no* outliers...

 

And I don't mind, anyway - I'd rather have/read reviews from people who aren't afraid to give effusive praise and also rip into games. It tends to wash out, when you get enough reviews.

And I don't mind, anyway - I'd rather have/read reviews from people who aren't afraid to give effusive praise and also rip into games.

 

Fine, here's my ratings for a couple of randomly picked Genesis titles to add to your database then:

 

0% Cyberball

0% EA Sports Double Header

0% Jerry Glanville's Pigskin Footbrawl

0% Joe Montana Football

0% John Madden Football

0% John Madden Football '92

0% John Madden Football '93

0% Madden NFL '94

0% Madden NFL '95

0% Madden NFL 96

0% Madden NFL 97

0% Madden NFL 98

0% Mutant League Football

0% NCAA Football

0% NFL Sports Talk Football '93 Starring Joe Montana

0% Powerball

75% Starflight

0% Super High Impact

0% Tecmo Super Bowl

0% Tecmo Super Bowl II: Special Edition

0% Tecmo Super Bowl III: Final Edition

0% Troy Aikman NFL Football

0% Unnecessary Roughness '95

 

I'm totally serious on these.

Have you played all those? Or are you just giving them zeroes since you hate sports games?

 

Because I've got no problem adding your pile of zeroes to the database (as long as you don't erase the post so I can link to them) but you've got to have played them.

 

My standards are loose, but not that loose.

Have you played all those? Or are you just giving them zeroes since you hate sports games?

 

I can play the few I'm missing so far tonight if it helps, but I'm pretty sure it won't change the rating much - after all they're all Football games...

Because I've got no problem adding your pile of zeroes to the database (as long as you don't erase the post so I can link to them) but you've got to have played them.

 

My standards are loose, but not that loose.

 

The point I was trying to bring accross is that I'm not competent enough to review a sports game, just like that hyper-action-kid wasn't competent enough to review Starflight.

Because I've got no problem adding your pile of zeroes to the database (as long as you don't erase the post so I can link to them) but you've got to have played them.

 

My standards are loose, but not that loose.

 

The point I was trying to bring accross is that I'm not competent enough to review a sports game, just like that hyper-action-kid wasn't competent enough to review Starflight.

Maybe...but a couple of things:

 

First, I have neither the time nor the inclination to editorially review all the scores. If I had tried to do that, the database would have about a tenth of the reviews and I would have given up trying to actually do this. Plus, it's too subjective - if I had a team of people who would each read every review/score and evaluate it, maybe...

Second, if you hate sports games, why is your opinion invalid? The test of any videogame is, "Is it fun?" If you don't have fun playing videogame baseball, what's wrong with that? Should Bridge be counted among the best 2600 games of all time; it is by all accounts a wonderful game of Bridge - just because almost nobody has any actual *fun* with it...

And third - not to belabor this point too much, but it really doesn't take many reviews before the outlying points are smoothed out. Take Starflight - you gave it a 75% after a thorough, recent play-through of the game. Even with that silly 7%, what is the rating in my db of Starflight? With just 8 reviews, 61%. Not too far off from your opinion.

 

And when I finish adding the reviews from gamefaqs.com, it will drag that rating up to 75%-80%, I think - about 7-8 people reviewed it there and they all gave it a score between 8 and 10.

Actually, you've motivated me - I'll take a break from the slow process of adding all the SNES games and add more Genesis reviews from gamefaqs.com

 

EDIT: Starting with Starflight.

 

EDIT II: New rating for Starflight: 15 reviews, 76%

 

Though, really, I think the 3 (!) 100% gamefaqs reviews are outliers just as much as the 7%...

Though, really, I think the 3 (!) 100% gamefaqs reviews are outliers just as much as the 7%...

 

It's a cult game. It has tons of fanpages out there and there's even a semi-official fanmade "Starflight 3" in the works, backed by the original Starflight designers as consultants. I can see some people being so obsessed with it to rate it 10/10, it sure has the potential. I don't think there can be upward outliers for this game, when even most professional reviewers gave it +85% scores back in the day

Ah, maybe so.

Second, if you hate sports games, why is your opinion invalid? The test of any videogame is, "Is it fun?" If you don't have fun playing videogame baseball, what's wrong with that? Should Bridge be counted among the best 2600 games of all time; it is by all accounts a wonderful game of Bridge - just because almost nobody has any actual *fun* with it...

 

I don't even consider these "things" video games. An accurate computer simulation of something like Football, Baseball, a Helicopter, a Ferrari or a Bridge game, is IMO more in the league of Excel or Word, than what I think of being a video game. Mario Tennis, now that is a video game. But Virtua Tennis - Blargh...

 

Considering the other aspect, I think my opinion is indeed invalid. Someone interested in reading a review of Madden Football, is probably interested in many aspects of this "game", but not wanting to hear that it sucks, "because" it is a Football game, or?