How exactly does [H]ard perform video card reviews/benchmarks?

[ShiShKaBoB]

So I am trying to find out [H]ard's method of testing/reviewing/benching video cards. If there is an editorial somewhere which is detailed please link, and I will close this thread.

I want to run some realworld benchmarks with some of my hardware and I've been using Fraps to record the data - I could be wrong but I believe [H]ard does this as well. However what I am really trying to find out is how to the replicate the exact same movements, and scenes, is there some sort of recording, or macro software which is used in conjuction with Fraps? Thanks any tips would be great. Im trying to compare differnet ATI and PhysX cards in Metro 2033 atm.

 

[Sycraft]

As far as I know they just play through the game in the same general fashion and log the framerate. There is no effort made to make sure it is point for point the same as that isn't possible and is beside the point. The idea isn't "Do we have two perfectly identical tests for FPS comparisons down to 4 significant digits." The idea is rather to see what the maximum playable settings are. That means that someone plays it, messes with the settings until they feel it is playable, and that is logged.

It's a different testing methodology and many people don't like it since the results are perfectly point for point replicatable, but I find it nicely replicates real world experience. That is to say if they claim X game is playable at Y settings on Z card, it is for me as well.

Their tests won't give you a precise "This card is X.X% faster than this other card," type of result. It gives you an idea of how they compare more generally.

 

[Nuby1Canuby]

They discuss their methods in the Benchmarking the Benchmarks article here:
http://www.hardocp.com/article/2008/02/11/benchmarking_benchmarks/

In the game or hardware performance article, they usually state what area of the level they are playing and what settings were used.


As an example in the PowerColor PCS+ HD5870 & HD5850 Video Card Review for BFBC2:
http://www.hardocp.com/article/2010/03/15/powercolor_pcs_hd5870_hd5850_video_card_review/3
"Using the Steam downloaded version of Battlefield: Bad Company 2 we played through the entire single player campaign. Finding that overall the "Heart of Darkness" campaign to be the most demanding, we used the first 10 minutes for our run-through. The run-through started out calmly with us strolling through a small village but soon enough we were ambushed and had to defend ourselves against automatic weapons, RPGs, and even a patrol boat. After crushing the ambushers we then took our own boat ride with dense forest all around we were eventually forced to fight on foot through the river. We then came upon another small village but this time we had the high ground and let our 50 cal sniper rifle take out the enemy from distance."

 

[ShiShKaBoB]

Okay cool, I was not sure if they did replicate the exact moments, movements and scenes, it sounds like they just play through. I thoroughly appreciated [H]ard's method of testing as I see the benefit of finding out best game play experience. I pretty much am doing the same thing, just was not sure if there was some better way of doing it. Thanks guys.

 

[Brent_Justice]

We don't benchmark video cards, we use them to play games and relate that gaming experience to you. Nothing better than evaluating a video card by using it to do exactly what it was intended for, playing games. Therefore, we play games, just like you do. No timedemos, no scripts, just real-world get-in-the-game gaming. Since people are still hung up on statistics we do fraps our run-throughs, to show you min/max/avg framerates, but these never determine the playability of the game, they are simply there to backup our conclusions.

Through practice, and repeatability, we get quite good at doing the run-throughs and being as "similar" each run-through as possible. If we see any wild variations, we do go back and do them again until the run-throughs match up as close as we can get them. Also note that the run-throughs do not determine playability either. We evaluate the entire game, literally playing through the whole game, on each video card, to find weak spots, and what graphics settings are playable with each video card. The run-through is just one little snipet of gameplay we can record, and we look hard for the most graphically challenging levels to use for the fraps data.

In a perfect world, we wouldn't need graphed data at all, only a table showing the highest playable settings, and commentary as to how the game played, what the weak spots are, the good, the bad, and how each video card delivered the gameplay experience. I don't think we are quite there yet with this understanding though, and so the numbers are there to look at if you wish, but they in no way shape or form determine the highest playable settings we find. You can't just look at one level in a game and determine how well video cards compare, you have to look at it as a whole.

It is this level of intensity, and resources, that other websites simply cannot duplicate. We have seen others try, and they've reverted back to doing timedemos, simply cause it is too much work. Well, we aren't that lazy, we aim to do it right, and that means playing the game just like gamers do, cause I'm a gamer too. When I get a new game the first thing I do is determine what resolution/aa/af/in-game settings I can enable, and I usually want them maxed out, but can't always. So that's exactly what we do in our evaluations, find out the best IQ possible for the performance, and which cards allows a better experience.

Hope that helps a bit.

 

[heatlesssun]

Thanks Brent! Nice to see your enthusiasm for your work and a good explanation of how you do it.

 

[ShiShKaBoB]

^^ Beautifully said, and I couldn't agree more, definitely brings a lot of clarity and sound reasoning. I've been reading the "evaluations" here for the last 10 years, and had it not been for the [H]ardOCP staff and editors, who knows where my career would have ended up, or what POS PC I would be using. I can't remember what the first review, or editorial I read here at [H]ard but I know it sparked an awaiting inferno within me. I can't thank you enough for your efforts. Always look forward to the next evaluation.

I like this "Moment of Clarity"

I was invited out to talk about the latest and greatest video card of the day on live TV, which was the NVIDIA GeForce FX 5800. Obviously, live TV is something you want to feel as though you are prepared for. We had tested the FX 5800 for days and days before I was to leave.

A couple of days before I was to be in San Francisco someone asked me, “How did you like gaming on the card?” I had no idea. I had never actually used the video card to play games, I had only "benchmarked" it using traditional methods.

 

[Eriksrocks]

If you're not using recorded actions then how is this so damn exact?

(From here)
I'm kinda confused.

 

[Brent_Justice]

Play the level and find out yourself.

We've done it so many times we're just that good. It is a very consistent level that we can repeat very well, and happens to be very graphically demanding. Level is "From Hells Heart Crysis" It's the same level we've been using since Warhead came out.