ATI Radeon HD 3870 X2 @ [H]

meaning no offense to many review sites, but I got sick of the graphs and charts and long ago.
[H] stood out to me because it wasnt just the absolute fastest the game could run, it was how fast it could run at acceptable quality and consistency.

Many times I looked at some chart, picked the card at the top and found out that I could run the game at some speed, but not play it.

Originally Posted by noseguard20

here ya go http://enthusiast.hardocp.com/news.h...VzaWFzdCwsLDE=

That is incorrect of Steve to say that and it has been fixed. Steve is our "news guy" and have nothing to do with editorial beyond the news page now days. Sorry about the misstatement.

Take it to PM - Kyle

Originally Posted by 67shelby

If you play through a level with FRAPS running and generate those numbers, fine, but how do you do that consistently? You cannot, as the user or game engine will never generate the same results twice. Therefore, this testing methodology is not accurate and is nothing more than an estimation of performance within a single gameplay setting. If we hang our results on card A being 2FPS faster than card B with this method, then it is useless as the results will vary too much to generate meaningful numbers.

Who is to say the levels they play through are the right ones? Do they test multiplayer? I would just like to see the "actual" gameplay sequences they are using to report the numbers. Can I recreate that gameplay to see how well my system performs or do I continue to count on my seats of the pants feeling about what my system is doing or blindly trust the results. One good thing about built-in benchmarks is the ability to get 90% of the way there when it comes to setting up a system quickly or having fun comparing numbers against the systems of my friends, which usually results in a credit card being used a few minutes afterwards.

Please stop ignoring the methods and reasoning we have specifically told you. You are basing your argument on what YOU THINK is going on instead of listening to what the authors TELL YOU exactly the process is.

Originally Posted by gljvd

You don't make sense though.

I would agree with you had a way to get the same run through each time. However you don't.

I could understand using a time demo that you guys created to run through a section of the game. That way it would be the same each time. However since you don't your using two diffrent sets of data to come to a flawed conclusion.

I'm still debating whether your being willfully ignorant here or you just can't see the mean average of statistics.

When you poll a group of people, you choose from a variety of demographics to get an average. If you poll a 100 golf pros at the local country club about which sport is the best I'm sure it will differ greatly then if you were to ask 100 people down at the skate park. Real world game testing while not constant does simulate day to day situations that most games find themselves in. Synthetic benchmarks while though constant and show off the hardware capabilities does not provide the real world gameplay experiance you experiance when you sit down to play, you can at least agree on this correct?

There are pros and cons to each test, I do however believe that the methodology behind real world testing to be better then the synthetic benchmarks and the reviews here at hard generally line up with what we see as evidence down the road. For the record, I'm sporting in my three rigs a 9700 PRO, X800XT, and a 1950XT so I'm far from being an NVIDIA fan boy.

I go with the best price/performance offering at the time I need to upgrade. I'm currently building a new system and the video card/mobo were the only two things I had not yet purchased. Based on this review I decided to buy the 3870 x2 because [H] showed that the price performance was there and it was a solidly made card.

Just because the review doesn't show it knocking the crap out of Nvidia doesn't make it a bad card, the simple fact that you can't let this go screams ulterior motive to me. I held off upgrading to a 2900 a while back because of [H] review of it, Anand Tech almost swayed me , but I held off because of the review methodolgy and to be honest I'm glad I did.

Nothing inherently wrong with the way AT does their reviews, I just prefer real world benches over synthetics, I prefer that in pretty much everything I can think of.

Originally Posted by Kyle_Bennett

But here is the rub, we DO NOT base our conclusions on the graphing you see of frames per second. We come to our conclusions based on real gameplay. We simply add the run through in as our proof to back up what we found while playing the game. The graphed run-through is simply to give our readers a look at what we are seeing while playing the game. Obviously it is also a tool to reinforce our findings as well, but graphed run-through is more for our readers than for us to form opinions with.

And we did discuss not doing those graphed run-throughs either, but I still think that is a bit more than the community could handle. Although I did think we would be beyond that five years later. The simple fact that we still think the "3dfx style" of benchmarking is simply a matter of economics. Sites just do not want to pay the costs of doing it right.

Thanks for the reply Kyle. I do not think you actually answered my question. I fully understand you play through the game, you then decide to take a particular level or section of the game that best represents overall gameplay, setup the system, (insert my question below here), you chart that data, report it for those of us that like numbers, and then come to an objective/subjective conclusion based on how well the game played utilizing a variety of settings (resolutions, AA/AF, etc). That is all fine and dandy and the experiences commented on during actual gameplay is wonderful in my opinion.

However, my question is how do you capture the performance data to create those charts and FPS results? Do you create timedemos while playing the game, then replay them with the game engine while FRAPS is running to collect the performance data or do you (your staff) play through a particular level with FRAPS running and collect that data for reporting the numbers. If it is the latter, how do you ensure consistency between test runs, what variability do you see between each run, and are your FPS results then averaged for a composite score? If you use timedemos, then how are yours different than other sites that use the same testing methodology with FRAPS or the game engine to report results. I think answering these questions would clear up a lot of what has been said today.

Thanks for the review Brent and Kyle.

Some of the canned benchmarks have as much physics and action as the rendering of a floor. Especially Crysis GPU Benchmark and that CoD4 intro. They could at least use the Assault benchmark in Crysis. I still believe real gameplay FPS recording is better, but I don't know who ever decided the GPU Benchmark gave a better indication of performance than the Assault Benchmark.

I honestly am a bit worried about ATi card performance with AA enabled possibly not being as good as it should be. I guess we can only hope for some improvements in drivers.

Originally Posted by 67shelby

Thanks for the reply Kyle. I do not think you actually answered my question. I fully understand you play through the game, you then decide to take a particular level or section of the game that best represents overall gameplay, setup the system, (insert my question below here), you chart that data, report it for those of us that like numbers, and then come to an objective/subjective conclusion based on how well the game played utilizing a variety of settings (resolutions, AA/AF, etc). That is all fine and dandy and the experiences commented on during actual gameplay is wonderful in my opinion.

However, my question is how do you capture the performance data to create those charts and FPS results? Do you create timedemos while playing the game, then replay them with the game engine while FRAPS is running to collect the performance data or do you (your staff) play through a particular level with FRAPS running and collect that data for reporting the numbers. If it is the latter, how do you ensure consistency between test runs, what variability do you see between each run, and are your FPS results then averaged for a composite score? If you use timedemos, then how are yours different than other sites that use the same testing methodology with FRAPS or the game engine to report results. I think answering these questions would clear up a lot of what has been said today.

NO YOU HAVE IT WRONG.

I fully understand you play through the game, you then decide to take a particular level or section of the game that best represents overall gameplay, setup the system, (insert my question below here), you chart that data, report it for those of us that like numbers, and then come to an objective/subjective conclusion based on how well the game played utilizing a variety of settings (resolutions, AA/AF, etc).

We play through the game, make notes of our experiences at specific settings then draw our conclusions. The graphed data you see has nothing to do with our conclusions. It is simply there as proof for our readers as to our experiences.

Now that I have said this for the umpteenth time, what are your questions?

If it is the latter, how do you ensure consistency between test runs,

...

Is it me or do you guys just keep ignoring everything they say?

Originally Posted by 67shelby

However, my question is how do you capture the performance data to create those charts and FPS results? Do you create timedemos while playing the game, then replay them with the game engine while FRAPS is running to collect the performance data or do you (your staff) play through a particular level with FRAPS running and collect that data for reporting the numbers. If it is the latter, how do you ensure consistency between test runs, what variability do you see between each run, and are your FPS results then averaged for a composite score? If you use timedemos, then how are yours different than other sites that use the same testing methodology with FRAPS or the game engine to report results. I think answering these questions would clear up a lot of what has been said today.

This question was answered in this thread, even if it is buried in 30 pages of posts, it is there. Probably why Kyle and Brent are so tired of answering it over and over. The gist of it though was that the reviewer, in this case Brent, plays the game, plays it a lot, and when you know a level/area enough, even with AI making different decisions, etc. You can still run through an area and basically do the same thing. Shoot 1st korean, hide behind cover, shoot 2nd korean, run to another area, throw grenade, duck, etc etc. I believe that's what the FPS graph is there for, to show that they played the same areas. On every graph I've seen in these reviews, the FPS rises and dips in almost the same identical spots on all video cards tested. That shows that the same areas, and very similar battles/whatever are happening on both video cards.

Edit: If that's wrong then please correct me.

Wow. I can't believe how many people still havn't really gotten a grasp of the [H] methodolgy. Sounds like we need a major front page write up re-explaining things in full. I personally don't see what the mystery is, as Kyle has explained until blue in the face.

I realize the underlying differences that exist between ALL hardware review sites. In fact, I believe that non of them preach the absolute gospel. So I would advise reading everything with a grain of salt and a little discernment. I don't think your hand needs to be held to understand where a card lies against it's competition.

Originally Posted by bluehaze013

Click the link that says discuss this in our forums at the end of the article


Here is the whole post, and there is some more information in that thread as well as in the review, each test is run three times per card and the results are averaged:



Thanks for the words of support today guys, much appreciated. Also be sure to check out the other reviews, we posted quite a few in the news today. I was a little disappointed with Hardocp and their stance today on their front page, stating that apparently everyone else is using "canned" benchmarks and they are the only ones who aren't.

We haven't been using premade benchmark scripts for a very long time (we also dropped 3dmark in reviews before hardocp did), stuart uses real game play. This is in fact how we noticed the minimum frame per second issue we noted (page 17 of the review), I don't think anyone else has even covered this in detail, take from that what you will! Not only that but we managed to get a statement direct from AMD explaining why. We have spent the last week working with AMD on the drivers, so I think its pretty fair to say we know what we are talking about.

That said, id rather not dwell on this, I hope our review has been informative, entertaining and educational.

As I said, I read the article. I saw the note that said they ran their tests three times and averaged the results. While that still is not specific as to how they test, it is the exact wording used to describe the canned benchmark method on most sites.

When he says, "oh no, we use real-world tests," does he mean they create their own time demos instead of using ones included with the game, or does he mean they FRAPS actual live gameplay like [H] does? There is a difference in the quality and accuracy of the results. For example, given how much higher the AA settings and framerates were on their tests for COD4 at the same resolution as [H] used, I would tend to believe that they used a testing method which did not capture all the elements of real gameplay.

On every graph I've seen in these reviews, the FPS rises and dips in almost the same identical spots on all video cards tested. That shows that the same areas, and very similar battles/whatever are happening on both video cards.

Apparently they just choose to overlook this fact. The AI doesn't chage drastically enough to even worry about this. The simple fact that I can sit down and play through a game a couple times and memorize the position and map of most levels makes this point so moot as to not even bring it up or worry about it in the least.

Originally Posted by Kyle_Bennett

Where exactly did HardOCP make that statement on our front page? Please give me a link and the quote please.

Link to which? The statement is here: http://enthusiast.hardocp.com/articl...50aHVzaWFzdA==

The guy that I was quoting was from DriverHeaven.net here: http://www.driverheaven.net/news/153...3870-x2-2.html

Now they are saying you retracted it, but it's still showing up for me? Dunno man...


Addressed on this page already. http://www.hardforum.com/showpost.ph...&postcount=562 - Kyle

Must be the fastest growing thread since "Valve sucks!"...

Strange, as there is not much to discuss - there is no discrepancy between the H review and the reviews on other sites, there is only a question of different technical approach. The consumers must decide for themselves which approach is most relevant for their use.

Main X2 selling point for me is 4 DVIs out of one PCI slot, and price/performance ratio. Noise may be a drawback. Superior ultra-high resolution gaming performance is actually the last criterion on my list.

Originally Posted by Commander Suzdal

As I said, I read the article. I saw the note that said they ran their tests three times and averaged the results. While that still is not specific as to how they test, it is the exact wording used to describe the canned benchmark method on most sites.

When he says, "oh no, we use real-world tests," does he mean they create their own time demos instead of using ones included with the game, or does he mean they FRAPS actual live gameplay like [H] does? There is a difference in the quality and accuracy of the results. For example, given how much higher the AA settings and framerates were on their tests for COD4 at the same resolution as [H] used, I would tend to believe that they used a testing method which did not capture all the elements of real gameplay.

It's in that thread man here I will quote it out for you:



I take it you aren't a regular, so ill break it down. We use real game play to test graphics cards, we have done so for a long time. if we used pre built benchmarks or time demos we wouldn't have found the minimum frame rate issue as the majority of sites who used these still haven't even found the issue. This is only seen with extended gameplay, or what id like to call "real life" gameplay. We play games for hours behind the scenes with new cards before we even start benchmarking. This is the best way to find or note anything unusual before we get into the indepth review testing. This is how we found issues that caused AMD to delay their NDA for a week. Issues we helped solve.

if you would like anything else explained please ask away.

Every single test in the review used real gameplay... i can confirm that as i was the one who tested the cards and wrote the article. We/I havent used timedemos, or built in benchmarks for years, and have never benched a cutscene in my life.

and heres a link to the thread:
http://www.driverheaven.net/news/153...3870-x2-2.html

Specifically, does anyone have an idea how timerecorded demos might skew the results if at all? They don't exactly seem the same thing as cutscenes but at the same time I would image that it takes some of the load off of the GPU from various things (AI, physics, etc).

Well that pretty much does it for me- After reading [H]'s review, Overclockers club review, PC Perspectives revew, extreme tech review and finalyl guru 3d's review and even Anands taken into count- this card in some certain games like Call of jaurez and Half life 2 in game will beat an 8800 ultra- but pretty much all the other games ive seen- its the complete opposite.

Take what you will- Im pretty amazed that G80's lasted this long- They are definitely the new ATI 9800...

btw. Noseguard- For calling "Nvidia Fanboys" out so much- you could be the biggest ATI Fanboy i have ever seen on the net....Please just stop and instead just debate over the cards. no need to call people out !

ati needs to realize that no matter what they do to it R600 will never beat the G80. and get to work on the R700. mabey it'll be somewhat competive with the gt200 or whatever the next gen geforce will be

Originally Posted by Patman08

ati needs to realize that no matter what they do to it R600 will never beat the G80. and get to work on the R700. mabey it'll be somewhat competive with the gt200 or whatever the next gen geforce will be

Theyre already at work on r700 and have been for a while now, it was reported the chip taped out a while back and was rumoured for a late q2 release.