Constructive Criticism for Futuremark

Ok, Tero owned up to his mistake.

And I've seen him on this forum A LOT today.....
see:

Lets use this chance to tell him what we like or dislike..... in a nice positive way.... maybe the next 3dMark2005 patch could include some revisions, or (more likely) 3dMark2007 might have more of the things WE WANT TO SEE. And as the cutting edge, hardcore tech people who are the ones that educate and maintain all of the joe six packs out there... our opinions should be valuable to him.

P.S. Criticism about 3dMark please, not Tero personally!! lol

Instead of focusing on a separate engine which gives no indication of performance in actual games on store shelves, why not move into making benchmarking software that takes advantage of pre-existing games as benchmarks...instead of a demo...why not make software that can run batches of your favorite games in benchmark loops and automatically format the output...

If crytek, id, valve, etc all contributed demos, that would lighten the load for 3dmark in creating game content and they could just focus on making the benchmark itself work...

If anyone has the power to pull a bunch of different gaming developers together under one roof to make a killer benchmark, futuremark can...

I don't have a problem with them using a proprietary engine, as long as it employs methods comparable to what's generally being used. I'd like to see more specifics in the results, though... being able to see what the bottleneck is in each instance.

Oh, and there's a B. Jovi on your list...

In total agreement with both of the previous posts. Mr. Sarkkinen are you reading this?

Real simple. Real game engines, real game. Let game cos sponser FM instead of Vid card makers.

As long as higher benchmarks / framerate equals more $$$, the days of legit comparable results are over. Lets face reality, there's nothing Futuremark and/or anyone else can do to correct what's happening.

Also, the idea of benching only games you are playing is not the answer either. Most people do not have the money to buy all of games and hardware just for testing.

It would be nice if there were test to verify manufacture specs with 3DMark, like bandwidth
fill rate and verticies/second more along the line of Sisoft Sandra for Ram...and be able to
move the camera.

There would also be options to turn on/off specific options, so as see whats happening
behind the scenes "so to speak".

Theres another thread going on already.
http://www.hardforum.com/showthread.php?t=817000

Myzhi said:

Most people do not have the money to buy all of games and hardware just for testing.

Click to expand...

Who said it had to require you to have all the games? I would actually prefer 1 or 2 hardcore games and have them implement mods that make performance crawl (like with D3 what CoD did from the Q3 engine).

Myzhi said:

As long as higher benchmarks / framerate equals more $$$, the days of legit comparable results are over.

Click to expand...

Absolutely. Futuremark chose their business model and all of the inherent weaknesses therein. I cannot in sincerety say 3DMark XXXX is not indicative of performance in all situations. It pointed out the flaws in the NVidia 5XXX series, while at the same time showing it's weakness as a true benchmark due to NVidia's blatent (yes BLATENT) cheating. A test cannot measure performance to any degree if the student cheats, same for video cards. NVidia effectively nullified 3DMark as a reliable indicator of performance (not that I specifically bash NVidia, ATI took their own liberties also, just not in a "hit you over the head style"). Anytime a driver team spends time optimising for a specific synthetic benchmark is time a driver team is NOT spending on improving drivers for GAMES.

I really do not know what 3DMark has to offer. I hate to sound like I am bashing Futuremark because they do make some pretty eyecandy demos and it takes time/talent and effort to produce such a product. I am just unsure of 3DMark as an actual measure of performance, which is what is intended to be.

My main suggestion would be that they code for OGL as well as D3D, I realize that may not be possible because of there involvment with Microsoft but it would be a nice way to compare the competing standards as well as having a gauge of what to expect in a good OGL game such as doom 3.

Another suggestion that I can think of off the top of my head is to make it so the CPU score isn't affected by driver revisions, the latest version of 3D mark is flawed in that respect. Currently, identical systems with either the vid card being different or the driver rev being different produces a higher CPU score for the system with the better video card. Don't ask me why, it just does.

My 2 cents.

My thoughts on how to improve 3dmark05:

3dmark05 fails as a real life gaming benchmark, but even worse it fails as the synthetic benchmark it is. 3dmark05 needs to be improved in the following way - it must use features of today's graphics cards extensively to demonstrate performance differences and image quality differences between this generation's video cards. 3dmark05 is no more than an intensive SM2.0 test. There is no extensive use of SM2.0B or SM3.0 (despite basic support of those modes) i.e. no geometry instancing, no dynamic branching, no longer shaders. There is no OpenEXR HDR support, which is the future of HDR lighting. No 3DC, so we can't see how that helps performance wise or IQ wise. Virtually none of the features introduced into this gen's GPUs were utilized significantly in 3dmark05 beyond a marketing standpoint. If Futuremark could not find a use for these things in its demos, I suggest they create different and more diverse demos, because some of the most popular real life games are using or will be using these features this year, and they have shown significant gains even from moderate use of them. I'd also recommend picking up OpenGL support if they truly want to be the "gamers" benchmark as Doom3 engine games all use OpenGL.

3dmark05 as a benchmark test best measures performance potential of the FX 5950 Ultra versus the 9800XT on games from 2003. Instead, it should be measuring the performance potential and image quality potential of the 6800 Ultra versus the X800XT on games from 2004 and beyond. This is where 3dmark05 fails, and this is where the improvement needs to be made. Even staunch supporters of synthetic benchmarks could not make a good argument that 3dmark05 is an accurate predictor for performance and IQ of today's cards, because it fails to significantly utilize most of the new technology today's cards brought to the table to enhance performance and image quality.

Dear Futuremark,

I love how your product incites the fan boys. Please keep up the good work.

Sincerely,
MrDigital

MrDigital said:

Dear Futuremark,

I love how your product incites the fan boys. Please keep up the good work.

Sincerely,
MrDigital

Click to expand...

lol. Thats all I can say.

The idea of Futuremark working with game companies and using their engines as the foundation for designing the bench mark tests sounds like a winner to me.

reposting from other thread hope this is ok

I haven't posted much less read hard|forum for quite a long time... Since I got banned quite a while ago for something that was a bit dumb, but no hard feelings.


Anyways I suggest to Futuremark that to improve the "usefulness" of their benchmark, they need to diversify the 3dmark scores into a few different tests.

One 3dmark number just isn't going to do it.

I suggest that when you start up 3dmark you pick the tests you want to run, or run all of them, but at the end you get a few different 3dmark points.

Just off the top of my head:

Futuristic Points:
Current Points:
Old School Points:

Just divide up the tests so that Futuristic is your best guess at what tech games in the future will use and how well cards do at that. Current is taliored to features and tech current game engines are using, and Old School is for year- 2 year old engines.

Then we can all use "current" as the most important 3dmark, and note "futuristic" as important, but not as important as you can't actually buy a game that would relate to.

Thanks for the time.

The "joe six packs" out there? Havent heard that one before....

a way to compare image quality.

maybe a stored bitmap of what it should look like and a toggle back and forth between what was rendered. also, hotkeys for showing mipmaps, trilinear or bilinear interpolation etc, (which should void the score upon usage).

the movie thing doesnt really help me determine where (or when and why) my system may suck at
i would want a simplistic scene that doesnt play out like a movie.

a free range area where we can control the camera, and then have points in that area where the camera can lock to a standardized (for fps comparison etc.) view of a particular effect. focal length adjustments, or house of mirrors, or walls of skin and flesh, or intense physics computation, and have a hotkey for a pause screen which would display info about what sort of techniques are being used or are being demonstrated for that area.

i want more info than just a number.

say there is a game that uses alot of one difficult effect, that one companies drivers really are great at. 3dmark may not give that particular effect alot of weight in its scoring. and then what of that game engine is reproduced and sold to 50 other games does that effect's scoring weight change? even if its a spectacular, yet computationally brillianty done? expand this idea to have a reference say that these *9* out of our *52* tests really reflect the comparative performance for *this* particular game.

give the 3dmark user the ability (somehow) to see the individual strengths and weaknesses of each driver version.

i would rather dick around for an hour with rthdribl.exe and judge fps than to watch the very scripted scenes. they were really boring this time around.

more added tomarrow

Ocean said:

a way to compare image quality.

maybe a stored bitmap of what it should look like and a toggle back and forth between what was rendered. also, hotkeys for showing mipmaps, trilinear or bilinear interpolation etc, (which should void the score upon usage).

Click to expand...

I like that idea. Instead of just reference images.... but an automated comparison test. Show the 2 scenes merged with the differences highlighted in white....

you know the black screen with white dots where the differences are....... welll add up the white dots and subtract from the final score. So if you're using optimizations to increase your score the "Quality Reference Comparison Test" will subtract roughly the amount of points off that you gained.

This will counter act cheating. If the differences are minor then only a minor subtraction, but if the scene is rendered very poorly b/c of all kinds of optimizations then subtract more points.

I'm not sure if it would catch artifacting from overclocking too far, but if it did that would be great too. B/c those arent valid scores if its so overclocked that you're getting all kinds of artifacts!!

- OK, I THINK MY IDEA IS PURE GENIUS!! - (patting myself on the back)