I’ve been doing some thinking about this whole nvidia/ati/Futuremark fiasco that’s been going on lately, and I was toying with the idea of trying to write a fair benchmark for modern graphics cards. I think, if nothing else, it would be a good learning experience.
Before I decide whether this is worth spending some time on, I’d like to get feedback from the developer community (you guys).
Here are some topics for discussion:
-
Is this worth doing? Are there already enough benchmarks?
-
Why is 3dMark so popular? eye candy? online comparison feature? results in a single number? Do people really base hardware purchasing decisions on it?
-
Expanding on number 2, is it a good idea to try to distill a fundamentally n-dimensional test into a single number so that people can compare easily? i.e. “My liquid cooled uber-computer get’s 235 ZenoMarks!”.
-
Is it better to write a code path for each popular card, or is it better to have them all run the same code? The former would show how different cards would perform under hand-optimization and vendor-specific extensions (which most game companies probably do) while the latter would give as much of an equal comparison as possible. I’d lean towards the latter, at least as a first pass.
-
Would people pay for it? $5? Or would trying to charge anything just make it unpopular?
-
What qualities/features would it have to have to get review sites to use it?
-
Would it be worthwhile to try to make it difficult to “cheat” on? I know it will always be possible to cheat, but I had the following thoughts: Keep everything that’s being rendered on-screen and moving. Optimize shaders as much as possible. Use some randomness (might necessitate statistical score). Create software renderer and compare pixel results (or, make a few screenshots before benchmark is released and store them for comparison. Raise a flag if stored and rendered images do not match).
-
I would like some sort of peer review for the project, but would not want to make it open source for a couple of reasons. Suggestions on this? I know there are a few people on this board with more experience than me whose input would be extremely valuable.
– Zeno