It might be useful to have some sort of official unit testing framework to help increase the quality of vendor drivers.
The unit testing framework would check which OpenGL version or extensions are available, and then proceed to test if each bit of functionality works as defined by the specs.
When some things are left open in the specs, such as precise pixel colors, pixels could be compared to reference colors together with certain thresholds (perhaps using SSIM/PSNR etc. etc.)
The framework could start small, only checking a couple of things and expanded over time.
The same framework could also be used by developers to double check if problems are caused by drivers or their own code…
Isn’t that what they call “conformance tests” ?
I have the impression they have not been updated much, and last I checked these tests are closed and costly.
It would basically be a conformance test, yes.
Not updated much? I’m assuming you’re referring to the vendors own private conformance tests?
The thing is, if there was an open source, more community involved test it would certainly be improved more over time and capture more bugs and problems than any private vendor created framework.
Not to mention that a common conformance testing framework among vendors would ensure that at least those things tested in the framework would work on all the drivers of those vendors.
(which would be a huge step forward)
A vendor’s primary concern is getting things out of the door, updating a framework, even if it improves the quality of their own code, comes second at best.
Second of all, we’ve all seen plenty of cases where drivers shipped with pretty obvious bugs.
In which case we can only guess that that vendor didn’t do any conformance testing, or that their conformance testing where lacking to say the least…
A community based/involved conformance test suite would be cheaper for vendors too. (at least for those parts that are not vendor specific I suppose)
There was a brief mention on the conformance suite status at the OpenGL BOF this past SIGGRAPH. See the last few slides in the Ecosystem Update presentation.
Sounds like there is a conformance suite for OpenGL ES 2.0 that was ported to desktop GL, and Khronos is working on funding to finish the project.
I agree this is a much needed effort, and I am looking forward to it.
There already is an automated testing framework for OpenGL. It’s called piglit and is used by the graphics driver development community (i.e. the community around the Mesa project) for them to ensure some level of quality of open source OpenGL drivers and to make writing drivers easier.
One year ago there was like a few hundreds of tests in this test suite, but today there are thousands and the number is rising! In addition to the automatic mode, most of the tests also support an interactive mode, so that you can see what is rendered wrong. Some tests also contain tens to thousands of subtests, e.g. they test various permutations of OpenGL states.
I think this is a pretty cool project that should not be overlooked.
NOTE to AMD: Your drivers fail some piglit tests. Would you be so kind as to fix them? Last time I tried Catalyst, the score was not good. There were about one hundred failures: http://people.freedesktop.org/~mareko/catalyst/problems.html
I guess NVIDIA are no better than AMD here, but I did not tested their drivers, so I can’t say for sure. However as a matter of fact, if they don’t use piglit, they certainly fail a good deal of tests there.
We do our best to ensure it’s really congruent to the OpenGL specification as well as various extension specifications in the registry, but if you expect it to be bug-free, well, we all know that bug-free projects don’t exist. It’s developed by the FOSS community, so if you find anything wrong with any of the tests, feel free to send patches.
Some big companies that contribute to piglit are Intel, VMWare, and Red Hat.
EDIT: Note to Eosie : if is was possible this project was mentioned/linked at the top of this page, it will increase its visibility a lot : http://www.mesa3d.org/conform.html