Testing an OpenGL Driver

My company will be recieving an OpenGL driver for use in an embedded application (avionics) and I’m responsible for testing the thing when it arrives. Because it’s for avionics, it needs very through testing – “All aspects of OpenGL” is what my manager said. This is testing in addition to the qualification testing that is required to use the ‘OpenGL’ name.

Unfortunately, I’m new to OpenGL (3 months or so of doing tutorials and reading manuals) and testing a drivers seems like a very advanced topic to me.

Can anyone suggest somewhere to look for an idea of what should be in these tests? I’ll need to adapt them to work on our operating system, but I’m hoping someone can give me some idea of where to start.

Much thanks.

That’s a big topic, but the most obvious test that should be run on every OpenGL implementation is conform, i.e., the OpenGL conformance suite. I’m not sure if conform is publicly available. WHQL includes an old version of it for OGL 1.1, but that one will be missing all the 1.2 tests.

• Matt

Maybe glean can help you. Try search it in sourceforge.

I thought that Avionics applications use cutdown versions of OpenGL to make thorough testing and predictability possible. This would make the OpenGL name incompatible because of the conformance tests.

Just passing the conformance tests will not be enough for your purposes although it is the technical prerequisite for using the OpenGL name, you seem to understand this. You require a much more rigorous level of testing which does not exist AFAIK. Testing OpenGL to see that it draws the right thing when asked, does not guarantee that, for example, when you call glTexSubImage2D imediately after glDrawPixels it won’t simply crash (my contrived example).
There are infinite numbers of possible scenarios like this.

I don’t see how you can test this after the fact. It requires an engineering methodology in writing the driver in the first place.

It is also probable that you don’t NEED OpenGL conformance, the dirty little secret is almost nobody REALLY has this, and there are OFFICIAL caveats in there like even conformant implementations are allowed to fail a certain number of tests.

What you need more than conformance, is stability and predictability. If a developer writes something and it doesn’t work or isn’t supported he will see that and can implement it differently. It won’t fail in mid flight, it’ll just cost a little bit of engineering time. OTOH, if you pass conformance but there’s some texture memory manager leak, or display list leak, then you’re in trouble.

So, you probably need a different type of test instead of or in addition to conformance.

You might want to limit the scope of what an application can do, for example your software engineering practices mean you won’t have memory issues, but if you’re using OpenGL display lists you’re relying on some memory management in the driver. So for instance this might be an OpenGL feature you want to avoid, this is highly speculative, I’m just trying to give you a flavor of what I mean.

Read this for more information:
http://www.opengl.org/developers/code/features/flight_decks.html

[This message has been edited by dorbie (edited 01-16-2002).]

If you wish to name the company that is providing the driver, maybe someone has run into their stuff in the past, and offer some advice.

I don’t think you can test every condition, unless you have lots of time on your hands, and you verify the results with known working drivers, or software emulation that is known to work 100% (although unless you have access to a VERY fast system, this is very slow.

P.S.

I have seen stress tests where you run every application you can find on an implementation, running about 9 different ones at a time for hours.

This is a different approach, it exercises the graphics system and forces a lot of graphics context switches but within those contexts only a limited number of paths are exercised.

If all you want to do is exercise every possible combination of state, this alone would be a significant task, but tractible, it might take a while to run. You are still left with issues of call order invariance and driver memory management among other things.

You need to get clearer on what you want and need to test before you tackle the how.

[This message has been edited by dorbie (edited 01-16-2002).]

More Details:

I thought that we’d be using a subset of openGL as well, but was informed otherwise when I enquired what the subset was. AFAIK this will be tested to DO178 level B or C – meaning complete requirements and code coverage but not branch coverage. Basically that means each line of code needs to trace to some requirement and each line of code must be executed within the testing but exercising only one condition of an if statement is okay. The requirements are, in this case, the opengl spec (1.2 I believe). Also required is functional coverage – which basically means all the major functions of the code have been exercised, usually this is covered by the previous two but sometimes an extra test is needed to cover the other way to cause a failure or some such thing. Oh, and robustness testing, which for software is testing timing and bad input values (or unexpected combinations)

I’m hoping that either my manager misunderstood what “all of OpenGL” meant and we’re actually getting a testable subset or else that I’ll only need to test the aspects of the driver that the developers actually end up using (the latter is unlikely, the FAA is strict about ‘dead code’). Or maybe the driver will arrive with a set of tests that can get me started.

In the meantime, could someone point me towards some basic functionality I can check for to at least assure that the driver is usable. Would something along the lines of drawing a cube on the screen at least cover a chunk of code? I guess, rotate it, flash the colors, turn lighting on and off . . . any suggestions?

Thanks much.

Like the man said try glean:
http://glean.sourceforge.net/

I hope the driver author is certifying the driver code. You can’t based on what you’ve said unless you have their code, and even then it sounds like an extremely tough task. The nature of the state engine it’s combinatorial permutations of state and the way a graphics pipeline operates in principal means that operations can be and are stitched together in what approaches self modifying code.

To exercise this it is not enough to say you have entered all relevant functions and branches. It is extremely call order dependent.

[This message has been edited by dorbie (edited 01-16-2002).]

As mentioned in the link above - http://www.opengl.org/developers/code/features/flight_decks.html - the Khronos group has been sanctioned by the OpenGL ARB to produce the official embedded subsets of OpenGL to suit the needs of the embedded market - including avionics.

The Khronos Group is close to finalizing its new participation agreement that will enable any company to join and contribute to the development of the embedded API specifications. If you are intested to participate you can join the embedded API mail groups and/or forums at www.khronos.org.

3Dlabs has produced a strawman proposal that will likely be starting point for the embedded API work - and it defines proper subsets of OpenGL for high-quality 2D, image manipulation without needing the full 3D texture machinery and a minimum footprint OpenGL-class 3D profile.

There has been significant interest in this initiative from the avionics and other embedded industry segments. The intention is that industry-agreed subsets of OpenGL will enable independant hardware and software dvelopment: chip companies such as 3Dlabs can produce cost and power-optimized accelerators for these profiles. Driver companies such as Seaweed can create DO-178 certifiable subset drivers. This will mean that individual display OEMs will no longer have to start from scratch in figuring out how to create a certified display subsystems.