PDA

View Full Version : New XML-based API Registry released



Jon Leech (oddhack)
06-13-2013, 02:42 PM
The old gl.spec & related files are being retired, replaced by an XML-based description of the GL, GLX, WGL, GL ES, and EGL APIs as has been promised since slightly before the Dark Ages. The XML files & Python scripts to process them and generate headers are in the public-access portion of Khronos Subversion (https://cvs.khronos.org/svn/repos/ogl/trunk/doc/registry/public/api/), and the GL, GLX, and WGL headers generated from them are also there and linked from the registry index (http://www.opengl.org/registry/#headers).

The GL, GLX, and WGL headers should be equivalent to the old headers generated from .spec, although they are structured differently and in some cases, enums and entry points have moved from one extension #if block to another (each extension / core version now references all the tokens and commands it introduces, even if some prior extension also references them, and if a token or entry point appears in more than one extension, it will only appear in the .h at the first place it's referred to). The GLES and EGL headers on the Khronos Registry website will probably be updated to be generated from the corresponding Registries fairly soon, if the corresponding Working Groups are OK with that.

The new format includes a Relax NG compact schema, and all the XML validates against that. There's a short description of the tags in the README.txt. It captures most, but not all of the information in the .spec files in a more semantically expressive way, describes types in C syntax instead of the generic type syntax used in .spec, and does not include array length information (which was often inaccurate in the .spec files). If you are using .spec to generate bindings to other languages, you'll have to translate the C types to your output types. The header generation scripts are intended to be extensible to other output formats fairly easily, though we haven't done anything with it but C headers.

The old .spec files and GL, GLX, and WGL headers generated from them are still available under a relocated Subversion path (https://cvs.khronos.org/svn/repos/ogl/trunk/doc/registry/public/oldspecs/), but we don't plan to update .spec in the future. A large number of .spec bugs that people had submitted have been addressed in the XML registry and closed, or in a few cases, WONTFIXed because they referred to specific properties of .spec that don't exist in the XML. It might be possible to do a mechanical translation from the XML back to .spec format - it would not be easy, because the XML is structured very differently and there are a lot of one-off hacks in .spec that aren't needed in the XML - but hopefully people can adapt their uses to work directly with the XML instead. It's possible the schema will need additions to accomodate some uses, feedback on that can be provided here.

Alfonse Reinheart
06-13-2013, 04:31 PM
Thank you for doing this.

Also, you now have your first bug report (http://www.khronos.org/bugzilla/show_bug.cgi?id=886) on the new spec files ;) Followed by two more ;)

thokra
06-14-2013, 02:43 AM
Relax NG compact schema

Most awesome! :)

Alfonse Reinheart
06-14-2013, 07:47 AM
Most awesome! :)

Even better, it has actual documentation in it explaining what the elements mean.

thokra
06-14-2013, 08:03 AM
it has actual documentation

*rubs eyes for about a minute*

Well damn, this is almost like solid software engineering. :D

Jon Leech (oddhack)
06-14-2013, 04:15 PM
The "actual documentation" is more complete (and prettier) now: see PDF (https://cvs.khronos.org/svn/repos/ogl/trunk/doc/registry/public/api/readme.pdf)

Jon Leech (oddhack)
06-24-2013, 04:32 AM
Since there was a good deal of demand for it, I've added the parameter/return type and array length annotations from gl.spec and enum*.spec into gl.xml. This required some schema additions, and pretty massive changes to gl.xml - I haven't done this for glx.xml or wgl.xml yet but will if this is well received.

There are a lot of caveats about the accuracy and completeness of these annotations, discussed in section 18.3 of the readme.pdf. I haven't done anything to make the annotations any more useful than they were in the .spec files, simply provided access to most of the same annotations (with exceptions as noted in the readme). If people do find these annotations useful, I hope they will be willing to build on this framework and fill in more enumerant group definitions and parameter tagging, since the vast majority of modern GL APIs don't have the annotations even in the .spec files.

I see there are some patches submitted in Bugzilla already and will get to those next.

Jon Leech (oddhack)
07-01-2013, 05:53 PM
Bruce Merry (of Bugle fame, among other things) wrote some scripts to detect possible command aliases and proposed a bunch for addition to gl.xml, which has been done now. There are a few more tweaks to the parameter annotations as well. Thanks, Bruce!

Nicolas Lelong
07-03-2013, 10:28 AM
Thanks for this !!

I was about to write yet another parser for gles headers, I'm happy not to have to :)

Lucretia
08-04-2013, 08:35 AM
Hi,

Well, it's about time this happened tbh, but from my cursory look I am surprised that:


You've kept the C stuff inside the main XML files where they really should be external to that and kept somewhere language specific.
You've added "GL_" and "gl" where the original specs didn't, this means all strongly typed languages have to go through the aggro of checking for and stripping them.
You seem to assume all enums are bitfields to be ANDed together where in the original specs you could easily see they were separate, had specific values and were not all enumerations, i.e. some are constants of a type to be ANDed.


I could be wrong, I hope I am, but this is what I saw.

Thanks,
Luke.

Jon Leech (oddhack)
08-05-2013, 05:05 PM
You seem to assume all enums are bitfields to be ANDed together where in the original specs you could easily see they were separate, had specific values and were not all enumerations, i.e. some are constants of a type to be ANDed.

I'm not sure what you mean by this. Enums that are bitmask values are tagged as such, e.g.

<enums namespace="GL" group="AttribMask" type="bitmask">
<enum value="0x00000001" name="GL_CURRENT_BIT"/>
...

You're correct on the other two points. OpenGL is explicitly defined as a C API and the XML registry reflects that now, while the .spec files pretended that wasn't the case - which resulted in a lot of pain for me, slow updates, and a fair number of errors creeping in over the years resulting from the typemap system. There is some discussion upthread about ways to help accomodate the differences if you want to mechanically translate the API to other language bindings, but it is true that you'll have to do some things differently than with the .spec files, and do some additional type translation work. Hopefully the benefits of a well-defined XML schema, more accurate description of the relationship between features and GL versions/profiles, ES support, and much more frequent updates will balance that out for most people.

bjzaba
08-06-2013, 10:42 PM
Would you be able to use a typemap like the old specs did?


https://cvs.khronos.org/svn/repos/ogl/trunk/doc/registry/public/oldspecs/gl.tm
https://cvs.khronos.org/svn/repos/ogl/trunk/doc/registry/public/oldspecs/glx.tm
https://cvs.khronos.org/svn/repos/ogl/trunk/doc/registry/public/oldspecs/wgl.tm


This would make it far easier for folks that are trying to generate loaders for other languages that don't follow the C declaration syntax.

Anyway, thanks so much for your work, you have made our lives much easier. :)

SaschaW
08-08-2013, 10:52 AM
As the (currently sole) maintainer of the Delphi (and Free Pascal) OpenGL headers over at delphigl.com I must admit that I'm pretty happy with the release of the XML-based API files. It only took me two weekends to write a converter that automatically creates a complete Delphi / Free Pascal OpenGL header with the click of one single button, and just half an hour ago I even got an OpenGL 4.x template working with that generated header :)

So from now one I don't have to do a diff with the C-headers to check what's changed and merge that manually into our headers anymore. Instead I now just download the latest XML definition, click a button and it's done. The only thing I'm currently struggling a bit with are type mappings, cause there are some differences between data types and their declarations between Delphi (Free Pascal) and C, and I also want the new header to be compatible to our old ones.

But I'd say the new XML definitions are a step in the right direction, especially for generating headers / wrappers for other languages.

Laar1
08-19-2013, 09:06 AM
Great to have a documented and up to date spec in xml format. But while writing a parser for it I stumbled on the point of the use of C-syntax for both types and functions.

Using C inside the spec is understandable from the point that OpenGL is a C API. Though parsing it to use it for something else than generating a C-like header it is a pain. Parsing the current format thus requires not only a xml-parser but also a parser for the C parts. Thus making the spec far more complicated to parse than a simple xml file. Assuming it is parsed the processable cases need to be distinguished. Simple cases like "typedef int <name>GLint</name>" can probably be handled by a translator, while complex cases like the "inttypes" cannot.

I think these problems all stem from allowing arbitrary C, while only a small part is needed for almost everything. Apart from "inttypes", "GLhandleARB" and the C-specific headers all types can be easily expressed without needing C. Furthermore I think that the same holds for the function parameters and prototypes.

Using xml for the standard case would have several advantages. The parsing would be easier, it would be clear what types are present and make C-specific types stand out from the normal ones. Overall this would make writing a parser for the spec format and using it a whole lot easier.

opengl4net
09-16-2013, 02:56 AM
Hi Jon,
it's really great the you've switched to the xml file, there were too many strange things in the old format. As I am working on wrapper for .NET (OpenGL4NET), I've started to go through your new format to rewrite the wrapper generator and I have these comments/questions right now:

1. Someone already wrote, that it would be great to have some more generic way of typemaps. This C declaration makes the process quite difficult as there are various kind of declarations (simple typedefs, callbacks etc.).
2. In some commands (e.g. glBufferStorage) there is no ptype (I know you've stated that ptype is optional, but again for people working on non-C implementation, this is a complication).
3. In the old format, there was a nice hint for parameters in/out value/array. I understand that for C language it does not make a big difference but for other languages this could help to create more language dependent calls. Furthermore having * directly in the <param> element makes the DOM somehow weird.
4. Just a small problem - having <return> tag instead of using <ptype> inside <proto> will again improve the quality of xml, and make it easier for other languages.

Please, would it be possible to make the .xml specification more C-independent.

Thank you

bjzaba
09-27-2013, 09:58 PM
Ok, here's our loader generator for Rust based on the XML registry: https://github.com/bjz/gl-rs :)

void256
02-03-2014, 05:11 PM
I always wanted to have a nice overview of which OpenGL functions or enums are supported in which OpenGL version. Something like the tables that are available for GLSL commands in the reference pages. However, I haven't found one ... so I've made my own :D

Using the XML registry as the base with Lucene and a simple Web-Frontend and API on top I've created: index.gl

Its still Beta but it works well, I think. The sourcecode is available at github too (can't post the link due to number of URLs restriction in a single post limit - but the link is at the bottom of the main website).

Maybe this will be helpful for others as well.