PDA

View Full Version : Interest in specialized GL function loader.



Alfonse Reinheart
10-04-2012, 01:37 AM
I'm interested in gauging the general interest in a specialized OpenGL function loader. What I mean is this.

Everyone has their own parts of OpenGL that they use. Everything else is superfluous.

Yet common loaders like GLEW and even my own GL Load will dump everything at you. If you're using core OpenGL 3.x+, there aren't a whole lot of extensions that interest you. Some of the non-hardware 4.x extensions are useful of course. But you don't need stuff like GL_EXT_shadow_funcs cluttering up your headers. If you're not using EXT_direct_state_access, do you really need it's 1001 functions clogging up your header?

And while one could create a core GL-only set of headers and loaders, there are extensions that people use that aren't core. Everyone has their own set.

Rather than trying to write a one-size-fits-all solution, what about a simple script, where you provide a basic version (with compatibility/core selection of course) and a list of extensions, and it creates a header and a .c or .cpp file for you to add to your project? The header imports everything you asked for. Something similar can be provided for WGL/GLX extensions.

Is that something that people might want? I've got most of the code-generation system working from my work with GL Load, which already knows how to break out functions by version and so forth. From there, it's simply a matter of modifying the front-end of the code generation system.

thokra
10-04-2012, 02:49 AM
Something like


genGLLoad [-core][-compat] [-extensions EXT, {EXT}]

?

Would be cool. One thing that annoys me constantly is that several IDEs I use aren't able to correctly parse glew.h and determine the function signature due to the indirections incurred by the #defines. Of course, this is basically the IDEs' fault but to help them provide accurate information, is there a simpler way of declaring stuff?

Also, if you're going to write that script, it's a perfect opportunity to provide a very basic GUI which lets you select the extension you need. I'd be willing to write that one up quickly if desired.

randall
10-04-2012, 04:10 AM
It would be very useful. Currently I am loading manually only the commands I need.

malexander
10-04-2012, 06:18 AM
That'd be useful. Currently for each new extension we decided to support, we copy the defines and function pointers from glext.h into our own header. For some extensions this is trivial, for others it's pretty tedious. Having a script that extracts these would be nice.

Could there be a flag to specify whether the function or function pointer declarations are output?

Alfonse Reinheart
10-04-2012, 11:42 AM
Could there be a flag to specify whether the function or function pointer declarations are output?

What do you mean by that exactly? Are you talking about doing static linking vs. dynamic or something?

malexander
10-04-2012, 05:03 PM
What do you mean by that exactly? Are you talking about doing static linking vs. dynamic or something?

Perhaps I've misunderstood what your script will be producing, but essentially the glext.h file has two ways to get at the underlying function (using GL_ARB_instanced_arrays as a simple example):

GLAPI void APIENTRY glVertexAttribDivisorARB (GLuint index, GLuint divisor);
typedef void (APIENTRYP PFNGLVERTEXATTRIBDIVISORARBPROC) (GLuint index, GLuint divisor);

A few ways to access this function come to mind:


Declare glVertexAttribDivisorARB() as a function in the header. Including a header with this file, without an underlying implementation in a GL lib, will fail to link (static) or fail at runtime (dynamic)
Declare a PFNGLVERTEXATTRIBDIVISORARBPROC glVertexAttribDivisorARB function pointer variable in the header, assigned by the .cpp file in an initialization routine. If the core feature / extension is not supported, it is NULL - no linker issues. Onus is on the developer to do proper checking before calling it.
Declare a wrapper function in the header, arglVertexAttribDivisorARB() (or some variant), which is defined in the .cpp file. This function asserts if the PFN variable in NULL (also implemented as a variable and initialized in the .cpp file), and calls it.

I'd prefer #2 for how we have things set up here, but I can see the value of the other two as well for other projects. So, a command line option to specify this behaviour.

Alfonse Reinheart
10-04-2012, 09:27 PM
I see what you're getting at now. #2 is the one that most widely actually works, so it's generally what I go with. I could implement the others, but as I work mostly on Windows, I wouldn't really know how to test it.

thokra
10-04-2012, 11:19 PM
I could implement the others, but as I work mostly on Windows, I wouldn't really know how to test it.

Since we have a pretty diverse here this shouldn't be a problem.

Aleksandar
10-06-2012, 03:17 AM
Rather than trying to write a one-size-fits-all solution, what about a simple script, where you provide a basic version (with compatibility/core selection of course) and a list of extensions, and it creates a header and a .c or .cpp file for you to add to your project? The header imports everything you asked for. Something similar can be provided for WGL/GLX extensions.

I'm not sure what parameters your script will use as an input, but it is much better to parse all files in the project searching for the GL calls and generate two files (.h/.cpp or .c) with appropriate pointers. That's the way I make the work done. I'm also keeping glext.h and wglext.h in the project. You could make the solution even more compact excluding those files, but it requires extracting appropriate enumerations and constant which requires additional work.


Having additional options, like: /V:GL3.3 /C, is welcome, if the project is targeting a concrete GL version. But the basic usage should be "give me all functions I use in the project".

Could you make command prompt applications instead of scripts? Some of us are allergic on LUA/PHP/Ruby solutions. ;)

thokra
10-06-2012, 03:53 AM
I'm not sure what parameters your script will use as an input, but it is much better to parse all files in the project searching for the GL calls and generate two files (.h/.cpp or .c) with appropriate pointers.

He said quite clearly, provide a basic version(core or compat) which would be a mutual exclusive switch and perhaps default to core if not specified and a list of extension.

The problem with your suggestion is that code needs to exist so you can run the script. What if you start a new project and you know you're gonna need a GL4.3 core header? What code is there to parse?


Could you make command prompt applications instead of scripts? Some of us are allergic on LUA/PHP/Ruby solutions.

I guess you mean a native executable. The thing is, at least for Python I can say that the wealth of the functionality and its ease of use is alleviating a lot of stuff when doing text processing or text generation. Not having to install an intrepreter is a reasonable suggestion, but it pales in comparison to 3rd party dependencies you need to get and maybe compile and general project prerequisites we OpenGL devs need on a regular basis. By excluding scripting languages you take many good development options away. I recently wrote a code generation script at work in Python - I cannot imagine how long this would have taken me in C++ so I'd advocate for Python.

Aleksandar
10-06-2012, 05:01 AM
He said quite clearly, provide a basic version(core or compat) which would be a mutual exclusive switch and perhaps default to core if not specified and a list of extension.
Do you think 90% of people trying to use OpenGL know what extensions they need? I wouldn't bet on it.


The problem with your suggestion is that code needs to exist so you can run the script. What if you start a new project and you know you're gonna need a GL4.3 core header? What code is there to parse?
Yes, it requires a code. A new project does not have any GL calls so there is no need for the extensions either. Only oglext.h, for example, should be included, and before compilation that function loader should be called (and it would fill oglext.h/.cpp). It could be some kind of plugin for VS, but it is better having a platform independent tool.


I guess you mean a native executable.
Yes, I still think it is a better solution. Alfonse would probably provide both code and executable for Windows. Knowing myself and many programmers around me, a executable is much better solution (if it works correctly), than having a code that have to be compiled, or script that requires its on environment.

Alfonse Reinheart
10-06-2012, 11:04 AM
it is much better to parse all files in the project searching for the GL calls and generate two files (.h/.cpp or .c) with appropriate pointers.

There are two major problems my tool is intended to solve:

1: Less-than-helpful IDE code completion. Using an all-in-one solution means that typing "gl" or "GL_" will cause your IDE to suggest everything. With this tool as I defined it, you can at least restrict it to a specific subset that you consider valuable, thus getting rid of 90+% of the IDE suggestions you won't be using.

2: Preventing you from accidentally using something you don't want to. If you forget that most 3.x hardware doesn't support ARB_draw_buffers_blend, and you use it, an all-in-one solution will just assume you know what you're doing.

The tool as you want it will undermine both of these. Your parse-generated header is based on existing functions/enums. OK, but that doesn't solve the problem of using a function I've never used before. The IDE will be massively unhelpful until I type the name and run the parse-generator again. If you already have a codebase that uses all of the functions you intend to use, then you're OK, but if you're trying to get to that point, it's all a big pain.

Further, your tool has no provisions to prevent you from accidentally using functions you don't want to use. Indeed, it will "helpfully" generate functions/enums for anything you ask of it. Thus it acts exactly like the all-in-one tool.

It's just not solving the problems of interest.


Do you think 90% of people trying to use OpenGL know what extensions they need? I wouldn't bet on it.

True. But I bet that those 90% also don't care. They're probably happy with a "give me all of OpenGL" loader. They also wouldn't want to set up an automated code-generator that they have to run every time they use a new OpenGL function. Nor would they likely be willing to install a plugin into their IDE of choice (assuming that their IDE supported such a plugin).

The people who don't know what they want already have a tool. This tool is for the people who do know what they want.


Could you make command prompt applications instead of scripts? Some of us are allergic on LUA/PHP/Ruby solutions.

There are limits to how much I'm willing to accommodate people's eccentricities. Small applications are already being written entirely in scripting languages. A major distributed VCS (http://en.wikipedia.org/wiki/Mercurial) is mostly written in Python, and the main GUI for it is almost 100% Python. So the sooner they get over this nonsense the better.

I'm not going to take my perfectly functioning code generation Lua code and port it all to C++ just because some people want to write "ExeName -options" instead of "lua LuaName.lua -options". And if someone is really that lazy (yet is somehow perfectly willing to build such an application), then I'm fine with them not using my code.

sqrt[-1]
10-06-2012, 04:39 PM
Having a targeted loader is something I have always wanted and had mused about doing it - so I am glad someone is going ahead.

Just a crazy thought, could this be run on a web server somewhere (perhaps - dare I suggest even Opengl.org?)
You have a web page that lists the versions/extensions with check boxes and then download the generated zip?

Alfonse Reinheart
10-06-2012, 04:59 PM
Could it? Yes.

I have absolutely no idea how to go about doing that of course. But it could certainly be done. It's going to be a simple command-line script, and the outputs should be readily identifiable via tool. The hard part (from my perspective) is the input/output part: building the web page, getting data from it to the server so that it runs the tool, and returning that data to the user. I know nothing about any of those processes.

Zyx_2000
10-07-2012, 03:14 AM
I would find such tool very useful. Especially if it could generate a C++:ish interface that puts everything in a gl namespace.

It would be even more useful if it had some support for multiple rendering paths using different OpenGL versions in the same program. So when I program for example the GL 2.1 path, it would prevent me from using GL 3 stuff, etc.

Aleksandar
10-07-2012, 08:32 AM
Sorry, Alfonse! I didn't get the point of your tool at first glance. I was not aware of the intention to confine IDE code completion to a predefined GL version. What I proposed is the absolute minimal function importer regardless of GL version.

I'm just curious, how do you plan to handle vendor specific and/or experimental extensions? Would it be used for the core functionality only?

Alfonse Reinheart
10-07-2012, 09:10 AM
It would be even more useful if it had some support for multiple rendering paths using different OpenGL versions in the same program. So when I program for example the GL 2.1 path, it would prevent me from using GL 3 stuff, etc.

I suppose you could generate two sets of headers/loaders, each exposing different versions of OpenGL and sets of extensions. To do this, I need to expose a way for two loaders to co-exist in the same exe (ie: don't stomp on each others' global variables/functions).

I should do that.


I'm just curious, how do you plan to handle vendor specific and/or experimental extensions? Would it be used for the core functionality only?

You list the extensions of interest, either on the command-line or via a file.

dukey
10-07-2012, 10:37 AM
I normally write my own loaders for opengl functions. Like I have a self contained FBO class that loads its own extensions. Just makes life easier for me. A lot easier re-using them in different projects too.

Alfonse Reinheart
10-12-2012, 10:15 PM
OK, so a basic implementation (https://bitbucket.org/alfonse/glloadgen/) is now available. Feel free to download it and have a go. The link goes to a page explaining the basics of how to use it.

Not much has been tested. I ported a basic GL 3.x "Hello World" application to it, so I can verify that it does compile and generate code well enough to load functions. I verified that the WGL generation compiles, but I haven't tested using it. I have absolutely no idea about the GLX stuff (I'm between Linux distros: fed up with Ubuntu's Unity garbage, but not enough free-time to install Linux Mint yet).

It's actually pretty amazing how small the OpenGL headers are when you're not importing hundreds of extensions that you really don't care about.

Feel free to drop off bug reports for anything that doesn't work. (https://bitbucket.org/alfonse/glloadgen/issues?status=new&status=open) Especially on Linux, which I haven't been able to test yet.

thokra
10-13-2012, 12:20 AM
fed up with Ubuntu's Unity garbage

Simple solution: Use Kubuntu. ;)

Gonna check it under Linux tonight.

kRogue
10-13-2012, 01:22 AM
Just one.. very morbid thought: does your tool require to be maintained as new GL versions and extensions are added? Is it possible to tweak as follows:

Same nice command line options as now
Takes as input a set of GL header files from which it generates the data. The idea is that glext.h (for example) has that functions for an extension foo are surrounded by #define GL_foo/#endif pair.


Though, I wonder about the horror of reused tokens and "dependent" extension functions (for example in an extension, "if GL_foo extension is supported then also the following functions for GL_bar are added: glBarFoo()" ).

At any rate, looking forward to futzing with this.

thokra
10-13-2012, 03:07 PM
Just played around a bit on Linux. The first example can be generated with Lua 5.1 and can be compiled with warnings using GCC 4.7.2. Runtime tests follow. I filed multiple bug reports concerning diverse stuff.

Alfonse Reinheart
10-13-2012, 10:56 PM
Takes as input a set of GL header files from which it generates the data. The idea is that glext.h (for example) has that functions for an extension foo are surrounded by #define GL_foo/#endif pair.

If you or someone else would like to do that, you may. But I'm not interested in writing a parser for a header file, just to reverse engineer data that already exists in a usable form. It's much easier to just run my existing process (https://bitbucket.org/alfonse/gl-xml-specs) on the .spec files, then use the Lua outputs in this project.

Also, I don't think that glext.h handles the whole core/compatibility thing, so you wouldn't be able to produce a clean GL core header. And it's missing all of the GL 1.1 enums and functions.


Though, I wonder about the horror of reused tokens and "dependent" extension functions (for example in an extension, "if GL_foo extension is supported then also the following functions for GL_bar are added: glBarFoo()" ).

There's not much that can be done about that. Though I did put in some special code for loading core functions that if it detects that the "core function" has EXT on the end of it (such as "glTextureStorage1DEXT", which is marked as version 4.2), then it won't count it as a "missing" function if that function wasn't found.


I filed multiple bug reports concerning diverse stuff.

I fixed a couple of those. The others will take a little time to resolve.

Alfonse Reinheart
10-15-2012, 02:01 PM
OK, I think I've fixed all of the outstanding bugs (those not entered by me). Also, I have a C++ style that does some nice namespace wrapping for everything.

So I'm calling this version 0.2 (https://bitbucket.org/alfonse/glloadgen/downloads). Take a look if you have the time.

Oh, and the documentation is a lot more complete (https://bitbucket.org/alfonse/glloadgen/wiki/Home). Also, I've documented the mechanism for creating your own writing styles (https://bitbucket.org/alfonse/glloadgen/wiki/Style_Creation), so that you can use the basic generation framework to generate whatever style of OpenGL loader you want.

I'm going to make a style that's compatible with the Unofficial OpenGL SDK (that is, it uses function pointers loaded by that system), so that you can use it's clean headers with them. I might do the same with GLEW, so that you can use it's clean headers with GLEW-based loading. In both cases, the loading calls will be forwarded to the other library, while the basic stuff will be handled here.

Alfonse Reinheart
10-30-2012, 10:24 AM
OK, so version 0.3 is up, which is mostly just a bug-fix release. I've finished a GLee-style loader (ie: no having to call a specific function to load function pointers), but it's not in 0.3 yet; it'll be going into the next release.

The main issue is that... I've kinda created something more than just a loader generator.

The differences between 0.2 and 0.3 are fairly minor from the outside perspective, just some bug fixes. The real changes are internal.

Basically, in creating a flexible system for building loaders, I had to create a flexible system for arbitrarily processing the .spec files. Which means that you can use it to build anything from that data. You could generate Python bindings. Lua bindings. Java or C# bindings. Pretty much anything you wanted, all in a relatively few lines of code.

I'm not sure where to go with this from here. I've documented some of what you need to know to yoke the potential of it (https://bitbucket.org/alfonse/glloadgen/wiki/Style_Creation), but is this something that people will actually use? Should I be advertising it as a general processing system, or contacting the OpenTK/LWJGL/etc people to let them know that I've basically done half their jobs for them?

What can be done with this system? Is this something people even want to do?

kyle_
10-30-2012, 11:28 AM
I had to create a flexible system for arbitrarily processing the .spec files.

Cool. I mean, really thats _almost_ what i need right now ;)
Now the sad part is, there are no spec files for EGL/GLES ...

nigels
11-03-2012, 04:31 PM
Now the sad part is, there are no spec files for EGL/GLES ...

There are some databases for EGL and GL ES as part of Regal
located at https://github.com/p3/regal/tree/master/scripts

See egl.py and gl.py in particular.

- Nigel

nigels
11-03-2012, 04:38 PM
The main issue is that... I've kinda created something more than just a loader generator.


That sounds familiar! :-)

I think I ought to mention here that GLEW does have a "subset" mode (undocumented, and
in a separate branch) which allows for opting-in on an extension-by-extension basis. This
has been especially useful for statically linking small applications.

An alternative approach that I have in mind for GLEW and Regal is to implement a pseudo
pre-processor to take a vanilla/stock GLEW, Regal or whatever loader and strip out all
the irrelevances such as vendor extensions, things long merged into core, etc etc.
Somehow I find it a bit more appealing to have a proliferation of alternatives derivable
from a complete source distribution, than having the code generation scripts handle
all the scenarios.

Anyway, the main suggestion I have for GL loading is to support Core context well,
it's a long-standing limitation of GLEW.

- Nigel

Alfonse Reinheart
11-03-2012, 08:20 PM
Somehow I find it a bit more appealing to have a proliferation of alternatives derivable
from a complete source distribution, than having the code generation scripts handle
all the scenarios.

A code generator is much easier to maintain. You way, you have to maintain a code generation system (to generate the original source) and this pre-processing system that removes the extra stuff.

Also, it's very easy to screw up the pre-processing system due to a simple lack of the proper information. For example, how do you decide to include the enums for ARB_tessellation_shader? Remember: it's a 4.0 core extension, so it's very much part of 4.0+. But it's also an extension.

The GLEW headers, for example, consider it an extension; it lives under a comment that says "GL_ARB_tessellation_shader". No mention is made of the fact that it's also core 4.0+ functionality and should be included in any headers that use 4.0+ regardless of whether the user asks for it or not. Without knowledge of what is and is not a core extension, your ability to get clean headers from 3.0+ with this system is diminished.

The OpenGL spec files, the source that the GLEW headers are built from, actually has this info. Enums from ARB_tessellation_shader are specifically stated to come from the extension and the version.

GLEW headers are a lossy code generation system. Generating code from such a system is probably not the best idea.

Granted, the .spec files alone don't have enough information to do this properly either. Enums are correctly classified, but functions are not. Which is why my loader systems have to keep a manually-updated list of what the core OpenGL extensions are.

Even so, it's still a lot easier to make this stuff from first principles than to parse someone else's headers.


Anyway, the main suggestion I have for GL loading is to support Core context well,
it's a long-standing limitation of GLEW.

GLEW is pretty much the only extension loader still supported today that doesn't support core contexts. All of my styles handle core contexts.

nigels
11-04-2012, 07:31 AM
A code generator is much easier to maintain.

I'd say - that depends. What we're finding with Regal are a whole lot of use-cases
that are fairly platform specific, along with the general desire to slim things in
various ways. Diagnostics in debug mode, for example.


For example, how do you decide to include the enums for ARB_tessellation_shader? Remember: it's a 4.0 core extension, so it's very much part of 4.0+. But it's also an extension.

You're right - it depends on having things sufficiently #ifdef-ed for preprocessing to work.
With GLEW, for example, I'd rather merge the subset patch to the mainline and preprocess
it away again, for release. Rather than complicate the code generation with the conditionals.

Regal already has a bunch of compile-time options, so pre-processing just ought to be equivalent
to whatever would have been specified for the compiler.

- Nigel