No more mapping layers, or arbitrary ones

Vertex inputs and fragment outputs are all defined in programs by strings (the name of the input/output variables). But because vertex attribute bindings and FBO bindings are done by integers, not strings, there is this mapping that has to be put into programs to map from vertex inputs and fragment outputs to attribute buffer indexes.

This is, in a shader world, a waste of time. The problem is that attribute bindings and FBO bindings are numbers rather than strings.

I’m sure that this is one of those things that isn’t going to be changed because it isn’t a big enough deal for the ARB to actually fix, but it is highly annoying. It makes autogenerated attribute bindings in programs meaningless, since you can’t rely on the values. So you can’t autogenerate attribute bindings if you want to use the same VAO with different programs. Though less of a concern for FBOs, since you won’t be having that many of them, it causes the same problem.

The right solution is to bind FBO attachments and VAO attributes to strings rather than numbers. However, that’s probably too radical… for some reason. So failing that, why not get rid of the limitation on the attribute and buffer indices?

After all, the attribute index is just a number. So long as you don’t bind more than MAX_ATTRIBUTES of then in a single VAO or program, it doesn’t really matter that the indices you used are 50, 329, 65536, and 2^20. What matters is that the number stored in the program object and the number stored in the VAO match. This doesn’t require a single API change; it’s just a re-specification of what numbers you are allowed to use.

The FBO issue is a bit more complex, because of the way that glDrawBuffers is defined. So doing it there would require an API change; rather than taking an array of entries, it would have to take two arrays of equal length of entries. Or even better, just have a glLinkBuffer or something that takes a number and an FBO/default framebuffer enum, much like you make a call for each attribute in a VAO.

Remember: the point of this is to allow the user to be able to take a VAO of unknown origin, a shader of unknown origin, and render with them by strings alone. If they happen to be compatible (the VAO uses the same attribute names as the shader), then it works. Allowing any integer solves the problem by allowing the user to make a simple hash-function for the strings, and assign program attribute names based on that. And the same goes for FBOs.

I would prefer having new entripoints to just use strings directly, but I suspect the ARB wouldn’t be interested in that. Maybe this is more palatable.

Perhaps I am missing something here,

  • glBindAttribLocation() is called prior to glLinkProgram() which maps names to a attribute “index”

  • program uses glEnableVertexAttribArray(index) and then glVertexAttribPointer() to setup VBO for attribute “index”.

So what is stopping someone from manually insuring that multiple shaders have specific names bound to specific “index”, so that you could use a VAO and shader of unknown origin. You would simply have to insure prior to link time that you do glBindAttribLocation() correctly.

So what is stopping someone from manually insuring that multiple shaders have specific names bound to specific “index”, so that you could use a VAO and shader of unknown origin.

It’s simple.

Let’s say you have two programs. And each uses the same set of vertex attributes. If you want to use the same VAO with both, you must manually ensure that both programs use the exact same indices for the exact same strings.

The simplest way of doing this is to make a hash-table where you assign each string a number. This is relatively easy. The problem is that GL as it currently stands only allows 16 different attribute integers, which means that if you’re using a lot of different attribute names, can easily cause collisions. And what happens if a shader uses two attribute names that happened to collide in the hash table? Badness.

The whole “bind strings to numbers” thing is an artifact that has long outlived its usefullness. Attributes and Output should just be strings. But since the ARB is unwilling to make real changes to the API, then at least they can make it possible to uniquely map arbitrary strings to arbitrary indices.

I’d rather have to do this manually from application side and not have it in the driver. IMO driver should expose physical index as it currently does and leave it up to the application to layout names/variables to index.

I’d rather have to do this manually from application side and not have it in the driver. IMO driver should expose physical index as it currently does and leave it up to the application to layout names/variables to index.

Part of the purpose of having an API is to abstract away needlessly complex or arcane hardware limitations. Drivers don’t force you to manage video memory or other such things. So why do they force you to manage attribute indices?

Actually, now that I think about it, this really isn’t going far enough. They need to do away with the mapping layers entirely and just use strings.

Actually, I’d rather have access to manage my own memory on the GPU. It is what I do for the consoles, and well something I really don’t like when I get back to PC/Mac land.

But seriously why not have this string to index mapper in some outside of GL userland API.

They need to do away with the mapping layers entirely and just use strings.

So you want something like:


glAttribSource("bitangent", m_vbo_Id, m_bitan_offset, GL_FLOAT, 3);

And then expect the driver to automatically match this attribute 
stream with any 

attribute vec3 bitangent;

it may find in the currently active shader?

This results in a fixed convention (defined by yourself) how attributes are named, both from the streams source side and from the shader source side. But in this case, attribute index numbers and attribute string both work equally well. With indexes it could look like this:



//global for your code 
const GLint BITANGENT_ATTRIBUTE = 4; 

// do this once, enforce your own fixed convention once the shader gets created
glBindAttribLocation(m_shader, BITANGENT_ATTRIBUTE, "bitangent");

//then somewhere else in the code:
glAttribSource(BITANGENT_ATTRIBUTE, m_vbo_Id, m_bitan_offset, GL_FLOAT, 3);


In order to make VAO easier to use, it should not be an error if a VAO provides more or less attributes than the shader needs.
If the shader doesn’t make use of an attribute, then it is just ignored and ideally causes no burden on the GPU. If the shader needs more attribues than the VAO provides, it should be possible to still render something and provide “default” attribute values via glVertexAttrib() calls.

So you want something like:

Basically, yes.

But in this case, attribute index numbers and attribute string both work equally well. With indexes it could look like this:

And what if you’re talking about a file-driven renderer? One where VAO/vertex formats and shaders are defined by files which can reference arbitrary attribute names? One that your program has never seen before?

Unless you sit there and parse every vertex shader to get the attributes and such out of them, you have no way to know what the attributes in a shader are. Which means that you can’t call “glBindAttribLocation” because you don’t know what the attributes are.

I agree that it’s easier to address by string - although it’s extremely easy to add that mapping function to your own code.

I think where you get an advantage with the current approach is once you cache those indices for subsequent render passes. If the shaders are not recompiled and relinked, no relatively expensive search needs to be done.

I think your example of a file-driven renderer fails because the setup code will still have to understand what that attribute is supposed to mean before binding to it. Convention can take you only so far.
That translation could be done in a domain specific language, included in the files, and interpreted at runtime, but the mapping would still have to be made ‘manually’, by whoever designed the shader.

I support T101’s approach, though it may not be universal. And besides, you can’t universally slap just any shader to any VAO with good results.
Imho file-driver renderers have no place in a release code. They should stay only in generic (and kinda useless) mesh-viewers. In my surfaces in modelers, I specify the name of shader to be used (i.e “surface1:BumpProgram” or “surface2:BumpShaderVtx:BumpShaderFrag” ). Then, the object-loader fills-in a structure like this:


class XSurface{
public:
	char* Name;
	int   IsDoubleSided;
	
	vec3  baseColor;// these here are frag-uniforms
	float luminosity; 
	float diffuse;
	float specularity;
	float glossiness;
	float reflection;
	float transparency;
	float refractionIdx;
	float translucency;
	float bumpIntensity;

	int	  NumPoly;
	U16*  AllPoly; // AllPoly[NumPoly*3]. Each triplet specifies a vertex-index

	int	  NumVerts; 
	vec3* AllVerts; // AllVerts[NumVerts];
	vec3* AllNorms; // AllNorms[NumVerts];

	int   NumUVCoords;
	vec2* UVCoords[8];


	int	  NumLayers; // number of specified texture, actually
	int   LayerClipID[8];
	float LayerOpacity[8];
	int	  LayerChannelType[8]; // 'COLR', 'DIFF','LUMI','SPEC','BUMP'

	int GetChannel(int ChanType){ // returns 0..7 on success, -1 on failure
		for(int i=0;i<NumLayers;i++){
			if(LayerChannelType[i]==ChanType)return i;
		}
		return -1; // failed
	}
	~XSurface(){
		xfree(Name);
		xfree(AllPoly);
		xfree(AllVerts);
		xfree(AllNorms);
		for(int i=0;i<NumUVCoords;i++)xfree(UVCoords[i]);
	}
};

This structure is filled by OBJ 3DS LWO etc. loaders. An array of XSurfaces is created from an object. Meanwhile, there’s a text-file that specifies per-shader: mapping of vtx-attribs, mapping of surface-variables (like “specularity”) to frag-shader uniforms. Optionally, a vertex-declaration is generated, if we want to put some attribs into i.e half-floats or normalized unsigned bytes. A function gets the XSurface and via those rules generates a nicely-packed array or file, to be used in release.
The artist can easily get a model-viewer, that shows how his meshes will look ingame, and takes only a Ctrl+S,Alt+Tab to view.

off-topic maybe, but what’s LayerClipID? Nothing to do with clipmapping is it?

I agree that it’s easier to address by string - although it’s extremely easy to add that mapping function to your own code.

If it’s “extremely easy”, then write one. Write one that has the following properties:

1: Does not require parsing a shader file for attributes.
2: Does not require re-binding the buffers in a VAO if you use two different program objects with the same VAO.
3: There is no explicit connection between a shader and the VAOs that will use it, thus providing for maximum flexibility.
4: Guarantees that it will always work, regardless of the combination of attribute strings (thus a string hash over 16 possible elements fails).
5: The only time attribute strings are available is when you’re building the VAO.

I think where you get an advantage with the current approach is once you cache those indices for subsequent render passes. If the shaders are not recompiled and relinked, no relatively expensive search needs to be done.

What “relatively expensive search” are you talking about? The search that the driver does to connect VAO names to program names? That “search” doesn’t even have to do string compairs; it can easily just do hashing over the small space of the attributes used in that VAO and program.

The principle disadvantage to the way things are now is that, without having real string mapping, I can’t actually use VAOs. Because my attribute indicies are being assigned by the driver, every time I bind buffers to render, I must query the attribute indices from the shader.

There’s already a parser for glslang; it’s built into the driver. I should not have to parse through a shader to find out what attributes it uses. At this point, the silly D3D method of using “semantics” to define what an attribute in a shader means is preferable; at least that one is automatic and doesn’t require parsing shader code to find the answer.

I think your example of a file-driven renderer fails because the setup code will still have to understand what that attribute is supposed to mean before binding to it.

Why? You have a VAO; it doesn’t care what the meaning of the data is, only what string it is called. The only thing that gives meaning to the data is the shader that uses it.

And besides, you can’t universally slap just any shader to any VAO with good results.

For any given set of attributes, there is a corresponding set of shader programs that use those attributes. So long as you stick to a particular naming convention, a VAO can be used with any program that uses the same attributes.

This structure is filled by OBJ 3DS LWO etc. loaders. An array of XSurfaces is created from an object. Meanwhile, there’s a text-file that specifies per-shader: mapping of vtx-attribs, mapping of surface-variables (like “specularity”) to frag-shader uniforms. Optionally, a vertex-declaration is generated, if we want to put some attribs into i.e half-floats or normalized unsigned bytes. A function gets the XSurface and via those rules generates a nicely-packed array or file, to be used in release.

That requires a lot of extra files just to establish basic conventions (mapping a specularity uniform to a value). Wouldn’t it make more sense to say implicitly, “The uniform named ‘specular’ means X” and just have your code set that named uniform when it changes?

It is certainly a functional way to deal with shaders and meshes, but it is certainly not the only way. Nor should it be the only viable way to do it.

Yup, just make your shaders and VAO use predefined attributes (or these same ones, but with changed names and attribIDs if you wish): gl_Vertex, gl_Normal, gl_Color, gl_TexCoord[0], etc. :D. Sure, there are only 16, which can introduce limitations on mapping, but 16 is enough room. I bet this “remapping” was what T101 meant as “extremely easy”.

That requires a lot of extra files just to establish basic conventions (mapping a specularity uniform to a value). Wouldn’t it make more sense to say implicitly, “The uniform named ‘specular’ means X” and just have your code set that named uniform when it changes?

Just 1 file, containing mappings for all 50-150 shaders. Typing code for each shader can be easy, but when you take into account the vertex-declarations (for those half/ubyte compressions), it turns into an M*N problem or a bit uglier solution.

It is certainly a functional way to deal with shaders and meshes, but it is certainly not the only way. Nor should it be the only viable way to do it.

I think we’d confuse Khronos and IHVs by first screaming for “only 1 fast-path”, and then asking for different ways to do the same thing. Right :slight_smile: ?
P.S.
I do realize that the string-based approach will be just a few extra hundred cycles/bind slower, and ~15k cycles slower at init, still I want “one true fast-path”. And I love semantics and control, so I’m very biased. And I’m already pissed on how some simplistic binding operations already take 8-15 freaking THOUSAND cycles :P.

why do you have to do this?
You can query the active attributes like this…


GLint activeAttribs=0;
glGetObjectParameteriv(hShader, GL_OBJECT_ACTIVE_ATTRIBUTES, &activeAttribs);
	
char attName[256];
GLsizei attNameLen=0;
GLint attSize=0;
GLenum attType=0;
GLint attLocation=0;

for (GLuint i=0; i<activeAttribs; ++i)
{
	glGetActiveAttrib(hShader, i, 256, &attNameLen, &attSize, &attType, attName);

	attLocation = glGetAttribLocation(hShader, attName);
}

I suppose I agree with korval on the fixed range of attrib locations - but i think the string thing is going a bit far.

Ah, no, it’s just TextureIndex, from an array of texture-filenames, that all surfaces/meshes in the current model use.

Hi Korval,

perhaps your issues are with rendering TO a VAO for intermediate results, and subsequently retrieving those intermediate results in another shader, and I can imagine that to be troublesome.

If that is the case, it wasn’t clear to me, so my response was based on the assumption that your VAO attributes are input only.
What I meant by “easy” then is simply calling a get…Location, and if successful followed by the actual bind by ID.
In this scheme you would offer your attributes to the program object - not have the program object ask for the attributes.

W.r.t. “relatively expensive”: you may well argue that calculating a hash function and possibly resolving a collision is cheap enough with modern CPU speeds, but it still takes longer than a single array lookup. If you always did the binding by string, the driver would have no way of eliminating that bit of overhead for you.

Presumably you don’t expose a raw VAO to the user, but you use some sort of 3D object that uses a VAO under the hood.
The class of your 3D object, and not the object itself would presumably define the meaning of an attribute/uniform. The shader would in some way use the attribute or uniform.
Any mapping would in that case have to be done only for the combination of Program object and 3D object class.
This is why the mapping could be cached, and why I think mapping by ID is preferable.

As long as you can control both sides, mapping purely by name will be fine. But what about third parties? (You did mention that ‘file-based renderer’ after all, which suggests extensibility).
That’s where the DSL suggestion comes in. It does require your user to make the mapping between attribute/uniform and shader explicitly, but if your DSL is powerful enough, you don’t need to hardcode as much, which means more flexibility.

Yup, just make your shaders and VAO use predefined attributes (or these same ones, but with changed names and attribIDs if you wish): gl_Vertex, gl_Normal, gl_Color, gl_TexCoord[0], etc.

Oh, you mean the attributes that won’t be there anymore because they’re deprecated.

Furthermore, the entire point of using a C-like language is to make the user’s job easier. If I have to pretend that a binormal or tangent is actually “gl_TexCoord[3]”, that basically throws the whole purpose in using C-style languages out the window. You may as well just use assembly.

If I have need of a new attribute in a shader, it should be no farther away than typing “in float someName;” in the vertex shader and using it. Doing anything more than that shows a problem with the system itself.

I think we’d confuse Khronos and IHVs by first screaming for “only 1 fast-path”, and then asking for different ways to do the same thing. Right :slight_smile: ?

Um, no. Because there would only be one path: using strings. As in no more integers, no more mapping layers. Like the title of the thread says.

Just 1 file, containing mappings for all 50-150 shaders.

And what about new shaders I make?

What you’re talking about is effectively a hash-table that can only have 16 unique entires in it. Once you do that, it is very easy to construct a scenario where shader A uses one set, shader B uses another, shader C uses another, and so on until it is impossible to create a 16-element mapping where two elements do not map to the same values.

That is, if you alias “position3d” with “position2d”, mapping them both to attribute 0, what happens if a shader tries to use both? It isn’t violating some hardware restriction; it’s only using 2 attributes. So you go back and poke at your mapping table and try again, and hope that the next time someone decides to use another set of attributes together, you don’t get a collision.

And this doesn’t deal with the problem of not knowing what attributes a particular shader uses without parsing the shader manually. Without that information, you can’t provide a string to call glBindAttribLocation with.

You can query the active attributes like this…

First of all, none of those GL functions exist. Maybe you’re looking for glGetProgramiv.

More importantly, those only work on a linked, finished program. That is, they do not work before linking, when is the point in time when you can do useful things like set the bindings based on a mapping table. From the spec: “BindAttribLocation has no effect until the program is linked. In particular, it doesn’t modify the bindings of active attribute variables in a program that has already been linked.”

W.r.t. “relatively expensive”: you may well argue that calculating a hash function and possibly resolving a collision is cheap enough with modern CPU speeds, but it still takes longer than a single array lookup.

Um, a hash table is an array.

If you always did the binding by string, the driver would have no way of eliminating that bit of overhead for you.

Nonsense. The absolute worst case is if (for some obscure hardware reason) reassigning indices was somehow a heavy-weight operation. That is, the driver internally assigns new numbers to these when a VAO is rendered with a different program object than was used before. In which case, all the VAO implementation has to do is store a mapping for each individual program that it gets used with. Use a new program, and it gets a new mapping that does the heavy-weight stuff exactly once. If the VAO gets changed, (a new attribute is added, or an old one removed), all the mappings are destroyed.

Presumably you don’t expose a raw VAO to the user

That kinda depends on what you’re making and who the “user” is. OpenGL should not enforce particular expectations on the user.

That’s where the DSL suggestion comes in.

Which exists for the sole purpose of getting around an idiotic API handicap. Basically, you’d be telling users of the tool that they have to jump through some nonsense hoop because the ARB couldn’t be bothered to do things the right way from the start.

You mean glGetObjectParameteriv doesn’t exist in the core, it is part of an old extension that is still widely supported. Not as dramatic as ‘none of those GL functions exist’, but the truth is often mundane.
But you’re right about it only being possible after linkage, but until a shader is linked it would be impossible for the driver to report the information. One approach would be if we could control the locations the linker binds named attributes to. But then again we might as well just use string bindings like you say.

That seems like quite a pretty good idea actually. However, why should it have to be done in the driver? It seems obvious from the replies in this thread that most people don’t have the same requirements as you do and are quite content with using fixed vertex attribute mappings.

Instead of forcing strings and their overhead (they do have an overhead, even with hashing tables) on every GLSL user, it seems pretty sensible to me to do what you suggest in the application. When a VAO container class instance is used with a new shader program, you could create a new VAO mapped from the strings and store it. As a bonus, if you know at some point that a particular VAO-shader program combination will not be used, you can release it from the VAO container class instance.

Anybody who is not implementing something as generic as you (who uses a VAO with a single fixed mapping) would not be slowed down by this mapping management. Besides, I would rather know precisely how the mapping works than rely on IHVs to roll their own potentially buggy/slow implementation.

Oh, you mean the attributes that won’t be there anymore because they’re deprecated.

I’m not in the process of moving any code to GL3, but I saw that and thought it was annoying.
But since these are the old ‘fixed’ attributes, with predefined indices, I guess they’re easy to reinstate in your own code.

Furthermore, the entire point of using a C-like language is to make the user’s job easier.

So you’re a convert to high level shading languages now? :wink:

Um, a hash table is an array.

Yes, indexed by the results of a hash function operating on the string you’re looking for.
And the string has to be compared after that because there is no such thing as a perfect hash function.
And in case of a hash value collision you will either have to rehash or treat every hash table entry as a list.

You can cache the final hash result somewhere (assuming you rehash), but that’s basically the same idea as caching the associated index.
Whether the hash function is called by you or the driver is not important, except if you want to reduce the number of times the function is called.

More importantly, those only work on a linked, finished program. That is, they do not work before linking, when is the point in time when you can do useful things like set the bindings based on a mapping table. From the spec: “BindAttribLocation has no effect until the program is linked. In particular, it doesn’t modify the bindings of active attribute variables in a program that has already been linked.”

This is where you have a point. If you cannot determine the attribute name until after the link, and you cannot bind the attribute after the link, you definitely have a catch-22.

As a workaround, I think you could perform a preprocessing step: compile and link shaders, then use glGetActiveAttrib to retrieve the attribute names.
Since glLinkProgram maps any attributes that have not been mapped explicitly, you end up with a list of attributes.
Then make the mapping and link again.
This does mean linking twice instead of once, but you only have to do it the first time you use that particular vertex format/vertex shader combination.
And it will let the driver do the parsing for you.

That kinda depends on what you’re making and who the “user” is. OpenGL should not enforce particular expectations on the user.

With user I mean the person (presumably) who supplies the original data for the array, not the one who translates this to the vertex array in memory.
(I think we’re in agreement here BTW)

Which exists for the sole purpose of getting around an idiotic API handicap.

Not just that. I suppose you could say it’s true for attributes, though your code may for example be setting certain attributes when loading 3D objects.
For uniforms, such as vectors related to lighting and observer, it allows you much more flexibility without having to recompile. (You will of course end up implementing matrix operations etc as intrinsic functions in that DSL).

Regarding vertex attributes, me being a C/C++ programmer I would personally prefer an explicit vertex format myself, and with some sort of validation step, instead of magically tying everything with the same name together.

I suppose to make it work the way you want, every attribute dereference would have to be kept track of after linking, a list per attribute name, and then the compiled and linked code would have to be patched at calling time to point to the corresponding index in the vertex structure. A bit like a loader.
It’s certainly flexible that way, but I wonder how much of a performance impact that patching step would have.

Question: how will you determine the attribute names in the original vertex array? If you have code that sets up the array, you have a fixed mapping and can just use the current mechanism. If you let the user decide what’s in the attributes, the user will have to specify it somehow. And I think that also allows you to use the current mechanism, doesn’t it?