Reading your post again, it’s definitely my bad reading comprehension, not your clarity. My apologies.
With instancing, the vertex shader is run multiple times over the vertices of the same base geometry (say a tree model). It’s up to you to vary certain elements of the model to achieve your desired effect, which is minimally the position of the model. Say all you want to do is translate your tree model. Using instanced arrays, you’d set up a vec3 vertex array which contains the various positions of the tree in addition to the vertex array containing the vertices of the tree model. Then you’d use glVertexAttribDivisor() to set the divisor to 1 for that attribute (named “instance_translate”, for example). This causes that vertex input to advance only one vec3 for each instance of the entire tree drawn. Then, in your shader, you’d do something like:
#version 330
in vec3 tree_vert; // tree model vertices
in vec3 instance_translate; // instance positions
uniform mat4 modelviewproject;
void main()
{
gl_Position = modelviewproject * vec4( tree_vert + instance_translate, 1.0);
}
Any geometry or fragment shaders would remain the same as for a non-instanced draw call - you’re going to be passing any instance-varying properties as outputs to the geometry or fragment shader.
Now, if you also wanted to tint the diffuse texture by some color, you’d create another VBO for the per-instance tint and also set the vertex attribute divisor to 1 for that array (if you wanted groups of 4 to share the same tint, you’d use 4 instead). Let’s say this vertex array contains a vec3 color to use as the tint. The vertex shader would then look something like:
#version 330
in vec3 tree_vert;
in vec3 instance_translate; // instanced
in vec3 tint; // instanced
out vec3 diffuse_tint;
uniform mat4 modelviewproject;
void main()
{
diffuse_tint = tint;
gl_Position = modelviewproject * vec4( tree_vert + instance_translate, 1.0);
}
Now generally you have about 16 vertex attribute locations to work with, and if you wanted to do a full transform matrix (mat4), this would take up 4 vertex attribute locations. So if you want to vary a lot of material properties, you may need to start packing them up into a single VBO (such as packing diffuse tint and specular gloss into a vec4) to avoid running out of vertex attribute locations.
If you are varying your instances per frame, this might be a lot of data to send to the GPU. If your varying instances are an ever-changing subset of a set of static instances, you might want to use a different approach. You could create a single instanced integer array containing the instance numbers to render, and then put all your diffuse_tint, transform and other material properties in constant Texture Buffer Objects (TBOs) that you only fill once. This way you only need to refill and upload the index array each frame instead of multiple material properties arrays per frame. As an example:
#version 330
in vec3 tree_vert;
in int instance_index; // instanced
out vec3 diffuse_tint;
uniform samplerBuffer instance_translate;
uniform samplerBuffer tint;
uniform mat4 modelviewproject;
void main()
{
diffuse_tint = texelFetch( tint, instance_index).rgb;
gl_Position = modelviewproject * vec4( tree_vert + texelFetch( instance_translate, instance_index).xyz, 1.0);
}
You can also use the built-in GLSL vertex shader variable gl_InstanceID to index the TBOs. This is useful if you want to keep your shader GL3.2 compatible, since instanced arrays are a GL3.3 feature (OSX is currently limited to GL3.2).
As always, you need to benchmark your specific cases to figure out which approach works best for your application.