I am currently rendering between 1000 and 10,000 instances of an 1000+ triangle object using instancing techniques (texture buffer object) and this works just fine. However, at any one time only a fraction of these instances are visible and I would now like to try and experiment with transform feedback and GPU frustum culling to try and generate a list of only visible instances.
My initial thoughts are to render each instance as a GL_POINT and use transform feedback to store the visible instance ID into a buffer object (storing the gl_InstanceID). By emitting the gl_InstanceID into a (texture)Buffer Object I can seamlessly integrate this new step into my existing pipeline and efficiently render the scene no matter how many instances are drawn.
I have no experience with transform feedback, but this is not the area wich I foresee as the main stumbling block. It's the Gemoetry shaders which are something completely new to me and I just don't have a clue how to write one. The goal of the Geometry shader is to frustum cull the Points and emit the gl_InstanceID. Each object instance would have the same radius which could be passed as a uniform parameter to aid in the culling process.
I've Googled a lot and can't really find any decent tutorials on Geometry shaders and/or Geometry view frustum culling. Transform feedback examples are on the net as someone has nicely written the OpenGL Samples pack series of GL 3/4 examples, and there's the Particle system transform feedback link on the front page of OpenGL.org.
So my question is: Can anyone help explain how to implement Gemoetry shader view frustum culling with a mind on instanced rendering, perhaps with some examples.
A secondary help would be if someone could post some reader-freindly geometry shader articles to help idiots like me with the syntax and rules (apart from the geometry_EXT spec).
OpenGL 3.3 compatibility profile
GLSL 1.5/330 compatibility shaders