More GPU Raycasting Questions

I’ve also been messing around with Peter Peter Trier’s GPU Raycasting setup, but I thought it would be better if I started my own thread, rather than hijacking herpaul’s earlier one, since I’m taking a slightly different tack.

I’ve got Peter’s setup semi-working as an isosurface renderer. Well, I say ‘semi-working’: it looks quite interesting, but is also quite broken.

As you can see from the screenshots below, there’s definitely a sense of depth, but the view ‘through’ each face of the cube doesn’t match up with the views through the other faces. Also, as the cubes are rotated, the the apparent depth of the isosurface, as viewed through each face decreases, until as the face is viewed more obliquely, the appearance is is a 2D slice through the isosurface, stuck to the surface of the face.
(The small graphics at bottom-left represent the front and back faces of the cube, and the vector created by subtracting colour values, as described on Peter’s blog).

Unfortunately, I don’t have access to a Windows PC, so I’ve not been able to see Peter’s demo application in action, so I don’t know if the ray volume is meant to be rotated. On the other hand, even when viewed without rotation, my version still doesn’t look quite right.




Here’s the shader code:

// VERTEX SHADER

varying vec4 pos;

void main()
{
	//Transform vertex by modelview and projection matrices
	gl_Position = gl_ModelViewProjectionMatrix * gl_Vertex;
	
	pos = gl_Position;
	
	//Forward current color and texture coordinates after applying texture matrix
	gl_FrontColor = gl_Color;
	gl_TexCoord[0] = gl_TextureMatrix[0] * gl_MultiTexCoord0;
}

// FRAGMENT SHADER

uniform sampler2D BackBuffer, VolTex;
uniform float StepSize;
uniform float IsoValue;
uniform vec4 IsoColor;
uniform vec3 IsoScale;
uniform vec3 IsoOffset;
uniform vec2 IsoThreshold;

varying vec4 pos;

void main()
{
	// Find the right place to lookup in the backside buffer
	vec2 texc = ((pos.xy / pos.w) + 1.0) / 2.0;
	vec4 start = gl_TexCoord[0];
	vec4 back_position = texture2D(BackBuffer, texc);
	vec3 dir = back_position.xyz - start.xyz;
	// The length from front to back is calculated and used to terminate the ray
	float len = length(dir.xyz);
	vec3 norm_dir = normalize(dir);
	float delta = StepSize;
	vec3 delta_dir = norm_dir * delta;
	float delta_dir_len = length(delta_dir);
	vec3 vec = start.xyz;
	vec4 col_acc = vec4(0.0);
	float alpha_acc = 0.0;
	float length_acc = 0.0;
	vec4 color_sample = vec4(0.0);
	float alpha_sample = 0.0;
	
	// Main raycast loop
	for(int i = 0; i < 450; i++)
	{
		vec3 vectmp = (vec * 2.0 - 1.0) * IsoScale + IsoOffset;
		
		// Borg surface formula from Paul Bourke's site
		// http://local.wasp.uwa.edu.au/~pbourke/geometry/borg/
		//float borg = sin(vectmp.x * vectmp.y) + sin(vectmp.y * vectmp.z) + sin(vectmp.z * vectmp.x);
		
		float blob =  pow(vectmp.x,2.0) + pow(vectmp.y,2.0) + pow(vectmp.z,2.0) +
					sin(4.0*vectmp.x) + sin(4.0*vectmp.y) + sin(4.0*vectmp.z) - 1.0;
		
		color_sample.rgb = IsoColor.rgb;
		//color_sample.a = (borg  > IsoThreshold[0] && borg < IsoThreshold[1]) ? 1.0 : 0.0;
		color_sample.a = (blob  > IsoThreshold[0] && blob < IsoThreshold[1]) ? 1.0 : 0.0;
		
		alpha_sample = color_sample.a * StepSize;
		col_acc   += (1.0 - alpha_acc) * color_sample * alpha_sample * 3.0;
		
		alpha_acc += alpha_sample;
		vec += delta_dir;
		length_acc += delta_dir_len;
		
		// Terminate if opacity > 1 or the ray is outside the volume
		if(length_acc >= len || alpha_acc > 1.0) break;	} 
	
	//Multiply color by texture
	gl_FragColor = vec4(col_acc);
}

The shader runs on the cube with backface-culling applied, taking the frontface-culled cube rendering as input to the ‘back_buffer’ uniform.

I’m sure there’s something simple I’m doing wrong, but I’ve no idea what.

Any help would be much appreciated!

Cheers,

a|x
http://machinesdontcare.wordpress.com

It does look like your texture coordinates are messed up a bit, though it’s a bit hard to see, try generating a volume texture that uses a bit simpler shapes.

It looks like maybe texture wrapping is set to GL_REPEAT try GL_CLAMP_TO_EDGE

Hi zeoverlord,

thanks for getting back to me!

I just tried manually clamping the texture coordinates for the texture-lookup to the 0 > 1 range, and this didn’t seem to make any difference, unfortunately. I agree though: I think the problem must be something to do with the texture-lookup, but I can’t work out what.

Or maybe it’s to do with the ‘color-cube’ rendering… I’m simply mapping cube vertices XYZ coordinates to RGB, with a 0.5 offset to bring them into the 0 > 1 range. Maybe this is where I’m going wrong…

I’d love to know if this setup should allow rotation of the volume.

Unfortunately, I can try rendering an actual 3D texture, as the application I’m using (Apple’s Quartz Composer) can’t deal with them. The eventual intention is to write a custom plugin for the application, to allow rendering of volume textures using this method, which is partly why I’m trying to get it running now, using QC as a prototyping environment, really.

a|x

Also, as the cubes are rotated, the the apparent depth of the isosurface, as viewed through each face decreases, until as the face is viewed more obliquely, the appearance is is a 2D slice through the isosurface, stuck to the surface of the face.

This is due to ray marching algorithm. To attenuate the 2D slices appearance, use a smaller step along the ray and I guess there is something wrong in your opacity and attenuation calculations.

Hi dletozeun,

ah… that makes a lot of sense. I will look into this. Peter’s setup wasn’t designed for rendering isosurfaces, so maybe there’s a problem with the way I’m using alpha-channels in my version.

Do you think the problem is to do with the way rays are cast from all the front faces of the cube, and I guess interact with each other in some way?

As a general rule, should I be rendering the volume onto the front-facing faces of a cube, or should I be using a plane? The writeup on the blog seems unclear on this.

Sorry to ask more questions,

a|x

Ah… I think it may be to do with the way the texture coordinates are setup on Quartz Composer’s Cube primitive. It looks like each face of the cube has texture coordinates going from 0 > 1 in the X and Y directions. Would this break the shader, do you think?

a|x

Ah… I think it may be to do with the way the texture coordinates are setup on Quartz Composer’s Cube primitive. It looks like each face of the cube has texture coordinates going from 0 > 1 in the X and Y directions. Would this break the shader, do you think?

I have never done gpu ray-casting yet and do not use Quartz Composer. Looking through the article, you need to set 3d texture coordinates to fetch a 3D texture, am I right?

Hi dletozeun,

thanks very much for having a look.

That’s the gist of Peter’s article. I’m trying to do something slightly different, ie use the 3D coordinates of the ray as it is cast through the volume as inputs to a 3D function and plotting the result as opacity values within the volume.

The texture coordinates of the front-faces of the cube are used as the starting position of the rays in Peter’s shader, but I think this may be causing problems when the volume is rotated.

The information on his blog is confusing on this, I think. He says

The origin is just the frontface values of the cube. So we have to do two renderpasses one for the front and one for the back. To render the backside we enable frontface culling. In my implementation i use a OpenGL framebuffer object (fbo) to store the result from the rendering of the backside, and use the frontface rendering to generate the fragments that starts the raycasting process

Which seems to imply he’s using the color-values of the frontface rendering, but in his shader, he simply uses the texture coordinates themselves.

I’m unclear as to whether I should be using the cube geometry to render the raycast or a simple quad. If I’m using a quad, I’d presumably need to take both frontface and backface-culled textures as inputs, and implement some kind of mechanism to prevent rays being cast from areas of the frontface texture representing ray-start positions outside the volume. There’s no sign of this in Peter’s shader, though…

Very confused…

a|x

Hmm, after a quick read of this article, i understood that to compute the ray direction, he needs to know the position of the ray entry point in the cube volume and the position of the ray exit point from the cube for each fragment.

Thus, he has to give two 2d texture to its shader, the first that contains all cube entry point positions and the second, all the cube exit points ( Note that imply to use 16 bit or 32 bit floating point textures).

To render these textures, he proceeds in two passes, the first with backface culling to render the front of the cube, and the second with frontface culling to render the back of the cube.
He says that he use a fbo to do this and I assume he also use a simple shader that writes into the texture each fragment world position.

That’s what had me confused, in fact. If you look at the shader code, though, he only has one sampler2D input, for the backface_buffer.

To render these textures, he proceeds in two passes, the first with backface culling to render the front of the cube, and the second with frontface culling to render the back of the cube.
He says that he use a fbo to do this and I assume he also use a simple shader that writes into the texture each fragment world position.

Yep, I think that’s correct. Except I think he’s using the raw vertex coordinates of the cube (in model-space? - I’m hazy on the terminology, which seems to vary anyway) rather than the coordinates in World/Eye space.

I’ve managed to get it to work, I think.
I used a plane quad to render the rays, rather than attempting to render them onto the faces of a cube, and instead of using the texture coordinates of the quad as ray start positions, I used the color-values of the backface-culled cube. I also added some code to discard fragments based on the alpha of the frontface texture, to avoid calculation of rays that start outside the volume. I’m sure there are other optimisations that can be made, too (if you’ve got any suggestions, I’d love to hear them).

Here’s my version:

/////////////////////
//  Vertex Shader  //
/////////////////////

varying vec4 pos;
uniform float ScaleMultiply;

void main()
{
	//Transform vertex by modelview and projection matrices
	gl_Position = gl_ModelViewProjectionMatrix * gl_Vertex;
	
	pos = gl_Position;
	pos.xy *= ScaleMultiply;

	gl_TexCoord[0] = gl_TextureMatrix[0] * gl_MultiTexCoord0;
}

/////////////////////
// Fragment Shader //
/////////////////////

uniform sampler2D RayStart, RayEnd;

uniform float StepSize;
uniform float IsoValue;
uniform vec4 IsoColor;
uniform vec3 IsoScale;
uniform vec3 IsoOffset;
uniform vec2 IsoThreshold;
uniform float Opacity;

varying vec4 pos;

void main()
{
	// Find the right place to lookup in the front and backside buffers
	vec2 texc = ((pos.xy / pos.w) + 1.0) / 2.0;
	
	// Ray start position from cube front-face render
	vec4 start = texture2D(RayStart, texc);
	
	// Determine if ray start-position is inside volume to be rendere
	if(start.a != 0.0) {
		// Lookup ray end in backface buffer
		vec4 back_position = texture2D(RayEnd, texc);
		// Ray vector
		vec3 dir = back_position.xyz - start.xyz;
		// The length from front to back is calculated and used to terminate the ray
		float len = length(dir.xyz);
		vec3 norm_dir = normalize(dir);
		float delta = StepSize;
		vec3 delta_dir = norm_dir * delta;
		float delta_dir_len = length(delta_dir);
		// Variables to hold accumulated color, alpha, ray-length
		vec3 vec = start.xyz;
		vec4 col_acc = vec4(0.0);
		float alpha_acc = 0.0;
		float length_acc = 0.0;
		vec4 color_sample = vec4(0.0);
		float alpha_sample = 0.0;
	
		// Main raycast loop
		for(int i = 0; i < 450; i++)
		{
			vec3 vectmp = (vec * 2.0 - 1.0) * IsoScale + IsoOffset;
			color_sample.rgb = IsoColor.rgb;
			
			//float blob =  pow(vectmp.x,2.0) + pow(vectmp.y,2.0) + pow(vectmp.z,2.0) +
					sin(4.0*vectmp.x) + sin(4.0*vectmp.y) + sin(4.0*vectmp.z) - 1.0;
			//color_sample.a = (blob  > IsoThreshold[0] && blob < IsoThreshold[1]) ? 1.0 : 0.0;
		
			// Borg surface formula from Paul Bourke's site
			// http://local.wasp.uwa.edu.au/~pbourke/geometry/borg/
			float borg = sin(vectmp.x * vectmp.y) + sin(vectmp.y * vectmp.z) + sin(vectmp.z * vectmp.x);
		
			color_sample.a = (borg  > IsoThreshold[0] && borg < IsoThreshold[1]) ? 1.0 : 0.0;
		
			alpha_sample = color_sample.a * StepSize;
		
			col_acc   += (1.0 - alpha_acc) * color_sample * alpha_sample * Opacity;
		
			alpha_acc += alpha_sample;
		
			vec += delta_dir;
			length_acc += delta_dir_len;
		
			// Terminate if opacity > 1 or the ray is outside the volume
			if(length_acc >= len || alpha_acc > 1.0) break;		}
	
	// Return fragment color
	gl_FragColor = vec4(col_acc);
	
	} else {
		// Ray start-position is outside volume
		discard;
	}
}

Here are a couple of screenshots of it in action:

I’m sure there is more optimisation to do done on the main raycasting loop, and I’m still a little confused as to how to scale the whole thing up and down, but it’s kinda working as it is.

Thanks once again for all your help on this,

a|x
http://machinesdontcare.wordpress.com

That’s what had me confused, in fact. If you look at the shader code, though, he only has one sampler2D input, for the backface_buffer.

Actually I think he does the final rendering on the cube front faces. In the second pass you can use the cube back faces position from the texture sampler, rendering the cube front faces and computing raymarching algorithm in the same pass.

I think you’re right, that’s what he does. I didn’t have any luck with that method though, for some reason. I think it might be to do with the way texture-coordinates are setup on the Cube primitive in Quartz Composer. I notice Peter just passes the texture coords through in the vertex shader, and uses them in the fragment shader to start the rays, but if I do the same, it doesn’t work for me. I guess I’ll just have pass another varying from the VS that mimics the colour-values I applied to the faces of the colour-cubes.

Thanks for unpicking Peter’s code for me. I’ll let you know how I get on swapping the quad for a cube again, and eliminating that extra texture-lookup.

Incidentally, there are some screenshots and videos of my quad-based setup on my blog now
http://machinesdontcare.wordpress.com/2009/06/15/gpu-raycasting-now-working/

Cheers,

a|x

Nice work! It looks very artistic. :slight_smile:

Though I am not sure to understand why you did not succeed to make it work with the cube.
What about performances, does it run smoothly? I would do kinda nice screensaver! :wink:

Cheers dletozeun!

I think I understand why it didn’t work with a cube initially. It’s to do with the fact that the cube primitive I was working with has texture coords going from 0 to 1 across each face, like this:

Whereas what I needed was something more like this:

So obviously with the cube’s original texture coords, the rays were starting in completely the wrong places.

The eventual code I came up with is this:


/*
	Volume raycaster VS
	Based on code by Peter Trier
	
	http://www.daimi.au.dk/~trier/?page_id=98
	
	Quartz Composer setup
	toneburst 2009
*/

varying vec4 vClipPos;	// Interpolated vertex-position in Clip (screen)-Space
varying vec4 vPos;		// Interpolated Vertex-position in object-space

void main()
{
	//Transform vertex by modelview and projection matrices
	gl_Position = gl_ModelViewProjectionMatrix * gl_Vertex;
	
	vClipPos = gl_Position;
	
	vPos = gl_Vertex + 0.5;
	vPos.w = 1.0;
	
	gl_TexCoord[0] = gl_TextureMatrix[0] * gl_MultiTexCoord0;
}

////////////////////////////////////////////////
////////////////////////////////////////////////

/*
	Volume raycaster FS
	Based on code by Peter Trier
	
	http://www.daimi.au.dk/~trier/?page_id=98
	
	Quartz Composer setup
	toneburst 2009
*/

uniform sampler2D RayStart, RayEnd;

uniform float StepSize;
uniform float IsoValue;
uniform vec4 IsoColor;
uniform vec3 IsoScale;
uniform vec3 IsoOffset;
uniform vec2 IsoThreshold;
uniform float Opacity;
uniform float RayPosMix;

varying vec4 vClipPos;	// Interpolated vertex-position in Clip (screen)-Space
varying vec4 vPos;		// Interpolated Vertex-position in Object-Space (0 > 1 range)

void main()
{
	// Ray start position
	vec4 start = vPos;
	// Find the right place to lookup in the backside buffer
	vec2 texc = ((vClipPos.xy / vClipPos.w) + 1.0) / 2.0;
	// Lookup ray end in backface buffer
	vec4 back_position = texture2D(RayEnd, texc);
	
	// Ray vector
	vec3 dir = back_position.xyz - start.xyz;
	vec3 norm_dir = normalize(dir);
	// The length from front to back is calculated and used to terminate the ray
	float len = length(dir.xyz);
	
	// delta is the 3D vector used to move the ray along each increment
	vec3 delta_dir = norm_dir * StepSize;
	float delta_dir_len = length(delta_dir);
	
	// Init variables to hold accumulated color, alpha, ray-length
	vec3 vec = start.xyz;
	vec4 col_acc = vec4(0.0);
	float alpha_acc = 0.0;
	float length_acc = 0.0;
	vec4 color_sample = vec4(0.0);
	float alpha_sample = 0.0;
	
	// Main raycast loop
	for(int i = 0; i < 200; i++) {
		
		// Scaled and offset ray position
		vec3 vectmp = (vec * 2.0 - 1.0) * IsoScale + IsoOffset;
		
		color_sample.rgb = IsoColor.rgb;
			
		// The Blog surface, by Paul Bourke
		// http://local.wasp.uwa.edu.au/~pbourke/geometry/blob/
		float blob =  pow(vectmp.x,2.0) + pow(vectmp.y,2.0) + pow(vectmp.z,2.0) +
					sin(4.0*vectmp.x) + sin(4.0*vectmp.y) + sin(4.0*vectmp.z) - 1.0;
			
		// Set alpha of color_sample at current ray position
		color_sample.a = (blob  > IsoThreshold[0] && blob < IsoThreshold[1]) ? 1.0 : 0.0;
		// Mix-in ray-position as RGB color
		color_sample.rgb += vec.xyz * RayPosMix;
		
		// Scale alpha_sample
		alpha_sample = color_sample.a * StepSize;
		
		// Accumulate color and alpha
		col_acc   += (1.0 - alpha_acc) * color_sample * alpha_sample * Opacity;
		alpha_acc += alpha_sample;
		
		// Accumulate delta and length
		vec += delta_dir;
		length_acc += delta_dir_len;
		
		// Terminate if opacity > 1 or the ray is outside the volume
		if(length_acc >= len || alpha_acc > 1.0) break;
	}
	
	// Return fragment color
	gl_FragColor = vec4(col_acc);
}

Which seems to work quite nicely, giving identical results to the quad method, in fact, but obviously removing the need to render front-faces to a texture and sample them.

I’m still not sure if all the alpha and colour operations in the ray loop are strictly necessary for this implementation, as opposed to actual 3D texture-rendering, but it seems to give a nice result, aesthetically, so I’m leaving them in for the moment.

Performance-wise, it runs quite slowly, really. On my MacBook Pro w/ ATI Radeon X1600 256mb GPU I get around 12fps rendering at 640 x 360 px. Having said that, I’m rendering the rendering (as it were) to a texture, and passing it through a cheap bloom effect, so removing that would probably speed things up a little bit.

I suspect Peter wrote the original shader code with clarity, more than speed in mind (understandably, since it was a tutorial), so I’m sure the code could also be optimised a little. If you have any ideas on any tweaks that could be made to speed things up, I’d love to hear them.

Thanks again for your help and encouragement with this little project.

Cheers,
a|x

Ok, I see. It looks like you were creating 2D texture coordinates instead of 3D one on the first picture.

This kind of algorithm is very time-consuming… you have to play with the number of samples on each ray cast and stop the computation when you realize that the contribution of a ray sample become too low.

Not sure how much difference it will make (depends on how smart the compiler is) but I think you should be able to write

float blob = pow(vectmp.x,2.0) + pow(vectmp.y,2.0) + pow(vectmp.z,2.0) + 
   sin(4.0*vectmp.x) + sin(4.0*vectmp.y) + sin(4.0*vectmp.z) - 1.0;

as

float blob = dot(vectmp, vectmp) + 
   dot(sin(4.0*vectmp), vec3(1.0)) - 1.0;

That’s the way texture coords are setup on the cube primitive in Quartz Composer. It’s setup to allow a different image to be displayed on each cube face, so UVs go from 0 to one across each face.

This kind of algorithm is very time-consuming… you have to play with the number of samples on each ray cast and stop the computation when you realize that the contribution of a ray sample become too low.

The shader already does some of that. It terminates rays as they leave the volume, and if the accumulated opacity reaches 1.

The problem with rendering dynamic equations (rather than volume textures that could be pre-processed) is that it’s hard to work out in advance if a particular ray is likely to accumulate 100% opacity before it leaves the volume.

In fact, with the Borg and Blog equations, a lot of the rays go all the way through the volume without accumulating enough opacity to terminate before reaching the back face. Some rays in fact don’t hit anything as they move through the space.

Not sure if there’s a way of getting around this, except maybe rendering a ‘pre-pass’ with a much larger step-size, then only rendering rays that accumulated more than a particular opacity amount on the previous pass. Dunno how this would work in practice though (or if it would).

a|x

Hi Lord crc

float blob = dot(vectmp, vectmp) +
dot(sin(4.0*vectmp), vec3(1.0)) - 1.0;

That’s cool, I will give that a go!

Could you do something similar with


float borg = sin(vectmp.x * vectmp.y) + sin(vectmp.y * vectmp.z) + sin(vectmp.z * vectmp.x);

perhaps…

Wish I was better at maths… :frowning:

a|x

Perhaps

float borg = dot(sin(vectmp * vectmp.yzx), vec3(1.0))

Though benchmark this, as I’m making assumtions here, and I have no idea if they hold or not :slight_smile:

It’s not magic, it just takes a bit of practice. Oh and read the spec, it details how each operator/function works :slight_smile:

You’re a genius!

I’ll give that a go, too.
I’ve got the GLSL Quick Reference pdf on a hair trigger, so I know basically what a lot of the builtin functions do. It’s understanding how/where to use a particular function that I’m a little hazy on. I’m picking stuff up though, thanks in no small part to assistance from you kind people.

Cheers,

a|x