PDA

View Full Version : Questions about outline fragment shader



felipearts
11-07-2011, 08:58 PM
Hi, excuse me if this is the wrong forum category to post about beginner questions(I am a new to this forum), but my doubts are specifically related to GLSL coding.

I had found and since then worked upon a cartoon outline fragment shader for my game, which acts upon an offscreen buffer which contains a texture of the current camera view(a common techinic in games for many effects); the original code is at the end of post #26 on the following link if you want to check how the code was like( Blenderartists cartoon shader thread (http://blenderartists.org/forum/showthread.php?172282-cartoon-in-GE/page2) )

But what really matters is the code as it is now:

/*
* Toon Lines shader by Jose I. Romero (cyborg_ar)
* Updated, modified and commented by felipearts on 18/08/2011
* Based on blender's built-in "prewitt" filter which is free software
* released under the terms of the GNU General Public License version 2
*
* The original code is (c) Blender Foundation.
*/
//Performance Tips: If possible use * and +(in that order) in the same calculation instead of / and -;
//use dot product and avoid unecessary calculations or info splitting
//(Ex: for a vec4 calculation use split.abcd instead of split.abc and split.d)

#version 120 //BGE GLSL version present when this shader was written

//get external info
uniform sampler2D bgl_RenderedTexture;
uniform sampler2D bgl_DepthTexture;
uniform vec2 bgl_TextureCoordinateOffset[9];

//constant variables representating the fragment distance from camera
//can use a custom uniform from a python file in edgeForce too
const float near = 0.100;
const float far = 100.0;
const float edgeThresh = 0.25;
const float edgeForce = 0.6;

//a custom function, similar to texture2D
float depth(in vec2 coord)
{
vec4 depth = texture2D(bgl_DepthTexture, coord);
return -near / ((-1.0+float(depth)) * ((far-near)/far));
}

//the fragment shader loop
void main(void)
{

//assign these variables now because they will be used next
vec4 sample[9];
vec4 texcol = texture2D(bgl_RenderedTexture, gl_TexCoord[0].st);
float pixZ = gl_FragCoord.z / gl_FragCoord.w;

//gets all neighboring fragments colors
for (int i = 0; i < 9; i++)
{
sample[i] = vec4(depth(gl_TexCoord[0].st + bgl_TextureCoordinateOffset[i]));
}


// The result fragment sample matrix is as below, where x is the current fragment(4)
// 0 1 2
// 3 x 5
// 6 7 8


//From all the neighbor fragments gets the one with the greatest and lowest colors
//in a pair so a subtract can be made later. The check is huge, but GLSL built-in functions
//are optimized for the GPU
vec4 areaMx = max(sample[0], max(sample[1], max(sample[2], max(sample[3], max(sample[5], max(sample[6], max(sample[7], sample [8])))))));

vec4 areaMn = min(sample[0], min(sample[1], min(sample[2], min(sample[3], min(sample[5], min(sample[6], min(sample[7], sample [8])))))));


//The dot below is the same as a sum of the areaMx - areaMn result RGB components, but is more GPU efficient.
//The result is the average difference amount of the RGB(note alpha was left alone) components group
//of the two select fragment samples above.
float colDifForce = ((dot(vec3(areaMx - areaMn), vec3(1)))/0.5);


//Check for heavy RGB difference to darken the current fragment;
//we do not want to mess with transparency, so leave alpha alone
//edgeForce can be changed below to make outline more transparent or opaque
// ? : is the same as if else
colDifForce > edgeThresh ? gl_FragColor = vec4(vec3(texcol*edgeForce), 1.0) : gl_FragColor = vec4(texcol);
}

As you can see I could understand a good portion of it, and made quite an improvement to the original code; The current code does produce the desired outlining if the object in question is near the camera, but the more the object gets far from the camera, as expected, the outline starts being applied to undesired spots, as this fragment shader works upon the fragment depth, and the depth differences between fragments increases as they are farther from the camera.

I then thought which the shader should too be sensitive to the fragment distance to the camera, to compensate the fragment depth at greater distances(operating at the colDifForce > edgeThresh part of the shader). It came to my knowledge about gl_FragCoord; and which the below code outputs the distance of the pixel to the camera:

float variable_name = gl_FragCoord.z / gl_FragCoord.w;

I understood which gl_FragCoord.z is gets the position of the fragment into the window z axis, but my first question is what does gl_FragCoord.w is supposed to represent? And why dividing that by the fragment z position results into the fragment distance to the camera?

Also I suppose the 'near' and 'far' variables from the code start(present since the original code) were meant by the original author to represent the camera start clipping value(where the drawing start) and the end clipping value(where nothing is draw), is my understanding correct about that?

Thanks for any advising into that, I will not bother you with additional questions until the ones above are answered first.

PS: Please explain the best you can, as I am not that experienced in GLSL.

Ed Daenar
11-08-2011, 12:04 PM
What this function:



float depth(in vec2 coord)
{
vec4 depth = texture2D(bgl_DepthTexture, coord);
return -near / ((-1.0+float(depth)) * ((far-near)/far));
}


is doing is linearizing the depth values from the Z buffer. You are right in thinking the near and far should represent your projection's near and far planes. If your points where projected with another set of planes, the result of linearizing the depth will be wrong.

I haven't really checked in depth the shader itself, but I notice you call the values fetched from depth buffer texture "colors". Ofcourse, be my guest and call them what you will, but they are technicaly not meant to be colors but non-linear distances between 0 and 1, where 0 is your near plane and 1 your far plane.

As I said, the function above linearizes the depth I assume to compare the difference values to a fixed threshold. If you didn't linearize it, comparing two values close to the near plane would yield a very small difference and two values close to the far plane would yield a higher difference, even if the linear difference was the same.

As for:


float pixZ = gl_FragCoord.z / gl_FragCoord.w;


I have no idea where you are using it or why you'd want to use it in this specific case, since what you seem to be doing is binding two surfaces to your samplers and rendering on a full screen quad.

However, ignoring the fact that it seems like a useless thing to do here, I'll explain what this means.

gl_FragCoord contains the x,y,z position in screen space. That means the x will go from 0 to ScreenWidth and y will go from 0 to ScreenHeight (where ScreenWidth and Height are your viewport's size). Z, however, will go from 0 to 1.

What does W contain then? To understand this you need to understand how perspective projection works, so I recomend you read a bit on that, but know that the process of projection is a two step one (at least). First, multiplying by the Projection matrix you have set, which you likely do manualy in your shaders and second, dividing by the "perspective correction" value. This is done implicitly by the pipeline, but there's times when you'll need to do it yourself. The "perspective correction" is literaly grabbing the resulting W coordinate and dividing X,Y and Z by it. Pseudo code, where position is a vec4 (in camera space) and projection is your projection mat4x4:



position = projection * position;
position.xyz /= position.w;
position.w = 1/position.w;


What GL will store in gl_FragCoord.w is the original W value inverted. That is, 1/position.w. So what the pipeline will do implicitly after you pass in the transformed position is divide XYZ by W, and store 1/W in the w coordinate. It will also apply the required transformation to go from clip space (-1..1) to screen space.

Edit: (Forgot to continue the idea...)

So, what the original fragment code was intending was to, for some reason, get the Z value of the fragment before the perspective correction. Whatever the use of that was meant to be is up to grabs, as there could be many.

Notice that Z / (1/W) is Z*W, since gl_FragCoord.w contains 1/W (being W the original w from the position after it was multiplied by the projection matrix, as explained above).

felipearts
11-08-2011, 02:24 PM
Thanks very much by the explaining Ed Daenar; I will research on my own about this 'linearization' you spoke of before asking more about that specific piece of code from the original author, as I want to know in details why the math for linearization is specifically that way(-near / ((-1.0+float(depth)) * ((far-near)/far)))


I have no idea where you are using it or why you'd want to use it in this specific case

As I said before, this fragment shader has issues, because if the fragment in question belong to an object which is far from the camera, undesired spots are outlined, because fragments far from the camera have a greater depth(from that depth map texture) difference then the ones near the near plane(at least that is what I think it is happening).

I then thought which I may need for the fragment shader to detect the fragment distance to the camera(using gl_FragCoord.z/gl_FragCoord.w since I saw someone doing this logic that way), so it increases the 'edgeThresh' variable for objects far from the near plane to avoid undesired outlining. I think I need both the 'linearized' depth value and also this fragment distance from the camera and do some calculation with that. Note I had not done that calculation yet, I just already put the variable which I thought I will need(pixZ); I do not know yet how I will do that.

Do you think this logic is valid/the best to avoid undesired outlining at objects far from the near plane? Note which all this outline effect should be done at the fragment shader only(fixed pipeline for the vertex shader only), because it is a limitation at the Game Engine I'm making this shader for.

The image below shows the issue I'm having:
http://img694.imageshack.us/img694/5564/depthissues.th.jpg (http://img694.imageshack.us/img694/5564/depthissues.jpg)

Ed Daenar
11-08-2011, 04:09 PM
I've never bothered to create an outline shader and the closests I've come to do it was by accident when doing a SSAO shader (as they share commonalities), so I won't be solving the issue you have completely. However, I can explain why using gl_FragCoord in this case won't work.

From what I understand, you are escencialy doing deferred shading. Your colored image is fully rendered in a texture and you decide in this post-process shader if you are using the color of the image itself or a black outline. To do this, you use the depth information from your depth texture.

Now consider what exactly gl_FragCoord would have inside in this case. To render this you are using some kind of full screen quad, so each fragment generated by the rasterization of that quad represents one pixel of your viewport. With that in mind, gl_FragCoord will simply carry the information of the quad you just rasterized. Because of that, Z is going to be the same number for every single fragment that comes in.

If you want to play around with the depth values of your fragments "as seen from the camera", you will need to only use the texels you can fetch from the depth texture. The information inside gl_FragCoord is useless for you (in this case, at least).

What I see from the code is that it's mixing two concepts: depth and color. My recomendation is that you trim down these aspects and separate them, until you understand fully the intention of the process. Then you can optimize it in whatever way you wish.

For example:



//gets all neighboring fragments colors
for (int i = 0; i < 9; i++)
{
sample[i] = vec4(depth(gl_TexCoord[0].st + bgl_TextureCoordinateOffset[i]));
}


This is fetching the neigboring depth values of the current fragment (as well as the current fragment's depth), as you ilustrate on the small comment later. However, it's placing the depth value on a vec4. Each of the xyzw components will store the same value. Remove this. Rewrite it so you only fetch the depth as depth, simple flat plain floats, which is what the depth values are.



//From all the neighbor fragments gets the one with the greatest and lowest colors
//in a pair so a subtract can be made later. The check is huge, but GLSL built-in functions
//are optimized for the GPU
vec4 areaMx = max(sample[0], max(sample[1], max(sample[2], max(sample[3], max(sample[5], max(sample[6], max(sample[7], sample [8])))))));

vec4 areaMn = min(sample[0], min(sample[1], min(sample[2], min(sample[3], min(sample[5], min(sample[6], min(sample[7], sample [8])))))));


Same here. Checking min and max on vec4s that contain the same information on each of the xyzw. Just do it with floats and keep on target what you are doing: comparing depths. In this case, finding the minimun and maximun depth values around the current fragment.



//The dot below is the same as a sum of the areaMx - areaMn result RGB components, but is more GPU efficient.
//The result is the average difference amount of the RGB(note alpha was left alone) components group
//of the two select fragment samples above.
float colDifForce = ((dot(vec3(areaMx - areaMn), vec3(1)))/0.5);


This block of code is confusing as hell. areaMx - areaMn is simply finding the difference between max and min, thus why the initial use of vec4s is already completely superfluos. Those could've been floats and you'd get the exact same result here.
Remember, you are comparing depth values. A depth value is a single float.

As to why it's doing a dot product, I have no real idea to be quite honest. What this reads, in simple straight math, is:
EDIT: missread the code. Corrected this.

float diff = areaMx - areaMn;
colDifForce = 1.5 * (diff);

I see no real intention to this block of code, as later on colDifForce is compared with your threshold and used to decide if you paint black or color. You could just altogether skip this and use your simple diff (which simply notes the difference between the max and min depth) and use it as a threshold.

From here on you could tweak and play with these values to see how you actualy want the effect.

My recomendation, when you play with someone else's algorythm which you don't fully understand is to trim it down to the bare working bones and remove superfluos stuff. Then you can work back upwards and add the extra little tweaks that make it better.


One last important note about your linearization function:

your near and far plane are set to



const float near = 0.100;
const float far = 100.0;


If these values are not the same as the actual camera used to render the color texture, all your work comparing depth will be for nothing. A far plane of 100.0 strikes me as very short, although ofcourse this means nothing as you could be considering each unit 1Km for all we know. But just saying, this smells a bit rotten so check that before you start.

Ed Daenar
11-08-2011, 05:18 PM
Ah yes, I realized I forgot something about that linearizing function you are using. It struck me as weird before and although it may indeed be linearizing the values, they are definitely not within 0 and 1. This may or may not cause problems with your algorythm, but you may want to use another way of linearizing them. For example, this snippet:



float LinearizeDepth(float z)
{
const float n = 1.0; // camera z near
const float f = 20000.0; // camera z far
return (2.0 * n) / (f + n - z * (f - n));
}


If you do change this, you'll have to re adjust your thershold.

felipearts
11-08-2011, 07:41 PM
As to why it's doing a dot product, I have no real idea to be quite honest.
I did that, as I read somewhere which its best to always use when possible dot products and multiplications in GLSL calculations, which it is handled faster by the GPU; was that information false? I think I read that at the OpenGL wiki. But as you are right, I will only need a float there, so I think I may not need the dot product there anymore.

About all that inneficience, I hadn't ever noticed it as I was too focused into getting rid of that unwanted outline, but very good point, I optimized the code before, but it seems it was not enough yet


If these values are not the same as the actual camera used to render the color texture, all your work comparing depth will be for nothing.

Do not worry about that, I had not updated the first code post, but I just used GLSL uniforms to pass the game camera current near plane and far plane values to the shader, so the shader will always use the correct value for that two variables.

I will do as you said, then I will post the code for further discussion.

felipearts
11-08-2011, 09:37 PM
Allright, thanks to you I narrowed my options to only the depth buffer, and am almost with a fully functional outline shader:

/*
* Toon Lines shader by Luiz Felipe M. Pereira(felipearts)
* Based on Toon Lines shader by Jose I. Romero (cyborg_ar)
* released under the terms of the GNU General Public License version 2
* updated 09/11/11
*
* The original code is (c) Blender Foundation.
*/
//Performance Tips: If possible use * and +(in that order) in the same calculation instead of / and -;
//use dot product and avoid unecessary calculations or info splitting
//(Ex: for a vec4 calculation use split.abcd instead of split.abc and split.d)

#version 120 //BGE GLSL version present when this shader was written

uniform float near; // The camera clipstart value, know in GLSL as near plane
uniform float far; // The camera clipend value, know in GLSL as far plane
uniform sampler2D bgl_RenderedTexture; // Gets the offscreen texture representating the current camera view contents
uniform sampler2D bgl_DepthTexture; // Gets the offscreen texture representating the current fragments depth
uniform vec2 bgl_TextureCoordinateOffset[9];


const float edgeThresh = 0.015; // Limit value for the fragment to be outlined
const float edgeForce = 0.6; // The force of the outline blackness


// A custom function, which returns the linearized depth value of the a given point in the depth texture,
// linearization seems to be a way of having more uniform values for the fragments far from the camera;
// as it is logical which greater depth values would give greater results, also linearization compensates
// lack of accurancy for fragments distant to the camera, as by default a lot of accurancy is allocated to
// fragments near the camera.
float LinearizeDepth(in float z)
{
return (2.0 * near) / (far + near - z * (far - near));
}

// The fragment shader loop
void main(void)
{

// Assign these variables now because they will be used next
float sample[9];
vec4 texcol = texture2D(bgl_RenderedTexture, gl_TexCoord[0].st);


// Gets all neighboring fragments depths stored into the depth texture
for (int i = 0; i < 9; i++)
{
sample[i] = LinearizeDepth( float( texture2D(bgl_DepthTexture, gl_TexCoord[0].st + bgl_TextureCoordinateOffset[i]) ) );
}


// The result fragment sample matrix is as below, where x is the current fragment(4)
// 0 1 2
// 3 x 5
// 6 7 8


// From all the neighbor fragments gets the one with the greatest and lowest depths and place them
// into two variables so a subtract can be made later. The check is huge, but GLSL built-in functions
// are optimized for the GPU
float areaMx = max(sample[0], max(sample[1], max(sample[2], max(sample[3], max(sample[5], max(sample[6], max(sample[7], sample [8])))))));

float areaMn = min(sample[0], min(sample[1], min(sample[2], min(sample[3], min(sample[5], min(sample[6], min(sample[7], sample [8])))))));


// Gets the average value between the maximum and minimum depths
float colDifForce = areaMx - areaMn;


//Check for heavy depth difference to darken the current fragment;
//we do not want to mess with transparency, so leave alpha alone
//edgeForce variable control the outline transparency, so 1.0 would be full black.
// ? : is the same as if else
colDifForce > edgeThresh * sample[4] ? gl_FragColor = vec4(vec3(texcol*edgeForce), 1.0) : gl_FragColor = vec4(texcol);
}

I thought a bit and then: 'I'm not using the sample[4]...', and as the code became with your help, sample[4] has the current fragment linearized depth, between 0 and 1, so I just got that and multiplied by the edgeThresh. The only issue is which some outlines disappears when the object gets far from the camera, but at least there are no undesired outlines(thought I am still missing something at that last line start).

I hope I'm not bothering you, but can you explain the math of your depth linearization? I know what the variables represents, but I do not know why you get the average distance between the near and far planes and multiply that by the sum of once again near and far planes subtracted by the fragment depth and then divides that by the double of near.

If you did not find bothering at all to explain that, if you can(since it is another author code), please also explain the previous math logic too( -near / ((-1.0+float(depth)) * ((far-near)/far)) )

PS: Good thinking of using a multiply instead of a division at the ColDifForce variable; thought in the end I didn't even remembered what that third value was doing at the ColDifForce original code; I then removed it and no issues happened, so it might be another useless value.

Ed Daenar
11-08-2011, 11:01 PM
About the linearization, it's basicly just redistributing values. I can explain in detail later, but time is short right now. However, this:




The only issue is which some outlines disappears when the object gets far from the camera


Happens because of this:



colDifForce > edgeThresh * sample[4] ? (etc)


See, sample[4] as you said is the current fragment's depth. It will go between 0 and 1, start at 0 on the near plane and incrementing (linearly, since you linearized it previosly) up to 1 as the object goes far away. What that means is that the conditional is asking for a bigger difference as objects go far away.

In other words, edgeThresh * sample[4] grows bigger as the object is further away, because sample[4] also grows.
So, colDifForce needs to be higher and higher to generate an edge.

This change:


const float edgeThresh = 0.00015;
colDifForce > edgeThresh ? gl_FragColor = vec4(vec3(texcol*edgeForce), 1.0) : gl_FragColor = vec4(texcol);


Shold give you about the same results. Notice I lowered edgeThresh significantly. Obviously, I pulled that number out of thin air, just move it a bit up or down, but you definitely needed lower values. Consider that sample[4] would have mostly values close to 0, if your objects were close to the camera.

Finaly, you don't really need to use sample[4] for anything as by sampling surrounding pixels and comparing min and max you are already "detecting an edge", so to speak. Ofcourse, you could use it for something if you so wanted, but shouldn't be necesary.



PS: Good thinking of using a multiply instead of a division at the ColDifForce variable; thought in the end I didn't even remembered what that third value was doing at the ColDifForce original code; I then removed it and no issues happened, so it might be another useless value.


It was a useless value, basicly. You were just scaling the difference for no special reason. I just modified the code to show you what you were doing, but there was no real purpose to that part.

felipearts
11-08-2011, 11:39 PM
(...)What that means is that the conditional is asking for a bigger difference as objects go far away.

That was the idea(it was intentional), because just with the sole edgeThresh comparison there are no changes to the undesired outlines appearing at far fragments issue, I really think I must do some math with edgeThresh and the current fragment depth in order to 'compensate' the value of edgeThresh for the farthest fragments, as just a sole edgeThresh with a fixed value isn't working(I'm open to ideas which may lead to a solution).

I'm not sure, but it may even be which the final issue is with ColDifForce or even both ColDifForce and edgeThresh.

Also take your time and explain later to me that other maths, no rush.

EDIT:

I was about to sleep when I remembered a kind of calculation teached at schools(do not know it's name in english), used to discover a value based on other three; I wanted the edgeThresh to be a proportional value based on the initial value of the near plane(0) and a given initial edgeThresh value, so:
near = initial Threshvalue
fragmentdepth = x

near*x = initial Threshvalue * fragmentdepth

x = ( 0.015 * fragmentdepth ) / near

So changing x for edgeThresh variable, I got a math which was supposed to which edgeThresh value is needed at a given depth based into a given initial edgeThresh value:

edgeThresh = ( 0.015 * fragmentdepth ) / near

I think this logic is quite valid, but yet some edges still dissapear at greater distance, if the outlined fragment is too near to the fragment behind it.

Ed Daenar
11-09-2011, 08:37 AM
Alright, I think I see the problem you are having. I'll describe it:

What happens is that, with the distance, even flat surfaces get shaded and edges get detected, when they really don't exist, so your flat surfaces that span into the screen are getting shaded down.

That's basicly a problem with edge detection on the depth buffer because two adjacent pixels can have a high depth difference despite pertaining to the same surface.

One way you can balance this out without having to do multi pass, which may or may not be enough for your application, is to account for possitive and negative contribution of the differences between the center fragment and the surroundings. Something like this (not guaranteed to compile):



//Current fragment's depth
float base = LinearizeDepth(texture2D(bgl_DepthTexture, gl_TexCoord[0].st).r);

//Accumulator
float colDifForce = 0.0;

for (int i = 0; i < 9; i++)
{
float depthSample = LinearizeDepth( float( texture2D(bgl_DepthTexture, gl_TexCoord[0].st + bgl_TextureCoordinateOffset[i]) ) );

colDifForce += (base - depthSample);

}

//Optional. May generate undesired edges against backgrounds.
colDifForce = abs(colDifForce);

if(colDifForce > edgeThresh)
fragment = vec4(vec3(texcol*edgeForce), 1.0);
else
fragment = vec4(texcol);



Code is neither optimal and, probably, won't even compile but should give you the idea.

Point of it is to combat abrupt depth differences due to far away perspective skewed objects and it's based in the notion that the surface will be balanced out between the fragments forming it in a way that if it has a member fragment that is very far away, it's likely to have a member fragment that is just as close to the camera, thus balancing out the edge detection.

You could also multipass an edge detection filter, but this is probably not what you'd prefer.

felipearts
11-09-2011, 04:36 PM
Never thought of it that way; I thinked the prewitt style sample calculation and difference was vital for this shader, but I did what you told; did it worked? Not at all, but, only if the edgeThresh value is a fixed value as I told before, using the math from my last post it does worked, the shader needed both the way you mentioned and the math I thought before to work, here is the fully functional(apparently) outliner code:

/*
* Toon Lines shader by Luiz Felipe M. Pereira(felipearts)
* Based on Toon Lines shader by Jose I. Romero (cyborg_ar)
* released under the terms of the GNU General Public License version 2
* updated 09/11/11
*
* The original code is (c) Blender Foundation.
*/
//Performance Tips: If possible use * and +(in that order) in the same calculation instead of / and -;
//use dot product and avoid unecessary calculations or info splitting
//(Ex: for a vec4 calculation use split.abcd instead of split.abc and split.d)

#version 120 //BGE GLSL version present when this shader was written

uniform float near; // The camera clipstart value, know in GLSL as near plane
uniform float far; // The camera clipend value, know in GLSL as far plane
uniform sampler2D bgl_RenderedTexture; // Gets the offscreen texture representating the current camera view contents
uniform sampler2D bgl_DepthTexture; // Gets the offscreen texture representating the current fragments depth
uniform vec2 bgl_TextureCoordinateOffset[9];

const float edgeForce = 0.6; // The force of the outline blackness
const float baseThresh = 0.001; // The initial(near value) edge threshold value for inking

// A custom function, which returns the linearized depth value of the a given point in the depth texture,
// linearization seems to be a way of having more uniform values for the fragments far from the camera;
// as it is logical which greater depth values would give greater results, also linearization compensates
// lack of accurancy for fragments distant to the camera, as by default a lot of accurancy is allocated to
// fragments near the camera.
float LinearizeDepth(in float z)
{
return (2.0 * near) / (far + near - z * (far - near));
}

// The fragment shader loop
void main(void)
{

// Assign these variables now because they will be used next
float sample[9];
vec4 texcol = texture2D(bgl_RenderedTexture, gl_TexCoord[0].st);

// Current fragment depth
float base = LinearizeDepth( float( texture2D(bgl_DepthTexture, gl_TexCoord[0].st) ) );

float colDifForce = 0.0;


// Gets all neighboring fragments depths stored into the depth texture
for (int i = 0; i < 9; i++)
{
sample[i] = LinearizeDepth( float( texture2D(bgl_DepthTexture, gl_TexCoord[0].st + bgl_TextureCoordinateOffset[i]) ) );

colDifForce += base - sample[i];
}


// The result fragment sample matrix is as below, where x is the current fragment(4)
// 0 1 2
// 3 x 5
// 6 7 8


// From all the neighbor fragments gets the one with the greatest and lowest depths and place them
// into two variables so a subtract can be made later. The check is huge, but GLSL built-in functions
// are optimized for the GPU
//float areaMx = max(sample[0], max(sample[1], max(sample[2], max(sample[3], max(sample[5], max(sample[6], max(sample[7], sample [8])))))));

//float areaMn = min(sample[0], min(sample[1], min(sample[2], min(sample[3], min(sample[5], min(sample[6], min(sample[7], sample [8])))))));


//float colDifForce = areaMx - areaMn; // Gets the average value between the maximum and minimum depths


//Check for heavy depth difference to darken the current fragment;
//we do not want to mess with transparency, so leave alpha alone
//edgeForce variable control the outline transparency, so 1.0 would be full black.
// ? : is the same as if else
// abs is short of absolute value, it tells to disconsider the negativity of a value if it exists
abs(colDifForce) > ( sample[4] * baseThresh ) / near ? gl_FragColor = vec4(vec3(texcol*edgeForce), 1.0) : gl_FragColor = vec4(texcol);
}

Not sure if I should get rid of the the difference calculation and the non variable overwriting depth offset calculation yet; so I just disabled the first code with a comment.

Now, will I stop here? Not at all. What you think the logic may be for one to control the size of the outline? To begin with I know I will use an uniform to have easily settable edge_size value without the need to edit the shader file. What I am aiming to is for slighter thinner lines the farthest from the near plane.

PS: When you have time don't forget that math explanation(and the one from the other author if you can). By the way, I had been noticing; is linearization just another word for normalization? It seems both keep values in 0,1 range.

PS2: If you want I can private message you the link to my game project thread, so you will know what you had been helping me with; you might find it interesting(or maybe not). Anyway surely I will put your nickname into the game credits.

Ed Daenar
11-10-2011, 09:42 AM
Glad to hear you got something working. Using the depth buffer for contour detection seems to bring a number of annoying problems always.

Anyway, I got some extra time now so let's see...

About the linearizing stuff:

No, it's not normalizing. As I'll show you later, the values that are created for Z when projecting a point end up in a non linear way. What that means is that the adjacent values will have a varying difference between them depending on how far they are from the origin. A non linear distribution would be, for example, a logarythm or an exponential. In other words, if F(x) is the non linear function, it means F(x+1) - F(x) != F(x+1+b) - F(x+b). A linear function is, obviously, a line :P

So how do you go from a non linear Z distribution to a linear Z distribution? By undoing what you did to get the original non linear Z. Look at your perspective matrix and you will find that the Z component will look like this:

Xp = some irrelevant stuff for this discussion
Yp = some irrelevant stuff for this discussion
Zp = ((f+n)*z + 2*n*f) / (n-f)
Wp = -z

Where:
Zp => final projected point's Z value
Wp => final projected point's W value
f ==> far plane value
n ==> near plane value
z ==> original z (in camera space, but this doesn't matter)

Apart from multiplying by the projection matrix, as I explained before, the pipeline will apply the perspective correction scaling which is diving XYZ by the resulting W from that original projection. So that:

Zp = Zp / Wp

Substituting:
Zp = (((f+n)*z + 2*n*f) / (n-f)) / (-z)

What use is this? So far not much. We know how to get the projected Z point, but we already have that value in the Z buffer. What we would want is that the original z was projected linearly instead, not like this... so what we'd like to have is:

Zl = (-z - n) / (f-n)

Where:
Zl ==> Linear "projected" value of the original z


So let's not lose track here. Which, of all these variables, do we actualy know for any given point? We know:

Zp ==> this comes from the depth buffer, it's the result of the original projection
f ==> this is the far plane, we set this value manualy
n ==> this is the near plane, also set manualy

What we don't know:
z ==> the original z before projection

What we want:
Zl ==> the projection of z, but linearly


So we got a formula that related Zl and z and another that relates Zp and z. By doing some simple school grade math juggling we can isolate z from the first formula:

Zp = (((f+n)*z + 2*n*f) / (n-f)) / (-z) --> becomes -->

z = 2*n*f/(z*(f-n)-(f+n))

Voila! We have reverted the projection and we have the original z (in camera space) now. This is already enough to do depth comparisions! The values of camera space z are already linear anyway, so to compare values you could just use this.

But these values are not within the 0..1 range. They are at the n..f scale. Which is ok if you just care about the differences. However, you may want to have these values normalized (now THIS is normalization :P) within 0..1 and thus you simply need the second formula:

Zl = (-z - n) / (f-n)

And there you go. Normalized linear Z values that went from the depth buffer non-linear values.

The original snip of code I pasted, btw, is one that has its origin at Humus blog or site or whatever and I've been unable to reduce these two formulas to the simple version he got. I can tell you one thing about it though:

That code sniped is linearizing (and normalizing) values between 0 and f, not between n and f. This means a value exactly in your near plane will not compute to 0 as it should, instead it will compute to whatever it's range is between 0 and f. It's basicly as if he was using this:

Zl = -z / f

You could probably try to do some math juggling by substituting the unprojected z within this formula and eventualy reach his code. I haven't really bothered much with it. These things give me headaches. In the end, you save a 1 instruction with that snip vs doing it in "two steps".

And if you most definitely need to normalize between n..f instead of 0..f, then you better do it the way I just showed.

------

As for the outline thickness thing:

Probably by moving around the threshold you'll get a thicker outline, though artifacts in the distance may start to appear.

You could consider an alternative way of doing this, if your framework allows it. Because you only really need to find the contour of an object and must do it on post processing pass, you could render an extra buffer where "ID"s of the objects are marked for each fragment.

Basicly, when you render an object originaly, you pass a uniform that identifies that individual object and output this uniform to the ID buffer. Later, on a post process pass like you do now, what you do instead of sampling the depth buffer to compare distances and try to find edges what you do is compare IDs. The current fragment ID vs the surrounding IDs. If any of the surrounding IDs are different it means the fragment is part of an edge and you can colorize it black. If all IDs are the same, then you are sampling the internal of an object and thus you can use the original color.

Consider this alternative if you want to experiment, it might give you less artifacts than a depth buffer approach. It does however mean you gotta "ID tag" your objects as you draw them, which may be out of your reach depending on your framework.

And sure, I'll check your game out ^^