scaling mesh along the normal

I tried to implement outlining of cell shade model by rendering Frontface-culled of the model using just black color.

I do this by scale the mesh along the normal using following vertex shader

#version 150 core

in vec3 vertex;
in vec3 normal;

uniform mat4 cameraMatrix;
uniform mat4 modelMatrix;
uniform mat4 projectionMatrix;

void main()
{	
        float scaleFactor = 0.005;
	vec3 scaleVertex = vertex + normal*scaleFactor;

	gl_Position = projectionMatrix*cameraMatrix*modelMatrix*vec4(scaleVertex.xyz,1.0);				
		
} 

but with constant-predefined scaleFactor the outline will only look good at certain distance.

When zoom in the outline will be very thick when zoom out the outline will be too fainted.

What should I do to be able to calculate scale factor on the fly so that I can control the thickness of outline across different distance ?

Thank

What about this :

{	
        float scaleFactor = 0.005;
	gl_Position = projectionMatrix*cameraMatrix*modelMatrix*vec4(vertex.xyz,1.0)+ normal*scaleFactor;				
		
} 

Assuming normal is transformed here, right?

• Chris

Indeed, my bad

Sorry to bother you again.

I modified the shader as suggest by ZBuffer (with normal transformed) but the result are still the same (different size when zooming).

#version 150 core

in vec3 vertex;
in vec3 normal;

uniform mat4 cameraMatrix;
uniform mat4 modelMatrix;
uniform mat4 projectionMatrix;

void main()
{	
	float scaleFactor = 0.005;
	
	mat4 modelviewMatrix = cameraMatrix*modelMatrix;

	mat3 normalMatrix = mat3(modelviewMatrix[0].xyz,
	modelviewMatrix[1].xyz,
	modelviewMatrix[2].xyz);

	vec3 viewSpaceNormal = normalMatrix*normal;
	vec4 replacementPosition = projectionMatrix*vec4(viewSpaceNormal,0.0)*scaleFactor;	

	gl_Position = projectionMatrix*cameraMatrix*modelMatrix*vec4(vertex.xyz,1.0)+replacementPosition;				
		
} 

what I really want to do is making the scaled pixel (the width of the outline) fixed at some value , like 2 pixel across different zooming distance.

Actually I could just use glPolygonMode(GL_FRONT_AND_BACK,GL_LINE);
and draw the line with out modifing the mesh. but my mesh has many unclosed surface (like back of character sleeve,hair).

What about a small normalize() here :

vec4 replacementPosition = normalize(projectionMatrix*vec4(viewSpaceNormal,0.0))*scaleFactor;

gl_Position = projectionMatrixcameraMatrixmodelMatrix*vec4(vertex.xyz,1.0)+replacementPosition;

Well it still look the same (this time with picture).

zoom out (click to enlarge)

zoom in (click to enlarge)

May be I have to re-calculate scaleFactor based on camera position or something but I dont really care now since the result will look ugly anyway(The thin line look too jaggy).

Can you guy suggest me some line rendering technique that work well with anime style cell shading technique ?

I also try using edge detection on normal g-buffer but cannot manage to make line wide more than 1~2 pixel. and the same edge detection kernel produced and ugly result when use with anti-aliased (resolved to NOAA texture) normal g-buffer (resolved to NOAA texture) , the detected line still look jaggy which destroy the purpose of anti-aliasing.

Edge detection line (NOAA , blend with source image)

Edge detection line (2XAA , blend with source image)

noticed that the line still look very jaggy and there also color leak

Thank.

I am very surprised that adding a normalize still give the same result.
Some thoughts :

vec4 projNormal = projectionMatrix*vec4(normalize(viewSpaceNormal),0.0)
vec4 replacementPosition = (normalize(projNormal)/projNormal)*scaleFactor;

Don’t you have to multiply that rotated normal by W or (Z*0.5+ 0.5) at some point?

The higher the Z, the larger the added displacement should be.

Btw, how about rendering wide lines, in wireframe, and cull=front?

That would be ideal solution (and should handle AA fine) but my mesh had many unclosed part (like sleeve , skirt and hair) which will result in visible wire frame and glLineWidth is also disable in OpenGL 3.2 core profile.

I am now try to improve my edge detection shader to better handle AA image (managed to fix color bleeding but still look jagged).

Thank

So you use GL3.2 core: use a geometry-shader to render only silhouette edges (triangle adjacency info required). Wide-lines were brought back in 3.2 . They are disabled not in “core” but “forward-compatibility” profile.

In all cases, the current approach with “scale along the normals” might do the trick, and you’re just a step away from it. Yet, edges with funky normals will probably become problematic.