glFog() axis problem

I’m trying to use glfog() to hide clipping points in my terrain editor.

However, because I am playing around with matrix multiplications, the glFog() seems to be working on the wrong axis.

Here is my renderFrame() method:


void RenderFrame()
{
    glEnable(GL_DEPTH_TEST);
    glEnable(GL_CULL_FACE);

    glViewport(0, 0, g_windowWidth, g_windowHeight);
    glClearColor(0.3f, 0.5f, 0.9f, 0.0f);
    glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

    glMatrixMode(GL_PROJECTION);
    glLoadIdentity();
    glMultMatrixf(&g_camera.getProjectionMatrix()[0][0]);

    glMatrixMode(GL_MODELVIEW);
    glLoadIdentity();
    glMultMatrixf(&g_camera.getViewMatrix()[0][0]);

    RenderTerrain();
	
    RenderText();
}

Here are a few screenshots to illustrate the problem:
image 1
image 2
image 3
As the camera turns to look down the Y axis, the fog takes affect. How can I fix this?

Any help would be much appreciated thank you!

The camera rotation should not affect GL_PROJECTION matrix, and currently it looks that it is the case.

So I should change my camera class to use gluLookAt() instead of returning a matrix?

No need to, but make sure g_camera.getProjectionMatrix() returns a real projection matrix, ie. no rotation/translation.

Can you show how you currently generate g_camera.getProjectionMatrix() ?

Here is the method I am using to generate the projection matrix.


void Camera::perspective(float fovx, float aspect, float znear, float zfar)
{
    // Construct a projection matrix based on the horizontal field of view
    // 'fovx' rather than the more traditional vertical field of view 'fovy'.

    float e = 1.0f / tanf(Math::degreesToRadians(fovx) / 2.0f);
    float aspectInv = 1.0f / aspect;
    float fovy = 2.0f * atanf(aspectInv / e);
    float xScale = 1.0f / tanf(0.5f * fovy);
    float yScale = xScale / aspectInv;

    m_projMatrix[0][0] = xScale;
    m_projMatrix[0][1] = 0.0f;
    m_projMatrix[0][2] = 0.0f;
    m_projMatrix[0][3] = 0.0f;

    m_projMatrix[1][0] = 0.0f;
    m_projMatrix[1][1] = yScale;
    m_projMatrix[1][2] = 0.0f;
    m_projMatrix[1][3] = 0.0f;

    m_projMatrix[2][0] = 0.0f;
    m_projMatrix[2][1] = 0.0f;
    m_projMatrix[2][2] = (zfar + znear) / (znear - zfar);
    m_projMatrix[2][3] = -1.0f;

    m_projMatrix[3][0] = 0.0f;
    m_projMatrix[3][1] = 0.0f;
    m_projMatrix[3][2] = (2.0f * zfar * znear) / (znear - zfar);
    m_projMatrix[3][3] = 0.0f;

    m_viewProjMatrix = m_viewMatrix * m_projMatrix;
    
    m_fovx = fovx;
    m_aspectRatio = aspect;
    m_znear = znear;
    m_zfar = zfar;
}

This is how I’m adjusting the modelview matrix.


void Camera::rotateFirstPerson(float headingDegrees, float pitchDegrees)
{
    // Implements the rotation logic for the first person style and
    // spectator style camera behaviors. Roll is ignored.

    m_accumPitchDegrees += pitchDegrees;

    if (m_accumPitchDegrees > 90.0f)
    {
        pitchDegrees = 90.0f - (m_accumPitchDegrees - pitchDegrees);
        m_accumPitchDegrees = 90.0f;
    }

    if (m_accumPitchDegrees < -90.0f)
    {
        pitchDegrees = -90.0f - (m_accumPitchDegrees - pitchDegrees);
        m_accumPitchDegrees = -90.0f;
    }
   
    Quaternion rot;

    // Rotate camera about the world y axis.
    // Note the order the quaternions are multiplied. That is important!
    if (headingDegrees != 0.0f)
    {
        rot.fromAxisAngle(WORLD_YAXIS, headingDegrees);
        m_orientation = rot * m_orientation;
    }

    // Rotate camera about its local x axis.
    // Note the order the quaternions are multiplied. That is important!
    if (pitchDegrees != 0.0f)
    {
        rot.fromAxisAngle(WORLD_XAXIS, pitchDegrees);
        m_orientation = m_orientation * rot;
    }
}


void Camera::updateViewMatrix()
{
    // Reconstruct the view matrix.

    m_viewMatrix = m_orientation.toMatrix4();

    m_xAxis.set(m_viewMatrix[0][0], m_viewMatrix[1][0], m_viewMatrix[2][0]);
    m_yAxis.set(m_viewMatrix[0][1], m_viewMatrix[1][1], m_viewMatrix[2][1]);
    m_zAxis.set(m_viewMatrix[0][2], m_viewMatrix[1][2], m_viewMatrix[2][2]);
    m_viewDir = -m_zAxis;

    if (m_behavior == CAMERA_BEHAVIOR_ORBIT)
    {
        // Calculate the new camera position based on the current
        // orientation. The camera must always maintain the same
        // distance from the target. Use the current offset vector
        // to determine the correct distance from the target.

        m_eye = m_target + m_zAxis * m_orbitOffsetDistance;
    }

    m_viewMatrix[3][0] = -Vector3::dot(m_xAxis, m_eye);
    m_viewMatrix[3][1] = -Vector3::dot(m_yAxis, m_eye);
    m_viewMatrix[3][2] = -Vector3::dot(m_zAxis, m_eye);
}

All of this code is coming from this demo.

Haven’t sifted it all, but this looks wrong. proj0,0 in the general perspective projection matrix is 2n/(r-l) (see this page at the end, Perspective Projection).

Your projection matrix computation code above obviously presumes a symmetric frustum, so r = -l. So 2n/(r-l) is the same as 2n/2r = n/r = 1/tan(0.5 * fovx). However, as you’ll notice your code above uses fovy instead of fovx here. That’s appears to be wrong.

It also seems to use (zfar + znear) / (znear - zfar) and (2.0f * zfar * znear) / (znear - zfar) instead of using -(zfar + znear) / (znear - zfar) and (-2.0f * zfar * znear) / (znear - zfar)

Why have they chosen to do this, and what effect does it have on the model view matrix?

If I change them to the correct values it seems to stretch the scene out at the edges.

No, you swapped a couple terms transcribing from the correct matrix in the book. It’s -(zfar + znear) / (zfar - znear) and (-2.0f * zfar * znear) / (zfar - znear).

If you negate the numerator and denominator of these corrected expressions, I believe you get what is in his code.

The problem was with my light settings, so what I thought was the black fog wasn’t.

here are the fog settings I am using


glFogi(GL_FOG_MODE, GL_LINEAR);		
glFogfv(GL_FOG_COLOR, fogColor);
glFogf(GL_FOG_DENSITY, 0.9f);
glHint(GL_FOG_HINT, GL_DONT_CARE);
glFogf(GL_FOG_START, 20.0f);
glFogf(GL_FOG_END, HEIGHTMAP_SIZE * HEIGHTMAP_GRID_SPACING);
glEnable(GL_FOG);	

Could it be that the shaders I am using to texture the terrain is preventing the fog from working?

as shown above the following method is called from my renderFrame() method.


void RenderTerrain()
{
glUseProgram(g_terrainShader);
UpdateTerrainShaderParameters();
glEnable(GL_LIGHTING);
glEnable(GL_LIGHT0);
glLightfv(GL_LIGHT0, GL_POSITION, g_lightDir);
......
glDisable(GL_LIGHT0);
glDisable(GL_LIGHTING);
glUseProgram(0);
}

This is also the only point where lighting is enabled.

Ok so glFog doesn’t work when using a shader, so I have adjusted my shader:


[vert]
uniform float tilingFactor;

varying vec4 normal;

varying float gl_FogFragCoord;

void main()
{
	gl_FogFragCoord = gl_Position.z;

    normal.xyz = normalize(gl_NormalMatrix * gl_Normal);
    normal.w = gl_Vertex.y;

    gl_Position = gl_ModelViewProjectionMatrix * gl_Vertex;
    gl_TexCoord[0] = gl_MultiTexCoord0 * tilingFactor;
}

[frag]
struct TerrainRegion
{
    float min;
    float max;
};

uniform TerrainRegion region1;
uniform TerrainRegion region2;
uniform TerrainRegion region3;
uniform TerrainRegion region4;
uniform TerrainRegion region5;

uniform sampler2D region1ColorMap;
uniform sampler2D region2ColorMap;
uniform sampler2D region3ColorMap;
uniform sampler2D region4ColorMap;
uniform sampler2D region5ColorMap;

varying vec4 normal;
varying float gl_FogFragCoord;

vec4 GenerateTerrainColor()
{
    vec4 terrainColor = vec4(0.0, 0.0, 0.0, 1.0);
    float height = normal.w;
    float regionMin = 0.0;
    float regionMax = 0.0;
    float regionRange = 0.0;
    float regionWeight = 0.0;
    
    // Terrain region 1.
    regionMin = region1.min;
    regionMax = region1.max;
    regionRange = regionMax - regionMin;
    regionWeight = (regionRange - abs(height - regionMax)) / regionRange;
    regionWeight = max(0.0, 0.5 * regionWeight);
    terrainColor += regionWeight * texture2D(region1ColorMap, gl_TexCoord[0].st);

    // Terrain region 2.
    regionMin = region2.min;
    regionMax = region2.max;
    regionRange = regionMax - regionMin;
    regionWeight = (regionRange - abs(height - regionMax)) / (regionRange * 2) ;
    regionWeight = max(0.0,regionWeight);
    terrainColor += regionWeight * texture2D(region2ColorMap, gl_TexCoord[0].st);

    // Terrain region 3.
    regionMin = region3.min;
    regionMax = region3.max;
    regionRange = regionMax - regionMin;
    regionWeight = (regionRange - abs(height - regionMax)) / regionRange;
    regionWeight = max(0.0, regionWeight);
    terrainColor += regionWeight * texture2D(region3ColorMap, gl_TexCoord[0].st);

    // Terrain region 4.
    regionMin = region4.min;
    regionMax = region4.max;
    regionRange = regionMax - regionMin;
    regionWeight = (regionRange - abs(height - regionMax)) / regionRange;
    regionWeight = max(0.0, regionWeight);
    terrainColor += regionWeight * texture2D(region4ColorMap, gl_TexCoord[0].st);

	// Terrain region 5.
    regionMin = region5.min;
    regionMax = region5.max;
    regionRange = regionMax - regionMin;
    regionWeight = (regionRange - abs(height - regionMax)) / regionRange;
    regionWeight = max(0.0, regionWeight);
    terrainColor += regionWeight * texture2D(region5ColorMap, gl_TexCoord[0].st);

    return terrainColor;
}

void main()
{ 	 
	float fog; 
    vec3 n = normalize(normal.xyz);

    float nDotL = max(0.0, dot(n, gl_LightSource[0].position.xyz));
        
    vec4 ambient = gl_FrontLightProduct[0].ambient;
    vec4 diffuse = gl_FrontLightProduct[0].diffuse * nDotL;
    vec4 color = gl_FrontLightModelProduct.sceneColor + ambient + diffuse;   

	fog = (gl_Fog.end - gl_FogFragCoord) * gl_Fog.scale;
    
    gl_FragColor = mix(gl_Fog.color, (color * GenerateTerrainColor()), fog);
}


The texture splatting is still working, but random patches are appearing each frame, can anyone point out what I am doing wrong please?