[2.5D Issue] - Fill z-buffer with grayscale texture

Hello everyone,
finally, after moving through many languages/frameworks I decided to implement my idea using only OpenGL.

My idea is a simple engine for 2.5D games (such old Alone in the dark, resident evil etc…), with:

• 2D backgrounds
• 3D actors/objects
• static cameras

Some months ago I completed my own 2.5D engine (actually a framework, with scene scripting and many more) using JMonkey engine, it works well, but I always had in my mind to create a tiny and faster version of this engine in pure OpenGL.
What I do in my current engine (JMonkey) is:

• Create a blender scene (high detail) and render in different layers.
• Create a very low poly blender scene only for collision, cameras and floor sectors.
• Import the low poly scene and cameras settings.

Here each layer has a fixed depth, and actors receive depth from sector where are in. At this point actors and
layer are enqueued and rendered with right order (depth testing disabled).

I tried to recreate this prototype in OpenGL and it works, it was simpler than I thought.

Now… this method for 2.5D is working well, and I think is the old school way to do it, but I would to implement it in a different (and modern) way now, my idea is:

• Render only one image from blender (to use as background) and NOT different layers.
• Render from the same camera a grayscale image to use as depth mask.
• Use always low poly scene for collisions and cameras (for cameras I already tested in OpenGL and it’s ok, for collision I will think later )

The question is… how to do it? The flow logic I think it’s simple (correct me if I’m wrong):

• Render a textured quad with the background
• Fill the Z-buffer with grayscale depth mask.

I’ve problem with second step. As I understood (I search for a month for a tutorial about this but no results found)
there are two ways for doing this (always correct me if I’m wrong)

• Create a quad over the camera (so no MVP transform for it), and in the vertex shader displace vertex according to depth texture.
• Fill directly the z-buffer with depth values from texture (fragment?).

I tried with first solution without success, but if it’s possible I would like to proceed with second solution, but the problem is…
how to do it? And how to adjust depth values according to near/far plane? It it possible to do it in a smart and flexible way?

I’m still studying OpenGL, so I apologize in advance for any stupid question.

I hope everything it’s clear.

Thank you very much

With legacy OpenGL, you can copy data directly to the depth buffer using glDrawPixels() with a format of GL_DEPTH_COMPONENT. That function doesn’t exist in the OpenGL 3+ core profile. You need to either use framebuffer objects or a fragment shader which writes to gl_FragDepth.

Either way, you aren’t realistically going to be able to directly upload 8-bit depth values when using a perspective projection. The non-linearity means that most of the scene only uses a fraction of the available depth range, so you need to use 16-bit or 24-bit values just to get reasonable precision in most of the scene. But if you’re using a shader, you can use that to convert linear depth to reciprocal depth.

See the glFrustum or gluPerspective reference pages for the matrices generated by these functions.

The portion related to depth is the lower-right 2x2 sub-matrix, i.e. the transformation of the Z and W components. Projective division converts clip-space coordinates to normalised device coordinates (NDC) by dividing X, Y and Z by W. The combination of the perspective transformation and projective division results in eye-space Z[sub]eye[/sub] being transformed to Z[sub]NDC[/sub] by Z[sub]NDC[/sub]=C+D/Z[sub]eye[/sub], where C=(near+far)/(near-far) and D=2nearfar/(near-far). This results in decreasing values of Z[sub]eye[/sub] being mapped to increasing values of Z[sub]NDC[/sub], i.e. the direction of the Z axis is flipped. Z[sub]eye[/sub]=-near is mapped to Z[sub]NDC[/sub]=-1 and Z[sub]eye[/sub]=-far is mapped to Z[sub]NDC[/sub]=+1.

The value stored in the depth buffer is Z[sub]NDC[/sub]*2-1, i.e. the Z[sub]NDC[/sub] range -1 to +1 is mapped to a depth range of 0 to 1 (i.e. 0 to 2n-1 where n is the number of depth bits).

Hi GClements,
first of all thanks a lot for your replay, than…
regarding the non-linearity of depth buffer I read about this, so thanks a lot for your technical explanation, it’s very clear.
Yes as you can saw I’m completing avoiding old fixed-pipeline functions, so I like your second idea, using a fragment to write the gl_FragDepth (also because I’m already using an FBO for an old style post-process pixelation, and I’d like to avoid using another one if no strictly required).

So in this case, let assume that I’ve:

• A background image of a room
• The same image in grayscale (I will test a bit with 16 or 24bit)
• For example a simple cube to put inside the scene.

So i think I need to put in my rendering queue:

1. A quad with screen size (no MVP it’s right?) with background texture
   2)??? Where to apply the depth texture? In the fragment of the same quad? Or does I need another quad (no colour but only depth writing)? Which is the right/better solution?
2. Render additional 3D objects (such my cube).

Thanks a lot again

Ray

Rendering the scene will require 2 stages, which can be done in either order. One is typical 3D rendering of the objects in the scene, the other is rendering a quad containing the background (colour and depth). The two stages will need different shaders.

The process for rendering the background will involve rendering a full-screen quad (or maybe not quite full screen; you shouldn’t assume that the screen has the same aspect ratio as the pre-rendered background). Two textures hold the colour and depth information for the background (or you could store the depth in the alpha channel). The fragment shader will copy the colour to either gl_FragColor or a user-defined output, and will transform the depth value and copy it to gl_FragDepth.

Provided that the depth texture stores linear depth, 8-bit resolution may be sufficient. It depends upon the depth range of your scene and how close 3D-rendered objects will get to objects which are part of the background. Bear in mind that the depth values only need to cover the range in which 3D-rendered objects can appear. Particularly distant objects only need a depth value greater than any rendered object; you don’t need to accurately represent their actual distance.

Dear GClements,
thank you very much for your explanation, it’s very clear, and I’m also happy to see that I was thinking in the right direction.
Probably 8bits for depth are sufficient, because for each camera view the walkable range “through depth” will be quite short, but in any case I will do a lot of test for this.

So thank you again

Ray

Ok this is what I tested:
First of all in my scene (a simple 1x1x1 cube) I correct the MVP matrix jus to be sure that the view from blender rendering and the view from OpenGL overlap correctly, and this is done.

Now there are few steps:

1. I start the scene rendering (let’s assume with a NEAR of 0.1, FAR of 100 and FoV_Y of 45 degrees), and inside the rendering page of blender I choose to show only the depth buffer (not with node editor, but in the rendering page there’s this useful option, I don’t know whether this is an issue of latest Blender versions), so at the end I got a 24bit depth texture, that is (I think) LINEAR. So in the fragment I should make it LOGARITHMIC as Z-Buffer works.

2. Render a quad with HD rendering.

3. Fill z-buffer with my depth texture

4. Render my 3D objects.

Ok, something works, because I see some of test 3D objects that will be occluded when they stay into some areas, but this is not done in a proper way, because depth is not precise as it should be, and this I think is a problem of precision of the non linearity of my z-buffer. This is my fragment.

#version 330 core

// Input data for UV.
in vec2 v2_UV;

// Output data for color.
out vec3 v3_Color;

// Sampler for depth mask.
uniform sampler2D tx_DepthTexture;

// Near and far, same of blender view, so 0.1 - 100.0.
uniform float f_Near;
uniform float f_Far;

void main()
{		
	float f_LinearDepth = texture2D(tx_DepthTexture, v2_UV).r;
	gl_FragDepth = -((f_Near + f_Far) * f_LinearDepth - (2 * f_Near)) / ((f_Near - f_Far) * f_LinearDepth);	
	v3_Color = vec3(linearDepth, linearDepth, linearDepth); // This is for test only, the final color will come from the real rendering texture.
}

I don’t know whether problem is inside fragment, or if my linear Z-texture rendered from blender is not right (I think is sufficient from blender to have same camera settings and projection settings).

There are three problems with this:

1. For a 24-bit depth texture, the value stored in the texture probably isn’t the absolute eye-space Z coordinate (which will be negative for points in front of the camera). It’s more likely to be either a depth value using the same equation as OpenGL (in which case, you should be able to use it directly, without any transformation) or a linear parameter which is 0 at the near plane and 1 at the far plane. In the latter case you would need:

Zeye = f_LinearDepth * (f_Near - f_Far) - f_Near.

2. The equation is wrong. The -(2f_Near) term should be +(2f_Far*f_Near). The mapping between eye-space Z and NDC Z is:

Zndc = -((f_Near + f_Far) * Zeye + (2 * f_Far * f_Near)) / ((f_Near - f_Far) * Zeye);

3. The NDC Z coordinate ranges from -1 at the near plane to +1 at the far plane, while gl_FragDepth ranges from 0 at the near plane to 1 at the far plane. So you need:

gl_FragDepth = (Zndc + 1) / 2;

If you’re unsure of how the depth is stored by Blender, render a scene consisting of some flat surfaces aligned to the X-Y plane (i.e. normal to the Z axis) with known Z coordinates and with the camera at the origin, and compare the values in the texture to the Z coordinates.

Thanks, thanks very much! I want to try soon as possible in this way, I think that I also got some problems because my method for loading textures loads with only 8 bit depth, that probably is too less, I will try with more precision and see what happens, for now thank you!

Dear GClements (and all interested people :)), I tried to render a 32bit depth texture with node compositing with blender, with the procedure you told me is not working, but, playing a bit with values I discovered an interesting point:

Using simply:

gl_FragDepth = (f_LinearDepth + 1) / 2;

without computing the non-linear depth (so taking the raw texture value from depth texture and just normalize it to range 0-1), this is working! I have a good filling inside the depth buffer and depth seems computed in a good way (when I have the time I want to make some test to ensure me that the depth is the real one) but at a first impression seems ok (but is also possible that I’m wrong).
Here the important issue is that I have to use a rasonable value for NEAR (in the projection matrix), in practical it’s working well only if I use 1.0 as value.

Now… I don’t want only yo have a working (also because I’m not sure it’s working… I just obtained something maybe near to the desired result) solution but I want also to understand… is it possible that blender already managed a lot of stuffs for me and provided me a good depth texture?

Thanks a lot

Ray

as I though I got the good result just for luck, but it’s not working, with your settings the fragment is this:

out vec3 outColor;
in vec2 fUv;
uniform sampler2D depthMask;
void main() {
   float f_Near = 1.0;
   float f_Far = 100.0;
   float f_LinearDepth = texture2D(depthMask, fUv).r;
   float Zeye = f_LinearDepth * (f_Near - f_Far) - f_Near;
   float Zndc = -((f_Near + f_Far) * Zeye + (2 * f_Far * f_Near)) / ((f_Near - f_Far) * Zeye);
   float final = (Zndc + 1) / 2;
   gl_FragDepth = final;
};

But the output data is not right, what I’m obtaining is too far related to near depth that I should obtain. I’m trying a lot of combination but seems not working properly.

[QUOTE=Ray1184;1279560]Using simply:

gl_FragDepth = (f_LinearDepth + 1) / 2;

without computing the non-linear depth (so taking the raw texture value from depth texture and just normalize it to range 0-1), this is working!
[/QUOTE]
This suggests that the depth information produced by blender is the NDC Z coordinate, i.e. it’s projected, not linear.

That will make matters simpler, assuming that you’re happy using the data as-is. But if you want to reduce the bit depth (e.g. to save space), you’ll probably need to linearise the data first. Projected depth values are unevenly distributed, with most of the range being used for values close to the near plane, so you typically need significantly more bits for projected depth than for linear depth.

Dear GClements, unfortunatly was a mistake, it’s almost (because is not very precise) working only with a very strict NEAR-FAR range, but with for example 1 and 100 I don’t see a result.

Currently with the shader of my last post (I report this below for simplify) is not working, because I see that non linear depth buffer loose a lot of detail, only very near is precise.

I still don’t know how blender do this, here a screenshot of my scene, and his depth texture (from blender). Maybe this can be give more information.

The scene
[ATTACH=CONFIG]1280[/ATTACH]

The depth buffer from blender
[ATTACH=CONFIG]1279[/ATTACH]

Fragment code (depth mask is from the depth buffer texture from blender):

in vec2 fUv;
uniform sampler2D depthMask;
void main() {
   float f_Near = 1.0;
   float f_Far = 100.0;
   float f_LinearDepth = texture2D(depthMask, fUv).r;
   float Zeye = f_LinearDepth * (f_Near - f_Far) - f_Near;
   float Zndc = -((f_Near + f_Far) * Zeye + (2 * f_Far * f_Near)) / ((f_Near - f_Far) * Zeye);
   float final = (Zndc + 1) / 2;
   gl_FragDepth = final;
};

How do you think I can fix this code?

Thanks a lot again

That is typical for projected depth (Zndc = C+D/Zeye). As a rough guide, the portion of the scene beyond N times the near distance gets 1/N of the depth range; the far distance has very little effect. So if e.g. Znear=1 and Zfar=100, half the depth range is used for Z values between -1 and -2 with the other half for Z values between -2 and -100.

You need to determine the exact mapping between eye-space Z and depth.

I suggest creating a scene with the camera at the origin facing along one of the axes, with a number of flat surfaces perpendicular to the axis at known distances in front of the camera. Compare those distances to the corresponding values in the generated depth buffer.

There are three main questions:

1. Whether the values in the depth buffer are signed (-0x80000000 to +0x7FFFFFFF corresponding to -1 to +1) or unsigned (0 to 0xFFFFFFFF corresponding to 0 to +1). Actually, they might even be floating-point.
2. Whether the depth values are linear (depth=A+B*Zeye) or projected (depth=C+D/Zeye).
3. The values of the two parameters in the equation.

Dear GClements,
I read some specification for blender Z-Buffer, and it seems that is already non-linear:

The Z-depth map; how far away each pixel is from the camera. Used for Depth-Of-Field (DOF). The depth map is inverse linear (1/distance) from the camera clip start.

Unfortunatly I’m still not able to find a more in depth specification about values stored inside the depth map. Another point is that my depth map rendered from blender (unlike blender spec said) seems linear (maybe I did something wrong in my rendering, or maybe it’s non-linear and seems like linear because FAR distance is high (100) and center of my 3D sample I think it’s more or less at 2 or 3, I must try with a smaller FAR)

Anyway I will try some tests, also applying some factors and trying to optaining a good mapping. I will let you say the result, and when it’s done I want also to write and publish some notes about this, I’m sure that other people will find this very interesting and useful.

Thanks for all support

Ray

Dear all,
finally something is working, at the end I could obtain a good result mixing what you explained me in this thread and a tutorial for pre-rendered backgrounds with a depth map in Unity-DirectX (I can easily adapt this for OpenGL)

This is what I done:

1) Render from blender:
I started in a wrong direction, because with my texture all values between 0 and 1 were covered, also if object were all in center of view (so more or less at half of FAR distance)
For correct this I found that the correct procedure is that from compositing:
1- from Z output socket SUBTRACT the value of NEAR (because default Z value starts from camera position, not from NEAR) and I obtain a value, (let’s call this A)
2- DIVIDE the value of A by (FAR - NEAR) and I obtain the final value that will be stored in my depth map (that is LINEAR).

2) Fill the depth buffer with texture:
In a nutshell I used this fragment shader:

#version 330 core
out vec3 outColor;
in vec2 fUv;
uniform mat4 proj; // The projection matrix.
uniform sampler2D depthMask;
void main() {
    float near = 1.0;
    float far = 10.0;
    float imageDepth = texture2D(depthMask, fUv).r;
    imageDepth = imageDepth * -1;
    vec4 temp = vec4(0.0, 0.0, ((imageDepth * (far - near)) - near), 1.0);
    vec4 clipSpace = proj * temp;
    clipSpace.z = clipSpace.z / clipSpace.w;
    // clipSpace.z = 0.5 * (clipSpace.z + 1.0);  // Should be this used only for DirectX?
    gl_FragDepth = clipSpace.z;
    outColor = vec3(imageDepth * -1, imageDepth * -1, imageDepth * -1); // Just for test I render the original input LINEAR depth as color.
};

It seems ok in this way (instead of do manual operations I use directly my projection matrix), but some stuff are still strange for me, i.e:

• Why must I remove this row “clipSpace.z = 0.5 * (clipSpace.z + 1.0);”? It means that is already normalized between 0 and 1 I guess?
• “clipSpace.z = clipSpace.z / clipSpace.w;” Why I need this? Sorry but is not so clear.

So do you think could be correct this way (I hope ;))? Because it seems a bit more complex that I though.

Thanks a lot

Ray

EDIT:
I tried a more specific test, with different planes in the scene (as GClement suggests) and depth writing seems almost right. I say almost because:
I have a cube that should be sliced in the half by the nearest plane, but is sliced not in the perfect half but a bit far. So I need to check some tricks for calibrate in a right way the depth writing.

If removing that line improves the result, it means that something else is wrong and this is compensating for it to an extent.

It would probably be quite simple to determine what the transformation should be if you’d post actual numbers, i.e. half a dozen (Zeye, depth) pairs.

This converts from homogeneous coordinates to Euclidean coordinates.

OpenGL (and most 3D rendering systems) use homogeneous coordinates, as this allows translations and projective transformations (e.g. perspective projection) to be represented as a linear transformation (i.e. a matrix).

A point (x,y,z) in Euclidean coordinates corresponds to the line w*(x,y,z,1) = (wx,wy,w*z,w) in homogeneous coordinates. Conversely, a point (x,y,z,w) in homogeneous coordinates corresponds to the point (x/w,y/w,z/w) in Euclidean coordinates.

OpenGL automatically performs the latter conversion as part of the rendering process. Consequently, you need to perform the same conversion in order to match the depth values calculated by OpenGL when rendering primitives.

Dear all,
after some days of inactivity (due to many troubles :D) I tried to read more in depth the tutorial that I found for unity and finally I got a working result (and this time really working! I did a lot of test with also quite complex scenes and depth buffer is managed in a good way).
In pratical the missing point was a very stupid issue from blender, in pratical when you render you must set RAW as output format, instead of default.
Did this is sufficient as I explained before in the compositing node editor to subtract NEAR to Z value and then divide it by (FAR-NEAR).
This would be the linear depth buffer texture. Did this the fragment shader it’s this

#version 330 core

/* The texture quad UV coords. */
in vec2 in_QuadUV;	

/* The projection matrix. */					
uniform mat4 u_ProjMatrix;		

/* The projection near distance. */	
uniform float u_NearDist;		

/* The projection far distance. */
uniform float u_FarDist;	

/* The texture sampler for depth map. */	
uniform sampler2D u_DepthMapSampler;	

/* The code. */
void main() {
    float f_ImageDepth = texture2D(u_DepthMapSampler, in_QuadUV).r;
    f_ImageDepth = -f_ImageDepth;
    vec4 v4_Temp = vec4(0.0, 0.0, ((f_ImageDepth* (u_FarDist - u_NearDist)) - u_NearDist), 1.0);
    vec4 v4_ClipSpace = u_ProjMatrix * v4_Temp;
    v4_ClipSpace.z = v4_ClipSpace.z / v4_ClipSpace.w;
    v4_ClipSpace.z = 0.5 * (v4_ClipSpace.z + 1.0); 
    gl_FragDepth = v4_ClipSpace.z; // Write the depth only. Background image it's wrote in a previous pass.
};

If someone would be interest and if there’s a related section inside the forum it would be a pleasure for me to make a more detailed report about this procedure.

Ray