PDA

View Full Version : z-buffer work around



GeorgeRH
03-22-2008, 02:39 PM
Ok, most my programming with graphics in the past has been strait to hardware. You know using the programmers guide to ega, vga svga..

Ok, so I am trying to rewrite or improve an old project I wrote almost 15years back. You would think wow that should be easy with newer technology opengl and so on. All I will say is I wish. So far massive disappointment. The Z-buffer implement in opengl in my opinion is atrocious at best. Great for something written 10+ years back but not what we could or should be at today.

Anyway. I need to display a massive seen. To give you a scale idea. Say the opengl value of 1 is equal to 3 feet. A person in game height would be 2 or the earth for instance would be 6,700,000 radius. With the Z-buffer that is implemented this is a massive issue because it does not have the precision and wastes a lot of what it does have. float *zbuffer; would be better than it is currently.

So my solutions so far I have came up with are based around 2 possible areas. Either using a loose Octree or sphere tree system along with cone shaped view frustum. The tree system either I use would also have to handle relative distance to size potential as well not just if the object is potentially visible. If you are sitting on mars looking at earth it is highly unlikely you would see the shuttle orbiting earth even if it was in front of earth in your view volume.

Basically because the Z-buffer has such a very large loss in precision in the distance I need to off load that work to the CPU and create a fairly fast way ensuring no surface will be behind another surface that would even include back face culling.

Which kind of all seems to defeat the point of why have a GPU to start if you have to off load so much work to the CPU. Simply making a better z-buffer would eliminate 90% of the issues and problems with complex and large scenes.

At least the good part is with multiprocessor cores we can farm that work to them.

So my two questions that I have is what workarounds have others came up with. No scaling does not fix the problem and what I am describing would require znear to be like 0.1 and zfar to be like a billion or more even.

Drawing the scene in multipass seems retarded since we already need to do multipass rendering for a host of other issues. The work even to do the multi pass seems to out weight the work to figure out what to display and not. Because you would still have to sort all the objects near to far and what about object that are both to other object then you have to split them up.

My second question. Why on earth are we still having such an antiquated z-buffer. I spent the last several years working in the processor industry and with the advancements we have gone through it makes no sense to me. Just the work load difference it would remove is far reason enough it could only improve the visual quality of what we have. So what if frame rate drops from 200+ down to 100 even 60 you can't see it. You shouldn't be trying to force the entire Z-buffer to fit in one scale either leave them as pure floats or doubles preferably 64 to 128 bit.

ZbuffeR
03-22-2008, 03:55 PM
Well if you know better, why not design yourself such a chip with 128bit float zbuffer ? :cool: Just joking.

To be more precise, it is not OpenGL that locks you on a 24bits zbuffer. It is only the hardware.

It has been almost 10 years of 24bits zbuffer on consumer cards, not sure why no progress have been made on this front. I guess all the depth tests optimisations such as compression, early-z, hierachical culling does not scale well with higher precision.

Some remarks :
- back face culling is done independantly from depth testing, don't worry about it
- doing multipass is not so retarded. You can simply provide the same scene multiple times, with disjoint znear/zfar couples. Using VBO will help reduce the performance cost (as geometry will not change during these passes)

Can you describe roughly this old project ? Maybe more specific hacks are possible ?

Humus
03-22-2008, 03:55 PM
I don't think the Z-buffer is antiquated in any way. It's done the way it is for a reason. We're trading non-optimal Z-distribution for simpler and faster hardware. Z is linear in screen-space (rather than eye-space), which makes it cheaper to interpolate the Z values, and very importantly for modern hardware, makes the Z values very compressible, which allows for bandwidth savings optimizations, which would be very hard and costly to implement for W-buffering. For most normal scenes the precision is good enough anyway.

Now if you're trying to render everything from 0.1 to a billion, then you're clearly falling outside of what's normal and what OpenGL was designed for. Maybe using a floating point buffer will help. Don't forget to reverse Z to get the Z distribution go hand in hand with float value distribution. If you can't use that, you could implement your own Z-buffer-like behaviour in the shader.

GeorgeRH
03-22-2008, 04:53 PM
What is normal is changing. Look at the vast number of projects that people are attempting planet rendering and even entire universes for game play.

As I made the point it isn't like we need 200fps anyway and sure as heck not 400fps. 99% of people can not see flicker above 30fps.

I think we should be doing object culling on cpu then surface or face culling and so on, on the GPU.

Just look at the large number of First Person shooters trying to go to larger and larger terrains. That alone or any other game for that matter needing large terrain.

Anyway I wasn't trying to create an argument just stating an opinion.

But I am more interested in what solutions people have tried besides the few bad choices I seen offered.

One idea on the sphere tree I had was to make this an orbital sphere tree. Objects have one of 3 primary states. Attached(such as attached to the surface), in orbit, or free. I figure most objects in the universe fall into those areas if not all.

The trick I think is to create a scene management / render manager that is intuitive. Such as what object may or may not be visible regardless if in line of site. Such as some stars can not be seen because they are dim, objects that are small or beyond X distance may not be seen, object behind other object or facing away may not be seen, and so on.

Even with such a system I am going to have to use a lot of false movement and scaling to get stuff done. Such as moving from one galaxy to another. Or even displaying a galaxy to be visible from another one will require using something like a projected back ground. Once an object comes in range then it is removed from the back ground and actually displayed. The problem with that is Every face on the projected back ground will have to get treated as an object so that if it sits behind another object it can be culled and modified to display the portion that isn't behind the object.

All of which becomes massive work wise. But would be extremely simplified with a proper z-buffer.

Anyway interested in what workarounds people have come up with for this issue.

GeorgeRH
03-22-2008, 05:22 PM
Well apparently the back face culling utterly fails when you have your znear and zfar to far apart from each other. Otherwise the objects that are culled should never bleed through and they do.

Why I say doing multipass is retarded. First it requires you to divide or order your scene from far to near. CPU intensive and time consuming especially if you have stuff moving in it which most games do.

You may already get limited to multipass for dealing with things like textures and shading, lighting and so on depending on the persons card to start with. It could take something that could be extremely playable and just make it not feasible at all.

The old project: was a simple display system. Used a pure float for the z buffer. it was done in 256 bit color so only used a byte for the color buffer. Back face culling was done using float vectors. It didn't use a sphere tree, quad tree or any of that not even a frustum to cull objects or surfaces. Very simple. It had one purpose to display the largest objects I could build from micro detail to massive detail. It wasn't designed to be fast.

I figured with the advancements we have had in CPU tech I would see what current hardware and APIs are capable of. Kind of disappointing to be honest. Some things I find awesome shader tech great. However, I still see issues with it.
Say you want to use the shaders to build some really nice land great. If you do then the CPU has to read from the Video card the height of the land or it can't place objects on it properly or it has to work it out itself also which may not be as accurate.

Reading from the video card should be a no-no unless copying for video or something.

Personally I think that one improvement fixing the z-buffer will be the biggest change they can make for next generation games they can possibly make. It would bring the level of detail to a entire new place and make entire universes feasible with relative ease.

CRasterImage
03-22-2008, 09:02 PM
You keep saying "Universes". Are you doing something space related? (planets, space ships, etc...)

If so, that is the render-order dream.

Also, I keep wondering if ZBuffer feels insulted by the criticism.
;-)

Mark Shaxted
03-22-2008, 09:39 PM
To the OP. I guess you're a young 'un. You seem to believe that a GRAPHICS API (ie putting pixels on the screen) is responsible for your physics engine. Well, that ain't gonna happen for a while.

The whole world as of now manages quite well with the current limitations. And from what you decribe, what you want is extremely easy. You just don't seem to have the ability to ask politely.

I think your inability to ask properly is hinderinhg you more than anything else. You're obviously not english/american because of your limited ability to speak the language - that's fine. But simple courtesy will take you a long way. I've already done what you're asking for. And I'll happily share mu knowledge if you can ask in a civilised manner.

Until then...

Mark Shaxted
03-22-2008, 09:47 PM
I spent the last several years working in the processor industry and with the advancements we have gone through it makes no sense to me

You were the mail boy yes? Otherwise you haven't got a clue what you are talking about.

sqrt[-1]
03-22-2008, 10:40 PM
Most projects that render universe scales don't seem to have a problem.

Some nice recent work
http://www.gamedev.net/community/forums/mod/journal/journal.asp?jn=263350

Some older work:
http://www.openuniverse.org/


And as long as you have a modern Geforce 8 you can use floating point depth buffers:
http://www.opengl.org/registry/specs/NV/depth_buffer_float.txt

Zengar
03-23-2008, 07:53 AM
George, the solution to your problem is to render only visible objects, recoding the coordinates on-the-fly. I fail to see problems with terrain rendering and z-buffer. Of course you don't just feed the whoel terrain into the renderer. But if you render only the visible part with appropriate cordinates, even the 24-bit z-buffer precision should be enough.

Humus
03-23-2008, 02:02 PM
As I made the point it isn't like we need 200fps anyway and sure as heck not 400fps. 99% of people can not see flicker above 30fps.

Many of us write projects that struggle with real-time performance on highend hardware. I wouldn't want the API to chance so that molecule to universe scale application can render effortlessly at the expense of rendering performance. We do have floating point Z-buffers available in hardware these days. That plus a reverse Z mapping actually even gives better precision than what linear distribution would. We don't have doubles though, if that's what you require. The first hardware to support doubles in any shape or form was released very recently, but that's in the shader only. I doubt we'll see double Z-buffers anytime soon.


All of which becomes massive work wise. But would be extremely simplified with a proper z-buffer.

Actually, I don't think you've defined what a "proper z-buffer" is in your opinion.

Humus
03-23-2008, 02:10 PM
Well apparently the back face culling utterly fails when you have your znear and zfar to far apart from each other. Otherwise the objects that are culled should never bleed through and they do.

Backface culling is done before rasterization, so z-buffering have nothing to do with any artifacts you may see.
The only reason I can see why that would happen is because your projection matrix might get messed up if you go really extreme, but then I'd be surprised if your rendering looks right otherwise. Have you tried reversing your depth range and using GL_GREATER for your depth test? Even if you use a fixed point z-buffer this helps with precision (at the expense of somewhat worse z-culling performance).

CRasterImage
03-23-2008, 07:17 PM
Well apparently the back face culling utterly fails when you have your znear and zfar to far apart from each other. Otherwise the objects that are culled should never bleed through and they do.

Backface culling is done before rasterization, so z-buffering have nothing to do with any artifacts you may see.

Ooh. Good point. Those two things have nothing to do with each other. Unless... Are znear and zfar factored into the view matrix? Or are they just applied to Z values after transformation?

If they are factored into the matrix, then it is possible that a triangle's normal vector could fail to have a length after transformation. Due to floating point precision error.

If a triangle's normal had 0.0 length, then it would fail to reject during the backface cull step.

Bob
03-24-2008, 04:02 AM
If they are factored into the matrix, then it is possible that a triangle's normal vector could fail to have a length after transformation. Due to floating point precision error.

If a triangle's normal had 0.0 length, then it would fail to reject during the backface cull step.
Backface culling is based on the screen space winding order of the vertices when projected onto the screen. The normal is completely irrelevant to backface culling.

CRasterImage
03-24-2008, 08:47 AM
Backface culling is based on the screen space winding order of the vertices when projected onto the screen. The normal is completely irrelevant to backface culling.

Really? I was under the impression that they test the surface normal's Z component for negative.

ZbuffeR
03-24-2008, 09:36 AM
Really? I was under the impression that they test the surface normal's Z component for negative.
It was a wrong impression, Bob is right. Only the screen winding is used, no need for a normal.
http://www.opengl.org/sdk/docs/man/xhtml/glCullFace.xml

CRasterImage
03-24-2008, 09:57 AM
That is interesting.

My instinct says that it would be faster to just add z's of the triangle's 3 normal vectors together and compare the result against < 0.

Calculating clockwise or counter-clockwise winding is more computationally intensive than that.

My idea assumes that the back-face culling (skipping the rasterization of triangles based on whether or not they face the camera or not) is happening after all vertices and vectors have been translated into screen-space.

Humus
03-24-2008, 11:45 AM
It may be faster, but it would also be unreliable as there's no guarantee that the provided normals really match the input geometry. Any form of smoothed normals would be prone to artifacts. Besides, in the age of vertex shaders, the shader may not be outputting any normal at all.

Xmas
03-24-2008, 01:50 PM
Have you tried reversing your depth range and using GL_GREATER for your depth test? Even if you use a fixed point z-buffer this helps with precision (at the expense of somewhat worse z-culling performance).
Why would reversing the depth range affect precision at all for a fixed-point depth buffer?

GeorgeRH
03-25-2008, 07:01 AM
sqrt[-1] There is no project in existence that has the ability to traverse a planet to another to 1 meter or yard or less on a PC.
I am familiar with those few works none of which render detail or move into less than 1 meter.

The new extension may just work for this project. But it isn't standard in fact it is written against the standards. Which I can't blame them on. All of which means large compatibility problems and can't be used on other manufacture cards.

Humus
Most peoples speed problems come from trying to get the work done on the CPU and then getting the correct stuff to the display. In all likelihood a lot of projects would become simpler and there for faster.


As to the Z-Buffer and back face culling. I thought as I guess most people do that when they back faced culled an object it removed it. Apparently the back face culling is tied into the Z-buffer system on both Nvidia and ATI cards.
The so called artifacts you claimed I saw are as I said the reverse facing surfaces blended into the front facing surfaces.
I made sure of this. By reducing the scale of the object then producing it normal then turned on blending and it was identical.

When the object is scaled to a smaller size it displays perfect. When enlarged to the proper scale the back face culling fails and the front and back surfaces are blended.

The solution was to remove the back faces before sending it to the display routine.

To make the point. The current Z-buffer that is a standard is fixed point math and is perspective orientated. meaning more bits are used up in the near field of view than the far which is why the errors I am talking about happen.

Xmas: I tried pretty much everything to see what stuff would happen and to localize the failure to ensure it was z-buffer and not my own code or coords being the problem.

CrasterImage:
Screen Space winding. For those of you who didn't read the name of the term. It says screen space winding not world coordinate winding. My point yes it has been processed at that point to screen space and with the lack of depth precision at far distances it fails.

ZbuffeR
03-25-2008, 11:02 AM
Apparently the back face culling is tied into the Z-buffer system on both Nvidia and ATI cards.

No.

The so called artifacts you claimed I saw are as I said the reverse facing surfaces blended into the front facing surfaces.
I made sure of this. By reducing the scale of the object then producing it normal then turned on blending and it was identical.
We are talking about "back" faces, not all hidden faces, ok ?
Most probably your are doing something wrong.

Screenshot and code please ?

CRasterImage
03-25-2008, 11:19 AM
My point yes it has been processed at that point to screen space and with the lack of depth precision at far distances it fails.

Why would precise z values effect backface culling?

Or, to be more descriptive, why would a 2D clockwise/counterclockwise algorithm ever pay attention to z values?

My theory is that you did not enable backface culling and your extreme z buffer setting revealed the artifacts.

Humus
03-25-2008, 02:19 PM
Why would reversing the depth range affect precision at all for a fixed-point depth buffer?

It doesn't help the final storage, but it helps the computations leading up to the final Z value. It helped a lot for Z-fighting in the distance in our engine, even though we use fixed point Z. I was actually a bit surprised myself how big impact it had, even with a fixed point buffer.

Humus
03-25-2008, 02:31 PM
Apparently the back face culling is tied into the Z-buffer system on both Nvidia and ATI cards.

No it's not. Period.
You're either misunderstanding something, or you have a bug.


My point yes it has been processed at that point to screen space and with the lack of depth precision at far distances it fails.

The depth is irrelevant to backface culling. X and Y is sufficient. Check the OpenGL spec, section 2.14.1, equation 2.6.

Lord crc
03-25-2008, 02:52 PM
The depth is irrelevant to backface culling. X and Y is sufficient. Check the OpenGL spec, section 2.14.1, equation 2.6.

If you got perspective projection, those screen-space X,Y coordinates depends on the depth of the vertex, no? Still, I would be very surprised if the Z-buffer precision affected the coordinate transformations in such a way (any way really).

GeorgeRH
03-25-2008, 02:52 PM
CRasterImage
http://www.opengl.org/resources/faq/technical/depthbuffer.htm
Why the issue happens 100% I am not sure. But I do know if I bring the object at a much smaller size very close to camera the z-buffer does work. Only when I expand the object either by me or using opengl scale and move it to a comparative view size and distance then the z-buffer and such fail.

When up close if you press the first surface you can not see the rear surfaces. But once expanded it does not hold true.

I am not sure so much if it is a pure clock wise counter clock wise check they do. My guess would be do to the limited bits assigned at that range the system is unable to determine which faces are closer. There for it samples from both. But it could be just crap code by both companies creating an identical problem, doesn't seem likely unless one copied the others code exactly. In other words I doubt that, I think it has to do with the precision issue.

Here is the test project only where I was trouble shooting the problem. Yes, I am 100% sure that the back face culling was enabled. So are a large number of programmers on other sites like gamedev.net where I first brought the issue up.

The reason I brought it here wasn't really to discuss if the issue exists it is to find out what workarounds people have came up with for it.

So to make the point I been through the code and so have a lot better programmers than myself. The Z-buffer problem is well a well known issue. I just want to see what workarounds people have came up with.

Things like oct-trees sphere trees and so on only work to a limited extent unless you are going to does so to a polygon level.

By the way the test code in the project above is just that nothing more. It is seriously a mess and has been up-down modified to test most aspects by commenting stuff in and out.
It is a VC++ 6.0 project.

CRasterImage
03-25-2008, 03:48 PM
I am not sure so much if it is a pure clock wise counter clock wise check they do. My guess would be do to the limited bits assigned at that range the system is unable to determine which faces are closer.


Backface culling doesn't try to determin which faces are closer/further away. It tries to determin whether or not any given triangle is facing towards the camera, or away from the camera.

Originally, I assumed that this was determined by testing a triangle's normal's z component for positive or negagtive Z. (assuming the normal has been translated into screen-space coordinates)

I was corrected and told that it uses a winding test against the x and y components. If the triangle winds one way, it is facing the camera. Otherwise it is facing away from the camera.



By the way the test code in the project above is just that nothing more. It is seriously a mess and has been up-down modified to test most aspects by commenting stuff in and out.
It is a VC++ 6.0 project.


Was there a link to your code?
I didn't see anything.


Also, what is the answer to the question:
Are you making a space-based game? (stars, planets, spaceships, etc..)

If so, is your problem only associated with rendering terrain while near a planet? Or a problem of rendering space-born objects from space? Or a combination?

GeorgeRH
03-26-2008, 09:49 AM
There was a link not sure why it didn't show it but showed the other one. weird.

http://www.grhmedia.com/GWorld5.zip

No not currently making a game. I was thinking of building just a graphics engine but wanted to know what the hardware limits were these days. From talking to others it looks like the z-buffer issue has been kind of sat on for a long time. A lot of game companies are trying to rewrite engines at present time to give better depth and implement larger scenes with more detail. Most are finding they are having to take advantage of the multiple CPU technology to handle this because the one place the GPU hasn't advanced in 10 years is the z-buffer.

I think the issue is a lot of the time referred to as z fighting.
You can force it to happen in games you can make your own levels such as unreal and so on. They have came up with a number of solutions most of which means part of the scene does not get drawn. Fog, multidrawing then clearing the z-buffer between hand.

All of which have limits.

If you want to test what I am saying try this.
set your near to .1 and far to 1,000,000,000
create a sphere with multi colored faces about 67,000,000 radius put it out at 300,000,000
enable back face culling
What you will see is a sphere that looks like blending is turned on and you can see the surfaces on the back side.

The sad part is I can do this entirely on software and create my own z-buffer and rasterizer which is what I did over 10 years back and it will display perfectly.

Anyway it boils down to if you want a massive scene you have to do a few things make sure no objects are behind one another,
if you have a partially obstructed object you will need to split or subdivided the surfaces to get better visibility and only display the ones not obstructed. You will also have to manually back face cull.

Which means all that work that was done for years to move work from the cpu to the gpu just went up in smoke and amounts to a hill of beans.

One place this fix would come in real handy is with medical imaging. They receive near real time data and need to go from far out to extremely close up. It would greatly improve images and cut cost of the equipment.

Not to mention game environments could be made far larger than we currently are making them and make the ease of creating them far easier and less cpu intensive.

Just my thoughts on it

Humus
03-26-2008, 02:47 PM
If you got perspective projection, those screen-space X,Y coordinates depends on the depth of the vertex, no?

Yes, it depends on W, but not on Z or Z/W that's used in the depth test.

Humus
03-26-2008, 02:59 PM
I am not sure so much if it is a pure clock wise counter clock wise check they do. My guess would be do to the limited bits assigned at that range the system is unable to determine which faces are closer.

The problem is that you're confusing the backface culling with the depth test. The backface culling is working perfectly from what I can tell running your app. The faces that bleed through are all front-faces. You have z-fighting though, but that's hardly a surprise at these settings.

Humus
03-26-2008, 03:21 PM
Most are finding they are having to take advantage of the multiple CPU technology to handle this because the one place the GPU hasn't advanced in 10 years is the z-buffer.

Why do you say that? The Z-buffer has advanced at least as much as the rest of the pipeline. 10 years ago a lot of hardware didn't even have one. The ones that did had a 16bit one. Since then we've added 24bit, 32bit floating point buffer, depth textures, a whole bunch of optimizations (HiZ, EarlyZ, Fast Z clear, Z-Compression), depth bounds test, depth clamp etc.


The sad part is I can do this entirely on software and create my own z-buffer and rasterizer which is what I did over 10 years back and it will display perfectly.

Sure, and it'll render at 1/1000 of the performance of a video card. Yes, software is more flexible. It has always been and will always be. If we made GPUs as flexible as CPUs then they would also run at the speed of the CPUs, which would make it pointless. GPUs are specialized to handled normal rendering tasks. Your rendering case is extreme, so no one has spent any effort trying to accelerate it. It's not normal to have a planet/pea ratio. If you need doubles then there's only RV6x0 based cards that supports that yet, and that's not even exposed in any API now other than for GPGPU, and you'd still have to implement your own depth buffer emulation. Keep in mind that it was only a couple of generations since we even got floats.

GeorgeRH
03-27-2008, 11:27 AM
First off, I am not getting depth testing or z-culling confused with back face culling.
Z-culling or depth testing simply puts the approximate depth of a pixel in a buffer or assigns it associated value based on depth as in opengl it either 0 to 1 or -1 to 1 depending on what articles you read. But the fact is they use perspective to assign bit values. so what is closer has greater depth.

This however is similar to w-buffer which is a pure float that is independent for each pixel and can actually give a better image in many cases.

Back face culling or winding tests work in one of two ways by either comparing 2 vectors one from the view point direction out ward to the surface normal. That is not used on most hardware in fact I can not think of a hardware implementation off the top of my head not saying it doesn't exist.

Then there is screen winding. Faster than the vector test method but has limitation that come to lighting and shading.

I hope that establishes I know what the difference is.
************************************************** ****************
Now my issue since you seem to miss the point over and over again I will step through it.
Use my test project if you must.

Take the same object make it small and view it close up you will see back face culling is working perfectly when you break the frontal surfaces you will not see the rear ones nor do you get artifacts or more precise see the rear surface.

That should establish the back face culling works fine.

Now enlarge the object and watch the surface start to show through and z-fighting begin.

Now that we know the back face culling is implemented correctly and yet we are getting z fighting.

Explain how that is happening.

Those other surfaces should not exist if the back face culling worked.

How is it then that the surfaces that should be back faced culled are still showing up if they are not tied into the screen depth algorithm?
************************************************** ***************
I know the settings are at fault for it z-near and z-far. They affect how far the z-buffer is spread out. However even if you change to 32 bit mode in my setup.txt file it isn't enough. it would require large number of bits or more to come close to that scale of variance. Which is funny a 32bit W-buffer would still out perform it. because most the detail would get wasted up close on the z-buffer. A 64bit w-buffer would slaughter it.
************************************************** ***************

One of the better ideas I heard to over come the limitation was a variable z-far and z-near and clear the z-buffer between them. It would mean each object or parts of objects would require being ordered from far to near then taking those objects and displaying them with different z-far and z-near values which would mean you would also need to scale the to bottom left right as well to get the correct perspective and screen position.

Then there is the idea of object culling, frustum culling, then back face culling surfaces and making sure partially covered surfaces are split and then only the visible surface displayed.

Anyway no matter what method is used to over come this problem it means dumping far more work back on the CPU that really could be better handle on the CPU with a different implementation.

Then there is a second down fall each requires even more calls to display an object.

Komat
03-27-2008, 01:36 PM
Now my issue since you seem to miss the point over and over again I will step through it.

Unfortunately you are the one who is missing the point.

Back face culling removes faces at which you look from the incorrect (back) side. It does not care if the face is hidden by something else. As long as you see the face from the correct (front) side, it will pass the back face culling. The correct ordering of faces is job of the depth testing.

Because your planet is not convex, there are situations when you see two faces from the front side while they overlap on the screen (those are the faces you see as the artifacts). Both faces will pass the back face culling (because you see them from the front side). If the depth test works, only the closest from them will be visible. If it fails (because of the extreme depth range and limited buffer precision), the one which is drawn as last will be visible resulting in artifact you see.

I will suggest simple experiment. Add a key to your program which will toggle on and off the depth test. Now use the small planet which looks correctly and disable the depth test. You will see similar kind of artifacts you are seeing on the big one. Now add a key which will toggle the back face culling. As before use the small planet and toggle it. You will see that the artifact does not appear.

ZbuffeR
03-27-2008, 03:07 PM
Anyway no matter what method is used to over come this problem it means dumping far more work back on the CPU that really could be better handle on the CPU with a different implementation.

Then there is a second down fall each requires even more calls to display an object.
GeorgeRH, you don't know what you are talking about, at all.
You can store an insane amount of geometry and texture and shaders on the GPU and draw even a very complex scene in just a few draw calls, with almost zero use of CPU, only GPU side will be used. Then simply repeat those few calls multiple times, each with specific znear/zfar ranges. Ie [1-1000[ then [1000 1000 000[ then [1000000000-1000000000000[ ... Hardly taxing on the CPU side, and the GPU will still be mostly asleep to draw your current scene :-)

Please don't behave as someone who know about it (you don't, everybody agrees on that) and stop complaining (as you have no reason to), that way you can still get good advice from experienced people.

CRasterImage
03-27-2008, 04:44 PM
Modern hardware is good, but it isn't perfect.

Would people like a higher resolution z-buffer? Sure!

Would we like to use 4-byte floats for color components rather than 8 bit unsigned integers that pretend to be floats? Sure!

Would we like more programming language features in shaders? Sure!

Would we like a blend shader? Sure! (this is what I want)

etc...

Many of these things are already possible, with the more recent GPUs. More of these things will become available over time.

The problem many here have with your post is:

You make it sound like the graphics industry went off in the wrong direction for the past decade. Somehow, forgetting to consult you first. (In spite of the fact that you haven't been a part of the graphics industry for the past decade)

You also imply that there is some large, faceless mob of angry programmers who meet on a regular basis just to bemoan the lack of floating point zbuffers.

On this board, I have seen many people seek help with various problems. I haven't heard anyone complain about zbuffer precision. That doesn't mean that no one has had that complaint, but just don't recall seeing it lately. I have no doubt that it has been an issue for a few individuals. However, it is not some industry-wide holocaust issue that we are all struggling with every day.

Everyone has their own personal "wish list".
A set of features that would suddenly make their projects much more strait-forward to code.

Life goes on. You either scale back your goals. Or write specific work-arounds for problems. Or reduce your target platform to just the high-end cards that support feature X, which solves your problem.

People on this board, who have many years of experience, have been trying to suggest work-arounds for your problem.

Listen to them.

The most common solution that has been suggested is:

Break your rendering pass into seperate passes based on depth levels. For each group, set the z_near and z_far to envelope that group's depth extents. These depth-groups should be sorted far-to-near.

How should you do this?
We can't tell you exactly.
It depends on what your data is like.

If it is a bunch of planets, moons and stars, you could practically render each heavenly body in it's own seperate z-range pass.

If it has lots of objects that partially occlude eachother and occupy the same space, (like small space craft flying near larger space craft) then you would need to group them into clusters of objects that have similar depth ranges and render them with a z-range that encompasses them.

If it involves rendering mountains and other distant terrain along with details right under your feet, then there are other methods you could use.

Tell us some specifics, without the attitude, and people will make more specific suggestions to solve your problem.

These people love this stuff.
It is often their hobby as well as their job.

GeorgeRH
03-28-2008, 08:20 AM
Komat and all: You are right on that issue, I missed your point. The surfaces showing through are just the ones that happen to face forward at the time because the elevation change.
Thanks, for restating it again. Guess I just needed it drilled in one more time. Sorry on that everyone.

Your right also on the bitching and complaining sorry again.

I do know as you said the geometry and much of the data can be left on the gpu side or video buffer.

Maybe, what I should be asking is a better question.
The shader technology today. Is it capable or being used to reorder the calling of the geometry.

In other words calling the objects from far to near or more precise the vertex data and the changing the z-near and z-far and so on? Then of course the z-buffer would be need to be cleared in between each new progressive step.

The project isn't aimed at one small area. Planets and so on is really a test data set. The reason for using it is it gives a good example of scale and change in perspective. Or something like the human brain and you wanted to go from seeing the entire thing to seeing a single cell.

I know ATI had on the rage model in 1999 16,24,32 bit z-buffers. Then I am not sure on Nvidia and 3dFX.

Guess I need to do some research into what can actually be done with newer shader technology.

Zengar
03-28-2008, 08:56 AM
To use the approach ZBuffeR and others suggested, you don't have to sort all the objects. You just have to determine if the objects fit into a particular depth interval. This is a O(n) operation and can be performed on the CPU really fast.

ZbuffeR
03-28-2008, 09:09 AM
To use the approach ZBuffeR and others suggested, you don't have to sort all the objects. You just have to determine if the objects fit into a particular depth interval.
Even not needed. Object outside the znear/zfar will be clipped away automatically by the GPU.
So you can choose your tradeoff between CPU usage, GPU usage, and complexity of code, with more or less precise CPU culling.

-NiCo-
03-28-2008, 10:07 AM
Hi guys,

Because of this thread I started experimenting with projection matrices and the impact of the 3rd row on the z-buffer accuracy. Just for fun I tried to replace the 3rd row of the projection matrix from glFrustum with the 3rd row of the projection matrix from a glOrtho call with the same near/far values.

Small note: It does appear to experience some weird behavior when resizing the window. Don't know why...

To my surprise, this seems to work. The problem is, mathematically this doesn't add up. In ortho mode near/far maps to [-1,1] (clip space) and the w coordinate remains 1 after projection so there's (theoretically anyway) no division to get to normalized device coordinates. However, in the matrix I set up near/far still maps to [-1,1] but the w-coordinate is equal to -Z. So there should be a division by w before the depth values get interpolated, no? If so, then why are the depth values I read out still in the range [0,1].

I'm only asking this question to get a better understanding of what calculations are actually being done by the hardware and at what times. I thought I figured everything out like perspective correct interpolation using barycentric coordinates while depth values are linearly interpolated in screen space, etc. but after encountering this issue it looks like I have to think twice :)

GeorgeRH
03-28-2008, 10:12 AM
Just my guess on this.
If I placed all the objects that were in side the view frustum and just started from far to near I could just call the entire buffer as Z-Buffer the stuff out side it would be automatically culled by hardware. Then as I step closer I correct the frustum and clear the Z-buffer.

To be honest I am having a bit of trouble implementing this. I want to do it manually then I'll move it over to a VBO and such to do what I can. When I do get it to display it still shows the same problems. So obviously I am doing it wrong.

When changing the frustum I need to first step back to projection view to start with?

glMatrixMode(GL_PROJECTION);
glLoadIdentity();

//back to model view to do my translations
glMatrixMode(GL_MODELVIEW);
glLoadIdentity();

//translations here

//clear z-buffer here
glClear(GL_DEPTH_BUFFER_BIT);
glLoadIdentity()

//then redraw here

GeorgeRH
03-28-2008, 10:32 AM
Not sure. Whatever they are doing on hardware is nothing like what I wrote ages ago on software.

Between the two I would have figured Z would have been treated the same.

Xmas
03-28-2008, 07:11 PM
To my surprise, this seems to work. The problem is, mathematically this doesn't add up. In ortho mode near/far maps to [-1,1] (clip space) and the w coordinate remains 1 after projection so there's (theoretically anyway) no division to get to normalized device coordinates. However, in the matrix I set up near/far still maps to [-1,1] but the w-coordinate is equal to -Z. So there should be a division by w before the depth values get interpolated, no? If so, then why are the depth values I read out still in the range [0,1].
The result of the vertex shader or the fixed function modelview-projection transformation is a 4D homogeneous position in clip space.

The hardware then takes three vertices for a triangle and clips it so that X, Y, and Z are between -W and +W. After that you divide X, Y, and Z by W to get normalized device coordinates, each in the range [-1, 1]. Then the viewport transformation maps Z from [-1, 1] to [n, f] where n and f are the parameters to glDepthRange, typically [0, 1].

-NiCo-
03-29-2008, 04:33 AM
After that you divide X, Y, and Z by W to get normalized device coordinates, each in the range [-1, 1]. Then the viewport transformation maps Z from [-1, 1] to [n, f] where n and f are the parameters to glDepthRange, typically [0, 1].


That's exactly my problem. It looks like the division by w is not happening for the case I explained in my previous post otherwise I would be seeing much smaller values (unless I draw at unit distance or less away from the camera of course).

GeorgeRH
03-29-2008, 07:18 AM
As I said above I have no idea what they do on hardware. Never had to divide by W for screen transformation on my manual rasterizer. Simply transformed the object to be inline with the view port in other words moved the objects not the view port. Then divided the x and y by angular value which was determined by what width of view angle you chose.
So for perspective view I could have a value such as .5 dividing all the x and y but if I wanted to change it to 1 then then it would give me ortho. The only thing z was used for at all was determining depth in the z buffer.

So for me no clue at all on that.

I did manage to get the other thing working. Works fine until I add a secondary sphere to the picture and reduce the terrain level to a realistic height. Then I get z-fighting between the two spheres. The primary sphere is easily fixed by dividing in half. But even stepping every 4000 which take minutes not seconds to render a frame does not fix the issue and so I will need to figure out a better way such as faking the perspective. Which so far early test indicate it will work just requires a lot more coding.

Lord crc
03-29-2008, 09:01 AM
Then divided the x and y by angular value which was determined by what width of view angle you chose.
So for perspective view I could have a value such as .5 dividing all the x and y but if I wanted to change it to 1 then then it would give me ortho.

Then you haven't used perspective projection at all, you've used orthographic projection with a different scale factor.

For perspective projection you must divide by z (or w if you use homogenous coordinates).

GeorgeRH
03-29-2008, 02:16 PM
Sorry, your right it was also divided by z but not w is what meant.
The .5 was for the width of angle you wanted to bring in. The Z in the ortho was only used to get proper depth.

GeorgeRH
03-29-2008, 02:36 PM
Wanted to say thanks again to those who helped out and bore the brunt of me being an #$$. Much appreciated and again sorry.

I did manage to get the concept working where it draws the back first then the near part of the object. This works very good on a single object to an extent it is still very limited.
The more you have over lapping surfaces or lawyers you have the less well it works and the closer those layers run the less it works as well.

Both of which means subdividing the object from back to front even more in thinner slices. The problem with that is something the size of a planet in scale value would take far to many calls to make a decent frame rate.

I did learn something from this the ability to change perspective is incredibly fast can even happen in the same frame and has no flicker effect.

I thought of a couple possible solutions. Change the perspective on the object over movement distances. The object can easily be scaled so as the perspective change is never noticed as it approaches. Only once the object reach very close up will the scale reach full size at which point in time the z fighting for any object will never be notice able because all the near surfaces and usual culling such as frustum and so forth.

This would only require keeping a real position of each object and a scaled position of each object along with scaled size and so on.

Humus
03-29-2008, 04:03 PM
However, in the matrix I set up near/far still maps to [-1,1] but the w-coordinate is equal to -Z. So there should be a division by w before the depth values get interpolated, no? If so, then why are the depth values I read out still in the range [0,1].

Final depth values will always be [0..1], but your actual near and far plane are probably not where you expect them to be from your calls, nor will your depth distribution be linear just because you copied the Z row from glOrtho. glOrtho being linear has more to do with the W row being [0 0 0 1].

-NiCo-
03-30-2008, 07:50 AM
Thanks for the replies, guys.

Actually I wasn't trying to linearize the depth, I already solved that years ago by using



uniform float near;
uniform float far;
void main()
{
gl_FragDepth = ((1.0f/gl_FragCoord.w)-near)/(far-near);
}


I know this can be optimized further and disables Early-Z but I was more concerned about not having to assign a varying to it.


Final depth values will always be [0..1], but your actual near and far plane are probably not where you expect them to be from your calls.


True. I recalculated what the equivalent near and far values would need to be to get the same 3rd row as the glOrtho call. It appears the far value is negative which might explain the flickering and weird behavior when resizing. However, it appears that the model I'm drawing is still being clipped at the near and far values I specify and not the recalculated near and far values. That's what confuses me... But since the equivalent far value is negative I'll just put this issue in the undefined behavior section :) Thanks for the help.