culling meshes across mulitple layers

My FBO is divided into 2 layers which are concentric. First layer is having radius 0.3 and second layer having radius 1 so that it covers the whole screen space. Now I want to avoid to render meshes in outer layer which are already in inner layer. So I calculate the distance between line of sight(where the eye vector hits the screen) and bounding sphere center and checking it against the radius of layers. Now for the meshes which are having the center in lower layer but part of that present in outer layer, those are not rendered in outer layer. To avoid this I thought of solution to transform bounding sphere radius and center of sphere in 0-1 and check the addition of them against radius of layer. but somebody pointed out to me that its not possible to transform radius into 0-1. How do I solve this problem? when I say inner layer, I am using glscissor of that radius.

Let me know if any part of the question is not clear.

I have implemented following culling algorithm which is only concerned with culling between the layers:

  1. I create a rectangles with min and max corner from the radius of 2 layers. Now these coordinates are in NDC.
  2. I have min and max corner of each mesh in local coordinate system. so I transform them into NDC using:

vec4 min=mvp * vec4(min_corner,1.0);
min=min.xyzw/min.w;
3. I create bounding box using min and max corners.
4. Check whether 2 rectangles overlap or not.

Now, when I transform into clip space, the x or y coordinates of min are sometimes greater than max corner. Same issue occurs when I divide them by w. Whats going wrong here?

You’re skipping the culling step.

Suppose your eye-space Z is 0. For a perspective projection, that makes min.w == 0, so your math blows up.

Suppose your eye-space Z is 10 (behind the eye). For a perspective projection, that makes min.w == -10. Dividing XYZ by this value negates (and scales) your XYZ, resulting in nonsense as this point has no analog in your space.

[QUOTE=Dark Photon;1281437]You’re skipping the culling step.

Suppose your eye-space Z is 0. For a perspective projection, that makes min.w == 0, so your math blows up.

Suppose your eye-space Z is 10 (behind the eye). For a perspective projection, that makes min.w == -10. Dividing XYZ by this value negates (and scales) your XYZ, resulting in nonsense as this point has no analog in your space.[/QUOTE]

does this mean I am passing the test for the primitives which are supposed to fail? But issue is that with above algorithm I don’t see some of the objects in the innermost layer as they are being falsely failing above test.
Also I get coordinates getting messed up even after multiplying with MVP matrix, why ?

After this test I have another test of frustum culling which is not based on culling between the layer. So without my algorithm, I see all the objects being rendered correctly but of-course there is overdraw of some of the objects in outer layer which are already rendered in innermost layer.

Is there anything wrong with my algorithm? What corrections needs to be done?

Okay. I found the problem, now I iterate over all the vertices transform each of them into NDC using MVP and then calculate min and max corner which works perfect. But this is much more overhead for each frame. Is there any optimization I can do here?

would there be any case that bounding box will change after transforming object into NDC space. Because when I use bounding box in every frame and calculate NDC using MVP I get incorrect output.

I am still not able to figure out why output is incorrect when I use bounding box calculated in object space. here is the data I got as output:


model matrix
  1.000000 0.000000 0.000000 0.000000  
  0.000000 0.000000 -1.000000 0.000000 
  0.000000 1.000000 0.000000 0.000000 
  0.000000 0.000000 0.000000 1.000000 

view* projection

 0.920664 -0.285565 -0.266194 -0.266141  
   0.039495 0.746434 -0.664418 -0.664286
  -0.388354 -0.601072 -0.698633 -0.698493
-0.063192 -1.194295 0.894055 1.093857

  MVP matrix 
.  0.920664 -0.285565 -0.266194 -0.266141  
   0.388354 0.601072 0.698633 0.698493
   0.039495 0.746434 -0.664418 -0.664286
   -0.063192 -1.194295 0.894055 1.093857

  bounding box in object space
  -23.376505 -23.376505 -0.023644
  23.376505 -23.376505 -0.023644
  -23.376505 23.376505 -0.023644
  23.376505 23.376505 -0.023644
  -23.376505 -23.376505 1.430214
  23.376505 -23.376505 1.430214
  -23.376505 23.376505 1.430214
  23.376505 23.376505 1.430214

  after transformation with MVP
 -30.664371 -8.587399 -9.199142 -8.997323
 12.379420 -21.938423 -21.644522 -21.440216
  -12.507672 19.514534 23.464052 23.659344
  30.536121 6.163512 11.018670 11.216449
  -30.606953 -7.502190 -10.165111 -9.963099
  12.436841 -20.853212 -22.610493 -22.405994
  -12.450252 20.599745 22.498081 22.693565
  30.593540 7.248722 10.052701 10.250672

 the bounding box after MinMax NDC min -0.577393 0.549507 1.000000 max 3.408166 1.023237 1.000000 
 after NDC 
  3.408166 0.954439 1.022431 1.000000
  -0.577393 1.023237 1.009529 1.000000
  -0.528657 0.824813 0.991746 1.000000
  2.722441 0.549507 0.982367 1.000000
  3.072031 0.752998 1.020276 1.000000
  -0.555068 0.930698 1.009127 1.000000
  -0.548625 0.907735 0.991386 1.000000
  2.984540 0.707146 0.980687 1.000000

.min -0.577393 0.549507 max 3.408166 1.023237 

whereas min and max corner for buffer is min: -0.53, -0.53 max: 0.53, 0.53. It is culling the meshes which are not supposed to.

what could be going wrong here?

Anyone? Let me know if anything is not clear.

I am struggling with this since long time. Any help would be greatly appreciated.