Creating a hole

I’d like to create a hole in some parts of a rendered image so that I can see what’s behind that hole. Unfortunately I can’t create the hole within the geometry (for some special reasons). Also I cannot use the scissor test because the geometry can be quiete complex. So I thought I could use the stencil test to do that. However I didn’t found any solution to it.

Let me try to explain better what I’d like to do. Let’s say I render a house from within that house. I’m in some room and seeing at a wall of this room where’s there’s a ‘door’. I’m also rendering what’s behind the wall and the door, but of course, this is not seenable unless the door is not rendered (erased).
Now, let’s say the user makes an action so that the door should be ‘erased’ from the drawing. But I cannot effectively erase this door from the geometry just because it is full part of the wall and the only thing I can know is where the door is and what is its aspect. (now for more information: that door might be purely virtual so that I can move it all along the wall, just in order to see what’s behind this imaginative door). Also, that wall might be very detailed: it’s not a pure simple quad and can have a lot of relief.
Now, when the door is ‘erased’, then we will be able to see what’s behind the wall (probably the other rooms of the house).

Well, I found a solution but that works only for pure flat walls, so this does not really help me in this sense.

Are there any ways to do so ? Maybe an efficient one (avoiding multipass could be very nice).

Hope this is understandable.

Why not simply store the door as a separate mesh in memory and then not render it??

Because I cannot. Geometry isn’t mine. Also the door should be able to move threw the wall.

Sounds like you need subtractive CSG, that is, cut some geometry out of some other geometry.

Here is a library that can render CSG geometry: http://www.opencsg.org/

It’s GPL. If you prefer to write your own code, I’m sure google will find tons of papers with different CSG algorithms.

The thing you propose sounds interresting. However, this does not seem to be what I was looking for. Or maybe did I misunderstood the purpose of CSG ? I’ll have a deeper look at that anyway.

What I’d have like is a way with GL to eliminate a portion of some existing and unmodifiable geometry in real-time but without changing this geometry, but only with using some technics in GL.

It appears that I can do such things using the stencil buffer, but unfortunately I failed. It seems it is due to the fact that all rendering that goes threw the stencil test also go to the framebuffer so I cannot eliminate some parts of the rendering in an easy way.
Isn’t there a way with the stencil to say: do not draw in that portion of screen ?

I’ll might have a better look at scissor since it appears it’s the best thing I have between my hands. But in another hand, isn’t that scissor a special case of the stencil test ?

For more information: The door could be added or removed from the wall in real-time, just for some specific purposes. When it is added, we can delete it (not the same as removed), so that we can see threw the wall at the location where the door was. When it is removed, then the wall hasn’t got any hole in it.

Thanks for the care.

From what I understood, the door is just part of a texture. You could preprocess that texture so that you have 2 versions of it. One texture has alpha 1.0 everywhere and another has 0.0 for the door and 1.0 everywhere else. Use blending or alphafunc or from GLSL the discard function.

Scissor works in window space and is square. There isn’t a inverse-scissor so that you could exclude rendering to the door.

You all have so nice ideas. But why do I have so bad circumstances ?

Yeah this is the problem with scissor test. I definately cannot use it for that purpose. So stencil seems to be my last resort.

The main problem with what I have is that the door isn’t only a texture. It can be a full geometry put on uppon the wall the latter that has absolutely no hole in it. The next stage of the development would be to animate that door so that it can open or close interactively (as also move on the wall).

The room is definately close and I cannot change that facts. The door(s) can move all along the wall(s) and when the door is open (or deleted for using what I said before), I need to see threw the part of the wall where the door was.
I cannot force modelers to put static holes in the rooms just because adjacent rooms might be added dynamically during the program life and so the doors can be set up at different locations depending on the newly added adjacent rooms.

To my point of view this is only a question about what we call in french ‘pochoir’ or in english ’ stencil key set’ (purely get from a translator).

Finally I’d like to add that I have a solution but that suits only for flat planes, which isn’t the case all the time. It is a really easy, fast and efficient one that does not require stencil. It is a harm that I cannot use it under other circumstances.

Originally posted by jide:

Isn’t there a way with the stencil to say: do not draw in that portion of screen ?

to disable drawing to the color buffer, still writing to stencil and/or depth buffer :
glColorMask(false,false,false,false);

If I understood well, you want to “boolean substract” a virtual door from a complex wall : seems really what can do this image-based opencsg lib like said Overmind.

jide - the stencil approach should work, but will of course only work if the wall is flat (ie. has no depth). You should of course only use the stencil-test for eliminating the unwanted part of the wall, and then use depth-test like always for rendering the visible part of the remaining objects.

So it goes a little something like this

• clear stencil-buffer.
• render a a boundingbox of the door to the stencil-buffer.
• render the wall to color/depth only where the stencil is not set.
• disable stencil-test, and draw everything else.

Hope this helps,

You understood well ZbuffeR.

The problem I found with OpenCSG is that phrase:
“CSG is often used as fundamental modeling technique in CAD/CAM applications”

I must do that in interactive framerate (and even better). The modeling stage has already been done and the geometry has been sent to the graphic hardware using static VBO. I’ll still have a better look at OpenCSG but at this time, I’m persuaded this won’t be of any kind of help for me (I really croos my fingers to be wrong, for example I could switch to dynamic VBO).

I know about ColorMask. This was the key for my ‘almost solution’. But when the door and the wall aren’t exactly parallel and do not have the same geometry, this didn’t helped me.

PS: I might seek in the wrong direction. This is so often the case when having difficulties in programming: there might have a so evident solution that I can’t see it. Using ColorMask was a good starting point, so I’m gonna see it more deeply.

PS: yeah, bounding, I’ll have a look at it also, thanks Mikkel Gjoel.

I guess I found a solution, thank you ZbuffeR for having let me see again to the color mask solution. I tried a different order to render things and here is a solution of what I made:

. Set depth func to less (was set before the l_equal).
. Draw what’s behind the wall first (this is not what I’d like but as this works and as I do not have any better solution for the moment…)
. Disable color masks
. Render the opening on the wall (the door when it is erased)
. Enable color masks back
. Render the walls.

I had this solution between my hands but for some few things it didn’t worked as expected.

Have you got any hints here for allowing me to render in the same order for all the parts ? I effectively prefer rendering from front to back for some reasons. And doing such a mixture does not help things.

I’ll still have a look at the stencil because I guess it will be more suitable for any circumstancies.

If you have any other ideas, please let me know

The “naive” stencil buffer approach works only in screen space, you can just “disable” some screen space pixels from being drawn.

You should really look at CSG again, I think it’s exactly what you want. It also uses the stencil buffer, and it does work in realtime.

“CSG is often used as fundamental modeling technique in CAD/CAM applications”
Where’s the problem?

You want to modify some models without actually modifying the vertex data, so CSG is exactly what you want, and you want it for exactly the same reason why it’s used in CAD/CAM applications

The only difference between your requirements and CAD is that you want it in realtime, but as I already said, the stencil-buffer based approach of OpenCSG works in realtime.

Okay I understand more about CSG. For the little story, in fact I did knew it a bit, some modelers can do that, but I didn’t knew its denomination and I’m not used to use them.

But I’ve got some problems: if the door and the part of the wall that will be erased are not made from exactly the same vertices, then none of the CSG operations will give a good result. Just imagine the wall is not purely flat (maybe wavy ?) but the door is purely flat. Then each of the CSG substractive operations will give hole only where both polygons from the wall and the door ‘meet’. So this will render a ‘puzzled opening’.

Anyway, I downloaded OpenCSG so that I can make tests in real situation and so that I can ensure if I understood CSG properly or not: maybe I can use ‘filled geometry’ which will definately be the solution to my problem. (With filled geometry I mean for example a ball not only container its surface but also all the points inside the surface, so some kind of a rock).
I know, I can become very annoying…

Also, the solution I told I had found showed exactly the same problems than the one I gave on example above.

Thanks.

CSG always works with filled geometry. You don’t define the door as a polygon, you define it as a box. This box simply has to be wider than the wall, including any surface features of the wall.

What you basically do is, define a 3D body that will be “cut out” of the wall. That way you can produce arbitrary complex cuts.

Thanks for that information. I’ll definately make a try with this library. Unfortunately I didn’t already done it. I was still looking for a solution using the stencil. And I almost have had something convicing. But as often with the stencil test, I encountered problems, here due to the depth test: I’m trying to achieve it letting the rendering order unchanged. I could do it properly with having another rendering ordering, efficiently but that does not suit my requirements, unfortunately.

I also think I would avoid CSG and would prefer the stencil solution which seems faster at my opinion. I will use CSG only if I can’t afford that issue directly threw GL.

Here is a little code example of what I have:

   glEnable (GL_STENCIL_TEST);
   glStencilFunc (GL_ALWAYS,0x1,0x1);
   glStencilOp (GL_ZERO,GL_ZERO,GL_REPLACE);
   glColorMask (GL_FALSE,GL_FALSE,GL_FALSE,GL_FALSE);
   glDepthMask (GL_FALSE);
   glColor3f (1,1,1);
   glutSolidCube (1);
   glDepthMask (GL_TRUE);
   glColorMask (GL_TRUE,GL_TRUE,GL_TRUE,GL_TRUE);
   
   glStencilFunc (GL_EQUAL,0x0,0x1);
   glColor3f (1,1,0);
   glBegin (GL_QUADS);
   glVertex3f (-1,-1,0);
   glVertex3f (1,-1,0);
   glVertex3f (1,1,0);
   glVertex3f (-1,1,0);
   glEnd();

   glPushMatrix();
   glColor3f (1,0,0);
   glTranslatef (0,0,-5);
   glRotatef (a,0,1,1);
   glutSolidCube (1);
   glPopMatrix();
   glDisable (GL_STENCIL_TEST);

You might best understand both what I’d like to do and what are my current problems.
Here, the rendering order is respected. The fact that I render the wall first, then the hole, or the reverse is not of importance. The main thing is that I render what’s behind the wall at last.

you could chuck the whole kit and kaboodle and portal render, front-to-back. but i won’t lay that one on you if you’re pinned down by that stencil buffer. might just complicate things. heck, maybe i misunderstood the whole thing.

No give a try.