Vista optimization problem

[LoopinFool]

We recently started testing one of our products on Vista 64. We encountered a problem I was unable to completely solve.
Our OpenGL preview window is a child of the main application's window. It has WS_CLIPCHILDREN and WS_CLIPSIBLINGS set. It gets re-drawn constantly because this is a real-time video application. Normally, the preview window is smaller than the main window and has various buttons around it.
We also have a mode where we re-size the preview window to fill the screen. Here's where the problem occurs. We noticed that all the buttons and windows that should be on top of the preview window had disappeared.
It appears Vista (or the OpenGL driver) detects that the window is now full-screen and enables "game mode" for us. Our OpenGL window then draws over all of its sibling windows. We do not make any API calls to enable this mode and certainly don't want it (since it breaks our application). We also occasionally put up progress dialogs and such, and those get obliterated, too.
I tried a number of things to avoid this "optimization" like setting PFD_SUPPORT_GDI or PFD_SUPPORT_OVERLAY, enabling/disabling Aero, opening a POPUP window above the app in the Z-order, etc. I was only able to work around it by sizing the preview window one pixel shy of the bottom of the screen.
I first realized what was happening because our app would return to normal when other applications' windows were on top of ours, or the Alt-Tab window was open. There's also a brief flash to black when it enters or exits the optimized mode.

This happened using an nVidia Quadro FX 4600 w/169.96 drivers.

I did some serious internet searching and couldn't find any references to this issue. So, has anyone encountered this and/or know a way to turn off this undesired "performance enhancement" being forced on us?

- LoopinFool

[Madoc]

We've encountered what I believe is the same issue in a windowed app of ours in the form of an apparent mode switch when none was requested. This doesn't affect our app or (custom) GUI but I tried to figure out what was going on anyway. Seems no matter what vista takes some additional steps when the window is made "full screen".

An older app displays a windows open dialogue (while full screen), this again causes an apparent mode switch and the dialogue does not appear. However, when alt+tab is pressed the dialogue appears...

Looks to me like vista might be screwing with the z-order when it does this and you may be able to encourage it to sort it out. Perhaps try (re)setting the z-order of some window in your app after a delay?

[LoopinFool]

Glad to hear someone else has seen this!
I looked into the Z order a bit and I don't believe that's it at all. Also, windows in front of the OpenGL window are still there and respond to mouse clicks.
It appears to have switched into a direct-to-screen ICD mode which simply doesn't do any clipping, so our OpenGL drawing obliterates the pixels of windows that should remain visible. They do flash on-screen a moment before they're drawn over.

- LoopinFool

[Madoc]

Oh, well I guess it's an additional issue then.

I don't know if it's feasable at all for you but you might consider implemeting a simple opengl rendered GUI system. Depending on what you do with the GUI (and whether it matters) you might also see a big performance benefit. At least this was the case with drivers years ago.

I can't say I'd reccommend implementing a very complex GUI system, I've done this and I have to say it becomes far more complicated than I would have imagined. A few buttons and checkboxes etc is trivial though.

[LoopinFool]

We've already thought of that, but it's really not worth it for this particular application. We just need it to run on systems shipped after XP is gone.

Unless we can find a magical window style (or something) that turns off this Vista "feature", we'll probably have to live with the one-pixel-too-short solution. It may work ok for this app.

- LoopinFool