3D window manager
Now that 3D HW is popular, it appears to me that at long last, it may be time to move to full 3D window management. A relatively easy first step might be to use one of the existing 3D object scene managers, and write an X11 screen driver that merely calls OpenGL functions in that scene manager instead of calling the 2D drawing routines.
This isn't the most efficient way possible, but there are very few primitives in X11 so it would be a good 'proof of concept'. The result would be the possibility of keeping multiple virtual X displays floating in a 3D space. Thus, the existing X11 window manageres would continue to manage the X screen without knowing the screen is virtual.
Of course, there's lots of work to follow up, including determining ways to handle the context switches and managing the virtual screens. This could be done with a program that is called with a special keystroke, for starters.
In the long term, an OpenGL display manager has the potential of being smaller and more efficient than X11, and could replace X11 under the open source OS. But it must support legacy X11 apps, and this is the way to do it.
Re: 3D window manager
Just to clarify two points - What I'm suggesting is essentially splitting XFree86 (for example) into the device driver layer and the drawing layers, and inserting OpenGL in between. This would move OpenGL closer to the hardware and X11 farther away.
At present, OpenGL pictures are drawn on an X frame managed by X11. The purpose here would be to invert that relationship. It would be possible (though silly) to run an OpenGL app in a window on an X display, which is being displayed on an OpenGL scene.
The main thrust is that X11 is big, old, clunky, and obsolete but ubiquitous. In particular event context is difficult. OpenGL (one hopes) is cleaner and better designed for interactive applications. So it makes a good candidate for a replacement to X, and has the 3D 'killer app' to make it popular enough to replace X.