Using OpenGL in a browser plug-in

Is this possible? How would the OpenGL context be created in something like an ActiveX control or a Firefox plug-in? Is there any particular reason why it would not be possible to build a browser plug-in that renders using OpenGL?

You might want to have a look at WebGL: http://www.khronos.org/webgl/

That looks very interesting except that Microsoft aren’t on board (no surprises there!) so no IE support.

Are there any reasons why OpenGL couldn’t be used in my own plug-in?

Well opengl can be used in your own plugin. Quake Live does this AFAIK.
These pages can provide more help :
https://developer.mozilla.org/en/Gecko_Plugin_API_Reference/Plug-in_Development_Overview
Maybe it is easier to create a windowless plugin, that create a GL window like a classic GL program.

Off-topic but still interesting, speaking of WebGL these a neat tool to play with GLSL shaders in your browser :
http://www.iquilezles.org/apps/shadertoy/