I was wondering if there is a way to do color keying in opengl
This can be an advanced question.
The short answer is that OpenGL uses a separate alpha channel for transparency. There is no automatic way to set this based on rgb colors at least in the classic OpenGL pipeline. Application software or other tools must compute alpha based on the RGB (and other cool information common to keying).
However with pixel shaders a lot more is possible w.r.t. chroma keying and you could do a lot more. Heck if you fed destination in as a texture you could probably even do triangulation in hardware bearing in mind you can do multiple taps with enough texture units.
Here’s a trick:
Set the color matrix to:
[ 1 0 0 0 ]
[ 0 1 0 0 ]
[ 0 0 1 0 ]
[ -.5 -.5 0 1 ]
…then upload your textures with an internal format of GL_RGBA, and when rendering set your alpha function to GL_GREATER, reference value 0.0.
Purple becomes transparent!
(Yes, this isn’t really chroma key, just a hacky simulation, I realize that, but hey hey, it’s not mainstream!)