Cubemap aliasing with mipmaps

[Seth Hoffert]

I've been noticing annoying aliasing at extreme angles when using (HDR) cubemaps for reflections. I'm using shaders.

In this screenshot, which depicts a portion of a torus, I am using an HDR cubemap with GL_CLAMP_TO_EDGE, and linear filtering with mipmapping. As I pivot the torus, the outlined portion of the screenshot sparkles/aliases/twinkles/whatever.

http://cse.unl.edu/~shoffert/aliased1b.png

In this next image, I've set the clamp mode to GL_CLAMP_TO_BORDER with a red border color. Apparently this nasty aliasing is coincident with the boundaries of each cubemap face.

http://cse.unl.edu/~shoffert/aliased2b.png

Finally, this image demonstrates the torus with no mipmapping at all. The twinkling is gone when I move it, but of course when I zoom out it looks sparkly all over.

http://cse.unl.edu/~shoffert/aliased3b.png

Anyone have any idea how to fix this? I'd really like to be able to use mipmaps for this... perhaps I need to tweak the LOD value in the lookup when it samples around the edges of the cubemap faces?

Thanks.

[ZbuffeR]

It might be solved by carefully building the cubemap mipmaps :
http://ati.amd.com/developer/cubemapgen/index.html
http://ati.amd.com/developer/gdc/GDC2005_CubeMapGen.pdf

[Seth Hoffert]

Thanks for the quick reply. I think you may be right; I was hoping that the mipmap generation performed by glGenerateMipmapEXT would do this but I can understand why it cannot, if this is the case.

[Seth Hoffert]

I've tried processing the cubemap with the utility mentioned by ZbuffeR, but now it appears even more pronounced - I'll try tweaking the params a bit, maybe I'm doing it wrong. This is what I've obtained:

http://cse.unl.edu/~shoffert/aliased4b.png

EDIT: I might be onto something... the lit portions near the offending area have values in excess of 8000.0, while the area around the lights are much less... although I am clamping in the shader, the extreme values propagated into the mipmaps when I generated them since they were unclamped at that time. I'm going to try clamping the values *before* I generate the mipmaps, in the original cubemap.

EDIT2: This helped a lot.

[Seth Hoffert]

For the life of me, I can't get this looking DECENT:

http://cse.unl.edu/~shoffert/highfreq2.png

I've tried all sorts of preprocessing on the cubemaps, and I still see these sparkles at edges.

[vs987]

on edges normals are changing a lot across pixels,
and so will do the reflection vector,
which results in a high derivative in your cubemap lookup,
and means a very lowlevel mipmap selection

try check if mipmapping really works
put different solid colors in ur mipmapchain (ie red in lvl0,green in lvl1,blue in lvl2 ....),
to see from which mipmap that sparkles are sampled from

be aware, that the inbuilt mipmap selection is based on the screen-space derivative of the (dependent) texture coordinates,
which is only computed at 2x2 pixels quads and not per-pixel.
maybe these sparkles are a result of this quarter-resolution mipmap selection, because your derivatives change too much per pixel

if this is all so, i would do like ur first idea
"perhaps I need to tweak the LOD value in the lookup when it samples around the edges of the cubemap faces?"
try to identify the problematic fragments in your shader and bias the LOD accordingly (means increasing), so you can at least avoid oversampling, but it may look blurry due to the undersampling

[Seth Hoffert]

Thanks vs987, I'll give it a try. :)