Edge texture filtering

[Aleksandar]

Hi All,

I've recently encountered a very peculiar problem. Edges of the triangles where texture changes (i.e. where level of texture array changes) expose noticeable artifacts. The artifacts are probably due to the texture filtering. I have to emphasize that texels on those edges are far from the border of the texture (level), and also I blend two adjacent levels. With or without blending, artifacts are the same. What could be the solution of the problem?

Images on the following links illustrate the problem (the same (zoomed) scene filled and wireframed).
Artifacts
Wireframe

Thank you in advance for any suggestion!

Aleksandar

[Dark Photon]

Are you sending the exact same vertex coordinates down for that edge in both triangles? Are those stray pixels being filled? Would clear the screen to 100% blue beforehand and then render the scene. Do you see blue holes? Try another clear color -- same result?

Also, are you rasterizing with MSAA?

Are your texture coordinates for that edge in both triangles the same? Do your texture coordinate have continuous derivatives at the edges of triangles? Are you using MIPmap filtering and/or anisotropic texture filtering? What happens if you revert to non-MIPmap filtering (NEAREST or LINEAR)?

[Aleksandar]

Thank you, Dark! Your questions help me to locate the problem, but it cannot be eliminated easily.
Well, let me first answer to all your questions. Maybe the answers could reveal deeper cause of the problem.

• Edges in question are boundary edges of two adjacent triangles of the same triangle strip. So, coordinates should be the same. In fact, the problem repeats on the same triangle strip whenever texture level (of the same texture array) is changed.
• Those pixels are filled (there is no holes, and the color is somehow related to used texture level)
• Yes, I’m using multisampling.
• The texture coordinates are not the same for both sides of the edge (different levels).
• Mipmaping is not used (not applicable in this case). GL_LINEAR is used for the MIN/MAX filtering, and GL_REPEAT for wrap in both direction.
• Anisotropic filtering IS used and THAT causing the problem!

If anisotropic filtering is disabled, the problem disappears. Unfortunately, I cannot permanently disable it, since it is a terrain rendering application, and the impact will be unacceptable.

Have you any suggestion how to solve it?

[Dark Photon]

I guess I'm not understanding something. You say MIPmapping is not used, but then you say that texture "level" varies across the edge. This doesn't make sense to me. Levels are MIPmap levels. If level varied, I could easily see how you might get filtering discontinuities.


In any case, what about the X and Y texture derivatives on either side. First, are they always valid? And are they always the same? Is the content of the texture level(s) they're looking up into the same?

Where you sometimes run into problems is if on the edges of your triangle the texture coordinates aren't defined outside of the triangle, so you end up with invalid (huge) texture derivatives for pixels (or samples) on the edge, which causes MIPmap filtering and/or anisotropic texture filtering to try and sample and filter over a large area of your texture. This can generate seam artifacts like this.

[Aleksandar]

OK, I'm sorry I was not clear. I'll try it better now.

I guess I'm not understanding something. You say MIPmapping is not used, but then you say that texture "level" varies across the edge. This doesn't make sense to me. Levels are MIPmap levels. If level varied, I could easily see how you might get filtering discontinuities.

When I say "level" I mean "texture array layer". I'm sorry for misuse of the word level (in my context those are levels of detail, hence the term).

In any case, what about the X and Y texture derivatives on either side. First, are they always valid? And are they always the same? Is the content of the texture level(s) they're looking up into the same?

I need a bit of coding to confirm values of the derivatives, but they are probably not the same since those are different textures (different layers of the same texture array).

Where you sometimes run into problems is if on the edges of your triangle the texture coordinates aren't defined outside of the triangle, so you end up with invalid (huge) texture derivatives for pixels (or samples) on the edge, which causes MIPmap filtering and/or anisotropic texture filtering to try and sample and filter over a large area of your texture. This can generate seam artifacts like this.

Texture coordinates are defined outside of the triangle (GL_REPEAT is used and always have a "buffer" of at least 10% of the texture size that is not used in the drawing process). The artifacts stays even if less than a half of the layer is used for drawing.

Here are some other examples of the artifacts:
- With anisotropic filtering (img1)
- With anisotropic filtering - artifacts marked (img1a)
- Without anisotropic filtering (img2)
- With anisotropic filtering - another example (img4)

[skynet]

I assume, this is some kind of virtual texturing?
It very much looks like you're using the wrong derivatives for fetching the texels. Mind you, the anisotropic filter is using the length of the derivatives to determine its footprint.
I guess the edges we're seeing are discontinuities in the texture coordinates (jumping from one tile to another?), which cause a sudden increase in the gradient length and therefor a different filtering than surrounding texels. Also, you should provide a large-enough border around your tiles to prevent sampling outside the tile when the filter gets close to the tile border.
More on VT stuff: https://mollyrocket.com/forums/viewf...59fc0160cb180b

[Aleksandar]

Thank you for the reply, skynet!

What I'm trying is some kind of clipmapping. Instead of using derivatives, I thought it was easier to use simple distance (in texture coord space) for choosing right layer. This is a fragment of my FS code (without blending, additional coloring, accessing static texture etc.)

float mL = max(abs(ex_TexCoord.x), abs(ex_TexCoord.y)); // ex_TexCoord – texture coord in ph. space
int layer;
float tSize; // size of a layer
float Dopt = Dmin * 0.9; // Dmin is a minimal size of a layer in physical coordinates
if(mL < (Dopt/2))
{
            layer = MaxLevel;
            tSize = Dmin;
}
else
{
            float fLayer = float(MaxLevel) + 1.0 + log2( mL / Dopt );
            layer = int(ceil(fLayer));
            tSize = Dmin * exp2(layer-MaxLevel);
}
//...
if(layer < (levelCount-1))
{
            vec3 texCoor = vec3(startTex[layer].x + 0.5 + ((ex_TexCoord.x + centTexOffset[layer].x) / tSize), startTex[layer].y + 0.5 + ((ex_TexCoord.y + centTexOffset[layer].y) / tSize), layer);
//...
}

At the first glance this is not a classical virtual texturing, but I have to take a deeper look at the material you have recommended. Thank you very much for the link!

I'll be grateful if you have any comment on the code. startTex[] and centTexOffset[] are offsets in a layer according to toroidal update and texture-mesh mismatch.

[skynet]

Could be that the texture coordinates are not continuous when you switch from one layer to another. When actually sampling the texture, try using explicit gradients (textureGrad()), based on the unmodified ex_TexCoord. You may need to scale the computed gradient to match the 'real' size each layer corresponds to.

textureGrad(clipmap, texCoor, dFdx(ex_TexCoord) * s, dFdy(ex_TexCoord) * s)

whereas 's' is some scale factor, probably something like (1.0/ (2^layer))

[GClements]

What I'm trying is some kind of clipmapping. Instead of using derivatives, I thought it was easier to use simple distance (in texture coord space) for choosing right layer. This is a fragment of my FS code (without blending, additional coloring, accessing static texture etc.)

I note that both layer and tSize are assigned within non-uniform control flow, and derivatives are undefined in this situation. IIUC, enabling anisotropic filtering can cause derivatives to be used even when mipmapping is disabled.

Multisamping presents another complication, as it can cause interpolation to occur other than at the fragment's centroid. Without multi-sampling, the fragment shader won't be run if the centroid is outside the primitive; with multi-sampling, the sample location is forced to lie inside the primitive. If the fragment shader is executed per-sample, the accuracy of derivatives may be impaired.

[Aleksandar]

Thank you again, skynet! And also thanks to Dark Photon (the first assumption on derivatives discontinuity)!

Assumptions are correct. The problem lies in wrong derivatives!

In this picture, coloring is done according to dFdy(ex_TexCoord). A discontinuity is easily seen.
Let's elaborate the problem a bit more. The main problem is in the way graphics card process fragments.
Usually, fragments are processed in small groups (2x2 blocks or so). Doing this GPU easily gets "approximated derivatives" by subtracting values of the adjacent fragments. On the edges of different texture layers (in my example), texture coordinates are totally different. That causes extreme derivatives if fragments from the same block belong to different textures. Artifacts are inevitable.

Using textureGrad() GLSL function for accessing texture with explicit partial derivatives can really help. Thank you again, skynet! But effect of the anisotropic filtering is spoiled. Take a look at the following pictures.
- With anisotropic
- With anisotropic + textureGrad(clipmap, texCoor, dFdx(ex_TexCoord), dFdy(ex_TexCoord))
- With anisotropic + textureGrad(clipmap, texCoor, dFdx(ex_TexCoord)*layer, dFdy(ex_TexCoord)*layer)
- Without anisotropic

Obviously, anisotropic filtering without textureGrad() fetching achieves best results. Calculating derivatives according to ex_TexCoord gives worse results (but better than without anisotropic filtering). Aliasing effects are quite apparent in this case. By multiplying derivatives with layer ID (higher number means wider spatial extent) gives much better results. Textures are much smoother, aliasing effects appear for much smaller viewing angles, and the scene appears more like using mipmaping than anisotropic filtering (too blurry). In any case, the standard anisotropic filtering with only factor 2 gives much better results than anything I've tried so far with textureGrad(). Probably partial derivatives dependency is more complex than we assumed at the start. I'll try to play with different coefficient, but any further suggestion would be more than welcome.