Linear texture interpolation in ARB_fp

[Zeno]

If I have an 8-bit linear filtered texture that I am sampling from a fragment program (running in 24 bit precision), can I get out a color value that is not representable by an 8-bit number?

For example, if I had a texture that had rgb(0,0,232) and rgb(0,0,233) in adjacent texels and I sampled half way between them, could I get rgb(0,0,232.5) out? (I realize that they will be mapped to [0,1], but it's easier to explain with ints).

Thanks,
Zeno

[jwatte]

Yes. The value will be truncated to framebuffer precision in a stage after the fragment program execution.

Note that the framebuffer may be a floating-point framebuffer if you have set that up; in that case, truncation doesn't have to happen (although clamping may happen, depending on set-up -- usually, you don't want this to happen when rendering to float).

[Zeno]

I'm using the value in a calculation, not to directly apply a color to the screen. I should have asked if the register I assign the lookup to can have > 256 unique values.

I think your answer was yes, the texture linear interpolator operates at > 8 bit precision (even on an 8 bit texture) when using arb_fp, correct?

[cass]

There are really two things to consider for your particular problem (probably in this order):

- texture filtering precision
- texture coordinate precision

Everybody does pretty high precision texture coordinates.

Texture filtering precision is probably 8-bit, so filtering between 232 and 233 will be effectively the same as nearest filtering.

On NVIDIA hardware, HILO textures (even HILO8 textures) are filtered at 16-bits, so you get a nice smooth transition between 232 and 233. You'll have to do the filtering in your shader program to do better than that - until higher precision filtering hardware shows up.

Hope this helps...
Thanks -
Cass

[Zeno]

That's exactly the answer I was looking for. I was getting some artifacts while doing math in the fragment program that weren't occuring when I did it on the CPU. An 8 bit filter explains it.

I'd like to use the HILO textures, but I'm running out of memory as-is, so my stuff will have to remain a pre-process. Thanks for the suggestion, though

-- Zeno

[cass]

Originally posted by Zeno:
I'd like to use the HILO textures, but I'm running out of memory as-is, so my stuff will have to remain a pre-process. Thanks for the suggestion, though

-- Zeno

HILO8 textures are only 8-bits per component. That shouldn't cost you more memory than any other 8-bit-per-component texture.

Glad that helped...

Cass