If I have an 8-bit linear filtered texture that I am sampling from a fragment program (running in 24 bit precision), can I get out a color value that is not representable by an 8-bit number?
For example, if I had a texture that had rgb(0,0,232) and rgb(0,0,233) in adjacent texels and I sampled half way between them, could I get rgb(0,0,232.5) out? (I realize that they will be mapped to [0,1], but it’s easier to explain with ints).
Yes. The value will be truncated to framebuffer precision in a stage after the fragment program execution.
Note that the framebuffer may be a floating-point framebuffer if you have set that up; in that case, truncation doesn’t have to happen (although clamping may happen, depending on set-up – usually, you don’t want this to happen when rendering to float).
I’m using the value in a calculation, not to directly apply a color to the screen. I should have asked if the register I assign the lookup to can have > 256 unique values.
I think your answer was yes, the texture linear interpolator operates at > 8 bit precision (even on an 8 bit texture) when using arb_fp, correct?
There are really two things to consider for your particular problem (probably in this order):
texture filtering precision
texture coordinate precision
Everybody does pretty high precision texture coordinates.
Texture filtering precision is probably 8-bit, so filtering between 232 and 233 will be effectively the same as nearest filtering.
On NVIDIA hardware, HILO textures (even HILO8 textures) are filtered at 16-bits, so you get a nice smooth transition between 232 and 233. You’ll have to do the filtering in your shader program to do better than that - until higher precision filtering hardware shows up.
That’s exactly the answer I was looking for. I was getting some artifacts while doing math in the fragment program that weren’t occuring when I did it on the CPU. An 8 bit filter explains it.
I’d like to use the HILO textures, but I’m running out of memory as-is, so my stuff will have to remain a pre-process. Thanks for the suggestion, though
Originally posted by Zeno:
[b]I’d like to use the HILO textures, but I’m running out of memory as-is, so my stuff will have to remain a pre-process. Thanks for the suggestion, though
– Zeno[/b]
HILO8 textures are only 8-bits per component. That shouldn’t cost you more memory than any other 8-bit-per-component texture.