View Full Version : Floating point accuracy in GLSL

darkforest

11-22-2010, 04:23 PM

I am working with uint8 and uint16 textures in GLSL ES.

What really happens when

float X = texture2D(TestTex, vec2(0.0,0.0) ).x;is called?

Let's say uint8 34 is stored at texel (0,0). Am I correct the above is equivalent to:

float X = float(34)/float(255);

But, the cast from uint8 to float introduces error, and so does the division.

So if I want to get 34 back exactly in my shader, can I trust either of

int X1 = round(X*255.0);

int X2 = int(X*255.0);

?

And what's this thing about GLSL being IEEE754 like? How different is it in a nutshell?

Thanks

aqnuep

11-22-2010, 04:48 PM

If you are using integer texture format, not a fixed point normalized texture format, you can directly do the appropriate integer texture fetches using integer samplers (usampler* or isampler*, the former could come handy in your case) as that way you get the actual integer texel values without the need of ALU instructions to scale it back up.

If you do use normalized texture formats (maybe because you are using a GPU generation that does not support integer texture formats), then your idea may work as well, however AFAIK there is no absolute guarantee about precision for pre-GL3 hardware.

david_f_knight

11-22-2010, 08:51 PM

But, the cast from uint8 to float introduces errorNo error is introduced casting ints or uints to floats unless an int's or uint's magnitude exceeds the maximum value representable with the number of bits in the float's mantissa. No uint8 can ever do that, even for a float16.

thekend

11-23-2010, 02:46 AM

If you are using integer texture format, not a fixed point normalized texture format, you can directly do the appropriate integer texture fetches using integer samplers (usampler* or isampler*, the former could come handy in your case) as that way you get the actual integer texel values without the need of ALU instructions to scale it back up.

For GLSL ES there are no integer samplers AFAIK. Is it possible to use them via any extensions for GL3 capable hadrware?

aqnuep

11-23-2010, 06:19 AM

Ah, sorry, didn't realize you look a solution for ES.

But yes, it is possible to use integer textures on GL3 capable hardware with GL_EXT_texture_integer that is already a core extension.

darkforest

11-23-2010, 06:51 AM

But, the cast from uint8 to float introduces errorNo error is introduced casting ints or uints to floats unless an int's or uint's magnitude exceeds the maximum value representable with the number of bits in the float's mantissa. No uint8 can ever do that, even for a float16.

How does this work?

A way to find the mantissa and exponent from an int is:

log2(34) ~= 5.0875

exponent = ceil(5.0875) = 6

mantissa = 34 / ( 2^(exponent) ) ~= 0.5313If I map back I get

0.5313 * 2^6 = 34.0032

Does IEEE754 use other method much more accurate or exactly accurate?

Also

http://wiki.flightgear.org/index.php/Howto:_Shader_Programming_in_FlightGear

says GLSL float is 'IEEE754 like'. What does that mean?

Thanks

david_f_knight

11-23-2010, 10:51 AM

How does this work?

A way to find the mantissa and exponent from an int is:

log2(34) ~= 5.0875

exponent = ceil(5.0875) = 6

mantissa = 34 / ( 2^(exponent) ) ~= 0.5313If I map back I get

0.5313 * 2^6 = 34.0032

Does IEEE754 use other method much more accurate or exactly accurate?

Yes, IEEE 754 uses another, exactly accurate (and faster and simpler) method. In most cases, IEEE 754 floating point numbers are encoded in normalized form. The log2 value of a number is never calculated, making the normalized encoding faster, simpler, and exact (as long as the magnitude of the number fits within the mantissa's bits).

IEEE 754 normalization means that the binary point position of a value is shifted one bit (and the exponent incremented or decremented by one) until the absolute value of the value is one or between one and two (except for the special values of zero, infinity, NaN, etc.). The exponent indicates how many positions the binary point was moved and the mantissa is not changed (except that the leading bit, which after normalization will be one, is removed to allow an extra bit of precision in the least-significant bit of the mantissa).

So, in your example, the binary value of 34 is 100010. Normalized, that is represented as 1.00010 with an exponent of five (added to a bias to allow both positive and negative exponents). After the leading one is removed the mantissa will be .00010 with however many trailing zeroes as are required to fill the remaining less significant bits of the mantissa for the particular floating point size (16, 32, 64, 80 bits).

Reconstructing the value, you concatenate the implicit leading one to the mantissa, for 1.00010, then shift the binary point five bits to the right from its position to the right of the leading one, for a value of 100010.0, which equals 34 exactly.

This can be implemented in hardware very efficiently and is very fast.

Also

http://wiki.flightgear.org/index.php/Howto:_Shader_Programming_in_FlightGear

says GLSL float is 'IEEE754 like'. What does that mean?

Well, the GLSL 4.0 spec says this about floats (in section 4.1.4):

As an input value to one of the processing units, a single-precision or double-precision floating-point variable is expected to match the corresponding IEEE 754 floating-point definition for precision and dynamic range. Floating-point variables within a shader are also encoded according to the IEEE 754 specification for single-precision floating-point values. However, it is not required that the precision of internal processing match the IEEE 754 floating-point specification for floating-point operations, but the guidelines for precision established by the OpenGL Graphics System Specification must be met. Similarly, treatment of conditions such as divide by 0 may lead to an unspecified result, but in no case should such a condition lead to the interruption or termination of processing. Generally, there are no signaling NaNs, and operating on NaNs (Not a Number) or infs (positive or negative infinities) gives undefined results.

So, to answer your question, IEEE 754-like in GLSL pretty much means that the special encoding and processing for the IEEE 754 special values NaN, plus infinity, and minus infinity isn't required.

Powered by vBulletin® Version 4.2.3 Copyright © 2018 vBulletin Solutions, Inc. All rights reserved.