YUV to RGB conversion using text_fragment_shader

OK, I have an application that decodes YUV video
that is textured in GL (on a mac), I would like to
use ATI_text_fragment_shader for video processing
but I need to do a YUV -> RGB color conversion
using a shader first (stupid ATI). Here is what I have come up
with so far (currently goes from VUY -> BGR):

!!ATIfs1.0

StartConstants;
CONSTANT c0 = {1.16400, 1.59600, 2.01800, 0.81300};
CONSTANT c1 = {1.08175, 0.39100, 0.87075, 0.28375};
EndConstants;

StartPrelimPass;
SampleMap r1, t0.str;

MUL r1.r, c0.r, r1.r; # 1.16400 * Y
MUL r1.g, c0.b, r1.g; # 2.01800 * U
MUL r1.b, c0.a, r1.b; # 0.81300 * V
MUL r2.g, c1.g, r1.b; # 0.39100 * U
MUL r2.b, c0.g, r1.b; # 1.59600 * V

ADD r2.r, r1.r, r1.g; # 1.16400 * Y + 2.01800 * U
SUB r2.r, r2.r, c1.r; # 1.16400 * Y + 2.01800 * U - 1.08175
MOV r0.b, r2.r;
EndPass;

StartOutputPass;
PassTexCoord r1, r1.str;
PassTexCoord r2, r2.str;

ADD r2.r, r1.r, r1.b; # 1.16400 * Y + 0.81300 * V
SUB r2.r, r2.r, r2.g; # 1.16400 * Y + 0.81300 * V - 0.39100 * U
ADD r2.r, r2.r, c1.a; # 1.16400 * Y + 0.81300 * V - 0.39100 * U + 0.28375
MOV r0.g, r2.r;

ADD r2.r, r1.r, r1.b; # 1.16400 * Y + 1.59600 * V
SUB r2.r, r2.r, c1.b; # 1.16400 * Y + 1.59600 * V - 0.87075
MOV r0.b, r2.r;
EndPass;

Heh, the problem is that this does not currently work (everything is green), so any help at all (a working
shader anyone ???) is greatly appreciated…

Constants in ATI_fragment_shader are only allowed to be in the range [0, 1], so this
CONSTANT c0 = {1.16400, 1.59600, 2.01800, 0.81300};
CONSTANT c1 = {1.08175, 0.39100, 0.87075, 0.28375};
is not legal and the values above 1 will most likely be clamped to 1. This might be an explanation for the effects you’re seeing.
You’ll have to use tricks to get around this limitation. For example, half the values in the constants and then scale them up by 2 again in the fragment shader.
Other than that, since a YUV->RGB conversion is nothing more than a 3x3 matrix multiply you should basically be able to do it with three DOT3 instructions.


Other than that, since a YUV->RGB conversion is nothing more than a 3x3 matrix multiply you should basically be able to do it with three DOT3 instructions.

Heh, that make sense (stupid me). I have come up with this:

!!ATIfs1.0

StartConstants;

CONSTANT c0 = {0.299, 0.587, 0.114};
CONSTANT c1 = {0.596, 0.275, 0.321};
CONSTANT c2 = {0.212, 0.523, 0.311};

EndConstants;

StartPrelimPass;

MOV r2.r, c1.r;
MOV r2.g, c1.g.neg;
MOV r2.b, c1.b.neg;

MOV r3.r, c2.r.neg;
MOV r3.g, c2.g.neg;
MOV r3.b, c2.b;

EndPass;

StartOutputPass;

SampleMap r1, t0.str;

PassTexCoord r2, r2.str;
PassTexCoord r3, r3.str;

DOT3 r0.r, r1, c0;
DOT3 r0.g, r1, r2;
DOT3 r0.b, r1, r3;

EndPass;

But I think that my matrix is way off because everything is a dark red… my current matirx
is:

0.299 0.587 0.114
0.596 -0.275 -0.321
-0.212 -0.523 0.311

Does anyone have another (better) color
conversion matrix ?

[This message has been edited by feelgood (edited 09-09-2003).]

If you are willing to use the imaging subset and do one CPU lookup transformation per component, you can easily do 32-bit YCrCb to RGB conversion in hardware without using pixel shaders, or at least without having to write shader code.

First construct luminance and chromiance lookup tables which take care of subtracting the bias of 16 from the input components, pre-clamping to [0,219] and [[0,224], and then scaling the result to [0,255].

I didn’t see any previous mention of clamping, but in the case of CCIR-601 YCrCb video data, the valid luminance range is 16-235 and the valid chromiance range is 16-240. On Nuon DVD players, the video data is often algorithmically converted from RGB to the native YCrCb frame buffer format. In this case, you must pre-clamp or else you almost certainly will experience underflow and or overflow for some pixels.

The lookup table entries also expand the range from [16,235] and [16,240] to [0,1]. OpenGL is going to want to operate on [0,1] component values, and the expansion can be done at the same time as clamping, so its easier just to do it there than try to hack OpenGL to work around it.

After transforming the components, the color vector is simply multiplied by the 3x3 color matrix containing the standard CCIR-601 conversion coefficients. After multiplication, the color components are post-biased to take care of the constant portion of the calculation.

The CCIR-601 equations subtract the bias of 16 from the compoents and then normalize them. Unlike luminance, which is normalized to [0,1], chromiance is normalized to
[-1/2,1/2]. I simply factored this into the post-bias values.

int8 LuminanceTable[256];
int8 ChromianceTable[256];

void CalculateTableEntries(uint8 *table, uint8 min, uint8 max)
{
uint8 clampedVal;

for(uint32 i = 0; i < 256; i++)
{
clampedVal = i;

if(clampedVal < min)
{
  clampedVal = min;
}
else if(clampedVal > max)
{
  clampedVal = max;
}

table[i] = (unsigned __int8)(((double)(clampedVal - min)) * (255.0/(max-min)));

}
}
GLfloat ycrcb2rgbColorMatrix = {
1.000, 1.000, 1.000, 0.000,
1.402, -0.700, 0.000, 0.000,
0.000, -0.340, 1.772, 0.000,
0.000, 0.000, 0.000, 1.000};

int8 LuminanceTable[256];
int8 ChromianceTable[256];

CalculateTableEntries((uint8 *)LuminanceTable,16,235);
CalculateTableEntries((uint8 *)ChromianceTable,16,240);

glMatrixMode(GL_COLOR);
glLoadMatrixf(ycrcb2rgbColorMatrix);

glPixelTransferf(GL_POST_COLOR_MATRIX_RED_BIAS,-1.402/2.0);
glPixelTransferf(GL_POST_COLOR_MATRIX_GREEN_BIAS,(0.70 + 0.34)/2.0);
glPixelTransferf(GL_POST_COLOR_MATRIX_BLUE_BIAS,-1.772/2.0);

for each 32-bit pixel in YCrCb format,
pixel[0] = LuminanceLookup[pixel[0]];
pixel[1] = ChromianceLookup[pixel[1]];
pixel[2] = ChromianceLookup[pixel[2]];
end for

render ycrcb texture

This is sufficient for rendering to screen. I can only assume that the color matrix method would also work for rendering to a texture/pbuffer.

The color matrix is accelerated on almost all nvidia cards according to their extensions table, and I know its supported on at least the recent ATI cards (9500 and above).

[This message has been edited by Riff (edited 09-10-2003).]