Off-screen and On-Screen frame buf rendering

Hello All,
This post is related to previous post on video frame freezing.Previous post was too long so I am writing a fresh post.

Situation: I have an AVI file. I am playing it using opengl under MFC. Now I want to capture current frame and apply shader program to operate on.

I used FBO and my technique is to render current frame data to a texture , copy it for shader to act on.

In my shader code, I am using gl_DrawBuffers(n, buf), where n = 4.

and the shader program(frag shader) writes back using gl_FragData[0] and glgl_FragData[1] — i.e it is writing back to two textures.

My FBO has 4 attachments. where attachment no. 1 and 2 (COLOR_ATTACHMENT_EXT0 and COLOR_ATTACHMENT_EXT1) are meant for the shader program to write on.

COLOR_ATTACHMENT_EXT2 is extra …useful for future and COLOR_ATTACHMENT_EXT3 is associated to texture where my video frame content falls on .

Now, After processing by the shader, I am switching back to main frame buffer by supplying argument 0 in glBindFrameBuffer().

The result is that sometimes the next frame appears on a yellow background and sometimes vanishes.

Q1. From your experience do you know the problem ?

Q2. Is there a better way where I can see my video frame uninterrupted and my shader code simultaneously act on every frame using FBO.

I had thought of attaching a Render Buffer Object so that the current scence contents fall on it and the this RBO may attach to FBO but I do not have any example to follow.

Can you help me?

Textures attached to FBO is write only. So… pseudo code for your app should be:


// initialistion
bool init()
{
 gen FBO
 bind FBO
 gen 4 textures and attach tem to FBO
 bind FBO 0

 gen shader
 initialise shader

 gen texture for video
 initialise texture

 return true;
}

// in frame
bool doFrame()
{
  update video texture
  bind video texture

  // switch to  offscreen rendering
  bind fbo, adjust matrices and viewport
  set draw buffers
  bind shader

  render quad.. result is in all 4 FBO attachments 
  bind shader 0
  bind fbo 0
  restore matrices and viewport

  // now you are back on backbuffer 
  bind texture fbo.attached_texture[0..3]
  render quad
  now you have one of fbo textures in backbufffer
}

Thanks a tonne yooyo, my view closely matched yours. I am not clear about the last two line.

i.e after switching back to main frame buffer by binding
glBindFrameBuffer(GL_FRAMEBUFFER, 0);

Q1. you wrote “now you are back on backbuffer” . Does “backbuffer” mean GL_BACK?

Q2. Moreover, after binding video frame texture to fbo , in render quad, do i need to supply tex coordinates ?

something like

drawQuad()
{
glTexture2f(x,y);
glVertex2f(0, 0) ;

} ??

Rest of the steps are ok to me.

Q3. What if I draw my video frame to RenderBuffer Object instead of TexBuf Obj?

Will that make difference?

Please comment. Thank you for your time.

A1. yes… backbuffer means GL_BACK
A2. Yes… you need to supply texture coordinates for each vertex

If you dont provide texture coordinates GL (as state machine) will use last known texture coordinate in all upcoming draw calls (if texturing is switched on)

One more thing… you need to understand difference between binding texture and attaching texture.
When you bind texture this mean you will use that texture in upcoming opetarions… as soruce (in case of render) or as destination in case of uploading teture data.

When you attach texture to fbo it means you are going to use that texture as destination for upcoming rendering operation.

You cant use same texture as sourca and destination in rendering operation. This mean… if texture is attached to fbo and fbo is binded then you cant bind same texture. And vice versa… if you bind texture and try to bind fbo who have same texture attached to it, then it will fail.

I tried but again it made very little difference. I exaplan as below with code steps

Here is my pesudocode, I can not talk ,much of shader code as it is immaterial here.

MFC Part –


OnPaint()
{
    // get screen context and set its properties
    //This is similar to glutInit() where viewport settings are done
      Get_and_Set_Context();

      //Copy frame data to temp object
      memcpy(temp, FrameData, sizeof(FrameData) );

    // Bind current context to FBO id "fb"
     glBindFrameBuffer(GL_FRAMEBUFFER , fb);

    //Call function that internally calls shader code
      MyFunction(); //I will expand this below

    //Switch Back to Main Frame Buffer
    glBindFrameBuffer(GL_FRAMEBUFFER , fb);


    //restore view port settings
    SetViewPort(Original_W, Original_H);

  
   //set up a textuer object's propoerteis and fill it with orignal frame data

   glActiveTexture(GL_TEXTURE0);
   glBindTexture(GL_TEXTURE_RECTANGLE_ARB, tex[3]); //where tex[4] are 4 texture ids created before hand
   
//set texture parameters
glTexParameteri(GL_TEXTURE_RECTANGLE_ARB, GL_TEXTURE_MIN_FILTER, GL_NEAREST);
glTexParameteri(GL_TEXTURE_RECTANGLE_ARB, GL_TEXTURE_MAG_FILTER, GL_NEAREST);
glTexParameteri(GL_TEXTURE_RECTANGLE_ARB, GL_TEXTURE_WRAP_S, GL_CLAMP);
glTexParameteri(GL_TEXTURE_RECTANGLE_ARB, GL_TEXTURE_WRAP_T, GL_CLAMP);
 
//copy back original frame data
memcpy( FrameData, temp, sizeof(FrameData) );

//delete temp
free(temp);
temp = null;

//attach FrameData to tex[3]
glTexImage2D(GL_TEXTURE_RECTANGLE_ARB, 0 , GL_RGB , FRameWidth , FrameHeight , 0 , GL_RGB, GL_UNSIGNED_BYTE, FrameData);

//draw to GL_FRONT buffer of the main frame buffer
glDrawBuffer(GL_FRONT); [I tried with GL_BLACK but again it waqs giving me twisted data with multiple copies on a yellow back ground. I tried with GL_FRONT and it gave me correct image.]

//Attach currently binded Frame data texture to FBO
glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0_EXT, GL_TEXTURE_RECTANGLE_ARB, tex[3],  0);

//drawQuad
drawQuad(FrameWidth, FrameHeight);  [Expanded later]

//swaps buffer .. similar to glutSwapBuffers()
SwapBuffers(m_hDC->GetSafeHdc()); 

//current REndering context is set to null ... as a good tip from MSDN
wglMakeCurrent(NULL, NULL );

//validation for error ..again tip taken from MSDN
ValidateRect(NULL); 
}//end of OnPaint()

The drawQuad Function


drawQuad(W,H)
{
glBegin(GL_QUADS);
glMultiTexCoord2f(GL_TEXTURE0, 0.0f , 0.0f);
glVertex2f( 0.0f, 0.0f);
glMultiTexCoord2f(GL_TEXTURE0, WD , 0.0f);
glVertex2f(WD , 0.0f);
glMultiTexCoord2f(GL_TEXTURE0, WD , HT);
glVertex2f( WD , HT );
glMultiTexCoord2f(GL_TEXTURE0, 0.0f , HT);
glVertex2f( 0.0f, HT );
glEnd();
}

and now the function that internally calls shader program


MyFunction()
{
  //Take three buffers that would be used as textures.
  float *A, *B, *C; //assume they are filled with some data, not necessarily with frame data...say for example non color data but corresponding to current frame under reference.

//Textures A,B and C have same format and dimension but unequal to that of current frame.

//Change viewport settings
SetViewPort(w,h); //specific to texsize

//set the properties of tex[0] ,tex[1] and tex[2] and associate them to A,B and C respectively.

SetTexture(tex[0],A);
SetTexture(tex[1],B);
SetTexture(tex[2],C);

//call shader
SetShader(); //does calculations and delivers results to first two out of four buffers.

//where buffers[4] has 4 color attachment points enabled

//read back data to A and B from shader
ReadBack(A, Buf[0]);
ReadBack(B, Buf[1]);


//copy A and B to some other temp variables accessible to other functions. and free A,B,C
}

The OnPaint() function is called for every frame hence you can assume that all three functions are called iteratively.

Observations:
For first frame everything works well. But for frame number 2 and onwards, the code crashes just after switching back to off-screen buffer from on-screen in OnPaint().

  1. SetShader uses glDrawBuffers(2, buffers) to write 2 streams of results on two of the 4 attached textures to FBO.

Help appreciated for above.Thanks.

This is wrong… and I cant understand whats going on.

  1. this is incorrect:
    //Switch Back to Main Frame Buffer
    glBindFrameBuffer(GL_FRAMEBUFFER , fb); 

it should be

    //Switch Back to Main Frame Buffer
    glBindFrameBuffer(GL_FRAMEBUFFER , 0); 
  1. You are uploading frameddata to tex[3], then attach that texture to which fbo? If you apply above fixes then this code

//Attach currently binded Frame data texture to FBO
glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0_EXT, GL_TEXTURE_RECTANGLE_ARB, tex[3],  0);

should fail.

Thank you for your time yooyo. :slight_smile:

Sorry that was my mistake in writing previous post. I mixed the two. But for now I am successful.

Though the method I implemented by writing to texture and drawing , is crude but I shall improve on it later.
For now, my ultimate aim is to see that shader code computes and it does not disturb viewing continuous frames.Unfortunately, this has not been met still.

As before, I am rendering current frame to a texture attached to off-screen FBO and then rendering it. [This is working now.The quality has suffered a bit in terms of brightness but still visible enough to deem it correct].

If previous post is referred to containing the code, the MyFucntion() calls setShader() that computes on second and third texture attached to the same FBO to which the first texture, containing the current frame, is attached.

To elaborate and make it clear, here is the picture:


OnPaint()
{
   //get current frame content
   getFrameData(&FrameData);

   //bind to off-screen frame buffer
   glBindFrameBuffer2D(GL_FRAMEBUFFER , fb);  //fb = 1
  
   //bind data to tex[0]
   ...

   //attach the above texture to fb
   ...

   //call MyFunction that internally calls SetShader()


   //restore view port settings

   //drawQuad()
..
//.. remaining OnPaint() , 
//swaps buffer .. similar to glutSwapBuffers()
SwapBuffers(m_hDC->GetSafeHdc()); 

//current REndering context is set to null ... as a good tip from MSDN
wglMakeCurrent(NULL, NULL );

//validation for error ..again tip taken from MSDN
ValidateRect(NULL);
}

MyFunction()


  //set second and third texture properties to be attached
  //to COLOR_ATTACHMENT_EXT1 and COLOR_ATTACHMENT_EXT2 resp.


//done so that previous settings are not affected  
glPushAttrib(GL_VIEWPORT_BIT | GL_COLOR_BUFFER_BIT);

  //change view port settings , for shader computation purpose
  setViewPort(new_width,new_height);

  //call shader
  SetShader(); // Bone of contention. I'll explain below.

  glPopAttrib();

 //free tex data
..
}

Inside the SetShader(), I am taking two streams and operating on them. Then the shader writes back results to

gl_FragData[0] & gl_FragData[1] resp.

Observation:
As soon as I activate SetShader() in MyFunction(), the screen at second frame turns yellow.

But if I comment out SetShader() in MyFunction(). I am able to view correct sequence of frames.

I guessed that since gl_FragData[0] and gl_FragData[1] are being used, then they may be whitewashing the original frame data in the FBO.

Then I changed the sequence to


gl_FragData[1] and gl_FragData[2], after attaching tex[0] to COLOR_ATTACHMENT_EXT1 and COLOR_ATTACHMENT_EXT2 respectively.

from


gl_FragData[0] and gl_FragData[1]

.

Observation:

No change, when SetShader() is activated, screen turns yellow.

Questions:

Q1. Is my guess right?

Q2. Do you suggest a solution for Q1.?

Again thank you for your time.

I see you are filling frame data to texture then attach that texture to fbo as render target. Invoking any rendering code on that fbo will overwrite original texture data. Im not sure is this what you want or not.

My suggestion is to create frame texture and fbo with additional textures. Then activate fbo, bind (not attach) frame texture and draw. Shader should read samples from frame texture, apply some math and writes to one or more gl_FragData (which is fbo attachments).

After this you have original frame texture intact and result of processing in fbo textures. Choose whatever you want to display on main backbuffer (render texture mapped quad) and swap buffers.

I see you are filling frame data to texture then attach that texture to fbo as render target. Invoking any rendering code on that fbo will overwrite original texture data. Im not sure is this what you want or not.

Exactly.That is what I am trying to do.

But now the problem still remains.Though I 100% agree on your views and that is what I expected.

But this is pointless… why do you fill texture data and then overwrite? Its something like…


int a;
a = some_heavy_calculation(); // uploading
a = another_calculation(); // fbo + shader

You should upload frame to texture, and use that texture in shader, and shader output should be written to another texture via fbo. Or…


int a, b;
a = some_heavy_calculation();
b = another_calculation(a);
// now you have regular frame (a) and filtered frame (b)

Sorry for the confusion, actually I am storing current frame to a texture that is bound to GL_TEXTURE0. This texture is not attached to FBO.

Then I am taking an FBO, binding to it glBindFrameBuffer(… , fb)
then taking 3 other textures ( allocating and initialized them with the proposed data to be acted upon by the shader program. )of equal sizes , format and target type.

After this I am attaching the last 3 textures to FBO and applying shader under MyFunction().

Now Inside MyFunction() the shader() if not called displays every frame correctly.

But when it is invoked, then the display changes and gives strange results such as pure yellow, sometimes green or black.

The shader program is using glDrawBuffers(3 , buffers), just before drawing quad to initiate shader execution.

Inside shader I am writing data to gl_FragData[0] and gl_FragData[1].

I suspect that gl_FragData() are playing spoilsport.

I am wondering why this should effect the display screen since I am restoring viewport settings to original and drawing quad with current frame texture data.

The agenda of rendering to unattached first texture tex[0], the frame is to temporarily store data, copy from it and then display it again using drawQuad() when on-screen frame buffer is re-activated.

I am doing all shader manipulation once the off-screen frame buffer is turned on using glBindFrameBuffer(… , fb)

MyFunction reads as


 glBindFrameBuffer(.. , fb)

glActiveTexture(GL_TEXTURE1); 
glBindTexture( GL_RECT.. , tex[1]);

//set pramertss
..

glTexImage2D(...);
glFrameBufferText(); //attach to FBO
glTexSubImage2D(..); //tranfer data to tex[1]

[above steps repeated for tex[2] and tex[3] ]

SetShader();

//retrieve data from buffers[0] and buffers[1] {equivalent to GL_COLOR_ATTACHMENT0 etc.}


//unbound from tex[1] , tex[2] and tex[3]
}


The calling function of MyFunction() switchs back to standard frame buffer.

//In parent function i.e OnPaint()
//drawQuad()

Thanks so much for your patience. I hope I make sense now.

Solved. My earlier approach was right but some how the RC were not getting changed. Than you yooyo for your time. :slight_smile:

This topic was automatically closed 183 days after the last reply. New replies are no longer allowed.