Skipping and Duplicating Frames

I wrote some code for use in my research which has worked well, however my timing constraints (EEG-based Brain-Computer Interfaces) are becoming more time critical, and I’ve discovered timing issues which I am posting here for help.

My application requires precise presentation of items on my monitor sync’ed with the refresh rate of 60fps. My problem might be one that people aren’t aware of because I doubt many would test with photo-detectors on the screen as I have done to see what is REALLY happening.

I’ve stripped away the application specific things, and provided a bare-bones test that anyone can reproduce. The effect happens on a variety of video cards and monitors, so it doesn’t seem to be vendor specific. However, I am using Linux (Ubuntu). Most people won’t have photo detectors and digital recording equipment, but I can test your code and post the results if you think you have a solution.

The test presents four different images repeatedly in sequence. Each image consists of two adjacent white or black squares. The frames are frame 0 through frame 3. In frame 0, both squares are black. In frame 1, only the right square is white. In frame two, the left square is white instead. In frame 3, they are both white. Each frame appears for 1/60 of a second and frames 0 to 3 are sequenced in order forever. The images are synchronized to the vertical sync of the video card/monitor.

Each square has a photo detector placed over it sampled, digitized, and recorded at 10,000 sps. By examining this recording, I can see what is really happening. What should happen is the outputs of the detectors may be viewed as a two-bit binary number progressing through 00, 01, 10, and 11 representing frame0, through frame3 respectively. At a sampling rate of 10,000 Hz, each frame is sampled almost 167 times by the recording device thereby capturing a clear record of what happened.

Rather than repeat all 167 samples for each 1/60 of a second, I will show only one number for each frame below to describe what SHOULD happen versus what ACTUALLY happens. We should see:

012301230123 …

Unfortunately, instead I see:

130012011201 …

So I miss frames, and duplicate frames with no pattern. I have the code below. If anyone has any suggestions about what to do to try to “fix” this, I can take your edits, and run them to show what the recorder generates for your solution. Please help! (And yes, if I slow this down to say one frame change per second by adding a one second sleep in the main loop, then I do faithfully get 60 0’s followed by 60 1’s followed by 60 2’s and then 60 3’s.)

At one point I put in some debugging code in my main loop to toggle a bit on an I/O port which I monitored both with a digital recorder as well as an oscilloscope to confirm that my code is indeed looping once every 1/60th of a second faithfully.


#include <iostream>
#include <cstdlib>
#include <GL/glut.h>
#include <GL/glx.h>
using namespace std;

#define SCREENWIDTH 800
#define SCREENHEIGHT 600

int hdc;
void display(void);
void keyboard(unsigned char key, int x, int y);
void configureWindow();
void configureGraphics();

static PFNGLXSWAPINTERVALSGIPROC swap_interval;

int num = 0;

int main(int argc, char **argv)
{
    glutInit(&argc, argv);
    configureWindow();
    configureGraphics();
    glutMainLoop();

    return 0;
}

void configureWindow()
{
    glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE);
    glutInitWindowSize(SCREENWIDTH, SCREENHEIGHT);
    hdc = glutCreateWindow("");
    glutFullScreen();
    glutDisplayFunc(display);
    glutKeyboardFunc(keyboard);
}

void configureGraphics()
{
    glDisable(GL_DEPTH_TEST);
    glColor3f(0.0,0.0,0.0);
    glClear(GL_COLOR_BUFFER_BIT);
    glColor3f(1.0,1.0,1.0);
    glMatrixMode(GL_PROJECTION);
    glLoadIdentity();
    glOrtho(0, SCREENWIDTH-1, 0, SCREENHEIGHT-1, -1, 1);
    PFNGLXSWAPINTERVALSGIPROC glXSwapIntervalSGI = (PFNGLXSWAPINTERVALSGIPROC)glXGetProcAddress((GLubyte*)"glXSwapIntervalSGI");
    glXSwapIntervalSGI(1);
}

void display(void)
{
    if (num/2 == 1) /* set the colour of the left square */
    {
            glColor3f(1.0, 1.0, 1.0);
    }
    else
    {
            glColor3f(0.0, 0.0, 0.0);
    }

     glClear(GL_COLOR_BUFFER_BIT);
     glBegin(GL_QUADS); /* draw the left square */
            glVertex2f(SCREENWIDTH/3 - 50,SCREENHEIGHT/2 - 50);
            glVertex2f(SCREENWIDTH/3 - 50,SCREENHEIGHT/2 + 50);
            glVertex2f(SCREENWIDTH/3 + 50,SCREENHEIGHT/2 + 50);
            glVertex2f(SCREENWIDTH/3 + 50,SCREENHEIGHT/2 - 50);
     glEnd();

    if (num%2 == 1) /* set the colour of the right square */
    {
            glColor3f(1.0, 1.0, 1.0);
    }
    else
    {
            glColor3f(0.0, 0.0, 0.0);
    }

     glBegin(GL_QUADS); /* draw the right square */
            glVertex2f(2*SCREENWIDTH/3 - 50,SCREENHEIGHT/2 - 50);
            glVertex2f(2*SCREENWIDTH/3 - 50,SCREENHEIGHT/2 + 50);
            glVertex2f(2*SCREENWIDTH/3 + 50,SCREENHEIGHT/2 + 50);
            glVertex2f(2*SCREENWIDTH/3 + 50,SCREENHEIGHT/2 - 50);
     glEnd();

    num = (num+1)%4; /* increase frame 0,1,2,3,0,1,2,3 */
    glutSwapBuffers();

     glutPostRedisplay();
}

void keyboard(unsigned char key, int x, int y)
{
    if (key == 27)
    {
            exit(0);
    }
}

p.s. I think the problem has something to do with the fifos, but I don’t know how to get around it … if I keep the fifos empty (by sleeping a bit in the loop), the problem goes away, but at the expense of not being there in time for the next frame. Then I get something like 0011223300112233 which isn’t a solution.

Your problem is trivial. (It almost sounds like you are testing us whether we are reading posted code carefully. :wink: )
The “problem” is in glXSwapIntervalSGI(2); It means you’ll get every second frame. glXSwapIntervalSGI() allows an application to specify a minimum periodicity of color buffer swaps, measured in video frame periods.

[QUOTE=Aleksandar;1256422]Your problem is trivial. (It almost sounds like you are testing us whether we are reading posted code carefully. :wink: )
The “problem” is in glXSwapIntervalSGI(2); It means you’ll get every second frame. glXSwapIntervalSGI() allows an application to specify a minimum periodicity of color buffer swaps, measured in video frame periods.[/QUOTE]

Whoops … I posted a test version of the code, 'sorry about that. I’ll edit it now, and you can have another look. While playing around with the problem, I set the swap interval to 2, but my recordings and results were made while it was still a (proper) 1.
(my apologies)

As I said in my first reply, I posted the wrong version of the code, however despite that, you might have misunderstood, since by your reply, it sounds like you saw the 0011223300112233 that I mentioned in my “p.s.” at the bottom of my original post, which is NOT what the (corrected) code did anyway. Look higher in the post and you’ll see that I was getting something more random than that. Thanks for helping - I hope you can help me get to the bottom of this now that I’ve clarified the problem.
-gt-

Before going any further with the analysis, you need to make some minor changes in the code and provide additional information.

  1. set glXSwapIntervalSGI(0); // disable vsync

  2. add glMatrixMode(GL_MODELVIEW); glLoadIdentity(); at the end of configureGraphics()

  3. add glClear(GL_COLOR_BUFFER_BIT); at the beginning of display() function

  4. forget about camera and 10K fps. Use CPU and GPU counters to get execution time

4.1. Read very precisely CPU time at the beginning of display() function - CPU_T1 (I have no idea how to do it in Linux; in Windows it is QueryPerformaceCounter() function.)

4.2. Read very precisely CPU time at the end of display() function, but before glutSwapBuffers() - CPU_T2

4.3. Read very precisely CPU time at the very end of display() function (after glutPostRedisplay()) - CPU_T3. I’m not familiar with GLUT so I don’t know whether this call is necessary or not.

4.4. Read very precisely GPU time at the beginning of display() function - GPU_T1 ( glQueryCounter(…, GL_TIMESTAMP); use glGetQueryObjectiv() to query the availability of the result, and read it with glGetQueryObjectui64v() only if it is available, to prevent stallings. )

4.5. Read very precisely GPU time at the end of display() function, but before glutSwapBuffers() - GPU_T2

CPU_T1 and GPU_T1 should be tracked also across multiple frames. For example GPU_T1(f+1) - GPU_T1(f) will give you a real frame-rate, while GPU_T2(f) - GPU_T1(f) will give you an actual work load.

It is very unlikely that OpenGL skips frames. Maybe it sometimes draws two (or more) frames before swapping buffers. Without previous counters we cannot say anything further.

[QUOTE=Aleksandar;1256454]Before going any further with the analysis, you need to make some minor changes in the code and provide additional information.

  1. set glXSwapIntervalSGI(0); // disable vsync

  2. add glMatrixMode(GL_MODELVIEW); glLoadIdentity(); at the end of configureGraphics()

  3. add glClear(GL_COLOR_BUFFER_BIT); at the beginning of display() function

  4. forget about camera and 10K fps. Use CPU and GPU counters to get execution time

4.1. Read very precisely CPU time at the beginning of display() function - CPU_T1 (I have no idea how to do it in Linux; in Windows it is QueryPerformaceCounter() function.)

4.2. Read very precisely CPU time at the end of display() function, but before glutSwapBuffers() - CPU_T2

4.3. Read very precisely CPU time at the very end of display() function (after glutPostRedisplay()) - CPU_T3. I’m not familiar with GLUT so I don’t know whether this call is necessary or not.

4.4. Read very precisely GPU time at the beginning of display() function - GPU_T1 ( glQueryCounter(…, GL_TIMESTAMP); use glGetQueryObjectiv() to query the availability of the result, and read it with glGetQueryObjectui64v() only if it is available, to prevent stallings. )

4.5. Read very precisely GPU time at the end of display() function, but before glutSwapBuffers() - GPU_T2

CPU_T1 and GPU_T1 should be tracked also across multiple frames. For example GPU_T1(f+1) - GPU_T1(f) will give you a real frame-rate, while GPU_T2(f) - GPU_T1(f) will give you an actual work load.

It is very unlikely that OpenGL skips frames. Maybe it sometimes draws two (or more) frames before swapping buffers. Without previous counters we cannot say anything further.[/QUOTE]

I’ll try this, but we’re at the end of the term here so I’ve got final exams to give and mark and so on, so I don’t know how soon I can get to it. But in the meantime, a few comments:

1/ I already used a function in Linux that returns the CPU time in microseconds, and according to that, everything is “on time” even though the actual output on the screen is not as it should be. I’ve checked the time at various strategic positions in my code including immediately before and after the blocking call. Also, there is overhead associated with trying to print that information, so to do it correctly, I need to populate a big array while taking the measurements and print it afterwards. If I try to print it during, there’s too much printing going on (trying to print several values per frame with the print window scrolling etc.) and I wind up not getting back from the call to the print function in a timely fashion, so debugging is not straight-forward in a time sensitive environment … one needs to be careful.

2/ I need to do some homework to find out how to get the GPU time. I didn’t do that.

3/ I don’t know how I can scrap the camera and digital recorder … its the only thing that reveals what’s really going on. By eye, you can tell “something” is just not quite right, but your eye can’t tell what is wrong, whereas the recorder captures the true story. Suppose the time stamps are as expected …ultimately the camera is telling the truth, since its oversampling 167 times for every frame.

4/ I’ll repeat the timestamp experiments following exactly what you described, however it doesn’t sound much different than what I’ve already tried with timestamps. What I found was evidence that the call doesn’t block a consistent amount of time for a frame as I expected it should. Rather, it sometimes returns quite quickly suggesting that some fifo is involved and the call thinks it okay to proceed without blocking because the frame is going to be put in some sort of queue instead of waiting for the vsync to come. Anyway, you’ll see what I mean when I conduct your experiment and give you the numbers.

5/ Can you clarify your direction to use:
set glXSwapIntervalSGI(0); // disable vsync
I don’t understand how I can disable vsync when that’s the only thing that orchestrates the timing of my code. Isn’t the main loop going to just going to race away wildly if I don’t have a blocking call in there to wait for vsync? I mean, that’s the only timing mechanism in my code, because I wanted my code to be synchronized to the frame rate precisely. If I do what you suggest here, my mail loop is just going to iterate as quickly as it can, which is far in excess of 200 loops per second. If I disable the blocking call to wait for vsync, what’s going to control the speed of my loop to make it execute once per frame?

-gt-

Off course that you have to allocate a vector and at each sample just increment index and write a value. Dumping to file should be done on app exit.

All homeworks have some purpose. :wink:

That’s ok. I just said to put it aside until we discover what’s going on.

Can you post some numbers. If frame-rate varies it means something is going on behind. Maybe it is because of other processes. Linux is a multitasking OS.

[QUOTE=George Townsend;1256457]5/ Can you clarify your direction to use:
set glXSwapIntervalSGI(0); // disable vsync
I don’t understand how I can disable vsync when that’s the only thing that orchestrates the timing of my code. Isn’t the main loop going to just going to race away wildly if I don’t have a blocking call in there to wait for vsync? I mean, that’s the only timing mechanism in my code, because I wanted my code to be synchronized to the frame rate precisely. If I do what you suggest here, my mail loop is just going to iterate as quickly as it can, which is far in excess of 200 loops per second. If I disable the blocking call to wait for vsync, what’s going to control the speed of my loop to make it execute once per frame?[/QUOTE]
I’ve asked you to remove synchronization just to be able to measure real time needed to accomplished some tasks. It won’t stay such. You can also compare intervals with and without vsync to see what’s happening. Vsync confines frame rate to some “very rounded” numbers like 60, 30, 20, etc. Even if the execution time is a microsecond longer than a frame-time frame-rate halves. So just disable it for the purpose of precise measuring.

[QUOTE=Aleksandar;1256459]
[much omitted]
Can you post some numbers. If frame-rate varies it means something is going on behind. Maybe it is because of other processes. Linux is a multitasking OS.

I’ve asked you to remove synchronization just to be able to measure real time needed to accomplished some tasks. It won’t stay such. You can also compare intervals with and without vsync to see what’s happening. Vsync confines frame rate to some “very rounded” numbers like 60, 30, 20, etc. Even if the execution time is a microsecond longer than a frame-time frame-rate halves. So just disable it for the purpose of precise measuring.[/QUOTE]

Yes, we worried about that in earlier testing, but confirmed by logging the time at strategic points that we DO get back in plenty of time. Having the sync enabled doesn’t hurt things actually, because with enough time stamps in enough places, you can see that we are faithfully back to the swap buffers call long before we have to, so not only is our code tight enough, but the background OS stuff manifests itself as some variation in the stamps, but never so long as to make us “late” for the next frame. Anyway, you’ll see this in the results as soon as I can get to them.

By the way, in the REAL code, which does significantly more work that this stripped down test code, we worried a lot about that, but time stamping here and there confirmed that even the real application related to this is definitely able to “keep up” even with the occasional “stall” from the OS demanding the CPU’s attention. We had to work hard to get the code sufficiently streamlined to accomplish that, so we’re aware of the issues.

As I said, once I’ve conducted your experiment, and showed you the results, you’ll see that although we are keeping up to the task, we are not getting the right stuff on the screen. What you will see though, are some odd (at least things I found odd) about the time that the blocking swap buffers takes to return. My assistant and I predicted that on the first call, it would block for a random amount of time (between 0 and 16666 usec) depending on when you “hit” the frame, but after that we expected it to consistently block for 16.666 msec, less the time our main loop took less a variable amount based on how much time the CPU spent off messing in the OS. That’s NOT, however what we saw, but I’ll post the details when I can.

Provided I time stamp the following points, and on the assumption that the time stamps clearly show we aren’t late for the next frame, do you think you can live with the sync NOT being disabled? Otherwise, I have to post a set of results that we know are not going to be useful apart from confirming that the code is small enough to easily execute in time, then I immediately need to rerun with the sync on to show the real problem. I mean, I’m okay with that if you feel its necessary, so you can let me know, but my suggestion is that if I do this, with the sync on, it should show you what you want to see:
i.e. time stamp the following with the sync enabled:

1/ the start of the main loop
2/ just before the blocking call to swapbuffers
3/ just after the swapbuffers
4/ the end of the loop (which isn’t much different from the start I guess)

My experience is that if I run the main loop precisely 10 times and terminate, there’s plenty of context to see what’s going on. Also, when I did this, I printed the actual time stamps in microseconds, and just let the time roll over to zero which it may or may not do in ten frames depending on where it starts, and I also printed the difference in time stamps since that’s easier to look at.

So that’s my suggestion, but I’m happy to do it with sync disabled first if you want. Let me know.
-gt-

This topic was automatically closed 183 days after the last reply. New replies are no longer allowed.