glFlush or glFinish with mulithreading?

[pjcozzi]

Hello,

We are doing multithreaded GL using a (shared) context per-thread with 3.2 core profile. The main thread renders and worker threads create and fill textures, vertex buffers, etc. The question is when a worker thread is done preparing a GL resource, should it call glFlush or glFinish so it can be used by the main thread?

After reading (and re-reading) this, I am convinced you only need to call glFlush. This works, as best as unit tests can show, on several different machines with different flavors of Windows and NVIDIA/ATI cards.

On one machine, it does not work consistently, e.g., textures have wrong colors. Although, it does work consistently if we replace glFlush with glFinish. This machine has 4 hyperthreaded cores, Windows 7 64, and NVIDIA drivers 197.45. We think there is a race condition somewhere that is most commonly exposed when the level of parallelism goes up.

We hope this is a problem in our code but it could be an issue in the drivers since even very simple tests can fail, e.g.:
- Create context for worker thread on the main thread
- Create shared context for main thread and set current
- Spawn worker thread
---- Set context current
---- Create and fill texture
---- glFlush (glFinish works!)
- Main thread blocks until worker thread is finished
- Verify texture contents - fails!

Source for this is here: TextureMultiThreadingTests and TextureFactory

So, should we be able to use glFlush? If not, is glFinish going to kill performance? It is certainty a problem in the single threaded world.

Regards,
Patrick

[overlay]

glFinish() makes sense, glFlush() does not guarantee the completion of the commands.

But with OpenGL 3.2, the recommended way is glFenceSync () with glWaitSync().


The 3.2 spec are pretty clear

"D.3.1 Determining Completion of Changes to an object" page 325.

"Completion of a command 1 may be determined either by calling Finish, or by calling FenceSync and executing a WaitSync command on the associated sync object. The second method does not require a round trip to the GL server and may be more efficient, particularly when changes to T in one context must be known to have completed before executing commands dependent on those changes in another context."


See also "5.2 Sync Objects and Fences" page 241--246.

[Bruce Wheaton]

I'm doing something similar, and it was a bit of a nightmare until recent drivers (Nvidia).

I do a glFlush, then poll for a fence to finish (old style nv fence, not the new sync fences). If things seem to be taking too long, I do wait for completion, but I often have other work to do.

I specifically don't glfinish - I think it will impose a big workload at an arbitrary time, when the screen threads might need to do work. If that makes sense.

Bruce

[pjcozzi]

Good call - for some reason I thought you couldn't share sync objects across contexts. That might have been true in an older version but that is certainty not the case now.

I have sync objects working (although I still need to test on the 4 core machine). I have a question/comment on ClientWaitSync vs WaitSync. At first only ClientWaitSync worked for me and WaitSync caused deadlock later in the tests. So I added a glFlush after WaitSync as Bruce mentioned and as shown in issue 20 in GL_ARB_sync.

Why exactly is this required? Is it because without flushing the worker thread, the main thread could wait on an unsignaled fence forever since the wait blocks the GL server?

Regards,
Patrick

[overlay]

I don't understand why it works with glClientWaitSync() without glFlush() but I may have an explanation for why there is a deadlock with glWaitSync() without glFlush().

OpenGL commands are accumulated in a command queue before being sent to the server. The commands are sent only in 2 conditions, a new command is added the queue, making the queue full, so the driver flushes it automatically. If you don't add an explicit glFlush() after your last command, then there is a chance that the queue is not full, so it will never be flushed.

If FenceSync() is the last command, then it will no be executed, hence the deadlock. If FenceSync() is not the last command, then the following commands will make the queue full and trigger an implicit flush.

In single thread mode, you usually don't care in double buffer mode because after rendering a buffer swap involves an implicit flush. But glFlush() is required in single buffer mode. It is the same for a thread that does not do OpenGL rendering but just OpenGL resource allocation/initialization.

Don't take my words for being the truth. This is just my understanding of the issue.

[pjcozzi]

Your explanation sounds pretty accurate given the behavior I am seeing. In the tests I am using, the worker thread(s) only issue a handful of GL commands to create and fill a GL resource then the last command they issue is glFenceSync. So your saying the glFenceSync was sitting around in the command queue, so back in the main thread glWaitSync was waiting on a sync object that was never signaled.

Thanks,
Patrick

[mfort]

I am also developing application with 2 OpenGL threads (main + worker).

I run into the same troubles with glFenceSync and glWait when there is no other OpenGL command following the glFenceSync. Actually I do not see a deadlock. This is probably caused by timeout in glWait.

What overlay said is probably correct explanation of this situation. But I do not think is it good to assume this behavour is correct. I read the ARB_sync spec several times and there is nothing about using glFlush in connection with glWait.

I am sure ARB was aware of glFlush problem because they solved it in glClientWait by using flag SYNC_FLUSH_COMMANDS_BIT. There is nothing similar in glWait.

I used the glFlush as a workaround. I believe the spec is either changed or NVIDIA drivers fixed. The flush is definitely bad solution as it can hurt performance when there are some other OpenGL commands after the glFenceSync.

I also noticed that glFlush after glFenceSync is more important on Windows XP then on Windows 7 (where it works most of the time).

[Alfonse Reinheart]

I used the glFlush as a workaround. I believe the spec is either changed or NVIDIA drivers fixed. The flush is definitely bad solution as it can hurt performance when there are some other OpenGL commands after the glFenceSync.

If I understand the behavior of glWaitSync correctly, it stops the GPU from getting any new commands until the sync object has signaled (ie: the fence completed). I'm pretty sure that's going to hurt performance more than doing a glFlush

What I'm most curious about is why people need glWaitSync to begin with? What is the use case for this function?

[pjcozzi]

What I'm most curious about is why people need glWaitSync to begin with? What is the use case for this function?

I can't speak for everyone but I am using it to synchronize between two threads - a worker thread that produces a GL resource like a texture and a main thread that consumes the resource for rendering. Even with application level synchronization, without GL synchronization, it appears rendering on the main thread can start before the resource was fully initialized.

Or are you asking why use glWaitSync instead of glClientWaitSync? Because I actually have the opposite question, why would someone want to block everything instead of just the GL server?

Regards,
Patrick

[Alfonse Reinheart]

Or are you asking why use glWaitSync instead of glClientWaitSync? Because I actually have the opposite question, why would someone want to block everything instead of just the GL server?

If you're using fences to, for example, check to see if a particular buffer object is still in use (so that you can use MapBufferRange's GL_UNSYNCRONIZED flag), then only glClientWaitSync will let you know when the fence has completed.

Now, the timeout value will be 0 in this case, so you're not really blocking.