FBO Multisample with crash

Hi, I use a multisample FBO and a normal FBO to save the openGL scene to an image on the disk.
If I render to a small size, e.g. 4096 x 4096, it properly works. I get a good antialias.
But if I set a size to e.g. 6000 x 6000 or 7000 x 7000, even if my GL_MAX_VIEWPORT_DIMS is 8192 x 8192, I get a garbage image.
If I set 8000 x 8000, sometimes I cannot initialize the normal FBO because I get the error GL_FRAMEBUFFER_UNSUPPORTED_EXT or sometimes I can properly initialize the FBOs but later, at the first draw, I get the following crash. Please note, if I don’t use the FBO Multisample, and use only the normal FBO, I can quite use the size 8192 x 8192.

Program received signal: “EXC_BAD_ACCESS”.
sharedlibrary apply-load-rules all
(gdb) backtrace
#0 0x8f0db3cf in gldAttachDrawable ()
#1 0x8f1a4478 in gldUpdateDispatch ()
#2 0x8f1a4a10 in gldFlush ()
#3 0x163259ba in glFlushRender_Exec ()
#4 0x9973ec28 in glFlushRenderAPPLE () I even tried the traditional glFlush(), same result.
#5 0x00063518 in ?? ()
#6 0x00066e19 in ?? ()
#7 0x0001e2fa in ?? ()
#8 0x00091d99 in ?? ()
#9 0x95289edd in invoking_ ()
#10 0x95289e48 in -[NSInvocation invoke] ()
#11 0x952c6698 in -[NSInvocation invokeWithTarget:] ()
#12 0x9528acd4 in forwarding ()
#13 0x9528a802 in forwarding_prep_0_ ()
#14 0x95289edd in invoking_ ()
#15 0x95289e48 in -[NSInvocation invoke] ()
#16 0x9939f8c6 in -[NSConnection dispatchInvocation:] ()
#17 0x9939e049 in -[NSConnection handleRequest:sequence:] ()
#18 0x99397e52 in -[NSConnection handlePortCoder:] ()
#19 0x99397a81 in -[NSConcretePortCoder dispatch] ()
#20 0x9937f3dd in __NSFireMachPort ()
#21 0x95252772 in __CFMachPortPerform ()
#22 0x9524e4db in __CFRunLoopRun ()
#23 0x9524c464 in CFRunLoopRunSpecific ()
#24 0x9524c291 in CFRunLoopRunInMode ()
#25 0x933bdf9c in RunCurrentEventLoopInMode ()
#26 0x933bdd51 in ReceiveNextEventCommon ()
#27 0x933bdbd6 in BlockUntilNextEventMatchingListInMode ()
#28 0x904f978d in _DPSNextEvent ()
#29 0x904f8fce in -[NSApplication nextEventMatchingMask:untilDate:inMode:dequeue:] ()
#30 0x904bb247 in -[NSApplication run] ()
#31 0x904b32d9 in NSApplicationMain ()
#32 0x00002756 in ?? ()
kill
quit

Would you please tell me what I do wrong here?
I create the FBOs this way:

// Create Multisample FBO__________________________
GLsizei antialiasSamples = 4; // my max samples is 6 (kCGLRPMaxSamples)
 
// Create multisample RBO Color
glGenRenderbuffersEXT(1, &mMultiSampleRBOColor);
glBindRenderbufferEXT(GL_RENDERBUFFER_EXT, mMultiSampleRBOColor);
glRenderbufferStorageMultisampleEXT(GL_RENDERBUFFER_EXT, antialiasSamples, GL_RGBA8, mWidth, mHeight);

// Create multisample RBO Depth
glGenRenderbuffersEXT(1, &mMultiSampleRBODepth);
glBindRenderbufferEXT(GL_RENDERBUFFER_EXT, mMultiSampleRBODepth);
glRenderbufferStorageMultisampleEXT(GL_RENDERBUFFER_EXT, antialiasSamples, GL_DEPTH_COMPONENT24, mWidth, mHeight);

// Create multisample FBO to render in
glGenFramebuffersEXT(1, &mMultiSampleFBO);
glBindFramebufferEXT(GL_FRAMEBUFFER_EXT, mMultiSampleFBO);
glFramebufferRenderbufferEXT(GL_FRAMEBUFFER_EXT, GL_COLOR_ATTACHMENT0_EXT, GL_RENDERBUFFER_EXT, mMultiSampleRBOColor);
glFramebufferRenderbufferEXT(GL_FRAMEBUFFER_EXT, GL_DEPTH_ATTACHMENT_EXT, GL_RENDERBUFFER_EXT, mMultiSampleRBODepth); 
// Check the validity of the Multisample FBO…


// Create Normal FBO__________________________

// Create RBO Color
glGenRenderbuffersEXT(1, &mRBOColor);
glBindRenderbufferEXT(GL_RENDERBUFFER_EXT, mRBOColor);
glRenderbufferStorageEXT(GL_RENDERBUFFER_EXT, GL_RGBA8, mWidth, mHeight);

// Create RBO Depth
glGenRenderbuffersEXT(1, &mRBODepth);
glBindRenderbufferEXT(GL_RENDERBUFFER_EXT, mRBODepth);
glRenderbufferStorageEXT(GL_RENDERBUFFER_EXT, GL_DEPTH_COMPONENT24, mWidth, mHeight);

// Create FBO to get the final image
glGenFramebuffersEXT(1, &mFBO);
glBindFramebufferEXT(GL_FRAMEBUFFER_EXT, mFBO);
glFramebufferRenderbufferEXT(GL_FRAMEBUFFER_EXT, GL_COLOR_ATTACHMENT0_EXT, GL_RENDERBUFFER_EXT, mRBOColor);
glFramebufferRenderbufferEXT(GL_FRAMEBUFFER_EXT, GL_DEPTH_ATTACHMENT_EXT, GL_RENDERBUFFER_EXT, mRBODepth);

// Check the validity of the FBO...

glBindFramebufferEXT(GL_FRAMEBUFFER_EXT, 0);
glBindRenderbufferEXT(GL_RENDERBUFFER_EXT, 0);


// This is my draw routine__________________________
glBindFramebufferEXT(GL_FRAMEBUFFER_EXT, mMultiSampleFBO);

// Draw the scene
glFlushRenderAPPLE();
glFinish();

glBindFramebufferEXT(GL_READ_FRAMEBUFFER_EXT, mMultiSampleFBO);	//Bind the Multisample FBO to read from
glBindFramebufferEXT(GL_DRAW_FRAMEBUFFER_EXT, mFBO);	//Bind the normal FBO to write to

glBlitFramebufferEXT(0, 0, mWidth, mHeight, 0, 0, mWidth, mHeight, GL_COLOR_BUFFER_BIT, GL_NEAREST);

//Bind the standard FBO for reading
glBindFramebufferEXT(GL_FRAMEBUFFER_EXT, mFBO);
glReadBuffer(GL_COLOR_ATTACHMENT0_EXT);
glReadPixels(0, 0, mWidth, mHeight, GL_RGBA, GL_UNSIGNED_BYTE, pixels);

glBindFramebufferEXT(GL_DRAW_FRAMEBUFFER_EXT, 0);
glBindFramebufferEXT(GL_READ_FRAMEBUFFER_EXT, 0);
glBindFramebufferEXT(GL_FRAMEBUFFER_EXT, 0);

I run MacOS X 10.6.5, NVIDIA GeForce GT 330M OpenGL Engine, 512MB VRAM, OpenGL Version: 2.1 NVIDIA-1.6.24

Did you checked the maximum size your FBO renderbuffer and your textures can handle ? Also, did you tried to stay with POT sizes ?

Thank you. Yes, I check

glGetIntegerv(GL_MAX_RENDERBUFFER_SIZE_EXT, &maxRenderbufferSize);
glGetIntegerv(GL_MAX_TEXTURE_SIZE, &maxTexSize);

and they both return 8192.
I lately tried to render with size 8192 x 4096 and it worked.
But 8192 x 8192 still crashes. The debugger shows
gldAttachDrawable
0x8f0db3cf <+2655> mov %eax,0x0
I am stuck. Any idea?

I think you run out of GPU memory.

one non multisampled FBO 8192x8192 RGBA8 + DEPTH(32bit) = 512MB
The multisampled FBO is larger (hard to say how much, since NVIDIA uses compression). Probably over 1024MB.

Your card has 512MB VRAM.

LeoGL: it is good practise to check for GL errors at leat after allocating new GL resources. I don’t see any error checks in your code. You should at least check glGetError() for GL_OUT_OF_MEMORY after calling glRenderBufferStorage().

Additionally, I strongly suggest you to use tiled rendring for creating such huge screenshots. This would then work even on gfx cards with much less than 512MB vram.

Thank you skynet and mfort.
Yes, I do tiled-rendering, but I need to know “when” I need to do it, that is, I need to know the maxOutSize I can render in.
So, I thought, if imageSize > maxOutSize then I do tile-rendering, otherwise I render normally (faster).
However, in order to test the FBO, for now, I turned off the tiled-rendering.
I do now check for the glError and yes, I “always” get an error when storing the multisample color buffer. “Always” sounds really strange.

glGenRenderbuffersEXT(1, &mMultiSampleRBOColor);
glBindRenderbufferEXT(GL_RENDERBUFFER_EXT, mMultiSampleRBOColor);
glRenderbufferStorageMultisampleEXT(GL_RENDERBUFFER_EXT, 4, GL_RGBA8, mWidth, mHeight);
glErr = glGetError();

When mWidth and mHeight are 1024 or 2048 I get GL_INVALID_ENUM, but I can quite render and save the image to the disk.
When mWidth and mHeight are 4096 or higher I get GL_INVALID_ENUM (and rarely GL_INVALID_VALUE) and the app crashes.

If I do a rough calculation I get
4096 * 4096 * (4 + 3) = 112MB
4 is bytesXpixel of the color buffer, 3 is bytesXpixel of the depth buffer (GL_DEPTH_COMPONENT24)

The other 2 buffers of the normal FBO (no multisample) should be the same 112MB
so I use 224MB. My VRAM is 512…

Should I add the window FrameBuffers too? Its size is now 10 x 10 pixels, same 4 + 3 bytesXpixel.
I am stuck.

Some info

VRAM 512 MB
GL_MAX_RENDERBUFFER_SIZE_EXT is 8192
GL_MAX_TEXTURE_SIZE is 8192
GL_MAX_VIEWPORT_DIMS is 8192 x 8192
kCGLRPMaxSamples is 8

Check for errors right before this code in order to make sure the error is not “dangling” from somewhere else.

Thanks skynet. The error indeed came from some other side. You are a wizard!
So, now, I get no error at all when glRenderbufferStorageEXT, but when I set the size to 4096 x 4096 I get that crash. All the time.

So no more forward? I still get that crash and no error memory. It’s a mystery.

I think your guestimates for GPU memory consumption are off. You’re trying to allocate a 4x MSAA framebuffer @ 4096x4096.

Theory says that’s 32 bytes/pixel for just the multisample buffer (RGBA8 color + depth). So total GPU memory consumption is 512MB, even before we start counting GPU memory space for your other needs (other render targets, downsample buffers, system framebuffer, Hi-Z, etc.) I think you mentioned your GPU has 512 GB. So that just ain’t gonna fit :frowning:

Use NVX_gpu_memory_info to verify what I’m saying with memory consumption is true on your platform. But I think you’re just trying to allocate more dedicated GPU memory than you GPU has.

Make sure you are checking for GL errors, especially after allocating those FBO render targets. Wouldn’t hurt to stick a glFinish() before these checks either, but you might have to render to the FBO to force the GPU to actually allocate the space for those render targets on the GPU and discover “Oh crap, I haven’t got that kind of space left”. :eek:

If you really need such insane viewport sizes, break the viewport down and render it in chunks.

Thank you, I understand. MSAA requires 32 bytes/pixel VRAM, so it’s a huge amount.
Well, so now I need to detect “when” I would run out of VRAM and turn on the tiled-rendering.

Unfortunately my card has not the extension GL_NVX_gpu_memory_info. Also, I don’t even know on which machines/GPU my app will run… so how can I get that info from any GPU? But most of all, how can I trap the error before the crash? I mean, actually I can indeed instantiate the large FBOs and get no error at all.

glRenderbufferStorageMultisampleEXT or glRenderbufferStorageEXT
glFinish();
glGetError();
reports no error;

The same no error with the
glFramebufferRenderbufferEXT
glFinish();
glGetError();

The same no error with the
glCheckFramebufferStatusEXT
glFinish();
glGetError();

glCheckFramebufferStatusEXT
reports no error too

One second later, I draw the scene into the FBO and I get a crash. Too bad!
While if I call the same procedure using a smaller size, everything works.
I must know “before” whether I can or I cannot use a large FBOs.
Any idea?

Yes, I can do that, but how large the tile should be?
Which size will guarantee I don’t get a crash?
1/4 or 1/8 or… of the GL_MAX_VIEWPORT_DIMS?
And if I need to process 10,000 images for a movie, don’t you think that using a small tile will multiply the waiting time by 10?
10 hours waiting instead of 1 hour only is a huge difference.
That’s why I need to maximize the tile size. That’s why I need to know whether the FBO with a given size will work. Otherwise I will retry to initiate the FBO with a smaller size…

But most of all, how can I trap the error before the crash?

In general, the best way to avoid this kind of thing is to not allocate incredibly large things. Simply don’t make a 4096x4096 16x multisample image. Ever. Even if you’re using a card with 2GB of RAM.

Ok thank you. How should I decide for the best tile size?

Yeah, ideally you’d like to get a GL error back that you can trap and use to tune your tile size. You might cook a short GLUT test program that illustrates your crash that folks can run and offer suggestions on. Could be a vendor driver bug that should be reported.

But I think the core issue in your way, even if the program didn’t crash, is that GL virtualizes GPU memory. It will do its best to continue to render even after you’ve blown past GPU memory (e.g. too many textures to fit on the GPU), and will swap things on and off the board to try and make it all render. So generally speaking using too much memory won’t fail – it’ll just slow down render perf.

So yeah, sometimes tuning your approach based on the amount of GPU memory you sense on the GPU is a really ugly but necessary evil. Besides NVX_gpu_memory_info (NVidia-specific) and ATI_mem_info (ATI-specific), other options you can use to query total GPU memory include OpenCL (cross-vendor), CUDA (NVidia-specific), and libXNVCtrl (NVidia-specific). If you need more details, just let us know.

This “tune based on GPU memory installed” approach won’t be ideal when you have multiple apps using significant amounts of GPU memory, but hey. What other options are there? Often we assume this case away because performance would suffer. About the only thing else you can do is assume your app has at least X MB of GPU memory available to it, and just hand-wave the results if not. Some apps take this approach.

Thank you Dark Photon. Unfortunately I don’t get any error, as I said above.
I don’t have NVX_gpu_memory_info extension, and openCL gives me only the total VRAM (CL_DEVICE_GLOBAL_MEM_SIZE) or 1/4 of it (CL_DEVICE_MAX_MEM_ALLOC_SIZE…) no matter how many buffers I have already allocated. The crash occurs just after allocating the normal color render buffer (after the multi-sampled buffers). To cut the problems here, I decided to set the max rendering size to MAX_DIM * 0.25. Someone bless me.