glMapBuffer(...) returns null pointer if another application occupied first 32G ram

OpenGL OpenGL: Advanced Coding
1

We have two applications A and B.
A is a CUDA computing app, allocate huge pinned memory.
B is an OpenGL based app.
The whole system host memory is 96G. Using GTX 1070.
When app A allocates more than 32G pinned memory, function glMapBuffer(…) just returns null pointer as there is no virtual memory. But definitely there is.
If app A allocates less than 32G pinned memory, then glMapBuffer(…) works properly.

My question: is there a limitation for OpenGL’s memory management? For example, it needs some buffer memory like page-locked memory to do data transfer and this part of memory must be located in the first 32G memory.

2

OpenGL itself doesn’t specify this kind of behaviour. This is going to be implementation-dependent.

Note the documentation for glMapBuffer: glMapBuffer

If the GL is unable to map the buffer object’s data store, glMapBuffer generates an error and returns NULL. This may occur for system-specific reasons, such as low virtual memory availability.

3

[QUOTE=mhagain;1287533]OpenGL itself doesn’t specify this kind of behaviour. This is going to be implementation-dependent.

Note the documentation for glMapBuffer: https://www.khronos.org/registry/OpenGL-Refpages/gl2.1/xhtml/glMapBuffer.xml[/QUOTE]

Hi mhagain, thank you for the reply.
Actually I noticed this line and was wondering why there is not enough virtual memory. Our virtual memory size is 96G.
That’s why I guess OpenGL’s driver under win7 cannot recognize the memory larger than 32G.
All the experiments proved that the pinned memory in app A must be lower than 32G to allow app B works properly.

4

[QUOTE=lzheng;1287532]The whole system host memory is 96G. Using GTX 1070.
When app A allocates more than 32G pinned memory, function glMapBuffer(…) just returns null pointer as there is no virtual memory. But definitely there is. If app A allocates less than 32G pinned memory, then glMapBuffer(…) works properly.[/QUOTE]

Ok first, glMapBuffer() returning NULL doesn’t necessarily mean there is no remaining virtual memory. But (on NVidia drivers), it may indicate that there is insufficient “pinned memory” left to satisfy the request.

AFAIK, the NVidia driver locates GL buffer objects either in GPU (on-device) memory or in CPU pinned memory (you can easily see this if you install a message callback using glDebugMessageCallback() – see this wiki link), and under some circumstances, they can migrate from one memory space to the other. Mapping a buffer on NVidia GL drivers automatically demotes the buffer object to pinned memory by default (IIRC), if it wasn’t already there.

Pinned memory is a special type of virtual memory (as you probably already know) which is not paged. Other names for “pinned memory” include: page-locked memory (on Linux) or non-paged memory or non-paged pool (on Windows). The opposite type of virtual memory is of course Paged memory (i.e. memory that can be paged out of physical RAM).

My question: is there a limitation for OpenGL’s memory management? For example, it needs some buffer memory like page-locked memory to do data transfer and this part of memory must be located in the first 32G memory.

For OpenGL in general, no (OpenGL is a spec defining an API, not a driver implementation). However, a specific GL driver could potentially have a limitation like this.

…that said, on a 64-bit CPU (which your numbers suggest you’re running on), I’ve not read or heard of a general limitation pn the amount of pinned memory being that low. In fact, on the net you can find folks that have much more pinned memory than that reserved.

How much non-paged memory is your Window 7 boxed configured for, and what does it show for the limit? (see Task Manager -> Performance -> Kernel Memory : Nonpaged, and/or Process Explorer -> System Information -> Memory). This is typically going to be a small subset of your total virtual memory.

Also, what’s your “NonPagedPoolSize” set to (not sure if setting this on Win7 works or not)?:

Check out Process Explorer or poolmon for tools that should let you monitor your non-paged pool usage dynamically.

According to this Microsoft blog post and reference page, the max non-paged pool memory on Win7 64-bit is min( 75% * Amount of RAM, 128 GB):

so with the right config, you should be able to fix your problem (**), collectively allocating at most 96GB of RAM for pinned memory with your current installed RAM.

(**) …that is, unless there’s some limitation in CUDA or the NVidia drivers that you hit up against.

5

the main point I guess is that this is going to be implementation-dependent.

A GTX 1070 is a consumer-class card, so (and I’m really entering speculation land here) I would suggest that it is neither designed for nor tested with a data set of this size. This problem most likely would not occur if using a Quadro.

6

[QUOTE=Dark Photon;1287557]Ok first, glMapBuffer() returning NULL doesn’t necessarily mean there is no remaining virtual memory. But (on NVidia drivers), it may indicate that there is insufficient “pinned memory” left to satisfy the request.

AFAIK, the NVidia driver locates GL buffer objects either in GPU (on-device) memory or in CPU pinned memory (you can easily see this if you install a message callback using glDebugMessageCallback() – see this wiki link), and under some circumstances, they can migrate from one memory space to the other. Mapping a buffer on NVidia GL drivers automatically demotes the buffer object to pinned memory by default (IIRC), if it wasn’t already there.

Pinned memory is a special type of virtual memory (as you probably already know) which is not paged. Other names for “pinned memory” include: page-locked memory (on Linux) or non-paged memory or non-paged pool (on Windows). The opposite type of virtual memory is of course Paged memory (i.e. memory that can be paged out of physical RAM).

For OpenGL in general, no (OpenGL is a spec defining an API, not a driver implementation). However, a specific GL driver could potentially have a limitation like this.

…that said, on a 64-bit CPU (which your numbers suggest you’re running on), I’ve not read or heard of a general limitation pn the amount of pinned memory being that low. In fact, on the net you can find folks that have much more pinned memory than that reserved.

How much non-paged memory is your Window 7 boxed configured for, and what does it show for the limit? (see Task Manager -> Performance -> Kernel Memory : Nonpaged, and/or Process Explorer -> System Information -> Memory). This is typically going to be a small subset of your total virtual memory.

Also, what’s your “NonPagedPoolSize” set to (not sure if setting this on Win7 works or not)?:

Check out Process Explorer or poolmon for tools that should let you monitor your non-paged pool usage dynamically.

According to this Microsoft blog post and reference page, the max non-paged pool memory on Win7 64-bit is min( 75% * Amount of RAM, 128 GB):

so with the right config, you should be able to fix your problem (**), collectively allocating at most 96GB of RAM for pinned memory with your current installed RAM.

(**) …that is, unless there’s some limitation in CUDA or the NVidia drivers that you hit up against.[/QUOTE]

Thank you so much for the reply.
I was disturbed by other thing for a while. Sorry for the delayed response. I really appreciated the detailed answer and want to do some testing before I asked you more questions which also takes some time.

Following the answer, I read the two links you provided. It seems that default nonpaged pool size is 75% of RAM or 128GB. The paged pool size in win 7 is min(system commit limit, 384GB).

In Registery Editor, the NonPagedPoolSize is a DWORD, limited to 32bit, I cannot make it larger than 4GB.
According to Process Explorer, this is how our win 7 with 32GB RAM or 96GB RAM look like:
[ATTACH=CONFIG]1505[/ATTACH]
So the operating system’s default Nonpaged Limit is 16,777,216
This confused me.

7

So now it seems to be Nvidia driver’s problem.