Part of the Khronos Group
OpenGL.org

The Industry's Foundation for High Performance Graphics

from games to virtual reality, mobile phones to supercomputers

Page 1 of 2 12 LastLast
Results 1 to 10 of 16

Thread: Regarding loading 3d meshes dynamically

  1. #1
    Advanced Member Frequent Contributor
    Join Date
    Mar 2009
    Location
    Singapore
    Posts
    800

    Regarding loading 3d meshes dynamically

    Hi all,
    I am working on an application where I need to load a lot of 3D textured meshes (these range in size from 5 mb to 20 mbs). Currently, I am loading around 20 of these models at initialization but it adds some latency (around 5 - 7 mins) as the application is busy loading the models and the screen is blank untill all of the models have been loaded. Now my client is asking me to add in more models (roughly from 150 to 600 models). Now if I continue with the current approach it will not only take ages to load but also it will overflow my GPU buffer object memory at some point. So now I am thinking of doing dynamic loading so i want to ask others what will be the best method to enable dynamic 3d textured mesh loading.

    Any pointers to relevant tutorials/docs/papers will be appreciated.

    Thanks,
    Mobeen
    Last edited by mobeen; 04-01-2013 at 10:06 PM.
    Regards,
    Mobeen

  2. #2
    Advanced Member Frequent Contributor
    Join Date
    Apr 2010
    Posts
    752
    Hmm, 5 minutes to load (less than) 400 MB of models/textures seems a bit longish - are you using optimized builds of your code and libraries?
    Regarding your question, I'm not sure what particular problems you are expecting, when there is a request to display a specific model, you load it and display it. Of course that causes some delay, since data has to move from disk to main memory to the GPU, depends on your application if that's acceptable, but you are not giving enough details to know.
    If it is a problem, you could use a separate OpenGL context and a "loading" thread to concurrently keep rendering your scene and add the new model once it's ready, but that is quite a bit more complicated. Or you make your loading code interruptible, where basically you give the loader a certain amount of time and it loads data until the time slice is used up (or it's done), remembers the state and gives control back to the caller. Next frame you continue loading where you left off - also a bit of work. So (as usual ) it depends on what you need.

  3. #3
    Advanced Member Frequent Contributor
    Join Date
    Mar 2009
    Location
    Singapore
    Posts
    800
    Hi Carsten,
    Thanks for your reply.
    I'm not sure what particular problems you are expecting, when there is a request to display a specific model, you load it and display it
    This was my initial approach but that gives a noticeable lag when the user is selecting his model at runtime which spoils the user experience so I turned to load all models at initialization approach which then moves the delay to startup.

    If it is a problem, you could use a separate OpenGL context and a "loading" thread to concurrently keep rendering your scene and add the new model once it's ready, but that is quite a bit more complicated.
    Hmm this sounds interesting any tutorials/references on this? By the way lets say I implement this approach, create two contexts, one for loading and another for normal work, how would the two GL contexts communicate? Would it be that the loading context will simply access a global datastructure which the rendering thread will render or is there any other approach you are referring to? and what about textures and buffer objects that the loading thread will be loading? will they be accessible by the rendering thread?

    EDIT: Found some similar threads on gamedev forums thought might be nice to share in my thread for others
    http://www.gamedev.net/topic/424705-...engl-contexts/
    http://www.gamedev.net/topic/395910-...ultithreading/
    http://www.gamedev.net/topic/394701-...le-data-loads/
    Last edited by mobeen; 04-02-2013 at 06:25 PM.
    Regards,
    Mobeen

  4. #4
    Senior Member OpenGL Guru Dark Photon's Avatar
    Join Date
    Oct 2004
    Location
    Druidia
    Posts
    3,213
    Quote Originally Posted by mobeen View Post
    ...it will overflow my GPU buffer object memory at some point. So now I am thinking of doing dynamic loading so i want to ask others what will be the best method to enable dynamic 3d textured mesh loading.
    carsten already gave you some good tips. So just a few more thoughts. First, you have several caches: CPU memory and GPU memory, with different sizes. You don't have to have everything loaded onto the GPU that you have loaded onto the CPU. Similarly, you don't have to have everything on the CPU that you have on disk. You know the sizes you have to work with here along with the sizes of your models, which factors into this. You also know where the bottlenecks are (e.g. is your mesh format an awkward CPU-intensive pain-in-the-butt to load, or is it largely a fast-to-load near memory image of your data).

    Also, I'd second the background thread idea for loading from disk. You prob don't want to be doing this disk I/O in your GL thread, unless maybe you're doing non-blocking I/O. But whatever you decide on, bench it before you buy into it! Next, keep in mind that that issue (loading from disk to CPU mem) is a completely separate issue from if/how you use a background thread to load some data form CPU mem to GPU. Decide what you need there, and why. You probably want to do some more reading on that; websearch Parallel OpenGL FAQ for starters.



    And yeah, 400MB/5 min = 1.3MB/sec is pretty bad. You can rip 50-150MB/sec off a cheap, non-RAIDed disk with sequential reads. Before you even think about any of the above, I'd get to the bottom of this perf problem and fix/rearchitect whatever you need to to get your load bandwidth up.

  5. #5
    Advanced Member Frequent Contributor
    Join Date
    Mar 2009
    Location
    Singapore
    Posts
    800
    Quote Originally Posted by Dark Photon View Post
    carsten already gave you some good tips. So just a few more thoughts. First, you have several caches: CPU memory and GPU memory, with different sizes. You don't have to have everything loaded onto the GPU that you have loaded onto the CPU. Similarly, you don't have to have everything on the CPU that you have on disk. You know the sizes you have to work with here along with the sizes of your models, which factors into this. You also know where the bottlenecks are (e.g. is your mesh format an awkward CPU-intensive pain-in-the-butt to load, or is it largely a fast-to-load near memory image of your data).

    Also, I'd second the background thread idea for loading from disk. You prob don't want to be doing this disk I/O in your GL thread, unless maybe you're doing non-blocking I/O. But whatever you decide on, bench it before you buy into it! Next, keep in mind that that issue (loading from disk to CPU mem) is a completely separate issue from if/how you use a background thread to load some data form CPU mem to GPU. Decide what you need there, and why. You probably want to do some more reading on that; websearch Parallel OpenGL FAQ for starters.



    And yeah, 400MB/5 min = 1.3MB/sec is pretty bad. You can rip 50-150MB/sec off a cheap, non-RAIDed disk with sequential reads. Before you even think about any of the above, I'd get to the bottom of this perf problem and fix/rearchitect whatever you need to to get your load bandwidth up.
    Hi Dark Photon,
    Thanks for your insights. Indeed before going with the parallelizing, I am going to first recheck what is causing this delay. Just a quick benchmarking of the code, I realize that the delay is coming from the SOIL image loading library (my mesh models load in pretty quickly though) so I think i would try to load models without texture and bench again to see if the loading improves.
    Regards,
    Mobeen

  6. #6
    Advanced Member Frequent Contributor
    Join Date
    Mar 2009
    Location
    Singapore
    Posts
    800
    Moderators there is a lot of span moving in these forums please clean it up.

    OK update from my side.
    I have tried to get this to work in a minimalistic opengl glut app. I am using Boost Threads and this is how i do it. First my main app window is initialized like this. I first init glut windowing stuff and then store the current DC and HGLRC from the main window. After attaching the display and other callbacks, I call my thread function to load models. This instantly loads the models in parallel.
    Code :
    void main(int argc, char** argv) {
       glutInit(&argc, argv);
       glutInitDisplayMode(GLUT_DOUBLE | GLUT_RGBA | GLUT_DEPTH);
       glutInitWindowSize(width, height);
       glutCreateWindow("GLUT Multithreaded OpenGL Demo");
       hrcOld = wglGetCurrentContext();
       hdc = wglGetCurrentDC();
       glutDisplayFunc(OnRender);
       glutReshapeFunc(OnReshape);
       glutIdleFunc(OnIdle);
       glutMouseFunc(OnMouseDown);
       glutMotionFunc(OnMouseMove);
       glutCloseFunc(OnShutdown);
       InitGL();
       boost::thread loader(LoadModels);
       glutMainLoop();
    }
    Now the load model function first makes a new context and then makes it current and then loads the models. So far this is working fine. Am i doing it properly?
    Code :
    boost::once_flag flag =  BOOST_ONCE_INIT;
    void init_context() {
        hrcNew = wglCreateContext(hdc);
        wglMakeCurrent(hdc, hrcNew);
    }
    void LoadModels() { 
       boost::call_once(&init_context, flag);
     
       float start = (float)glutGet(GLUT_ELAPSED_TIME);
     
       for(int i=0;i<NUM_MODELS;i++) {     
            boost::mutex::scoped_lock lock(mutex);
            //model loading code here//
            modelList.push_back(model); 
        }    
        start = ((float)glutGet(GLUT_ELAPSED_TIME)-start);
        std::cout<<"Models loaded in "<<start<<" msecs."<<endl;
        wglMakeCurrent(hdc, hrcOld);
        wglDeleteContext(hrcNew);
    }
    Last edited by Dark Photon; 04-04-2013 at 05:30 AM.
    Regards,
    Mobeen

  7. #7
    Advanced Member Frequent Contributor
    Join Date
    Apr 2010
    Posts
    752
    An OpenGL context can be current in at most one thread at a time. When you reach the end of LoadModels() you try to make the "main" context current, violating that constraint. I believe you also need to instruct your contexts to share resources (see wglShareLists).

    Also, note that Dark Photon made a suggestion that does not require the complications of multiple OpenGL contexts: Load data from disk into CPU memory (i.e. into a suitable data structure) in the thread, then tell the main thread to create the OpenGL objects for that model. That way only the main thread needs an OpenGL context - there'll still be a bit of a delay, since before rendering the data has to be moved to GPU memory, but the CPU -> GPU memory transfer is orders of magnitude faster than the disk -> CPU memory transfer.

  8. #8
    Advanced Member Frequent Contributor
    Join Date
    Mar 2009
    Location
    Singapore
    Posts
    800
    Thanks for the detailed response carsten. OK so i think i will go with the one context approach. Load models in a CPU datastruct. When the load is complete, i signal the main context that the data is ready and so it can carryout the rendering.

    The reason why i create a new context in my load function is because I am using SOIL to load model textures. My models are stored in a CPU datastructure and I make no gl calls (glBufferData etc.). However SOIL makes a gl call for checking for the availability of non-power of two texture extension using glGetString (line 1880 in SOIL.c). As soon as the code comes there, the application crashes if i dont create a new context in the LoadModels function.
    An OpenGL context can be current in at most one thread at a time. When you reach the end of LoadModels() you try to make the "main" context current, violating that constraint.
    Agreed but as soon as I am done with the loading thread, shouldn't I make the main context current and then delete the new context? I thought I cant delete a current rendering context can I?
    Regards,
    Mobeen

  9. #9
    Advanced Member Frequent Contributor
    Join Date
    Apr 2010
    Posts
    752
    Agreed but as soon as I am done with the loading thread, shouldn't I make the main context current and then delete the new context? I thought I cant delete a current rendering context can I?
    I don't know off the top of my head about deleting a current context, but you can make no context current (i.e. wglMakeCurrent(hdc, NULL)).

  10. #10
    Advanced Member Frequent Contributor
    Join Date
    Mar 2009
    Location
    Singapore
    Posts
    800
    Quote Originally Posted by carsten neumann View Post
    I don't know off the top of my head about deleting a current context, but you can make no context current (i.e. wglMakeCurrent(hdc, NULL)).
    I am not deleting the current context, i first restore the old context and then delete the new context. Besides, wouldn't this be a rendering context deletion leak? The new gl context is never deleted. This seems to be very similar to this code example
    Code :
    int* pData=new int[1000];
     
    //work on pData once finished
     
    pData = NULL; //we should call delete [] pData right?
    Regards,
    Mobeen

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •