PDA

View Full Version : Parallel loading of object files.



Waleed
12-24-2011, 01:17 PM
Ok, so I'm trying to load really huge object files using the Nate Robin's GLM drawModel() function and I was wondering if any of you know if there's an existent way for me to utilize numerous cores in loading the model instead of just one. I thought if not then I'd figure out a way to split the object file into numerous smaller ones, maybe write a small program that does so or something. Then I'd use multiple threads to load the files concurrently. Any thoughts, suggestions or preferably knowledge of existent methods that do the previous or something similar?

Your help is much appreciated,
-W

Alfonse Reinheart
12-24-2011, 02:44 PM
In general, the primary cost of loading any large file is in loading the file. Which is a fundamentally single-threaded operation. Splitting it into multiple files will only make it take longer.

If you want to make model loading take less time, then compress the file and decompress it after loading.

Simon Arbon
12-24-2011, 04:47 PM
The best thing that you can do is put it in one big (compressed) file and then de-fragment your hard disk to minimise the seek time.
The only time that splitting the file might improve load time is if you had more than one physical hard disk and put one file on each.

Waleed
12-25-2011, 02:38 AM
In general, the primary cost of loading any large file is in loading the file. Which is a fundamentally single-threaded operation. Splitting it into multiple files will only make it take longer.

If you want to make model loading take less time, then compress the file and decompress it after loading.


The best thing that you can do is put it in one big (compressed) file and then de-fragment your hard disk to minimise the seek time.
The only time that splitting the file might improve load time is if you had more than one physical hard disk and put one file on each.

I understand what you both mean. Thanks for the heads up.

What I don't understand though is your talk about compressing the large obj file.

@Alfonse:
What do you mean by compress the file and then decompress it after loading? Please elaborate.

Alfonse Reinheart
12-25-2011, 11:24 AM
Compression. (http://en.wikipedia.org/wiki/Data_compression) Zip files. (http://en.wikipedia.org/wiki/Zip_%28file_format%29) zlib. (http://en.wikipedia.org/wiki/Zlib) Do any of these things ring a bell?

Waleed
12-25-2011, 11:52 AM
Compression. (http://en.wikipedia.org/wiki/Data_compression) Zip files. (http://en.wikipedia.org/wiki/Zip_%28file_format%29) zlib. (http://en.wikipedia.org/wiki/Zlib) Do any of these things ring a bell?

I understand the concepts but not how to use them in this specific situation. Excuse my ignorance, I'm fairly new at this. Thank you for your patience as well as your willingness to help so far.