View Full Version : Game Model Format

01-31-2008, 11:53 AM
I looked into Collada, .x, and a few other file formats, and couldn't find a fast-loading file format with support for hierarchal meshes and skinned animation. So I wrote my own.

Here is a preliminary spec:

The idea is that you can load this straight into your engine, without any of the fiddling around that other formats require. Tangent-bitanget arrays are pre-calculated, animation keys are already interpolated at regular intervals, and the data can be read straight into vertex arrays to be sent to the GPU. It's easy to read and can be expanded to support new data without breaking old loaders.

At this point all I have is a .b3d to .gmf converter, but I am talking to a few people about adding exporters for various 3D packages once the file spec is finalized.

This isn't just vaporware, I have animated skinned models working right now:

01-31-2008, 12:03 PM
great, just what the world needs, yet another geometry file format.

01-31-2008, 12:19 PM
Indeed. I evaluated tons of them and could not find a single one that supported skinned animation and hierarchies, and didn't require tons of preprocessing.

01-31-2008, 12:53 PM
BTW, Collada is not meant to be a final format. It is an interchange format. It is intended to represent 90% of the data in a tool like Max or Maya. It is for toolchain work; it makes it easier to switch between tools like Max and Maya. It also makes it easier to write converters that go from a tool to the end format that you want.

And I have to agree with Knackered. The world doesn't need another geometry file format. You may find it useful, but I wouldn't expect it to be considered generally useful. In my experience, the final geometry format should be designed based on need anyway.

01-31-2008, 04:30 PM
A simple format like that isn't a bad idea. Many small projects don't have the time to implement universal format like collada. Unfortunately the known formats that supports bones aren't very GPU friendly. For example MD5 and cal3d are designed for CPU based animation.
So a simple GPU friendly format could be helpful for small projects, if exporters will be available.

01-31-2008, 04:45 PM
From my experience, portability and performance are mutually exclusive. You always end up with some more or less internal binary format that suits your own needs. I'm no game developer, but I guess that this is especially true for game data, hm?


01-31-2008, 05:16 PM
If you like it, use it and give me feedback. If you don't, I don't care. It boggles my mind that there is not one widely-supported model format that simply stores the data the way you need to store and upload it to the GPU; I don't believe in the idea that we are all doing such different things that no model format can be shared; We're all just pushing matrices and uploading vertex arrays, and I purposely design the vertex array format so it could be suited for any use.

Believe me, if there was another file format like this out there I would have gladly used it!

01-31-2008, 05:31 PM
It boggles my mind that there is not one widely-supported model format that simply stores the data the way you need to store and upload it to the GPU

That's because there's no one "way you need to store it". There are a number of different ways to store data, and the particular situation determines how this data gets stored.

We're all just pushing matrices and uploading vertex arrays

Except that some of us want to be able to change texture coordinates by substituting one set of texture coordinate mappings with another. Or a different color array (for having different sides in an RTS). Or the z-component of the position.

The number of bones that the user expects to upload for rendering a piece of a mesh is an implicit part of the model format, because it must inform the process of creating that mesh. If your shader can only index 32 bones, but you use 64 in your mesh creation, you have a broken mesh.

We are not all just doing the same thing. Not everyone is doing something as simple as "push matrix, upload some verts."

01-31-2008, 05:45 PM
You can store up to 255 bones if you use bytes for the variable type, and if you really wanted to you could use words or integers.

Most engines will support much fewer than 255 bones. In my own engine, I just have an error message pop up if the number of bones exceeds the engine's limit, and I just tell my users they need to keep it under a certain threshold.

01-31-2008, 06:46 PM
Most engines will support much fewer than 255 bones.

Actually, most engines of note will take whatever the shader-defined matrix limits are and break your skinned mesh up into bit-sized pieces that use no more than that number. At render time, it then puts these pieces back together.

01-31-2008, 11:24 PM
What about the Ogre format? It is probably alright.
For a while I was working on taking their code and modifying it to my liking but I suddenly stopped.

02-01-2008, 10:37 AM
Is their format documented? I think I looked at it a while ago and couldn't find a spec.

I think I will move things like material name and limb names into a custom properties block. It would just be a list of keynames and values, like this:


I think the really important data is the matrix hierarchy, mesh data, animations, and weighting. Everything else is just fluff that may vary depending on the implementation.

Also going to replace the position/quat stuff with local 4x4 matrices, because I think that is less ambiguous.

I'll make the exporter sources open and free so people can modify and add to them as they see fit. As it is now, you can have things like multiple texcoord sets and you can expand the format without breaking existing loaders.

The thing I have had problems with was getting animated skinned mesh data from modeling packages to my engine, so I think if that core functionality is provided, this can help a lot of people. Then they can modify the exporters however they see fit to add their own functionality.

02-01-2008, 11:22 AM

Good for you, I agree collada isn't straight forward, and frequently you need something that "just works". I looked at your spec, and quickly decided it wasn't for me,I need more supporting documetation such as quick referances, example code, etc... to me there is no payoff for using what you did. Don't get me wrong, I would probably use it if it had good documentation or looked like it would "just work" for me.
In summary, from an outsiders perspective, your spec and collada are the same, no friendly documentation. Collada is more advance but in the end it, the only good tools for it seem to be a pay to play solution. Don't let the flametards get to you, remeber the people running collada are now running opengl, for better or worse.

All hail the might KRONOS.

02-01-2008, 12:51 PM
Thanks for the feedback. I am still fiddling around with this, but I will get some example code up in the next couple weeks. I think it needs an OpenGL example that loads the model and displays animation.

Collada is an interchange format, so I don't consider the two formats to be related. In fact a Collada -> GMF converter might be a good idea, as it would allow support for any modeling package that properly exports a Collada file.

When I brought up the idea of a game-ready format on the Collada forum, the attitude seemed to be that everyone on the planet requires their own unique file format, and needs to write their own exporters. I don't buy that. I can at least provide a good easy-to-read animated file format, along with plugin and converter source code so the format can be extended according to one's own needs.

02-01-2008, 03:04 PM
I don't buy that.

Whether you buy it or not, it's basically true. I've given examples of the variety of stuff that some users will want that others will consider (at best) cruft.

Now, if you want a "hobbiest" format like .X files, which are useful for getting something up and running, but useless in professional code, that's one thing. A good hobbiest toolpath going from Collada to a game mesh loader/renderer wouldn't be a bad idea.

02-01-2008, 04:29 PM
You sure make a lot of effort to subtly disparage my work. You are quick to label it as a "hobbyist" project, and as "useless for professional code", and you promote nonsense like the idea of a game using more than 255 bones in a model, as if you are the expert on all these subjects. Then you quickly take the opportunity to promote Collada for my clearly inferior format, since I don't have a 100+ page spec or a bloated SDK that gets completely rewritten every six months. At the same time, I have made no comment on Collada other than agreeing it was an interchange format, and was not designed for loading directly into a game.

Thank you for helping me to see the error of my ways, I was under the impression that loading vertex data straight into memory and to the GPU would be faster, but clearly it would be better to do things your way.

Thank you for barging in and derailing this discussion you claim to have no interest in. If you truly had no interest in this, I think you would just not bother to comment so many times. I am not selling anything or forcing you to do things my way, I am only asking for ideas and putting out some free tools, which people may take or leave as they see fit.

It took a lot of work to implement this, and I thought it might be helpful to others, but clearly that idea does not sit well with you.

02-01-2008, 05:40 PM
I've read your spec, and I actually think it's quite nice that someone's formalised something similar to our in-house file format. If you produce a max and maya exporter I'll add support for it to my renderer.
Just a little criticism, you should support interleaved vertex formats somehow, I wouldn't want to be interleaving every load.

02-01-2008, 06:09 PM
That's a good idea. I think I can add interleaved data without complicating things too much.

02-01-2008, 07:00 PM
Yes, you'd probably just introduce another sub-chunk type to the vertexarray chunk in your system. Something like this:



long ;data type (GMF_POSITION, GMF_NORMAL,
long ;number of elements
long ;offset in bytes from data origin

long ;number of vertices
long ;vertex stride
data[n] ;vertex data.

You'll need the vertex stride to support padding schemes, and maybe if it's 0 the stride would default to the total byte footprint of all attribs.

02-01-2008, 07:06 PM
You could add vertex attribute name, so you can easy "link" array with proper attribute in vertex shader (if names match).

02-01-2008, 07:12 PM
if it's to be extensible, yes, I think the attribs should have actual names rather than being a fixed documented constant. Another chunk could be added to the start mapping attrib names to numbers, then only the number need be used for each vertexarray to save space.
Easy to see how these things get complicated when you try and be all things to all men.
BTW, how do I get this forum to always not add my bloody signature to every post without having to hit the checkbox every time?

02-01-2008, 09:43 PM
Is their format documented? I think I looked at it a while ago and couldn't find a spec.

I asked on their forum but the answer was no so that's why I was trying to understand their code.
I settled for taking my own format and hacked in the support and just changed the file extension.
I have to admit there is a big lack of good cross platform file and export, ...

02-02-2008, 06:34 PM
I am thinking something like this:

Int - number of interleaved data types (1 for non-interleaved arrays)

Then for each interleaved data type:
int - data type (GMF_POSITION, NORMAL, etc.)
int - variable type (GMF_FLOAT, etc.)
int - number of elements

Then the data.

So for non-interleaved arrays, this just adds a single integer "1" into the format.

02-02-2008, 11:03 PM
I have predefined vertex formats like

All components are floats.
V = means 3D vertex
N = 3D normal
T = 2D texcoord
T3 = 3D texcoord

I suggest that you create a few predefined types.
If the user wants to have his own format, he can use some #define starting from some high value.

02-03-2008, 10:21 AM
Why do you guys use interleaved data? I was under the impression that this was non-optimal and/or being phased out.

02-03-2008, 10:54 AM
Why do you think that?

02-03-2008, 11:19 AM
I guess I was thinking of when using shadow passes, how you don't need all the vertex arrays. I think that was why I used separate arrays, plus I find them more intuitive.

02-03-2008, 04:20 PM
Why do you guys use interleaved data? I was under the impression that this was non-optimal and/or being phased out.
no its the opposite interleaved is more optimal, though from my testing not by much just a couple of percentage points

02-03-2008, 06:10 PM
for me I got about 3% improvement with interleaving the v3f,n3f,t2f attribute format, probably because each vertex is then 32 byte aligned.

02-04-2008, 10:08 AM
IIRC, I read in some NVidia papers that you should enable all vertex client states no matter is your vertex shader use it or not. Driver is smart enough to not fetch unused attributes.

Generally, Im splitting attributes in static and dynamic data. Static attributes are uploaded once and dynamic frequently. In my character renering code, Im doing skinning on CPU and transfer only skinned (dynamic) data. Texture coordinates and vertrex color are static, but position, normal, tangent and binormal are dynamic and transfered after their update. At the first time, I did complete hardware skinning, but in multipass rendering, hardware have to skin same model several times, which is not optimal.

Today, I can switch to complete hardware skinnig using latest features (transform feedback) but Im not sure is it implemented on ATI. Another option is to use pixel shader to do vertex processing job and use MRT (4x RGB-f32 rendering targets) to calculate Position, T, B and N in four different textures. Later I can readback texture data into PBO and rebind PBO as VBO, setup vertex attributes pointers and start series of draw calls.

02-04-2008, 11:45 AM
IIRC, I read in some NVidia papers that you should enable all vertex client states no matter is your vertex shader use it or not. Driver is smart enough to not fetch unused attributes.

The driver code behind glEnableClientState and gl****Pointer knows when you have interlaced. If you call glDisableClientState(GL_NORMAL_ARRAY), it will not care. Normals are still read into the cache because that's how the cache circuit works. Also, the bus is wide enough to read several interlaced vertices into the cache at once.

02-06-2008, 06:31 AM

I don't think it's a good idea to use long for the type, but rather you should use int. As I think on 64bit cpu's long == 64bit, but on 32bit systems long == 32bit.

Hmm, well I think this is so anyhows.