New texture compression methods and format

Texutre compression methods atm are very simple and can’t compress much the data. S3’s DXTn is good but compress like 4:1 ratio… Almost all cards now can help decompressing MPEG2. Could be possible to support something like JPG/JPG2000/WMP textures or to make a new serious-high-compression algorithm by hardware? ( for example wavelets and decompress a pixel using dot-products ). I think that will be good because:

  1. Less VRAM will be required by textures, so graphic cards will be cheaper. I don’t know you but I am not able to pay 900$ for 1Gb VRAM and games are using tons and tons of normal maps and textures and will use even more. For example, if you attended the N. Tatarchuk’s presentation in the GDC about the toy ATI demo, she said that demo would be -impossible- to do without 3Dc compression, because the texture could occupy more than 1Gb… and is only a small demo, not a complete game level…

  2. These kind of graphic cards could compress/decompress by hardware our JPG/JPEG2000/WMP photos so we will save CPU cycles. After all as I said they are already helping MPEG2 decompression so…

I think will be possible ( and not saying optimal ) to uncompress/cache the JPG lines used by the current triangle using the
jpeg_read_scanlines() function ( fron the IJP JPG free library )

We should try to define a new and much more powerful texture compression algorithm for realtime graphics based on Haar-vq-wavelets so a random pixel will be uncompressed using a few dot products.

And I finish with a thought… If the photographic companies like Kodak, Ricoh, Nikkon, etc meet and define image compression formats(say,JPG)… why the game industry can’t meet and define a good compressed, mip/clipmap, tiled, animated and royalty-free texture format? So far we got is the Microsoft DirectX Texture format … and is propietary and blah blah

So, to resume… I want to define a new texture format standard which supports all the mentioned features(tiling,mipmaps,cubemaps,animation,etc) with a new texture compression algorithm that could reach JPG-JPG2000-WMP ratios and, ofc, an API to read/write/manage this new format. This new format and features must have the corresponding HW-support ( compress/decompress in realtime, cubemaps, etc… )

thx

1)partially true, but only to some degree, and most of this can be solved if game developers only wrote a good solid texture manager, like they do for consoles.

  1. the trick is to to that without affecting GPU speed, CPU speed is not affected any texture compression algorithms, save for the transfer of the texture data that is, but that is marginal and only done once(per texture loaded), but GPU decompression is done five times per fragment, that’s where you want to optimize things.

About the Windows Media Photo format, it’s a lossy format and thus no good for things like normal textures and such.
Besides it’s a microsoft beast, and probably wrapped in layer after layer of patents.

I think some extra compression could be exposed, after all, are we bandwidth limited or not? Trading GPU cycles for bandwidth seems like a good idea for me but I hardly believe this is going to happen any time soon.
There won’t be a glLoadJpeg by sure however.

Texutre compression methods atm are very simple and can’t compress much the data.
You misunderstand what texture compression is.

Texture compression means that the texture is stored in videoram compressed. S3TC/DXTC is a compression format that is designed to work in that environment. It guarentees a specific texture size, and for any random pixel you may want to access, you can get the location of the block that contains it very easily.

JPEG can’t do that. Indeed, as I understand it, if you want a particular pixel of a JPEG image, you need to decompress the entire thing. Which, of course, defeats the purpose of texture compression.

Now, that’s not to say that we shouldn’t have more and/or better compressed texture formats. But they won’t be common image formats; they must be specially designed formats.

See, for example, the Windows Media Photo new format in
Please refrain from shilling Microsoft products where inappropriate.

Originally posted by zeoverlord:
About the Windows Media Photo format, it’s a lossy format and thus no good for things like normal textures and such.
Besides it’s a microsoft beast, and probably wrapped in layer after layer of patents. [/QB]
In http://www.microsoft.com/whdc/xps/wmphoto.mspx you can see it says

Originally posted by Korval:
JPEG can’t do that. Indeed, as I understand it, if you want a particular pixel of a JPEG image, you need to decompress the entire thing. Which, of course, defeats the purpose of texture compression

Well I see a jpeg_read_scanlines() function in the Independent Joint JPEG Group C++ SDK library… If you see a few the sources you can see it appears is possible to encode/decode data in NxN blocks of the DCT or even by scanlines or am I wrong?

Prolly you are right though JPG is not the best format to store in VRAM the texture but was only an example…

Well, to conclude, I think we need to define new and much powerful compression formats for textures… For example one based in haar-wavelets like I said before so any pixel could be decompressed easy and fast using a few dot products. In fact, some PRT texture methods start to use this to compress the radiance per pixel.

Also we could use psycho-accustic(sorry i dont know how to translate this) methods like the MPEG3-Audio does but applyed to the texture colors, etc…

Originally posted by Obli:
I think some extra compression could be exposed, after all, are we bandwidth limited or not? Trading GPU cycles for bandwidth seems like a good idea for me but I hardly believe this is going to happen any time soon.
There won’t be a glLoadJpeg by sure however.

Memory bandwidth limited, not the PCI-E bus bandwith(or at least should not be).
With a higher memory bandwith you can have more texture reads, compression does not change this since you allways have to read from the memory, compressed or not, infact compression might even use up more bandwidth.

Originally posted by santyhammer:
[b] In http://www.microsoft.com/whdc/xps/wmphoto.mspx you can see it says

• Fixed or floating point high dynamic range image encoding

• Lossless or high quality lossy compression

However you probably are right about the patent part, etc...
[/b]

As far as i can tell wmp is a container format + a few compression algoritms.
But even jpeg can do “high quality lossy compression” it’s just a matter of how large the file will become.

What’s wrong with S3TC/DXTC?

4:1/8:1 ain’t good enough for ya? :wink:

On a personal note, I’m actually quite fond of it (I shill you not :-)).

P.S. What I like is having a single file format that has decent compression, stores 2d, 3d, cubes and mipmaps, is easy to work with and hardware ready straight from the disk. If they were to do something internally to increase the compression ratio, that’s fine, as long as it doesn’t add complexity or delays to my side of things. Oh, but then it’s just an implementation detail :wink:

P.P.S. I’d like to formally request a new set of forum gremlins.

Well I see a jpeg_read_scanlines() function in the Independent Joint JPEG Group C++ SDK library…
The first step in texturing is to turn the texel coordinate, in texture space, into a memory address into the compressed texture. I don’t see a function in the jpeg API that does that.

Remember: texture compression works because the textures are stored and used compressed. The hardware may decompress texels (as they are fetched) into the texture cache, so that the texture unit doesn’t have to know how to decompress them. As such, the blocks have to be small. They cannot, under any circumstances, be scanlines.

And you need to be able to random-access without having to decompress the previous data.

why the game industry can’t meet and define a good compressed, mip/clipmap, tiled, animated and royalty-free texture format?
What does tiling and animation have to do with a texture format? Textures, by definition, are static images; they should not be animated.

4:1/8:1 ain’t good enough for ya? :wink:
It’s OK, if you’re not using floating-point textures. But it does make using FP textures even more expensive by comparison.

What we don’t have is a good mechanism for compression FP textures. And I’m not sure if such a mechanism even exists that works at all the way a texture unit wants to work.

DXT5 is not very good for normal maps. A thing like 3Dc or the upcoming DX10 BC1-5 formats are much better. Or, for example, the new Humus compression format.

On the other hand a ratio of 4:1 is pretty bad compared with JPEG2000 and WMP. Again ATI toy demo could wipe any graphic card without 3Dc. S3/DXT is not enough. As Korval said DXT and fp formats are not good.

Let me put you some examples why we need tiling and animation.

Tiling. Imagine you want to use a custom re-cubemaper like used in NVIDIA Mike’s demo or as described in ShaderX3. This texture is big, something like 3k x 3k ( 6 cube face of 1024 in a 3x3 ). Now imagine I wanna map an object using only the face X+ … With a tiling system the texture could be loaded from system memory to AGP and then used. Without tiling the entire texture should be uploaded again to VRAM. Texture tiling-in-memory will be good to render-to-cubemap and update only the affected faces.

Other example of tiling can be a 32k x 32k terrain like is going to be used in Quake Wars ( aka Megatexture ). To manage this “texture atlas” you need a tiling system to edit and manage each sub-texture or your graphic card and RAM will wipe.

And other… Imagine you can’t really decompress only a pixel in a JPG. Then with the “tiling” you could divide the texture in 32x32 tiles and uncompress & cache completely the portion of the texture the triangle texels use.

Texture animation. Have you played Everquest? In this game torches and other fire and particles effects uses like 20 different textures as frames ( like fire1_frame1.tga, fire1_frame2.tga ). This is bad because you need to interpolate between frames to achieve a smooth effect. Putting fast different frames won’t show a good effect. We need some way to interpolate and compress texture playback changes in an effective way ( like a MPEG2 but for textures ). On the other hand current OpenGL implementation don’t describe a method to use “video textures”. The common way to achieve this is to decompress to a dynamic texture the AVI frame and then use fragment shaders to interpolate. With video textures you can achieve very good effects like described, for example, here:

      [http://www.psxextreme.com/scripts/previews2/preview.asp?PrevID=285](http://www.psxextreme.com/scripts/previews2/preview.asp?PrevID=285)          

I think is much better to have all the data stored in a file like runAnimation.tex than 200 files like runAnimation_frame1.tga, runAnimation_frame2.tga blah blah… So thats why I ask to include animation frames in our texture format, just to avoid to use 200 different files… Is much more easy to do

   OGLTexture tex;
   tex.Load(L"fire.tex");
   OGLTexture_Frame frame = tex.GetFrame(18);

You can think each frame is part of a 3D volume texture too, so this could be used to store 3D texture slices, etc…

Animation can be just a syntax-sugar for an atlas.

The Texture SDK could perform some funcitonality to interpolate, blit, scale, atlas, generate mipmaps, etc… using software like the GLU/D3DX library. The driver should ONLY add support for HW-compression/decompression, render-to-cubemap, etc…

So what I ask is a thing like the OpenEXR but extended to textures and not only to images. A format and a SDK to simplify all the atlas, animation, compression, tiling, hdr-fp, etc… with the appropiate hw support. DirectX incorporates some of these things but lacks others.

For the (texture) memory part, I think the future is in procedural texture/geometry generation. The memory battle is a lost cause (long term), and raw content will never really scale with available memory and compression algorithms unless more of it is generated on the fly.

For animations, I’d use a texture atlas. Heck, a simple interpolation isn’t that big a deal, is it?

Other example of tiling can be a 32k x 32k terrain like is going to use in Quake Wars ( aka Megatexture ). To manage this “texture atlas” you need a tiling system to edit and manage each sub-texture or your graphic card and RAM will wipe.
That doesn’t explain what it has to do with actual texture formats. If GL as it stands gives you the tools to do this, so be it. And if Quake Wars is doing it, then it does give them the tools to do it. No need for the driver to even know it’s happenning.

Equally importantly, the handling of so-called “Megatextures” is an issue that can be resolved with a caching system that doesn’t require anyone to change their GL code. All it is is a hardware and driver change, and the allocation of a really big texture.

Either way, there is no need for a new, special kind of texture or texture definition.

Have you played Everquest?
No. I don’t play crappy games.

In this game torches and other fire and particles effects uses like 20 different textures as frames ( like fire1_frame1.tga, fire1_frame2.tga ). This is bad because you need to interpolate between frames to achieve a smooth effect.
Like I said, crappy game. There are ways to solve this problem that don’t involve demanding texture formats from the driver; indeed most reasonable games actually implement them.

You can just have frames of animation all in one texture and switch them by bumping the texture coordinates in the vertex shader. This is so simple, I probably could have implemented it in the time it took to write this.

And if you need to interpolate, feel free to throw that into the shader as well. I don’t see what the need is for having a driver-defined feature for this.

Or, maybe you use a volume texture with 3DTC, and get the exact same results. It makes no difference; the tools are already there. That some crappy programmers at Verant/Sony have no idea what they’re doing is really irrelevant.

I think the future is in procedural texture/geometry generation.
We keep hearing this year after year. But it never really happens. Not in the general case.

Game development is going to proceed like CG movie development. And they don’t throw procedural textures onto everything. They’ll be used where needed, and that’s all.

Trying to texture a terrain with a 32x32K texture just strikes me as really pushing it. It’s clever that it can be done, but what about 64x64, 256x256?

Where the procedural stuff is needed is in terrain rendering–for stuff that’s organic in nature. The man-made stuff, on the other hand, is another matter entirely; but I still think it could benefit from some clever procedural thinking down the road.

The problem right now is that the hardware just isn’t quite there for an all out procedural assault (but it will be).

I don’t play crappy games.
That bad? Never played it myself, but I’ve met some folks that would sooner be skinned alive than give up playing the game. It seems to be a role playing game syndrome (those people always scared me a little) :wink:

Other example of tiling can be a 32k x 32k terrain like is going to be used in Quake Wars ( aka Megatexture ).
The other thing about this megatexture stuff: 32x32K is puny for a terrain, even by the most conservative estimates. Obviously for Quake stuff, it’s fine–their worlds have been around that size since the beginning. I’m sure it’ll look great, but I just can’t see it scaling much beyond that.

Originally posted by zeoverlord:
[quote]Originally posted by Obli:
I think some extra compression could be exposed, after all, are we bandwidth limited or not? Trading GPU cycles for bandwidth seems like a good idea for me but I hardly believe this is going to happen any time soon.
There won’t be a glLoadJpeg by sure however.

Memory bandwidth limited, not the PCI-E bus bandwith(or at least should not be).[/QUOTE]With memory speeds growing linearly and GPU processing rates super-linearly, I don’t see how this could make a difference. Maybe this generation is not because often fragment-processing bound but sooner or later it’ll be. Replacing communication with computation seems a long tradition to me.

Originally posted by zeoverlord:
With a higher memory bandwith you can have more texture reads, compression does not change this since you allways have to read from the memory, compressed or not, infact compression might even use up more bandwidth.
I don’t understand. If so, why should DXTC be used? In the case of DXTC, as far as I know, the texels ARE compressed and interpolated somewhere (say, inside texture lookups calls). This trades the (slow) communication to external memory for (fast) GPU internal computation.
How this, or another similar trade, could increase memory bandwidth consumation takes me to badly designed access patterns killing the caches really badly or other arcane ways. I cannot realize other drawback and I would like to know how this could happen.

Saving ‘connection’ bandwidth is less interesting to me, although it has some potential applications, I’m not opening this can of worms.

Originally posted by santyhammer:
Well I see a jpeg_read_scanlines() function in the Independent Joint JPEG Group C++ SDK library… If you see a few the sources you can see it appears is possible to encode/decode data in NxN blocks of the DCT or even by scanlines or am I wrong?
That’s just an API function for that library, obviously aimed at making use of JPEG files easy. That doesn’t mean that JPEGs are easily decoded to scanlines, in fact, it isn’t, it’s just that the library takes care of that (probably just decoding as much as needed and caching some results for future scanline requests). JPEG files are encoded in 8x8 blocks, but these blocks are variable length and tightly packed. Given a coordinate you can’t just go grab the pixel, you have to if not decode everything at least parse the file to the block you’re looking for.

Originally posted by santyhammer:
Also we could use psycho-accustic(sorry i dont know how to translate this) methods like the MPEG3-Audio does but applyed to the texture colors, etc…
I think the word psycho-visual is what you’re looking for. I’d say that’s what I did in my latest demo . You don’t neccesarily need more hardware, but you can often take advantage of existing formats and use some math.

Originally posted by zeoverlord:
With a higher memory bandwith you can have more texture reads, compression does not change this since you allways have to read from the memory, compressed or not, infact compression might even use up more bandwidth.
Certainly not the case for the DXT or 3Dc formats. The performance increase comes primarily from the memory bandwidth you save by using these formats.

Originally posted by Leghorn:
[b] What’s wrong with S3TC/DXTC?

4:1/8:1 ain’t good enough for ya? ;-)[/b]
Nothing wrong with the compression rate. Sometimes the quality isn’t good enough though.

Originally posted by Korval:
[QUOTE]What we don’t have is a good mechanism for compression FP textures. And I’m not sure if such a mechanism even exists that works at all the way a texture unit wants to work.
There are some new FP formats in DX10, but I can’t recall if there are any compressed such formats. But you could always use L16 + DXT1 and decode with three instructions in the shader.

Sometimes the quality isn’t good enough though.
You mean like for normals? I’d agree. But that’s yet another argument for the procedural case :wink:

P.S. I really hate having to fool with normal maps :frowning: