High resolution image flipbook animation APIs/toolkits?

I already put this up on the beginner’s list with no real responses.

Say, “for theory”, that I have a large collection of time-series imagery, 16 bit depth FITS images, 16 megapixels each.

-requirements:
->= 20 frames per second minimum framerate
-adjustable color mapping
-adjustable alpha
-rotatable

Eventually, I might stack and separate several “layers” with alpha, and maybe do additional things like finer grained textured polygon meshes, …

I’ve worked in OpenGL and Performer in the past (been a while), so I want to hit the ground running and I’m hoping to find a toolkit/API for doing this type of thing:

autoconfig for # CPUs and gfx pipe(s)
specify (FITS) image set

Loop:
prefetch images to keep pipeline filled
(separate thread/proc)
do whatever processing needed
render image
swap buffers

Sooo…

Anybody have any APIs/toolkits or sample apps to do this already?

For doing the rendering and manip/motion/rotations, I intended to just take the images and map as a texture onto a surface.
Eventually, there might be multiple images/layers which are separated by some adjustable value, and possibly connective geometry between the “layers”.

In some ways, this is similar to some of the old space to face zoom demos by SGI and others with dynamic texture paging, but I believe that my bandwidth requirements are a wee bit higher and continuous.

Think about your bandwidth problems first!
20 fps at 16 megapixels in 16 bit…
In uncompressed raw format you will find no medium to actually load all that data, not to mention displaying 16 megapixels resolution at once…

You will have to lower your input resolution to do anything in 20fps.
There is a lot of FITS code (loading, processing, etc) around I think… the OpenGL stuff should be straight forward, pixelshaders for processing and displaying the data.

While I don’t know what FITS is, at a required 640MB/s sustained data upload speed… I’m not surprised noone answered in the beginners section. Your h/w alone must be very, very expensive, and therefore very rare.

I’m not aware of any OpenGL output device (gfx card) able to display 4096x4096 pixels, and even if there was one, I’m not aware of any visualization devices (monitor, projector etc) able to do this - except, possibly the very, very highest end (f.ex. digital cinema projectors). But then, I don’t see how 16bpp would be enough.

Disregarding my ignorance of such highest-end-and-beyond hardware, at 640MB/s you either have the majority of data already in RAM (27 minutes of data/TB of RAM by my calculations), why uploading that to alternate textures shouldn’t be a problem, or you have an insanely fast I/O subsystem (you most certainly isn’t using x86 ). This kind of hardware isn’t exactly common, why I think your best bet finding libraries for it would be to contact the few organizations or companies that might have already done something similar.

++luck;
(it would be interesting to know what kind of h/w you’re using and have considered for something like this)

I am largely aware of the bandwidth issues involved, so that’s why I’m inquiring.

As for the resolution of 16 megapixels being displayed, that is the point of using texture mapped polygons, to allow for the hardware to do the scaling/transformations to display on whatever res display you happen to be on. Also, these images will be rotated and translated in 3D, rather than just a flat on 2D projection.

FITS is a standard astronomy/science data format. A bit old/dated in some ways, but in common use throughout the field. I am much less concerned about loading a FITS file than the throughput, pipelining and rendering issues, as there are FITS loaders aplenty.

I strongly prefer to load straight FITS files and map to internal texture format suitable for OpenGL rather than doing a preprocessing conversion step.

A lot of these files can be preloaded/mapped to memory. The machines in question will have 8GB or more memory, so at 32MB per file, that comes out to 250+ frames. We have some machines with a lot more memory, but they don’t have the fast gfx pipes.

I’m supposing that one of the better things to do would be to have the app query the display resolution and on loading, downsample the textures to suit the display being used (we are using some 9 megapixel TFTs and some 8 megapixel projectors and lots of other displays too). The downside to this is that zooming in will lose fidelity unless I employ LODs, but that will certainly incur more memory and CPU costs itself.

Mipmaps consume additional texture memory as well.

Faster machines now please.

While I don’t know of any existing code or libraries to do what you need, is texture compression an option?

Assuming S3TC (and my understanding of it is correct) this could save you 75% memory and bandwidth for the 16bpp textures. For 32bpp textures the savings are 1:8. Even if you add mipmaps to the compressed data the savings would be 1:3 or 1:6 for 16/32bpp respectively.

This compression could possibly even be done in near realtime at loadtime by even software, but likely more cost effective using dedicated gfx h/w (incl. the option of having a number COTS PCs doing it).

I don’t know what you’re trying to accomplish specifically, but if your textures are smaller than 4096 x 4096 that can make things easier. If they’re larger than that you’re going to have to chop them up into tiles. To get the size down you could use texture compression. I use DDS compression (google it) in some apps. Works like a charm. nVidia has a DDS SDK and ATI has a Windows app called the compressinator to make DDS files from images. This sounds like a relatively straightforward app to implement. I’d start with a basic glut framework and build something out of that. Quick for prototyping.

If you really need a “toolkit” I’d suggest looking into OpenScenegraph (AKA OSG) http://www.openscenegraph.org

OSG has a large following and you can also easily find that know OSG that can offer advice. There are also many people on the OSG mailing list that offer contracting services if you’re so inclined.

I’d suggest you start with a simple glut app and hack something together yourself though.

Something that just struck me (I hope this isn’t OT for OP):

Is anyone aware of a combination of texture compression and a video en/decoder? Something that could run (at least) at roughly the same CPU cycle count as a plain video decoder, but as the output data would be 6 times smaller (assuming 24bpp is the “normal” output, even that I know video usually uses YUV) than uncompressed output I think at least four times as fast as uncompressed data output wouldn’t be too much to ask for.

Furthermore, isn’t it that quite a few of the gfx chips of today (at least from “the large two”, but possibly most or even all nowadays?) already contain hardware MPEG (1/2, maybe even parts of 4?) decoders? If so, wouldn’t it make sense to be able to make use of this already existing functionality by uploading MPEG streams (with a compression ratio 100:1-200:1 !) from host to server, have it decompress the frames, and then put the result in a specified RGB texture?

Should that be possible, OP could store 20-40 minutes instead of 12.5 seconds of streaming video in the same amount of RAM, and reduce bandwidth demands by a factor 100-200!

/me thinks this could warrant a new thread.

Lossless texture compression is certainly an option, or perhaps lossy at a very low loss rate.

I’m not too worried about the basics of writing the app and getting it to render. I’m concerned mostly about the bandwidth and throughput. Perhaps I was a bit too tongue in cheek when I said “a wee bit higher” on the bandwidth requirements.

Anyhow, I’m just going to plow through it and see what I come up with.

Probably have a pipeline of a texture loader thread to prefetch as much as possible.
Image conversion and downsampling and saving to an intermediate format look almost necessary due to the large amount of CPU/time that consumes on its own.

A couple more restraints being set (of course are:
-portability (OS/X, Linux, Windows and possibly IRIX)
-9000+ frames of playback (ugh!

BTW- The images are 4096x4096, but some border can be cropped.

Just be glad we’re not using 84 megapixel CCDs (yet…

Ok.
It’s been a while, and I’ve been a busy bee and I knocked out an application prototype and it works great (even on relatively modest hardware) to an extent. My app currently easily allows panning (time and space), zooming, scaling, transforming, blending, … and works great, as long as you can fit everything in RAM

I’m working on adding features and revising it to handle all of the new features I’ve been asked to add.

Currently, I can load a bunch of channels of data and I use alpha to blend the channels together. I know that this is not the right way of doing this and I’m currently designing a fragment shader to do proper blending of data channels.

Also, since nobody currently makes an affordable machine with 4 - 10 terabytes of RAM, I am also making a system for streaming the data in, since with the amount of data we are talking about, you very quickly overwhelm any computer’s RAM resources.
I keep mentioning to various computer manufacturers how nice it would be to have a computer with 10Tb of RAM, mostly, but not wholly, in jest, and I get lots of laughs. Seriously, it would be awesome to have that though
Daily data volume is going to be about 4+ Tb, every day the Sun shines in space…

A bit more detail:
The application is to play time series image data (4Kx4K: 4096x4096) as movies at 20Hz or faster. These images are the result of blending one or more channels (up to 8) of 16 megapixel 16 bit grayscale images together.

16 megapixels @ 16bpp per channel --> 32 megabytes/image/channel
16Mpixel @ 30 Hz --> 960 Mbytes/second/channel
16Mpixel @ 20 Hz --> 640 Mbytes/second/channel

And that is PER channel, and we have up to 10 channels…
The data rate is unbelievably high, even for just one channel, and we will usually be visualizing at least 3 or 4 channels simultaneously (2.88 - 3.84 Gbytes/second), so here’s how I plan to cheat:

-have a set of spatially subsampled 1Kx1K @ 8bpp images that can be streamed much more feasibly:
1024x1024 * 8bpp @ 30Hz --> 30 megabytes/second/channel
1024x1024 * 8bpp @ 20Hz --> 20 megabytes/second/channel

When the user pauses the movie, the application will fetch the 4Kx4K images for each channel and display those instead of the 1Kx1K.

For 4Kx4K display, I am thinking about using some Toshiba 8 megapixel LCD displays, if they can ever get them to work in OS/X and Linux.
We are considering using the CGLX cluster rendering toolkit.

The fragment shader that I am working on will be used for performing various arithmetic blending operations between data channels and for colorization of the final image.
I also want to support colormaps via a fragment shader. Ie, upload a colormap as a texture and based on the blended value at a fragment, index into the colormap texture to look up the final fragment color.

Sounds like you would benefit from parallel graphic clustering.
Things like :
http://chromium.sourceforge.net/ (this one seems too old)
http://www.equalizergraphics.com/ (this one looks good, with sort-first and sort-last load balancing)
http://www.equalizergraphics.com/gallery.html

yep.
That is the plan.
CGLX is a clustered rendering toolkit that also provides much of what is most commonly needed out of GLUT.
Even so, there are still very serious bottlenecks

I forgot to include a link for CGLX:

http://vis.ucsd.edu/~cglx/

Have you tried to use some of compressed formats, like http://developer.nvidia.com/object/real-time-ycocg-dxt-compression.html

I thing idSoftware use similar tech for their upcoming game with Megatexture engine.

I’ve been thinking about using some native as well as other compressed formats (JPEG 2000, YUV, MPEG 4, …), as long as the hardware could assist in the decode and the image fidelity is high.

I read the link you provided, but it is talking about 4:1 compression, which will not be nearly adequate for streaming in data from disk, at least at full resolution.

One of the ideas behind the application I am developing is to look for features and phenomena in the data, so lossy compression could be bad, depending upon the amount of artifacts that it generates.

The daily volume of data is going to be about 1200+ gigapixels a day, every day.

DirectX 11 hw is going to have texture compression for floating point formats. So if you wait for the hardware, and then some more years for Khronos to standardize it in OpenGL, you might save some bandwith.