Streaming several (YUV) videos using OpenGL

imported_sampsa · January 23, 2018, 7:28am

Hi,

I’m trying to stream video using OpenGL textures. Some features of my programming architecture are:

[ul]
[li] Direct GPU memory access using PBOs - works great [/li][li] I am using a stack of pre-reserved PBO that are being recycled - works great [/li][li] Textures present LUMA and CHROMA planes that given to a shader program. Shader interpolates from YUV to RGB - works nicely [/li][/ul]

However, there is an bottleneck that is driving me nuts when I’m copying from PBO to textures.

I have traced this issue into the format/internal_format pair in

glTexImage2D(GL_TEXTURE_2D, 0, internal_format, w, h, 0, format, GL_UNSIGNED_BYTE, 0)

As we know, OpenGL converts everything to RGBA. The documentation states that:

glTexImage2D - OpenGL 4 Reference Pages : “GL_RED : each element is a single red component. The GL converts it to floating point and assembles it into an RGBA element by attaching 0 for green etc.”

My problem is that I don’t want RGB-whatever, just single channel …! The “texture” here is just a blob of bytes that is given to the shader program. I don’t need any conversion.

The only format/internal_format pair that gives me decent results, is GL_RGBA/GL_RGBA (but I can’t use that).

My case would be GL_RED/GL_RED … but that sucks big time. With that pair, “glTexSubImage2D” is hundred times slower than with GL_RGBA/GL_RGBA.

So, I tried to drop PBOs alltogether and start using TBOs (texture buffer objects) … with TBOs there is no conversion - they are more like plain byte buffers, right?

However, TBOs don’t give be dma to the GPU (this works with PBOs). This one:

payload = (GLubyte*)glMapBuffer(GL_TEXTURE_BUFFER, GL_WRITE_ONLY)

gives me a null pointer.

I am starting to run out of ideas - please help!

A small stand-alone test program can be found here … it just benchmarks texture uploading (does not visualize anything):

https://github.com/elsampsa/opengl-texture-streaming
Regards,

Sampsa

P. S. A related stack overflow question is here: c++ - Streaming several (YUV) videos using OpenGL - Stack Overflow

GClements · January 23, 2018, 2:50pm

If format and type match internalFormat, there shouldn’t be any conversion.

The “conversion” to floating-point RGBA is only conceptual. It allows the internal and external formats to be discussed separately, rather than needing to discuss each possible combination of format and internalFormat.

Conceptually, the external format is converted to floating-point RGBA (converting signed or unsigned normalised values to floats, setting missing components to their default values), then that is converted to the internal format (converting floats to normalised values, discarding unused components). But I wouldn’t expect it to actually convert to/from float if neither the internal nor external format is float, nor add components only to discard them.

If you’re using 3.2+ core profile, you can’t use unsized internal formats; you need to use e.g. GL_R8 or GL_RGBA8.

glTexSubImage2D doesn’t have an internalFormat parameter; was that a typo for glTexImage2D?

Does it make a difference if you use GL_R8 rather than GL_RED? (I’m assuming that type is GL_UNSIGNED_BYTE.)

Buffer textures essentially just allow buffers to be accessed with texelFetch(); the data is treated as a 1D array of texels in any format available to textures (between one and four components consisting of signed or unsigned integers, signed or unsigned normalised values, or floats).

Is anything bound to GL_TEXTURE_BUFFER? Note that GL_TEXTURE_BUFFER is a convenience target, similar to GL_COPY_READ_BUFFER and GL_COPY_WRITE_BUFFER. Binding a buffer to that target doesn’t have any effect beyond associating the buffer with that target.

A buffer texture is created using glTexBuffer() or glTextureBuffer(), which just takes an existing buffer and wraps a buffer texture around it. The data isn’t copied; the buffer texture references the underlying buffer, so changes to the buffer affect subsequent reads from the texture.

Dark_Photon · January 24, 2018, 5:48am

[QUOTE=sampsa;1290245]I’m trying to stream video using OpenGL textures. Some features of my programming architecture are:

[ul]
[li] Direct GPU memory access using PBOs - works great … [/li][li] Textures present LUMA and CHROMA planes that given to a shader program. … [/li][/ul]
However, there is an bottleneck that is driving me nuts when I’m copying from PBO to textures.

I have traced this issue into the format/internal_format pair in

glTexImage2D(GL_TEXTURE_2D, 0, internal_format, w, h, 0, format, GL_UNSIGNED_BYTE, 0)

[/QUOTE]

If you’re searching for texel transfer “fast paths” in the driver, I’d recommend that you use glGetInternalFormativ() and just ask your GL driver what they are. This API lets you query (among other things):

[ol]
[li]what internal formats it prefers (i.e. which are likely the ones natively supported by the GPU+GL driver), [/li][li]which format/type you should use when feeding data into and out of that internal format (to avoid internal conversions) when using different transfer APIs. [/li][/ol]
Here’s some code to copy/paste:

[highlight=c++]
const GLenum target = GL_TEXTURE_2D;
GLint supported, preferred, format[ 3 ], type[ 3 ];

glGetInternalformativ( target, intFormat, GL_INTERNALFORMAT_SUPPORTED , 1, &supported );
glGetInternalformativ( target, intFormat, GL_INTERNALFORMAT_PREFERRED , 1, &preferred );
glGetInternalformativ( target, intFormat, GL_TEXTURE_IMAGE_FORMAT , 1, &format[0] );
glGetInternalformativ( target, intFormat, GL_TEXTURE_IMAGE_TYPE , 1, &type [0] );
glGetInternalformativ( target, intFormat, GL_READ_PIXELS_FORMAT , 1, &format[1] );
glGetInternalformativ( target, intFormat, GL_READ_PIXELS_TYPE , 1, &type [1] );
glGetInternalformativ( target, intFormat, GL_GET_TEXTURE_IMAGE_FORMAT , 1, &format[2] );
glGetInternalformativ( target, intFormat, GL_GET_TEXTURE_IMAGE_TYPE , 1, &type [2] );



You don't really care about the glReadPixels and glGetTex*Image() behavior, so ignore the last 4.  I'm just including those for completeness.  The info you care about should be provided by the first 4 lines.

If you use this to buzz out various internal formats on an NVidia GL driver for instance, you'll find that it really doesn't like internal formats for 2- and 3-component uncompressed textures.  So prefer 1- and 4-component internal texture formats.  And of course when possible, you want the data you upload (format/type) to already match the internal format.


> My case would be GL_RED/GL_RED .. but that sucks big time.  With that  pair, "glTexSubImage2D" is hundred times slower than with  GL_RGBA/GL_RGBA.


As GClements mentioned, I'd try buzzing out internal format GL_R8 (with format/type = GL_RED / GL_UNSIGNED_BYTE).  If your driver's just weird/old, check out the legacy internal format GL_LUMINANCE8 (with format/type = GL_LUMINANCE / GL_UNSIGNED_BYTE).


[quote="GClements"]
If [i]format[/i] and [i]type[/i] match [i]internalFormat[/i], there shouldn't be any conversion.
[/quote]

In my experience, that depends on the driver.  Not all supported internal formats seem to be natively supported by the GPU+driver without extra expense in texel transfers to/from those formats.

imported_sampsa · January 24, 2018, 11:57am

Hi,

Thanks for the comments. Eventually I found out that there is not that much difference between GL_RED and, say, GL_BGR (the latter is around 1/3 faster in my intel-gfx based laptop).

The PBO=>TEX seems to take a few milliseconds for a 1080p frame. I guess that’s the best we can do?

The bottleneck in my case seems to be elsewhere … please do take a look at my follow-up question:

For texture uploading, I put some tests in a single file here: