Part of the Khronos Group
OpenGL.org

The Industry's Foundation for High Performance Graphics

from games to virtual reality, mobile phones to supercomputers

Page 2 of 2 FirstFirst 12
Results 11 to 19 of 19

Thread: Flashes on ARM Mali

  1. #11
    Senior Member OpenGL Guru Dark Photon's Avatar
    Join Date
    Oct 2004
    Location
    Druidia
    Posts
    4,478
    Quote Originally Posted by Utumno View Post
    A string of bad news:

    1) my question on Mali dev forum (https://community.arm.com/graphics/f...i-t880-flashes ) remains unanswered
    That stinks. Yeah, I had similar experience on the Qualcomm forums a few years back: LINK Come to find out, their driver sometimes generates GL_OUT_OF_MEMORY when their shader compiler fouls up or fails to detect an error: ANOTHER LINK.

    However, you've established that you can get rid of your flashing merely by stalling the pipeline a while, and you're on Mali, so it's not likely that it has anything to do with a shader error.

    You might try posting about your problem on the the OpenGL ES Forum on khronos.org. It's possible you might catch someone's eye there that has tripped over something similar on their mobile GPU driver.

    3) I have tried to introduce the 'round-robin' strategy with FBOs, and with the Atomic counter, and with the SSBO ( I've tried 2, 3, 5 of everything ) ---> this does not make one bit of a difference. Still flashes once-twice per second. (although with the SSBO what I tried is only 1 fragment shader with 1 SSBO binding point, and 5 SSBOs being bound there in a round-robin strategy - ...

    ...maybe I should try 5 fragment shaders with the only difference being SSBO binding points - and 5 SSBOs permanently bound 1-to-1?)
    No, I don't think that's likely to produce a different result.

    Re your GL_OUT_OF_MEMORY problem, I'd take a look at your buffer and texture usage to make sure there's absolutely no way you're ghosting them on blocking on them such that they might trigger a fragment flush. That "can" lead to legitimate GL_OUT_OF_MEMORY on mobile with some drivers, even if it doesn't look to you like you're using that much memory. Yes, I realize it's a different GPU vendor (so buffer and texture usage contention may be handled differently on Mali drivers), but for general idea, see this link:


    6) KHR or debug output does not seem to be possible on Android. In fact, I don't even know how to create a debug context....
    It's possible your platform doesn't support KHR_debug. Have you tried printing the output of glGetString(GL_EXTENSIONS)? If so, look in there.

    Also, one of the helpful things about KHR_debug is you don't necessarily have to create a debug GL context. You can just call glEnable( GL_DEBUG_OUTPUT ) and start using it, if supported. That said, if you want to go the "create a GL debug context" route, you can do that with EGL by calling eglCreateContext() and passing the EGL_CONTEXT_OPENGL_DEBUG_BIT_KHR bit within the EGL_CONTEXT_FLAGS_KHR flags. For more detail, see EGL_KHR_create_context

    EDIT: debug output is there, but it is only available starting with OpenGL ES 3.2. Fortunately the Mali T880 is 3.2 capable, let's try!

    EDIT2: Well great. KHR debug output seems to be there in OpenGL ES 3.2 headers (at least the function and constant definitions are there) but trying to use them - i.e. calling glDebugMessageCallbackKHR() - results in

    Code :
    java.lang.UnsupportedOperationException: not yet implemented

    on Android 7.0. Supposedly Java bindings were added in Android 8.0 (this was claimed by https://stackoverflow.com/questions/...d-in-android-6, turned out to be not true)

    EDIT3: Even better. Now I tried this on another phone running Android 8.1.0 updated today - still 'not yet implemented'. No wonder there's absolutely no info online about how to use it...
    There's got to be some way to use it. If not, that's surprisingly lame.

    But also one surprising discovery - glFlush() does NOT make this bug go away!

    I have added some 20 glFlushes() all over the code now, 1 after each major step. It's still flashing just the same.
    That's interesting, though I'm not too surprised about that. Depending on how glFlush() is implemented and where in the frame you do it, it could actually cause "more" flashing. I know on another mobile GPU's drivers, a:

    Code cpp:
    glFenceSync( GL_SYNC_GPU_COMMANDS_COMPLETE, ... );
    ...
    glWaitSync();

    mid-frame will cause flashing, as this generated a full GPU pipeline flush including tile rasterization for the partially-queued work for the currently bound render target. A flush at the beginning of the frame (in between render target renders) is more likely not to induce flashing than one done mid-render-target-render.

    If I add just one synchronised 'mapBufferRange - unmapBuffer' at the beginning of my render loop, the bug is completely gone though:

    Code cpp:
    glBindBuffer(GL_SHADER_STORAGE_BUFFER, mSSBO[0] );
    glMapBufferRange(GL_SHADER_STORAGE_BUFFER, 0, length, GL_MAP_READ_BIT);
    glUnmapBuffer(GL_SHADER_STORAGE_BUFFER);
    glBindBuffer(GL_SHADER_STORAGE_BUFFER, 0);
    Another good data point. I think if you keep whacking at this, you'll figure it out.

    If your driver blocks the draw thread whenever you map a buffer that's in-use by the pipeline, that should stall the draw thread at the Map for a while. That could be enough to prevent the circumstances that later that cause the flashing.

    And the fact that you're doing it at the beginning of the render loop makes it less likely to cause flashing, if in-fact it's triggering a full fragment flush (which since you're using it in the fragment shader, I'd bet that it will).

    You really need more visibility as to what's going on down in that driver.
    Last edited by Dark Photon; 06-08-2018 at 06:27 PM.

  2. #12
    Senior Member OpenGL Guru Dark Photon's Avatar
    Join Date
    Oct 2004
    Location
    Druidia
    Posts
    4,478
    Here's a Mali dev support guy talking about using KHR_debug under EGL / OpenGL ES on their GPUs 4 years ago.

    Easier OpenGL ES debugging on ARM Mali GPUs with GL_KHR_debug

    You might see if this helps any.

    This (mali_kernel_common.h) among other places talks about the MALI driver having a mali_debug_level that can take values from 0 though 6 (6 being the highest debug level). This might give you more detailed mali driver info in logcat.

    Here's a link related to it on android (LINK) which suggests that you might be able to set this on Android from a shell with:

    Code :
    # mali_debug_level: Disabling it, you can gain some performance improvements.
     
    echo "0" > /sys/module/mali/parameters/mali_debug_level

    There are references elsewhere that you might also be able to set this as a parameter to the mali kernel module when it's loaded into the kernel on boot-up:

    Code :
    insmod /boot/mali.ko mali_debug_level=2

    And here's a post talking about it on the ARM forums: LINK
    Last edited by Dark Photon; 06-08-2018 at 06:50 PM.

  3. #13
    Intern Contributor
    Join Date
    Nov 2015
    Posts
    50
    Thanks for the answers, again, I really appreciate this!

    I have read before the Mali dev guy page about KHR_debug from 4 years ago. Unfortunately he's talking about native development, i.e. NDK. Like I said, KHR_debug is defined in OpenGL ES 3.2 Java imports ('headers'), but the methods are 'not yet implemented'. Maybe I'll rewrite the app to Native and then I'll be able to squeeze some more info from the driver - but given the fact that the minimal 'bug reproducing' app ATM is about 5k lines - I am not that desperate yet.

    I have also rooted my phone and I am looking at Linux filesystem. Unfortunately the structure is different than the one described in those links from few years ago (that was about the previous architecture, so called Utgard) - now there's no loadable module anymore, everything is statically compiled into the kernel and possibility to dynamically load modules is compiled out (probably for security reasons). But I can see there are unofficial ROMs for my device (Samsung Galaxy S7), maybe I'll try that. There is some info in /sys about memory Mali allocates when my app is running.

    The '/sys/module/mali/parameters/mali_debug_level' unfortunately does not exist any more (not surprising, since there is no module). My question in the Mali dev forum asking how to enable more debugs from the Mali driver ( https://community.arm.com/graphics/f...om-mali-driver ) of course remains unanswered.

    I have also asked in OpenGL ES forum (thanks for the link) : https://forums.khronos.org/showthrea...-Mali-T880-GPU . So far not answered.

    I had some discussion in StackOverflow with someone who, having read the symptoms, agreed that the only possibility is either a 'full pipeline flush' or a bug in the driver.

    #####################################
    Maybe I'll show what I exactly mean by 'flashes'. Here's how the main 'bug-reproducing' app should look like (95% of frames look like this):



    The above is rendered in the following way:
    1) zero out our Atomic counter and SSBO 'per-pixel head pointers'. (AKA 'Order Independent Transparency Pass1 - Clear')
    2) take a textured cube
    3) render it to a FBO1 with a special fragment shader that takes a color parameter (in this case RED) which 'pulls' each pixel color towards itself
    4) blur FBO1 ( this takes two passes - first FBO1 is the input, blur horizontally to another FBO2, then back to our initial FBO1 blurring vertically) (blur is masked by stencil for speed)
    5) copy this blurred red cube to another FBO3, again with a special fragment shader that copies all opaque (frag.a >0.95 ) fragments directly to FBO3, and the transparent fragments get inserted to the per-pixel linked lists of triplets (pointer to next element of linked list, depth,rgba) - this is done already sorting by depth. (AKA 'Order Independent Transparency Pass2 - Build')
    6) Repeat steps 2-3-4-5 with another cube, this time giving YELLOW in step 3 (and a slightly different Model View Matrix, of course)
    7) Repeat steps 2-3-4-5 with another cube, this time giving GREEN in step 3 (and again a slightly different Model View Matrix)
    8) Render a quad, with depth, color and stencil writes off, with fragment shader which goes through the per-pixel linked lists in SSBO and cuts off those that are occluded by opaque pixels from FBO3 (AKA 'Order Independent Transparency Pass3 - Cut')
    9) Render a quad, color+depth writes on, stencil off, Blending on, this time going through the linked lists in SSBO, blending them in order, and finally blending with color from FBO3 (AKA 'Order Independent Transparency Pass4 - Render')
    10) blit FBO3 to screen

    #####################################
    Here are some unusual frames:

    1. Green cube completely missing:

    2. The whole Red cube and the opaque part of the Yellow cube missing:

    3. Both Red and Yellow completely missing:

    4. The whole Red, and transparent parts of Yellow and Green missing:

    5. And IMHO the most interesting example, transparent parts of Red and Yellow show, even though they should be occluded (just as if Order Independent Transparency Pass3 'Cut' pass did not run at all??)
    Last edited by Utumno; 06-09-2018 at 01:33 PM.

  4. #14
    Intern Contributor
    Join Date
    Nov 2015
    Posts
    50
    And another interesting example. Here's another app, a unit test of the graphics library. With this app its very hard to reproduce this bug, 99.9% of frames look like this:

    This is rendered in the following way:

    1. Take a texture of a leaf, render 8 quads textured with this to a FBO1 forming the 'inner ring' with the fragment shader pulling pixel colours towards RED
    2. Blur FBO1
    4. Blit FBO1 to the center of another, larger FBO2
    3. Render 8 more quads textured with the very same leaf texture to FBO2 (forming the 'outer ring'), this time pulling their fragments towards GREEN
    5. Blur FBO2 (so the inner ring gets blurred twice)
    6. Blit FBO2 to the screen.

    Of course the above is also done with the Order Independent Transparency way, when copying we put transparent fragments to the SSBO and merge the SSBO in the last step.

    I managed to see a frame like this:


    So the outer ring is gone, and what's even more interesting, the inner ring got distorted in some strange way. One can see the bottom of FBO2 (the distorted inner ring is cut in the lower part - that's because FBO2 ends there).
    This kind of got me thinking that maybe the objects do not 'disappear' but get distorted and moved; in this particular example we were lucky that the movement was small enough that the new position was still on the screen.

    But on the other hand this would not explain the last case from the previous post - the one where parts of the transparent rings around the cubes showed, even though they should have been cut by the 'Order Independent Transparency Pass3 - Cut' pass.
    Last edited by Utumno; 06-09-2018 at 01:25 PM.

  5. #15
    Intern Contributor
    Join Date
    Nov 2015
    Posts
    50
    I have updated my phone from stock Samsung Android 7.0.0-based firmware to an unofficial LineageOS 15.1 (Android 8.1.0-based).

    Unfortunately the Mali driver is still the same, version 'r12p1-03dev0' and, unsurprisingly, behaves identically.
    The aim is to install the latest Mali driver and see - I guess I can spin my own kernel here, compile Mali with highest debug level and see...
    Last edited by Utumno; 06-10-2018 at 03:29 PM.

  6. #16
    Intern Contributor
    Join Date
    Nov 2015
    Posts
    50
    Summarizing this investigation: mixed, mostly bad, news.

    Flashing my Samsung Galaxy S7 with Lineage OS 15.1 (even though version of the Mali driver in there is still be same) proved partly useful, because the nature of the bug changed slightly. The bug, for some reason, became more predictable. This allowed for the 'let's keep removing code part by part and seeing if the bug is still there' approach; albeit with a long, statistical test (each time 50 runs + script measuring frequency of crashes / flashes ).

    So at the end of this process I concluded that the issue undoubtedly has something to do with the SSBO; sometimes reads from the SSBO

    Code :
    layout (std430,binding=1) buffer linkedlist  
      {                                       
      uint u_Records[];           
      };
    would either return garbage or block (I can't tell those cases apart, but for sure something very shady happens when reading even though the SSBO is filled up with values in 1pass --> memoryBarrier --> read from in another pass). I've tried 'coherent' 'volatile' 'restrict' over there, nothing helps.

    Then I did some more testing on different devices and I gave up. Results:

    Qualcomm's Adreno 418: works wonderfully
    NVidia's Tegra K1: works wonderfully
    PowerVR GX6240: works, although slowly
    PowerVR GE8100: shader fails to compile ( reported here: https://forums.imgtec.com/t/ge8100-i...compile/2708/2 )
    ARM Mali T880: some instability with SSBO, flashes, occasional crashes (with Samsung's original Android 7.0-based OS mostly flashes)

    Looks like in light of this driver situation I'll have to give up trying to implement A-buffer for Order Independent Transparency for mobile devices and think of something simpler. Hurray for Qualcomm and NVidia, down with ARM and Imagination...

    I am also rethinking my choice of platform. Looks like writing any advanced graphics app for Android, with so many possible GPUs, is not going to be easy. In iPhone camp at least they have to deal with only 1 brand of GPU...
    Last edited by Utumno; 06-16-2018 at 05:14 PM.

  7. #17
    Intern Contributor
    Join Date
    Nov 2015
    Posts
    50
    Wow! 3 weeks into the hunt and I finally solved it. I essentially managed to split the SSBO into two parts and make one of the parts (the smaller one, fortunately) into a circular queue of 5 SSBOs, each one used once every 5 frames.
    The test I was talking about in the previous post confirms (10000 runs) that the bug is gone.

    Now the main problem is the PowerVR GE8100 compiler bug, fortunately on their forum they are a bit more responsive than ARM...

  8. #18
    Senior Member OpenGL Guru Dark Photon's Avatar
    Join Date
    Oct 2004
    Location
    Druidia
    Posts
    4,478
    Congrats! Persistence pays off!

    Do you have any insight as to why that particular combination seems to have avoided the instability? Or just chalk one up to another crazy driver bug that makes no sense yet.

    Now the main problem is the PowerVR GE8100 compiler bug, fortunately on their forum they are a bit more responsive than ARM...
    Yes, I was very happy with Imagination Technologies' dev support team a few years back, in how well they investigated problem reports and responded to forum posts. I'd point you to a few names, but the folks that I worked with there have all moved onto other companies since then. In the last 2 years, there's been quite a brain drain at ImgTech, with them losing a number of really solid dev tech and driver engineers.

    In iPhone camp at least they have to deal with only 1 brand of GPU...
    Not quite.

    Before 2017, sure: it was Imagination Tech's PowerVR GPUs. Really solid hardware, drivers, dev tools, and developer support IMO.

    Since then, read up on the Applie A11 Bionic GPU, Iphone 8+, and the new St. Albans, UK office Apple set up near ImgTech.
    Last edited by Dark Photon; 06-22-2018 at 06:10 PM.

  9. #19
    Intern Contributor
    Join Date
    Nov 2015
    Posts
    50
    I need to soften my criticism of ARM here; a very kind ARM engineer has answered (https://community.arm.com/graphics/f...i-t880-flashes ) my questions in the forum and confirmed that the observed behaviour was a bug in the 'r12p1-03dev0' version of the Mali driver. The bug has since been fixed - on r22 the flashing cannot be replicated any more. So answering your question, yes - another crazy driver bug that makes no sense yet.

    However now, for a change, the guys in Imagination forum went AWOL
    Last edited by Utumno; 06-26-2018 at 09:45 AM.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •