|
|
Created:
4 years, 9 months ago by liberato (no reviews please) Modified:
4 years, 8 months ago CC:
chromium-reviews, darin-cc_chromium.org, feature-media-reviews_chromium.org, jam, mcasas+watch_chromium.org, piman+watch_chromium.org, posciak+watch_chromium.org Base URL:
https://chromium.googlesource.com/chromium/src.git@master Target Ref:
refs/pending/heads/master Project:
chromium Visibility:
Public. |
DescriptionMade full screen video with DevTools not flash black on Android.
When DevTools draws to capture the screen output for the remote
side, the video draw quad isn't processed as an overlay since
it's not in the root render pass. However, on android, overlays
aren't optional since Chrome doesn't have the pixels. So, it
uses the PictureBuffer texture to draw, which obscures the surface
view by removing the transparent box that the overlay put there.
Plus, drawing also happens normally, with overlays (no idea why),
and both results end up on the device display. The result is that
the user sees a visible flicker as the transparent overlay box
shows up and goes away. This normal draw has the side-effect of
updating the SurfaceView's contents.
It's unclear to me why this draw happens if we're also going to
display the one for DevTools. The frame rate is ~halved.
This CL attaches changes the PictureBuffer textures to be 2D, and
attaches a 1x1 transparent textures to them, in the case we're using
SurfaceView. When DevTools draw uses this texture to draw the video
quad, it draws the transparent box instead of obscuring it.
In the long run, we should probably transition away from SurfaceView
if we ever detect that we're drawing the video quads rather than
processing them for overlays. This would have the benefit that one
could see the video in DevTools, too.
BUG=592798
Committed: https://crrev.com/2d8fd4af4de3a87aeabe6d3bc86e9b15962b36da
Cr-Commit-Position: refs/heads/master@{#383828}
Patch Set 1 #
Total comments: 4
Patch Set 2 : comments and member fn ptr to base::Callback #
Total comments: 2
Patch Set 3 : switched to 2d textures #
Total comments: 8
Patch Set 4 : cl feedback #Patch Set 5 : request 1x1 2D textures for SV #
Total comments: 5
Patch Set 6 : LOG_IF and rebased (sorry) #
Messages
Total messages: 39 (9 generated)
liberato@chromium.org changed reviewers: + dalecurtis@chromium.org, watk@chromium.org
dalecurtis@chromium.org changed reviewers: + sievers@chromium.org
I'm not strongly familiar with all the ins and outs of EGL images, so +sievers@ to review. https://codereview.chromium.org/1785153004/diff/1/content/common/gpu/media/an... File content/common/gpu/media/android_deferred_rendering_backing_strategy.cc (right): https://codereview.chromium.org/1785153004/diff/1/content/common/gpu/media/an... content/common/gpu/media/android_deferred_rendering_backing_strategy.cc:102: DLOG(ERROR) << "Error destroying transparent EGLImage: " Return? https://codereview.chromium.org/1785153004/diff/1/content/common/gpu/media/an... File content/common/gpu/media/android_deferred_rendering_backing_strategy.h (right): https://codereview.chromium.org/1785153004/diff/1/content/common/gpu/media/an... content/common/gpu/media/android_deferred_rendering_backing_strategy.h:86: EGLImageKHR CreateImageFromTexture(bool ( Use callback instead?
https://codereview.chromium.org/1785153004/diff/1/content/common/gpu/media/an... File content/common/gpu/media/android_deferred_rendering_backing_strategy.cc (right): https://codereview.chromium.org/1785153004/diff/1/content/common/gpu/media/an... content/common/gpu/media/android_deferred_rendering_backing_strategy.cc:102: DLOG(ERROR) << "Error destroying transparent EGLImage: " On 2016/03/11 19:23:36, DaleCurtis wrote: > Return? we want to continune with cleanup. i'll add a comment. https://codereview.chromium.org/1785153004/diff/1/content/common/gpu/media/an... File content/common/gpu/media/android_deferred_rendering_backing_strategy.h (right): https://codereview.chromium.org/1785153004/diff/1/content/common/gpu/media/an... content/common/gpu/media/android_deferred_rendering_backing_strategy.h:86: EGLImageKHR CreateImageFromTexture(bool ( On 2016/03/11 19:23:36, DaleCurtis wrote: > Use callback instead? since the strategy isn't refcounted, didn't seem to add anything. i'll update to base::Callback and see how it looks. originally, i wanted to send in a lambda, but i couldn't get base::Bind to work with it (even though i found a discussion on it that says it should).
@sievers: do you see any problems with how this CL uses EGLImage? thanks -fl
sievers, PTAL - are we using eglimages in an okay way? do you see any potential problems with this approach? thanks -fl
DevTools uses CopyFromCompositingSurface() right? That works through putting a copy request on the (Surface)Layer which in the next browser composite will be copied out (by introducing an intermediate render surface in the compositor).
On 2016/03/25 21:57:13, sievers wrote: > DevTools uses CopyFromCompositingSurface() right? > > That works through putting a copy request on the (Surface)Layer which in the > next browser composite will be copied out (by introducing an intermediate render > surface in the compositor). So I guess what I was trying to understand is why then this problem would be related to SurfaceView, EGLimage or what not. I'd think that there would just be no promotion when you introduce the readback with the intermediate surface.
On 2016/03/25 21:58:42, sievers wrote: > On 2016/03/25 21:57:13, sievers wrote: > > DevTools uses CopyFromCompositingSurface() right? > > > > That works through putting a copy request on the (Surface)Layer which in the > > next browser composite will be copied out (by introducing an intermediate > render > > surface in the compositor). > > So I guess what I was trying to understand is why then this problem would be > related to SurfaceView, EGLimage or what not. > I'd think that there would just be no promotion when you introduce the readback > with the intermediate surface. correct, overlays aren't promoted. the problem is that it tries to draw the stream video draw quad, drawing a black patch over the area that used to be transparent for the SV. that causes every other frame or so to flash black on the device. this patch initializes that texture to be transparent, so the SV shows through in all cases.
On 2016/03/25 22:04:27, liberato wrote: > On 2016/03/25 21:58:42, sievers wrote: > > On 2016/03/25 21:57:13, sievers wrote: > > > DevTools uses CopyFromCompositingSurface() right? > > > > > > That works through putting a copy request on the (Surface)Layer which in the > > > next browser composite will be copied out (by introducing an intermediate > > render > > > surface in the compositor). > > > > So I guess what I was trying to understand is why then this problem would be > > related to SurfaceView, EGLimage or what not. > > I'd think that there would just be no promotion when you introduce the > readback > > with the intermediate surface. > > correct, overlays aren't promoted. the problem is that it tries to draw the > stream video draw quad, drawing a black patch over the area that used to be > transparent for the SV. that causes every other frame or so to flash black on > the device. > > this patch initializes that texture to be transparent, so the SV shows through > in all cases. Ok I see, I think that makes sense then as long as we can't really do the dynamic and deferred real overlay promotion on Android. But why do you need an EGLimage for the 1x1 transparent texture and don't just call texImage2D() instead?
On 2016/03/25 22:24:43, sievers wrote: > On 2016/03/25 22:04:27, liberato wrote: > > On 2016/03/25 21:58:42, sievers wrote: > > > On 2016/03/25 21:57:13, sievers wrote: > > > > DevTools uses CopyFromCompositingSurface() right? > > > > > > > > That works through putting a copy request on the (Surface)Layer which in > the > > > > next browser composite will be copied out (by introducing an intermediate > > > render > > > > surface in the compositor). > > > > > > So I guess what I was trying to understand is why then this problem would be > > > related to SurfaceView, EGLimage or what not. > > > I'd think that there would just be no promotion when you introduce the > > readback > > > with the intermediate surface. > > > > correct, overlays aren't promoted. the problem is that it tries to draw the > > stream video draw quad, drawing a black patch over the area that used to be > > transparent for the SV. that causes every other frame or so to flash black on > > the device. > > > > this patch initializes that texture to be transparent, so the SV shows through > > in all cases. > > Ok I see, I think that makes sense then as long as we can't really do the > dynamic and deferred real overlay promotion on Android. > > But why do you need an EGLimage for the 1x1 transparent texture and don't just > call texImage2D() instead? does that work with external textures? i've never tried. i thought they were eglimage only.
On 2016/03/25 23:00:25, liberato wrote: > On 2016/03/25 22:24:43, sievers wrote: > > On 2016/03/25 22:04:27, liberato wrote: > > > On 2016/03/25 21:58:42, sievers wrote: > > > > On 2016/03/25 21:57:13, sievers wrote: > > > > > DevTools uses CopyFromCompositingSurface() right? > > > > > > > > > > That works through putting a copy request on the (Surface)Layer which in > > the > > > > > next browser composite will be copied out (by introducing an > intermediate > > > > render > > > > > surface in the compositor). > > > > > > > > So I guess what I was trying to understand is why then this problem would > be > > > > related to SurfaceView, EGLimage or what not. > > > > I'd think that there would just be no promotion when you introduce the > > > readback > > > > with the intermediate surface. > > > > > > correct, overlays aren't promoted. the problem is that it tries to draw the > > > stream video draw quad, drawing a black patch over the area that used to be > > > transparent for the SV. that causes every other frame or so to flash black > on > > > the device. > > > > > > this patch initializes that texture to be transparent, so the SV shows > through > > > in all cases. > > > > Ok I see, I think that makes sense then as long as we can't really do the > > dynamic and deferred real overlay promotion on Android. > > > > But why do you need an EGLimage for the 1x1 transparent texture and don't just > > call texImage2D() instead? > > does that work with external textures? i've never tried. i thought they were > eglimage only. Hmm no you'd have to get the compositor to use GL_TEXTURE_2D for the quad, which then probably requires the resource from the renderer to use that in the first place. And that maybe doesn't make sense the way you are transitioning in and out of fullscreen?
lgtm https://codereview.chromium.org/1785153004/diff/20001/content/common/gpu/medi... File content/common/gpu/media/android_deferred_rendering_backing_strategy.cc (right): https://codereview.chromium.org/1785153004/diff/20001/content/common/gpu/medi... content/common/gpu/media/android_deferred_rendering_backing_strategy.cc:99: if (have_context && transparent_image_ != EGL_NO_IMAGE_KHR) { does it really need a context for the EGL call?
> And that maybe doesn't make sense the way you are transitioning in and out of fullscreen? right now, it doesn't. after some though, maybe we could do that, since you have a really good point. in full screen mode when using a SV, the deferred rendering backing strategy could request 2D textures instead and use glTexImage2D to initialize them. avoiding EGLImages seems like a good thing. simplifies cleanup and potential flakiness on older platforms, if nothing else. the only thing that scares me is that it would stop using the stream video draw quads, which i think is fine but i'd have to check. i don't think that anything happens that would affect overlays differently, but there are a lot of layers. :) it adds a dependency on our current behavior of restarting the VDA on the transition, else we'd be forced to dismiss the picture buffers to change targets. that invalidates pending frames when using unowned textures, causing visible glitches, but is required for security reasons without additional work. i'm okay with that, since we do restart the VDA now, and shouldn't be using SV when there's a readback anyway. i'll try it out and see. https://codereview.chromium.org/1785153004/diff/20001/content/common/gpu/medi... File content/common/gpu/media/android_deferred_rendering_backing_strategy.cc (right): https://codereview.chromium.org/1785153004/diff/20001/content/common/gpu/medi... content/common/gpu/media/android_deferred_rendering_backing_strategy.cc:99: if (have_context && transparent_image_ != EGL_NO_IMAGE_KHR) { On 2016/03/25 23:23:00, sievers wrote: > does it really need a context for the EGL call? thanks! i just assumed it needed one since creating the image does. i didn't realize that the context was needed only for the source texture, which makes quite a bit of sense in retrospect :)
Description was changed from ========== Made full screen video with DevTools not flash black on Android. When DevTools draws to capture the screen output for the remote side, the video draw quad isn't processed as an overlay since it's not in the root render pass. However, on android, overlays aren't optional since Chrome doesn't have the pixels. So, it uses the PictureBuffer texture to draw, and obscure the surface view by removing the transparent box that the overlay put there. Plus, the second draw ends up being displayed on the device too (no idea why), so the user sees a visible flicker. Since the normal draw also happens, which selects the video quad as an overlay, the SurfaceView does actually get updated. It's unclear to me why this draw happens if we're also going to display the one for DevTools. So, this CL attaches a 1x1 transparent EGLImage to the PictureBuffer textures, so that the DevTools draw draws the transparent box, instead of obscuring it. In the long run, we should probably transition away from SurfaceView if we ever detect that we're drawing the video quads rather than processing them for overlays. This would have the benefit that one could see the video in DevTools, too. BUG=592798 ========== to ========== Made full screen video with DevTools not flash black on Android. When DevTools draws to capture the screen output for the remote side, the video draw quad isn't processed as an overlay since it's not in the root render pass. However, on android, overlays aren't optional since Chrome doesn't have the pixels. So, it uses the PictureBuffer texture to draw, which obscures the surface view by removing the transparent box that the overlay put there. Plus, drawing also happens normally, with overlays (no idea why), and both results end up on the device display. The result is that the user sees a visible flicker as the transparent overlay box shows up and goes away. This normal draw has the side-effect of updating the SurfaceView's contents. It's unclear to me why this draw happens if we're also going to display the one for DevTools. The frame rate is ~halved. This CL attaches changes the PictureBuffer textures to be 2D, and attaches a 1x1 transparent textures to them, in the case we're using SurfaceView. When DevTools draw uses this texture to draw the video quad, it draws the transparent box instead of obscuring it. In the long run, we should probably transition away from SurfaceView if we ever detect that we're drawing the video quads rather than processing them for overlays. This would have the benefit that one could see the video in DevTools, too. BUG=592798 ==========
Patchset #3 (id:40001) has been deleted
yay -- seems to work with 2d textures. i'm going to double-check the power numbers just to be sure that the switch to 2D textures didn't break something subtle. thanks -fl
On 2016/03/28 15:12:44, liberato wrote: > yay -- seems to work with 2d textures. i'm going to double-check the power > numbers just to be sure that the switch to 2D textures didn't break something > subtle. > > thanks > -fl power is the same between WMPI and WMPA. -fl
This isn't allocating 2D textures the size of the video is it?
On 2016/03/28 17:51:28, DaleCurtis wrote: > This isn't allocating 2D textures the size of the video is it? nope, 1x1.
Hmm, how is it not doing that? You changed the return type for GetTextureTarget() which is used by AVDA to send to ProvidePictureBuffers() using |size_| ? https://codereview.chromium.org/1785153004/diff/60001/content/common/gpu/medi... File content/common/gpu/media/android_deferred_rendering_backing_strategy.cc (right): https://codereview.chromium.org/1785153004/diff/60001/content/common/gpu/medi... content/common/gpu/media/android_deferred_rendering_backing_strategy.cc:108: return (surface_texture_ ? GL_TEXTURE_EXTERNAL_OES : GL_TEXTURE_2D); No unnecessary parens. https://codereview.chromium.org/1785153004/diff/60001/content/common/gpu/medi... content/common/gpu/media/android_deferred_rendering_backing_strategy.cc:203: unsigned char rgba[] = {0, 0, 0, 0}; static const, does uint8_t work? https://codereview.chromium.org/1785153004/diff/60001/content/common/gpu/medi... content/common/gpu/media/android_deferred_rendering_backing_strategy.cc:204: const gfx::Size size(1, 1); I'd just inline each value, but up to you. https://codereview.chromium.org/1785153004/diff/60001/content/common/gpu/medi... File content/common/gpu/media/android_video_decode_accelerator.cc (right): https://codereview.chromium.org/1785153004/diff/60001/content/common/gpu/medi... content/common/gpu/media/android_video_decode_accelerator.cc:805: bool have_context = false; const bool have_context = make_context_current_.Run() and then check.
lgtm
Patchset #4 (id:80001) has been deleted
https://codereview.chromium.org/1785153004/diff/60001/content/common/gpu/medi... File content/common/gpu/media/android_deferred_rendering_backing_strategy.cc (right): https://codereview.chromium.org/1785153004/diff/60001/content/common/gpu/medi... content/common/gpu/media/android_deferred_rendering_backing_strategy.cc:108: return (surface_texture_ ? GL_TEXTURE_EXTERNAL_OES : GL_TEXTURE_2D); On 2016/03/28 18:02:02, DaleCurtis wrote: > No unnecessary parens. Done. https://codereview.chromium.org/1785153004/diff/60001/content/common/gpu/medi... content/common/gpu/media/android_deferred_rendering_backing_strategy.cc:203: unsigned char rgba[] = {0, 0, 0, 0}; On 2016/03/28 18:02:02, DaleCurtis wrote: > static const, does uint8_t work? Done. https://codereview.chromium.org/1785153004/diff/60001/content/common/gpu/medi... content/common/gpu/media/android_deferred_rendering_backing_strategy.cc:204: const gfx::Size size(1, 1); On 2016/03/28 18:02:02, DaleCurtis wrote: > I'd just inline each value, but up to you. i think that it's clearer than glTexImage2D(..., 1, 1, 0, GL_RGBA,...), so i'll keep it as is. https://codereview.chromium.org/1785153004/diff/60001/content/common/gpu/medi... File content/common/gpu/media/android_video_decode_accelerator.cc (right): https://codereview.chromium.org/1785153004/diff/60001/content/common/gpu/medi... content/common/gpu/media/android_video_decode_accelerator.cc:805: bool have_context = false; On 2016/03/28 18:02:02, DaleCurtis wrote: > const bool have_context = make_context_current_.Run() and then check. Done.
Can you elaborate on the correctness given my previous statement "Hmm, how is it not doing that? You changed the return type for GetTextureTarget() which is used by AVDA to send to ProvidePictureBuffers() using |size_| ?" I want to make sure we're not regressing memory usage.
On 2016/03/28 22:04:06, DaleCurtis wrote: > Can you elaborate on the correctness given my previous statement "Hmm, how is it > not doing that? You changed the return type for GetTextureTarget() which is used > by AVDA to send to ProvidePictureBuffers() using |size_| ?" I want to make sure > we're not regressing memory usage. sorry, missed your comment. it's true that the PictureBuffer's texture is being initialized with an image that's big enough for the frame. however, we're replacing it via glTexImage2D with a 1x1 in AssignOne. between the time the PictureBuffer textures are generated and AssignOne is called on each, it will use more memory. after AssignOne has been called, it'll use the same amount (approximately) as it did before the CL. are you concerned about it? if so, we could reduce it by making RequestPictureBuffers not assume the size. we'd also have to fix the PictureBuffer's internal size to be correct via something like the existing Strategy::UpdatePictureBufferSize, since GVD would think that it's a 1x1 picture otherwise.
Video frames are pretty large, so a spike of say 4x 720p textures seems problematic. Will all four be allocated before any are replaced?
On 2016/03/28 22:33:39, DaleCurtis wrote: > Video frames are pretty large, so a spike of say 4x 720p textures seems > problematic. Will all four be allocated before any are replaced? yes, they will exist concurrently. i'll see if requesting 1x1 works.
Patchset #5 (id:120001) has been deleted
https://codereview.chromium.org/1785153004/diff/140001/content/common/gpu/med... File content/common/gpu/media/android_deferred_rendering_backing_strategy.cc (right): https://codereview.chromium.org/1785153004/diff/140001/content/common/gpu/med... content/common/gpu/media/android_deferred_rendering_backing_strategy.cc:118: // texture when building the quad for it. as an aside, it currently does not actually scale the texture; 1x1 would work fine. this is because the scaling matrix is being ignored in the AVDA path in favor of the real texture matrix. toby noticed it when removing the texture matrix IPC. i think that it's okay for use with SurfaceTexture as is, but we're going to make it consistent with 2D textures anyway.
lgtm % question https://codereview.chromium.org/1785153004/diff/140001/content/common/gpu/med... File content/common/gpu/media/android_deferred_rendering_backing_strategy.cc (right): https://codereview.chromium.org/1785153004/diff/140001/content/common/gpu/med... content/common/gpu/media/android_deferred_rendering_backing_strategy.cc:206: if (!surface_texture_ && have_context) { Is this part necessary anymore since you're allocating a 2D texture already? https://codereview.chromium.org/1785153004/diff/140001/content/common/gpu/med... File content/common/gpu/media/android_video_decode_accelerator.cc (right): https://codereview.chromium.org/1785153004/diff/140001/content/common/gpu/med... content/common/gpu/media/android_video_decode_accelerator.cc:807: if (!have_context) LOG_IF(!have_context, "") if you want.
The CQ bit was checked by liberato@chromium.org
The patchset sent to the CQ was uploaded after l-g-t-m from sievers@chromium.org, watk@chromium.org, dalecurtis@chromium.org Link to the patchset: https://codereview.chromium.org/1785153004/#ps160001 (title: "LOG_IF and rebased (sorry)")
https://codereview.chromium.org/1785153004/diff/140001/content/common/gpu/med... File content/common/gpu/media/android_deferred_rendering_backing_strategy.cc (right): https://codereview.chromium.org/1785153004/diff/140001/content/common/gpu/med... content/common/gpu/media/android_deferred_rendering_backing_strategy.cc:206: if (!surface_texture_ && have_context) { On 2016/03/29 17:38:41, DaleCurtis wrote: > Is this part necessary anymore since you're allocating a 2D texture already? yes. it uploads the 1x1 image. the picture buffer texture is backed by an uninitialized image of the correct size. https://codereview.chromium.org/1785153004/diff/140001/content/common/gpu/med... File content/common/gpu/media/android_video_decode_accelerator.cc (right): https://codereview.chromium.org/1785153004/diff/140001/content/common/gpu/med... content/common/gpu/media/android_video_decode_accelerator.cc:807: if (!have_context) On 2016/03/29 17:38:41, DaleCurtis wrote: > LOG_IF(!have_context, "") if you want. Done.
CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/patch-status/1785153004/160001 View timeline at https://chromium-cq-status.appspot.com/patch-timeline/1785153004/160001
Message was sent while issue was closed.
Committed patchset #6 (id:160001)
Message was sent while issue was closed.
Description was changed from ========== Made full screen video with DevTools not flash black on Android. When DevTools draws to capture the screen output for the remote side, the video draw quad isn't processed as an overlay since it's not in the root render pass. However, on android, overlays aren't optional since Chrome doesn't have the pixels. So, it uses the PictureBuffer texture to draw, which obscures the surface view by removing the transparent box that the overlay put there. Plus, drawing also happens normally, with overlays (no idea why), and both results end up on the device display. The result is that the user sees a visible flicker as the transparent overlay box shows up and goes away. This normal draw has the side-effect of updating the SurfaceView's contents. It's unclear to me why this draw happens if we're also going to display the one for DevTools. The frame rate is ~halved. This CL attaches changes the PictureBuffer textures to be 2D, and attaches a 1x1 transparent textures to them, in the case we're using SurfaceView. When DevTools draw uses this texture to draw the video quad, it draws the transparent box instead of obscuring it. In the long run, we should probably transition away from SurfaceView if we ever detect that we're drawing the video quads rather than processing them for overlays. This would have the benefit that one could see the video in DevTools, too. BUG=592798 ========== to ========== Made full screen video with DevTools not flash black on Android. When DevTools draws to capture the screen output for the remote side, the video draw quad isn't processed as an overlay since it's not in the root render pass. However, on android, overlays aren't optional since Chrome doesn't have the pixels. So, it uses the PictureBuffer texture to draw, which obscures the surface view by removing the transparent box that the overlay put there. Plus, drawing also happens normally, with overlays (no idea why), and both results end up on the device display. The result is that the user sees a visible flicker as the transparent overlay box shows up and goes away. This normal draw has the side-effect of updating the SurfaceView's contents. It's unclear to me why this draw happens if we're also going to display the one for DevTools. The frame rate is ~halved. This CL attaches changes the PictureBuffer textures to be 2D, and attaches a 1x1 transparent textures to them, in the case we're using SurfaceView. When DevTools draw uses this texture to draw the video quad, it draws the transparent box instead of obscuring it. In the long run, we should probably transition away from SurfaceView if we ever detect that we're drawing the video quads rather than processing them for overlays. This would have the benefit that one could see the video in DevTools, too. BUG=592798 Committed: https://crrev.com/2d8fd4af4de3a87aeabe6d3bc86e9b15962b36da Cr-Commit-Position: refs/heads/master@{#383828} ==========
Message was sent while issue was closed.
Patchset 6 (id:??) landed as https://crrev.com/2d8fd4af4de3a87aeabe6d3bc86e9b15962b36da Cr-Commit-Position: refs/heads/master@{#383828} |