16 bpp video stream capture, render and WebGL usage - Realsense R200 & SR300 support.

16 bpp video stream capture, render and WebGL usage - Realsense R200 & SR300 support.

It Supports 16 bit depth and infrared video streams from R200 and SR300.
Verified to work on Windows 10 and Ubuntu 16.04.

Rendering video element:
========================
16bpp frames are uploaded to RG textures, respectively - for RG textures
lower byte goes to 'R' and higher byte to 'G'.
Rendered output is original RG texture. Screenshots of demo are available on
crbug.com/624436 page.

RG88 path required custom bilinear filter in shaders.cc.

WebGL:
======
Upload video to texture:
In all the textures, R, G, B components have the same value (as luminance).
- RGBA UNSIGNED_BYTE: R, G and B components presents upper byte of 16 bit video,
  so, with precision loss.
- RGB/RGBA FLOAT: Normalized 16 bit value in R, G and B component - no precision loss.

Getting full 16 bit or float data to javascript:
===============================================
1. on start, create framebuffer, attach (COLOR_ATTACHMENT0) RGBA FLOAT texture to it.
2. In loop, when video is ready, after texImage2D video to the texture, bind
   the framebuffer and call readpixels to FloatArray. This way the data is
   accessible from javascript.

Complete examples are available on Issue 624436 page.

2D canvas + getImageData returns 8 bit color components (higher bit value) in RGBA.

BUG=624436
CQ_INCLUDE_TRYBOTS=master.tryserver.blink:linux_precise_blink_rel

[aleksandar.stojiljkovic, 2016-07-04 17:44:26]

Description was changed from

==========
16 bit video stream capture and render - Realsense SR300 support.

This is initial code with Realsense SR300 camera on Windows.
Supports only 16 bit depth video stream from SR300. Verified to work on Windows
10.

16bit frames are uploaded to RG textures - lower byte to 'R' and higher byte to
'G'.
Rendered output is original RG texture. Screenshot should be available on Issue
624436 page soon.

There are multiple TODO(astojilj) in the code. I'm going to address them in
follow up patches. The initial patch is used to get early feedback.

BUG=624436
==========

to

==========
16 bit video stream capture and render - Realsense SR300 support.

This is initial code with Realsense SR300 camera on Windows.
Supports only 16 bit depth video stream from SR300. Verified to work on Windows
10.

16bit frames are uploaded to RG textures - lower byte to 'R' and higher byte to
'G'.
Rendered output is original RG texture. Screenshot should be available on Issue
624436 page soon.

There are multiple TODO(astojilj) in the code. I'm going to address them in
follow up patches. The initial patch is used to get early feedback.

BUG=624436
CQ_INCLUDE_TRYBOTS=tryserver.blink:linux_blink_rel
==========

[aleksandar.stojiljkovic, 2016-07-04 18:09:54]

aleksandar.stojiljkovic@intel.com changed reviewers:
+ dnicoara@chromium.org, dongseong.hwang@intel.com, hubbe@chromium.org,
kbr@google.com

[aleksandar.stojiljkovic, 2016-07-04 18:09:55]

Patch 1 is fully functioning proof of concept code.
Continuing work on it but early feedback is more than welcome.
Thanks.

[dshwang, 2016-07-04 18:23:17]

nice progress

https://codereview.chromium.org/2121043002/diff/1/cc/layers/video_layer_impl.cc
File cc/layers/video_layer_impl.cc (right):

https://codereview.chromium.org/2121043002/diff/1/cc/layers/video_layer_impl....
cc/layers/video_layer_impl.cc:323: YVideoDrawQuad* y_video_quad =
IMO, existing TextureDrawQuad can handle it.

[aleksandar.stojiljkovic, 2016-07-05 19:13:16]

Description was changed from

==========
16 bit video stream capture and render - Realsense SR300 support.

This is initial code with Realsense SR300 camera on Windows.
Supports only 16 bit depth video stream from SR300. Verified to work on Windows
10.

16bit frames are uploaded to RG textures - lower byte to 'R' and higher byte to
'G'.
Rendered output is original RG texture. Screenshot should be available on Issue
624436 page soon.

There are multiple TODO(astojilj) in the code. I'm going to address them in
follow up patches. The initial patch is used to get early feedback.

BUG=624436
CQ_INCLUDE_TRYBOTS=tryserver.blink:linux_blink_rel
==========

to

==========
8 and 16 bpp video stream capture and render - Realsense R200 & SR300 support.

This is an initial code supporting depth and infrared camera devices capture and
rendering with Realsense R200 and SR300 camera on Windows.

The initial patch is used to get early feedback while working on it.
It Supports 8 and 16 bit depth and infrared video streams from R200 and SR300.
Verified to work on Windows 10.

8 and 16bpp frames are uploaded to R and RG textures, respectively - for RG
textures
lower byte goes to 'R' and higher byte to 'G'.
Rendered output is original R/RG texture. Screenshots of demo are be available
on Issue 624436 page.

There are multiple TODO(astojilj) in the code. I'm going to address them in
follow up
patches here.

BUG=624436
==========

[aleksandar.stojiljkovic, 2016-07-05 19:54:17]

aleksandar.stojiljkovic@intel.com changed reviewers:
+ danakj@chromium.org, enne@chromium.org, mcasas@chromium.org

[aleksandar.stojiljkovic, 2016-07-05 19:54:18]

patchset 2: 8 bpp and Realsense R200 support added.
Works with depth and infrared device streams.
Demo screenshots: crbug.com/624436#c16

Including more owners:

cc/layers:
cc/resources:
 +enne@, +danakj@

media/capture/video/win:
 +mcasas@
(there are some TODOs there but before adding more code your opinion about
approach with GUIDs, devices, preferred streams would help a lot. Thanks)

https://codereview.chromium.org/2121043002/diff/1/cc/layers/video_layer_impl.cc
File cc/layers/video_layer_impl.cc (right):

https://codereview.chromium.org/2121043002/diff/1/cc/layers/video_layer_impl....
cc/layers/video_layer_impl.cc:323: YVideoDrawQuad* y_video_quad =
On 2016/07/04 18:23:16, dshwang wrote:
> IMO, existing TextureDrawQuad can handle it.

Done.

https://codereview.chromium.org/2121043002/diff/20001/cc/resources/video_reso...
File cc/resources/video_resource_updater.h (right):

https://codereview.chromium.org/2121043002/diff/20001/cc/resources/video_reso...
cc/resources/video_resource_updater.h:45: Y_RESOURCE,
maybe not the best naming - stands for Y8 and Y16.

[aleksandar.stojiljkovic, 2016-07-11 22:15:14]

https://codereview.chromium.org/2121043002/diff/20001/content/browser/rendere...
File content/browser/renderer_host/media/video_capture_device_client.cc (right):

https://codereview.chromium.org/2121043002/diff/20001/content/browser/rendere...
content/browser/renderer_host/media/video_capture_device_client.cc:172:
memcpy(buffer->data(), data, length);
normalized unsigned short -| half_float conversion could happen here.

[aleksandar.stojiljkovic, 2016-07-15 11:45:00]

Description was changed from

==========
8 and 16 bpp video stream capture and render - Realsense R200 & SR300 support.

This is an initial code supporting depth and infrared camera devices capture and
rendering with Realsense R200 and SR300 camera on Windows.

The initial patch is used to get early feedback while working on it.
It Supports 8 and 16 bit depth and infrared video streams from R200 and SR300.
Verified to work on Windows 10.

8 and 16bpp frames are uploaded to R and RG textures, respectively - for RG
textures
lower byte goes to 'R' and higher byte to 'G'.
Rendered output is original R/RG texture. Screenshots of demo are be available
on Issue 624436 page.

There are multiple TODO(astojilj) in the code. I'm going to address them in
follow up
patches here.

BUG=624436
==========

to

==========
8 and 16 bpp video stream capture and render - Realsense R200 & SR300 support.

This is an initial code supporting depth and infrared camera devices capture and
rendering with Realsense R200 and SR300 camera on Windows.

The initial patch is used to get early feedback while working on it.
It Supports 8 and 16 bit depth and infrared video streams from R200 and SR300.
Verified to work on Windows 10.

8 and 16bpp frames are uploaded to R and RG textures, respectively - for RG
textures
lower byte goes to 'R' and higher byte to 'G'.
Rendered output is original R/RG texture. Screenshots of demo are be available
on Issue 624436 page.

There are multiple TODO(astojilj) in the code. I'm going to address them in
follow up
patches here.

BUG=624436
CQ_INCLUDE_TRYBOTS=tryserver.blink:linux_blink_rel
==========

[aleksandar.stojiljkovic, 2016-07-15 12:29:06]

Patch #3: bilinear filter for rg88, float and half float added.
Patch is used for power and performance measurement - results are to be
published at https://crbug.com/624436 soon.

https://codereview.chromium.org/2121043002/diff/40001/cc/output/shader.cc
File cc/output/shader.cc (right):

https://codereview.chromium.org/2121043002/diff/40001/cc/output/shader.cc#new...
cc/output/shader.cc:2167: vec4 TextureLookupBilinear(sampler2D sampler, highp
vec2 texCoord) {
custom bilinear filter used for RG88.

https://codereview.chromium.org/2121043002/diff/40001/cc/resources/resource_p...
File cc/resources/resource_provider.cc (right):

https://codereview.chromium.org/2121043002/diff/40001/cc/resources/resource_p...
cc/resources/resource_provider.cc:453: // TODO(astojilj): Temporary code for
performance and power consumption
Uncomment line bellow to measure F16 and F32.

https://codereview.chromium.org/2121043002/diff/40001/cc/resources/video_reso...
File cc/resources/video_resource_updater.cc (right):

https://codereview.chromium.org/2121043002/diff/40001/cc/resources/video_reso...
cc/resources/video_resource_updater.cc:109: uint16_t
ushort_to_half_float(uint16_t value) {
Approach used for uint16_t to half float conversion - idea for log2 approach is
from here: 
https://graphics.stanford.edu/~seander/bithacks.html#IntegerLogLookup

[aleksandar.stojiljkovic, 2016-08-15 21:02:37]

Description was changed from

==========
8 and 16 bpp video stream capture and render - Realsense R200 & SR300 support.

This is an initial code supporting depth and infrared camera devices capture and
rendering with Realsense R200 and SR300 camera on Windows.

The initial patch is used to get early feedback while working on it.
It Supports 8 and 16 bit depth and infrared video streams from R200 and SR300.
Verified to work on Windows 10.

8 and 16bpp frames are uploaded to R and RG textures, respectively - for RG
textures
lower byte goes to 'R' and higher byte to 'G'.
Rendered output is original R/RG texture. Screenshots of demo are be available
on Issue 624436 page.

There are multiple TODO(astojilj) in the code. I'm going to address them in
follow up
patches here.

BUG=624436
CQ_INCLUDE_TRYBOTS=tryserver.blink:linux_blink_rel
==========

to

==========
8 and 16 bpp video stream capture and render - Realsense R200 & SR300 support.

This is an initial code supporting depth and infrared camera devices capture and
rendering with Realsense R200 and SR300 camera on Windows.

The initial patch is used to get early feedback while working on it.
It Supports 8 and 16 bit depth and infrared video streams from R200 and SR300.
Verified to work on Windows 10.

Rendering video element:
========================
8 and 16bpp frames are uploaded to R and RG textures, respectively - for RG
textures
lower byte goes to 'R' and higher byte to 'G'.
Rendered output is original R/RG texture. Screenshots of demo are available on
crbug.com/624436 page.

RG88 path required custom bilinear filter in shaders.cc.

WebGL:
======
Upload video to texture:
- RGBA UNSIGNED_BYTE: only R component presents upper byte of 16 bit video,
- R16UI UNSIGNED_SHORT: 16 bits go to R,
- RG8 UNSIGNED_BYTE: upper byte to G, lower to R.

Getting full 16 bit data to javascript.
===========================
The approach:
1. on start, create framebuffer, attach (COLOR_ATTACHMENT0) R16UI texture to it.
2. In loop, when video is ready, after texImage2D video to R16UI texture, bind
   the framebuffer and call readpixels to Uint16Array. This way the data is
   accessible in javascript.

jsfiddle examples are available on Issue 624436 page.

2D canvas + getImageData returns 8 bit R in RGBA.

There are multiple TODO(astojilj) in the code. I'm going to address them in
follow up
patches here.

BUG=624436
==========

[aleksandar.stojiljkovic, 2016-08-15 21:10:22]

aleksandar.stojiljkovic@intel.com changed reviewers:
+ zmo@chromium.org

[aleksandar.stojiljkovic, 2016-08-15 21:10:23]

+zmo

(to review approach in third_party/WebKit/Source/modules/webgl, Video
skcanvas_video_renderer.cc and VideoMediaPlayer(MS) added in this patch).

The patch adds support for following:

WebGL Upload 16 bit video to texture:
- RGBA UNSIGNED_BYTE: only R component presents upper byte of 16 bit video,
- R16UI UNSIGNED_SHORT: 16 bits go to R,
- RG8 UNSIGNED_BYTE: upper byte to G, lower to R.

Getting full 16 bit data to javascript.
===========================
The approach:
1. on start, create framebuffer, attach (COLOR_ATTACHMENT0) R16UI texture to it.
2. In loop, when video is ready, after texImage2D video to R16UI texture, bind
   the framebuffer and call readpixels to Uint16Array. This way the data is
   accessible in javascript.

details, jsfiddle demos used for verification, screenshots and explanations are
available on crbug.com/624436

[Ken Russell (switch to Gerrit), 2016-08-16 00:22:25]

kbr@chromium.org changed reviewers:
+ kbr@chromium.org

[Ken Russell (switch to Gerrit), 2016-08-16 00:22:27]

It's encouraging to see your progress on this. A few questions.

https://codereview.chromium.org/2121043002/diff/60001/cc/resources/resource_p...
File cc/resources/resource_provider.cc (right):

https://codereview.chromium.org/2121043002/diff/60001/cc/resources/resource_p...
cc/resources/resource_provider.cc:455: // LUMINANCE_8;
Does this really use a luminance texture containing 16-bit floats? Luminance
textures are deprecated in OpenGL ES 3.0. Did you try a GL_R16F type texture?

https://codereview.chromium.org/2121043002/diff/60001/gpu/command_buffer/serv...
File gpu/command_buffer/service/texture_manager.cc (right):

https://codereview.chromium.org/2121043002/diff/60001/gpu/command_buffer/serv...
gpu/command_buffer/service/texture_manager.cc:174: {GL_R16UI, GL_R16UI,
GL_UNSIGNED_SHORT},
This one isn't mentioned in the switch statement below. Is that deliberate?

https://codereview.chromium.org/2121043002/diff/60001/media/renderers/skcanva...
File media/renderers/skcanvas_video_renderer.cc (right):

https://codereview.chromium.org/2121043002/diff/60001/media/renderers/skcanva...
media/renderers/skcanvas_video_renderer.cc:775: data = frame->visible_data(0);
Is there ever a situation where the video frame's data will already be stored in
a texture on the GPU, and the call to writeable_data() or visible_data() will
cause a GPU-to-CPU readback? Obviously, for performance reasons, it's crucial to
avoid such readbacks.

[aleksandar.stojiljkovic, 2016-08-16 12:02:31]

On 2016/08/16 00:22:27, Ken Russell wrote:
> It's encouraging to see your progress on this. A few questions.
Thanks. 
I'll move to OSX support and fix the issues you identified while cleaning the
code...

https://codereview.chromium.org/2121043002/diff/60001/cc/resources/resource_p...
File cc/resources/resource_provider.cc (right):

https://codereview.chromium.org/2121043002/diff/60001/cc/resources/resource_p...
cc/resources/resource_provider.cc:455: // LUMINANCE_8;
On 2016/08/16 00:22:26, Ken Russell wrote:
> Does this really use a luminance texture containing 16-bit floats? Luminance
> textures are deprecated in OpenGL ES 3.0. Did you try a GL_R16F type texture?

It doesn't. I just reused YUV path for the prototype. Will fix it.

https://codereview.chromium.org/2121043002/diff/60001/gpu/command_buffer/serv...
File gpu/command_buffer/service/texture_manager.cc (right):

https://codereview.chromium.org/2121043002/diff/60001/gpu/command_buffer/serv...
gpu/command_buffer/service/texture_manager.cc:174: {GL_R16UI, GL_R16UI,
GL_UNSIGNED_SHORT},
On 2016/08/16 00:22:27, Ken Russell wrote:
> This one isn't mentioned in the switch statement below. Is that deliberate?

Code change in this file isn't needed here. I'll remove it. Thanks.

On a related note, and it is relevant for the discussion here:

It was part of another prototype effort (copyTextureChromium for R16F and R32F)
but then I decided not to go that path - packing (copying) to skia format
RGBA16F happening in SkCanvasVideoRenderer::ConvertVideoFrameToRGBPixels looked
redundant.
So, the code is now using newly added WebMediaPlayerMS::texImageImpl.

For the reference, copyTextureChromium works fine on Windows for R16F when R16F
is color-renderable (when EXT_color_buffer_half_float&OES_texture_half_float are
present).

https://codereview.chromium.org/2121043002/diff/60001/media/renderers/skcanva...
File media/renderers/skcanvas_video_renderer.cc (right):

https://codereview.chromium.org/2121043002/diff/60001/media/renderers/skcanva...
media/renderers/skcanvas_video_renderer.cc:775: data = frame->visible_data(0);
On 2016/08/16 00:22:27, Ken Russell wrote:
> Is there ever a situation where the video frame's data will already be stored
in
> a texture on the GPU, and the call to writeable_data() or visible_data() will
> cause a GPU-to-CPU readback? Obviously, for performance reasons, it's crucial
to
> avoid such readbacks.

Of course. In this case VideoFrame::storage_type() is STORAGE_SHMEM.
If video_frame->HasTextures(), it would be handled using
copyVideoTextureToPlatformTexture, but there is a need to check that here.

[Ken Russell (switch to Gerrit), 2016-08-16 18:27:37]

https://codereview.chromium.org/2121043002/diff/60001/media/renderers/skcanva...
File media/renderers/skcanvas_video_renderer.cc (right):

https://codereview.chromium.org/2121043002/diff/60001/media/renderers/skcanva...
media/renderers/skcanvas_video_renderer.cc:775: data = frame->visible_data(0);
On 2016/08/16 12:02:31, aleksandar.stojiljkovic wrote:
> On 2016/08/16 00:22:27, Ken Russell wrote:
> > Is there ever a situation where the video frame's data will already be
stored
> in
> > a texture on the GPU, and the call to writeable_data() or visible_data()
will
> > cause a GPU-to-CPU readback? Obviously, for performance reasons, it's
crucial
> to
> > avoid such readbacks.
> 
> Of course. In this case VideoFrame::storage_type() is STORAGE_SHMEM.
> If video_frame->HasTextures(), it would be handled using
> copyVideoTextureToPlatformTexture, but there is a need to check that here.

OK. Do you plan to add this code path?

If this code path is added, is it practical to support a few types of packing
when copying from the source to the destination texture? For example, RG8 ->
R16UI? Or has that been the plan all along?

[aleksandar.stojiljkovic, 2016-08-17 07:08:58]

Description was changed from

==========
8 and 16 bpp video stream capture and render - Realsense R200 & SR300 support.

This is an initial code supporting depth and infrared camera devices capture and
rendering with Realsense R200 and SR300 camera on Windows.

The initial patch is used to get early feedback while working on it.
It Supports 8 and 16 bit depth and infrared video streams from R200 and SR300.
Verified to work on Windows 10.

Rendering video element:
========================
8 and 16bpp frames are uploaded to R and RG textures, respectively - for RG
textures
lower byte goes to 'R' and higher byte to 'G'.
Rendered output is original R/RG texture. Screenshots of demo are available on
crbug.com/624436 page.

RG88 path required custom bilinear filter in shaders.cc.

WebGL:
======
Upload video to texture:
- RGBA UNSIGNED_BYTE: only R component presents upper byte of 16 bit video,
- R16UI UNSIGNED_SHORT: 16 bits go to R,
- RG8 UNSIGNED_BYTE: upper byte to G, lower to R.

Getting full 16 bit data to javascript.
===========================
The approach:
1. on start, create framebuffer, attach (COLOR_ATTACHMENT0) R16UI texture to it.
2. In loop, when video is ready, after texImage2D video to R16UI texture, bind
   the framebuffer and call readpixels to Uint16Array. This way the data is
   accessible in javascript.

jsfiddle examples are available on Issue 624436 page.

2D canvas + getImageData returns 8 bit R in RGBA.

There are multiple TODO(astojilj) in the code. I'm going to address them in
follow up
patches here.

BUG=624436
==========

to

==========
8 and 16 bpp video stream capture and render - Realsense R200 & SR300 support.

This is an initial code supporting depth and infrared camera devices capture and
rendering with Realsense R200 and SR300 camera on Windows.

The initial patch is used to get early feedback while working on it.
It Supports 8 and 16 bit depth and infrared video streams from R200 and SR300.
Verified to work on Windows 10.

Rendering video element:
========================
8 and 16bpp frames are uploaded to R and RG textures, respectively - for RG
textures
lower byte goes to 'R' and higher byte to 'G'.
Rendered output is original R/RG texture. Screenshots of demo are available on
crbug.com/624436 page.

RG88 path required custom bilinear filter in shaders.cc.

WebGL:
======
Upload video to texture:
- RGBA UNSIGNED_BYTE: only R component presents upper byte of 16 bit video,
- R16UI UNSIGNED_SHORT: 16 bits go to R,
- RG8 UNSIGNED_BYTE: upper byte to G, lower to R.

Getting full 16 bit data to javascript.
===========================
The approach:
1. on start, create framebuffer, attach (COLOR_ATTACHMENT0) R16UI texture to it.
2. In loop, when video is ready, after texImage2D video to R16UI texture, bind
   the framebuffer and call readpixels to Uint16Array. This way the data is
   accessible in javascript.

jsfiddle examples are available on Issue 624436 page.

2D canvas + getImageData returns 8 bit R in RGBA.

There are multiple TODO(astojilj) in the code. I'm going to address them in
follow up
patches here.

BUG=624436
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.win:win_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel
==========

[aleksandar.stojiljkovic, 2016-08-17 07:09:00]

Description was changed from

==========
8 and 16 bpp video stream capture and render - Realsense R200 & SR300 support.

This is an initial code supporting depth and infrared camera devices capture and
rendering with Realsense R200 and SR300 camera on Windows.

The initial patch is used to get early feedback while working on it.
It Supports 8 and 16 bit depth and infrared video streams from R200 and SR300.
Verified to work on Windows 10.

Rendering video element:
========================
8 and 16bpp frames are uploaded to R and RG textures, respectively - for RG
textures
lower byte goes to 'R' and higher byte to 'G'.
Rendered output is original R/RG texture. Screenshots of demo are available on
crbug.com/624436 page.

RG88 path required custom bilinear filter in shaders.cc.

WebGL:
======
Upload video to texture:
- RGBA UNSIGNED_BYTE: only R component presents upper byte of 16 bit video,
- R16UI UNSIGNED_SHORT: 16 bits go to R,
- RG8 UNSIGNED_BYTE: upper byte to G, lower to R.

Getting full 16 bit data to javascript.
===========================
The approach:
1. on start, create framebuffer, attach (COLOR_ATTACHMENT0) R16UI texture to it.
2. In loop, when video is ready, after texImage2D video to R16UI texture, bind
   the framebuffer and call readpixels to Uint16Array. This way the data is
   accessible in javascript.

jsfiddle examples are available on Issue 624436 page.

2D canvas + getImageData returns 8 bit R in RGBA.

There are multiple TODO(astojilj) in the code. I'm going to address them in
follow up
patches here.

BUG=624436
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.win:win_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel
==========

to

==========
8 and 16 bpp video stream capture and render - Realsense R200 & SR300 support.

This is an initial code supporting depth and infrared camera devices capture and
rendering with Realsense R200 and SR300 camera on Windows.

The initial patch is used to get early feedback while working on it.
It Supports 8 and 16 bit depth and infrared video streams from R200 and SR300.
Verified to work on Windows 10.

Rendering video element:
========================
8 and 16bpp frames are uploaded to R and RG textures, respectively - for RG
textures
lower byte goes to 'R' and higher byte to 'G'.
Rendered output is original R/RG texture. Screenshots of demo are available on
crbug.com/624436 page.

RG88 path required custom bilinear filter in shaders.cc.

WebGL:
======
Upload video to texture:
- RGBA UNSIGNED_BYTE: only R component presents upper byte of 16 bit video,
- R16UI UNSIGNED_SHORT: 16 bits go to R,
- RG8 UNSIGNED_BYTE: upper byte to G, lower to R.

Getting full 16 bit data to javascript.
===========================
The approach:
1. on start, create framebuffer, attach (COLOR_ATTACHMENT0) R16UI texture to it.
2. In loop, when video is ready, after texImage2D video to R16UI texture, bind
   the framebuffer and call readpixels to Uint16Array. This way the data is
   accessible in javascript.

jsfiddle examples are available on Issue 624436 page.

2D canvas + getImageData returns 8 bit R in RGBA.

There are multiple TODO(astojilj) in the code. I'm going to address them in
follow up
patches here.

BUG=624436
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel
==========

[aleksandar.stojiljkovic, 2016-08-17 07:09:02]

Description was changed from

==========
8 and 16 bpp video stream capture and render - Realsense R200 & SR300 support.

This is an initial code supporting depth and infrared camera devices capture and
rendering with Realsense R200 and SR300 camera on Windows.

The initial patch is used to get early feedback while working on it.
It Supports 8 and 16 bit depth and infrared video streams from R200 and SR300.
Verified to work on Windows 10.

Rendering video element:
========================
8 and 16bpp frames are uploaded to R and RG textures, respectively - for RG
textures
lower byte goes to 'R' and higher byte to 'G'.
Rendered output is original R/RG texture. Screenshots of demo are available on
crbug.com/624436 page.

RG88 path required custom bilinear filter in shaders.cc.

WebGL:
======
Upload video to texture:
- RGBA UNSIGNED_BYTE: only R component presents upper byte of 16 bit video,
- R16UI UNSIGNED_SHORT: 16 bits go to R,
- RG8 UNSIGNED_BYTE: upper byte to G, lower to R.

Getting full 16 bit data to javascript.
===========================
The approach:
1. on start, create framebuffer, attach (COLOR_ATTACHMENT0) R16UI texture to it.
2. In loop, when video is ready, after texImage2D video to R16UI texture, bind
   the framebuffer and call readpixels to Uint16Array. This way the data is
   accessible in javascript.

jsfiddle examples are available on Issue 624436 page.

2D canvas + getImageData returns 8 bit R in RGBA.

There are multiple TODO(astojilj) in the code. I'm going to address them in
follow up
patches here.

BUG=624436
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel
==========

to

==========
8 and 16 bpp video stream capture and render - Realsense R200 & SR300 support.

This is an initial code supporting depth and infrared camera devices capture and
rendering with Realsense R200 and SR300 camera on Windows.

The initial patch is used to get early feedback while working on it.
It Supports 8 and 16 bit depth and infrared video streams from R200 and SR300.
Verified to work on Windows 10.

Rendering video element:
========================
8 and 16bpp frames are uploaded to R and RG textures, respectively - for RG
textures
lower byte goes to 'R' and higher byte to 'G'.
Rendered output is original R/RG texture. Screenshots of demo are available on
crbug.com/624436 page.

RG88 path required custom bilinear filter in shaders.cc.

WebGL:
======
Upload video to texture:
- RGBA UNSIGNED_BYTE: only R component presents upper byte of 16 bit video,
- R16UI UNSIGNED_SHORT: 16 bits go to R,
- RG8 UNSIGNED_BYTE: upper byte to G, lower to R.

Getting full 16 bit data to javascript.
===========================
The approach:
1. on start, create framebuffer, attach (COLOR_ATTACHMENT0) R16UI texture to it.
2. In loop, when video is ready, after texImage2D video to R16UI texture, bind
   the framebuffer and call readpixels to Uint16Array. This way the data is
   accessible in javascript.

jsfiddle examples are available on Issue 624436 page.

2D canvas + getImageData returns 8 bit R in RGBA.

There are multiple TODO(astojilj) in the code. I'm going to address them in
follow up
patches here.

BUG=624436
CQ_INCLUDE_TRYBOTS=master.tryserver.blink:linux_precise_blink_rel
==========

[aleksandar.stojiljkovic, 2016-08-18 22:29:14]

On 2016/08/16 18:27:37, Ken Russell wrote:
>
https://codereview.chromium.org/2121043002/diff/60001/media/renderers/skcanva...
> File media/renderers/skcanvas_video_renderer.cc (right):
> 
>
https://codereview.chromium.org/2121043002/diff/60001/media/renderers/skcanva...
> media/renderers/skcanvas_video_renderer.cc:775: data = frame->visible_data(0);
> On 2016/08/16 12:02:31, aleksandar.stojiljkovic wrote:
> > On 2016/08/16 00:22:27, Ken Russell wrote:
> > > Is there ever a situation where the video frame's data will already be
> stored
> > in
> > > a texture on the GPU, and the call to writeable_data() or visible_data()
> will
> > > cause a GPU-to-CPU readback? Obviously, for performance reasons, it's
> crucial
> > to
> > > avoid such readbacks.
> > 
> > Of course. In this case VideoFrame::storage_type() is STORAGE_SHMEM.
> > If video_frame->HasTextures(), it would be handled using
> > copyVideoTextureToPlatformTexture, but there is a need to check that here.
> 
> OK. Do you plan to add this code path?
> 
> If this code path is added, is it practical to support a few types of packing
> when copying from the source to the destination texture? For example, RG8 ->
> R16UI? Or has that been the plan all along?
Yes, the plan is to support both.
dshwang@ already started work on adding support for RG8 format here [2], though
the patch is extending to cover also replacing replace LUMINANCE_F16 by RG8.
However, since using GpuMemoryBuffers for Video Capture is still experimental
[1], this work started with shared memory. 

[1]
https://cs.chromium.org/chromium/src/media/base/media_switches.cc?rcl=1471534...
[2]
https://codereview.chromium.org/2122573003


I don't know yet what is available to get R16UI texture from RG8 texture.
Guess we cannot use glTextureView/ARB_texture_view.
Obvious choice would be to do the same as for YUV planes (conversion on GPU) in
[3]

[3]
https://cs.chromium.org/chromium/src/third_party/WebKit/Source/modules/webgl/...
 

Anyway, didn't try it yet - started work on it and will upload the code here
once it is ready.
Thanks.

[aleksandar.stojiljkovic, 2016-09-20 11:19:22]

Description was changed from

==========
8 and 16 bpp video stream capture and render - Realsense R200 & SR300 support.

This is an initial code supporting depth and infrared camera devices capture and
rendering with Realsense R200 and SR300 camera on Windows.

The initial patch is used to get early feedback while working on it.
It Supports 8 and 16 bit depth and infrared video streams from R200 and SR300.
Verified to work on Windows 10.

Rendering video element:
========================
8 and 16bpp frames are uploaded to R and RG textures, respectively - for RG
textures
lower byte goes to 'R' and higher byte to 'G'.
Rendered output is original R/RG texture. Screenshots of demo are available on
crbug.com/624436 page.

RG88 path required custom bilinear filter in shaders.cc.

WebGL:
======
Upload video to texture:
- RGBA UNSIGNED_BYTE: only R component presents upper byte of 16 bit video,
- R16UI UNSIGNED_SHORT: 16 bits go to R,
- RG8 UNSIGNED_BYTE: upper byte to G, lower to R.

Getting full 16 bit data to javascript.
===========================
The approach:
1. on start, create framebuffer, attach (COLOR_ATTACHMENT0) R16UI texture to it.
2. In loop, when video is ready, after texImage2D video to R16UI texture, bind
   the framebuffer and call readpixels to Uint16Array. This way the data is
   accessible in javascript.

jsfiddle examples are available on Issue 624436 page.

2D canvas + getImageData returns 8 bit R in RGBA.

There are multiple TODO(astojilj) in the code. I'm going to address them in
follow up
patches here.

BUG=624436
CQ_INCLUDE_TRYBOTS=master.tryserver.blink:linux_precise_blink_rel
==========

to

==========
8 and 16 bpp video stream capture and render - Realsense R200 & SR300 support.

This is an initial code supporting depth and infrared camera devices capture and
rendering with Realsense R200 and SR300 camera on Windows/Linux/ChromeOS.

The initial patch is used to get early feedback while working on it.
It Supports 8 and 16 bit depth and infrared video streams from R200 and SR300.
Verified to work on Windows 10 and Ubuntu 16.04.

Rendering video element:
========================
8 and 16bpp frames are uploaded to R and RG textures, respectively - for RG
textures
lower byte goes to 'R' and higher byte to 'G'.
Rendered output is original R/RG texture. Screenshots of demo are available on
crbug.com/624436 page.

RG88 path required custom bilinear filter in shaders.cc.

WebGL:
======
Upload video to texture:
- RGBA UNSIGNED_BYTE: only R component presents upper byte of 16 bit video,
- R16UI UNSIGNED_SHORT: 16 bits go to R,
- RG8 UNSIGNED_BYTE: upper byte to G, lower to R.
- R32F FLOAT: Normalized value.

Getting full 16 bit or float data to javascript:
===============================================
1. on start, create framebuffer, attach (COLOR_ATTACHMENT0) R32F/R16UI texture
to it.
2. In loop, when video is ready, after texImage2D video to R32F/R16UI texture,
bind
   the framebuffer and call readpixels to Uint16Array. This way the data is
   accessible in javascript.

Note: R32F works on all the platforms, R16UI only on Windows (not supported when
gpu memory buffers are enabled).

Complete examples are available on Issue 624436 page.

2D canvas + getImageData returns 8 bit R in RGBA.

There are multiple TODO(astojilj) in the code. I'm going to address them in
follow up
patches here.

BUG=624436
==========

[aleksandar.stojiljkovic, 2016-09-20 11:19:23]

aleksandar.stojiljkovic@intel.com changed reviewers:
+ sandersd@chromium.org

[aleksandar.stojiljkovic, 2016-09-20 11:20:25]

aleksandar.stojiljkovic@intel.com changed reviewers:
+ jbauman@chromium.org

[aleksandar.stojiljkovic, 2016-09-20 11:25:46]

aleksandar.stojiljkovic@intel.com changed reviewers:
+ reveman@chromium.org

[aleksandar.stojiljkovic, 2016-09-20 12:22:55]

Patch Set 6 : gpu memory buffers support, reinterpret RG8->R32F on GPU, Linux
and ChromeOS support added.

CC-ing additional reviewers:
+reveman@
for gpu memory buffer related changes and cc.

+jbauman@
for gpu/command_buffer/* and skcanvas_video_renderer.cc.

+sandersd@ for all changes in media.

+enne@
for cc/quads.

Went astray few times while doing gpu memory buffer support. Wanted to have it
working on OSX so I started adding libusb based capture device (no other driver
available on OSX for realsense) before figuring out that modified fake capture
device can be used. I think libusb based capture support could be useful, but
let's do it in later patches. 

Added some of the code from dshwang@'s:
https://codereview.chromium.org/2122573003/
The patch here is not replacing LUMINANCE_F16 by RG_88, it is only using RG_88
for 16 bit video.

Disclaimer: didn't verify GPU memory buffer support on ChromeOS, only on OSX
with modified fake capture device. Shared memory buffer path works on ChromeOS.

I was using manual tests (jsfibble pages on the bug pages). Plan to add tests in
the next patch.

https://codereview.chromium.org/2121043002/diff/60001/cc/resources/resource_p...
File cc/resources/resource_provider.cc (right):

https://codereview.chromium.org/2121043002/diff/60001/cc/resources/resource_p...
cc/resources/resource_provider.cc:455: // LUMINANCE_8;
On 2016/08/16 00:22:26, Ken Russell wrote:
> Does this really use a luminance texture containing 16-bit floats? Luminance
> textures are deprecated in OpenGL ES 3.0. Did you try a GL_R16F type texture?

If software planes (shared memory) are used, the case when there is no
ext_texture_rg support is disabled now (by not supporting 16 bit video in
VideoResourceUpdater::CreateForSoftwarePlanes). So, we support it for ES3 and
for ES2 when ext_texture_rg extension is available.

https://codereview.chromium.org/2121043002/diff/60001/gpu/command_buffer/serv...
File gpu/command_buffer/service/texture_manager.cc (right):

https://codereview.chromium.org/2121043002/diff/60001/gpu/command_buffer/serv...
gpu/command_buffer/service/texture_manager.cc:174: {GL_R16UI, GL_R16UI,
GL_UNSIGNED_SHORT},
On 2016/08/16 00:22:27, Ken Russell wrote:
> This one isn't mentioned in the switch statement below. Is that deliberate?

Done. As part of work on gpu buffers support, this got updated.

https://codereview.chromium.org/2121043002/diff/60001/media/renderers/skcanva...
File media/renderers/skcanvas_video_renderer.cc (right):

https://codereview.chromium.org/2121043002/diff/60001/media/renderers/skcanva...
media/renderers/skcanvas_video_renderer.cc:775: data = frame->visible_data(0);
On 2016/08/16 18:27:37, Ken Russell wrote:
> On 2016/08/16 12:02:31, aleksandar.stojiljkovic wrote:
> > On 2016/08/16 00:22:27, Ken Russell wrote:
> > > Is there ever a situation where the video frame's data will already be
> stored
> > in
> > > a texture on the GPU, and the call to writeable_data() or visible_data()
> will
> > > cause a GPU-to-CPU readback? Obviously, for performance reasons, it's
> crucial
> > to
> > > avoid such readbacks.
> > 
> > Of course. In this case VideoFrame::storage_type() is STORAGE_SHMEM.
> > If video_frame->HasTextures(), it would be handled using
> > copyVideoTextureToPlatformTexture, but there is a need to check that here.
> 
> OK. Do you plan to add this code path?
> 
> If this code path is added, is it practical to support a few types of packing
> when copying from the source to the destination texture? For example, RG8 ->
> R16UI? Or has that been the plan all along?

Done. Reinterpreting RG8 to R32F when storage is GPU native memory buffer is
done now on GPU in this file.

https://codereview.chromium.org/2121043002/diff/100001/media/renderers/skcanv...
File media/renderers/skcanvas_video_renderer.cc (right):

https://codereview.chromium.org/2121043002/diff/100001/media/renderers/skcanv...
media/renderers/skcanvas_video_renderer.cc:889:
DoCopyRG8ToRedTextureData(local_gl, source_texture, intermediate_texture,
On 2016/08/16 18:27:37, Ken Russell wrote:
>
https://codereview.chromium.org/2121043002/diff/60001/media/renderers/skcanva...
> File media/renderers/skcanvas_video_renderer.cc (right):
> 
>
https://codereview.chromium.org/2121043002/diff/60001/media/renderers/skcanva...
> media/renderers/skcanvas_video_renderer.cc:775: data = frame->visible_data(0);
> On 2016/08/16 12:02:31, aleksandar.stojiljkovic wrote:
> > On 2016/08/16 00:22:27, Ken Russell wrote:
> > > Is there ever a situation where the video frame's data will already be
> stored
> > in
> > > a texture on the GPU, and the call to writeable_data() or visible_data()
> will
> > > cause a GPU-to-CPU readback? Obviously, for performance reasons, it's
> crucial
> > to
> > > avoid such readbacks.
> > 
> > Of course. In this case VideoFrame::storage_type() is STORAGE_SHMEM.
> > If video_frame->HasTextures(), it would be handled using
> > copyVideoTextureToPlatformTexture, but there is a need to check that here.
> 
> OK. Do you plan to add this code path?
> 
> If this code path is added, is it practical to support a few types of packing
> when copying from the source to the destination texture? For example, RG8 ->
> R16UI? Or has that been the plan all along?
It is implemented here - 16 bit value laid out in two RG bytes is normalized and
converted (rendered) to R32F texture.
The approach follows YUV planes conversion approach - local canvas's GL is used
for conversion, output then promoted to destination GL and  CopyTextureCHROMIUM
used to copy to target texture.
+sandersd, +dshwang, +jbauman, 
could you please review this file changes? Thanks.

[enne (OOO), 2016-09-20 19:03:45]

Where is the testing for this feature? I would have expected to see a layout
test, a cc pixel test that would exercise these shaders, and also a gl renderer
shader program compilation test for the new shaders.

[aleksandar.stojiljkovic, 2016-09-20 20:04:06]

On 2016/09/20 19:03:45, enne wrote:
> Where is the testing for this feature? I would have expected to see a layout
> test, a cc pixel test that would exercise these shaders, and also a gl
renderer
> shader program compilation test for the new shaders.

enne@, thanks for help, let me add those in following patches.
As the feature is large, started review process early, while still prototyping
and early feedback from reviewers, here and in related patches, helped to steer
it.
I was using manual tests (jsfiddle pages mentioned in the bug comments) for
verification.

[enne (OOO), 2016-09-20 20:12:05]

Ok! Sorry, I misunderstood your previous message.  It looked like you were CCing
a bunch of folks for a review and I made the wrong assumption that this was
closer to a final state.  The cc end of things looks reasonable to me, other
than missing testing.

[jbauman, 2016-09-20 20:51:22]

https://codereview.chromium.org/2121043002/diff/100001/media/renderers/skcanv...
File media/renderers/skcanvas_video_renderer.cc (right):

https://codereview.chromium.org/2121043002/diff/100001/media/renderers/skcanv...
media/renderers/skcanvas_video_renderer.cc:951: rg8_to_red_program_ =
gl->CreateProgram();
I don't think the lifetime of this is correct if the main thread context is lost
(GPU process crashes, etc.). I'm not sure where to put it otherwise, though.
Previously I've put it into skia, where the GrContext has the correct lifetime.
Another option is to somehow put it into CopyTextureCHROMIUM.

[aleksandar.stojiljkovic, 2016-09-20 21:30:08]

On 2016/09/20 20:51:22, jbauman wrote:
>
https://codereview.chromium.org/2121043002/diff/100001/media/renderers/skcanv...
> File media/renderers/skcanvas_video_renderer.cc (right):
> 
>
https://codereview.chromium.org/2121043002/diff/100001/media/renderers/skcanv...
> media/renderers/skcanvas_video_renderer.cc:951: rg8_to_red_program_ =
> gl->CreateProgram();
> I don't think the lifetime of this is correct if the main thread context is
lost
> (GPU process crashes, etc.). I'm not sure where to put it otherwise, though.
> Previously I've put it into skia, where the GrContext has the correct
lifetime.
> Another option is to somehow put it into CopyTextureCHROMIUM.

Yes, it is not correct.
Noticed those "//GPU Process crashed" comments in webmediaplayer_ms.cc, e.g.

auto* provider =
  RenderThreadImpl::current()->SharedMainThreadContextProvider().get();
  // GPU Process crashed.
  if (!provider)
    return false; 

and plan to use provider->GrContext().uniqueId() to check if program is still
valid. As GrContext::abandonContext() is clearing created resources, I believe
there is no problem in keeping dangling program & shader ids. "Dangling" with
the meaning that there's no glDeleteProgram/glDeleteShader calls.

Initially started and then abandoned implementing this in CopyTextureChromium -
doesn't fit the requirements that color components are preserved there.
I also considered implementing another extension - CopyImageSubDataChromium that
would copy from RG6 to R16UI in gpu service process, but with skia context
support for restoring state after rendering and since it is possible to follow
approach of YUV conversion without changing skia bitmap type interfaces (No RG8,
R16UI, R32F support there) this is IMHO  
good approach.

So, I'll upload a patch using GrContext::uniqueID to verify cache/context
validity (also extend the resources cached) and then CC someone from skia to
verify. Thanks.

[aleksandar.stojiljkovic, 2016-09-30 12:20:26]

Description was changed from

==========
8 and 16 bpp video stream capture and render - Realsense R200 & SR300 support.

This is an initial code supporting depth and infrared camera devices capture and
rendering with Realsense R200 and SR300 camera on Windows/Linux/ChromeOS.

The initial patch is used to get early feedback while working on it.
It Supports 8 and 16 bit depth and infrared video streams from R200 and SR300.
Verified to work on Windows 10 and Ubuntu 16.04.

Rendering video element:
========================
8 and 16bpp frames are uploaded to R and RG textures, respectively - for RG
textures
lower byte goes to 'R' and higher byte to 'G'.
Rendered output is original R/RG texture. Screenshots of demo are available on
crbug.com/624436 page.

RG88 path required custom bilinear filter in shaders.cc.

WebGL:
======
Upload video to texture:
- RGBA UNSIGNED_BYTE: only R component presents upper byte of 16 bit video,
- R16UI UNSIGNED_SHORT: 16 bits go to R,
- RG8 UNSIGNED_BYTE: upper byte to G, lower to R.
- R32F FLOAT: Normalized value.

Getting full 16 bit or float data to javascript:
===============================================
1. on start, create framebuffer, attach (COLOR_ATTACHMENT0) R32F/R16UI texture
to it.
2. In loop, when video is ready, after texImage2D video to R32F/R16UI texture,
bind
   the framebuffer and call readpixels to Uint16Array. This way the data is
   accessible in javascript.

Note: R32F works on all the platforms, R16UI only on Windows (not supported when
gpu memory buffers are enabled).

Complete examples are available on Issue 624436 page.

2D canvas + getImageData returns 8 bit R in RGBA.

There are multiple TODO(astojilj) in the code. I'm going to address them in
follow up
patches here.

BUG=624436
==========

to

==========
8 and 16 bpp video stream capture and render - Realsense R200 & SR300 support.

This is an initial code supporting depth and infrared camera devices capture and
rendering with Realsense R200 and SR300 camera on Windows/Linux/ChromeOS.

The initial patch is used to get early feedback while working on it.
It Supports 8 and 16 bit depth and infrared video streams from R200 and SR300.
Verified to work on Windows 10 and Ubuntu 16.04.

Rendering video element:
========================
8 and 16bpp frames are uploaded to R and RG textures, respectively - for RG
textures
lower byte goes to 'R' and higher byte to 'G'.
Rendered output is original R/RG texture. Screenshots of demo are available on
crbug.com/624436 page.

RG88 path required custom bilinear filter in shaders.cc.

WebGL:
======
Upload video to texture:
- RGBA UNSIGNED_BYTE: only R component presents upper byte of 16 bit video,
- R16UI UNSIGNED_SHORT: 16 bits go to R,
- RG8 UNSIGNED_BYTE: upper byte to G, lower to R.
- R32F FLOAT: Normalized value.

Getting full 16 bit or float data to javascript:
===============================================
1. on start, create framebuffer, attach (COLOR_ATTACHMENT0) R32F/R16UI texture
to it.
2. In loop, when video is ready, after texImage2D video to R32F/R16UI texture,
bind
   the framebuffer and call readpixels to Uint16Array. This way the data is
   accessible in javascript.

Note: R32F works on all the platforms, R16UI only on Windows (not supported when
gpu memory buffers are enabled).

Complete examples are available on Issue 624436 page.

2D canvas + getImageData returns 8 bit R in RGBA.

There are multiple TODO(astojilj) in the code. I'm going to address them in
follow up
patches here.

BUG=624436
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.win:win_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel
==========

[aleksandar.stojiljkovic, 2016-09-30 12:20:28]

Description was changed from

==========
8 and 16 bpp video stream capture and render - Realsense R200 & SR300 support.

This is an initial code supporting depth and infrared camera devices capture and
rendering with Realsense R200 and SR300 camera on Windows/Linux/ChromeOS.

The initial patch is used to get early feedback while working on it.
It Supports 8 and 16 bit depth and infrared video streams from R200 and SR300.
Verified to work on Windows 10 and Ubuntu 16.04.

Rendering video element:
========================
8 and 16bpp frames are uploaded to R and RG textures, respectively - for RG
textures
lower byte goes to 'R' and higher byte to 'G'.
Rendered output is original R/RG texture. Screenshots of demo are available on
crbug.com/624436 page.

RG88 path required custom bilinear filter in shaders.cc.

WebGL:
======
Upload video to texture:
- RGBA UNSIGNED_BYTE: only R component presents upper byte of 16 bit video,
- R16UI UNSIGNED_SHORT: 16 bits go to R,
- RG8 UNSIGNED_BYTE: upper byte to G, lower to R.
- R32F FLOAT: Normalized value.

Getting full 16 bit or float data to javascript:
===============================================
1. on start, create framebuffer, attach (COLOR_ATTACHMENT0) R32F/R16UI texture
to it.
2. In loop, when video is ready, after texImage2D video to R32F/R16UI texture,
bind
   the framebuffer and call readpixels to Uint16Array. This way the data is
   accessible in javascript.

Note: R32F works on all the platforms, R16UI only on Windows (not supported when
gpu memory buffers are enabled).

Complete examples are available on Issue 624436 page.

2D canvas + getImageData returns 8 bit R in RGBA.

There are multiple TODO(astojilj) in the code. I'm going to address them in
follow up
patches here.

BUG=624436
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.win:win_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel
==========

to

==========
8 and 16 bpp video stream capture and render - Realsense R200 & SR300 support.

This is an initial code supporting depth and infrared camera devices capture and
rendering with Realsense R200 and SR300 camera on Windows/Linux/ChromeOS.

The initial patch is used to get early feedback while working on it.
It Supports 8 and 16 bit depth and infrared video streams from R200 and SR300.
Verified to work on Windows 10 and Ubuntu 16.04.

Rendering video element:
========================
8 and 16bpp frames are uploaded to R and RG textures, respectively - for RG
textures
lower byte goes to 'R' and higher byte to 'G'.
Rendered output is original R/RG texture. Screenshots of demo are available on
crbug.com/624436 page.

RG88 path required custom bilinear filter in shaders.cc.

WebGL:
======
Upload video to texture:
- RGBA UNSIGNED_BYTE: only R component presents upper byte of 16 bit video,
- R16UI UNSIGNED_SHORT: 16 bits go to R,
- RG8 UNSIGNED_BYTE: upper byte to G, lower to R.
- R32F FLOAT: Normalized value.

Getting full 16 bit or float data to javascript:
===============================================
1. on start, create framebuffer, attach (COLOR_ATTACHMENT0) R32F/R16UI texture
to it.
2. In loop, when video is ready, after texImage2D video to R32F/R16UI texture,
bind
   the framebuffer and call readpixels to Uint16Array. This way the data is
   accessible in javascript.

Note: R32F works on all the platforms, R16UI only on Windows (not supported when
gpu memory buffers are enabled).

Complete examples are available on Issue 624436 page.

2D canvas + getImageData returns 8 bit R in RGBA.

There are multiple TODO(astojilj) in the code. I'm going to address them in
follow up
patches here.

BUG=624436
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel
==========

[aleksandar.stojiljkovic, 2016-09-30 12:20:30]

Description was changed from

==========
8 and 16 bpp video stream capture and render - Realsense R200 & SR300 support.

This is an initial code supporting depth and infrared camera devices capture and
rendering with Realsense R200 and SR300 camera on Windows/Linux/ChromeOS.

The initial patch is used to get early feedback while working on it.
It Supports 8 and 16 bit depth and infrared video streams from R200 and SR300.
Verified to work on Windows 10 and Ubuntu 16.04.

Rendering video element:
========================
8 and 16bpp frames are uploaded to R and RG textures, respectively - for RG
textures
lower byte goes to 'R' and higher byte to 'G'.
Rendered output is original R/RG texture. Screenshots of demo are available on
crbug.com/624436 page.

RG88 path required custom bilinear filter in shaders.cc.

WebGL:
======
Upload video to texture:
- RGBA UNSIGNED_BYTE: only R component presents upper byte of 16 bit video,
- R16UI UNSIGNED_SHORT: 16 bits go to R,
- RG8 UNSIGNED_BYTE: upper byte to G, lower to R.
- R32F FLOAT: Normalized value.

Getting full 16 bit or float data to javascript:
===============================================
1. on start, create framebuffer, attach (COLOR_ATTACHMENT0) R32F/R16UI texture
to it.
2. In loop, when video is ready, after texImage2D video to R32F/R16UI texture,
bind
   the framebuffer and call readpixels to Uint16Array. This way the data is
   accessible in javascript.

Note: R32F works on all the platforms, R16UI only on Windows (not supported when
gpu memory buffers are enabled).

Complete examples are available on Issue 624436 page.

2D canvas + getImageData returns 8 bit R in RGBA.

There are multiple TODO(astojilj) in the code. I'm going to address them in
follow up
patches here.

BUG=624436
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel
==========

to

==========
8 and 16 bpp video stream capture and render - Realsense R200 & SR300 support.

This is an initial code supporting depth and infrared camera devices capture and
rendering with Realsense R200 and SR300 camera on Windows/Linux/ChromeOS.

The initial patch is used to get early feedback while working on it.
It Supports 8 and 16 bit depth and infrared video streams from R200 and SR300.
Verified to work on Windows 10 and Ubuntu 16.04.

Rendering video element:
========================
8 and 16bpp frames are uploaded to R and RG textures, respectively - for RG
textures
lower byte goes to 'R' and higher byte to 'G'.
Rendered output is original R/RG texture. Screenshots of demo are available on
crbug.com/624436 page.

RG88 path required custom bilinear filter in shaders.cc.

WebGL:
======
Upload video to texture:
- RGBA UNSIGNED_BYTE: only R component presents upper byte of 16 bit video,
- R16UI UNSIGNED_SHORT: 16 bits go to R,
- RG8 UNSIGNED_BYTE: upper byte to G, lower to R.
- R32F FLOAT: Normalized value.

Getting full 16 bit or float data to javascript:
===============================================
1. on start, create framebuffer, attach (COLOR_ATTACHMENT0) R32F/R16UI texture
to it.
2. In loop, when video is ready, after texImage2D video to R32F/R16UI texture,
bind
   the framebuffer and call readpixels to Uint16Array. This way the data is
   accessible in javascript.

Note: R32F works on all the platforms, R16UI only on Windows (not supported when
gpu memory buffers are enabled).

Complete examples are available on Issue 624436 page.

2D canvas + getImageData returns 8 bit R in RGBA.

There are multiple TODO(astojilj) in the code. I'm going to address them in
follow up
patches here.

BUG=624436
CQ_INCLUDE_TRYBOTS=master.tryserver.blink:linux_precise_blink_rel
==========

[aleksandar.stojiljkovic, 2016-09-30 14:26:32]

Patch Set 7 : Tests: cc, skcanvas_video_renderer, wrtcrecorder... Fake capture
supports Y16

The patch adds tests or enables support for 16bpp Y16 format in existing tests:
- cc_unittests: pixeltest & drawquadtest,
- media_unittests: test lifecycle management of cached GPU resources when
GrContext is lost/restored.
- capture_unittests:
  Fake capture device now provides 2 devices by default- Y16 is second - so some
of the tests run using this device.
- content_browsertest: WebRtcMediaRecorderTest (html page using Y16 capture from
fake capture device and recording it, but not a  pixel test).
- content_unittests: capture buffer_pool, controller and client unit tests.

I need a suggestion where to put a layout test that would use fake capture
device (command line activation on launch) and WebGL canvas on the page.
content_browsertests folder, or to blink layout tests, or to some other place?

Thanks.

Started splitting the patch to smaller patches, will provide links later.

https://codereview.chromium.org/2121043002/diff/100001/media/renderers/skcanv...
File media/renderers/skcanvas_video_renderer.cc (right):

https://codereview.chromium.org/2121043002/diff/100001/media/renderers/skcanv...
media/renderers/skcanvas_video_renderer.cc:951: rg8_to_red_program_ =
gl->CreateProgram();
On 2016/09/20 20:51:22, jbauman wrote:
> I don't think the lifetime of this is correct if the main thread context is
lost
> (GPU process crashes, etc.). I'm not sure where to put it otherwise, though.
> Previously I've put it into skia, where the GrContext has the correct
lifetime.
> Another option is to somehow put it into CopyTextureCHROMIUM.

Done.
Using GrContext::uniqueID to control the lifecycle of cached resources.
Added unit test to cover this:
SkCanvasVideoRendererTest, Y16CachedGLResourcesContextRestore
https://codereview.chromium.org/2121043002/diff/120001/media/renderers/skcanv...

https://codereview.chromium.org/2121043002/diff/120001/cc/output/renderer_pix...
File cc/output/renderer_pixeltest.cc (right):

https://codereview.chromium.org/2121043002/diff/120001/cc/output/renderer_pix...
cc/output/renderer_pixeltest.cc:1079: TYPED_TEST(IntersectingQuadGLPixelTest,
YVideoQuads) {
On 2016/09/20 19:03:45, enne wrote:
> Where is the testing for this feature? I would have expected to see a layout
> test, a cc pixel test that would exercise these shaders, and also a gl
renderer
> shader program compilation test for the new shaders.
cc pixel test:
Intersecting (and two color rendering) test added here.
enne@, do we need edge bleed implementation for Y16 video? Thanks.
Not sure if we need to clamp sampling coordinates to visible rect in bilinear
sampling for Y16.

[Ken Russell (switch to Gerrit), 2016-09-30 23:01:30]

There were quite a few issues in the command buffer changes so I stopped after
going through them. When you've resolved these and feel that the Blink side
changes are ready for review, please follow up on the CL.

https://codereview.chromium.org/2121043002/diff/120001/gpu/command_buffer/com...
File gpu/command_buffer/common/gles2_cmd_utils.cc (right):

https://codereview.chromium.org/2121043002/diff/120001/gpu/command_buffer/com...
gpu/command_buffer/common/gles2_cmd_utils.cc:1094: return GL_FLOAT;
I think this change is actually wrong. For these floating-point formats the
default supported ReadPixels type is GL_FLOAT. This function gives the
"alternate", or "implementation", read type, and the command buffer typically
chooses to make this GL_UNSIGNED_BYTE. Hopefully this change would have caused
some tests to fail (at least some in the WebGL conformance suite). If not then
more tests should be added.

https://codereview.chromium.org/2121043002/diff/120001/gpu/command_buffer/ser...
File gpu/command_buffer/service/feature_info.cc (right):

https://codereview.chromium.org/2121043002/diff/120001/gpu/command_buffer/ser...
gpu/command_buffer/service/feature_info.cc:1292: }
Instead of adding these values in separate if-statements, could you move the
block above (regarding  feature_flags_.ext_texture_rg) above the block higher
starting with "if (enable_texture_storage)"?

That way these could just be folded into the existing tests during the enabling
of the texture_storage extension (where they ostensibly belong).

https://codereview.chromium.org/2121043002/diff/120001/gpu/command_buffer/ser...
File gpu/command_buffer/service/gles2_cmd_copy_texture_chromium.cc (right):

https://codereview.chromium.org/2121043002/diff/120001/gpu/command_buffer/ser...
gpu/command_buffer/service/gles2_cmd_copy_texture_chromium.cc:435:
!GL::IsIntegerFormat(dest_internal_format)));
This addition to the code is very highly specialized to your use case. Can you
simplify and generalize it, and still keep it working for your uses?

https://codereview.chromium.org/2121043002/diff/120001/gpu/command_buffer/ser...
File gpu/command_buffer/service/gles2_cmd_decoder.cc (right):

https://codereview.chromium.org/2121043002/diff/120001/gpu/command_buffer/ser...
gpu/command_buffer/service/gles2_cmd_decoder.cc:10602:
accepted_types.push_back(GL_FLOAT);
Where is it specified that ReadPixels with type GL_RED and type GL_FLOAT is
legal if a GL_R32F color buffer is bound? Reading the ES 3.0.4 spec,
https://www.khronos.org/registry/gles/extensions/EXT/EXT_color_buffer_float.txt
, and https://www.khronos.org/registry/gles/extensions/EXT/EXT_texture_rg.txt, I
don't see what makes this a legal readback format/type combination. I do see
that GL_RGBA/GL_FLOAT would be legal per EXT_color_buffer_float.

https://codereview.chromium.org/2121043002/diff/120001/gpu/command_buffer/ser...
gpu/command_buffer/service/gles2_cmd_decoder.cc:15027:
feature_info_->feature_flags().ext_texture_rg);
EXT_texture_rg doesn't provide the GL_R16UI format. That's only supported by
ES3. The logic here which gates the support for these internal formats must be
correct.

https://codereview.chromium.org/2121043002/diff/120001/gpu/command_buffer/ser...
gpu/command_buffer/service/gles2_cmd_decoder.cc:15038: source_internal_format ==
GL_RG_EXT ||
Why is support for GL_RG_EXT here not gated by
feature_info_->feature_flags().ext_texture_rg? Is it because if the source
texture was allocated successfully with that format, it means that feature test
already happened elsewhere?

https://codereview.chromium.org/2121043002/diff/120001/gpu/command_buffer/ser...
gpu/command_buffer/service/gles2_cmd_decoder.cc:15041:
validators_->texture_internal_format.IsValid(source_internal_format));
Why does this check of source_internal_format against the validators_ have to be
done? Is it because the texture may have been mailboxed over into our context,
so this context's share group's validators might not support the source internal
format?

https://codereview.chromium.org/2121043002/diff/120001/gpu/command_buffer/ser...
gpu/command_buffer/service/gles2_cmd_decoder.cc:15044: // It is, for now,
checked in Extensions3DUtil::canUseCopyTextureCHROMIUM.
This is a pretty significant TODO. Could you please work on resolving it?

https://codereview.chromium.org/2121043002/diff/120001/gpu/command_buffer/ser...
gpu/command_buffer/service/gles2_cmd_decoder.cc:15047:
validators_->texture_internal_format.IsValid(dest_internal_format);
Why is this logic separate from the computation of valid_dest_format above?
Could you at least move it closer to the other code?

https://codereview.chromium.org/2121043002/diff/120001/gpu/command_buffer/ser...
File gpu/command_buffer/service/texture_manager.cc (right):

https://codereview.chromium.org/2121043002/diff/120001/gpu/command_buffer/ser...
gpu/command_buffer/service/texture_manager.cc:169: {GL_R16F, GL_R16F,
GL_HALF_FLOAT},
These entries are flat-out wrong. Please remove them.

https://codereview.chromium.org/2121043002/diff/120001/gpu/command_buffer/ser...
gpu/command_buffer/service/texture_manager.cc:2911: switch (format) {
This mapping back of illegal formats to correct ones is not OK. Please figure
out some other solution that doesn't involve making this change.

[aleksandar.stojiljkovic, 2016-10-03 10:56:53]

Patch Set 9 : Review #36 fix. Thanks kbr@.
Resolved by reverting the code.
Decided to edit the approach and drop webgl2 & R32F approach.
WebGL1 via RGB/RGBA & FLOAT works just fine.
Related to R16UI/R32F - we can just add it later, once this patch gets in and
webgl2 is there by default.
In details:
Apologize for the time waste on reviewing this.
It turned out that most of these were exploratory hacks I did to make the
prototype work with experimental WebGL2 and R16UI and R32F - so I reverted them
all.
The problem was that (in addition to code being new to me)  rendering command
buffer context is still es2, even webgl2 context and --enable-unsafe-es3-apis in
command line are used. Texture data needs to pass through multiple contexts,
some of them are GLES3 capable some are not.

https://codereview.chromium.org/2121043002/diff/120001/gpu/command_buffer/com...
File gpu/command_buffer/common/gles2_cmd_utils.cc (right):

https://codereview.chromium.org/2121043002/diff/120001/gpu/command_buffer/com...
gpu/command_buffer/common/gles2_cmd_utils.cc:1094: return GL_FLOAT;
On 2016/09/30 23:01:29, Ken Russell wrote:
> I think this change is actually wrong. For these floating-point formats the
> default supported ReadPixels type is GL_FLOAT. This function gives the
> "alternate", or "implementation", read type, and the command buffer typically
> chooses to make this GL_UNSIGNED_BYTE. Hopefully this change would have caused
> some tests to fail (at least some in the WebGL conformance suite). If not then
> more tests should be added.

Done. Wasn't important. GL_IMPLEMENTATION_COLOR_READ_XXXX behavior on OSX
(driver returns UNSIGNED_BYTE) differed from Windows&Linux (on those I believe
driver returned FLOAT).

https://codereview.chromium.org/2121043002/diff/120001/gpu/command_buffer/ser...
File gpu/command_buffer/service/feature_info.cc (right):

https://codereview.chromium.org/2121043002/diff/120001/gpu/command_buffer/ser...
gpu/command_buffer/service/feature_info.cc:1292: }
tOn 2016/09/30 23:01:29, Ken Russell wrote:
> Instead of adding these values in separate if-statements, could you move the
> block above (regarding  feature_flags_.ext_texture_rg) above the block higher
> starting with "if (enable_texture_storage)"?
> 
> That way these could just be folded into the existing tests during the
enabling
> of the texture_storage extension (where they ostensibly belong).

Done. For future reference (if submitting patch related to this), the code is
wrong - texture_internal_format_storage validator should be modified, not
texture_internal_format.

https://codereview.chromium.org/2121043002/diff/120001/gpu/command_buffer/ser...
File gpu/command_buffer/service/gles2_cmd_copy_texture_chromium.cc (right):

https://codereview.chromium.org/2121043002/diff/120001/gpu/command_buffer/ser...
gpu/command_buffer/service/gles2_cmd_copy_texture_chromium.cc:435:
!GL::IsIntegerFormat(dest_internal_format)));
On 2016/09/30 23:01:29, Ken Russell wrote:
> This addition to the code is very highly specialized to your use case. Can you
> simplify and generalize it, and still keep it working for your uses?
> 

Done. This would be useful later - edited out from this patch as not WebGL1
related.

[Ken Russell (switch to Gerrit), 2016-10-05 01:27:28]

Thanks for revising this patch. I only reviewed a few parts; gpu/command_buffer,
ui/gl, and third_party/WebKit; and they look good overall. Could you indicate
whether you think this is ready to be fully reviewed at this point?

[aleksandar.stojiljkovic, 2016-10-05 03:57:33]

Patch Set 10 : rebase after 2376293003 land.

On 2016/10/05 01:27:28, Ken Russell wrote:
> Thanks for revising this patch. I only reviewed a few parts;
gpu/command_buffer,
> ui/gl, and third_party/WebKit; and they look good overall. Could you indicate
> whether you think this is ready to be fully reviewed at this point?

Thanks.
After 2376293003: land (some of the code was duplicated) here and I could split
the patch to "capture and gpu buffers", cc and WebKit patch and I would send
them for review in that order.
If you saw something critical missing in WebKit code (I still need to add layout
tests - have the code but not sure where to put it in folder structure), please
advise.

[aleksandar.stojiljkovic, 2016-10-05 04:08:00]

Capture (content&media) patch: https://codereview.chromium.org/2398463003 

On 2016/10/05 03:57:33, aleksandar.stojiljkovic wrote:
> Patch Set 10 : rebase after 2376293003 land.
> 
> On 2016/10/05 01:27:28, Ken Russell wrote:
> > Thanks for revising this patch. I only reviewed a few parts;
> gpu/command_buffer,
> > ui/gl, and third_party/WebKit; and they look good overall. Could you
indicate
> > whether you think this is ready to be fully reviewed at this point?
> 
> Thanks.
> After 2376293003: land (some of the code was duplicated) here and I could
split
> the patch to "capture and gpu buffers", cc and WebKit patch and I would send
> them for review in that order.
> If you saw something critical missing in WebKit code (I still need to add
layout
> tests - have the code but not sure where to put it in folder structure),
please
> advise.

[Ken Russell (switch to Gerrit), 2016-10-05 19:38:40]

On 2016/10/05 04:08:00, aleksandar.stojiljkovic wrote:
> Capture (content&media) patch: https://codereview.chromium.org/2398463003 
> 
> On 2016/10/05 03:57:33, aleksandar.stojiljkovic wrote:
> > Patch Set 10 : rebase after 2376293003 land.
> > 
> > On 2016/10/05 01:27:28, Ken Russell wrote:
> > > Thanks for revising this patch. I only reviewed a few parts;
> > gpu/command_buffer,
> > > ui/gl, and third_party/WebKit; and they look good overall. Could you
> indicate
> > > whether you think this is ready to be fully reviewed at this point?
> > 
> > Thanks.
> > After 2376293003: land (some of the code was duplicated) here and I could
> split
> > the patch to "capture and gpu buffers", cc and WebKit patch and I would send
> > them for review in that order.
> > If you saw something critical missing in WebKit code (I still need to add
> layout
> > tests - have the code but not sure where to put it in folder structure),
> please
> > advise.

The Blink-side code looks good. Look at
src/third_party/WebKit/LayoutTests/VirtualTestSuites for examples of which
subdirectories of layout tests are run with which command line arguments. I'm
not sure how this code will be tested at that level -- will there be a way for
you to load up some pre-recorded file from a RealSense camera for testing
purposes? Regardless, we can probably put the tests under media/ or
fast/canvas/webgl/ .

We should also consider adding tests to the WebGL conformance suite at
github.com/KhronosGroup/WebGL .

[aleksandar.stojiljkovic, 2016-10-07 15:10:04]

aleksandar.stojiljkovic@intel.com changed reviewers:
+ dcastagna@chromium.org, tommi@chromium.org

[aleksandar.stojiljkovic, 2016-10-07 15:10:05]

Patch Set 11 : scoping Y8 out.

On 2016/10/05 19:38:40, Ken Russell wrote:
> On 2016/10/05 04:08:00, aleksandar.stojiljkovic wrote:
> > Capture (content&media) patch: https://codereview.chromium.org/2398463003 
> > 
> > On 2016/10/05 03:57:33, aleksandar.stojiljkovic wrote:
> > > Patch Set 10 : rebase after 2376293003 land.
> > > 
> > > On 2016/10/05 01:27:28, Ken Russell wrote:
> > > > Thanks for revising this patch. I only reviewed a few parts;
> > > gpu/command_buffer,
> > > > ui/gl, and third_party/WebKit; and they look good overall. Could you
> > indicate
> > > > whether you think this is ready to be fully reviewed at this point?
> > > 
> > > Thanks.
> > > After 2376293003: land (some of the code was duplicated) here and I could
> > split
> > > the patch to "capture and gpu buffers", cc and WebKit patch and I would
send
> > > them for review in that order.
> > > If you saw something critical missing in WebKit code (I still need to add
> > layout
> > > tests - have the code but not sure where to put it in folder structure),
> > please
> > > advise.
> 
> The Blink-side code looks good. Look at
> src/third_party/WebKit/LayoutTests/VirtualTestSuites for examples of which
> subdirectories of layout tests are run with which command line arguments. I'm
> not sure how this code will be tested at that level -- will there be a way for
> you to load up some pre-recorded file from a RealSense camera for testing
> purposes? Regardless, we can probably put the tests under media/ or
> fast/canvas/webgl/ .
> 
> We should also consider adding tests to the WebGL conformance suite at
> github.com/KhronosGroup/WebGL .

Thanks. Working on it.
To edit code for review, removing prototyped Y8 support.

My idea to split capture part to separate patch [1] failed -
content_browsertests are using fake capture device and rendering Y16 video to 2D
canvas (e.g. WebRtcGetUserMediaBrowserTest.GetUserMediaWithMandatorySourceID).
So, back to effort on making this whole patch going through review.

Some areas have tests and can be reviewed:

+tommi@, mcasas@: 
content/browser/renderer_host/media/*
media/capture/*

+dcastagna@
hubbe@ advised to ask dcastagna@chromium.org for Owner review of
media/video/gpu_memory_buffer_video_frame_pool.cc.
PTAL.

Thanks

[1] 
crrev.com/2398463003 16 bit capture and GPU&CPU memory buffer support.

[aleksandar.stojiljkovic, 2016-10-07 15:11:20]

Description was changed from

==========
8 and 16 bpp video stream capture and render - Realsense R200 & SR300 support.

This is an initial code supporting depth and infrared camera devices capture and
rendering with Realsense R200 and SR300 camera on Windows/Linux/ChromeOS.

The initial patch is used to get early feedback while working on it.
It Supports 8 and 16 bit depth and infrared video streams from R200 and SR300.
Verified to work on Windows 10 and Ubuntu 16.04.

Rendering video element:
========================
8 and 16bpp frames are uploaded to R and RG textures, respectively - for RG
textures
lower byte goes to 'R' and higher byte to 'G'.
Rendered output is original R/RG texture. Screenshots of demo are available on
crbug.com/624436 page.

RG88 path required custom bilinear filter in shaders.cc.

WebGL:
======
Upload video to texture:
- RGBA UNSIGNED_BYTE: only R component presents upper byte of 16 bit video,
- R16UI UNSIGNED_SHORT: 16 bits go to R,
- RG8 UNSIGNED_BYTE: upper byte to G, lower to R.
- R32F FLOAT: Normalized value.

Getting full 16 bit or float data to javascript:
===============================================
1. on start, create framebuffer, attach (COLOR_ATTACHMENT0) R32F/R16UI texture
to it.
2. In loop, when video is ready, after texImage2D video to R32F/R16UI texture,
bind
   the framebuffer and call readpixels to Uint16Array. This way the data is
   accessible in javascript.

Note: R32F works on all the platforms, R16UI only on Windows (not supported when
gpu memory buffers are enabled).

Complete examples are available on Issue 624436 page.

2D canvas + getImageData returns 8 bit R in RGBA.

There are multiple TODO(astojilj) in the code. I'm going to address them in
follow up
patches here.

BUG=624436
CQ_INCLUDE_TRYBOTS=master.tryserver.blink:linux_precise_blink_rel
==========

to

==========
16 bpp video stream capture, render and WebGL usage - Realsense R200 & SR300
support.

It Supports 16 bit depth and infrared video streams from R200 and SR300.
Verified to work on Windows 10 and Ubuntu 16.04.

Rendering video element:
========================
16bpp frames are uploaded to RG textures, respectively - for RG textures
lower byte goes to 'R' and higher byte to 'G'.
Rendered output is original RG texture. Screenshots of demo are available on
crbug.com/624436 page.

RG88 path required custom bilinear filter in shaders.cc.

WebGL:
======
Upload video to texture:
In all the textures, R, G, B components have the same value (as luminance).
- RGBA UNSIGNED_BYTE: R, G and B components presents upper byte of 16 bit video,
  so, with precision loss.
- RGB/RGBA FLOAT: Normalized 16 bit value in R, G and B component - no precision
loss.

Getting full 16 bit or float data to javascript:
===============================================
1. on start, create framebuffer, attach (COLOR_ATTACHMENT0) RGBA FLOAT texture
to it.
2. In loop, when video is ready, after texImage2D video to the texture, bind
   the framebuffer and call readpixels to FloatArray. This way the data is8 and
16 bpp video stream capture and render - Realsense R200 & SR300 support.

This is an initial code supporting depth and infrared camera devices capture and
rendering with Realsense R200 and SR300 camera on Windows/Linux/ChromeOS.

The initial patch is used to get early feedback while working on it.
It Supports 8 and 16 bit depth and infrared video streams from R200 and SR300.
Verified to work on Windows 10 and Ubuntu 16.04.

Rendering video element:
========================
8 and 16bpp frames are uploaded to R and RG textures, respectively - for RG
textures
lower byte goes to 'R' and higher byte to 'G'.
Rendered output is original R/RG texture. Screenshots of demo are available on
crbug.com/624436 page.

RG88 path required custom bilinear filter in shaders.cc.

WebGL:
======
Upload video to texture:
- RGBA UNSIGNED_BYTE: only R component presents upper byte of 16 bit video,
- R16UI UNSIGNED_SHORT: 16 bits go to R,
- RG8 UNSIGNED_BYTE: upper byte to G, lower to R.
- R32F FLOAT: Normalized value.

Getting full 16 bit or float data to javascript:
===============================================
1. on start, create framebuffer, attach (COLOR_ATTACHMENT0) R32F/R16UI texture
to it.
2. In loop, when video is ready, after texImage2D video to R32F/R16UI texture,
bind
   the framebuffer and call readpixels to Uint16Array. This way the data is
   accessible in javascript.

Note: R32F works on all the platforms, R16UI only on Windows (not supported when
gpu memory buffers are enabled).

Complete examples are available on Issue 624436 page.

2D canvas + getImageData returns 8 bit R in RGBA.

There are multiple TODO(astojilj) in the code. I'm going to address them in
follow up
patches here.

BUG=624436
CQ_INCLUDE_TRYBOTS=master.tryserver.blink:linux_precise_blink_rel
   accessible from javascript.

Complete examples are available on Issue 624436 page.

2D canvas + getImageData returns 8 bit color components (higher bit value) in
RGBA.

BUG=624436
==========

[aleksandar.stojiljkovic, 2016-10-07 15:11:54]

Description was changed from

==========
16 bpp video stream capture, render and WebGL usage - Realsense R200 & SR300
support.

It Supports 16 bit depth and infrared video streams from R200 and SR300.
Verified to work on Windows 10 and Ubuntu 16.04.

Rendering video element:
========================
16bpp frames are uploaded to RG textures, respectively - for RG textures
lower byte goes to 'R' and higher byte to 'G'.
Rendered output is original RG texture. Screenshots of demo are available on
crbug.com/624436 page.

RG88 path required custom bilinear filter in shaders.cc.

WebGL:
======
Upload video to texture:
In all the textures, R, G, B components have the same value (as luminance).
- RGBA UNSIGNED_BYTE: R, G and B components presents upper byte of 16 bit video,
  so, with precision loss.
- RGB/RGBA FLOAT: Normalized 16 bit value in R, G and B component - no precision
loss.

Getting full 16 bit or float data to javascript:
===============================================
1. on start, create framebuffer, attach (COLOR_ATTACHMENT0) RGBA FLOAT texture
to it.
2. In loop, when video is ready, after texImage2D video to the texture, bind
   the framebuffer and call readpixels to FloatArray. This way the data is8 and
16 bpp video stream capture and render - Realsense R200 & SR300 support.

This is an initial code supporting depth and infrared camera devices capture and
rendering with Realsense R200 and SR300 camera on Windows/Linux/ChromeOS.

The initial patch is used to get early feedback while working on it.
It Supports 8 and 16 bit depth and infrared video streams from R200 and SR300.
Verified to work on Windows 10 and Ubuntu 16.04.

Rendering video element:
========================
8 and 16bpp frames are uploaded to R and RG textures, respectively - for RG
textures
lower byte goes to 'R' and higher byte to 'G'.
Rendered output is original R/RG texture. Screenshots of demo are available on
crbug.com/624436 page.

RG88 path required custom bilinear filter in shaders.cc.

WebGL:
======
Upload video to texture:
- RGBA UNSIGNED_BYTE: only R component presents upper byte of 16 bit video,
- R16UI UNSIGNED_SHORT: 16 bits go to R,
- RG8 UNSIGNED_BYTE: upper byte to G, lower to R.
- R32F FLOAT: Normalized value.

Getting full 16 bit or float data to javascript:
===============================================
1. on start, create framebuffer, attach (COLOR_ATTACHMENT0) R32F/R16UI texture
to it.
2. In loop, when video is ready, after texImage2D video to R32F/R16UI texture,
bind
   the framebuffer and call readpixels to Uint16Array. This way the data is
   accessible in javascript.

Note: R32F works on all the platforms, R16UI only on Windows (not supported when
gpu memory buffers are enabled).

Complete examples are available on Issue 624436 page.

2D canvas + getImageData returns 8 bit R in RGBA.

There are multiple TODO(astojilj) in the code. I'm going to address them in
follow up
patches here.

BUG=624436
CQ_INCLUDE_TRYBOTS=master.tryserver.blink:linux_precise_blink_rel
   accessible from javascript.

Complete examples are available on Issue 624436 page.

2D canvas + getImageData returns 8 bit color components (higher bit value) in
RGBA.

BUG=624436
==========

to

==========
16 bpp video stream capture, render and WebGL usage - Realsense R200 & SR300
support.

It Supports 16 bit depth and infrared video streams from R200 and SR300.
Verified to work on Windows 10 and Ubuntu 16.04.

Rendering video element:
========================
16bpp frames are uploaded to RG textures, respectively - for RG textures
lower byte goes to 'R' and higher byte to 'G'.
Rendered output is original RG texture. Screenshots of demo are available on
crbug.com/624436 page.

RG88 path required custom bilinear filter in shaders.cc.

WebGL:
======
Upload video to texture:
In all the textures, R, G, B components have the same value (as luminance).
- RGBA UNSIGNED_BYTE: R, G and B components presents upper byte of 16 bit video,
  so, with precision loss.
- RGB/RGBA FLOAT: Normalized 16 bit value in R, G and B component - no precision
loss.

Getting full 16 bit or float data to javascript:
===============================================
1. on start, create framebuffer, attach (COLOR_ATTACHMENT0) RGBA FLOAT texture
to it.
2. In loop, when video is ready, after texImage2D video to the texture, bind
   the framebuffer and call readpixels to FloatArray. This way the data is
   accessible from javascript.

Complete examples are available on Issue 624436 page.

2D canvas + getImageData returns 8 bit color components (higher bit value) in
RGBA.

BUG=624436
==========

[Daniele Castagna, 2016-10-10 18:42:31]

https://codereview.chromium.org/2121043002/diff/200001/media/video/gpu_memory...
File media/video/gpu_memory_buffer_video_frame_pool.cc (right):

https://codereview.chromium.org/2121043002/diff/200001/media/video/gpu_memory...
media/video/gpu_memory_buffer_video_frame_pool.cc:162: VideoPixelFormat
output_format_;
This is changing slightly with crrev.com/2382443008 and we'll have to rebase it.

Could you please extract this changes in its own CL and test it in
gpu_memory_buffer_video_frame_pool_unittest.cc ?

[aleksandar.stojiljkovic, 2016-10-11 18:29:54]

https://codereview.chromium.org/2121043002/diff/200001/media/video/gpu_memory...
File media/video/gpu_memory_buffer_video_frame_pool.cc (right):

https://codereview.chromium.org/2121043002/diff/200001/media/video/gpu_memory...
media/video/gpu_memory_buffer_video_frame_pool.cc:162: VideoPixelFormat
output_format_;
On 2016/10/10 18:42:31, Daniele Castagna wrote:
> This is changing slightly with crrev.com/2382443008 and we'll have to rebase
it.
> 
> Could you please extract this changes in its own CL and test it in
> gpu_memory_buffer_video_frame_pool_unittest.cc ?

Done.
CL split to:
crrev.com/2410923002: GpuMemoryBufferVideoFramePool: Y16 support.
Except new format enum, patches look orthogonal - shouldn't be a problem to
merge them.

[aleksandar.stojiljkovic, 2016-10-18 20:45:33]

ccameron@'s non-lgtm on misusing RG88 for 16 bit plane in YUV :
https://crrev.com/2122573003

ccameron@ raised important concern affecting this and split patches too:

On 2016/10/17 22:28:31, ccameron wrote:
> This approach does not lgtm. We should not simulate 1=channel images using
> 2-channel images.
> 
> It is possible to create R16 unorm GMBs. We should use these. I verified this
on
> Mac with IOSurfaces, and I can send you a test application which demonstrates
> this.
> 
> Alternatively, unless we prove that the short<->half-float conversion is the
> critical bottleneck for a critical use-case, then we should consider using
R16F
> (half-float single channel).

Could you please clarify the concern, why it is not ok to use RG88 for 16
bit plane and in which cases it could be considered as a hack?
Is is about lower layers loosing the semantics or the need for custom bilinear
filtering?

Measurements: 
Did some measurements here, comparing CPU, GPU power consumption for conversion,
glTexImage and rendering for float, half_float and RG88.
https://bugs.chromium.org/p/chromium/issues/detail?id=624436#c28

And the conclusion was that RG88 used for Depth16 internally only is acceptable
(hack) if we hide this as implementation detail from end user...
End user shouldn't see the value split to color components. So, this patch is
handling that, too,... I mean converting RG88 to float on GPU in
SkCanvasVideoRenderer or doing it on CPU when shared memory data gets texImage2D
to float texture.

In a way, float would be preferred as users could use WebGL1 with no precision
loss to get the data available on GPU (FLOAT RGBA texture) or CPU (from
javascript accessible through gl.readPixels from texture color-attached to
framebuffer).

IOSurface R16 or R16UI is not usable here - OSX is not a target platform now for
Realsense cameras or Kinect2 camera. In later plases, once libusb backend is
added to capture, we could use it.

If using RG88 for 16 bit depth is not fine, this patch would get simplified and
implemented on the same way on all of the target platforms (Win, ChromeOS,
Linux) using shared memory CPU buffers. I'll do it that way than - split this
patch to minimal support (using CPU shared memory) and consider GPU buffer
format as an optimization for patches.
Thanks ccameron@.

[aleksandar.stojiljkovic, 2016-10-19 12:27:22]

Description was changed from

==========
16 bpp video stream capture, render and WebGL usage - Realsense R200 & SR300
support.

It Supports 16 bit depth and infrared video streams from R200 and SR300.
Verified to work on Windows 10 and Ubuntu 16.04.

Rendering video element:
========================
16bpp frames are uploaded to RG textures, respectively - for RG textures
lower byte goes to 'R' and higher byte to 'G'.
Rendered output is original RG texture. Screenshots of demo are available on
crbug.com/624436 page.

RG88 path required custom bilinear filter in shaders.cc.

WebGL:
======
Upload video to texture:
In all the textures, R, G, B components have the same value (as luminance).
- RGBA UNSIGNED_BYTE: R, G and B components presents upper byte of 16 bit video,
  so, with precision loss.
- RGB/RGBA FLOAT: Normalized 16 bit value in R, G and B component - no precision
loss.

Getting full 16 bit or float data to javascript:
===============================================
1. on start, create framebuffer, attach (COLOR_ATTACHMENT0) RGBA FLOAT texture
to it.
2. In loop, when video is ready, after texImage2D video to the texture, bind
   the framebuffer and call readpixels to FloatArray. This way the data is
   accessible from javascript.

Complete examples are available on Issue 624436 page.

2D canvas + getImageData returns 8 bit color components (higher bit value) in
RGBA.

BUG=624436
==========

to

==========
16 bpp video stream capture, render and WebGL usage - Realsense R200 & SR300
support.

It Supports 16 bit depth and infrared video streams from R200 and SR300.
Verified to work on Windows 10 and Ubuntu 16.04.

Rendering video element:
========================
16bpp frames are uploaded to RG textures, respectively - for RG textures
lower byte goes to 'R' and higher byte to 'G'.
Rendered output is original RG texture. Screenshots of demo are available on
crbug.com/624436 page.

RG88 path required custom bilinear filter in shaders.cc.

WebGL:
======
Upload video to texture:
In all the textures, R, G, B components have the same value (as luminance).
- RGBA UNSIGNED_BYTE: R, G and B components presents upper byte of 16 bit video,
  so, with precision loss.
- RGB/RGBA FLOAT: Normalized 16 bit value in R, G and B component - no precision
loss.

Getting full 16 bit or float data to javascript:
===============================================
1. on start, create framebuffer, attach (COLOR_ATTACHMENT0) RGBA FLOAT texture
to it.
2. In loop, when video is ready, after texImage2D video to the texture, bind
   the framebuffer and call readpixels to FloatArray. This way the data is
   accessible from javascript.

Complete examples are available on Issue 624436 page.

2D canvas + getImageData returns 8 bit color components (higher bit value) in
RGBA.

BUG=624436
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.win:win_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel
==========

[aleksandar.stojiljkovic, 2016-10-19 12:27:24]

Description was changed from

==========
16 bpp video stream capture, render and WebGL usage - Realsense R200 & SR300
support.

It Supports 16 bit depth and infrared video streams from R200 and SR300.
Verified to work on Windows 10 and Ubuntu 16.04.

Rendering video element:
========================
16bpp frames are uploaded to RG textures, respectively - for RG textures
lower byte goes to 'R' and higher byte to 'G'.
Rendered output is original RG texture. Screenshots of demo are available on
crbug.com/624436 page.

RG88 path required custom bilinear filter in shaders.cc.

WebGL:
======
Upload video to texture:
In all the textures, R, G, B components have the same value (as luminance).
- RGBA UNSIGNED_BYTE: R, G and B components presents upper byte of 16 bit video,
  so, with precision loss.
- RGB/RGBA FLOAT: Normalized 16 bit value in R, G and B component - no precision
loss.

Getting full 16 bit or float data to javascript:
===============================================
1. on start, create framebuffer, attach (COLOR_ATTACHMENT0) RGBA FLOAT texture
to it.
2. In loop, when video is ready, after texImage2D video to the texture, bind
   the framebuffer and call readpixels to FloatArray. This way the data is
   accessible from javascript.

Complete examples are available on Issue 624436 page.

2D canvas + getImageData returns 8 bit color components (higher bit value) in
RGBA.

BUG=624436
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.win:win_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel
==========

to

==========
16 bpp video stream capture, render and WebGL usage - Realsense R200 & SR300
support.

It Supports 16 bit depth and infrared video streams from R200 and SR300.
Verified to work on Windows 10 and Ubuntu 16.04.

Rendering video element:
========================
16bpp frames are uploaded to RG textures, respectively - for RG textures
lower byte goes to 'R' and higher byte to 'G'.
Rendered output is original RG texture. Screenshots of demo are available on
crbug.com/624436 page.

RG88 path required custom bilinear filter in shaders.cc.

WebGL:
======
Upload video to texture:
In all the textures, R, G, B components have the same value (as luminance).
- RGBA UNSIGNED_BYTE: R, G and B components presents upper byte of 16 bit video,
  so, with precision loss.
- RGB/RGBA FLOAT: Normalized 16 bit value in R, G and B component - no precision
loss.

Getting full 16 bit or float data to javascript:
===============================================
1. on start, create framebuffer, attach (COLOR_ATTACHMENT0) RGBA FLOAT texture
to it.
2. In loop, when video is ready, after texImage2D video to the texture, bind
   the framebuffer and call readpixels to FloatArray. This way the data is
   accessible from javascript.

Complete examples are available on Issue 624436 page.

2D canvas + getImageData returns 8 bit color components (higher bit value) in
RGBA.

BUG=624436
CQ_INCLUDE_TRYBOTS=master.tryserver.blink:linux_precise_blink_rel
==========

[aleksandar.stojiljkovic, 2016-10-19 12:29:12]

The CQ bit was checked by aleksandar.stojiljkovic@intel.com to run a CQ dry run

[commit-bot: I haz the power, 2016-10-19 12:30:01]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-10-19 12:33:42]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-10-19 12:33:44]

Dry run: Try jobs failed on following builders:
  mac_chromium_compile_dbg_ng on master.tryserver.chromium.mac (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_comp...)
  mac_chromium_rel_ng on master.tryserver.chromium.mac (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_...)

[aleksandar.stojiljkovic, 2016-10-19 19:17:34]

The CQ bit was checked by aleksandar.stojiljkovic@intel.com to run a CQ dry run

[commit-bot: I haz the power, 2016-10-19 19:18:33]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-10-19 19:57:05]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-10-19 19:57:07]

Dry run: Try jobs failed on following builders:
  chromium_presubmit on master.tryserver.chromium.linux (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/chromium_presub...)

[Ken Russell (switch to Gerrit), 2016-10-20 00:47:29]

On 2016/10/18 20:45:33, aleksandar.stojiljkovic wrote:
> ccameron@'s non-lgtm on misusing RG88 for 16 bit plane in YUV :
> https://crrev.com/2122573003
> 
> ccameron@ raised important concern affecting this and split patches too:
> 
> On 2016/10/17 22:28:31, ccameron wrote:
> > This approach does not lgtm. We should not simulate 1=channel images using
> > 2-channel images.
> > 
> > It is possible to create R16 unorm GMBs. We should use these. I verified
this
> on
> > Mac with IOSurfaces, and I can send you a test application which
demonstrates
> > this.
> > 
> > Alternatively, unless we prove that the short<->half-float conversion is the
> > critical bottleneck for a critical use-case, then we should consider using
> R16F
> > (half-float single channel).
> 
> Could you please clarify the concern, why it is not ok to use RG88 for 16
> bit plane and in which cases it could be considered as a hack?
> Is is about lower layers loosing the semantics or the need for custom bilinear
> filtering?
> 
> Measurements: 
> Did some measurements here, comparing CPU, GPU power consumption for
conversion,
> glTexImage and rendering for float, half_float and RG88.
> https://bugs.chromium.org/p/chromium/issues/detail?id=624436#c28
> 
> And the conclusion was that RG88 used for Depth16 internally only is
acceptable
> (hack) if we hide this as implementation detail from end user...
> End user shouldn't see the value split to color components. So, this patch is
> handling that, too,... I mean converting RG88 to float on GPU in
> SkCanvasVideoRenderer or doing it on CPU when shared memory data gets
texImage2D
> to float texture.
> 
> In a way, float would be preferred as users could use WebGL1 with no precision
> loss to get the data available on GPU (FLOAT RGBA texture) or CPU (from
> javascript accessible through gl.readPixels from texture color-attached to
> framebuffer).
> 
> IOSurface R16 or R16UI is not usable here - OSX is not a target platform now
for
> Realsense cameras or Kinect2 camera. In later plases, once libusb backend is
> added to capture, we could use it.
> 
> If using RG88 for 16 bit depth is not fine, this patch would get simplified
and
> implemented on the same way on all of the target platforms (Win, ChromeOS,
> Linux) using shared memory CPU buffers. I'll do it that way than - split this
> patch to minimal support (using CPU shared memory) and consider GPU buffer
> format as an optimization for patches.
> Thanks ccameron@.

I'd like to focus on the highest performance path. If developers can keep all of
the data on the GPU by using RG88 textures then I think we should enable that
possibility. I pushed back on ccameron's review on
https://codereview.chromium.org/2122573003 . Let's see what the resolution is.

[aleksandar.stojiljkovic, 2016-10-20 21:37:06]

On 2016/10/20 00:47:29, Ken Russell wrote:
> On 2016/10/18 20:45:33, aleksandar.stojiljkovic wrote:
> > ccameron@'s non-lgtm on misusing RG88 for 16 bit plane in YUV :
> > https://crrev.com/2122573003
> > 
> > ccameron@ raised important concern affecting this and split patches too:
> > 
> > On 2016/10/17 22:28:31, ccameron wrote:
> > > This approach does not lgtm. We should not simulate 1=channel images using
> > > 2-channel images.
> > > 
> > > It is possible to create R16 unorm GMBs. We should use these. I verified
> this
> > on
> > > Mac with IOSurfaces, and I can send you a test application which
> demonstrates
> > > this.
> > > 
> > > Alternatively, unless we prove that the short<->half-float conversion is
the
> > > critical bottleneck for a critical use-case, then we should consider using
> > R16F
> > > (half-float single channel).
> > 
> > Could you please clarify the concern, why it is not ok to use RG88 for 16
> > bit plane and in which cases it could be considered as a hack?
> > Is is about lower layers loosing the semantics or the need for custom
bilinear
> > filtering?
> > 
> > Measurements: 
> > Did some measurements here, comparing CPU, GPU power consumption for
> conversion,
> > glTexImage and rendering for float, half_float and RG88.
> > https://bugs.chromium.org/p/chromium/issues/detail?id=624436#c28
> > 
> > And the conclusion was that RG88 used for Depth16 internally only is
> acceptable
> > (hack) if we hide this as implementation detail from end user...
> > End user shouldn't see the value split to color components. So, this patch
is
> > handling that, too,... I mean converting RG88 to float on GPU in
> > SkCanvasVideoRenderer or doing it on CPU when shared memory data gets
> texImage2D
> > to float texture.
> > 
> > In a way, float would be preferred as users could use WebGL1 with no
precision
> > loss to get the data available on GPU (FLOAT RGBA texture) or CPU (from
> > javascript accessible through gl.readPixels from texture color-attached to
> > framebuffer).
> > 
> > IOSurface R16 or R16UI is not usable here - OSX is not a target platform now
> for
> > Realsense cameras or Kinect2 camera. In later plases, once libusb backend is
> > added to capture, we could use it.
> > 
> > If using RG88 for 16 bit depth is not fine, this patch would get simplified
> and
> > implemented on the same way on all of the target platforms (Win, ChromeOS,
> > Linux) using shared memory CPU buffers. I'll do it that way than - split
this
> > patch to minimal support (using CPU shared memory) and consider GPU buffer
> > format as an optimization for patches.
> > Thanks ccameron@.
> 
> I'd like to focus on the highest performance path. If developers can keep all
of
> the data on the GPU by using RG88 textures then I think we should enable that
> possibility. I pushed back on ccameron's review on
> https://codereview.chromium.org/2122573003 . Let's see what the resolution is.

Thanks a lot.
In order to get the patch reviewed faster, I've split the CPU/shared memory
support and tests here:
https://crrev.com/2428263004: 16 bpp video stream capture, render and WebGL
usage using (CPU) shared memory buffers

There is significantly less code to review than here and the code is simpler.
Since there is no GPU buffer support anyway on Windows, I hope we can get the
code in sooner with https://crrev.com/2428263004.
The plan is to to rebase this one later to it.