Pepper: add PPB_VideoEncoder interface

Pepper: add PPB_VideoEncoder interface

The purpose of this extension is to give the ability for applications
using the Pepper API to encode video frames drawn using OpenGL. A
typical use case would be to encode frames composed of different video
streams and/or graphic elements.

The application is expected to draw into offscreen textures and send
these textures to the encoder through the PPB_VideoEncoder
interface. The encoder then returns bitstream buffers using
PPB_Buffer_Dev. These buffers have to be discarded once the
application has processed them. The application will be notified
through the initialization step of how many textures need to be
allocated in order to feed the encoder at all time. The encoder will
notify the application once a video texture has been processed and can
be reused by the application.

This new PPB_VideoEncoder interface tries to reproduce the same
patterns as the PPB_VideoDecoder interface.

BUG=390964

[llandwerlin-old, 2014-07-04 10:36:38]

Here is a CL just for the PPB_VideoEncoder API.

[bbudge, 2014-07-04 16:22:17]

Just a few high level comments based on the issue description and the IDL file.

1) With PPB_VideoDecoder, we went to some trouble to not expose the shared
memory buffers to the plugin, so PPB_Buffer_Dev didn't have to be made a stable
API. Instead, the proxy creates and manages the buffers and accepts data from
the plugin's regular memory buffer. In your case, this should be the opposite:
the plugin should receive encoded buffers as a pointer and size (into a shared
memory buffer in a pool managed by the proxy code.)
e2) PPB_VideoDecoder has two functions that call back to the plugin, Decode and
GetPicture, rather than having an explicit PictureReady function pointer. Could
your API do something similar, with Encode and GetBitstreamBuffer functions? It
would be nice if these two APIs matched each other that way.

[llandwerlin-old, 2014-07-04 16:46:40]

On 2014/07/04 16:22:17, bbudge (OOO July 1-8) wrote:
> Just a few high level comments based on the issue description and the IDL
file.
> 
> 1) With PPB_VideoDecoder, we went to some trouble to not expose the shared
> memory buffers to the plugin, so PPB_Buffer_Dev didn't have to be made a
stable
> API. Instead, the proxy creates and manages the buffers and accepts data from
> the plugin's regular memory buffer. In your case, this should be the opposite:
> the plugin should receive encoded buffers as a pointer and size (into a shared
> memory buffer in a pool managed by the proxy code.)

Sure.
So no more RecycleBitstreamBuffer() and the application is expected to copy the
buffer passed to the callback. The buffer is only valid for the duration of the
call to the GetBitstreamBuffer() callback.

> e2) PPB_VideoDecoder has two functions that call back to the plugin, Decode
and
> GetPicture, rather than having an explicit PictureReady function pointer.
Could
> your API do something similar, with Encode and GetBitstreamBuffer functions?
It
> would be nice if these two APIs matched each other that way.

Would you like to see the GetDismissedPicture() call disappear? And instead,
have
the Encode() callback to be fired once the picture is available again?

Thanks

[bbudge, 2014-07-09 17:10:37]

On 2014/07/04 16:46:40, llandwerlin wrote:
> On 2014/07/04 16:22:17, bbudge (OOO July 1-8) wrote:
> > Just a few high level comments based on the issue description and the IDL
> file.
> > 
> > 1) With PPB_VideoDecoder, we went to some trouble to not expose the shared
> > memory buffers to the plugin, so PPB_Buffer_Dev didn't have to be made a
> stable
> > API. Instead, the proxy creates and manages the buffers and accepts data
from
> > the plugin's regular memory buffer. In your case, this should be the
opposite:
> > the plugin should receive encoded buffers as a pointer and size (into a
shared
> > memory buffer in a pool managed by the proxy code.)
> 
> Sure.
> So no more RecycleBitstreamBuffer() and the application is expected to copy
the
> buffer passed to the callback. The buffer is only valid for the duration of
the
> call to the GetBitstreamBuffer() callback.
> 
> > e2) PPB_VideoDecoder has two functions that call back to the plugin, Decode
> and
> > GetPicture, rather than having an explicit PictureReady function pointer.
> Could
> > your API do something similar, with Encode and GetBitstreamBuffer functions?
> It
> > would be nice if these two APIs matched each other that way.
> 
> Would you like to see the GetDismissedPicture() call disappear? And instead,
> have
> the Encode() callback to be fired once the picture is available again?

Yes, this is how the decoder API keeps the plugin from pushing too much data.

> 
> Thanks

[llandwerlin-old, 2014-07-09 17:17:42]

On 2014/07/09 17:10:37, bbudge (OOO July 1-8) wrote:
> 
> Yes, this is how the decoder API keeps the plugin from pushing too much data.
> 

Ok, good. Though I realized that getting rid of RecycleBitstreamBuffer() is
tricky.
Because of the way the plugin's callbacks are fired to avoid reentrancy.
I'll update the patchset reintroducing that method.

I'm also reworking the renderer side to have a VideoEncoderShim like you did a
VideoDecoderShim.

Let me know if that's ok, thanks.

[bbudge, 2014-07-09 17:26:43]

On 2014/07/09 17:17:42, llandwerlin wrote:
> On 2014/07/09 17:10:37, bbudge (OOO July 1-8) wrote:
> > 
> > Yes, this is how the decoder API keeps the plugin from pushing too much
data.
> > 
> 
> Ok, good. Though I realized that getting rid of RecycleBitstreamBuffer() is
> tricky.
> Because of the way the plugin's callbacks are fired to avoid reentrancy.
> I'll update the patchset reintroducing that method.

I haven't had a chance to really think this through. It does seem unnecessary,
since the plugin only needs to copy the buffer, unlike the decoder, which needs
to render the texture.

In general we tried to make the decoder API easy to use and foolproof for the
plugin author.

> 
> I'm also reworking the renderer side to have a VideoEncoderShim like you did a
> VideoDecoderShim.
> 
> Let me know if that's ok, thanks.

That sounds good. Thanks for tackling this!

[llandwerlin-old, 2014-07-15 18:58:56]

On 2014/07/09 17:26:43, bbudge wrote:
> On 2014/07/09 17:17:42, llandwerlin wrote:
> > On 2014/07/09 17:10:37, bbudge (OOO July 1-8) wrote:
> > > 
> 
> I haven't had a chance to really think this through. It does seem unnecessary,
> since the plugin only needs to copy the buffer, unlike the decoder, which
needs
> to render the texture.
> 
> In general we tried to make the decoder API easy to use and foolproof for the
> plugin author.
> 

Last patchset introduce the PPP_VideoEncoder interface that the plugin instance
needs to implement in order to receive the bitstream buffers.
Buffers are now entirely managed by the internal proxy resource and from the
developer's point of view, the handed buffers should only be considered valid
for the duration of the vfunc call.

I have the implementation living in a branch on top of this set, ready to be
uploaded if you want to have a look at it.

[dmichael (off chromium), 2014-07-15 20:39:07]

On 2014/07/15 18:58:56, llandwerlin wrote:
> On 2014/07/09 17:26:43, bbudge wrote:
> > On 2014/07/09 17:17:42, llandwerlin wrote:
> > > On 2014/07/09 17:10:37, bbudge (OOO July 1-8) wrote:
> > > > 
> > 
> > I haven't had a chance to really think this through. It does seem
unnecessary,
> > since the plugin only needs to copy the buffer, unlike the decoder, which
> needs
> > to render the texture.
> > 
> > In general we tried to make the decoder API easy to use and foolproof for
the
> > plugin author.
> > 
> 
> Last patchset introduce the PPP_VideoEncoder interface that the plugin
instance
> needs to implement in order to receive the bitstream buffers.
> Buffers are now entirely managed by the internal proxy resource and from the
> developer's point of view, the handed buffers should only be considered valid
> for the duration of the vfunc call.
> 
> I have the implementation living in a branch on top of this set, ready to be
> uploaded if you want to have a look at it.
I'm sorry, we're not using PPP interfaces that way anymore. It has a few
problems, key among them being that the plugin must handle callbacks on the main
pepper thread. With PP_CompletionCallback, they can receive the callback on any
thread they choose (or even do a blocking PPB call if calling from a background
thread).

I haven't looked carefully at the API. If it turns out that you really do need a
PPP interface, it should instead be modeled after the dev-channel PPB_Messaging
and PPP_MessageHandler, where the PPP interface is passed as a parameter. But
maybe bbudge will have good advice as to whether a PPP interface or completion
callbacks is better.

[bbudge, 2014-07-15 20:53:06]

Kind of implicit in my comment (2) was that PPB_VideoDecoder avoids the use of a
PPP_ interface or callback function by having Push / Pull functions, Decode and
GetPicture calls. I was imagining you could have Encode and GetBitstreamBuffer
functions to match this.

This works pretty well. The plugin pushes stream data with Decode and pulls
pictures with GetPicture. The only strangeness is when the plugin calls Flush.
It continues to call GetPicture and we signal end of video with a
PP_ERROR_ABORTED result.

I don't see a reason why you can't do the same here.

[llandwerlin-old, 2014-07-15 21:54:53]

On 2014/07/15 20:53:06, bbudge wrote:
> Kind of implicit in my comment (2) was that PPB_VideoDecoder avoids the use of
a
> PPP_ interface or callback function by having Push / Pull functions, Decode
and
> GetPicture calls. I was imagining you could have Encode and GetBitstreamBuffer
> functions to match this.
> 
> This works pretty well. The plugin pushes stream data with Decode and pulls
> pictures with GetPicture. The only strangeness is when the plugin calls Flush.
> It continues to call GetPicture and we signal end of video with a
> PP_ERROR_ABORTED result.
> 
> I don't see a reason why you can't do the same here.

Ok. I'm having trouble to understand how this is going to work for the encoder
case.

The push/pull functions assume a transfer of ownership of the textures in the
decoder API.
In (2) you asked for the proxy to create and manage the buffers. That implies
the proxy needs to know when the buffer is not used by the plugin anymore. I'm
struggling to see how this works when the plugin can receive callbacks on any
thread.

Maybe I'm missing some technicality here.
Thanks for looking into this.

[bbudge, 2014-07-16 15:52:54]

On 2014/07/15 21:54:53, llandwerlin wrote:
> On 2014/07/15 20:53:06, bbudge wrote:
> > Kind of implicit in my comment (2) was that PPB_VideoDecoder avoids the use
of
> a
> > PPP_ interface or callback function by having Push / Pull functions, Decode
> and
> > GetPicture calls. I was imagining you could have Encode and
GetBitstreamBuffer
> > functions to match this.
> > 
> > This works pretty well. The plugin pushes stream data with Decode and pulls
> > pictures with GetPicture. The only strangeness is when the plugin calls
Flush.
> > It continues to call GetPicture and we signal end of video with a
> > PP_ERROR_ABORTED result.
> > 
> > I don't see a reason why you can't do the same here.
> 
> Ok. I'm having trouble to understand how this is going to work for the encoder
> case.
> 
> The push/pull functions assume a transfer of ownership of the textures in the
> decoder API.
> In (2) you asked for the proxy to create and manage the buffers. That implies
> the proxy needs to know when the buffer is not used by the plugin anymore. I'm
> struggling to see how this works when the plugin can receive callbacks on any
> thread.
> 
> Maybe I'm missing some technicality here.
> Thanks for looking into this.


Pepper resources aren't accessed from multiple threads normally, but it should
be possible for them to 'live' on any thread the plugin creates. That's why PPP
interfaces are problematic for us.

Your API allows the plugin to signal that it has copied the data using the
RecycleBitstreamBuffer function. But it doesn't really seem necessary for the
proxy to "loan" out shm buffers to the plugin like PPB_VideoDecoder must do with
textures. The plugin should be able to copy the data into its own buffer as soon
as it receives it. So the RecycleBitstreamBuffer function doesn't seem necessary
to me.

Textures are a problem though. The plugin needs to be able to send multiple
textures to Encode so as not to stall the encoder, and it needs to know when the
encoder has actually read the data and the texture is safe to be used again.

In your design, PPB_VideoEncoder manages the bitstream buffers, but the plugin
creates and manages the textures. This is different from PPB_VideoDecoder, which
creates and manages both textures and bitstream buffers. It would be nice if we
could design PPB_VideoEncoder to be more like PPB_VideoDecoder, or vice versa,
for consistency.

Also, looking at VideoEncodeAccelerator and VideoEncoder classes in Chrome,
these use media::VideoFrame, which may be a problem. I'm not sure why there is
this asymmetry (VideoDecodeAccelerator returns textures, VideoDecoder returns
VideoFrames) but it would be a good idea to ask some domain experts about the
performance implications if we have to convert textures to VideoFrames and
whether the gains from using GL textures as a kind of shared memory are worth
the cost of reading textures back to the CPU. I'll do that and post a separate
comment about that.

After thinking more about encoding, this is how I think it could work (assuming
we do use GL textures to transport pictures instead of VideoFrames):

1) Create the encoder.
2) Call Initialize with a callback (the proxy will create a pool of textures
that it owns.)
3) Call GetBitstreamBuffer with a callback to start pulling encoded buffers (the
plugin must be able to call GBB again inside the callback, to keep pulling
bitstream buffers.)
4) Call GetPictureBuffer with a callback to get a texture to render into (the
plugin should be able to call GPB again inside the callback.)
5) Render a frame of video to the picture (texture).
6) Call Encode with the finished picture.

The pool of textures is necessary so the plugin can be encoding multiple video
frames concurrently. The GetPictureBuffer call is needed to rate-limit the
plugin. Encode doesn't need a callback since GetPictureBuffer rate-limits the
plugin. The API should make it clear the picture can only be used once, and the
plugin has to call GetPictureBuffer before it can call Encode again.

How the bitstream (shm) buffers are created is another matter. You could create
them as you need them in the host (right before sending back an encoded buffer.)
You could also just create them all up-front, but we've preferred creating them
on demand elsewhere.

[llandwerlin-old, 2014-07-17 14:59:34]

On 2014/07/16 15:52:54, bbudge wrote:
> On 2014/07/15 21:54:53, llandwerlin wrote:

Thanks a lot of taking the time to look at this stuff.

> 
> 
> Pepper resources aren't accessed from multiple threads normally, but it should
> be possible for them to 'live' on any thread the plugin creates. That's why
PPP
> interfaces are problematic for us.
> 
> Your API allows the plugin to signal that it has copied the data using the
> RecycleBitstreamBuffer function. But it doesn't really seem necessary for the
> proxy to "loan" out shm buffers to the plugin like PPB_VideoDecoder must do
with
> textures. The plugin should be able to copy the data into its own buffer as
soon
> as it receives it. So the RecycleBitstreamBuffer function doesn't seem
necessary
> to me.

Sorry to bring this up again, but this is were I have a problem (I might be
missing something obvious).
Assuming we're using shared memory buffers, once a buffer is ready to be handed
over to the plugin, the proxy needs to know when that buffer can then either be
sent back to the host for reuse, or disposed (and its associated memory
unmapped).
Looking at the TrackedCallback class, when the proxy calls Run() on this class,
it will queue the callback to be run on the main loop the plugin called the API
from. But there is no way to know when that actually happens. Leaving the proxy
with no clear timing for discarding the memory buffer.

> 
> Textures are a problem though. The plugin needs to be able to send multiple
> textures to Encode so as not to stall the encoder, and it needs to know when
the
> encoder has actually read the data and the texture is safe to be used again.
> 
> In your design, PPB_VideoEncoder manages the bitstream buffers, but the plugin
> creates and manages the textures. This is different from PPB_VideoDecoder,
which
> creates and manages both textures and bitstream buffers. It would be nice if
we
> could design PPB_VideoEncoder to be more like PPB_VideoDecoder, or vice versa,
> for consistency.

Sure, that can be done pretty easily.

> 
> Also, looking at VideoEncodeAccelerator and VideoEncoder classes in Chrome,
> these use media::VideoFrame, which may be a problem. I'm not sure why there is
> this asymmetry (VideoDecodeAccelerator returns textures, VideoDecoder returns
> VideoFrames) but it would be a good idea to ask some domain experts about the
> performance implications if we have to convert textures to VideoFrames and
> whether the gains from using GL textures as a kind of shared memory are worth
> the cost of reading textures back to the CPU. I'll do that and post a separate
> comment about that.

That's interesting because with another colleague we're looking at pushing GL
textures as video frames where it make sense.
On Intel platforms we have hardware encoders available. The constrain being that
the hardware will work with NV12 (YUV) format, so there is a transformation pass
needed.
We've noticed that the GLHelper class has some shaders for converting RGBA to
YUV, our idea was to use the GPU to convert the textures to something suitable
for the hardware, that would prevent readback for cases where you have hardware
available. 

> 
> After thinking more about encoding, this is how I think it could work
(assuming
> we do use GL textures to transport pictures instead of VideoFrames):
> 
> 1) Create the encoder.
> 2) Call Initialize with a callback (the proxy will create a pool of textures
> that it owns.)
> 3) Call GetBitstreamBuffer with a callback to start pulling encoded buffers
(the
> plugin must be able to call GBB again inside the callback, to keep pulling
> bitstream buffers.)
> 4) Call GetPictureBuffer with a callback to get a texture to render into (the
> plugin should be able to call GPB again inside the callback.)
> 5) Render a frame of video to the picture (texture).
> 6) Call Encode with the finished picture.
> 
> The pool of textures is necessary so the plugin can be encoding multiple video
> frames concurrently. The GetPictureBuffer call is needed to rate-limit the
> plugin. Encode doesn't need a callback since GetPictureBuffer rate-limits the
> plugin. The API should make it clear the picture can only be used once, and
the
> plugin has to call GetPictureBuffer before it can call Encode again.
> 
> How the bitstream (shm) buffers are created is another matter. You could
create
> them as you need them in the host (right before sending back an encoded
buffer.)
> You could also just create them all up-front, but we've preferred creating
them
> on demand elsewhere.

Sure.

[bbudge, 2014-07-17 17:41:25]

On 2014/07/17 14:59:34, llandwerlin wrote:
> On 2014/07/16 15:52:54, bbudge wrote:
> > On 2014/07/15 21:54:53, llandwerlin wrote:
> 
> Thanks a lot of taking the time to look at this stuff.
> 
> > 
> > 
> > Pepper resources aren't accessed from multiple threads normally, but it
should
> > be possible for them to 'live' on any thread the plugin creates. That's why
> PPP
> > interfaces are problematic for us.
> > 
> > Your API allows the plugin to signal that it has copied the data using the
> > RecycleBitstreamBuffer function. But it doesn't really seem necessary for
the
> > proxy to "loan" out shm buffers to the plugin like PPB_VideoDecoder must do
> with
> > textures. The plugin should be able to copy the data into its own buffer as
> soon
> > as it receives it. So the RecycleBitstreamBuffer function doesn't seem
> necessary
> > to me.
> 
> Sorry to bring this up again, but this is were I have a problem (I might be
> missing something obvious).
> Assuming we're using shared memory buffers, once a buffer is ready to be
handed
> over to the plugin, the proxy needs to know when that buffer can then either
be
> sent back to the host for reuse, or disposed (and its associated memory
> unmapped).
> Looking at the TrackedCallback class, when the proxy calls Run() on this
class,
> it will queue the callback to be run on the main loop the plugin called the
API
> from. But there is no way to know when that actually happens. Leaving the
proxy
> with no clear timing for discarding the memory buffer.

Actually, I think the fact that these are shared memory buffers should be an
implementation detail. Instead of passing them over to the plugin, the plugin
could pass in a PP_ArrayOutput parameter to Encode that you could use to copy
from the shared memory buffer to the plugin's buffer. That way, you would know
when the shared memory can be used again. There are several examples of this in
the proxy code. One that I can think of is PPB_TrueTypeFont_Dev (look at
ppapi/proxy/truetype_font_resource.cc. Or just search for PP_ArrayOutput in
Chrome.

> 
> > 
> > Textures are a problem though. The plugin needs to be able to send multiple
> > textures to Encode so as not to stall the encoder, and it needs to know when
> the
> > encoder has actually read the data and the texture is safe to be used again.
> > 
> > In your design, PPB_VideoEncoder manages the bitstream buffers, but the
plugin
> > creates and manages the textures. This is different from PPB_VideoDecoder,
> which
> > creates and manages both textures and bitstream buffers. It would be nice if
> we
> > could design PPB_VideoEncoder to be more like PPB_VideoDecoder, or vice
versa,
> > for consistency.
> 
> Sure, that can be done pretty easily.
> 
> > 
> > Also, looking at VideoEncodeAccelerator and VideoEncoder classes in Chrome,
> > these use media::VideoFrame, which may be a problem. I'm not sure why there
is
> > this asymmetry (VideoDecodeAccelerator returns textures, VideoDecoder
returns
> > VideoFrames) but it would be a good idea to ask some domain experts about
the
> > performance implications if we have to convert textures to VideoFrames and
> > whether the gains from using GL textures as a kind of shared memory are
worth
> > the cost of reading textures back to the CPU. I'll do that and post a
separate
> > comment about that.
> 
> That's interesting because with another colleague we're looking at pushing GL
> textures as video frames where it make sense.
> On Intel platforms we have hardware encoders available. The constrain being
that
> the hardware will work with NV12 (YUV) format, so there is a transformation
pass
> needed.
> We've noticed that the GLHelper class has some shaders for converting RGBA to
> YUV, our idea was to use the GPU to convert the textures to something suitable
> for the hardware, that would prevent readback for cases where you have
hardware
> available. 

With the video folks, there is an interesting email thread about how frames
should be passed around in Chrome. I'll cc you on those and/or post a summary
here.

> 
> > 
> > After thinking more about encoding, this is how I think it could work
> (assuming
> > we do use GL textures to transport pictures instead of VideoFrames):
> > 
> > 1) Create the encoder.
> > 2) Call Initialize with a callback (the proxy will create a pool of textures
> > that it owns.)
> > 3) Call GetBitstreamBuffer with a callback to start pulling encoded buffers
> (the
> > plugin must be able to call GBB again inside the callback, to keep pulling
> > bitstream buffers.)
> > 4) Call GetPictureBuffer with a callback to get a texture to render into
(the
> > plugin should be able to call GPB again inside the callback.)
> > 5) Render a frame of video to the picture (texture).
> > 6) Call Encode with the finished picture.
> > 
> > The pool of textures is necessary so the plugin can be encoding multiple
video
> > frames concurrently. The GetPictureBuffer call is needed to rate-limit the
> > plugin. Encode doesn't need a callback since GetPictureBuffer rate-limits
the
> > plugin. The API should make it clear the picture can only be used once, and
> the
> > plugin has to call GetPictureBuffer before it can call Encode again.
> > 
> > How the bitstream (shm) buffers are created is another matter. You could
> create
> > them as you need them in the host (right before sending back an encoded
> buffer.)
> > You could also just create them all up-front, but we've preferred creating
> them
> > on demand elsewhere.
> 
> Sure.

[bbudge, 2014-07-17 18:07:10]

Some information from an internal email thread on issues with a PPAPI
VideoEncoder:

----------------

On Wed, Jul 16, 2014 at 9:13 AM, Bill Budge <bbudge@google.com> wrote:
Hi Antoine,

An external contributor is proposing a PPB_VideoEncoder API and we're reviewing
his CL.

https://codereview.chromium.org/365153003/

One thing that came up is that Chrome encoders (hardware accelerated and not)
take VideoFrames as input. The CL proposes using GL textures to transport
pictures from the plugin to the renderer. I have two questions I'm hoping you
can answer:

1) Does the savings from avoiding a copy of frame data offset the cost of
downloading the contents of the texture into a VideoFrame, or is it possible to
avoid this?

My understanding is that while a VideoFrame can encapsulate a GL texture, today
none of the accelerated encoders support using that as an input (at least as
implemented in Chrome. It is possible that they support it in hardware). Reading
back from a GL texture is going to be slow no matter what.
Given that, I think it's a mistake to expose the encoder uniquely based on GL
textures:
1- if the source of data is in software, using the API means uploading to a
texture (costs at least one if not 2 copies), and then reading back from it
(depending on where it's done, an extra 1 or 2 copies, plus synchronization,
plus a potentially slow GPU->CPU readback).
2- if the source of data is in a texture, we have to readback anyway, it doesn't
really matter if it's done by the plugin or by the API, it will be slow anyway.

It does make sense to expose GL textures once the accelerated encoders can take
them as an input. The savings then would be significant.
 
2) Is this a good API from the plugin perspective (does a plugin normally want
to encode a series of GL textures, or would VideoFrame be more useful)?

I'm not sure of the use case. The main use cases for encoding today are:
1- video conference, the source is in software, coming from the camera (note
that the Pepper video capture API gives you software bits)
2- tab casting, the source can be either in hardware or software, but it's all
done inside of chrome today. I'm not sure about a similar use case from within a
plugin, I guess why not in theory, though a follow-up question is whether or not
we believe such a use case is important enough that we want the maintenance
burden. I don't think there is any similar API exposed to the web, which is a
good sniff test of whether or not we should expose to pepper.

Antoine

----------------

Pawel Osciak <posciak@google.com>

Antoine is correct on all points.

VideoFrame can encapsulate shared memory and dmabufs as well, and those are
actually used for encode in camera capture/webrtc cases. The texture path is
used for casting, when we do do a download, but then use shared memory in VF
again and then copy out from it to codec buffers. The only path where we
actually do use VF textures is for decode output.

The problem with casting is that we don't setup the textures that we may have to
cast from (we don't know what may get casted), so we can't prepare "special
shareable" buffers beforehand. We definitely want to improve this situation, one
issue is though that not all textures allocated by DRM/GPU can be shared with
codecs, for example on some ARMs (Exynos), which may have special requirements
for the memory used by the codec.

I definitely agree having textures as encoder input is not good. We need a
universal transport for shareable buffers though and at one point the idea was
to use textures, which, as Antoine mentioned, were supposed to be shareable with
the codec via EGL_EXT_image_dma_buf_import (this is not good for casting though,
because it's backing a texture with EGLImage created from dmabuf, not exporting
dmabufs from an existing texture). Then we wanted to use dmabufs, but right now
it seems that we end up using GLimage, which will be able to encapsulate dmabufs
among other things. 

 
2) Is this a good API from the plugin perspective (does a plugin normally want
to encode a series of GL textures, or would VideoFrame be more useful)?

I'm not sure of the use case. The main use cases for encoding today are:
1- video conference, the source is in software, coming from the camera (note
that the Pepper video capture API gives you software bits)
2- tab casting, the source can be either in hardware or software, but it's all
done inside of chrome today. I'm not sure about a similar use case from within a
plugin, I guess why not in theory, though a follow-up question is whether or not
we believe such a use case is important enough that we want the maintenance
burden. I don't think there is any similar API exposed to the web, which is a
good sniff test of whether or not we should expose to pepper.

This is correct. The goal is to back VideoFrames with dmabufs of HW buffers (on
CrOS at least, and maybe something else on Android and elsewhere). If the source
is coming from a CPU buffer, we'd upload to a codec buffer backed by a dmabuf,
mmapping its dmabuf fd.

So I think the goal should be to have an abstract entity like GLImage as the
input to the codec.

I'm sure others will like to add to this.

Thanks,
Pawel

[dmichael (off chromium), 2014-07-18 20:15:49]

Hmm, I'm feeling myself leaning towards piman@'s way of thinking about this. I
think it's a useful API, but there is no analogous w3c API. Decode is a little
weird, because there is the <video> tag that will do decode for JS, and our
PPB_VideoDecode is a bit more fully featured. But it's not a fundamentally new
capability, either.

I think we might want to put this CL on hold for now. I'll ask around to see if
this aligns with some of our own platform work. I really appreciate you stepping
up to do this work, but (a) it seems like it strikes out too far ahead of what's
available via WhatWG/W3C specs and (b) I don't want to take on the maintenance
burden for this API right now if I can help it.

Meantime, you might want to consider taking this idea to the WebRTC working
group. If it gets consideration as a DOM API, that will give you one possible
path forward for encoding, and it would make us more likely to want to implement
this in Pepper, too.

[llandwerlin-old, 2014-07-19 18:33:14]

On 2014/07/18 20:15:49, dmichael wrote:
> Hmm, I'm feeling myself leaning towards piman@'s way of thinking about this. I
> think it's a useful API, but there is no analogous w3c API. Decode is a little
> weird, because there is the <video> tag that will do decode for JS, and our
> PPB_VideoDecode is a bit more fully featured. But it's not a fundamentally new
> capability, either.
> 
> I think we might want to put this CL on hold for now. I'll ask around to see
if
> this aligns with some of our own platform work. I really appreciate you
stepping
> up to do this work, but (a) it seems like it strikes out too far ahead of
what's
> available via WhatWG/W3C specs and (b) I don't want to take on the maintenance
> burden for this API right now if I can help it.
> 
> Meantime, you might want to consider taking this idea to the WebRTC working
> group. If it gets consideration as a DOM API, that will give you one possible
> path forward for encoding, and it would make us more likely to want to
implement
> this in Pepper, too.

Right,

The other way I was thinking about getting a similar feature was to implement
this first spec :

https://dvcs.w3.org/hg/dap/raw-file/tip/media-stream-capture/MediaRecorder.html

And also the Canvas Recording section of this one :

https://dvcs.w3.org/hg/audio/raw-file/tip/streams/StreamProcessing.html

It is a bit less flexible because from my understanding it pushes the muxing
into the browser.
Let me know if that's something you're more likely to accept.
Thanks.

[joshtriplett, 2014-07-21 15:02:50]

Following up on several of these points:

One of the major reasons to use GL textures in this interface is specifically so
that Chrome apps can use GL to construct the input and have some hope of it
feeding into a hardware encoder with zero copies.  If the encoder *has* to do
readback or make a copy, it can, but the interface should not *force* it to do
so.  An interface that requires making a full copy of every frame, let alone
more than one such copy, would severely limit performance, especially on
lower-end Chrome devices.

We're working with ISVs who are specifically looking to use GL to construct an
image to feed into the encoder, and who really need to use hardware encoding
rather than software encoding.  The source of their data is specifically GL, not
software.  There is, today, no way for them to do either of those things.

(If you'd like to talk further about specific applications, I'd be happy to do
so; drop me an email.)

Regarding the use cases:

1) Ideally, the video capture interface should provide something that can be
used as a GL texture without making any copies.  That would allow compositing
the camera image with other things before feeding it into the encoder.

2) For tab casting, the image provided by Chrome should by definition always
come from the GPU as an already-rendered-to buffer ready for the GPU to use
further.

Several use cases do exist for both of those, such as screencasting/recording,
application sharing (for collaboration/presentation/demos/training/etc), live
streaming, etc.

The purpose of using GL in these cases is to work with the image before encoding
it (e.g. composite several things into a frame).

All of these use cases will work fine whether the API uses GL textures directly
or uses VideoFrame as a wrapper around a hardware buffer, as long as the
application has a zero-copy way to get GL textures out of the decoder interface
and feed them into the encoder interface.  Having to copy every frame kills
performance.

I do like the idea of seeing this using the new style that doesn't deeply care
about the main NaCl thread.

Could we discuss this further, and see if we can come up with an API that we're
all happy with?  (Also, let us know if you'd like to discuss this via a medium
other than email.)

[joshtriplett, 2014-07-21 15:10:02]

The problem is that we're shipping Chrome hardware *today* that has hardware
video encoding capabilities, but we have absolutely no way for applications to
make use of that.  That leaves applications stuck with undesirable alternatives
like embedding ffmpeg in NaCl, which introduces numerous problems in addition to
poor performance and battery life.