Pepper: Define PPB_VideoEncoder API.

ppapi/api/ppb_video_encoder.idl

Index: ppapi/api/ppb_video_encoder.idl
diff --git a/ppapi/api/ppb_video_encoder.idl b/ppapi/api/ppb_video_encoder.idl
new file mode 100644
index 0000000000000000000000000000000000000000..f0b6631c9e8a765227525301eab230203592738c
--- /dev/null
+++ b/ppapi/api/ppb_video_encoder.idl
@@ -0,0 +1,137 @@
+/* Copyright (c) 2015 The Chromium Authors. All rights reserved.
+ * Use of this source code is governed by a BSD-style license that can be
+ * found in the LICENSE file.
+ */
+
+/**
+ * This file defines the <code>PPB_VideoEncoder</code> interface.
+ */
+
+[generate_thunk]
+
+label Chrome {
+  M42 = 0.1
+};
+
+/**
+ * Video encoder interface.
+ *
+ * Typical usage:
+ * - Call Create() to create a new video encoder resource.
+ * - Call GetSupportedFormats() to determine which codecs and profiles are
+ * - available.
+ * - Call Initialize() to initialize the encoder for a supported profile.
+ * - Call GetVideoFrame() to get a video frame resource, copy the frame into
+ * - its buffer, and call Encode() to push the video frame to the encoder. Do
+ * - this continuously, waiting for completion of each call.
+ * - Call GetBitstreamBuffer() continuously (waiting for each previous call to
+ *   complete) to pull encoded pictures from the encoder.
+ * - Call RecycleBitstreamBuffer() after consuming the data in the bitstream
+ * - buffer.
+ * - To destroy the decoder, the plugin should release all of its references to
+ *   it. Any pending callbacks will abort before the decoder is destroyed.
+ *
+ * Available video codecs vary by platform.
+ * All: theora, vorbis, vp8.
+ * Chrome and ChromeOS: aac, h264.
+ * ChromeOS: mpeg4.
+ */
+interface PPB_VideoEncoder {
+  /**
+   * Creates a new video encoder resource.
+   *
+   * @param[in] instance A <code>PP_Instance</code> identifying the instance
+   * with the video encoder.
+   *
+   * @return A <code>PP_Resource</code> corresponding to a video encoder if
+   * successful or 0 otherwise.
+   */
+  PP_Resource Create(
+      [in] PP_Instance instance);
+
+  /**
+   * Determines if the given resource is a video encoder.
+   *
+   * @param[in] resource A <code>PP_Resource</code> identifying a resource.
+   *
+   * @return <code>PP_TRUE</code> if the resource is a
+   * <code>PPB_VideoEncoder</code>, <code>PP_FALSE</code> if the resource is
+   * invalid or some other type.
+   */
+  PP_Bool IsVideoEncoder(
+      [in] PP_Resource resource);
+
+  /**
+   * Gets an array of supported video encoder profiles.
+   * These can be used to choose profiles when calling Initialize().
+   *
+   * @param[in] video_encoder A <code>PP_Resource</code> identifying the video
+   * encoder.
+   * @param[in] output A <code>PP_ArrayOutput</code> to hold the supported
+   * <code>PP_SupportedVideoProfile</code> structs.
+   * @param[in] callback A <code>PP_CompletionCallback</code> to be called upon
+   * completion of GetSupportedProfiles.
+   *
+   * @return If >= 0, the number of supported profiles returned, otherwise an
+   * error code from <code>pp_errors.h</code>.
+   */
+  int32_t GetSupportedProfiles(
+      [in] PP_Resource video_encoder,
+      [in] PP_ArrayOutput output,
+      [in] PP_CompletionCallback callback);
+
+  /**
+   * Initializes a video encoder resource. This should be called after Create()
+   * and before any functions below.
+   *
+   * @param[in] video_encoder A <code>PP_Resource</code> identifying the video
+   * encoder.
+   * @param[in] input_format The <code>PP_VideoFrame_Format</code> of the
+   * frames which will be encoded.
+   * @param[in] input_visible_size A <code>PP_Size</code> specifying the
+   * dimensions of the visible part of the input frames.
+   * @param[in] output_profile A <code>PP_VideoProfile</code> specifying the
+   * codec profile of the encoded output stream.
+   * @param[in] acceleration A <code>PP_HardwareAcceleration</code> specifying
+   * whether to use a hardware accelerated or a software implementation.
+   * @param[in] callback A <code>PP_CompletionCallback</code> to be called upon
+   * completion.
+   *
+   * @return An int32_t containing an error code from <code>pp_errors.h</code>.
+   * Returns PP_ERROR_NOTSUPPORTED if video encoding is not available, or the
+   * requested codec profile is not supported.
+   */
+  int32_t Initialize(
+      [in] PP_Resource video_decoder,
+      [in] PP_VideoFrame_Format input_format,
+      [in] PP_Size input_visible_size,
+      [in] PP_VideoProfile output_profile,
+      [in] uint32_t initial_bitrate,
+      [in] PP_HardwareAcceleration acceleration,
+      [in] PP_CompletionCallback callback);
+
+  int32_t GetVideoFrame(

[llandwerlin-old, 2015/01/13 12:43:55]

How would we like to have GetVideoFrame() behave when called too often?
There will be a limited amount of frames allocated. In the case where all the
frames have been submitted to the encoder, but none has returned, should we
return PP_ERROR_NOMEMORY or PP_OK_COMPLETIONPENDING?

I thinking the later would be better.

Also, should we expose the number of frame allocated to the application?

[bbudge, 2015/01/14 02:01:13]

This should work like PPB_VIdeoDecoder or PPB_MediaStreamVideoTrack, which
throttle the plugin by returning PP_OK_COMPLETIONPENDING when no more calls to a
function can be satisfied. In this case, the API would throttle the plugin's
calls to GetVideoFrame by returning PP_OK_COMPLETIONPENDING when the last
available frame is committed. The plugin gets unblocked when Encodes complete.
It's not clear to me how VEA notifies the client about this though.

As for returning the number of frames that the encoder requires, that points to
a potential problem with sharing frames between PPB_MediaStreamVideoTrack and
PPB_VideoEncoder. MSVT currently allows a maximum of 8 frames and it's possible
that an encoder might require more than that number. In that case, the plugin
would have to copy data from MSVT frames to VE frames (otherwise it could hang
with all MSVT frames committed and no encodes complete.)

That might imply that it's too difficult to allow sharing. I still need to ask
whether this is a real-world use case.

[llandwerlin-old, 2015/01/14 11:56:13]

As far I as I can tell, the Vaapi/V4L2/Android encoder accelerators all copy the
input frames into a format suitable for them (NV12), so with the current
implementations we shouldn't encounter any blocking situation.

[bbudge, 2015/01/14 16:38:10]

Here's a VEA that shares the frame with the GPU process, implying that we have
to hold the frame for a while:
https://code.google.com/p/chromium/codesearch#chromium/src/content/common/gpu...

It's not clear to me how the client can tell when the frame is safe to use
again.

[llandwerlin-old, 2015/01/14 16:57:08]

The frame is kept in a hash table :

https://code.google.com/p/chromium/codesearch#chromium/src/content/common/gpu...

It is released in GpuVideoEncodeAcceleratorHost::OnNotifyInputDone

media::VideoFrame is created with a callback that is fired upon destruction,
that's what the VEAClient should use to know when to recycle them.

[bbudge, 2015/01/14 17:20:11]

Thanks for the explanation! Then we can distinguish between our frame or a MSVT
frame. I need to add an 'out' param for Encode so the plugin can know when to
recycle a MSVT frame.

[llandwerlin-old, 2015/01/14 20:21:26]

Maybe I'm missing something, but I would have thought the callback was enough.
We just need to map the frames to the callbacks in the proxy and call
accordingly.

[bbudge, 2015/01/14 22:07:46]

You're correct, the plugin can keep track of which frame it sent with the
callback.

+      [in] PP_Resource video_encoder,
+      [out] PP_Resource video_frame,

[dmichael (off chromium), 2015/01/14 00:25:20]

I take it this is a PPB_VideoFrame?

[bbudge, 2015/01/14 02:01:13]

Yes, owned by the proxy and loaned to the plugin, similarly to
PPB_MediaStreamVideoTrack.

+      [in] PP_CompletionCallback callback);
+
+  int32_t Encode(

[llandwerlin-old, 2015/01/12 11:48:04]

PPB_VideoDecoder has a |decode_id| parameter on its Decode() method :

https://code.google.com/p/chromium/codesearch#chromium/src/ppapi/c/ppb_video_...
Any reason why the encoder wouldn't have that parameter too?

[bbudge, 2015/01/12 18:38:53]

I don't see any way to pass in such id's to the encoder, so I'm not sure how
we'd associate bitstream buffers with video frames. I'd be surprised if they
were in 1:1 correspondence.
https://code.google.com/p/chromium/codesearch#chromium/src/media/video/video_...

[llandwerlin-old, 2015/01/13 12:02:11]

Looking at the Vaapi (Intel accelerator) code, I thought that would be the case,
but I can't really tell for the other drivers.
So nope.

+      [in] PP_Resource video_encoder,
+      [in] PP_Resource video_frame,

[llandwerlin-old, 2015/01/12 11:48:04]

Will Encode() allow to pass video frame coming out of a
PPB_MediaStreamVideoTrack?

[bbudge, 2015/01/12 18:38:53]

Great question.

This seems like a common use case. I suppose we should allow the
PPB_VideoEncoder implementation to accept PPB_VideoFrame resources that are not
part of its own pool. I still think it should create its own pool though so the
plugin doesn't have to create and manage pools of VideoFrame resources (as
implied by the VEA::Client interface). Perhaps the implementation can lazily
create the pool so we don't waste system resources if the plugin Configures the
MSVT with enough frames to satisfy the Encoder. We would only create the pool of
frames when the plugin called GetVideoFrame.

[llandwerlin-old, 2015/01/13 12:02:10]

Sounds good.

+      [in] PP_Bool force_keyframe,
+      [in] PP_CompletionCallback callback);
+
+  int32_t GetBitstreamBuffer(
+      [in] PP_Resource video_encoder,
+      [out] PP_BitstreamBuffer bitstream_buffer,
+      [in] PP_CompletionCallback callback);
+
+  void RecycleBitstreamBuffer(
+      [in] PP_Resource video_decoder,
+      [in] PP_BitstreamBuffer bitstream_buffer);
+
+  void RequestEncodingParametersChange(

[dmichael (off chromium), 2015/01/14 00:25:20]

Does there need to be a CompletionCallback for this, so you know if it succeeds
and when it takes effect?

[bbudge, 2015/01/14 02:01:13]

media::VideoEncodeAccelerator doesn't return a value or have a callback and the
docs state it's a request only:
https://code.google.com/p/chromium/codesearch#chromium/src/media/video/video_...

+      [in] PP_Resource video_decoder,
+      [in] uint32_t bitrate,
+      [in] uint32_t framerate);
+};

[llandwerlin-old, 2015/01/12 11:48:04]

Should there be a Reset() method?
We could use Destroy() on media::VideoEncodeAccelerator to implement it.

[bbudge, 2015/01/12 18:38:53]

I tried to match 1:1 with VideoDecodeAccelerator where possible. Is there a use
case for this? 'Reset' and 'Flush' add a lot of complexity to PPB_VideoDecoder
and it would be nice to avoid that.

[llandwerlin-old, 2015/01/13 12:02:11]

I'm thinking that if we allow frames from the MediaStreamVideoTrack to be
submitted and at some point we want to stop the encoding and get the encoder to
drop all the resources it's currently working with, we might need to call the
Reset() method of the encoder before with return the video frame to the
MediaStreamVideoTrack with RecycleFrame().

[bbudge, 2015/01/14 02:01:13]

I don't think VEA has a Reset method. See above for other difficulties w.r.t
sharing frames.