Pepper VideoDecoder API definition.

ppapi/api/ppb_media_codec_video_decoder.idl

+/* Copyright (c) 2014 The Chromium Authors. All rights reserved.
+ * Use of this source code is governed by a BSD-style license that can be
+ * found in the LICENSE file.
+ */
+
+/**
+ * This file defines the <code>PPB_MediaCodecVideoDecoder</code> interface.
+ */
+
+[generate_thunk]
+
+label Chrome {
+  [channel=dev] M36 = 0.1
+};
+
+/**
+ * Video decoder interface.
+ *
+ * Typical usage:
+ * - Call Create() to create a new video decoder resource.
+ * - Call Initialize() to initialize it with a 3d graphics context.
+ * - Call GetBitstreamBuffer() to get a buffer to hold bitstream data.
+ * - Fill the bitstream buffer with video data to decode.
+ * - Call Decode() to send the bitstream data to the decoder.
+ * - Call GetPictureBuffer() to get the next decoded picture.
+ * - To signal end of stream to the decoder: call Flush() and wait for the
+ *   callback.
+ * - To reset the decoder (e.g. to implement Seek): call Reset() and wait for
+ *   the callback.
+ */
+interface PPB_MediaCodecVideoDecoder {
+  /**
+   * Creates a new video decoder resource.
+   *
+   * @param[in] instance A <code>PP_Instance</code> identifying the instance
+   * with the video decoder.
+   *
+   * @return A <code>PP_Resource</code> corresponding to a video decoder if
+   * successful or 0 otherwise.
+   */
+  PP_Resource Create(
+      [in] PP_Instance instance);
+
+  /**
+   * Determines if the given resource is a video decoder.
+   *
+   * @param[in] resource A <code>PP_Resource</code> identifying a resource.
+   *
+   * @return <code>PP_TRUE</code> if the resource is a
+   * <code>PPB_MediaCodecVideoDecoder</code>, <code>PP_FALSE</code> if the
+   * resource is invalid or some other type.
+   */
+  PP_Bool IsMediaCodecVideoDecoder(
+      [in] PP_Resource resource);
+
+  /**
+   * Initializes a video decoder resource. This should only be called once,
+   * after Create() and before any other functions. Hardware video decoding is

[Ami GONE FROM CHROMIUM, 2014/04/09 22:12:41]

I thought the point of this take on the API was to hide the unavailability of HW
capabilities behind SW fallback, no?

[bbudge, 2014/04/11 17:15:16]

We aren't attempting to provide software fallback because it's hard (the API
returns pictures as GL textures). If it's not hard, we could consider it.

+   * not supported on all platforms.
+   *
+   * @param[in] video_decoder A <code>PP_Resource</code> identifying the video
+   * decoder.
+   * @param[in] context A <code>PPB_Graphics3D</code> resource to use for
+   * decoding.
+   * @param[in] profile A <code>PP_MediaCodec_Profile</code> specifying the
+   * video stream profile.
+   * @param[in] callback A <code>PP_CompletionCallback</code> to be called upon
+   * completion.
+   *
+   * @return An int32_t containing an error code from <code>pp_errors.h</code>.
+   * Returns PP_ERROR_NOTSUPPORTED on platforms where hardware video decoding is

[Ami GONE FROM CHROMIUM, 2014/04/09 22:12:41]

I'm surprised status is not returned in the callback instead.
(comment applies elsewhere below as well)

[bbudge, 2014/04/11 17:15:16]

Comment is a bit misleading since calling this with good params will return
PP_OK_COMPLETIONPENDING. The user's callback will receive PP_ERROR_NOTSUPPORTED.

Fixed the comment.

+   * not available.
+   */
+  int32_t Initialize(
+      [in] PP_Resource video_decoder,
+      [in] PP_Resource context,

[dmichael (off chromium), 2014/04/09 21:54:07]

might be good to name graphics3d_context

[bbudge, 2014/04/11 17:15:16]

Done.

+      [in] PP_MediaCodec_Profile profile,

[dmichael (off chromium), 2014/04/09 21:54:07]

Do you have a function yet for getting supported profiles?

Do we want to consider a "RecommendProfile" kind of function, where you can pass
some information about the video you have and have the decoder tell you what
profile you should use? Or:
1) Treat this profile parameter as a suggestion, and the decoder automatically
figures out the best applicable profile the hardware can support
or...
2) Detect the appropriate profile automatically from the stream

(I have to admit I know little enough about decoding to know if 1 and 2 are
insane)

[Ami GONE FROM CHROMIUM, 2014/04/09 22:18:38]

profile is a property of the video to be decoded.  It does you no good to init
the decoder at a higher profile, and it is likely to fail during decode if you
initialize with a lower profile.
There is active harm in init'ing at a higher-than-needed profile because HW
resources will be consumed (reducing the available decoding power for other
videos concurrently decoding), so I don't think we want to do #1.

The profile is usually stored in the muxing container (e.g. mp4), not in the
media bitstream (e.g. h.264) so I don't think #2 works in general.  Also, you
don't want to find out that the decoder can't handle your stream part-way
through (because mid-stream fallback is a hard problem), so Initialize() is the
right time to specify the profile.

[bbudge, 2014/04/16 00:51:05]

So based on this discussion, I don't think we can provide assistance to the
plugin here. The plugin processes its stream to determine the codec profile and
then attempts to Initialize at that level.

+      [in] PP_CompletionCallback callback);
+
+  /**
+   * Gets a bitstream buffer to hold video data to be decoded. The bitstream
+   * buffer is a read-only struct describing the buffer. The plugin can have
+   * multiple requests for bitstream buffers pending. It should bound the number
+   * of decodes it has in flight to some constant number by throttling its calls
+   * to GetBitstreamBuffer().

[dmichael (off chromium), 2014/04/09 21:54:07]

I wonder if it might be possible for Chrome/PPAPI to handle the throttling?

One possible way is to only return a BitstreamBuffer to the plugin *only* when
it's appropriate to call Decode again. Then the plugin doesn't have to think at
all about keeping the pipe full; they keep pushing data so long as they have a
place to write it.

[Ami GONE FROM CHROMIUM, 2014/04/09 22:12:41]

why GBB() and not Decode() in this sentence?

[bbudge, 2014/04/11 17:15:16]

I tried to think of a way to do this automatically but there are difficulties.

1) Since we want to allow multiple pending Decode calls, we need multiple
pending GBB calls, and so we have to be prepared to queue up requests if the
plugin calls GBB really fast. I think failing in this case would make writing
plugin callbacks a pain.
2) Need a magic number of buffers to allow before we start throttling. I don't
know what a good value would be and likely the plugin knows better.
3) We could add this number to the parameters for Initialize(), and have some
hard limit in case the plugin sets it too large. It adds complexity to the proxy
though.

[bbudge, 2014/04/11 17:15:16]

Throttling GBB would limit the number of pending decodes. I've eliminated the
GetBitstreamBuffer function, so it is now Decode that needs throttling.

[bbudge, 2014/04/16 00:51:05]

I've changed the API to eliminate GBB. I've also changed the semantics of Decode
so only one call can be pending. Internally, the proxy now throttles requests
for new shm buffers to solve the problem. It's better to do this in the proxy,
where we can better determine appropriate limits, rather than leaving this to
the plugin developer to guess.

+   *
+   * @param[in] video_decoder A <code>PP_Resource</code> identifying the video
+   * decoder.
+   * @param[in] size The size of the buffer in bytes.

[Ami GONE FROM CHROMIUM, 2014/04/09 22:12:41]

s/size/capacity/ ?

[bbudge, 2014/04/11 17:15:16]

Eliminated GBB.

+   * @param[out] bitstream_buffer A <code>PP_MediaCodec_BitstreamBuffer</code>
+   * to hold the bitstream buffer.
+   * @param[in] callback A <code>PP_CompletionCallback</code> to be called upon
+   * completion.
+   *
+   * @return An int32_t containing an error code from <code>pp_errors.h</code>.
+   */
+  int32_t GetBitstreamBuffer(
+      [in] PP_Resource video_decoder,
+      [in] uint32_t size,
+      [out] PP_MediaCodec_BitstreamBuffer bitstream_buffer,
+      [in] PP_CompletionCallback callback);
+
+  /**
+   * Decodes a bitstream buffer. The plugin should have first called
+   * GetBitstreamBuffer() and filled the returned buffer with bitstream data.
+   * Decode() takes back ownership of the bitstream buffer. The plugin should
+   * not use |bitstream_buffer| after calling Decode().
+   *
+   * @param[in] video_decoder A <code>PP_Resource</code> identifying the video
+   * decoder.
+   * @param[in] bitstream_buffer A <code>PP_MediaCodec_BitstreamBuffer</code>
+   * holding the bitstream data. This contains a |buffer_id| field that can be
+   * used to associate bitstream buffers with picture buffers.
+   * @param[in] callback A <code>PP_CompletionCallback</code> to be called upon
+   * completion.

[Ami GONE FROM CHROMIUM, 2014/04/09 22:12:41]

what does "completion" mean in this context?  
- decoder accepting the buffer
- buffer is on the IPC wire to the decoder
- decoder has finished with the buffer
- something else?

[bbudge, 2014/04/11 17:15:16]

Yes, completion means the buffer has been decoded. (proxy received
NotifyEndOfBitstreamBuffer).
Made the comment say this.

[bbudge, 2014/04/16 00:51:05]

I've changed the semantics slightly. Now this means the plugin can release its
buffer and make another call to Decode.

+   *
+   * @return An int32_t containing an error code from <code>pp_errors.h</code>.
+   */
+  int32_t Decode(
+      [in] PP_Resource video_decoder,
+      [in] PP_MediaCodec_BitstreamBuffer bitstream_buffer,
+      [in] PP_CompletionCallback callback);
+
+  /**
+   * Gets the next picture buffer from the decoder. When the plugin is finished
+   * with the picture, it should call RecyclePictureBuffer() to return it to the
+   * system. The plugin can call GetPictureBuffer() again in its callback to

[Ami GONE FROM CHROMIUM, 2014/04/09 22:12:41]

what callback?

[bbudge, 2014/04/11 17:15:16]

The user-provided callback.
Made the comment clearer.

+   * request the next picture.
+   *
+   * @param[in] video_decoder A <code>PP_Resource</code> identifying the video
+   * decoder.
+   * @param[out] picture_buffer A <code>PP_MediaCodec_PictureBuffer</code> to
+   * hold the decoded picture.
+   * @param[in] callback A <code>PP_CompletionCallback</code> to be called upon
+   * completion.
+   *
+   * @return An int32_t containing an error code from <code>pp_errors.h</code>.
+   */
+  int32_t GetPictureBuffer(
+      [in] PP_Resource video_decoder,
+      [out] PP_MediaCodec_PictureBuffer picture_buffer,
+      [in] PP_CompletionCallback callback);
+
+  /**
+   * Recycles a picture buffer that the plugin has received from the decoder.
+   * The plugin should call this as soon as it has finished using the texture so
+   * the system can decode more pictures.
+   *
+   * @param[in] video_decoder A <code>PP_Resource</code> identifying the video
+   * decoder.
+   * @param[in] picture_buffer A <code>PP_MediaCodec_PictureBuffer</code> to
+   * return to the system.
+   */
+  void RecyclePictureBuffer(
+      [in] PP_Resource video_decoder,
+      [in] PP_MediaCodec_PictureBuffer picture_buffer);
+
+  /**
+   * Flushes the decoder.  Any pending decodes are completed including callbacks
+   * to the plugin. Once done flushing, the decoder will call |callback|. While
+   * flushing, the plugin should not attempt to decode more video.

[dmichael (off chromium), 2014/04/09 21:54:07]

You could maybe flesh this comment out a bit with why you might want to use
flush and what happens when it's done. If I understand correctly, something
like:
" After Flush() completes, the plugin may begin decoding frames again. This can
be useful for seeking in the video stream."

Also, what happens to BitStreamBuffers the plugin already has? Are they still
valid to use after a Flush?

And...  maybe all of what I just said really belongs on "Reset" instead...

[Ami GONE FROM CHROMIUM, 2014/04/09 22:12:41]

what about calling reset during flush?

[bbudge, 2014/04/11 17:15:16]

I'm going to say the plugin can't call anything during Flush or Reset (until its
callback is run)

[bbudge, 2014/04/16 00:51:05]

Also, Flush is to signal end-of-stream, so the plugin can shut down a video
gracefully.

+   *
+   * @param[in] video_decoder A <code>PP_Resource</code> identifying the video
+   * decoder.
+   * @param[in] callback A <code>PP_CompletionCallback</code> to be called upon
+   * completion.
+   *
+   * @return An int32_t containing an error code from <code>pp_errors.h</code>.
+   */
+  int32_t Flush(
+      [in] PP_Resource video_decoder,
+      [in] PP_CompletionCallback callback);
+
+  /**
+   * Resets the decoder as quickly as possible.  Pending decodes are aborted
+   * and the decoder is put back into a state ready to receive further decode
+   * calls. Once all decodes are aborted and reset is complete, |callback| will

[Ami GONE FROM CHROMIUM, 2014/04/09 22:12:41]

Forbid decode & flush from being called during reset?

[bbudge, 2014/04/11 17:15:16]

Done.

+   * be called.

[Ami GONE FROM CHROMIUM, 2014/04/09 22:12:41]

doco impact on ownership of picture and bitstream buffers.

[bbudge, 2014/04/11 17:15:16]

Done.

+   *
+   * @param[in] video_decoder A <code>PP_Resource</code> identifying the video
+   * decoder.
+   * @param[in] callback A <code>PP_CompletionCallback</code> to be called upon
+   * completion.
+   *
+   * @return An int32_t containing an error code from <code>pp_errors.h</code>.
+   */
+  int32_t Reset(
+      [in] PP_Resource video_decoder,
+      [in] PP_CompletionCallback callback);
+};

[Ami GONE FROM CHROMIUM, 2014/04/09 22:12:41]

How does teardown work?
(plugin needs to be able to indicate that its pending callbacks should not fire)

[bbudge, 2014/04/11 17:15:16]

The plugin should just release its references on the video decoder. Pepper proxy
code aborts all callbacks at that point. I removed the old Destroy method, since
it leaves the resource in a zombie state.

[igorc, 2014/04/11 22:16:28]

Would be great to document this.

[bbudge, 2014/04/16 00:51:05]

It's a Pepper convention, but I'll add a note at the top usage summary.