MJPEG acceleration for video capture using VAAPI

content/browser/renderer_host/gpu_jpeg_decoder.h

+// Copyright 2015 The Chromium Authors. All rights reserved.

[wuchengli, 2015/05/04 14:14:41]

I know the current patchset has a bug and the performance data is not final. But
please add what you have to the change description so people know the rough
figure. We need two numbers -- CPU usage and latency. Should we use Broadwell or
Haswell Chromebooks for benchmark?

[kcwu, 2015/05/08 14:42:42]

Done.

+// Use of this source code is governed by a BSD-style license that can be
+// found in the LICENSE file.
+
+#ifndef CONTENT_BROWSER_RENDERER_HOST_GPU_JPEG_DECODER_H_
+#define CONTENT_BROWSER_RENDERER_HOST_GPU_JPEG_DECODER_H_
+
+#include "base/callback.h"
+#include "base/memory/ref_counted.h"
+#include "base/memory/scoped_ptr.h"
+#include "content/common/content_export.h"
+#include "media/video/capture/video_capture_device.h"
+#include "media/video/jpeg_decode_accelerator.h"
+
+namespace base {
+class MessageLoopProxy;
+}
+
+namespace media {
+class VideoFrame;
+}
+
+namespace content {
+
+// Adapter to GpuJpegDecodeAccelerator for VideoCaptureDevice::Client. It takes
+// care of GpuJpegDecodeAccelerator creation, shared memory, and threading
+// issues. If decoding fails, it will fallback to software decode for current
+// frame and reject upcoming frames.
+//
+// There are two threads involved in this class. The constructor,
+// destructor, and the decoding responses (JpegDecodeAccelerator::Client
+// functions) run on IO thread. Initialize() and DecodeCapturedData() run on
+// the other thread (i.e., device thread for Linux).
+class CONTENT_EXPORT GpuJpegDecoder
+    : public media::JpegDecodeAccelerator::Client {
+ public:
+  typedef base::Callback<void(
+      scoped_ptr<media::VideoCaptureDevice::Client::Buffer>,
+      const scoped_refptr<media::VideoFrame>&,
+      const base::TimeTicks&)> DecodeDoneCB;
+
+  // Returns true if JPEG hardware decoding is supported on this device.
+  static bool Supported();
+
+  // |device_client| is where we request VideoCaptureDevice::Client::Buffer as
+  // output buffer from and send decoded result to.
+  GpuJpegDecoder(media::VideoCaptureDevice::Client* device_client,
+                 const DecodeDoneCB& decode_done_cb);

[wuchengli, 2015/05/04 14:14:41]

Can GpuJpegDecoder call device_client_->OnIncomingCapturedVideoFrame() and let
OnIncomingCapturedVideoFrame call DoIncomingCapturedVideoFrameOnIOThread
directly if it is on IO thread? 

First, we already pass media::VideoCaptureDevice::Client to the constructor and
it's convenient to use it. Second, passing a VideoCaptureController and
|controller_| is harder to think of the lifecycle. For example,
VideoCaptureController is one layer further and is |controller_| always valid
when GpuJpegDecoder calls it?

[kcwu, 2015/05/08 14:42:42]

I found ReserveOutputBuffer can be done outside of DecodeCapturedData, so we
don't need to pass device_client into GpuJpegDecoder.

+  ~GpuJpegDecoder();
+
+  // Creates and intializes decoder in GPU side. Do nothing if already

[wuchengli, 2015/05/04 14:14:41]

"Do nothing if already initialized" is strange. The knowledge should be
maintained in the client.

[kcwu, 2015/05/08 14:42:42]

Done.

+  // initialized. Return falses if failed.

[wuchengli, 2015/05/04 14:14:41]

s/Return falses/Returns false/

[kcwu, 2015/05/08 14:42:42]

Done.

+  bool Initialize();
+
+  // Decodes a JPEG picture. Returns true iff GpuJpegDecoder is not in failed
+  // state and accepts this decode request.
+  bool DecodeCapturedData(const uint8* data,
+                          size_t in_buffer_size,
+                          const media::VideoCaptureFormat& frame_format,
+                          int rotation,
+                          const base::TimeTicks& timestamp);
+
+  // JpegDecodeAccelerator::Client implementation.
+  // These will be called on IO thread.
+  virtual void VideoFrameReady(int32_t buffer_id) override;
+  virtual void NotifyError(int32_t buffer_id,
+                           media::JpegDecodeAccelerator::Error error) override;
+
+ private:
+  // Returns true if the decoding of last frame is not finished yet.
+  bool IsDecoding_Locked();
+
+  // The "customer" of GpuJpegDecoder.
+  media::VideoCaptureDevice::Client* device_client_;
+
+  // The callback when decode succeed.
+  DecodeDoneCB decode_done_cb_;
+
+  // The capture thread.

[wuchengli, 2015/05/04 14:14:41]

Add documentation what a capture thread is. Without some context, a developer
(like me) does not know what that is.

[kcwu, 2015/05/08 14:42:42]

Done.

+  const scoped_refptr<base::MessageLoopProxy> capture_task_runner_;
+
+  // The underlying JPEG decode accelerator.
+  scoped_ptr<media::JpegDecodeAccelerator> decoder_;
+
+  // Guards |failed_|, |decode_done_closure_|, and |in_buffer_|.
+  base::Lock lock_;
+
+  bool initialized_;
+  bool failed_;

[wuchengli, 2015/05/04 14:14:41]

Add a status callback so you don't need |failed_|. See |status_cb| of
GpuVideoDecoder::Initialize. The client should decide what to do if decode
fails.

[wuchengli, 2015/05/06 07:59:18]

See the comment in DecodeCapturedData. If we return an enum from
DecodeCapturedData, we don't need the status callback and
video_capture_device_client.cc can clean |external_jpeg_decoder_| if returned
status is FATAL_ERROR.

[kcwu, 2015/05/08 14:42:42]

Done.

[kcwu, 2015/05/08 14:42:42]

Acknowledged.

+
+  // The closure of |decode_done_cb_| with bound parameters.
+  base::Closure decode_done_closure_;
+
+  // Next id for |in_buffer_|.
+  int32 next_bitstream_buffer_id_;
+
+  // Shared memory to store JPEG stream buffer. |in_buffer_| is backed by this.
+  scoped_ptr<base::SharedMemory> in_shared_memory_;
+
+  // JPEG stream buffer as input to JpegDecodeAccelerator.
+  media::BitstreamBuffer in_buffer_;
+
+  DISALLOW_COPY_AND_ASSIGN(GpuJpegDecoder);
+};
+
+}  // namespace content
+
+#endif  // CONTENT_BROWSER_RENDERER_HOST_GPU_JPEG_DECODER_H_