MJPEG acceleration for video capture using VAAPI

content/browser/renderer_host/gpu_jpeg_decode_accelerator_adapter.h

Index: content/browser/renderer_host/gpu_jpeg_decode_accelerator_adapter.h
diff --git a/content/browser/renderer_host/gpu_jpeg_decode_accelerator_adapter.h b/content/browser/renderer_host/gpu_jpeg_decode_accelerator_adapter.h
new file mode 100644
index 0000000000000000000000000000000000000000..345ef1e72091741ce57b6e49e6effcd676fab0fc
--- /dev/null
+++ b/content/browser/renderer_host/gpu_jpeg_decode_accelerator_adapter.h
@@ -0,0 +1,124 @@
+// Copyright 2015 The Chromium Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style license that can be
+// found in the LICENSE file.
+
+#ifndef CONTENT_BROWSER_RENDERER_HOST_GPU_JPEG_ACCELERATOR_ADAPTER_H_
+#define CONTENT_BROWSER_RENDERER_HOST_GPU_JPEG_ACCELERATOR_ADAPTER_H_
+
+#include "base/memory/ref_counted.h"
+#include "base/memory/scoped_ptr.h"
+#include "content/common/content_export.h"
+#include "media/video/capture/video_capture_device.h"
+#include "media/video/jpeg_decode_accelerator.h"
+
+namespace base {
+class MessageLoopProxy;
+}
+
+namespace content {
+
+// Adapter to GpuJpegDecodeAccelerator for VideoDeviceDevice::Client. It takes

[mcasas, 2015/04/16 23:55:14]

s/VideoDeviceDevice/VideoCaptureDevice/

[kcwu, 2015/04/20 17:48:00]

Done.

+// care GpuJpegDecodeAccelerator creation, shared memory, and thread issues.

[mcasas, 2015/04/16 23:55:14]

"care of"
"threading issues"

[kcwu, 2015/04/20 17:48:00]

Done.

+// If decoding failed, it will fallback to software decode for current frame

[mcasas, 2015/04/16 23:55:14]

s/failed/fails/

[kcwu, 2015/04/20 17:48:00]

Done.

+// and reject upcoming frames.
+//
+// This class is mainly accessed by device thread. The decoding responses

[mcasas, 2015/04/16 23:55:14]

Please also mention that it's created and destroyed on Capture Thread.

[kcwu, 2015/04/20 17:48:00]

Done.

+// (JpegDecodeAccelerator::Client) are on IO thread. Since it only decodes a
+// frame at a time, the access is almost mutual exclusive: device thread can
+// only access members if IsDecoding() is false. IO thread can only access if
+// IsDecoding() is true.
+class CONTENT_EXPORT GpuJpegDecodeAcceleratorAdapter
+    : public media::JpegDecodeAccelerator::Client {
+ public:
+  // A lightweight check for caller to avoid IPC latency for known unsupported
+  // platform. Initialize() can do the real platform supporting check but it
+  // requires an IPC.
+  static bool Supported();
+
+  // |device_client| is where we request VideoCaptureDevice::Client::Buffer as
+  // output buffer from and send decoded result to.
+  GpuJpegDecodeAcceleratorAdapter(
+      media::VideoCaptureDevice::Client* device_client);
+  ~GpuJpegDecodeAcceleratorAdapter();
+
+  // Create and intialize decoder in GPU side. Return false if failed.
+  bool Initialize();
+  // Return true if in failed state. If so, the caller,
+  // VideoCaptureDevice::Client, should do software decoding by itself.
+  bool IsFailed();
+
+  // Decode JPEG stream. Parameters are modeled after
+  // VideoCaptureDeviceClient::OnIncomingCapturedData.
+  void DecodeCapturedData(const uint8* data,
+                          int length,
+                          const media::VideoCaptureFormat& frame_format,
+                          int rotation,
+                          const base::TimeTicks& timestamp);
+
+  // JpegDecodeAccelerator::Client implementation.
+  // These will be called in IO thread.
+  virtual void VideoFrameReady(int32_t buffer_id) override;
+  virtual void NotifyError(int32_t buffer_id,
+                           media::JpegDecodeAccelerator::Error error) override;
+
+ private:
+  // Fail state indicats to use software decode. See comment in

[mcasas, 2015/04/16 23:55:14]

s/indicats/indicates/

[kcwu, 2015/04/20 17:48:00]

Done.

+  // FallbackToSoftwareDecode for detail.
+  enum FailState { NOT_FAIL, FAILING, FAILED };
+
+  // For keeping parameters of DecodeCapturedData. In case we need to pass them
+  // to software decoder if hardware decoding failed.
+  struct CapturedData {
+    uint8* data;
+    int length;
+    media::VideoCaptureFormat frame_format;
+    int rotation;
+    base::TimeTicks timestamp;
+  };
+
+  bool IsDecoding();
+  // Check |bitstream_buffer_id| from GPU process is expected.
+  bool IsExpectedDecodeResponse(int32 bitstream_buffer_id);
+  // Current frame decoding is done. Allow DecodeCapturedData() to accept next
+  // frame.
+  void DecodeDone();
+  // Fallback current frame to software decode. Set flag to reject future
+  // upcoming frames.
+  void FallbackToSoftwareDecode();

[mcasas, 2015/04/16 23:55:14]

I don't think this is really needed: If the current frame does not
get decoded correctly, just discard it and mark the error
condition. 

Moreover, that would allow you to not hold on to |captured_data_|,
note that all VideoCaptureDevice assume that the Client is done
with |data| when returning from OnIncomingCapturedXXX(), so 
|captured_data_.data| accessing should be avoided. In linux case
in particular, this pointer is returned to the V4L2 driver.

[kcwu, 2015/04/20 17:48:00]

That means for certain device or (external) camera, the first frame is always
got drop (and thus the latency is 33ms longer than others). Is this acceptable?
Before OnIncomingCapturedXXX() returns, the data is copied to
|in_shared_memory_| and |captured_data_.data| is replaced by the pointer to
|in_shared_memory_|. So accessing |captured_data_.data| is not a problem.

[mcasas, 2015/04/28 00:04:49]

IIUC, the drop-first-frame would be an error condition on those 
devices, correct? IOW if the decoder accelerator works, that
frame would not be lost. So dropping the first frame is a corner
case of a particular hardware support, and then it doesn't matter
much if the delay is increased by a frame, and it simplifies the
code.
So |captured_data|.data| points to two different locations? 
I'll take a look on the latest patchset so you don't have to
come back in history to this PS.

[kcwu, 2015/05/08 14:42:40]

Acknowledged.

+
+  // The "customer" of GpuJpegDecodeAcceleratorAdapter.
+  media::VideoCaptureDevice::Client* device_client_;
+  // The main working thread. Used for DCHECK current thread.
+  scoped_refptr<base::MessageLoopProxy> device_thread_;
+  // All decoding requests are sent to |decoder_|.
+  scoped_ptr<media::JpegDecodeAccelerator> decoder_;
+
+  // Below fields are protected by |lock_|.
+  base::Lock lock_;
+
+  FailState fail_state_;
+
+  // Keep the captured data from camera. If hardware decode failed, pass this
+  // to software decoder as fallback.
+  CapturedData captured_data_;

[mcasas, 2015/04/16 23:55:14]

See my comments above, I think we could get rid of this.

[kcwu, 2015/05/08 14:42:40]

Done.

+
+  // Next id for |in_buffer_|.
+  int32 next_bitstream_buffer_id_;
+
+  // Shared memory to store JPEG stream buffer. |in_buffer_| is backed by this.
+  scoped_ptr<base::SharedMemory> in_shared_memory_;
+  // JPEG stream buffer as input to JpegDecodeAccelerator.
+  media::BitstreamBuffer in_buffer_;
+
+  // Buffer to hold decoded output of JPEG decoder.
+  // |out_frame_| is backed by this.
+  scoped_refptr<media::VideoCaptureDevice::Client::Buffer> out_buffer_;
+  // VideoFrame to receive decoded output of JpegDecodeAccelerator.
+  scoped_refptr<media::VideoFrame> out_frame_;
+
+  DISALLOW_COPY_AND_ASSIGN(GpuJpegDecodeAcceleratorAdapter);
+};
+
+}  // namespace content
+
+#endif  // CONTENT_BROWSER_RENDERER_HOST_GPU_JPEG_ACCELERATOR_ADAPTER_H_