ScreenCapture for Android phase1, part I

media/capture/content/android/screen_capture_machine_android.cc

Index: media/capture/content/android/screen_capture_machine_android.cc
diff --git a/media/capture/content/android/screen_capture_machine_android.cc b/media/capture/content/android/screen_capture_machine_android.cc
new file mode 100644
index 0000000000000000000000000000000000000000..02d9c8861a0dc1177db9c88a6c967a8c057e4b55
--- /dev/null
+++ b/media/capture/content/android/screen_capture_machine_android.cc
@@ -0,0 +1,146 @@
+// Copyright 2015 The Chromium Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style license that can be
+// found in the LICENSE file.
+
+#include "media/capture/content/android/screen_capture_machine_android.h"
+
+#include "base/android/jni_android.h"
+#include "base/android/scoped_java_ref.h"
+#include "jni/ScreenCapture_jni.h"
+#include "media/base/video_capture_types.h"
+#include "media/base/yuv_convert.h"
+#include "media/capture/content/android/screen_capture_factory_android.h"
+#include "media/capture/content/video_capture_oracle.h"
+#include "third_party/libyuv/include/libyuv.h"
+
+using base::android::AttachCurrentThread;
+
+namespace media {
+
+// static
+bool ScreenCaptureMachineAndroid::RegisterScreenCaptureMachine(JNIEnv* env) {
+  return media::RegisterNativesImpl(env);
+}
+
+ScreenCaptureMachineAndroid::ScreenCaptureMachineAndroid() {}
+
+ScreenCaptureMachineAndroid::~ScreenCaptureMachineAndroid() {}
+
+void ScreenCaptureMachineAndroid::OnFrameAvailable(JNIEnv* env,
+                                                   jobject obj,
+                                                   jbyteArray data,
+                                                   jint cropWidth,
+                                                   jint cropHeight,
+                                                   jlong timestamp) {
+  const media::VideoCaptureOracle::Event event =
+      media::VideoCaptureOracle::kCompositorUpdate;
+  uint64_t absolute_micro =
+      static_cast<int64>(timestamp / base::Time::kNanosecondsPerMicrosecond);
+  const base::TimeTicks start_time =
+      base::TimeTicks::FromInternalValue(absolute_micro);
+  scoped_refptr<media::VideoFrame> frame;
+  media::ThreadSafeCaptureOracle::CaptureFrameCallback capture_frame_cb;
+
+  bool oracle_decision = oracle_proxy_->ObserveEventAndDecideCapture(

[miu, 2016/04/27 05:24:53]

I'm not familiar with the API provided by the Android platform, but would it be
possible for this decision to be made *before* the platform does all the work to
read-back the screen frame from the GPU?

For example, tab capture (WebContentsVideoCaptureDevice) uses an event-driven
"subscriber" model, where the compositor notifies it whenever compositing will
occur, and provides it with the damage region. Then, the oracle makes a decision
whether it will capture the next frame. So, the expensive operation of read-back
only happens when it absolutely needs to (saving CPU/GPU/power, smoother/faster
performance, etc.).

[braveyao, 2016/05/04 18:49:41]

Android will send a copy of the whole display whenever it updates the rendering
to screen. We just wait for the incoming frames, kind of same as camera capture.
So I suppose there is no way to do the decision beforehand. Any other concern on
this?

[miu, 2016/05/10 00:29:00]

Acknowledged. Seems the API designer assumed the platform knows better than the
application as to whether to do the expensive work. That's unfortunate. ;-)

+      event, gfx::Rect(), start_time, &frame, &capture_frame_cb);
+
+  if (!oracle_decision)
+    return;
+
+  jbyte* buffer = env->GetByteArrayElements(data, NULL);
+  if (!buffer)
+    return;
+
+  uint8* source_argb = reinterpret_cast<uint8_t*>(buffer);
+  int source_stride = cropWidth * 4;
+  // If needed, scale the captured frame to the required size.
+  if (gfx::Size(cropWidth, cropHeight) != frame->visible_rect().size()) {
+    std::unique_ptr<uint8[]> scaled_frame_data;
+    scaled_frame_data.reset(new uint8[frame->visible_rect().width() *
+                                      frame->visible_rect().height() * 4]);
+    libyuv::ARGBScale(source_argb, source_stride, cropWidth, cropHeight,

[miu, 2016/04/27 05:24:53]

Can the platform do this? Usually scaling is much more performant in the
GPU/compositing side of things, rather than on one thread on the CPU.

IIUC, the Java side of this implementation could create a VirtualDisplay with
the desired width/height/dpi to accomplish the scaling operation.

Also, how is screen rotation handled?  You may need to flip and/or
letterbox/pillarbox to account for 90/180/270 degrees.

[braveyao, 2016/05/04 18:49:41]

Exactly. You can do the setting to VirtualDisplay to get required frame size,
also call resize() to resize it on the fly.In phase 1 I only guarantee the
default setting working. Passing into frame size constraints may work, but I
haven't tested it.
Currently we will always get a VGA size capture in landscape position by
default. When device is in portrait position, the capture frame will have black
padding on both side. 

This piece of code doesn't run here. So I replace it with a DCHECK. Will study
it more in phase 2.

[miu, 2016/05/10 00:29:00]

On second thought, if the frame->visible_rect() is not known until this point
(because it is determined by the oracle), we always have to scale here. (The
frame->visible_rect() can change dynamically during a session.)

However, do consider setting the VirtualDisplay size to the maximum size
requested in the media::VideoCaptureParams passed to
ScreenCaptureMachineAndroid::Start(). There's no sense in copying/reading-back
more data than is absolutely needed.

[braveyao, 2016/05/18 00:41:46]

Done.

+                      scaled_frame_data.get(),
+                      frame->visible_rect().width() * 4,
+                      frame->visible_rect().width(),
+                      frame->visible_rect().height(), libyuv::kFilterBilinear);
+    source_argb = scaled_frame_data.get();
+    source_stride = frame->visible_rect().width() * 4;
+  }
+
+  // Populate the VideoFrame.
+  DCHECK(frame->format() == PIXEL_FORMAT_I420 ||
+         frame->format() == PIXEL_FORMAT_YV12);
+
+  media::ConvertRGB32ToYUV(

[miu, 2016/04/27 05:24:53]

Is it possible to push colorspace conversion upstream as well?  On the Java
side, ImageReader could be created with ImageFormat.YUV_420_888 format.

[braveyao, 2016/05/04 18:49:41]

Thanks for the advice. I didn't consider this before. With a quick test, I got a
same issue as reported in
https://code.google.com/p/android/issues/detail?id=180348 on N9, which has no
such problem on N5.
Anyway, add a TODO for a further study in phase 2.

[miu, 2016/05/10 00:29:00]

In Chromium code, we have to work on many different hardware configurations.
It's typical to have to purposely write extra code to workaround issues like
this.

So, I would suggest you always attempt to use the ImageReader w/ YV12 output,
and if you catch the exception indicated in the bug, re-create the reader in
RGBA format to allow the pipeline in a "fall-back mode" where it will work at
lower performance on buggy platforms.

Your code here could simply be:

  if (data_is_in_argb) {
    media::ConvertRGB32ToYUV(...);
  } else {
    for each plane in (Y, U, V) {
      memcpy(...);
    }
  }

[braveyao, 2016/05/18 00:41:46]

Done.

BTW: YUV420 has better performance, but worse compatibility. There is only one
device in five at my hands that supports YUV. Even on a N5 android L, it will
silently fail. So I have to only enable YUV on M devices. Need future
investigation.

+      source_argb, frame->visible_data(media::VideoFrame::kYPlane),
+      frame->visible_data(media::VideoFrame::kUPlane),
+      frame->visible_data(media::VideoFrame::kVPlane),
+      frame->visible_rect().width(), frame->visible_rect().height(),
+      source_stride, frame->stride(media::VideoFrame::kYPlane),
+      frame->stride(media::VideoFrame::kUPlane));
+
+  // Deliver the populated VideoFrame.
+  capture_frame_cb.Run(frame, start_time, true);
+
+  env->ReleaseByteArrayElements(data, buffer, JNI_ABORT);
+}
+
+void ScreenCaptureMachineAndroid::OnActivityResult(JNIEnv* env,
+                                                   jobject obj,
+                                                   jint result) {
+  if (result == 0) {
+    oracle_proxy_->ReportError(FROM_HERE, "The user denied screen capture");
+    return;
+  }
+
+  media::Java_ScreenCapture_startCapture(env, obj);
+}
+
+void ScreenCaptureMachineAndroid::Start(
+    const scoped_refptr<media::ThreadSafeCaptureOracle>& oracle_proxy,
+    const media::VideoCaptureParams& params,
+    const base::Callback<void(bool)> callback) {
+  DCHECK(oracle_proxy.get());
+  oracle_proxy_ = oracle_proxy;
+
+  j_capture_.Reset(
+      media::ScreenCaptureFactoryAndroid::createScreenCaptureMachineAndroid(
+          reinterpret_cast<intptr_t>(this)));
+
+  if (j_capture_.obj() == NULL) {
+    LOG(ERROR) << "Failed to createScreenCaptureAndroid,"
+                  " Maybe android version is too low";
+    callback.Run(false);
+    return;
+  }
+
+  capture_format_ = params.requested_format;
+  // This capturer always outputs ARGB, non-interlaced.
+  capture_format_.pixel_format = media::PIXEL_FORMAT_ARGB;

[miu, 2016/04/27 05:24:53]

I believe the |params| would specify I420 or YV12, since that's the format of
the VideoFrames being produced by ScreenCaptureMachineAndroid.

[braveyao, 2016/05/04 18:49:41]

Yes, |params| asks for I420. Same as above comment, I will check if we could get
MediaProjection/ImageReader work with that format in phase 2. And here we only
care about the frame size. So remove the unnecessary stuff.

[miu, 2016/05/10 00:29:00]

This is orthogonal to the other comment. The |capture_format_.pixel_format|
property is referring to the format of the frames being output by this
media::VideoCaptureDevice, and has nothing to do with how they are sourced from
the operating system. In other words, if you get RGBA frames from Android, the
client is the one requiring that they be converted to YV12 for consumption by
downstream Chromium code.

Actually, ScreenCaptureDeviceCore already checks for I420-only format:
https://code.google.com/p/chromium/codesearch#chromium/src/media/capture/cont...

Anyway, it seems you've removed this code in the latest Patch Set, so nothing to
do here.

+
+  CHECK(capture_format_.frame_size.GetArea() > 0);
+  CHECK(!(capture_format_.frame_size.width() % 2));
+  CHECK(!(capture_format_.frame_size.height() % 2));
+
+  JNIEnv* env = AttachCurrentThread();
+  jboolean ret = media::Java_ScreenCapture_startPrompt(
+      env, j_capture_.obj(), capture_format_.frame_size.width(),
+      capture_format_.frame_size.height());
+  if (!ret) {
+    callback.Run(false);
+    return;
+  }
+
+  callback.Run(true);
+}
+
+void ScreenCaptureMachineAndroid::Stop(const base::Closure& callback) {
+  JNIEnv* env = AttachCurrentThread();
+  Java_ScreenCapture_stopCapture(env, j_capture_.obj());
+
+  callback.Run();
+}
+
+void ScreenCaptureMachineAndroid::MaybeCaptureForRefresh() {}

[miu, 2016/04/27 05:24:53]

Depending on the behavior of the platform, you may need to implement this
properly.

Ideal: Platform only calls OnFrameAvailable() whenever the screen changed. That
means a non-changing screen would effectively "pause" capture. If this can
happen, then MaybeCaptureForRefresh() needs to force frame captures to satisfy
downstream consumers needing the extra "refresh" frames (e.g., video encoder,
streaming implementations, etc.).

Power-hungry/Wasteful behavior: Platform always calls OnFrameAvailable() at a
constant framerate, regardless of whether the screen content has changed. In
this case, you don't need to implement MaybeCaptureForRefresh() because a frame
will be produced in the near future anyway.

[braveyao, 2016/05/04 18:49:41]

Thanks for explaining this new method.
Android also only calls OnFrameAvailable() whenever the screen changed. So the
current consequence is the sending fps will drop to 0 in some cases. What's the
impact here?
I see in ThreadSafeCaptureOracle::AttemptPassiveRefresh() if it returns true,
then it won't call to MaybeCaptureForRefresh(). Since Android works in passive
way too, e.g. no API to ask for one frame from it, so I think we can make it
stop there. Is that right? How can I guarantee it?

Anyway it's also a phase 2 job. Will give a further study later.

[miu, 2016/05/10 00:29:00]

There are concerns on the consuming-end of these video frames. This is explained
in detail here:
https://code.google.com/p/chromium/codesearch#chromium/src/media/capture/vide...

There were several important user-facing bugs resolved by this new API, so not
implementing this is asking for trouble. ;-)
No. The passive refresh will not always succeed. It depends on whether the
buffer pool can "resurrect" the last frame that was sent. Timing, performance,
and environmental conditions will influence that, which means it is not
guaranteed on any platform.

Is there an API for force-capturing the screen? Otherwise, you'll want to always
hold onto the last Image's bytes so that "active" refresh can be handled
correctly.
How so? IIUC, "phase 1" is about implementing video frame capture and "phase 2"
is about connecting that into the existing video stack framework in libcontent.

[braveyao, 2016/05/18 00:41:46]

Done.

+
+}  // namespace media