ScreenCapture for Android phase1, part I

media/base/android/java/src/org/chromium/media/ScreenCapture.java

Index: media/base/android/java/src/org/chromium/media/ScreenCapture.java
diff --git a/media/base/android/java/src/org/chromium/media/ScreenCapture.java b/media/base/android/java/src/org/chromium/media/ScreenCapture.java
new file mode 100644
index 0000000000000000000000000000000000000000..c33e06c34ce4c1f44be08f362451fc225c1bd78d
--- /dev/null
+++ b/media/base/android/java/src/org/chromium/media/ScreenCapture.java
@@ -0,0 +1,281 @@
+// Copyright 2015 The Chromium Authors. All rights reserved.

[mcasas, 2016/04/27 01:19:26]

2016

[miu, 2016/04/27 05:24:53]

Hmm...these files are so far away (in the directory hierarchy) from their C++
modules. Is this common practice when doing Java in the Chromium project? Would
it be easy to move these into media/capture/content/android/...?

[braveyao, 2016/05/04 18:49:40]

Done.

[braveyao, 2016/05/04 18:49:40]

It seems like a common practice to gather all the JAVA files under one place for
each component of chromium, such as chrome/android/java, mojo/public/java, also
media/base/android/java. As to media/base/android, once all the Java files for
audio/video/mediacodec are located there. It's very recently that the Java files
for video capture are moved to media/capture/vdieo/android, while all others
remain at same place. So I'm a little confusing about the moving of video
capture. But still think it's good to put the screen capture java files with
others. WDYT?

[mcasas, 2016/05/04 19:56:47]

What do you mean "once all the Java files ..." 
Java files, like any other, are grouped in any way
that makes sense due to functionality, not language. 
I agree with miu@ and my comment in 
media/base/android/media_jni_registrar.cc
that these files should go elsewhere.

[whywhat, 2016/05/04 20:22:39]

FWIW, in general the corresponding file for src/foo/bar/boo/far/xyz.cc would
live in src/foo/android/java/src/org/chromium/foo/bar/boo/far/XYZ.java

So in this case and all other media Java files, the paths would look like
src/media/android/java/src/org/chromium/{base,capture}/...

However, it seems like currently Java files under media/ use a different
structure:

src/media/foo/boo/bar.cc corresponds to
src/media/foo/boo/android/java/src/org/chromium/media/Bar.java (examples are
media/base and media/capture/video except that two Java files corresponding to
media/audio/ counterparts are placed under media/base/android/java with all the
rest -- I agree it may look confusing).

Therefore the consistent path to this Java file seems to be
src/media/capture/content/android/java/src/org/chromium/media/ScreenCapture[Machine].java

I agree that until the files under media follow the general rule, this makes
more sense than putting them under media/base/android/java.

[braveyao, 2016/05/18 00:41:46]

Done.

+// Use of this source code is governed by a BSD-style license that can be
+// found in the LICENSE file.
+
+package org.chromium.media;
+
+import android.annotation.TargetApi;
+import android.app.Activity;
+import android.app.Fragment;
+import android.app.FragmentManager;
+import android.app.FragmentTransaction;
+import android.content.Context;
+import android.content.Intent;
+import android.graphics.PixelFormat;
+import android.hardware.display.DisplayManager;
+import android.hardware.display.VirtualDisplay;
+import android.media.Image;
+import android.media.ImageReader;
+import android.media.projection.MediaProjection;
+import android.media.projection.MediaProjectionManager;
+import android.os.Build;
+import android.os.Handler;
+import android.os.HandlerThread;
+import android.util.DisplayMetrics;
+import android.view.Display;
+import android.view.Surface;
+
+import org.chromium.base.ApplicationStatus;
+import org.chromium.base.Log;
+import org.chromium.base.annotations.CalledByNative;
+import org.chromium.base.annotations.JNINamespace;
+
+import java.nio.ByteBuffer;
+
+/**
+ * This class implements Screen Capture using projection API, introduced in Android
+ * API 21 (L Release). Capture takes place in the current Looper, while pixel
+ * download takes place in another thread used by ImageReader.
+ **/
+@JNINamespace("media")
+@TargetApi(Build.VERSION_CODES.LOLLIPOP)
+public class ScreenCapture extends Fragment {
+    // Internal class implementing the ImageReader listener. Gets pinged when a
+    // new frame is been captured and downloaded to memory-backed buffers.
+    private class CrImageReaderListener implements ImageReader.OnImageAvailableListener {

[mcasas, 2016/04/27 01:19:26]

This is _very_ similar class to [1] except for the
synchronized part and that here we only accept 
RGBA_8888. Please do not duplicate code, extend it
here.

https://code.google.com/p/chromium/codesearch#chromium/src/media/capture/vide...

[braveyao, 2016/05/04 18:49:40]

As we discussed offline. I understand the concern here. But not very comfortable
to link ScreenCapture with VideoCaptureCamera2. It's too troublesome to extend
it here and not good for readability as I imagine. Maybe we just keep it so far.

[mcasas, 2016/05/04 19:56:47]

It's not complicated. You're probably confused as to
where to put this common class, since content/ and video/
captures are in different folders, but in Java, everything
is reorganised as packages! (See line 5 of this file).

You need to make a new Java file, call it
CrImageReaderListener, and make sure it goes to
package org.chromium.media, then you can import
it from any other files, including this one, and
just extend it here by doing:

private class MyCrImageReaderListener extends CrImageReaderListener {

    @Override 
    public void OnImageAvailable(ImageReader reader) {
        // copy here the synchronised part...

        super.OnImageAvailable( pixelFormat )
    }
}

where you have also added a parameter pixelFormat to
CrImageReaderListener.OnImageAvailable and you're using
it in the comparison of [1].

[1]
https://code.google.com/p/chromium/codesearch#chromium/src/media/capture/vide...

[braveyao, 2016/05/18 00:41:46]

Tried a bit and saw your point. Added a TODO for improvement after we settle
down all the buffer-format/jni-interface stuffs. This deserves a single cl.

+        @Override
+        public void onImageAvailable(ImageReader reader) {
+            synchronized (mCaptureStateLock) {
+                if (mCaptureState != CaptureState.STARTED) {
+                    Log.e(TAG, "Get captured frame in unexpected state.");
+                    return;
+                }
+            }
+
+            Image image = null;
+            try {
+                image = reader.acquireLatestImage();
+                if (image == null) return;
+                if (image.getFormat() != PixelFormat.RGBA_8888) {
+                    Log.e(TAG, "Unexpected image format: " + image.getFormat() + " or #planes: "
+                                    + image.getPlanes().length);
+                    return;
+                }
+
+                Image.Plane[] planes = image.getPlanes();
+                ByteBuffer buffer = (ByteBuffer) planes[0].getBuffer();
+                buffer.get(mCapturedData);
+
+                nativeOnFrameAvailable(mNativeScreenCaptureMachineAndroid, mCapturedData,
+                        image.getCropRect().width(), image.getCropRect().height(),
+                        image.getTimestamp());
+            } catch (IllegalStateException ex) {
+                Log.e(TAG, "acquireLatestImage():" + ex);
+                return;
+            } finally {
+                if (image != null) image.close();
+            }
+        }
+    }
+
+    private class MediaProjectionCallback extends MediaProjection.Callback {
+        @Override
+        public void onStop() {
+            changeCaptureStateAndNotify(CaptureState.STOPPED);
+            mMediaProjection = null;
+            if (mVirtualDisplay == null) return;
+            mVirtualDisplay.release();
+            mVirtualDisplay = null;
+        }
+    }
+
+    private byte[] mCapturedData;
+    private final Context mContext;
+
+    private MediaProjection mMediaProjection;
+    private MediaProjectionManager mMediaProjectionManager;
+    private VirtualDisplay mVirtualDisplay;
+    private static final int REQUEST_MEDIA_PROJECTION = 1;
+
+    private Surface mSurface;
+    private ImageReader mImageReader = null;
+    private int mScreenDensity;
+    private int mWidth;
+    private int mHeight;
+    private int mResultCode;
+    private Intent mResultData;
+
+    // Native callback context variable.
+    private final long mNativeScreenCaptureMachineAndroid;
+
+    private static enum CaptureState { ATTACHED, ALLOWED, STARTED, STOPPING, STOPPED }
+    private CaptureState mCaptureState = CaptureState.STOPPED;
+    private final Object mCaptureStateLock = new Object();
+
+    private static final String TAG = "ScreenCaptureMachine";
+
+    ScreenCapture(Context context, long nativeScreenCaptureMachineAndroid) {
+        mContext = context;
+        mNativeScreenCaptureMachineAndroid = nativeScreenCaptureMachineAndroid;
+        try {
+            Activity activity = ApplicationStatus.getLastTrackedFocusedActivity();
+            if (activity == null) {
+                Log.e(TAG, "activity is null");
+                return;
+            }
+
+            FragmentManager fragmentManager = activity.getFragmentManager();
+            FragmentTransaction fragmentTransaction = fragmentManager.beginTransaction();
+            fragmentTransaction.add(this, "screencapture");
+            fragmentTransaction.commit();
+
+            DisplayMetrics metrics = new DisplayMetrics();
+            Display display = activity.getWindowManager().getDefaultDisplay();
+            display.getMetrics(metrics);
+            mScreenDensity = metrics.densityDpi;
+        } catch (Exception e) {

[mcasas, 2016/04/27 01:19:26]

No-no, don't catch generic exceptions out of a 
large code chunk [1] (Chromium Java style guide
follows the Android one).

[1]
https://source.android.com/source/code-style.html#dont-catch-generic-exception

[braveyao, 2016/05/04 18:49:40]

Done.

[mcasas, 2016/05/04 19:56:47]

Still you need to not catch Exception, but
each individual one being thrown, like in

https://code.google.com/p/chromium/codesearch#chromium/src/media/capture/vide...

[braveyao, 2016/05/18 00:41:46]

I'm aware of this. But as to the methods here, there is no specified exceptions
named in the API guidance.
For example, to FragmentTransaction.commit(), it only says " If the commit is
attempted after that point, an exception will be thrown.",
http://developer.android.com/reference/android/app/FragmentTransaction.html#c....
Then maybe catch the general exception is still a better option?

+            Log.e(TAG, "ScreenCaptureExcaption " + e);
+        }
+    }
+
+    @Override
+    public void onAttach(Context context) {
+        super.onAttach(context);
+        Log.d(TAG, "onAttach");
+        changeCaptureStateAndNotify(CaptureState.ATTACHED);
+    }
+
+    // This method was deprecated in API level 23 by onAttach(Context).

[mcasas, 2016/04/27 01:19:26]

// TODO(braveyao)
make a crbug.com for it and link it here.

[braveyao, 2016/05/04 18:49:40]

Done.
Not add a crbug though, since this will exist for many years before API 23 is
the minSdkVersion.

+    @SuppressWarnings("deprecation")
+    @Override
+    public void onAttach(Activity activity) {
+        super.onAttach(activity);
+        Log.d(TAG, "onAttach");
+        changeCaptureStateAndNotify(CaptureState.ATTACHED);
+    }
+
+    @Override
+    public void onDetach() {
+        super.onDetach();
+        Log.d(TAG, "onDetach");
+        stopCapture();
+    }
+
+    @CalledByNative
+    public boolean startPrompt(int width, int height) {
+        Log.d(TAG, "startPrompt");
+        synchronized (mCaptureStateLock) {
+            while (mCaptureState != CaptureState.ATTACHED) {
+                try {
+                    mCaptureStateLock.wait();
+                } catch (InterruptedException ex) {
+                    Log.e(TAG, "ScreenCaptureEvent: " + ex);

[mcasas, 2016/04/27 01:19:26]

It's not an event, is an exception.

[braveyao, 2016/05/04 18:49:40]

Done.

+                }
+            }
+        }
+
+        try {
+            mWidth = width;
+            mHeight = height;
+            int expectedFrameSize = mWidth * mHeight * 4;
+            mCapturedData = new byte[expectedFrameSize];
+
+            mMediaProjectionManager = (MediaProjectionManager) mContext.getSystemService(
+                    Context.MEDIA_PROJECTION_SERVICE);
+            if (mMediaProjectionManager == null) {
+                Log.e(TAG, "mMediaProjectionManager is null");
+                return false;
+            }
+
+            startActivityForResult(
+                    mMediaProjectionManager.createScreenCaptureIntent(), REQUEST_MEDIA_PROJECTION);
+        } catch (Exception e) {
+            Log.e(TAG, "ScreenCaptureExcaption " + e);

[mcasas, 2016/04/27 01:19:26]

typo

[braveyao, 2016/05/04 18:49:40]

Done.

+            return false;
+        }
+        return true;
+    }
+
+    @Override
+    public void onActivityResult(int requestCode, int resultCode, Intent data) {
+        if (requestCode == REQUEST_MEDIA_PROJECTION) {
+            int result;
+            if (resultCode != Activity.RESULT_OK) {
+                result = 0;
+            } else {
+                result = 1;

[mcasas, 2016/04/27 01:19:26]

Maybe |result| should be boolean?

[braveyao, 2016/05/04 18:49:40]

Done.

+                mResultCode = resultCode;
+                mResultData = data;
+                changeCaptureStateAndNotify(CaptureState.ALLOWED);
+            }
+            nativeOnActivityResult(mNativeScreenCaptureMachineAndroid, result);
+        }
+    }
+
+    // start screen capture.

[mcasas, 2016/04/27 01:19:26]

s/start/Starts/
But, honestly, this comment and l.248 are quite
superfluous.

[braveyao, 2016/05/04 18:49:40]

Done.

+    @CalledByNative
+    public void startCapture() {
+        Log.d(TAG, "startCapture");
+        synchronized (mCaptureStateLock) {
+            if (mCaptureState != CaptureState.ALLOWED) {
+                Log.e(TAG, "startCapture() invoked without user permission.");
+                return;
+            }
+        }
+        mMediaProjection = mMediaProjectionManager.getMediaProjection(mResultCode, mResultData);
+        if (mMediaProjection == null) {
+            Log.d(TAG, "mMediaProjection is null");

[mcasas, 2016/04/27 01:19:26]

Log.e

[braveyao, 2016/05/04 18:49:40]

Done.

+            return;
+        }
+        mMediaProjection.registerCallback(new MediaProjectionCallback(), null);
+
+        final int maxImages = 2;
+        mImageReader = ImageReader.newInstance(mWidth, mHeight, PixelFormat.RGBA_8888, maxImages);
+        mSurface = mImageReader.getSurface();
+        HandlerThread thread = new HandlerThread("Screen");
+        thread.start();
+        final Handler backgroundHandler = new Handler(thread.getLooper());
+        final CrImageReaderListener imageReaderListener = new CrImageReaderListener();
+        mImageReader.setOnImageAvailableListener(imageReaderListener, backgroundHandler);
+
+        mVirtualDisplay = mMediaProjection.createVirtualDisplay("ScreenCapture", mWidth, mHeight,
+                mScreenDensity, DisplayManager.VIRTUAL_DISPLAY_FLAG_AUTO_MIRROR, mSurface, null,
+                null);
+
+        changeCaptureStateAndNotify(CaptureState.STARTED);
+    }
+
+    // Stops screen capture.
+    @CalledByNative
+    public void stopCapture() {
+        Log.d(TAG, "stopCapture");
+        synchronized (mCaptureStateLock) {
+            if (mMediaProjection != null && mCaptureState == CaptureState.STARTED) {
+                mMediaProjection.stop();
+                changeCaptureStateAndNotify(CaptureState.STOPPING);
+            }
+
+            while (mCaptureState != CaptureState.STOPPED) {
+                try {
+                    mCaptureStateLock.wait();
+                } catch (InterruptedException ex) {
+                    Log.e(TAG, "ScreenCaptureEvent: " + ex);
+                }
+            }
+        }
+    }
+
+    private void changeCaptureStateAndNotify(CaptureState state) {
+        synchronized (mCaptureStateLock) {
+            mCaptureState = state;
+            mCaptureStateLock.notifyAll();
+        }
+    }
+
+    // Method for ScreenCapture implementations to call back native code.
+    private native void nativeOnFrameAvailable(long nativeScreenCaptureMachineAndroid, byte[] data,
+            int cropWidth, int cropHeight, long timestamp);
+
+    // Method for ScreenCapture implementations to call back native code.
+    private native void nativeOnActivityResult(long nativeScreenCaptureMachineAndroid, int result);
+}