DBus MediaAnalyticsClient and media_perception pb.

extensions/common/api/media_perception_private.idl

Index: extensions/common/api/media_perception_private.idl
diff --git a/extensions/common/api/media_perception_private.idl b/extensions/common/api/media_perception_private.idl
new file mode 100644
index 0000000000000000000000000000000000000000..4b99c9d8c9147b4eeafa4c2caa2083f2fc1575bf
--- /dev/null
+++ b/extensions/common/api/media_perception_private.idl
@@ -0,0 +1,142 @@
+// Copyright 2017 The Chromium Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style license that can be
+// found in the LICENSE file.
+
+// Private API for communicating with and receiving real-time media perception
+// information from an computer vision + machine intelligence binary.
+[implemented_in = "extensions/browser/api/media_perception_private/media_perception_private_api.h"]

[tbarzic, 2017/04/27 20:37:36]

this is chromeos only, right?
can you add
platforms=("chromeos")

[Luke Sorenson, 2017/05/03 23:56:07]

Done.

+
+namespace mediaPerceptionPrivate {
+  enum Status {
+    ERROR, // An error occurred that needs to be propagated to the frontend.

[tbarzic, 2017/04/27 20:37:36]

Can you put the comments above the values?
Also, I'd add a new line after each enum value (for readability).

[Luke Sorenson, 2017/05/03 23:56:07]

Done.

+    TIMEOUT,  // Unable to reach media analysis process.
+    UNINITIALIZED, // Media analytics process waiting to be started.
+    STARTED,  // Analysis process running but not recieving frames.
+    RUNNING,  // Analysis process running and injesting frames.
+    SUSPENDED
+  };
+
+  // The system and configuration state of the analysis process and v4lplugin.
+  dictionary State {
+    Status? status;
+    // Optionally add device context to setState command for starting
+    // media analytics process, so that the media analytics process can
+    // better select the right video device to open.
+    DOMString? deviceContext;
+  };
+
+  dictionary Point {
+    // x represents the horizontal distance from the top left corner of the
+    // image to the point.
+    double? x;
+    // y represents the vertical distance from the top left corner of the
+    // image to the point.
+    double? y;
+  };
+
+  dictionary BoundingBox {
+    // Specifies whether the points are normalized to the size of the image.
+    // If not set, the points are not normalized.
+    boolean? normalized;
+    // The two points that define the corners of a bounding box.
+    Point? topLeft;
+    Point? bottomRight;
+  };
+
+  enum EntityType {
+    UNSPECIFIED,
+    FACE,
+    PERSON
+  };
+
+  dictionary Entity {
+    // A unique id associated with the detected entity, which can be used to
+    // track the entity over time.
+    long? id;
+
+    EntityType? type;
+
+    // Minimum box which captures entire detected entity.
+    BoundingBox? boundingBox;
+
+    // A value for the quality of this detection.
+    double? confidence;
+  };
+
+  // The set of computer vision metadata for an image frame.
+  dictionary FramePerception {
+    long? frameId;
+
+    long? frameWidthInPx;
+    long? frameHeightInPx;
+    // The timestamp associated with the frame (when its recieved by the
+    // analysis process).
+    double? timestamp;
+
+    // The list of entities detected in this frame.
+    Entity[]? entities;
+  };
+
+  dictionary MediaPerception {
+    // The timestamp attached with when this data originated from the analysis
+    // process.
+    double? timestamp;
+    // A single framePerception or possibly array of framePerceptions.

[tbarzic, 2017/04/27 20:37:36]

The type indicates that this will always be an array.

[Luke Sorenson, 2017/05/03 23:56:07]

Done.

+    FramePerception[]? framePerceptions;
+  };
+
+  dictionary ImageFrame {
+    long? width;
+    long? height;
+    // colorspace is defined in the same way as SimpleImage::ColorSpace.

[tbarzic, 2017/04/27 20:37:36]

What's SimpleImage?

[Luke Sorenson, 2017/05/03 23:56:07]

It's a class defined within google3, this entire dictionary will actually change
because we want to use JPEG encoded images but haven't yet figured out how we
want to represent them in idl. Adding a TODO for now.

+    long? colorspace;
+    // By default, 1 channel means Grayscale, 2 channels meangs Grayscale + Alpha,

[tbarzic, 2017/04/27 20:37:36]

Can the meaning of these values be different in any case?

[Luke Sorenson, 2017/05/03 23:56:07]

See above comment, this dictionary is not final and will change.

+    // 3 channels means RGB, and 4 channels means RGBA.
+    long? channels;
+    // TODO(lasoren): Add compression format marker.
+    // The bytes of the image frame.
+    ArrayBuffer? frame;
+  };
+
+  dictionary PerceptionSample {
+    // The video analytics FramePerception for the associated image frame
+    // data.
+    FramePerception? framePerception;
+    // The image frame data for the associated FramePerception object.
+    ImageFrame? imageFrame;
+  };
+
+  dictionary Diagnostics {
+    // A buffer of image frames and the associated video analytics to be sent
+    // for diagnostics (when a user reports malfunction).
+    PerceptionSample[]? perceptionSamples;
+  };
+
+  callback StateCallback = void(State state);
+
+  callback DiagnosticsCallback = void(Diagnostics diagnostics);
+
+  interface Functions {
+    // Get the status of the media perception process.
+    // |callback| : The current State of the system.
+    static void getState(StateCallback callback);

[tbarzic, 2017/04/27 20:37:36]

nit: new line after methods

[Luke Sorenson, 2017/05/03 23:56:07]

Done.

+    // Set the desired state of the system.
+    // |state| : A dictionary with the desired new state.
+    // |callback| : The State of the system after setting it. Verifies that
+    // the state was set as desired.
+    static void setState(State state, StateCallback callback);

[tbarzic, 2017/04/27 20:37:36]

So, what would be the meaning of calling setState({status:
ERROR/TIMEOUT/RUNNING}).
Is this intended to start/stop the mediaPerception daemon? Maybe it would be
better to replace this with a clearer API, e.g.
start(DeviceContext)
stop()

[fraz, 2017/05/02 21:57:29]

The only accestable setState values for State should be SUSPENDED and RUNNING,
all other states are transitional and informational from getState. The
transitional states should result in an error result from setState. The setState
model retains more flexibility inthe API as the map gorws to include other state
characteristics (eg runtime control) without exploding the top-level API.

[Luke Sorenson, 2017/05/03 23:56:07]

Echoing what Matt said, we prefer our API to SetState and GetState generically
so that we can easily extend the API to support different possible states for
the media analytics process, which we suspect will get more complex over time.

By using a generic SetState, we only need to modify the idl and media_perception
proto, rather than adding a whole new API Function.

[tbarzic, 2017/05/05 21:10:40]

Yes, having this generic setState is fine with me (unless you expect the set of
settable states not to change - which seems not to be the case).
Though, can you please document settable states here (or use different states
enum).

Also, what do you think about renaming this to requestState (logic being that
depending on it's state the platform may refuse to change the state) - this is
just a suggestion, feel free to disregard :)

[Luke Sorenson, 2017/05/08 19:06:06]

I expect that the settable states will almost certainly grow over time. Added
documentation to the idl in https://codereview.chromium.org/2860203002/

I can definitely see the name being requestState (something I discussed with
team) but as with all APIs, its never a guarantee that a request will succeed
and not return some error, so I think I will leave it as setState.

+    // Get a diagnostics buffer out of the video analytics process.
+    // |callback| : Returns a Diagnostics dictionary object which
+    // contains image frame data and associated detections to be logged.
+    static void getDiagnostics(DiagnosticsCallback callback);
+  };
+
+  interface Events {
+    // Fired when the analysis process has passed back to Chrome the current
+    // mediaPerception information.
+    // |mediaPerception| : The dictionary which contains a dump of everything
+    // the analysis process has detected or determined from the incoming media
+    // streams.
+    static void onMediaPerception(MediaPerception mediaPerception);
+  };
+};