MediaPerceptionPrivate IDL and skeleton.

extensions/common/api/media_perception_private.idl

Index: extensions/common/api/media_perception_private.idl
diff --git a/extensions/common/api/media_perception_private.idl b/extensions/common/api/media_perception_private.idl
new file mode 100644
index 0000000000000000000000000000000000000000..f08007eb5e0246e867d6bdc9991933feeaa25ddf
--- /dev/null
+++ b/extensions/common/api/media_perception_private.idl
@@ -0,0 +1,158 @@
+// Copyright 2017 The Chromium Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style license that can be
+// found in the LICENSE file.
+
+// Private API for communicating with and receiving real-time media perception

[tbarzic, 2017/05/08 22:21:07]

Private API for receiving real-time media perception information.

(the rest seems more like implementation detail)

[Luke Sorenson, 2017/05/09 17:41:05]

Done.

+// information from an computer vision + machine intelligence binary.
+[platforms=("chromeos")]
+

[tbarzic, 2017/05/08 22:21:07]

nit: no new line

[Luke Sorenson, 2017/05/09 17:41:05]

Done.

+namespace mediaPerceptionPrivate {
+  enum Status {
+    // An error occurred.
+    ERROR,
+
+    // Unable to reach media analysis process.

[tbarzic, 2017/05/08 22:21:08]

Attempt to reach media analysis process failed.

[Luke Sorenson, 2017/05/09 17:41:05]

Done.

+    TIMEOUT,
+
+    // Media analytics process waiting to be launched.
+    UNINITIALIZED,
+
+    // Analysis process running but is not yet receiving image frames. This is
+    // a transitional state between SUSPENDED and RUNNING, for the time it
+    // takes to warm up the media analysis pipeline.
+    STARTED,
+
+    // Analysis process running and injesting image frames. At this point,
+    // MediaPerception signals should be coming over D-Bus.
+    RUNNING,
+
+    // Analysis process is running and ready to be set to state RUNNING.

[tbarzic, 2017/05/08 22:21:07]

How about "Media analytics process is running, but not receiving image frames"?
And remove this sentence from the STARTED comment?
Or am I misunderstanding what the states represent?

[Luke Sorenson, 2017/05/09 17:41:05]

I added more clarifying comments to try to help make clear what each state
represents.

+    SUSPENDED
+  };
+
+  // The system and configuration state of the analysis process and v4lplugin.
+  dictionary State {
+    Status? status;

[tbarzic, 2017/05/08 22:21:07]

So, if understand this correctly, status will always be set here?
Maybe make it required, so the function call fails early in case someone tries
to call it with only deviceContext set?

[Luke Sorenson, 2017/05/09 17:41:05]

Done.

+    // Optionally add device context to setState command for starting
+    // media analytics process, so that the media analytics process can
+    // better select the right video device to open. Provide this value with
+    // status RUNNING when the process is currently in an UNINITIALIZED or
+    // SUSPENDED state.
+    DOMString? deviceContext;
+  };
+
+  dictionary Point {
+    // The horizontal distance from the top left corner of the image.
+    double? x;

[tbarzic, 2017/05/08 22:21:07]

It feels a little weird that the API allows a point with an unset coordinate,
but OK.
Would the app chose a default value to handle this, just ignore the point, or
ignore the whole message?

[Luke Sorenson, 2017/05/09 17:41:05]

We will be careful to always check that a value exists where we expect it to.
The app would most likely ignore the whole message in that case and log an error
because it really shouldn't happen, by having an optional value here we can more
easily identify when a member that really should be there is somehow not getting
through.

+    // The vertical distance from the top left corner of the image.
+    double? y;
+  };
+
+  dictionary BoundingBox {
+    // Specifies whether the points are normalized to the size of the image.
+    boolean? normalized;
+    // The two points that define the corners of a bounding box.
+    Point? topLeft;
+    Point? bottomRight;
+  };
+
+  enum EntityType {
+    UNSPECIFIED,
+    FACE,
+    PERSON
+  };
+
+  dictionary Entity {
+    // A unique id associated with the detected entity, which can be used to
+    // track the entity over time.
+    long? id;
+
+    EntityType? type;
+
+    // Minimum box which captures entire detected entity.
+    BoundingBox? boundingBox;
+
+    // A value for the quality of this detection.
+    double? confidence;
+  };
+
+  // The set of computer vision metadata for an image frame.
+  dictionary FramePerception {
+    long? frameId;
+
+    long? frameWidthInPx;
+    long? frameHeightInPx;
+    // The timestamp associated with the frame (when its recieved by the
+    // analysis process).
+    double? timestamp;
+
+    // The list of entities detected in this frame.
+    Entity[]? entities;
+  };
+
+  dictionary MediaPerception {
+    // The timestamp attached with when this data originated from the analysis

[tbarzic, 2017/05/08 22:21:07]

nit: The time the media perception data was emitted by the analysis process.

[Luke Sorenson, 2017/05/09 17:41:05]

Done.

+    // process. This value will be greater than the timestamp stored within the
+    // FramePerception dictionary and the difference between them can be viewed
+    // as the processing time for a single frame.
+    double? timestamp;
+    // An array of framePerceptions, often just one.
+    FramePerception[]? framePerceptions;
+  };
+
+  // TODO(lasoren): Change this interface based on the compressed images coming
+  // from the media analytics process.

[tbarzic, 2017/05/08 22:21:07]

the comments keep switching between using analytics process and analysis
process.. can you switch one of them to another.

[Luke Sorenson, 2017/05/09 17:41:05]

Done.

+  dictionary ImageFrame {
+    long? width;
+    long? height;
+
+    // TODO(lasoren): Add compression format marker.
+    // The bytes of the image frame.
+    ArrayBuffer? frame;
+  };
+
+  dictionary PerceptionSample {
+    // The video analytics FramePerception for the associated image frame
+    // data.
+    FramePerception? framePerception;
+    // The image frame data for the associated FramePerception object.
+    ImageFrame? imageFrame;
+  };
+
+  dictionary Diagnostics {
+    // A buffer of image frames and the associated video analytics to be sent
+    // for diagnostics (when a user reports malfunction).

[tbarzic, 2017/05/08 22:21:07]

"to be sent" seems like the app implementation detail (and is a little
misleading in this context, given that the API received this).

Maybe:
A buffer of image frames and the associated video analytics infrormation that
can be used to diagnose a malfunction.

[Luke Sorenson, 2017/05/09 17:41:05]

Done.

+    PerceptionSample[]? perceptionSamples;
+  };
+
+  callback StateCallback = void(State state);
+
+  callback DiagnosticsCallback = void(Diagnostics diagnostics);
+
+  interface Functions {
+    // Get the status of the media perception process.
+    // |callback| : The current State of the system.
+    static void getState(StateCallback callback);
+
+    // Set the desired state of the system.
+    // |state| : A dictionary with the desired new state. Settable states are
+    // RUNNING and SUSPENDED.
+    // |callback| : The State of the system after setting it. Verifies that
+    // the state was set as desired.

[tbarzic, 2017/05/08 22:21:07]

nit: s/Verifies that/Can be used to verify/

[Luke Sorenson, 2017/05/09 17:41:05]

Done.

+    static void setState(State state, StateCallback callback);
+
+    // Get a diagnostics buffer out of the video analytics process.
+    // |callback| : Returns a Diagnostics dictionary object which
+    // contains image frame data and associated detections to be logged.

[tbarzic, 2017/05/08 22:21:07]

I think it's evident from other comments what the diagnostics dictionary
contains :)

[Luke Sorenson, 2017/05/09 17:41:04]

Done.

+    static void getDiagnostics(DiagnosticsCallback callback);
+  };
+
+  interface Events {
+    // Fired when the analysis process has passed back to Chrome the current

[tbarzic, 2017/05/08 22:21:07]

"Fired when media perception information is received from media analytics
process."?

[Luke Sorenson, 2017/05/09 17:41:05]

Done.

+    // mediaPerception information.
+    // |mediaPerception| : The dictionary which contains a dump of everything
+    // the analysis process has detected or determined from the incoming media
+    // streams.
+    static void onMediaPerception(MediaPerception mediaPerception);
+  };
+};