[Mojo Video Capture] Replace RESOURCE_UTILIZATION with interface ReceiverLoadObserver

media/capture/video/video_capture_device.h

 // Copyright (c) 2012 The Chromium Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file.
 //
 // VideoCaptureDevice is the abstract base class for realizing video capture
 // device support in Chromium. It provides the interface for OS dependent
 // implementations.
 // The class is created and functions are invoked on a thread owned by
 // VideoCaptureManager. Capturing is done on other threads, depending on the OS
 // specific implementation.
@@ skipping 23 matching lines @@
 #include "media/mojo/interfaces/image_capture.mojom.h"
 #include "mojo/public/cpp/bindings/array.h"
 #include "ui/gfx/gpu_memory_buffer.h"
 
 namespace tracked_objects {
 class Location;
 }  // namespace tracked_objects
 
 namespace media {
 
-class CAPTURE_EXPORT VideoCaptureDevice {
+class CAPTURE_EXPORT ConsumerFeedbackObserver {

[mcasas, 2016/12/03 01:35:48]

As a general naming comment, "ConsumerFeedbackObserver"
is a bit tenuous in the media:: namespace; perhaps sth
like PipelineUtilizationObserver or UpstreamUsageObserver
or even madness: {Pipeline,Usptream}UllageObserver :-)
(VideoCapturePipelineUtilizationObserver
is just ridiculous, but very definitory).

Same for the method's name:
s/OnConsumerReportingUtilization/OnUtilizationReport/ ?
since this class doesn't need to know if it's the Consumer
or someone else reporting the pipeline fillup level.

[chfremer, 2016/12/05 18:17:00]

Excellent point.
miu@ suggested "feedback" instead of "utilization" anticipating that we may in
the future want to add other forms of feedback from frame consumers to
producers. We can keep the term "utilization" in the method name. I am proposing
to name the interface "VideoFrameConsumerFeedbackObserver" and the method
"OnUtilizationReport".

We now have a ton of classes with "VideoFrame..." prefix in the media namespace.
This seems to indicate that it would be useful to create a namespace like
media::video_capture.

+ public:
+  virtual ~ConsumerFeedbackObserver() {}
+
+  // During processing of a video frame, consumers may report back their
+  // utilization level to the source device. The device may use this information
+  // to adjust the rate of data it pushes out. Values are interpreted as
+  // follows:
+  // Less than 0.0 is meaningless and should be ignored.  1.0 indicates a
+  // maximum sustainable utilization.  Greater than 1.0 indicates the consumer
+  // is likely to stall or drop frames if the data volume is not reduced.
+  //
+  // Example: In a system that encodes and transmits video frames over the
+  // network, this value can be used to indicate whether sufficient CPU
+  // is available for encoding and/or sufficient bandwidth is available for
+  // transmission over the network.  The maximum of the two utilization
+  // measurements would be used as feedback.
+  //
+  // The parameter |frame_feedback_id| must match a |frame_feedback_id|
+  // previously sent out by the VideoCaptureDevice we are giving feedback about.
+  // It is used to indicate which particular frame the reported utilization
+  // corresponds to.
+  virtual void OnConsumerReportingUtilization(int frame_feedback_id,
+                                              double utilization) {}

[mcasas, 2016/12/03 01:35:48]

I'm dubious as to |frame_feedback_id|. Although
I see the point of frames being discarded out of 
(sent) order, what is a humble implementation to
do about it? Adding the parameter essentially 
forces the implementers to track a potentially 
unordered stream of utilization notifications, but
in all likelihood they shouldn't care too much
but just take the last such notification face value,
right?  

Or, perhaps the current final destination of the
utilization reports (i.e. the Oracles) are super 
sophisticated and indeed track individual reports 
when they don't really need to and can be 
simplified? (in another CL woud be ok ofc.).

[chfremer, 2016/12/05 18:17:00]

Good point. And I share the feeling.
I also agree that if we want to simplify, we should do it separately from this
CL, since this CL is trying to be a refactoring-only CL.

[mcasas, 2016/12/05 18:26:44]

Yup.

+
+  static constexpr double kNoUtilizationRecorded = -1.0;
+};
+
+class CAPTURE_EXPORT VideoCaptureDevice : public ConsumerFeedbackObserver {
  public:
 
   // Interface defining the methods that clients of VideoCapture must have. It
   // is actually two-in-one: clients may implement OnIncomingCapturedData() or
   // ReserveOutputBuffer() + OnIncomingCapturedVideoFrame(), or all of them.
   // All clients must implement OnError().
   class CAPTURE_EXPORT Client {
    public:
     // Memory buffer returned by Client::ReserveOutputBuffer().
     class CAPTURE_EXPORT Buffer {
      public:
       virtual ~Buffer() = 0;
       virtual int id() const = 0;
+      virtual int frame_feedback_id() const = 0;
       virtual gfx::Size dimensions() const = 0;
       virtual size_t mapped_size() const = 0;
       virtual void* data(int plane) = 0;
       void* data() { return data(0); }
 #if defined(OS_POSIX) && !(defined(OS_MACOSX) && !defined(OS_IOS))
       virtual base::FileDescriptor AsPlatformFile() = 0;
 #endif
       virtual bool IsBackedByVideoFrame() const = 0;
       virtual scoped_refptr<VideoFrame> GetVideoFrame() = 0;
     };
 
     virtual ~Client() {}
 
     // Captured a new video frame, data for which is pointed to by |data|.
     //
     // The format of the frame is described by |frame_format|, and is assumed to
     // be tightly packed. This method will try to reserve an output buffer and
     // copy from |data| into the output buffer. If no output buffer is
     // available, the frame will be silently dropped. |reference_time| is
     // system clock time when we detect the capture happens, it is used for
     // Audio/Video sync, not an exact presentation time for playout, because it
     // could contain noise. |timestamp| measures the ideal time span between the
     // first frame in the stream and the current frame; however, the time source
     // is determined by the platform's device driver and is often not the system
     // clock, or even has a drift with respect to system clock.
+    // |frame_feedback_id| is an identifier that allows clients to refer back to
+    // this particular frame when reporting consumer feedback via
+    // OnConsumerReportingUtilization(). This identifier is needed because
+    // frames are consumed asynchronously and multiple frames can be "in flight"
+    // at the same time.
     virtual void OnIncomingCapturedData(const uint8_t* data,
                                         int length,
                                         const VideoCaptureFormat& frame_format,
                                         int clockwise_rotation,
                                         base::TimeTicks reference_time,
-                                        base::TimeDelta timestamp) = 0;
+                                        base::TimeDelta timestamp,
+                                        int frame_feedback_id = 0) = 0;
 
     // Reserve an output buffer into which contents can be captured directly.
     // The returned Buffer will always be allocated with a memory size suitable
     // for holding a packed video frame with pixels of |format| format, of
     // |dimensions| frame dimensions. It is permissible for |dimensions| to be
     // zero; in which case the returned Buffer does not guarantee memory
     // backing, but functions as a reservation for external input for the
     // purposes of buffer throttling.
     //
     // The output buffer stays reserved and mapped for use until the Buffer
     // object is destroyed or returned.
     virtual std::unique_ptr<Buffer> ReserveOutputBuffer(
         const gfx::Size& dimensions,
         VideoPixelFormat format,
-        VideoPixelStorage storage) = 0;
+        VideoPixelStorage storage,
+        int frame_feedback_id) = 0;
 
     // Captured new video data, held in |frame| or |buffer|, respectively for
     // OnIncomingCapturedVideoFrame() and  OnIncomingCapturedBuffer().
     //
     // In both cases, as the frame is backed by a reservation returned by
     // ReserveOutputBuffer(), delivery is guaranteed and will require no
     // additional copies in the browser process.
     // See OnIncomingCapturedData for details of |reference_time| and
     // |timestamp|.
     // TODO(chfremer): Consider removing one of the two in order to simplify the
     // interface.
     virtual void OnIncomingCapturedBuffer(
         std::unique_ptr<Buffer> buffer,
         const VideoCaptureFormat& frame_format,
         base::TimeTicks reference_time,
         base::TimeDelta timestamp) = 0;
     virtual void OnIncomingCapturedVideoFrame(
         std::unique_ptr<Buffer> buffer,
         scoped_refptr<VideoFrame> frame) = 0;
 
     // Attempts to reserve the same Buffer provided in the last call to one of
     // the OnIncomingCapturedXXX() methods. This will fail if the content of the
     // Buffer has not been preserved, or if the |dimensions|, |format|, or
     // |storage| disagree with how it was reserved via ReserveOutputBuffer().
     // When this operation fails, nullptr will be returned.
     virtual std::unique_ptr<Buffer> ResurrectLastOutputBuffer(
         const gfx::Size& dimensions,
         VideoPixelFormat format,
-        VideoPixelStorage storage) = 0;
+        VideoPixelStorage storage,
+        int new_frame_feedback_id) = 0;
 
     // An error has occurred that cannot be handled and VideoCaptureDevice must
     // be StopAndDeAllocate()-ed. |reason| is a text description of the error.
     virtual void OnError(const tracked_objects::Location& from_here,
                          const std::string& reason) = 0;
 
     // VideoCaptureDevice requests the |message| to be logged.
     virtual void OnLog(const std::string& message) {}
 
     // Returns the current buffer pool utilization, in the range 0.0 (no buffers
     // are in use by producers or consumers) to 1.0 (all buffers are in use).
     virtual double GetBufferPoolUtilization() const = 0;
   };
 
-  virtual ~VideoCaptureDevice();
+  ~VideoCaptureDevice() override;
 
   // Prepares the video capturer for use. StopAndDeAllocate() must be called
   // before the object is deleted.
   virtual void AllocateAndStart(const VideoCaptureParams& params,
                                 std::unique_ptr<Client> client) = 0;
 
   // In cases where the video capturer self-pauses (e.g., a screen capturer
   // where the screen's content has not changed in a while), consumers may call
   // this to request a "refresh frame" be delivered to the Client.  This is used
   // in a number of circumstances, such as:
@@ skipping 72 matching lines @@
 
  private:
   // Gets the power line frequency from the current system time zone if this is
   // defined, otherwise returns 0.
   PowerLineFrequency GetPowerLineFrequencyForLocation() const;
 };
 
 }  // namespace media
 
 #endif  // MEDIA_CAPTURE_VIDEO_VIDEO_CAPTURE_DEVICE_H_