VideoToolbox encoder for cast senders.

media/cast/sender/video_sender.cc

 // Copyright 2014 The Chromium Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file.
 
 #include "media/cast/sender/video_sender.h"
 
 #include <algorithm>
 #include <cstring>
 
 #include "base/bind.h"
 #include "base/debug/trace_event.h"
 #include "base/logging.h"
 #include "base/message_loop/message_loop.h"
 #include "media/cast/cast_defines.h"
 #include "media/cast/net/cast_transport_config.h"
 #include "media/cast/sender/external_video_encoder.h"
 #include "media/cast/sender/video_encoder_impl.h"
 
+#if defined(OS_MACOSX)
+#include "media/cast/sender/h264_vt_encoder.h"
+#endif
+
 namespace media {
 namespace cast {
 
 // The following two constants are used to adjust the target
 // playout delay (when allowed). They were calculated using
 // a combination of cast_benchmark runs and manual testing.
 
 // This is how many round trips we think we need on the network.
 const int kRoundTripsNeeded = 4;
 // This is an estimate of all the the constant time needed
@@ skipping 28 matching lines @@
       last_bitrate_(0),
       playout_delay_change_cb_(playout_delay_change_cb),
       weak_factory_(this) {
   cast_initialization_status_ = STATUS_VIDEO_UNINITIALIZED;
   VLOG(1) << "max_unacked_frames is " << max_unacked_frames_
           << " for target_playout_delay="
           << target_playout_delay_.InMilliseconds() << " ms"
           << " and max_frame_rate=" << video_config.max_frame_rate;
   DCHECK_GT(max_unacked_frames_, 0);
 
+#if defined(OS_MACOSX)
+  // On Apple platforms, use the hardware H.264 encoder if possible. It is the
+  // only reasonable option for iOS.
+  if (!video_config.use_external_encoder &&

[miu, 2014/09/24 00:10:14]

|use_external_encoder| is set to true when the hardware encoder should be used,
so did you mean to omit the bang?

Also, will video toolbox provide a software encoder on Mac OS, if the hardware
support does not exist?  Perhaps the value of |use_external_encoder| should be
ignored here, and handled inside H264VideoToolboxEncoder::Initialize()?

[jfroy, 2014/09/24 01:00:58]

I understood |use_external_encoder| from the point of view of cast, which is to
say that it should use the client-provided callbacks to allocate an external
encoder managed by an instance of |ExternalVideoEncoder|. This is in fact what
my first implementation used.

However, |VideoEncodeAccelerator| (which is used by |ExternalVideoEncoder|) is a
very poor interface for VideoToolbox (it forces unnecessary buffer allocations
into unnecessarily large shared memory buffers). The basic |VideoEncoder|
interface was much better, and this is what |H264VideoToolboxEncoder|
implements. So from this point of view, this check is correct.

VideoToolbox is opaque from the POV of the client. On OS X, it will indeed
provide software encoding if there is no hardware encoder or if the hardware
encoder is busy. On iOS, it only provides access to a hardware encoder and will
fail with a "resource busy" error if someone else is already using it. This
could be changed in the future by Apple on specific hardware.

So in summary, |use_external_encoder| implies using the |ExternalVideoEncoder|
class and using the client-provided vea and memory callbacks, which this
implementation does not use.

+      video_config.codec == CODEC_VIDEO_H264) {
+    video_encoder_.reset(new H264VideoToolboxEncoder(
+        cast_environment,
+        video_config,
+        base::Bind(&VideoSender::OnEncoderInitialized,
+                   weak_factory_.GetWeakPtr(),
+                   initialization_cb)));
+  }
+#endif  // defined(OS_MACOSX)
+#if !defined(OS_IOS)
   if (video_config.use_external_encoder) {
     video_encoder_.reset(new ExternalVideoEncoder(
         cast_environment,
         video_config,
         base::Bind(&VideoSender::OnEncoderInitialized,
                    weak_factory_.GetWeakPtr(), initialization_cb),
         create_vea_cb,
         create_video_encode_mem_cb));
   } else {
     // Software encoder is initialized immediately.
     congestion_control_.reset(
         NewAdaptiveCongestionControl(cast_environment->Clock(),
                                      video_config.max_bitrate,
                                      video_config.min_bitrate,
                                      max_unacked_frames_));
     video_encoder_.reset(new VideoEncoderImpl(

[miu, 2014/09/24 00:10:14]

On desktop Mac OS, |video_encoder_| will be recreated here (after being created
on line 76 above).  Perhaps lines 85-103 should be guarded with:

  if (!video_encoder_) {
    ...
  }

[jfroy, 2014/09/24 01:00:58]

Oh man that is embarrassing. Thanks for catching that.

[jfroy, 2014/09/24 18:43:53]

I also wanted to mention why I did not integrate the VideoToolbox encoder in
|VideoEncoderImpl|. The reason is that the |SoftwareVideoEncoder| interface used
by |VideoEncoderImpl| imposes synchronous frame encoding and a dedicated video
encoding thread. VideoToolbox already spawns its own threads and operates
asynchronously.

Furthermore, the encoder may not emit encoded frames until a few input frames
have been provided, putting the encoder at risk of deadlocking: the
implementation would have to block the video thread until the encoded frame
corresponding to a submitted input frame is emitted, thus blocking submission of
further input frames, thus potentially blocking encoded frames from being
emitted.

I also wanted to avoid the cost of an unnecessary thread and the latency of
posting from the main thread to the video thread.

         cast_environment, video_config, max_unacked_frames_));
     cast_initialization_status_ = STATUS_VIDEO_INITIALIZED;
   }
+#endif  // !defined(OS_IOS)
 
   if (cast_initialization_status_ == STATUS_VIDEO_INITIALIZED) {
     cast_environment->PostTask(
         CastEnvironment::MAIN,
         FROM_HERE,
         base::Bind(initialization_cb, cast_initialization_status_));
   }
 
   media::cast::CastTransportRtpConfig transport_config;
   transport_config.ssrc = video_config.ssrc;
@@ skipping 98 matching lines @@
   DCHECK(cast_environment_->CurrentlyOn(CastEnvironment::MAIN));
 
   frames_in_encoder_--;
   DCHECK_GE(frames_in_encoder_, 0);
 
   SendEncodedFrame(encoder_bitrate, encoded_frame.Pass());
 }
 
 }  // namespace cast
 }  // namespace media