MediaCaptureFromElement: add support for audio captureStream().

content/renderer/renderer_blink_platform_impl.cc

 // Copyright (c) 2012 The Chromium Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file.
 
 #include "content/renderer/renderer_blink_platform_impl.h"
 
 #include <utility>
 
 #include "base/command_line.h"
 #include "base/files/file_path.h"
@@ skipping 43 matching lines @@
 #include "content/renderer/device_sensors/device_light_event_pump.h"
 #include "content/renderer/device_sensors/device_motion_event_pump.h"
 #include "content/renderer/device_sensors/device_orientation_absolute_event_pump.h"
 #include "content/renderer/device_sensors/device_orientation_event_pump.h"
 #include "content/renderer/dom_storage/local_storage_cached_areas.h"
 #include "content/renderer/dom_storage/local_storage_namespace.h"
 #include "content/renderer/dom_storage/webstoragenamespace_impl.h"
 #include "content/renderer/gamepad_shared_memory_reader.h"
 #include "content/renderer/media/audio_decoder.h"
 #include "content/renderer/media/canvas_capture_handler.h"
+#include "content/renderer/media/html_audio_element_capturer_source.h"
 #include "content/renderer/media/html_video_element_capturer_source.h"
 #include "content/renderer/media/image_capture_frame_grabber.h"
 #include "content/renderer/media/media_recorder_handler.h"
 #include "content/renderer/media/renderer_webaudiodevice_impl.h"
 #include "content/renderer/media/renderer_webmidiaccessor_impl.h"
 #include "content/renderer/media/rtc_certificate_generator.h"
 #include "content/renderer/mojo/blink_service_registry_impl.h"
 #include "content/renderer/render_thread_impl.h"
 #include "content/renderer/renderer_clipboard_delegate.h"
 #include "content/renderer/screen_orientation/screen_orientation_observer.h"
@@ skipping 894 matching lines @@
   AddVideoTrackToMediaStream(
       HtmlVideoElementCapturerSource::CreateFromWebMediaPlayerImpl(
           web_media_player,
           content::RenderThread::Get()->GetIOMessageLoopProxy()),
       false,  // is_remote
       false,  // is_readonly
       web_media_stream);
 #endif
 }
 
+void RendererBlinkPlatformImpl::createHTMLAudioElementCapturer(
+    WebMediaStream* web_media_stream,
+    WebMediaPlayer* web_media_player) {
+#if defined(ENABLE_WEBRTC)

[miu, 2016/05/13 23:40:35]

Consider omitting the "#if defined(ENABLE_WEBRTC)" since this should all compile
and run w/o webrtc dependencies.

[mcasas, 2016/05/14 02:23:47]

It should be like that indeed, but by
extension (and history), all MediaStream 
and GetUserMedia APIs and code are considered
behind this WebRtc-grouping. 

IOW, if a (possibly, downstream) build has no 
WebRtc, it would have no MediaStreams, hence 
no use for this API.

[miu, 2016/05/17 21:00:41]

This may have been true in the past, but not anymore. For example, in
content/content_renderer.gypi, there are media_stream_*.* files listed in both
'public_renderer_sources' and 'private_renderer_sources'; both of which are
built even when ENABLE_WEBRTC is false.

Historically, the MediaStream API was designed and built at a time when WebRTC
was the only game in town. That's why there's all this code that mixes-in WebRTC
dependencies and conditionally compiles around ENABLE_WEBRTC. Nowadays, there
are multiple other non-WebRTC product features that rely on it (e.g., Cast
Streaming, WebAudio, etc.). So, it's a general TODO/FixIt issue in the
content/renderer/media code to start enabling more of the MediaStream code (the
modules without WebRTC dependencies), moving their compile/link outside of the
protected "ENABLE_WEBRTC" realm so that the code can be re-used by other product
features.

OOC, I tried building your patch locally with GN arg 'enable_webrtc=false'.
Everything compiled, except for one issue: Linking failed on the call to
AddAudioTrackToMediaStream() (line 991 in this change). It seems that's only
because media_stream_utils.cc is not in 'public_renderer_sources', which is an
oversight of mine. It should be moved since my recent "big 8000 line change"
removed the last of the WebRTC dependencies from the media_stream_utils
functions.

However, this is all a moot point: Take a look at my comment below, and you
*definitely* won't need to conditionally compile this code via "#ifdef
ENABLE_WEBRTC" anymore.

Generally, I would suggest that if code can compile and link successfully when
ENABLE_WEBRTC is false, you shouldn't add this guard. It's self-documenting code
for a problem that doesn't actually exist, which can mislead future maintainers.

[mcasas, 2016/05/18 00:00:10]

Acknowledged.

+  DCHECK(web_media_stream);
+  DCHECK(web_media_player);
+  // These are the MediaStream parameters, hardcoded and pervasive
+  // TODO(mcasas): fetch them from some place and/or ignore these.
+  const int kInputNumChannels = 1;

[miu, 2016/05/13 23:40:35]

Actually, this is the *output* format (from ExternalMediaStreamAudioSource). It
will call HtmlAudioElementCapturerSource::Initialize() with these params. So,
you can make this whatever format you would want the MediaStreamTrack to
provide.

Also, note that you could eliminate the need for any audio conversion if you
just used the audio format of the HTMLMediaElement. It feels like it should be
easy to pass that information to this method.

[mcasas, 2016/05/14 02:23:47]

Yeah, |output| from the Source but |input| from
the Track POV, and since we're adding a Track in
l.991, I thought this naming was more representative.
Turns out it's actually quite hard since is buried in 
the pipeline decoding the actual data, and depending
on the video, the decoding library, and the network,
it can take a while before we actually could get these
parameters, even up to the time of the first frame
arrival...

[miu, 2016/05/17 21:00:41]

Oh! I don't know why I didn't notice this before: You have
HtmlAudioElementCapturerSource extending media::AudioCapturerSource. That's
requiring you to fix the output audio format ahead of time and have to convert
whatever happens to come in from the WebMediaPlayerImpl.

Instead, what you want to do is have HtmlAudioElementCapturerSource extend
MediaStreamAudioSource (i.e., don't use ExternalMediaStreamAudioSource, but
write your own). The format of the audio can then change at any time. Also, the
whole class becomes much simpler:

class CONTENT_EXPORT HtmlAudioElementSource final
    : NON_EXPORTED_BASE(public MediaStreamAudioSource) {
 public:
  ...ctor...

  ~HtmlAudioElementSource() final {
    EnsureSourceIsStopped();
  }

  bool EnsureSourceIsStarted() final {
    if (audio_source_ && !is_started_) {
      audio_source_->SetCopyAudioCallback(
          base::Bind(OnAudioBus, this));
      is_started_ = true;
    }
    return is_started_;
  }

  void EnsureSourceIsStopped() final {
    if (is_started_) {
      if (audio_source_) {
        audio_source_->ClearCopyAudioCallback();
        audio_source_.reset();
      }
      is_started_ = false;
    }
  }

  void OnAudioBus(std::unique_ptr<media::AudioBus> audio_bus,
                  uint32_t delay_milliseconds,
                  int sample_rate) {
    const base::TimeTicks capture_time = base::TimeTicks::Now() -
        base::TimeDelta::FromMilliseconds(delay_milliseconds);

    if (sample_rate != last_sample_rate_ ||
        audio_bus->channels() != last_num_channels_ ||
        audio_bus->frames() != last_bus_frames_) {
      MediaStreamAudioSource::SetFormat(media::AudioParameters(
          media::AudioParameters::AUDIO_PCM_LOW_LATENCY,
          media::GuessChannelLayout(audio_bus->channels()),
          sample_rate, 16, audio_bus->frames()));
      last_sample_rate_ = sample_rate;
      last_num_channels_ = audio_bus->channels();
      last_bus_frames_ = audio_bus->frames();
    }

    MediaStreamAudioSource::DeliverDataToTracks(*audio_bus_, capture_time);
  }

  base::WeakPtr<media::WebAudioSourceProviderImpl> audio_source_;
  bool is_started_ = false;

  base::ThreadChecker thread_checker_;

  DISALLOW_COPY_AND_ASSIGN(HtmlAudioElementSource);
};

[mcasas, 2016/05/18 00:00:10]

Brilliant! SO -oh- Done!

+  const int kInputSamplingRate = 48000;
+  const int kBufferDurationMs = 10;
+
+  AddAudioTrackToMediaStream(

[miu, 2016/05/13 23:40:35]

Nice! :)

[mcasas, 2016/05/14 02:23:47]

Acknowledged.

+      HtmlAudioElementCapturerSource::CreateFromWebMediaPlayerImpl(
+          web_media_player),
+      kInputSamplingRate, media::GuessChannelLayout(kInputNumChannels),
+      kBufferDurationMs * kInputSamplingRate /
+          base::Time::kMillisecondsPerSecond,
+      false /* is_remote */, false /* is_readonly */, web_media_stream);
+#endif
+}
+
 //------------------------------------------------------------------------------
 
 WebImageCaptureFrameGrabber*
 RendererBlinkPlatformImpl::createImageCaptureFrameGrabber() {
 #if defined(ENABLE_WEBRTC)
   return new ImageCaptureFrameGrabber();
 #else
   return nullptr;
 #endif  // defined(ENABLE_WEBRTC)
 }
@@ skipping 342 matching lines @@
   return &trial_token_validator_;
 }
 
 void RendererBlinkPlatformImpl::workerContextCreated(
     const v8::Local<v8::Context>& worker) {
   GetContentClient()->renderer()->DidInitializeWorkerContextOnWorkerThread(
       worker);
 }
 
 }  // namespace content