[PPAPI] Pepper MediaStream API audio track implementation and example.

content/renderer/pepper/pepper_media_stream_audio_track_host.cc

Index: content/renderer/pepper/pepper_media_stream_audio_track_host.cc
diff --git a/content/renderer/pepper/pepper_media_stream_audio_track_host.cc b/content/renderer/pepper/pepper_media_stream_audio_track_host.cc
new file mode 100644
index 0000000000000000000000000000000000000000..fecd255f4502a8bd919b7da82b4e38bdbb4d185e
--- /dev/null
+++ b/content/renderer/pepper/pepper_media_stream_audio_track_host.cc
@@ -0,0 +1,161 @@
+// Copyright (c) 2014 The Chromium Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style license that can be
+// found in the LICENSE file.
+
+#include "content/renderer/pepper/pepper_media_stream_audio_track_host.h"
+
+#include "base/bind.h"
+#include "base/location.h"
+#include "base/logging.h"
+#include "base/macros.h"
+#include "base/message_loop/message_loop_proxy.h"
+#include "ppapi/c/pp_errors.h"
+#include "ppapi/c/ppb_audio_frame.h"
+#include "ppapi/shared_impl/media_stream_frame.h"
+
+using media::AudioParameters;
+
+namespace {
+
+// TODO(penghuang): make this configurable.
+const int32_t kNumberOfFrames = 4;
+
+}  // namespace
+
+namespace content {
+
+PepperMediaStreamAudioTrackHost::AudioSink::AudioSink(
+    PepperMediaStreamAudioTrackHost* host)
+    : host_(host),
+      frame_data_size_(0),
+      main_message_loop_proxy_(base::MessageLoopProxy::current()) {
+}
+
+PepperMediaStreamAudioTrackHost::AudioSink::~AudioSink() {
+  DCHECK_EQ(main_message_loop_proxy_, base::MessageLoopProxy::current());
+}
+
+void PepperMediaStreamAudioTrackHost::AudioSink::EnqueueFrame(int32_t index) {

[no longer working on chromium, 2014/02/01 21:28:09]

The thread model is very complicated in this class, please add thread check to
all methods.
For the audio thread, you can capture_thread_checker_.DetachFromThread() on
PepperMediaStreamAudioTrackHost::AudioSink::OnSetFormat, and
DCHECK(capture_thread_checker_.CalledOnValidThread()) on
PepperMediaStreamAudioTrackHost::AudioSink::OnData

[Peng, 2014/02/02 02:11:26]

Done.

+  DCHECK_GE(index, 0);
+  DCHECK_LT(index, host_->frame_buffer()->number_of_frames());
+  base::AutoLock lock(mutex_);
+  frames_.push_back(index);
+}
+
+void PepperMediaStreamAudioTrackHost::AudioSink::InitFramesOnMainThread(
+    uint32_t number_of_frames, uint32_t frame_size) {

[bbudge, 2014/01/31 22:13:01]

DCHECK_EQ(main_message_loop_proxy_, base::MessageLoopProxy::current());
?

[no longer working on chromium, 2014/02/01 21:28:09]

The number_of_frames here and  number_of_frames in AudioSink::OnData mean
different things. Can you change the name to remove the confusion, for example,
number_of_buffers,  buffer_size

Also do you really need uint32_t? how about size_t? 

[Peng, 2014/02/02 02:11:26]

Done.

[Peng, 2014/02/02 02:11:26]

Because the base class is used for both Audio and Video Track. So we decided use
name AudioFrame and VideoFrame. So everywhere in audio and video tracks, we use
frame. But after reading MediaStreamAudioSink. I understand buffer is better
name than frame here. So I think it is better to rename InitFrames of the base
class to InitBuffers, and AudioFrame to AudioBuffer, but keep VideoFrame, etc.
But I would like to do it in a separate CL, because it need change base classes
as well. WDYT?
I changed it to int32_t, because this function call base class's InitFrames()
which takes int32_t.

[no longer working on chromium, 2014/02/03 09:50:44]

sgtm.

[no longer working on chromium, 2014/02/03 09:50:44]

Curiously, does the overall pepper code prefer int32_t over int?

[Peng, 2014/02/03 14:38:54]

I think so. I did not find size_t in ppapi/api.

+  bool result = host_->InitFrames(number_of_frames, frame_size);
+  DCHECK(result);
+  base::AutoLock lock(mutex_);
+  for (uint32_t i = 0; i < number_of_frames; ++i) {
+    int32_t index = host_->frame_buffer()->DequeueFrame();
+    DCHECK_GE(index, 0);
+    frames_.push_back(index);
+  }
+}
+
+void PepperMediaStreamAudioTrackHost::AudioSink::OnData(const int16* audio_data,
+                                                        int sample_rate,
+                                                        int number_of_channels,
+                                                        int number_of_frames) {
+  DCHECK(audio_data);
+  DCHECK_EQ(sample_rate, audio_params_.sample_rate());
+  DCHECK_EQ(number_of_channels, audio_params_.channels());
+  DCHECK_EQ(number_of_frames, audio_params_.frames_per_buffer());
+  int32_t index = -1;
+  {
+    base::AutoLock lock(mutex_);
+    if (!frames_.empty()) {
+      index = frames_.front();
+      frames_.pop_front();
+    }
+  }
+
+  if (index != -1) {
+    // TODO(penghuang): support re-sampling, etc.
+    ppapi::MediaStreamFrame::Audio* frame =

[no longer working on chromium, 2014/02/01 21:28:09]

in OnSetFormat() you post a task InitFramesOnMainThread on the main render
thread and call InitFrames there, but before the task is executed, Ondata with
the new format might have come. Is it a problem if host_->frame_buffer() is
still using the old format?

[Peng, 2014/02/02 02:11:26]

I remember you told me the OnSetFormat will only be called once. So before
InitFrameOnMianThead() being finished, the index will be always -1, So we will
simply drop all data until underlying buffer is ready.
It should not have any problem, right?

[no longer working on chromium, 2014/02/03 09:50:44]

No, today this can be called more than once. Everytime the format of the audio
data has changed, it will trigger OnSetFormat. But OnSetFormat won't be called
frequently.

Probably in the future we can make sure OnSetFormat can be called once, but not
now.

[Peng, 2014/02/03 14:38:54]

That is a bad news. Could you give me some real cases for the change?

Right now, the base class does not support re-allocating underlying shared
memory. I am think could we allocate the buffer with bigger frame size which is
big enough for all kinds of formats. So we can avoid re-allocating shared
memory.

[no longer working on chromium, 2014/02/03 20:23:59]

Each time if a WebRtcAudioCapturer::SetCapturerSource is called, it will trigger
the SetFormat.
https://code.google.com/p/chromium/codesearch#chromium/src/content/renderer/m...

You can work around the problem if you can allocate a buffer of 10ms data, that
is the the maximum buffer size of a packet that the sink will get.
I would suggest you add a CHECK here to make sure it is the case.

In the future, I am going to refactor the code to make sure SetFormat will be
called on once.

+        &(host_->frame_buffer()->GetFramePointer(index)->audio);
+    frame->header.size = host_->frame_buffer()->frame_size();
+    frame->header.type = ppapi::MediaStreamFrame::TYPE_AUDIO;
+    frame->timestamp = timestamp_.InMillisecondsF();
+    frame->sample_rate = static_cast<PP_AudioFrame_SampleRate>(sample_rate);
+    frame->number_of_channels = number_of_channels;
+    frame->number_of_samples = number_of_channels * number_of_frames;
+    frame->data_size = frame_data_size_;
+    memcpy(frame->data, audio_data, frame_data_size_);
+
+    // This function is called from the audio thread, but
+    // |SendEnqueueFrameMessageToPlugin()| doesn't use any sync IPC,
+    // so it is safe to call it from the audio thread directly.
+    host_->SendEnqueueFrameMessageToPlugin(index);
+  }
+  timestamp_ += frame_duration_;

[no longer working on chromium, 2014/02/01 21:28:09]

nit, you don't need this frame_duration_ member, frame_duration_ equals to
number_of_frames in the callback.

[Peng, 2014/02/02 02:11:26]

I think we need calculate duration from the number_of_frames base on the sample
rate.
I think the number_of_frames is always a same value. So probably using an extra
can has slightly better performance. ?

[no longer working on chromium, 2014/02/03 09:50:44]

OK, it is fine to use the member then.

+}
+
+void PepperMediaStreamAudioTrackHost::AudioSink::OnSetFormat(
+    const AudioParameters& params) {
+  DCHECK(!audio_params_.IsValid());
+  DCHECK(params.IsValid());
+  DCHECK_EQ(params.bits_per_sample(), 16);
+  DCHECK((params.sample_rate() == AudioParameters::kTelephoneSampleRate) ||
+         (params.sample_rate() == AudioParameters::kAudioCDSampleRate));
+
+  COMPILE_ASSERT(AudioParameters::kTelephoneSampleRate == 8000,
+                 audio_sample_rate_does_not_match);
+  COMPILE_ASSERT(AudioParameters::kAudioCDSampleRate == 44100,
+                 audio_sample_rate_does_not_match);
+
+  // TODO(penghuang): support setting format more than once.
+  audio_params_ = params;
+  frame_duration_ = audio_params_.GetBufferDuration();
+  frame_data_size_ = audio_params_.GetBytesPerBuffer();
+
+  // The size is slightly bigger than necessary, because 8 extra bytes are added
+  // into the struct. Also see |MediaStreamFrame|.
+  int32_t size = sizeof(ppapi::MediaStreamFrame::Audio) + frame_data_size_;

[bbudge, 2014/01/31 22:13:01]

since sizeof is a size_t, this may not compile on all platforms.

[Peng, 2014/02/02 02:11:26]

Done.

+
+  // This function is called from the audio thread and |InitFrames()| uses some
+  // sync IPC. So we have to call |InitFrames()| from the main thread.

[no longer working on chromium, 2014/02/01 21:28:09]

I think you need some synchronization between OnSetFormat and
InitFramesOnMainThread. Probably you should drop the audio buffer until the
buffer in the main thread is setup properly.
For example, empty the frames_ here, and set it up in InitFramesOnMainThread

[Peng, 2014/02/02 02:11:26]

I assume this function will be only called once. And InitFrameOnMianThread()
uses a lock
to protect the frame queue |frames_|. and the |frames_| will be always empty
before InitFrameOnMainThread is finished. So it should be OK.

+  main_message_loop_proxy_->PostTask(
+      FROM_HERE,
+      base::Bind(&AudioSink::InitFramesOnMainThread,
+                 AsWeakPtr(), kNumberOfFrames, size));
+}
+
+PepperMediaStreamAudioTrackHost::PepperMediaStreamAudioTrackHost(
+    RendererPpapiHost* host,
+    PP_Instance instance,
+    PP_Resource resource,
+    const blink::WebMediaStreamTrack& track)
+    : PepperMediaStreamTrackHostBase(host, instance, resource),
+      track_(track),
+      connected_(false),
+      audio_sink_(this) {
+  DCHECK(!track_.isNull());
+}
+
+PepperMediaStreamAudioTrackHost::~PepperMediaStreamAudioTrackHost() {
+  OnClose();
+}
+
+void PepperMediaStreamAudioTrackHost::OnClose() {
+  if (connected_) {
+    MediaStreamAudioSink::RemoveFromAudioTrack(&audio_sink_, track_);
+    connected_ = false;
+  }
+}
+
+void PepperMediaStreamAudioTrackHost::OnNewFrameEnqueued() {
+  int32_t index = frame_buffer()->DequeueFrame();
+  DCHECK_GE(index, 0);
+  audio_sink_.EnqueueFrame(index);
+}
+
+void PepperMediaStreamAudioTrackHost::DidConnectPendingHostToResource() {
+  if (!connected_) {
+    MediaStreamAudioSink::AddToAudioTrack(&audio_sink_, track_);
+    connected_ = true;
+  }
+}
+
+}  // namespace content