Adds volume level measurements to the AudioInputController for low-latency clients

media/audio/audio_input_controller.cc

 // Copyright (c) 2012 The Chromium Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file.
 
 #include "media/audio/audio_input_controller.h"
 
 #include "base/bind.h"
 #include "base/threading/thread_restrictions.h"
+#include "base/time/time.h"
 #include "media/base/limits.h"
 #include "media/base/scoped_histogram_timer.h"
 #include "media/base/user_input_monitor.h"
 
+using base::TimeDelta;
+
 namespace {
 const int kMaxInputChannels = 3;
 
 // TODO(henrika): remove usage of timers and add support for proper
 // notification of when the input device is removed.  This was originally added
 // to resolve http://crbug.com/79936 for Windows platforms.  This then caused
 // breakage (very hard to repro bugs!) on other platforms: See
 // http://crbug.com/226327 and http://crbug.com/230972.
 // See also that the timer has been disabled on Mac now due to
 // crbug.com/357501.
 const int kTimerResetIntervalSeconds = 1;
 // We have received reports that the timer can be too trigger happy on some
 // Mac devices and the initial timer interval has therefore been increased
 // from 1 second to 5 seconds.
 const int kTimerInitialIntervalSeconds = 5;
+
+// Time constant for AudioPowerMonitor.  See AudioPowerMonitor ctor comments for
+// semantics.  This value was arbitrarily chosen, but seems to work well.

[tommi (sloooow) - chröme, 2014/05/17 13:45:00]

maybe you can elaborate on how you tested this... seems to work well for...?
(would also be good to mention the platform since this might work fine on
desktop but not as well on mobile?)

[henrika (OOO until Aug 14), 2014/05/19 11:58:51]

It is a time constant taken from the initial implementation of the
AudioPowerMonitor and it gives a nice balance between how fast changes are
detected and how much the output fluctuates. There is no difference in this
behavior between desktop and mobile. It is the same value as is used for the
power measurements on the render side.

+const int kPowerMeasurementTimeConstantMillis = 10;

[tommi (sloooow) - chröme, 2014/05/17 13:45:00]

nit: Milliseconds or Ms

[henrika (OOO until Aug 14), 2014/05/19 11:58:51]

I had done modifications before this review was finalized. This is changed on
the new version.

+
+// Time between two successive measurements of audio power levels.
+const int kPowerMonitorLogIntervalMilliSeconds = 500;

[tommi (sloooow) - chröme, 2014/05/17 13:45:00]

nit: Milliseconds

[henrika (OOO until Aug 14), 2014/05/19 11:58:51]

Done.

 }
 
 namespace media {
 
 // static
 AudioInputController::Factory* AudioInputController::factory_ = NULL;
 
 AudioInputController::AudioInputController(EventHandler* handler,
                                            SyncWriter* sync_writer,
                                            UserInputMonitor* user_input_monitor)
@@ skipping 128 matching lines @@
 void AudioInputController::SetAutomaticGainControl(bool enabled) {
   task_runner_->PostTask(FROM_HERE, base::Bind(
       &AudioInputController::DoSetAutomaticGainControl, this, enabled));
 }
 
 void AudioInputController::DoCreate(AudioManager* audio_manager,
                                     const AudioParameters& params,
                                     const std::string& device_id) {
   DCHECK(task_runner_->BelongsToCurrentThread());
   SCOPED_UMA_HISTOGRAM_TIMER("Media.AudioInputController.CreateTime");
+
+  // Create the audio (power) level meter given the provided audio parameters.
+  audio_level_.reset(new media::AudioPowerMonitor(params.sample_rate(),
+      TimeDelta::FromMilliseconds(kPowerMeasurementTimeConstantMillis)));
+  audio_bus_ = AudioBus::Create(params);

[tommi (sloooow) - chröme, 2014/05/17 13:45:00]

how cpu intensive is it to add this?  This seems like a lot when all we want to
do is to detect signal absense - right?

[henrika (OOO until Aug 14), 2014/05/19 11:58:51]

The plan was to perform measurements at a very low rate, like once every other
second or so. Can't see that is could be an issue. On the output side, the same
is done for each audio frame (not on mobile).

Yes, one could come up with another type of detector which looks for signal
absence but given that we might pick up background noise etc. it feels more
efficient to use this type of level meter IMO. Then we can also detect states
like: "Look, there is a clear input signal (non-zero) but it does not reach the
other side, or the renderer does not get it".

+  audio_params_ = params;
+
   // TODO(miu): See TODO at top of file.  Until that's resolved, assume all
   // platform audio input requires the |no_data_timer_| be used to auto-detect
   // errors.  In reality, probably only Windows needs to be treated as
   // unreliable here.
   DoCreateForStream(audio_manager->MakeAudioInputStream(params, device_id),
                     true);
 }
 
 void AudioInputController::DoCreateForStream(
     AudioInputStream* stream_to_control, bool enable_nodata_timer) {
@@ skipping 73 matching lines @@
 void AudioInputController::DoClose() {
   DCHECK(task_runner_->BelongsToCurrentThread());
   SCOPED_UMA_HISTOGRAM_TIMER("Media.AudioInputController.CloseTime");
 
   if (state_ == CLOSED)
     return;
 
   // Delete the timer on the same thread that created it.
   no_data_timer_.reset();
 
+  if (audio_level_)
+    audio_level_->Reset();
+
   DoStopCloseAndClearStream();
   SetDataIsActive(false);
 
   if (SharedMemoryAndSyncSocketMode())
     sync_writer_->Close();
 
   if (user_input_monitor_)
     user_input_monitor_->DisableKeyPressMonitoring();
 
   state_ = CLOSED;
@@ skipping 91 matching lines @@
     key_pressed = current_count != prev_key_down_count_;
     prev_key_down_count_ = current_count;
     DVLOG_IF(6, key_pressed) << "Detected keypress.";
   }
 
   // Use SharedMemory and SyncSocket if the client has created a SyncWriter.
   // Used by all low-latency clients except WebSpeech.
   if (SharedMemoryAndSyncSocketMode()) {
     sync_writer_->Write(data, size, volume, key_pressed);
     sync_writer_->UpdateRecordedBytes(hardware_delay_bytes);
+
+    if (!audio_level_)
+      return;
+
+    // Perform periodic audio (power) level measurements.
+    if ((base::TimeTicks::Now() - last_audio_level_log_time_).InMilliseconds() >
+        kPowerMonitorLogIntervalMilliSeconds) {
+      last_audio_level_log_time_ = base::TimeTicks::Now();
+      audio_bus_->FromInterleaved(

[tommi (sloooow) - chröme, 2014/05/17 13:45:00]

safe to access these variables on a different thread without a lock? Maybe add
comments for how this is guaranteed to be safe.

[henrika (OOO until Aug 14), 2014/05/19 11:58:51]

I have rewritten this part in the latest version. Let me check again and come
back.

+          data, audio_bus_->frames(), audio_params_.bits_per_sample() / 8);
+      audio_level_->Scan(*audio_bus_, audio_bus_->frames());
+      std::pair<float, bool> result = audio_level_->ReadCurrentPowerAndClip();
+
+      // Get the audible level in the range [0.0,1.0], where 0.0 means
+      // the audio signal is silent and 1.0 means it is at maximum volume.
+      float level_dbfs = result.first;
+      float level = 0;
+      static const float kSilenceThresholdDBFS = -72.24719896f;

[tommi (sloooow) - chröme, 2014/05/17 13:45:00]

how does this translate to a data packet of all zeros vs a data packet that has
one non-zero frame with a very low value?
For how manu packets is the value calculated?

[henrika (OOO until Aug 14), 2014/05/19 11:58:51]

Assume that you by packet mean all audio frames contained on one Ondata
callback.

We will not detect one single audio frame (4 bytes) being zero since the power
meter uses an exponential MA filter
(http://en.wikipedia.org/wiki/Exponential_smoothing). You need a longer sequence
of zeros to reset the filter. Takes a few milliseconds. If the signal drops from
max to zero, it should take about 10ms for the measurement to reflect it. But
again, we must see a sequence (full packet) of low values. 

+      if (level_dbfs < kSilenceThresholdDBFS)
+        level = 0.0f;
+      else if (level_dbfs > 0.0f)
+        level = 1.0f;
+      else
+        level = 1.0f - level_dbfs / kSilenceThresholdDBFS;
+      DVLOG(1) << "audio_level: " << level;
+
+      audio_level_->Reset();
+    }
+
     return;

[tommi (sloooow) - chröme, 2014/05/17 13:45:00]

nit: remove?

[henrika (OOO until Aug 14), 2014/05/19 11:58:51]

has been modified.

   }
 
   // TODO(henrika): Investigate if we can avoid the extra copy here.
   // (see http://crbug.com/249316 for details). AFAIK, this scope is only
   // active for WebSpeech clients.
   scoped_ptr<uint8[]> audio_data(new uint8[size]);
   memcpy(audio_data.get(), data, size);
 
   // Ownership of the audio buffer will be with the callback until it is run,
   // when ownership is passed to the callback function.
@@ skipping 29 matching lines @@
 
 void AudioInputController::SetDataIsActive(bool enabled) {
   base::subtle::Release_Store(&data_is_active_, enabled);
 }
 
 bool AudioInputController::GetDataIsActive() {
   return (base::subtle::Acquire_Load(&data_is_active_) != false);
 }
 
 }  // namespace media