Adding WebRTC Auto Gain Control Test (linux)

The auto gain control test plays a file into the fake microphone. Then it
sets up a one-way WebRTC call with audio only and records Chrome's output on
the receiving side using the audio loopback provided by the quality test
(see the class comments for more details).

Then both the recording and reference file are split on silence. This creates
a number of segments with speech in them. The reason for this is to provide
a kind of synchronization mechanism so the start of each speech segment is
compared to the start of the corresponding speech segment. This is because we
will experience inevitable clock drift between the system clock (which runs
the fake microphone) and the sound card (which runs play-out). Effectively
re-synchronizing on each segment mitigates this.

The silence splitting is inherently sensitive to the sound file we run on.
Therefore the reference file must have at least 500 ms of pure silence
between speech segments; the test will fail if the output produces more
segments than the reference.

The test reports the difference in dBFS units between the reference and
output file per 10 ms interval in each speech segment. A value of 6 means the
output was 6 dBFS units louder than the reference, presumably because the
AGC applied gain to the signal.

We record in CD format here (44.1 kHz) because that's what the fake input
device currently supports, and we want to be able to compare directly.

This test is currently Linux only. It should work on the other platforms
too but I'll try to get this running first. It will run on the WebRTC
internal bots and report results to the perf dashboard, so we can look at
the data it produces and see if the data on the bots is as consistent as
on my workstation.

BUG=303259
Committed: https://crrev.com/fd442ed0421ff475ed3db26ec6d5ab61de7021f0
Cr-Commit-Position: refs/heads/master@{#309196}

[phoglund_chromium, 2014-12-17 18:52:00]

phoglund@chromium.org changed reviewers:
+ henrika@chromium.org

[phoglund_chromium, 2014-12-17 18:52:52]

https://codereview.chromium.org/799983002/diff/60001/chrome/browser/media/chr...
File chrome/browser/media/chrome_webrtc_audio_quality_browsertest.cc (right):

https://codereview.chromium.org/799983002/diff/60001/chrome/browser/media/chr...
chrome/browser/media/chrome_webrtc_audio_quality_browsertest.cc:347: const char*
kTreshold = "3%";
5% turned out to be too aggressive for the new input file, but this works for
the old one too.

https://codereview.chromium.org/799983002/diff/100001/chrome/browser/media/ch...
File chrome/browser/media/chrome_webrtc_audio_quality_browsertest.cc (right):

https://codereview.chromium.org/799983002/diff/100001/chrome/browser/media/ch...
chrome/browser/media/chrome_webrtc_audio_quality_browsertest.cc:370: // Looks
for 0.3-second silences (under 1% audio power) and splits the input
I'm thinking about moving some of these new functions into
webrtc_browsertest_audio in the next patch. In that case I'll probably move
existing helper functions as well, and that would have made this patch harder to
understand so I'll wait with that.

[phoglund_chromium, 2014-12-17 18:53:13]

An early Christmas present for you, mr. Andreasson.

[phoglund_chromium, 2014-12-17 18:54:55]

https://codereview.chromium.org/799983002/diff/100001/media/audio/fake_audio_...
File media/audio/fake_audio_input_stream.cc (right):

https://codereview.chromium.org/799983002/diff/100001/media/audio/fake_audio_...
media/audio/fake_audio_input_stream.cc:218: // Stop playing if we've played out
the whole file.
The AGC test is the only user of this anyway so I can bluntly change it from
looping to not looping.

[bjornv, 2014-12-18 11:30:27]

bjornv@chromium.org changed reviewers:
+ bjornv@chromium.org

[bjornv, 2014-12-18 11:30:28]

Sorry, Patrik. I thought we agreed on calculating the overall energy of the last
x segments and compare.

https://codereview.chromium.org/799983002/diff/100001/chrome/browser/media/ch...
File chrome/browser/media/chrome_webrtc_audio_quality_browsertest.cc (right):

https://codereview.chromium.org/799983002/diff/100001/chrome/browser/media/ch...
chrome/browser/media/chrome_webrtc_audio_quality_browsertest.cc:512: // fail the
the test if the difference is too big (if it is, we're likely
Remove one "the"

https://codereview.chromium.org/799983002/diff/100001/chrome/browser/media/ch...
chrome/browser/media/chrome_webrtc_audio_quality_browsertest.cc:516: // file is
6 dBFS units louder than the reference for one particular 10ms
dBFS here is wrong. It is simply dB. The reference to Full Scale was lost in the
subtraction.

The math:
10*log(Energy_A/Energy_full_scale) - 10*log(Energy_B/Energy_full_scale)
= 10*log(Energy_A/Energy_full_scale * Energy_full_scale/Energy_B)
= 10*log(Energy_A/Energy_B)

AND
I thought we agreed on computing the energy for the whole segment and not
multiple 10 ms chunks. That was only used for verify there was no weird effects
when AGC was disabled.

https://codereview.chromium.org/799983002/diff/100001/chrome/browser/media/ch...
chrome/browser/media/chrome_webrtc_audio_quality_browsertest.cc:558: for (size_t
i = 0; i < ref_segments.size(); i++) {
Don't compare the first 1-2 segments, due to AGC convergence time.

https://codereview.chromium.org/799983002/diff/100001/chrome/browser/media/we...
File chrome/browser/media/webrtc_browsertest_audio.cc (right):

https://codereview.chromium.org/799983002/diff/100001/chrome/browser/media/we...
chrome/browser/media/webrtc_browsertest_audio.cc:65: // Set up the bus to read
10 ms at a time.
Could we set up the bus to read the entire segment, or controlled by an input
parameter?

[henrika (OOO until Aug 14), 2014-12-18 11:32:20]

Not adding any extra comments at this stage since bjornv proposes rather large
changes.

[phoglund_chromium, 2014-12-18 12:05:56]

> 
> AND
> I thought we agreed on computing the energy for the whole segment and not
> multiple 10 ms chunks. That was only used for verify there was no weird
effects
> when AGC was disabled.
> 

Oh, right. I tried that out and I'm not convinced the results are actually
useful. These are results with the change:

-25 -28, RESULT agc_energy_mod_with_agc:
speech_44kHz_16bit_stereo.wav_segment_0= [3] dBFS
-38 -41, RESULT agc_energy_mod_with_agc:
speech_44kHz_16bit_stereo.wav_segment_1= [2] dBFS
-36 -38, RESULT agc_energy_mod_with_agc:
speech_44kHz_16bit_stereo.wav_segment_2= [2] dBFS
-40 -39, RESULT agc_energy_mod_with_agc:
speech_44kHz_16bit_stereo.wav_segment_3= [-2] dBFS
-41 -39, RESULT agc_energy_mod_with_agc:
speech_44kHz_16bit_stereo.wav_segment_4= [-2] dBFS
-40 -40, RESULT agc_energy_mod_with_agc:
speech_44kHz_16bit_stereo.wav_segment_5= [0] dBFS

These are the results without the change:

RESULT agc_energy_modwith_agc: speech_44kHz_16bit_stereo.wav_segment_0=
[-3,-6,-8,-7,-2,2,3,2,2,6,10,15,18,16,11,7,4,5,5,6,6,5,5,6,6,6,5,3,2,1,2,4,5,4,5,5,7,10,13,6,1,-2,-3,4,5,7,7,8]
dBFS

MEAN: 4.5625

RESULT agc_energy_modwith_agc: speech_44kHz_16bit_stereo.wav_segment_1=
[8,3,5,8,8,10,10,9,11,13,17,19,20,19,15,8,-6,-9,-1,0,0,1,2,3,4,1,-3,-6,-9,-10,-9,-7,-5,-4,-4,-4,-2,3,1,12,11,10,12,14,13,13,10,5,6,3,0,-1,0,0,-1,1,5,16,20,24,26,25,26,25,27,33]
dBFS

MEAN: 6.8787878787878789

RESULT agc_energy_modwith_agc: speech_44kHz_16bit_stereo.wav_segment_2=
[5,0,8,5,7,8,5,6,6,6,7,6,4,2,3,6,5,5,5,3,3,3,3,3,3,4,3,4,8,9,6,1,6,8,8,9,7,8,9,8,6,5,6,6,5,7,8,10,11,11,10,7,6,5,6,7,8,6,2,4,9,7,5,5,6,4,5,4,4,4,5,5,4,4,5,5,6,7,7,8,10,12,13,12,10,10,8,3,-1,2,8,6,2,1,1,3,4,6,9,10,10,11,11,8,-1,-3,-2,3,8,2,-2,-3,2,2,5,8,7,5,7,8,9,11,11,6,4,2,1,2,2,3,6,5,6,9,12]
dBFS

MEAN: 5.659259259259259

RESULT agc_energy_modwith_agc: speech_44kHz_16bit_stereo.wav_segment_3=
[4,12,2,-3,4,3,2,3,3,4,3,2,3,4,4,3,4,4,4,7,9,7,6,7,7,4,3,2,7,9,11,11,9,8,5,7,5,5,4,6,9,12,14,9,5,4,6,8,10,12,12,11,8,6,6,9,0,-3,-4,5,4,4,5,7,9,11,9,6,6,7,10,10,8,5,4,4,4,4,5,6,7,6,6,8,10,14,15,13,12,5,-1,-2,0,2,11,11,8,8,8,5,4,5,5,8,8,9,11,12,8,5,4,1,3,5,7,7,9,6,2,2,1,6,7,6,6,7,10,10,11,9,11,11,14]
dBFS

MEAN 6.3684210526315788

RESULT agc_energy_modwith_agc: speech_44kHz_16bit_stereo.wav_segment_4=
[6,7,7,6,7,6,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,6,7,6,6,7,6,7,7,7,7,6,7,6,7,7,7,7,7,7,8,8,8,8,8,9,11,6,4,6,9,9,8,8,6,5,6,4,6,6,3,5,7,10,10,11,7,2,-6,1,5,4,5,5,6,6,6,8,10,10,11,11,10,10,10,5,2,6,7,2,10,9,6,10,6,0,1,2,2,4,2,5,6,6,4,7,4,4,7,-2,-2,6,9,10,11,12,11,5,5,3,5,7,8,8,9,10,9,-6,-9,1,1,-2,-7,1,8,9,11,10,9,9,9,9,8,8,7,8,9,11,12,14]
dBFS

MEAN 6.4131736526946108

RESULT agc_energy_modwith_agc: speech_44kHz_16bit_stereo.wav_segment_5=
[7,6,6,5,4,3,3,3,2,1,2,2,3,3,4,4,5,6,5,7,6,7,7,7,7,7,7,7,6,7,7,7,7,7,7,7,7,6,6,6,6,5,3,-3,-3,-2,-5,-4,2,4,7,12,13,11,9,0,-1,6,10,11,10,8,8,8,7,7,6,6,7,3,5,6,7,7,6,7,6,5,7,12,11,9,9,11,10,7,6,6,9,10,7,9,10,12]
dBFS

MEAN: 5.96

The mean values calculated from the list of 10 ms segments strike me as more
believable rather than 3,2,2,-2,-2,0. When I listen to the file I think it
sounds rather evenly amplified. I think the problem might be that the power
algorithm works better on shorter segments, e.g. more like 10 ms rather than two
seconds. I can experiment with making the intervals a bit longer, like 50 or 100
ms.

[phoglund_chromium, 2014-12-18 12:12:57]

Here are results when I turn up from 10 to 100 ms for the audio bus:

RESULT agc_energy_mod_with_agc: speech_44kHz_16bit_stereo.wav_segment_0=
[6,5,1,6,15] dBFS
MEAN 6.5999999999999996
RESULT agc_energy_mod_with_agc: speech_44kHz_16bit_stereo.wav_segment_1=
[14,0,-11,12,2,24,51] dBFS
MEAN 13.142857142857142
RESULT agc_energy_mod_with_agc: speech_44kHz_16bit_stereo.wav_segment_2=
[6,3,9,8,11,3,3,8,2,11,3,7,3,14] dBFS
MEAN 6.5
RESULT agc_energy_mod_with_agc: speech_44kHz_16bit_stereo.wav_segment_3=
[4,8,9,5,12,5,7,6,4,4,4,-2,9,20] dBFS
MEAN 6.7857142857142856
RESULT agc_energy_mod_with_agc: speech_44kHz_16bit_stereo.wav_segment_4=
[7,7,7,7,7,8,7,7,5,8,8,5,9,8,-3,6,18] dBFS
MEAN 7.117647058823529
RESULT agc_energy_mod_with_agc: speech_44kHz_16bit_stereo.wav_segment_5=
[3,0,2,7,5,8,3,10,10,27] dBFS
MEAN 7.5

[phoglund_chromium, 2014-12-18 12:18:38]

I'll send you the actual recorded files offline so you can figure out which of
the three alternatives above are the most correct.

[bjornv, 2014-12-18 12:59:59]

On 2014/12/18 12:18:38, phoglund wrote:
> I'll send you the actual recorded files offline so you can figure out which of
> the three alternatives above are the most correct.

I looked at the power_monitor and it is a weighted moving average. So if you use
the same time constant you'll basically end up with the last 10 ms being
effective.
And as the name says, it's a _monitor_ like a real-time display of the power,
with smoothing.

Try increasing the time constant kPowerMeasurementTimeConstantMillis to 100 or
1000 for the all segment case.

Otherwise, we need to compare the AVERAGE(10 ms chunks with 10 ms time constant)

[phoglund_chromium, 2014-12-18 14:30:24]

> I looked at the power_monitor and it is a weighted moving average. So if you
use
> the same time constant you'll basically end up with the last 10 ms being
> effective.
> And as the name says, it's a _monitor_ like a real-time display of the power,
> with smoothing.
> 
> Try increasing the time constant kPowerMeasurementTimeConstantMillis to 100 or
> 1000 for the all segment case.
> 
> Otherwise, we need to compare the AVERAGE(10 ms chunks with 10 ms time
constant)

Actually, with 1000 it looks a lot better: 

RESULT agc_energy_mod_with_agc: speech_44kHz_16bit_stereo.wav_segment_0= [6]
dBFS
RESULT agc_energy_mod_with_agc: speech_44kHz_16bit_stereo.wav_segment_1= [7]
dBFS
RESULT agc_energy_mod_with_agc: speech_44kHz_16bit_stereo.wav_segment_2= [6]
dBFS
RESULT agc_energy_mod_with_agc: speech_44kHz_16bit_stereo.wav_segment_3= [6]
dBFS
RESULT agc_energy_mod_with_agc: speech_44kHz_16bit_stereo.wav_segment_4= [6]
dBFS
RESULT agc_energy_mod_with_agc: speech_44kHz_16bit_stereo.wav_segment_5= [5]
dBFS

2000 also seems to work well. Would it make sense to make the constant actually
equal to the segment length for each segment?

[phoglund_chromium, 2014-12-18 14:57:47]

Results where I set the measurement value to the file's duration. It appears to
work and I assume this will work best if we ever run a file with longer speech
segments:

RESULT agc_energy_diff_with_agc: speech_44kHz_16bit_stereo.wav_segment_0=
5.807039260864258 dB
RESULT agc_energy_diff_with_agc: speech_44kHz_16bit_stereo.wav_segment_1=
5.478204727172852 dB
RESULT agc_energy_diff_with_agc: speech_44kHz_16bit_stereo.wav_segment_2=
5.556024551391602 dB
RESULT agc_energy_diff_with_agc: speech_44kHz_16bit_stereo.wav_segment_3=
5.471179962158203 dB
RESULT agc_energy_diff_with_agc: speech_44kHz_16bit_stereo.wav_segment_4=
6.036399841308594 dB
RESULT agc_energy_diff_with_agc: speech_44kHz_16bit_stereo.wav_segment_5=
5.422649383544922 dB

Without audio processing:

RESULT agc_energy_diff_no_agc: speech_44kHz_16bit_stereo.wav_segment_0=
-.8436851501464844 dB
RESULT agc_energy_diff_no_agc: speech_44kHz_16bit_stereo.wav_segment_1=
-.2960071563720703 dB
RESULT agc_energy_diff_no_agc: speech_44kHz_16bit_stereo.wav_segment_2=
-.17702865600585938 dB
RESULT agc_energy_diff_no_agc: speech_44kHz_16bit_stereo.wav_segment_3=
-.5085678100585938 dB
RESULT agc_energy_diff_no_agc: speech_44kHz_16bit_stereo.wav_segment_4=
-.6086807250976562 dB
RESULT agc_energy_diff_no_agc: speech_44kHz_16bit_stereo.wav_segment_5=
-.1779022216796875 dB

I didn't see any good reasons to round the numbers anymore since there's only
one number now.

The code got a lot simpler now as well; patch coming up right away.

[phoglund_chromium, 2014-12-18 15:02:38]

Ok, PTAL!

[henrika (OOO until Aug 14), 2014-12-19 10:00:15]

Very impressive CL. Nice work indeed. 
LGTM with nits ;-)

https://codereview.chromium.org/799983002/diff/120001/chrome/browser/media/ch...
File chrome/browser/media/chrome_webrtc_audio_quality_browsertest.cc (right):

https://codereview.chromium.org/799983002/diff/120001/chrome/browser/media/ch...
chrome/browser/media/chrome_webrtc_audio_quality_browsertest.cc:185: // on
success. We record in CD format unless record_cd is false, in which case
|record_cd| and please don't use CD/DAT format. Be more specific in terms of
sample rate, #bits etc.

https://codereview.chromium.org/799983002/diff/120001/chrome/browser/media/ch...
chrome/browser/media/chrome_webrtc_audio_quality_browsertest.cc:316: CommandLine
MakeSoxCommandLine() {
Could you add some comments and reference to SoX?

https://codereview.chromium.org/799983002/diff/120001/chrome/browser/media/ch...
chrome/browser/media/chrome_webrtc_audio_quality_browsertest.cc:346: const char*
kAbovePeriods = "1";
How essential are these exact parameter values for a working test? Can it be
flaky?

https://codereview.chromium.org/799983002/diff/120001/chrome/browser/media/ch...
chrome/browser/media/chrome_webrtc_audio_quality_browsertest.cc:495: // Splits
|to_split| into sub-files base on silence. The file you use must have
nit, 'based on'

https://codereview.chromium.org/799983002/diff/120001/chrome/browser/media/ch...
chrome/browser/media/chrome_webrtc_audio_quality_browsertest.cc:649: * device
currently supports, and we want to be able to compare directly.
Any link to a crbug here?

https://codereview.chromium.org/799983002/diff/120001/chrome/browser/media/ch...
chrome/browser/media/chrome_webrtc_audio_quality_browsertest.cc:727: // The AGC
should apply gain here on the order of 5-6 dBFS units.
How do you know this gain? is it digital only?

[phoglund_chromium, 2014-12-19 10:34:21]

https://codereview.chromium.org/799983002/diff/120001/chrome/browser/media/ch...
File chrome/browser/media/chrome_webrtc_audio_quality_browsertest.cc (right):

https://codereview.chromium.org/799983002/diff/120001/chrome/browser/media/ch...
chrome/browser/media/chrome_webrtc_audio_quality_browsertest.cc:185: // on
success. We record in CD format unless record_cd is false, in which case
On 2014/12/19 10:00:14, henrika wrote:
> |record_cd| and please don't use CD/DAT format. Be more specific in terms of
> sample rate, #bits etc.

Done.

https://codereview.chromium.org/799983002/diff/120001/chrome/browser/media/ch...
chrome/browser/media/chrome_webrtc_audio_quality_browsertest.cc:316: CommandLine
MakeSoxCommandLine() {
On 2014/12/19 10:00:14, henrika wrote:
> Could you add some comments and reference to SoX?

Done.

https://codereview.chromium.org/799983002/diff/120001/chrome/browser/media/ch...
chrome/browser/media/chrome_webrtc_audio_quality_browsertest.cc:346: const char*
kAbovePeriods = "1";
On 2014/12/19 10:00:14, henrika wrote:
> How essential are these exact parameter values for a working test? Can it be
> flaky?

It depends a lot on the files we feed in. With the files we currently use, I
think these values give us a good margin. Particularly for this new test I
inserted extra silence between the speech segments to make silence splitting
more reliable. We can't be sure until we run on the bots though.

https://codereview.chromium.org/799983002/diff/120001/chrome/browser/media/ch...
chrome/browser/media/chrome_webrtc_audio_quality_browsertest.cc:495: // Splits
|to_split| into sub-files base on silence. The file you use must have
On 2014/12/19 10:00:14, henrika wrote:
> nit, 'based on'

Done.

https://codereview.chromium.org/799983002/diff/120001/chrome/browser/media/ch...
chrome/browser/media/chrome_webrtc_audio_quality_browsertest.cc:649: * device
currently supports, and we want to be able to compare directly.
On 2014/12/19 10:00:14, henrika wrote:
> Any link to a crbug here?

Sure, I can link to the fake device bug where the reasons for this are
documented.

https://codereview.chromium.org/799983002/diff/120001/chrome/browser/media/ch...
chrome/browser/media/chrome_webrtc_audio_quality_browsertest.cc:727: // The AGC
should apply gain here on the order of 5-6 dBFS units.
On 2014/12/19 10:00:14, henrika wrote:
> How do you know this gain? is it digital only?

I guess I don't really, but those are the results I'm getting and I think that
sounds reasonable according to how much louder the recording sounds than the
ref. I realize this is a circular argument though so I'll just remove the
comment :)

[phoglund_chromium, 2014-12-19 10:44:34]

The CQ bit was checked by phoglund@chromium.org

[commit-bot: I haz the power, 2014-12-19 10:45:28]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/799983002/140001

[commit-bot: I haz the power, 2014-12-19 11:22:24]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2014-12-19 11:22:26]

Try jobs failed on following builders:
  win_gpu on tryserver.chromium.gpu
(http://build.chromium.org/p/tryserver.chromium.gpu/builders/win_gpu/builds/10...)

[phoglund_chromium, 2014-12-19 12:33:39]

The CQ bit was checked by phoglund@chromium.org

[commit-bot: I haz the power, 2014-12-19 12:34:03]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/799983002/160001

[commit-bot: I haz the power, 2014-12-19 14:28:42]

Committed patchset #9 (id:160001)

[commit-bot: I haz the power, 2014-12-19 14:29:34]

Patchset 9 (id:??) landed as
https://crrev.com/fd442ed0421ff475ed3db26ec6d5ab61de7021f0
Cr-Commit-Position: refs/heads/master@{#309196}