|
|
Created:
4 years, 10 months ago by Tima Vaisburd Modified:
4 years, 9 months ago CC:
chromium-reviews, feature-media-reviews_chromium.org Base URL:
https://chromium.googlesource.com/chromium/src.git@spitzer-aad-test Target Ref:
refs/pending/heads/master Project:
chromium Visibility:
Public. |
DescriptionAdded unit test from MediaCodecPlayerAndroid
Modified AndrodDecoderTest to accept delay in both decode
and output callbacks. Changed MediaCodecAudioDecoder to
pass the test.
Only Opus file is included in this CL.
BUG=542910
Committed: https://crrev.com/6f344af54443c7e169e2b6969174f4d5986e3335
Cr-Commit-Position: refs/heads/master@{#379647}
Patch Set 1 #
Total comments: 16
Patch Set 2 : Fixed bug in MediaCodecAudioDecoder::QueueInput() #
Total comments: 7
Patch Set 3 : Replaced a polling loop with a nested RunLoop, added comments #Patch Set 4 : Rebase only #Patch Set 5 : Skipped tests if MediaCodec is needed but not available on device #
Messages
Total messages: 27 (9 generated)
timav@chromium.org changed reviewers: + dalecurtis@google.com, liberato@chromium.org, qinmin@chromium.org, watk@chromium.org, xhwang@chromium.org
This change has the following fixed to MediaCodecAudioDecoder: 1. Uses AudioTimestampHalper to calculate timestamps, 2. Tries to enqueue all input DecoderBuffer's instead of just one - if the prior succeeded, tried the next if available, 3. Checks for valid time stamp in the input. Because if these changes I include most people who reviewed the original implementation as the reviewers, watk@, liberato@ qinmin@: fyi, and please provide your valuable comments, as always.
Description was changed from ========== Added unit test from MediaCodecPlayerAndroid Modified AndrodDecoderTest to accept delay in both decode and output callbacks. Changed MediaCodecAudioDecoder to pass the test. BUG=542910 ========== to ========== Added unit test from MediaCodecPlayerAndroid Modified AndrodDecoderTest to accept delay in both decode and output callbacks. Changed MediaCodecAudioDecoder to pass the test. Only Opus file is included in this CL. BUG=542910 ==========
I forgot to mention that this CL only includes the Opus. AAC is still unsolved.
https://codereview.chromium.org/1740313002/diff/1/media/filters/android/media... File media/filters/android/media_codec_audio_decoder.cc (right): https://codereview.chromium.org/1740313002/diff/1/media/filters/android/media... media/filters/android/media_codec_audio_decoder.cc:226: return did_work; this will always return false, else the loop didn't exit. https://codereview.chromium.org/1740313002/diff/1/media/filters/audio_decoder... File media/filters/audio_decoder_unittest.cc (right): https://codereview.chromium.org/1740313002/diff/1/media/filters/audio_decoder... media/filters/audio_decoder_unittest.cc:148: void WaitForDecodeResponse() { "do work for 100msec" seems flaky. i see that the async nature of MediaCodec makes this somewhat difficult to avoid, but 100msec seems way too short. what would you think about mocking out MediaCodec, maybe with a software decoder? or, does this test run on multiple android devices? is its goal to be more of an integration test? https://codereview.chromium.org/1740313002/diff/1/media/filters/audio_decoder... media/filters/audio_decoder_unittest.cc:152: for (int i = 0; i < timeout / sleep_time; ++i) { not sure if ::Sleep guarantees not to wake up early, but "Now() < start+timeout" makes it not matter. https://codereview.chromium.org/1740313002/diff/1/media/filters/audio_decoder... media/filters/audio_decoder_unittest.cc:362: } while (decoded_audio_size() < kDecodeRuns); why did this change? does MediaCodec have a variable # buffer delay, or is it some side-effect of the async issues (WaitForDecodeResponse)? https://codereview.chromium.org/1740313002/diff/1/media/filters/audio_decoder... media/filters/audio_decoder_unittest.cc:370: ASSERT_LE(kDecodeRuns, decoded_audio_size()); why can't we guarantee EQ here? is it because one must RunUntilIdle, which might finish more than one Decode request in one call?
https://codereview.chromium.org/1740313002/diff/1/media/filters/android/media... File media/filters/android/media_codec_audio_decoder.cc (right): https://codereview.chromium.org/1740313002/diff/1/media/filters/android/media... media/filters/android/media_codec_audio_decoder.cc:226: return did_work; On 2016/02/29 17:08:57, liberato wrote: > this will always return false, else the loop didn't exit. Yes, thank you, it was a bug. Rewritten. https://codereview.chromium.org/1740313002/diff/1/media/filters/audio_decoder... File media/filters/audio_decoder_unittest.cc (right): https://codereview.chromium.org/1740313002/diff/1/media/filters/audio_decoder... media/filters/audio_decoder_unittest.cc:148: void WaitForDecodeResponse() { On 2016/02/29 17:08:57, liberato wrote: > "do work for 100msec" seems flaky. i see that the async nature of MediaCodec > makes this somewhat difficult to avoid, but 100msec seems way too short. > > what would you think about mocking out MediaCodec, maybe with a software > decoder? I think that mocking MediaCodec is a good idea, as it would allow to write the tests that check input/output timestamps, delays for input and output buffers etc. For mocking, maybe, we can even have pre-canned output and do not have to use a software decoder. On the other hand, the async nature of MediaCodec seems to be the most important thing to simulate, and we would have to wait for response somehow even for the mocked version of decoder. > or, does this test run on multiple android devices? is its goal to be more of > an integration test? Yes, I think this is true. https://codereview.chromium.org/1740313002/diff/1/media/filters/audio_decoder... media/filters/audio_decoder_unittest.cc:152: for (int i = 0; i < timeout / sleep_time; ++i) { On 2016/02/29 17:08:57, liberato wrote: > not sure if ::Sleep guarantees not to wake up early, but "Now() < start+timeout" > makes it not matter. Yes, I was not sure whether to use the Sleep(), or to implement timeout with a timer, as in https://code.google.com/p/chromium/codesearch#chromium/src/media/base/android... There I call RunUnitilIdle() in a loop until a condition happens or a timer fires. The version with the Sleep seems simpler? What do you think? https://codereview.chromium.org/1740313002/diff/1/media/filters/audio_decoder... media/filters/audio_decoder_unittest.cc:362: } while (decoded_audio_size() < kDecodeRuns); On 2016/02/29 17:08:57, liberato wrote: > why did this change? > > does MediaCodec have a variable # buffer delay, or is it some side-effect of the > async issues (WaitForDecodeResponse)? I think first. See below. https://codereview.chromium.org/1740313002/diff/1/media/filters/audio_decoder... media/filters/audio_decoder_unittest.cc:370: ASSERT_LE(kDecodeRuns, decoded_audio_size()); On 2016/02/29 17:08:57, liberato wrote: > why can't we guarantee EQ here? is it because one must RunUntilIdle, which > might finish more than one Decode request in one call? As far as I understand, this is the consequence of OpenMax interface which MediaCodec is based upon. There are no guarantees wrt when output buffers will be available relative to the moments we enqueue input buffers. Even after you waited for the first output buffer, there is no guarantee, in principle, that there will be "one in - one out". I think several output buffers can be available at once, and our decoder implementation returns them all at once. The only way I see to have ASSERT_EQ is send the EOS first: decode 3 buffers, then SendEndOfStream(), then wait for 3 output buffers to come. I can do this if you think the equality is important.
dalecurtis@chromium.org changed reviewers: + dalecurtis@chromium.org
https://codereview.chromium.org/1740313002/diff/20001/media/filters/android/m... File media/filters/android/media_codec_audio_decoder.cc (right): https://codereview.chromium.org/1740313002/diff/20001/media/filters/android/m... media/filters/android/media_codec_audio_decoder.cc:538: audio_buffer->set_timestamp(timestamp_helper_->GetTimestamp()); I believe instead of a timestamp helper you actually want a discard helper here so that MSE's partial append window trimming will function correctly. See how OpusAudioDecoder and FFmpegAudioDecoder use their AudioDiscardHelper classes. https://codereview.chromium.org/1740313002/diff/20001/media/filters/audio_dec... File media/filters/audio_decoder_unittest.cc (right): https://codereview.chromium.org/1740313002/diff/20001/media/filters/audio_dec... media/filters/audio_decoder_unittest.cc:144: WaitForDecodeResponse(); Can we instead have DecodeFinished() take a closure from run_loop.QuitClosure() and execute that once called?
https://codereview.chromium.org/1740313002/diff/20001/media/filters/android/m... File media/filters/android/media_codec_audio_decoder.cc (right): https://codereview.chromium.org/1740313002/diff/20001/media/filters/android/m... media/filters/android/media_codec_audio_decoder.cc:538: audio_buffer->set_timestamp(timestamp_helper_->GetTimestamp()); On 2016/02/29 21:51:30, DaleCurtis wrote: > I believe instead of a timestamp helper you actually want a discard helper here > so that MSE's partial append window trimming will function correctly. See how > OpusAudioDecoder and FFmpegAudioDecoder use their AudioDiscardHelper classes. I have a question. As far as I can get these decoders modify the output buffer using all input buffers that contributed to that output buffer. In MediaCodec, however, the association between input and output buffers is lost. We can build it ourselves based on timestamps, but this is not completely reliable. Do you see an easier way to accomplish this and, do we have to have this discard for this decoder?
https://codereview.chromium.org/1740313002/diff/20001/media/filters/android/m... File media/filters/android/media_codec_audio_decoder.cc (right): https://codereview.chromium.org/1740313002/diff/20001/media/filters/android/m... media/filters/android/media_codec_audio_decoder.cc:538: audio_buffer->set_timestamp(timestamp_helper_->GetTimestamp()); On 2016/03/01 at 01:08:07, Tima Vaisburd wrote: > On 2016/02/29 21:51:30, DaleCurtis wrote: > > I believe instead of a timestamp helper you actually want a discard helper here > > so that MSE's partial append window trimming will function correctly. See how > > OpusAudioDecoder and FFmpegAudioDecoder use their AudioDiscardHelper classes. > > I have a question. > As far as I can get these decoders modify the output buffer using all input buffers that contributed to that output buffer. In MediaCodec, however, the association between input and output buffers is lost. We can build it ourselves based on timestamps, but this is not completely reliable. Do you see an easier way to accomplish this and, do we have to have this discard for this decoder? Right now it's fine to just use the timestamp helper, but without it gapless playback using MSE will not work (used by Play Music and likely transitively by Cast Android receiver). Likely to make this work we'll need to extend AudioDiscardHelper's ability to deal with discards that span multiple buffers and (as you say) attach the input buffers based on frame count (or timing).
https://codereview.chromium.org/1740313002/diff/20001/media/filters/audio_dec... File media/filters/audio_decoder_unittest.cc (right): https://codereview.chromium.org/1740313002/diff/20001/media/filters/audio_dec... media/filters/audio_decoder_unittest.cc:144: WaitForDecodeResponse(); On 2016/02/29 21:51:30, DaleCurtis wrote: > Can we instead have DecodeFinished() take a closure from run_loop.QuitClosure() > and execute that once called? I'm sorry, I did not understand. Could you, please, elaborate? Do you suggest to stop the run loop? The reason we did not get the DecodeCB right away (and it was not even posted before decoder_->Decode() returns) is that no input buffers are available at this time. Our decoder will succeed the second attempt, in 10 ms. I have seen this happens after Reset().
https://codereview.chromium.org/1740313002/diff/20001/media/filters/audio_dec... File media/filters/audio_decoder_unittest.cc (right): https://codereview.chromium.org/1740313002/diff/20001/media/filters/audio_dec... media/filters/audio_decoder_unittest.cc:144: WaitForDecodeResponse(); On 2016/03/01 at 01:55:49, Tima Vaisburd wrote: > On 2016/02/29 21:51:30, DaleCurtis wrote: > > Can we instead have DecodeFinished() take a closure from run_loop.QuitClosure() > > and execute that once called? > > I'm sorry, I did not understand. Could you, please, elaborate? Do you suggest to stop the run loop? > > The reason we did not get the DecodeCB right away (and it was not even posted before decoder_->Decode() returns) is that no input buffers are available at this time. Our decoder will succeed the second attempt, in 10 ms. I have seen this happens after Reset(). I mean instead of having a spin-loop do: base::RunLoop run_loop; decoder_->Decode( buffer, base::Bind(&AudioDecoderTest::DecodeFinished, base::Unretained(this), run_loop.QuitClosure())); run_loop.Run(); Inside DecodeFinished you can call QuitClosure().
fun with tests :) -fl https://codereview.chromium.org/1740313002/diff/1/media/filters/audio_decoder... File media/filters/audio_decoder_unittest.cc (right): https://codereview.chromium.org/1740313002/diff/1/media/filters/audio_decoder... media/filters/audio_decoder_unittest.cc:148: void WaitForDecodeResponse() { On 2016/02/29 21:01:03, Tima Vaisburd wrote: > On 2016/02/29 17:08:57, liberato wrote: > > "do work for 100msec" seems flaky. i see that the async nature of MediaCodec > > makes this somewhat difficult to avoid, but 100msec seems way too short. > > > > what would you think about mocking out MediaCodec, maybe with a software > > decoder? > > I think that mocking MediaCodec is a good idea, as it would allow to write the > tests that check input/output timestamps, delays for input and output buffers > etc. For mocking, maybe, we can even have pre-canned output and do not have to > use a software decoder. On the other hand, the async nature of MediaCodec seems > to be the most important thing to simulate, and we would have to wait for > response somehow even for the mocked version of decoder. > > > or, does this test run on multiple android devices? is its goal to be more of > > an integration test? > > Yes, I think this is true. i was thinking about a very simple MediaCodec mock. make outputs available immediately. that lets us test the correctness of a bunch of MCAD code. i don't think that what we have here really tests that very reliably. it's unpredictable what delays we'll see in practice, and also correspondingly hard to fix if the test is flaky. the remainder of these tests do not seem to have that sort of variation in them (at least, not from my quick reading of them). if this is the first such test to depend on timing, then maybe the mock is more appropriate. if these tests already have weird timing stuff happening for other platforms / codecs, then i don't see a problem with adding another. https://codereview.chromium.org/1740313002/diff/1/media/filters/audio_decoder... media/filters/audio_decoder_unittest.cc:370: ASSERT_LE(kDecodeRuns, decoded_audio_size()); On 2016/02/29 21:01:03, Tima Vaisburd wrote: > On 2016/02/29 17:08:57, liberato wrote: > > why can't we guarantee EQ here? is it because one must RunUntilIdle, which > > might finish more than one Decode request in one call? > > As far as I understand, this is the consequence of OpenMax interface which > MediaCodec is based upon. There are no guarantees wrt when output buffers will > be available relative to the moments we enqueue input buffers. Even after you > waited for the first output buffer, there is no guarantee, in principle, that > there will be "one in - one out". I think several output buffers can be > available at once, and our decoder implementation returns them all at once. > > The only way I see to have ASSERT_EQ is send the EOS first: decode 3 buffers, > then SendEndOfStream(), then wait for 3 output buffers to come. > > I can do this if you think the equality is important. i don't think eq is important, but it's probably worth documenting why it's not eq.
https://codereview.chromium.org/1740313002/diff/1/media/filters/audio_decoder... File media/filters/audio_decoder_unittest.cc (right): https://codereview.chromium.org/1740313002/diff/1/media/filters/audio_decoder... media/filters/audio_decoder_unittest.cc:148: void WaitForDecodeResponse() { On 2016/03/01 22:20:39, liberato wrote: > i was thinking about a very simple MediaCodec mock. make outputs available > immediately. that lets us test the correctness of a bunch of MCAD code. Even if the MediaCodec returns the output buffer immediately, MCAD posts it, so some code would have to be modified anyway. See my reply to your second comment. > i don't think that what we have here really tests that very reliably. it's > unpredictable what delays we'll see in practice, and also correspondingly hard > to fix if the test is flaky. Well, in theory the tests should never be flaky. The reality is that we have to deal with this issue on Android platform. > if this is the first such test to depend on timing, then maybe the mock is more > appropriate. > > if these tests already have weird timing stuff happening for other platforms / > codecs, then i don't see a problem with adding another. I can only say that on Android platform media_unittests sometimes show flakiness, and we fight it. As for mock, I'm still thinking that the mock is a valuable addition, but should not be a replacement of the real test on device. I see the value of this test in that it can run on real device and we can verify that it properly decodes. Besides, I think that even the simplest mock of the MediaCodec goes beyond the scope of this CL. Where do you envision the flakiness? Maybe I can improve the code? https://codereview.chromium.org/1740313002/diff/1/media/filters/audio_decoder... media/filters/audio_decoder_unittest.cc:370: ASSERT_LE(kDecodeRuns, decoded_audio_size()); On 2016/03/01 22:20:39, liberato wrote: > On 2016/02/29 21:01:03, Tima Vaisburd wrote: > > On 2016/02/29 17:08:57, liberato wrote: > > > why can't we guarantee EQ here? is it because one must RunUntilIdle, which > > > might finish more than one Decode request in one call? > > > > As far as I understand, this is the consequence of OpenMax interface which > > MediaCodec is based upon. There are no guarantees wrt when output buffers will > > be available relative to the moments we enqueue input buffers. Even after you > > waited for the first output buffer, there is no guarantee, in principle, that > > there will be "one in - one out". I think several output buffers can be > > available at once, and our decoder implementation returns them all at once. > > > > The only way I see to have ASSERT_EQ is send the EOS first: decode 3 buffers, > > then SendEndOfStream(), then wait for 3 output buffers to come. > > > > I can do this if you think the equality is important. > > i don't think eq is important, but it's probably worth documenting why it's not > eq. I went a little deeper to see why we cannot ASSERT_EQ, or rather, why do I think it's not worth it. The original code asserts on l.344 in GetDecodedAudioMD5(j) because the output buffer is not available immediately (which in my experiment happens because the output buffer is posted by MediaCodecAudioDecoder internals). We could decode |kDecodeRuns| buffers and wait for them on the decoded side, but we do not know whether |kDecoderRuns| on input would be enough to get the same number on output. That's why I decided to go for a simpler algorithm. I tried to explain this in comments. https://codereview.chromium.org/1740313002/diff/20001/media/filters/audio_dec... File media/filters/audio_decoder_unittest.cc (right): https://codereview.chromium.org/1740313002/diff/20001/media/filters/audio_dec... media/filters/audio_decoder_unittest.cc:144: WaitForDecodeResponse(); On 2016/03/01 20:25:13, DaleCurtis wrote: > On 2016/03/01 at 01:55:49, Tima Vaisburd wrote: > > On 2016/02/29 21:51:30, DaleCurtis wrote: > > > Can we instead have DecodeFinished() take a closure from > run_loop.QuitClosure() > > > and execute that once called? > > > > I'm sorry, I did not understand. Could you, please, elaborate? Do you suggest > to stop the run loop? > > > > The reason we did not get the DecodeCB right away (and it was not even posted > before decoder_->Decode() returns) is that no input buffers are available at > this time. Our decoder will succeed the second attempt, in 10 ms. I have seen > this happens after Reset(). > > I mean instead of having a spin-loop do: > > base::RunLoop run_loop; > decoder_->Decode( > buffer, > base::Bind(&AudioDecoderTest::DecodeFinished, base::Unretained(this), > run_loop.QuitClosure())); > run_loop.Run(); > > Inside DecodeFinished you can call QuitClosure(). Done.
lgtm, yay for more tests! https://codereview.chromium.org/1740313002/diff/1/media/filters/audio_decoder... File media/filters/audio_decoder_unittest.cc (right): https://codereview.chromium.org/1740313002/diff/1/media/filters/audio_decoder... media/filters/audio_decoder_unittest.cc:148: void WaitForDecodeResponse() { On 2016/03/01 23:35:33, Tima Vaisburd wrote: > On 2016/03/01 22:20:39, liberato wrote: > > i was thinking about a very simple MediaCodec mock. make outputs available > > immediately. that lets us test the correctness of a bunch of MCAD code. > > Even if the MediaCodec returns the output buffer immediately, MCAD posts it, so > some code would have to be modified anyway. See my reply to your second comment. > > > i don't think that what we have here really tests that very reliably. it's > > unpredictable what delays we'll see in practice, and also correspondingly hard > > to fix if the test is flaky. > > Well, in theory the tests should never be flaky. The reality is that > we have to deal with this issue on Android platform. > > > if this is the first such test to depend on timing, then maybe the mock is > more > > appropriate. > > > > if these tests already have weird timing stuff happening for other platforms / > > codecs, then i don't see a problem with adding another. > > I can only say that on Android platform media_unittests sometimes show > flakiness, and we fight it. > > As for mock, I'm still thinking that the mock is a valuable addition, but should > not be a replacement of the real test on device. I see the value of this test in > that it can run on real device and we can verify that it properly decodes. > > Besides, I think that even the simplest mock of the MediaCodec goes beyond the > scope of this CL. > > Where do you envision the flakiness? Maybe I can improve the code? flakiness: a delay of 100msec seems like something that could easily be flaky. i was going to suggest "wait indefinitely", but it looks like dale's suggestion does that already. flakiness avoided. MCAD posts it: but RunUntilIdle would, i think, always catch that. mock: without the potential flakiness due to timeout, i agree that it's out of scope. as for end-to-end testing, i really don't think that what this test does is a substitute for actually checking that the output plays, arrives in time, etc. however, that's _definitely_ out of scope. :) https://codereview.chromium.org/1740313002/diff/1/media/filters/audio_decoder... media/filters/audio_decoder_unittest.cc:152: for (int i = 0; i < timeout / sleep_time; ++i) { On 2016/02/29 21:01:03, Tima Vaisburd wrote: > On 2016/02/29 17:08:57, liberato wrote: > > not sure if ::Sleep guarantees not to wake up early, but "Now() < > start+timeout" > > makes it not matter. > > Yes, I was not sure whether to use the Sleep(), or to implement timeout with a > timer, as in > https://code.google.com/p/chromium/codesearch#chromium/src/media/base/android... > > There I call RunUnitilIdle() in a loop until a condition happens or a timer > fires. The version with the Sleep seems simpler? What do you think? i think that i like dale's suggestion better than either of ours :)
lgtm
The CQ bit was checked by timav@chromium.org
CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/patch-status/1740313002/60001 View timeline at https://chromium-cq-status.appspot.com/patch-timeline/1740313002/60001
The CQ bit was unchecked by commit-bot@chromium.org
Try jobs failed on following builders: linux_android_rel_ng on tryserver.chromium.android (JOB_FAILED, https://build.chromium.org/p/tryserver.chromium.android/builders/linux_androi...)
The CQ bit was checked by timav@chromium.org
The patchset sent to the CQ was uploaded after l-g-t-m from liberato@chromium.org, dalecurtis@chromium.org Link to the patchset: https://codereview.chromium.org/1740313002/#ps80001 (title: "Skipped tests if MediaCodec is needed but not available on device")
CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/patch-status/1740313002/80001 View timeline at https://chromium-cq-status.appspot.com/patch-timeline/1740313002/80001
Message was sent while issue was closed.
Description was changed from ========== Added unit test from MediaCodecPlayerAndroid Modified AndrodDecoderTest to accept delay in both decode and output callbacks. Changed MediaCodecAudioDecoder to pass the test. Only Opus file is included in this CL. BUG=542910 ========== to ========== Added unit test from MediaCodecPlayerAndroid Modified AndrodDecoderTest to accept delay in both decode and output callbacks. Changed MediaCodecAudioDecoder to pass the test. Only Opus file is included in this CL. BUG=542910 ==========
Message was sent while issue was closed.
Committed patchset #5 (id:80001)
Message was sent while issue was closed.
Description was changed from ========== Added unit test from MediaCodecPlayerAndroid Modified AndrodDecoderTest to accept delay in both decode and output callbacks. Changed MediaCodecAudioDecoder to pass the test. Only Opus file is included in this CL. BUG=542910 ========== to ========== Added unit test from MediaCodecPlayerAndroid Modified AndrodDecoderTest to accept delay in both decode and output callbacks. Changed MediaCodecAudioDecoder to pass the test. Only Opus file is included in this CL. BUG=542910 Committed: https://crrev.com/6f344af54443c7e169e2b6969174f4d5986e3335 Cr-Commit-Position: refs/heads/master@{#379647} ==========
Message was sent while issue was closed.
Patchset 5 (id:??) landed as https://crrev.com/6f344af54443c7e169e2b6969174f4d5986e3335 Cr-Commit-Position: refs/heads/master@{#379647} |