MediaStreamRecorder: Separate video/audio origin of timestamps for WebM muxer

MediaStreamRecorder: Separate video/audio origin of timestamps for WebM muxer

In certain conditions (see bug), the origin of
timestamps for audio and video is quite separated
which causes a jump on the WebM frame timestamps.
This seems to be an issue for certain players (chrome)
and not so much for others (vlc).

This CL solves the problem by keeping a different
origin-of-timestamps for audio and video separately,
and passing a TimeDelta to the AddFrame() method.

See the bug for an investigation as to why this
started happening recently.

BUG=597034
TEST=On my Win7 laptop, navigate to [1], record
stream for a few seconds, then stop recording and
try playing back. With this CL, no freeze.

[1] https://webrtc.github.io/samples/src/content/getusermedia/record/

Committed: https://crrev.com/aa575cd9100984ba7cf7fc0377d7c5d40abbdb5f
Cr-Commit-Position: refs/heads/master@{#385899}

[mcasas, 2016-04-06 00:32:31]

Description was changed from

==========
MediaStreamRecorder: Separate video/audio origin of timestamps for WebM muxer

BUG=597034
==========

to

==========
MediaStreamRecorder: Separate video/audio origin of timestamps for WebM muxer

In certain conditions (see bug), the origin of
timestamps for audio and video is quite separated
which causes a jump on the WebM frame timestamps.
This seems to be an issue for certain players (chrome)
and not so much for others (vlc).

This CL solves the problem by keeping a different
origin-of-timestamps for audio and video separately,
and passing a TimeDelta to the AddFrame() method.

BUG=597034
TEST=On my Win7 laptop, navigate to [1], record
stream for a few seconds, then stop recording and
try playing back. With this CL, no freeze.

[1] https://webrtc.github.io/samples/src/content/getusermedia/record/
==========

[mcasas, 2016-04-06 00:33:42]

mcasas@chromium.org changed reviewers:
+ miu@chromium.org

[mcasas, 2016-04-06 00:33:43]

miu@ PTAL

[miu, 2016-04-06 00:49:01]

lgtm, HOWEVER: The reason the timestamps are far apart is because there is
more/less playout buffering going on for audio versus video.  The timestamps
should be reflecting the playout time at which each frame should be displayed on
screen (or, for audio, when each frame should come out the speaker).

With this change, I'm willing to bet A/V sync is now a problem.  At least, it
will be on some platforms.  Solving this may involve having your WebM muxer
queuing-up either audio or video (or both), and then only output both to the
WebM container once they've been aligned w.r.t. their playout timestamps.

So, I'm lgtm'ing this, and leave it up to you to decide whether this change
should be committed as-is, and whether the A/V sync performance aspects of your
feature should be addressed now or later.

[mcasas, 2016-04-06 19:45:05]

On 2016/04/06 00:49:01, miu wrote:
> lgtm, HOWEVER: The reason the timestamps are far apart is because there is
> more/less playout buffering going on for audio versus video.  The timestamps
> should be reflecting the playout time at which each frame should be displayed
on
> screen (or, for audio, when each frame should come out the speaker).
> 
> With this change, I'm willing to bet A/V sync is now a problem.  At least, it
> will be on some platforms.  Solving this may involve having your WebM muxer
> queuing-up either audio or video (or both), and then only output both to the
> WebM container once they've been aligned w.r.t. their playout timestamps.
> 
> So, I'm lgtm'ing this, and leave it up to you to decide whether this change
> should be committed as-is, and whether the A/V sync performance aspects of
your
> feature should be addressed now or later.

I don't think this is a case of audio/video sync adjustments. I 
printed out the base::TimeTicks timestamps of the incoming video
and audio recorded frames and I got the following numbers. 

First a bunch of Video Frames arrive with correlative timestamps

media::WebmMuxer::OnEncodedVideo - 59928B @ 2149924 bogo-microseconds
media::WebmMuxer::OnEncodedVideo: delaying until audio track ready.
media::WebmMuxer::OnEncodedVideo - 108B @ 2183258 bogo-microseconds
media::WebmMuxer::OnEncodedVideo: delaying until audio track ready.
media::WebmMuxer::OnEncodedVideo - 143B @ 2216591 bogo-microseconds
media::WebmMuxer::OnEncodedVideo: delaying until audio track ready.

and now the first Audio arrives with completely different timestamps:

media::WebmMuxer::OnEncodedAudio - 386B @ 1269455645930 bogo-microseconds
media::WebmMuxer::AddAudioTrack format: 1 channel_layout: 2 channels: 1
sample_rate: 48000 bits_per_sample: 16 frames_per_buffer: 2880 effects: 0
mic_positions:

and now they start mingling, each with their own reference.

media::WebmMuxer::OnEncodedVideo - 214B @ 2249924 bogo-microseconds
media::WebmMuxer::OnEncodedVideo - 217B @ 2283258 bogo-microseconds
media::WebmMuxer::OnEncodedVideo - 308B @ 2316591 bogo-microseconds
media::WebmMuxer::OnEncodedAudio - 383B @ 1269455771097 bogo-microseconds
media::WebmMuxer::OnEncodedVideo - 348B @ 2349924 bogo-microseconds
media::WebmMuxer::OnEncodedVideo - 309B @ 2383257 bogo-microseconds
media::WebmMuxer::OnEncodedVideo - 640B @ 2416591 bogo-microseconds
media::WebmMuxer::OnEncodedAudio - 383B @ 1269455878194 bogo-microseconds
media::WebmMuxer::OnEncodedVideo - 809B @ 2449924 bogo-microseconds
media::WebmMuxer::OnEncodedVideo - 977B @ 2483257 bogo-microseconds
media::WebmMuxer::OnEncodedVideo - 1118B @ 2516591 bogo-microseconds
media::WebmMuxer::OnEncodedAudio - 383B @ 1269455988642 bogo-microseconds
media::WebmMuxer::OnEncodedVideo - 1296B @ 2549924 bogo-microseconds

(I also checked that these timestamps correlate with
those coming from the Tracks, the recorders did not
interfere with them significantly).

[mcasas, 2016-04-06 21:05:55]

Description was changed from

==========
MediaStreamRecorder: Separate video/audio origin of timestamps for WebM muxer

In certain conditions (see bug), the origin of
timestamps for audio and video is quite separated
which causes a jump on the WebM frame timestamps.
This seems to be an issue for certain players (chrome)
and not so much for others (vlc).

This CL solves the problem by keeping a different
origin-of-timestamps for audio and video separately,
and passing a TimeDelta to the AddFrame() method.

BUG=597034
TEST=On my Win7 laptop, navigate to [1], record
stream for a few seconds, then stop recording and
try playing back. With this CL, no freeze.

[1] https://webrtc.github.io/samples/src/content/getusermedia/record/
==========

to

==========
MediaStreamRecorder: Separate video/audio origin of timestamps for WebM muxer

In certain conditions (see bug), the origin of
timestamps for audio and video is quite separated
which causes a jump on the WebM frame timestamps.
This seems to be an issue for certain players (chrome)
and not so much for others (vlc).

This CL solves the problem by keeping a different
origin-of-timestamps for audio and video separately,
and passing a TimeDelta to the AddFrame() method.

See the bug for an investigation as to why this
started happening recently.

BUG=597034
TEST=On my Win7 laptop, navigate to [1], record
stream for a few seconds, then stop recording and
try playing back. With this CL, no freeze.

[1] https://webrtc.github.io/samples/src/content/getusermedia/record/
==========

[mcasas, 2016-04-06 21:06:02]

The CQ bit was checked by mcasas@chromium.org

[commit-bot: I haz the power, 2016-04-06 21:06:49]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1863893002/1
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1863893002/1

[mcasas, 2016-04-06 21:10:00]

On 2016/04/06 21:06:49, commit-bot: I haz the power wrote:
> CQ is trying da patch. Follow status at
>  https://chromium-cq-status.appspot.com/patch-status/1863893002/1
> View timeline at
>  https://chromium-cq-status.appspot.com/patch-timeline/1863893002/1

Please see the bug for an explanation of why
this desynchronization happens and why this
CL is a solution to it.

[commit-bot: I haz the power, 2016-04-07 00:35:11]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-04-07 00:35:12]

Exceeded global retry quota

[mcasas, 2016-04-07 16:02:01]

The CQ bit was checked by mcasas@chromium.org

[commit-bot: I haz the power, 2016-04-07 16:02:15]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1863893002/1
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1863893002/1

[commit-bot: I haz the power, 2016-04-07 17:09:51]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-04-07 17:09:52]

Try jobs failed on following builders:
  linux_chromium_chromeos_rel_ng on tryserver.chromium.linux (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...)

[mcasas, 2016-04-07 17:15:19]

The CQ bit was checked by mcasas@chromium.org

[commit-bot: I haz the power, 2016-04-07 17:15:41]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1863893002/1
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1863893002/1

[commit-bot: I haz the power, 2016-04-07 18:24:26]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-04-07 18:24:28]

Try jobs failed on following builders:
  linux_chromium_chromeos_rel_ng on tryserver.chromium.linux (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...)

[mcasas, 2016-04-07 21:38:49]

The CQ bit was checked by mcasas@chromium.org

[commit-bot: I haz the power, 2016-04-07 21:39:23]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1863893002/1
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1863893002/1

[commit-bot: I haz the power, 2016-04-07 22:15:58]

Description was changed from

==========
MediaStreamRecorder: Separate video/audio origin of timestamps for WebM muxer

In certain conditions (see bug), the origin of
timestamps for audio and video is quite separated
which causes a jump on the WebM frame timestamps.
This seems to be an issue for certain players (chrome)
and not so much for others (vlc).

This CL solves the problem by keeping a different
origin-of-timestamps for audio and video separately,
and passing a TimeDelta to the AddFrame() method.

See the bug for an investigation as to why this
started happening recently.

BUG=597034
TEST=On my Win7 laptop, navigate to [1], record
stream for a few seconds, then stop recording and
try playing back. With this CL, no freeze.

[1] https://webrtc.github.io/samples/src/content/getusermedia/record/
==========

to

==========
MediaStreamRecorder: Separate video/audio origin of timestamps for WebM muxer

In certain conditions (see bug), the origin of
timestamps for audio and video is quite separated
which causes a jump on the WebM frame timestamps.
This seems to be an issue for certain players (chrome)
and not so much for others (vlc).

This CL solves the problem by keeping a different
origin-of-timestamps for audio and video separately,
and passing a TimeDelta to the AddFrame() method.

See the bug for an investigation as to why this
started happening recently.

BUG=597034
TEST=On my Win7 laptop, navigate to [1], record
stream for a few seconds, then stop recording and
try playing back. With this CL, no freeze.

[1] https://webrtc.github.io/samples/src/content/getusermedia/record/
==========

[commit-bot: I haz the power, 2016-04-07 22:15:58]

Committed patchset #1 (id:1)

[commit-bot: I haz the power, 2016-04-07 22:18:12]

Description was changed from

==========
MediaStreamRecorder: Separate video/audio origin of timestamps for WebM muxer

In certain conditions (see bug), the origin of
timestamps for audio and video is quite separated
which causes a jump on the WebM frame timestamps.
This seems to be an issue for certain players (chrome)
and not so much for others (vlc).

This CL solves the problem by keeping a different
origin-of-timestamps for audio and video separately,
and passing a TimeDelta to the AddFrame() method.

See the bug for an investigation as to why this
started happening recently.

BUG=597034
TEST=On my Win7 laptop, navigate to [1], record
stream for a few seconds, then stop recording and
try playing back. With this CL, no freeze.

[1] https://webrtc.github.io/samples/src/content/getusermedia/record/
==========

to

==========
MediaStreamRecorder: Separate video/audio origin of timestamps for WebM muxer

In certain conditions (see bug), the origin of
timestamps for audio and video is quite separated
which causes a jump on the WebM frame timestamps.
This seems to be an issue for certain players (chrome)
and not so much for others (vlc).

This CL solves the problem by keeping a different
origin-of-timestamps for audio and video separately,
and passing a TimeDelta to the AddFrame() method.

See the bug for an investigation as to why this
started happening recently.

BUG=597034
TEST=On my Win7 laptop, navigate to [1], record
stream for a few seconds, then stop recording and
try playing back. With this CL, no freeze.

[1] https://webrtc.github.io/samples/src/content/getusermedia/record/

Committed: https://crrev.com/aa575cd9100984ba7cf7fc0377d7c5d40abbdb5f
Cr-Commit-Position: refs/heads/master@{#385899}
==========

[commit-bot: I haz the power, 2016-04-07 22:18:13]

Patchset 1 (id:??) landed as
https://crrev.com/aa575cd9100984ba7cf7fc0377d7c5d40abbdb5f
Cr-Commit-Position: refs/heads/master@{#385899}