Import MSE tests from web-platform-tests

Import MSE tests from web-platform-tests

Some of the upstream tests originate from http/tests/media/media-source/
and in the long run these should be consolidated. Importing is the first
step, and may reveal interop problems in need of attention.

BUG=624037, 633669, 651765
R=chcunningham@chromium.org,wolenetz@chromium.org

Review-Url: https://codereview.chromium.org/2830203003
Cr-Commit-Position: refs/heads/master@{#466370}
Committed: https://chromium.googlesource.com/chromium/src/+/0452bcf300c90ef3707f530cdec61c5397627cc9

[foolip, 2017-04-21 05:43:19]

The CQ bit was checked by foolip@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-04-21 05:43:34]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-04-21 07:14:47]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-04-21 07:14:48]

Dry run: This issue passed the CQ dry run.

[wolenetz, 2017-04-21 10:05:53]

I think Chrome will fail some of the wpt MSE tests that use MP4 due to need to
fix conflation of DTS for PTS in web app visible buffered range reporting. This
is a known group of issues that I'm working on now.

Detecting that group of failures may not happen if these tests are not run on a
Chrome build containing proprietary codecs (e.g. MP4).

Will this CL actually *do* the import? I note that the tests are not currently
in LayoutTests/external/wpt/media-source folder.

[wolenetz, 2017-04-21 10:20:45]

On 2017/04/21 10:05:53, wolenetz wrote:
> Will this CL actually *do* the import? I note that the tests are not currently
> in LayoutTests/external/wpt/media-source folder.

Note to self: yes, this CL will enable the import.

LGTM

[foolip, 2017-04-21 16:59:50]

On 2017/04/21 10:20:45, wolenetz wrote:
> On 2017/04/21 10:05:53, wolenetz wrote:
> > Will this CL actually *do* the import? I note that the tests are not
currently
> > in LayoutTests/external/wpt/media-source folder.
> 
> Note to self: yes, this CL will enable the import.

Yep, that's right, and the goal is that automatic imports run at least once
every 24 hours. Don't be surprised if you see a longer delay, but do poke
qyearsley@ if it becomes a hindrance.

In that automatic import, failing tests will automatically get *-expected.txt
files expecting failure, and from there it should be easier to work towards
getting more tests passing. See
https://chromium.googlesource.com/chromium/src/+/master/docs/testing/web_plat...
for more details on the whole 2-way sync setup.

[foolip, 2017-04-21 16:59:53]

The CQ bit was checked by foolip@chromium.org

[commit-bot: I haz the power, 2017-04-21 17:00:20]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-04-21 17:05:59]

CQ is committing da patch.

Bot data: {"patchset_id": 1, "attempt_start_ts": 1492793993896680, "parent_rev":
"8a9c23403a7f9840cf4d9cbc517535ad6b8de953", "commit_rev":
"0452bcf300c90ef3707f530cdec61c5397627cc9"}

[commit-bot: I haz the power, 2017-04-21 17:06:16]

Description was changed from

==========
Import MSE tests from web-platform-tests

Some of the upstream tests originate from http/tests/media/media-source/
and in the long run these should be consolidated. Importing is the first
step, and may reveal interop problems in need of attention.

BUG=624037,633669,651765
R=chcunningham@chromium.org,wolenetz@chromium.org
==========

to

==========
Import MSE tests from web-platform-tests

Some of the upstream tests originate from http/tests/media/media-source/
and in the long run these should be consolidated. Importing is the first
step, and may reveal interop problems in need of attention.

BUG=624037,633669,651765
R=chcunningham@chromium.org,wolenetz@chromium.org

Review-Url: https://codereview.chromium.org/2830203003
Cr-Commit-Position: refs/heads/master@{#466370}
Committed:
https://chromium.googlesource.com/chromium/src/+/0452bcf300c90ef3707f530cdec6...
==========

[commit-bot: I haz the power, 2017-04-21 17:06:22]

Committed patchset #1 (id:1) as
https://chromium.googlesource.com/chromium/src/+/0452bcf300c90ef3707f530cdec6...

[chcunningham, 2017-04-21 17:20:06]

On 2017/04/21 16:59:50, foolip_UTC7 wrote:
> In that automatic import, failing tests will automatically get *-expected.txt
> files expecting failure, and from there it should be easier to work towards
> getting more tests passing.


Will there be any alert/bug to go with auto-generated *-expected.txt files. Or
do we just manually monitor these as they accumulate?

[foolip, 2017-04-21 17:36:18]

On 2017/04/21 17:20:06, chcunningham wrote:
> On 2017/04/21 16:59:50, foolip_UTC7 wrote:
> > In that automatic import, failing tests will automatically get
*-expected.txt
> > files expecting failure, and from there it should be easier to work towards
> > getting more tests passing.
> 
> 
> Will there be any alert/bug to go with auto-generated *-expected.txt files. Or
> do we just manually monitor these as they accumulate?

qyearsley@, I tried to find the bug for this, but couldn't find one?

In any event, the current state is that new failures won't result in any email,
and that keeping the test suite in good shape is therefore a bit of a challenge.

I think that sending email or auto-filing bugs actually wouldn't be enough to
solve this problem, because it'd be a trickle of issues where it's easy to miss
some of them. Rather, some quick way of seeing the current status and what needs
addressing would be helpful. That'd mean flagging anything that has a
*-expected.txt file, anything being skipped in W3CImportExpectations, and
anything with entries in TestExpectations.

[qyearsley, 2017-04-21 18:12:53]

On 2017/04/21 at 17:36:18, foolip wrote:
> On 2017/04/21 17:20:06, chcunningham wrote:
> > On 2017/04/21 16:59:50, foolip_UTC7 wrote:
> > > In that automatic import, failing tests will automatically get
*-expected.txt
> > > files expecting failure, and from there it should be easier to work
towards
> > > getting more tests passing.
> > 
> > 
> > Will there be any alert/bug to go with auto-generated *-expected.txt files.
Or
> > do we just manually monitor these as they accumulate?
> 
> qyearsley@, I tried to find the bug for this, but couldn't find one?
> 
> In any event, the current state is that new failures won't result in any
email, and that keeping the test suite in good shape is therefore a bit of a
challenge.
> 
> I think that sending email or auto-filing bugs actually wouldn't be enough to
solve this problem, because it'd be a trickle of issues where it's easy to miss
some of them. Rather, some quick way of seeing the current status and what needs
addressing would be helpful. That'd mean flagging anything that has a
*-expected.txt file, anything being skipped in W3CImportExpectations, and
anything with entries in TestExpectations.

I think the most relevant bug is http://crbug.com/676672 ("Notify owners for new
failures in imported wpt directories").

There was some discussion before of filing bugs for newly imported failing
tests, although there are so many that this could be to noisy. There's also the
idea of having a bug for failing tests in each wpt directory, and updating those
bugs when new tests are failing.

[foolip, 2017-04-21 18:33:51]

On 2017/04/21 18:12:53, qyearsley wrote:
> On 2017/04/21 at 17:36:18, foolip wrote:
> > On 2017/04/21 17:20:06, chcunningham wrote:
> > > On 2017/04/21 16:59:50, foolip_UTC7 wrote:
> > > > In that automatic import, failing tests will automatically get
> *-expected.txt
> > > > files expecting failure, and from there it should be easier to work
> towards
> > > > getting more tests passing.
> > > 
> > > 
> > > Will there be any alert/bug to go with auto-generated *-expected.txt
files.
> Or
> > > do we just manually monitor these as they accumulate?
> > 
> > qyearsley@, I tried to find the bug for this, but couldn't find one?
> > 
> > In any event, the current state is that new failures won't result in any
> email, and that keeping the test suite in good shape is therefore a bit of a
> challenge.
> > 
> > I think that sending email or auto-filing bugs actually wouldn't be enough
to
> solve this problem, because it'd be a trickle of issues where it's easy to
miss
> some of them. Rather, some quick way of seeing the current status and what
needs
> addressing would be helpful. That'd mean flagging anything that has a
> *-expected.txt file, anything being skipped in W3CImportExpectations, and
> anything with entries in TestExpectations.
> 
> I think the most relevant bug is http://crbug.com/676672 ("Notify owners for
new
> failures in imported wpt directories").
> 
> There was some discussion before of filing bugs for newly imported failing
> tests, although there are so many that this could be to noisy. There's also
the
> idea of having a bug for failing tests in each wpt directory, and updating
those
> bugs when new tests are failing.

Ah thanks. I was looking for bugs assigned to you, that's why I couldn't find
it.

Reviewers, feedback on your preferred workflow for wpt would be much appreciated
here or anywhere.

[wolenetz, 2017-04-21 18:47:07]

On 2017/04/21 18:33:51, foolip_UTC7 wrote:
> On 2017/04/21 18:12:53, qyearsley wrote:
> > On 2017/04/21 at 17:36:18, foolip wrote:
> > > On 2017/04/21 17:20:06, chcunningham wrote:
> > > > On 2017/04/21 16:59:50, foolip_UTC7 wrote:
> > > > > In that automatic import, failing tests will automatically get
> > *-expected.txt
> > > > > files expecting failure, and from there it should be easier to work
> > towards
> > > > > getting more tests passing.
> > > > 
> > > > 
> > > > Will there be any alert/bug to go with auto-generated *-expected.txt
> files.
> > Or
> > > > do we just manually monitor these as they accumulate?
> > > 
> > > qyearsley@, I tried to find the bug for this, but couldn't find one?
> > > 
> > > In any event, the current state is that new failures won't result in any
> > email, and that keeping the test suite in good shape is therefore a bit of a
> > challenge.
> > > 
> > > I think that sending email or auto-filing bugs actually wouldn't be enough
> to
> > solve this problem, because it'd be a trickle of issues where it's easy to
> miss
> > some of them. Rather, some quick way of seeing the current status and what
> needs
> > addressing would be helpful. That'd mean flagging anything that has a
> > *-expected.txt file, anything being skipped in W3CImportExpectations, and
> > anything with entries in TestExpectations.
> > 
> > I think the most relevant bug is http://crbug.com/676672 ("Notify owners for
> new
> > failures in imported wpt directories").
> > 
> > There was some discussion before of filing bugs for newly imported failing
> > tests, although there are so many that this could be to noisy. There's also
> the
> > idea of having a bug for failing tests in each wpt directory, and updating
> those
> > bugs when new tests are failing.
> 
> Ah thanks. I was looking for bugs assigned to you, that's why I couldn't find
> it.
> 
> Reviewers, feedback on your preferred workflow for wpt would be much
appreciated
> here or anywhere.

To help wpt become the baseline for Blink LayoutTests, and reduce divergence, I
would prefer an auto-filer system (perhaps tag wpt suites with OWNERs that
receive such assigned bugs).
Yes, it introduces potential for noise from bad stuff upstream, but fixing that
noise helps the overall web platform.

[foolip, 2017-05-05 12:32:05]

On 2017/04/21 18:47:07, wolenetz wrote:
> On 2017/04/21 18:33:51, foolip_UTC7 wrote:
> > On 2017/04/21 18:12:53, qyearsley wrote:
> > > On 2017/04/21 at 17:36:18, foolip wrote:
> > > > On 2017/04/21 17:20:06, chcunningham wrote:
> > > > > On 2017/04/21 16:59:50, foolip_UTC7 wrote:
> > > > > > In that automatic import, failing tests will automatically get
> > > *-expected.txt
> > > > > > files expecting failure, and from there it should be easier to work
> > > towards
> > > > > > getting more tests passing.
> > > > > 
> > > > > 
> > > > > Will there be any alert/bug to go with auto-generated *-expected.txt
> > files.
> > > Or
> > > > > do we just manually monitor these as they accumulate?
> > > > 
> > > > qyearsley@, I tried to find the bug for this, but couldn't find one?
> > > > 
> > > > In any event, the current state is that new failures won't result in any
> > > email, and that keeping the test suite in good shape is therefore a bit of
a
> > > challenge.
> > > > 
> > > > I think that sending email or auto-filing bugs actually wouldn't be
enough
> > to
> > > solve this problem, because it'd be a trickle of issues where it's easy to
> > miss
> > > some of them. Rather, some quick way of seeing the current status and what
> > needs
> > > addressing would be helpful. That'd mean flagging anything that has a
> > > *-expected.txt file, anything being skipped in W3CImportExpectations, and
> > > anything with entries in TestExpectations.
> > > 
> > > I think the most relevant bug is http://crbug.com/676672 ("Notify owners
for
> > new
> > > failures in imported wpt directories").
> > > 
> > > There was some discussion before of filing bugs for newly imported failing
> > > tests, although there are so many that this could be to noisy. There's
also
> > the
> > > idea of having a bug for failing tests in each wpt directory, and updating
> > those
> > > bugs when new tests are failing.
> > 
> > Ah thanks. I was looking for bugs assigned to you, that's why I couldn't
find
> > it.
> > 
> > Reviewers, feedback on your preferred workflow for wpt would be much
> appreciated
> > here or anywhere.
> 
> To help wpt become the baseline for Blink LayoutTests, and reduce divergence,
I
> would prefer an auto-filer system (perhaps tag wpt suites with OWNERs that
> receive such assigned bugs).
> Yes, it introduces potential for noise from bad stuff upstream, but fixing
that
> noise helps the overall web platform.

I've commented on https://crbug.com/676672 , will see where that goes.