WPTServe: Run .any.js and .worker.js variations

WPTServe: Run .any.js and .worker.js variations

When .any.js and .worker.js files are seen during MANIFEST generation
they are mapped to .any.html, .any.worker.html, and .worker.html files
as appropriate which WPTServe will generate on-the-fly. This allows
for easy testing of [Exposed=(Window,Worker)] APIs and tests that are
simply script with no boilerplate HTML. These are now supported in
run-webkit-tests.

When specifying tests to run, the source name should be specified, 
for example:

  $ run-webkit-test external/wpt/storage/interfaces.worker.js

But expectations will use the generated name, e.g.:

  external/wpt/storage/interfaces.worker-expected.txt

And failures will be reported under the generated name:

  external/wpt/storage/interfaces.worker.html

BUG=653514
R=qyearsley@chromium.org

[jsbell, 2017-03-21 18:30:29]

Running locally, failures are in two classes:

`*.worker.html` files which are now being run - yay! Most of these seem like
legitimate issues, e.g. ErrorEvent is missing [Exposed=(Window,Worker)]

Files in html/syntax/parsing/html5lib_*' with the suffix `?run_type=XXX` where
XXX is 'uri', 'write', or 'write_single'. I haven't seen these before. These
would have been filtered out before. I guess we should continue filtering these
for now?

[jsbell, 2017-03-21 18:38:23]

On 2017/03/21 18:30:29, jsbell wrote:
> Files in html/syntax/parsing/html5lib_*' with the suffix `?run_type=XXX` where
> XXX is 'uri', 'write', or 'write_single'. I haven't seen these before. These
> would have been filtered out before. I guess we should continue filtering
these
> for now?

FYI, those are from: <meta name="variant" content="?run_type=uri">

And it seems like my pessimism was premature - we seem to pass most of them!

[jsbell, 2017-03-21 19:08:23]

Description was changed from

==========
WPTServe: Run .any.js and .worker.js variations

When .any.js and .worker.js files are seen during MANIFEST generation
they are mapped to .any.html, .any.worker.html, and .worker.html files
as appropriate which WPTServe will generate on-the-fly. This allows
for easy testing of [Exposed=(Window,Worker)] APIs and tests that are
simply script with no boilerplate HTML.

BUG=653514
R=qyearsley@chromium.org
==========

to

==========
WPTServe: Run .any.js and .worker.js variations

When .any.js and .worker.js files are seen during MANIFEST generation
they are mapped to .any.html, .any.worker.html, and .worker.html files
as appropriate which WPTServe will generate on-the-fly. This allows
for easy testing of [Exposed=(Window,Worker)] APIs and tests that are
simply script with no boilerplate HTML. These are now supported in
run-webkit-tests.

When specifying tests to run, the source name should be specified, e.g.
run-webkit-test external/wpt/storage/interfaces.worker.js but expectations and
results will use the generated name, e.g.
external/wpt/storage/interfaces.worker-expected.txt and
external/wpt/storage/interfaces.worker.html.

BUG=653514
R=qyearsley@chromium.org
==========

[jsbell, 2017-03-21 19:11:16]

Description was changed from

==========
WPTServe: Run .any.js and .worker.js variations

When .any.js and .worker.js files are seen during MANIFEST generation
they are mapped to .any.html, .any.worker.html, and .worker.html files
as appropriate which WPTServe will generate on-the-fly. This allows
for easy testing of [Exposed=(Window,Worker)] APIs and tests that are
simply script with no boilerplate HTML. These are now supported in
run-webkit-tests.

When specifying tests to run, the source name should be specified, e.g.
run-webkit-test external/wpt/storage/interfaces.worker.js but expectations and
results will use the generated name, e.g.
external/wpt/storage/interfaces.worker-expected.txt and
external/wpt/storage/interfaces.worker.html.

BUG=653514
R=qyearsley@chromium.org
==========

to

==========
WPTServe: Run .any.js and .worker.js variations

When .any.js and .worker.js files are seen during MANIFEST generation
they are mapped to .any.html, .any.worker.html, and .worker.html files
as appropriate which WPTServe will generate on-the-fly. This allows
for easy testing of [Exposed=(Window,Worker)] APIs and tests that are
simply script with no boilerplate HTML. These are now supported in
run-webkit-tests.

When specifying tests to run, the source name should be specified, 
for example:

  $ run-webkit-test external/wpt/storage/interfaces.worker.js

But expectations will use the generated name, e.g.:

  external/wpt/storage/interfaces.worker-expected.txt

And failures will be reported under the generated name:

  external/wpt/storage/interfaces.worker.html

BUG=653514
R=qyearsley@chromium.org
==========

[jsbell, 2017-03-21 19:16:40]

jsbell@chromium.org changed reviewers:
+ jeffcarp@chromium.org

[jsbell, 2017-03-21 19:16:41]

This should work and pass, with one exception... which is a tricky one:

external/wpt/workers/nested_worker.worker.js

This test times out because we don't support nested workers (crbug.com/31666). 

* You can't [Skip] it in TestExpectations as .../nested_worker.worker.js because
the name doesn't match

* You can't [Skip] it in TestExpectations as .../nested_worker.worker.html
because it fails the lint test as that file does not exist

We could [Skip] in W3CImportExpectations, but I believe the above limitations
mean any new failing .worker.js or .any.js would cause the autoroller to fail.

[jsbell, 2017-03-21 21:00:52]

Failures from the try run:

external/wpt/html/dom/interfaces.worker.html - flaky? maybe I messed up the
baseline?
https://storage.googleapis.com/chromium-layout-test-archives/linux_chromium_r...

external/wpt/workers/nested_worker.worker.html - as expected

[jeffcarp, 2017-03-21 21:16:17]

On 2017/03/21 at 21:00:52, jsbell wrote:
> Failures from the try run:
> 
> external/wpt/html/dom/interfaces.worker.html - flaky? maybe I messed up the
baseline?
https://storage.googleapis.com/chromium-layout-test-archives/linux_chromium_r...
> 
> external/wpt/workers/nested_worker.worker.html - as expected

My only idea to get around this is to add special logic to
layout_tests.models.test_expectations.TestExpectationParser._lint_line for
*.any.html and *.worker.html tests (solving problem #2 above):
https://cs.chromium.org/chromium/src/third_party/WebKit/Tools/Scripts/webkitp...

[jeffcarp, 2017-03-21 21:18:49]

On 2017/03/21 at 21:16:17, jeffcarp wrote:
> On 2017/03/21 at 21:00:52, jsbell wrote:
> > Failures from the try run:
> > 
> > external/wpt/html/dom/interfaces.worker.html - flaky? maybe I messed up the
baseline?
https://storage.googleapis.com/chromium-layout-test-archives/linux_chromium_r...
> > 
> > external/wpt/workers/nested_worker.worker.html - as expected
> 
> My only idea to get around this is to add special logic to
layout_tests.models.test_expectations.TestExpectationParser._lint_line for
*.any.html and *.worker.html tests (solving problem #2 above):
>
https://cs.chromium.org/chromium/src/third_party/WebKit/Tools/Scripts/webkitp...

Great, qyearsley suggested a better alternative to check against MANIFEST.json
in the lint: https://bugs.chromium.org/p/chromium/issues/detail?id=653514#c16

[qyearsley, 2017-03-21 21:19:35]

On 2017/03/21 at 21:16:17, jeffcarp wrote:
> On 2017/03/21 at 21:00:52, jsbell wrote:
> > Failures from the try run:
> > 
> > external/wpt/html/dom/interfaces.worker.html - flaky? maybe I messed up the
baseline?
https://storage.googleapis.com/chromium-layout-test-archives/linux_chromium_r...
> > 
> > external/wpt/workers/nested_worker.worker.html - as expected
> 
> My only idea to get around this is to add special logic to
layout_tests.models.test_expectations.TestExpectationParser._lint_line for
*.any.html and *.worker.html tests (solving problem #2 above):
>
https://cs.chromium.org/chromium/src/third_party/WebKit/Tools/Scripts/webkitp...

Hopefully, I don't think there needs to be special logic for ".any.html" and
".worker.html" in particular... but I think maybe we'll need to change the
semantics so that a "test" or "test path" is not the same as a "file" or "file
path" (more discussion in https://crbug.com/653514#c16).