When rebaselining testharness tests with new all-PASS results, remove the baseline.

When rebaselining testharness tests with new all-PASS results, remove the baseline.

For testharness.js tests where the result just contains PASS lines with no
console errors or warnings, our way of representing that is to have no baseline
(no -expected.txt file). So, when rebaselining, the proper "baseline" for tests
like this is to have no baseline.

Example case for how I think this might work: Let's imagine there's a
testharness test which fails on mac and linux, but passes on windows. The
existing baselines may be:

    LayoutTests/platform/mac/foo/bar-expected.txt
    LayoutTests/platform/linux/foo/bar-expected.txt

If we rebaseline and it now passes on linux, then I imagine
that the copy-existing-baselines baselines step would copy
linux/foo/bar-expected.txt to linux-precise/, and the
baselines in both linux/ and linux-precise/ would be deleted
on the rebaseline-test step. Then, we'd be left with just
LayoutTests/platform/mac/foo/bar-expected.txt.

Although I haven't really tested out this whole flow yet.

This CL also extracts a function import is_all_pass_testharness_result from
webkitpy.layout_tests.models.testharness_results to avoid duplicating code.

BUG=654030
Committed: https://crrev.com/0bb11a3670a64f56774881350f8f4b0035ba2eeb
Cr-Commit-Position: refs/heads/master@{#424509}

[qyearsley, 2016-10-11 00:24:29]

qyearsley@chromium.org changed reviewers:
+ dpranke@chromium.org, wkorman@chromium.org

[Dirk Pranke, 2016-10-11 00:30:50]

lgtm, but you have some typos in your CL description: the second paragraph ends
abruptly, and there is no platform/linux-trusty (trusty uses platform/linux).

[qyearsley, 2016-10-11 03:10:38]

Description was changed from

==========
When rebaselining testharness tests with new all-PASS results, remove the
baseline.

For testharness.js tests where the result just contains PASS lines with no
console errors or warnings, our way of representing that is to have no baseline
(no -expected.txt file). So, when rebaselining, the proper "baseline" for tests
like this is to

Example case for how I think this might work: Let's imagine there's a
testharness test which fails on mac and linux, but passes on windows. The
existing baselines may be:

    LayoutTests/platform/mac/foo/bar-expected.txt
    LayoutTests/platform/linux/foo/bar-expected.txt

If we rebaseline and it now passes on linux, then I imagine
that the copy-existing-baselines baselines step would copy
linux/foo/bar-expected.txt into the more specific linux-precise
and linux-trusty locations, which would then be deleted
on the rebaseline-test step. Then, we'd be left with just
LayoutTests/platform/mac/foo/bar-expected.txt, which would be correct.

Although I haven't really tested out this whole flow yet.

This CL also extracts a function import is_all_pass_testharness_result from
webkitpy.layout_tests.models.testharness_results to avoid duplicating code.

BUG=654030
==========

to

==========
When rebaselining testharness tests with new all-PASS results, remove the
baseline.

For testharness.js tests where the result just contains PASS lines with no
console errors or warnings, our way of representing that is to have no baseline
(no -expected.txt file). So, when rebaselining, the proper "baseline" for tests
like this is to have no baseline.

Example case for how I think this might work: Let's imagine there's a
testharness test which fails on mac and linux, but passes on windows. The
existing baselines may be:

    LayoutTests/platform/mac/foo/bar-expected.txt
    LayoutTests/platform/linux/foo/bar-expected.txt

If we rebaseline and it now passes on linux, then I imagine
that the copy-existing-baselines baselines step would copy
linux/foo/bar-expected.txt into the more specific linux-precise
and linux-trusty locations, which would then be deleted
on the rebaseline-test step. Then, we'd be left with just
LayoutTests/platform/mac/foo/bar-expected.txt, which would be correct.

Although I haven't really tested out this whole flow yet.

This CL also extracts a function import is_all_pass_testharness_result from
webkitpy.layout_tests.models.testharness_results to avoid duplicating code.

BUG=654030
==========

[qyearsley, 2016-10-11 03:50:02]

Description was changed from

==========
When rebaselining testharness tests with new all-PASS results, remove the
baseline.

For testharness.js tests where the result just contains PASS lines with no
console errors or warnings, our way of representing that is to have no baseline
(no -expected.txt file). So, when rebaselining, the proper "baseline" for tests
like this is to have no baseline.

Example case for how I think this might work: Let's imagine there's a
testharness test which fails on mac and linux, but passes on windows. The
existing baselines may be:

    LayoutTests/platform/mac/foo/bar-expected.txt
    LayoutTests/platform/linux/foo/bar-expected.txt

If we rebaseline and it now passes on linux, then I imagine
that the copy-existing-baselines baselines step would copy
linux/foo/bar-expected.txt into the more specific linux-precise
and linux-trusty locations, which would then be deleted
on the rebaseline-test step. Then, we'd be left with just
LayoutTests/platform/mac/foo/bar-expected.txt, which would be correct.

Although I haven't really tested out this whole flow yet.

This CL also extracts a function import is_all_pass_testharness_result from
webkitpy.layout_tests.models.testharness_results to avoid duplicating code.

BUG=654030
==========

to

==========
When rebaselining testharness tests with new all-PASS results, remove the
baseline.

For testharness.js tests where the result just contains PASS lines with no
console errors or warnings, our way of representing that is to have no baseline
(no -expected.txt file). So, when rebaselining, the proper "baseline" for tests
like this is to have no baseline.

Example case for how I think this might work: Let's imagine there's a
testharness test which fails on mac and linux, but passes on windows. The
existing baselines may be:

    LayoutTests/platform/mac/foo/bar-expected.txt
    LayoutTests/platform/linux/foo/bar-expected.txt

If we rebaseline and it now passes on linux, then I imagine
that the copy-existing-baselines baselines step would copy
linux/foo/bar-expected.txt to linux-precise/, and the
baselines in both linux/ and linux-precise/ would be deleted
on the rebaseline-test step. Then, we'd be left with just
LayoutTests/platform/mac/foo/bar-expected.txt.

Although I haven't really tested out this whole flow yet.

This CL also extracts a function import is_all_pass_testharness_result from
webkitpy.layout_tests.models.testharness_results to avoid duplicating code.

BUG=654030
==========

[qyearsley, 2016-10-11 03:54:04]

On 2016/10/11 at 00:30:50, dpranke wrote:
> lgtm, but you have some typos in your CL description: the second paragraph
ends abruptly, and there is no platform/linux-trusty (trusty uses
platform/linux).

I really need to start reading through things before I post :-/

Also, I think I don't yet properly understand how the baseline fallback graph
looks; so now I'm going to take
a closer look and try to write them out:

  <version>: <baseline directory fallback path>
  mac10.11: mac/ -> generic
  mac10.10: mac-mac10.10/ -> mac/ -> generic
  mac10.9: mac-mac10.9/ -> mac-mac10.10/ -> mac/ -> generic
  retina: mac-retina/ -> mac/ -> generic
  win10: win/ -> generic
  win7: win7/ -> win/ -> generic
  trusty: linux -> win/ -> generic
  precise: linux-precise/ -> linux/ -> win/ -> generic
  icecreamsandwich: android/ -> linux/ -> win/ -> generic

So, there are now nine "versions" (icecreamsandwich, precise, trusty, win7,
win10, mac10.9, mac10.10, mac10.11, retina), and there are also exactly nine
non-generic directories (android, linux-precise, linux, win7, win, mac-mac10.9,
mac-mac10.10, mac, retina); each of these directories is the
"primary"/"first-choice" directory for some version. Is that right?

Also, is there some important reason why linux platforms fall back to win/
before falling back to the generic baseline?

[qyearsley, 2016-10-11 18:05:22]

The CQ bit was checked by qyearsley@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2016-10-11 18:05:49]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[Dirk Pranke, 2016-10-11 18:18:12]

On 2016/10/11 03:54:04, qyearsley wrote:
> On 2016/10/11 at 00:30:50, dpranke wrote:
> > lgtm, but you have some typos in your CL description: the second paragraph
> ends abruptly, and there is no platform/linux-trusty (trusty uses
> platform/linux).
> 
> I really need to start reading through things before I post :-/
> 
> Also, I think I don't yet properly understand how the baseline fallback graph
> looks; so now I'm going to take
> a closer look and try to write them out:
> 
>   <version>: <baseline directory fallback path>
>   mac10.11: mac/ -> generic
>   mac10.10: mac-mac10.10/ -> mac/ -> generic
>   mac10.9: mac-mac10.9/ -> mac-mac10.10/ -> mac/ -> generic
>   retina: mac-retina/ -> mac/ -> generic
>   win10: win/ -> generic
>   win7: win7/ -> win/ -> generic
>   trusty: linux -> win/ -> generic
>   precise: linux-precise/ -> linux/ -> win/ -> generic
>   icecreamsandwich: android/ -> linux/ -> win/ -> generic
> 
> So, there are now nine "versions" (icecreamsandwich, precise, trusty, win7,
> win10, mac10.9, mac10.10, mac10.11, retina), and there are also exactly nine
> non-generic directories (android, linux-precise, linux, win7, win,
mac-mac10.9,
> mac-mac10.10, mac, retina); each of these directories is the
> "primary"/"first-choice" directory for some version. Is that right?

Yes. 

There's a now-out-of-date actual graph of this here:

https://docs.google.com/drawings/d/1KBTihR80H42GB0be0qK2CyM-pPUoMdnHqYaOsNK85...

But the concept is the same. Quite some time ago -- before Blink even existed --
I wrote down 
why things work the way they do:

http://trac.webkit.org/wiki/LayoutTestsSearchPath

It might be add some equivalent of this doc to the current docs somewhere.

> 
> Also, is there some important reason why linux platforms fall back to win/
> before falling back to the generic baseline?

In theory this allowed us to share a lot of baselines, because we tried to have
linux match
the windows results. I'm not sure if this is still true, though; it's easy
enough to check by
seeing how many tests on linux actually use the win baseline.

[qyearsley, 2016-10-11 18:48:14]

On 2016/10/11 at 18:18:12, dpranke wrote:
> On 2016/10/11 03:54:04, qyearsley wrote:
> > On 2016/10/11 at 00:30:50, dpranke wrote:
> > > lgtm, but you have some typos in your CL description: the second paragraph
> > ends abruptly, and there is no platform/linux-trusty (trusty uses
> > platform/linux).
> > 
> > I really need to start reading through things before I post :-/
> > 
> > Also, I think I don't yet properly understand how the baseline fallback
graph
> > looks; so now I'm going to take
> > a closer look and try to write them out:
> > 
> >   <version>: <baseline directory fallback path>
> >   mac10.11: mac/ -> generic
> >   mac10.10: mac-mac10.10/ -> mac/ -> generic
> >   mac10.9: mac-mac10.9/ -> mac-mac10.10/ -> mac/ -> generic
> >   retina: mac-retina/ -> mac/ -> generic
> >   win10: win/ -> generic
> >   win7: win7/ -> win/ -> generic
> >   trusty: linux -> win/ -> generic
> >   precise: linux-precise/ -> linux/ -> win/ -> generic
> >   icecreamsandwich: android/ -> linux/ -> win/ -> generic
> > 
> > So, there are now nine "versions" (icecreamsandwich, precise, trusty, win7,
> > win10, mac10.9, mac10.10, mac10.11, retina), and there are also exactly nine
> > non-generic directories (android, linux-precise, linux, win7, win,
mac-mac10.9,
> > mac-mac10.10, mac, retina); each of these directories is the
> > "primary"/"first-choice" directory for some version. Is that right?
> 
> Yes. 
> 
> There's a now-out-of-date actual graph of this here:
> 
>
https://docs.google.com/drawings/d/1KBTihR80H42GB0be0qK2CyM-pPUoMdnHqYaOsNK85...
> 
> But the concept is the same. Quite some time ago -- before Blink even existed
-- I wrote down 
> why things work the way they do:
> 
> http://trac.webkit.org/wiki/LayoutTestsSearchPath
> 
> It might be add some equivalent of this doc to the current docs somewhere.

Yeah -- for now, another page under
https://www.chromium.org/developers/testing/webkit-layout-tests/ (like
https://www.chromium.org/developers/testing/webkit-layout-tests/testexpectations)
might be a good place to put this. (I plan to add this later today.)

> > Also, is there some important reason why linux platforms fall back to win/
> > before falling back to the generic baseline?
> 
> In theory this allowed us to share a lot of baselines, because we tried to
have linux match
> the windows results. I'm not sure if this is still true, though; it's easy
enough to check by
> seeing how many tests on linux actually use the win baseline.

Makes sense :-)

[qyearsley, 2016-10-11 18:48:21]

The CQ bit was unchecked by qyearsley@chromium.org

[qyearsley, 2016-10-11 18:48:22]

The CQ bit was checked by qyearsley@chromium.org

[commit-bot: I haz the power, 2016-10-11 18:48:37]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-10-11 19:21:32]

Committed patchset #3 (id:40001)

[commit-bot: I haz the power, 2016-10-11 19:23:06]

Description was changed from

==========
When rebaselining testharness tests with new all-PASS results, remove the
baseline.

For testharness.js tests where the result just contains PASS lines with no
console errors or warnings, our way of representing that is to have no baseline
(no -expected.txt file). So, when rebaselining, the proper "baseline" for tests
like this is to have no baseline.

Example case for how I think this might work: Let's imagine there's a
testharness test which fails on mac and linux, but passes on windows. The
existing baselines may be:

    LayoutTests/platform/mac/foo/bar-expected.txt
    LayoutTests/platform/linux/foo/bar-expected.txt

If we rebaseline and it now passes on linux, then I imagine
that the copy-existing-baselines baselines step would copy
linux/foo/bar-expected.txt to linux-precise/, and the
baselines in both linux/ and linux-precise/ would be deleted
on the rebaseline-test step. Then, we'd be left with just
LayoutTests/platform/mac/foo/bar-expected.txt.

Although I haven't really tested out this whole flow yet.

This CL also extracts a function import is_all_pass_testharness_result from
webkitpy.layout_tests.models.testharness_results to avoid duplicating code.

BUG=654030
==========

to

==========
When rebaselining testharness tests with new all-PASS results, remove the
baseline.

For testharness.js tests where the result just contains PASS lines with no
console errors or warnings, our way of representing that is to have no baseline
(no -expected.txt file). So, when rebaselining, the proper "baseline" for tests
like this is to have no baseline.

Example case for how I think this might work: Let's imagine there's a
testharness test which fails on mac and linux, but passes on windows. The
existing baselines may be:

    LayoutTests/platform/mac/foo/bar-expected.txt
    LayoutTests/platform/linux/foo/bar-expected.txt

If we rebaseline and it now passes on linux, then I imagine
that the copy-existing-baselines baselines step would copy
linux/foo/bar-expected.txt to linux-precise/, and the
baselines in both linux/ and linux-precise/ would be deleted
on the rebaseline-test step. Then, we'd be left with just
LayoutTests/platform/mac/foo/bar-expected.txt.

Although I haven't really tested out this whole flow yet.

This CL also extracts a function import is_all_pass_testharness_result from
webkitpy.layout_tests.models.testharness_results to avoid duplicating code.

BUG=654030

Committed: https://crrev.com/0bb11a3670a64f56774881350f8f4b0035ba2eeb
Cr-Commit-Position: refs/heads/master@{#424509}
==========

[commit-bot: I haz the power, 2016-10-11 19:23:07]

Patchset 3 (id:??) landed as
https://crrev.com/0bb11a3670a64f56774881350f8f4b0035ba2eeb
Cr-Commit-Position: refs/heads/master@{#424509}