More correct TestExpectations

Our old TestExpectations file had problems. I like to know the effect CRs have on our test coverage. The problems prevented this:

1) 125 passing tests were marked as failing.
 
With 125 stealth passing tests, I would not know what different my CR would make.

2) [Crash/Failing] tests were all marked as [Skip]

Because [Skip] tests are ignored by bots, any changes in status to skipped tests
had to be manually checked by developer.

This patch correctly marks all virtual/layout_ng tests:
- all passing tests are no longer skipped.
- all failing tests are explicitly marked as such:

[ Failure ] means test has failed
[ Crashed ] means that bad thing
[ Skip ] means test has missing expectations, and cannot be run

I've cleared with dpranke and quearsley that we can mark our tests as such.

I've also created an html page that can be run inside test-layout-results that generates these TestExpectations automatically. We will all be able to use it to automate updating of TestExpectations. This will come in a separate patch.

BUG=635619
Review-Url: https://codereview.chromium.org/2905933002
Cr-Commit-Position: refs/heads/master@{#475164}
Committed: https://chromium.googlesource.com/chromium/src/+/8ae2026e07808d63a1722123f46a5d33e05064a0

[atotic, 2017-05-25 21:44:20]

Description was changed from

==========
More correct TestExpectations

BUG=635619
==========

to

==========
Our old TestExpectations file had problems. I like to know the effect CRs have
on our test coverage. The problems prevented this:

1) 125 passing tests were marked as failing.
 
With 125 stealth passing tests, I would not know what different my CR would
make.

2) [Crash/Failing] tests were all marked as [Skip]

Because [Skip] tests are ignored by bots, any changes in status to skipped tests
had to be manually checked by developer.

This patch correctly marks all virtual/layout_ng tests:
- all passing tests are no longer skipped.
- all failing tests are explicitly marked as such:

[ Failure ] means test has failed
[ Crashed ] means that bad thing
[ Skip ] means test has missing expectations, and cannot be run

I've cleared with dpranke and quearsley that we can mark our tests as such.

I've also created an html page that can be run inside test-layout-results that
generates these TestExpectations automatically. We will all be able to use it to
automate updating of TestExpectations. This will come in a separate patch.

BUG=635619
==========

[atotic, 2017-05-25 21:44:20]

atotic@chromium.org changed reviewers:
+ cbiesinger@chromium.org, glebl@chromium.org, ikilpatrick@chromium.org

[atotic, 2017-05-25 21:56:58]

ptal

[Gleb Lanbin, 2017-05-26 00:15:44]

The CQ bit was checked by glebl@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-05-26 00:16:27]

Dry run: CQ is trying da patch.

Follow status at:
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[ikilpatrick, 2017-05-26 16:39:25]

lgtm

[atotic, 2017-05-26 16:40:15]

The CQ bit was checked by atotic@chromium.org

[commit-bot: I haz the power, 2017-05-26 16:40:50]

CQ is trying da patch.

Follow status at:
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[cbiesinger, 2017-05-26 16:41:54]

lgtm

https://codereview.chromium.org/2905933002/diff/20001/third_party/WebKit/Layo...
File third_party/WebKit/LayoutTests/TestExpectations (right):

https://codereview.chromium.org/2905933002/diff/20001/third_party/WebKit/Layo...
third_party/WebKit/LayoutTests/TestExpectations:276: ### flakey expectations
cannot be generated automatically
flakey -> flaky

[commit-bot: I haz the power, 2017-05-26 18:48:27]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-05-26 18:48:28]

Try jobs failed on following builders:
  mac_chromium_rel_ng on master.tryserver.chromium.mac (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_...)

[atotic, 2017-05-26 20:58:25]

The CQ bit was checked by atotic@chromium.org

[atotic, 2017-05-26 20:58:25]

The patchset sent to the CQ was uploaded after l-g-t-m from
cbiesinger@chromium.org, ikilpatrick@chromium.org
Link to the patchset: https://codereview.chromium.org/2905933002/#ps40001
(title: "Added Mac colorspace/font failures")

[commit-bot: I haz the power, 2017-05-26 20:59:21]

CQ is trying da patch.

Follow status at:
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-05-26 22:22:45]

CQ is committing da patch.

Bot data: {"patchset_id": 40001, "attempt_start_ts": 1495832305182870,
"parent_rev": "293e93eb3bcfbcc23fc6a1c644a624a1ca0d4bdc", "commit_rev":
"8ae2026e07808d63a1722123f46a5d33e05064a0"}

[commit-bot: I haz the power, 2017-05-26 22:22:57]

Description was changed from

==========
Our old TestExpectations file had problems. I like to know the effect CRs have
on our test coverage. The problems prevented this:

1) 125 passing tests were marked as failing.
 
With 125 stealth passing tests, I would not know what different my CR would
make.

2) [Crash/Failing] tests were all marked as [Skip]

Because [Skip] tests are ignored by bots, any changes in status to skipped tests
had to be manually checked by developer.

This patch correctly marks all virtual/layout_ng tests:
- all passing tests are no longer skipped.
- all failing tests are explicitly marked as such:

[ Failure ] means test has failed
[ Crashed ] means that bad thing
[ Skip ] means test has missing expectations, and cannot be run

I've cleared with dpranke and quearsley that we can mark our tests as such.

I've also created an html page that can be run inside test-layout-results that
generates these TestExpectations automatically. We will all be able to use it to
automate updating of TestExpectations. This will come in a separate patch.

BUG=635619
==========

to

==========
Our old TestExpectations file had problems. I like to know the effect CRs have
on our test coverage. The problems prevented this:

1) 125 passing tests were marked as failing.
 
With 125 stealth passing tests, I would not know what different my CR would
make.

2) [Crash/Failing] tests were all marked as [Skip]

Because [Skip] tests are ignored by bots, any changes in status to skipped tests
had to be manually checked by developer.

This patch correctly marks all virtual/layout_ng tests:
- all passing tests are no longer skipped.
- all failing tests are explicitly marked as such:

[ Failure ] means test has failed
[ Crashed ] means that bad thing
[ Skip ] means test has missing expectations, and cannot be run

I've cleared with dpranke and quearsley that we can mark our tests as such.

I've also created an html page that can be run inside test-layout-results that
generates these TestExpectations automatically. We will all be able to use it to
automate updating of TestExpectations. This will come in a separate patch.

BUG=635619

Review-Url: https://codereview.chromium.org/2905933002
Cr-Commit-Position: refs/heads/master@{#475164}
Committed:
https://chromium.googlesource.com/chromium/src/+/8ae2026e07808d63a1722123f46a...
==========

[commit-bot: I haz the power, 2017-05-26 22:22:59]

Committed patchset #3 (id:40001) as
https://chromium.googlesource.com/chromium/src/+/8ae2026e07808d63a1722123f46a...