Fix testharness result parsing, make "Harness error" count as failure.

Fix testharness result parsing, make "Harness error" count as failure.

BUG=703331, 705125
Review-Url: https://codereview.chromium.org/2766823003
Cr-Commit-Position: refs/heads/master@{#459924}
Committed: https://chromium.googlesource.com/chromium/src/+/13c19c95a727c5dbb003bcc089e6089bdcbe96aa

[qyearsley, 2017-03-21 22:18:11]

qyearsley@chromium.org changed reviewers:
+ jeffcarp@chromium.org

[qyearsley, 2017-03-21 22:18:11]

https://codereview.chromium.org/2766823003/diff/1/third_party/WebKit/Tools/Sc...
File
third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/models/testharness_results.py
(left):

https://codereview.chromium.org/2766823003/diff/1/third_party/WebKit/Tools/Sc...
third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/models/testharness_results.py:56:
line.startswith('Harness Error. harness_status = ')):
This turned out to be a very minor error. This was looking for "harness_status",
but in reality when there is a harness error it appears to actually say
"harness_status.status"...

[qyearsley, 2017-03-21 22:19:53]

Note, I'm expecting that there may need to be several -expected.txt files added
for tests that have "Harness error" but no "FAIL". Running try jobs to see if
this is the case.

[Raymond Toy, 2017-03-21 22:26:24]

On 2017/03/21 22:19:53, qyearsley wrote:
> Note, I'm expecting that there may need to be several -expected.txt files
added
> for tests that have "Harness error" but no "FAIL". Running try jobs to see if
> this is the case.

Just ran this with the webaudio tests.  49 tests now fail.  Probably because of
duplicate test names.

[jeffcarp, 2017-03-22 22:53:28]

lgtm

[qyearsley, 2017-03-22 23:08:40]

The CQ bit was checked by qyearsley@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-03-22 23:09:15]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-03-23 00:30:19]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-03-23 00:30:20]

Dry run: Try jobs failed on following builders:
  win_chromium_rel_ng on master.tryserver.chromium.win (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_...)

[qyearsley, 2017-03-23 01:28:21]

Looks like a lot of tests had testharness errors! About ~200 tests, including
tests in:

accessibility/
animations/
css3/
custom-elements/
editing/
external/csswg-test/
external/wpt/html/
external/wpt/shadow-dom/
external/wpt/streams/
fast/alignment/
fast/canvas/
fast/css/
fast/dom/
fast/...
http/tests/budget/
http/tests/credentialmanager/
http/tests/dom/
http/tests/fetch/
http/tests/media/
http/tests/origin_trials/
http/tests/security/
http/tests/serviceworker/
http/tests/shapedetection/
http/tests/streams/
http/tests/w3c/webperf/http/tests/webaudio/
http/tests/xmlhttprequest/
intersection-observer/media/
payments/
shadow-dom/
shapedetection/
svg/
transforms/
vibration/

Next step: Add text baselines for these tests in this CL, and try to let people
know by mentioning this on blink-dev.

[qyearsley, 2017-03-24 20:41:30]

The CQ bit was checked by qyearsley@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-03-24 20:41:55]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-03-24 21:37:35]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-03-24 21:37:36]

Dry run: Try jobs failed on following builders:
  linux_chromium_rel_ng on master.tryserver.chromium.linux (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...)

[qyearsley, 2017-03-24 22:46:07]

Description was changed from

==========
Fix testharness result parsing, make "Harness error" count as failure.

BUG=703331
==========

to

==========
Fix testharness result parsing, make "Harness error" count as failure.

BUG=703331,705125
==========

[qyearsley, 2017-03-25 23:05:35]

The CQ bit was checked by qyearsley@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-03-25 23:05:43]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-03-26 01:33:25]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-03-26 01:33:26]

Dry run: Try jobs failed on following builders:
  win_chromium_rel_ng on master.tryserver.chromium.win (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_...)

[qyearsley, 2017-03-27 16:56:37]

The CQ bit was checked by qyearsley@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-03-27 16:57:05]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-03-27 18:14:27]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-03-27 18:14:28]

Dry run: This issue passed the CQ dry run.

[qyearsley, 2017-03-27 18:53:55]

I used git blame to check who might be interested in these new expectations, and
who might be able to improve/fix some tests:

http/tests/: jsbell@, mkwst@, yhirano@
http/tests/streams/: domenic@, ricea@
fast/canvas/: zakerinasab@, jinho.bang@samsung.com
svg/: fs@opera.com
fast/dom/: hs1217.lee@samsung.com

And sent a message on blink-dev about this:
https://groups.google.com/a/chromium.org/forum/#!topic/blink-dev/gg7aZdcrJG4

Hopefully this helpful and isn't too spammy...

Thanks!

[qyearsley, 2017-03-27 20:10:35]

The CQ bit was checked by qyearsley@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-03-27 20:11:05]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-03-27 21:25:24]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-03-27 21:25:25]

Dry run: This issue passed the CQ dry run.

[qyearsley, 2017-03-27 22:53:39]

The CQ bit was checked by qyearsley@chromium.org

[qyearsley, 2017-03-27 22:53:40]

The patchset sent to the CQ was uploaded after l-g-t-m from
jeffcarp@chromium.org
Link to the patchset: https://codereview.chromium.org/2766823003/#ps130001
(title: "One more test with nondeterministic message order")

[commit-bot: I haz the power, 2017-03-27 22:54:20]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-03-27 23:08:05]

CQ is committing da patch.

Bot data: {"patchset_id": 130001, "attempt_start_ts": 1490655219397410,
"parent_rev": "6c18543914d549e025ef4200e25e09945eac692d", "commit_rev":
"13c19c95a727c5dbb003bcc089e6089bdcbe96aa"}

[commit-bot: I haz the power, 2017-03-27 23:09:02]

Description was changed from

==========
Fix testharness result parsing, make "Harness error" count as failure.

BUG=703331,705125
==========

to

==========
Fix testharness result parsing, make "Harness error" count as failure.

BUG=703331,705125

Review-Url: https://codereview.chromium.org/2766823003
Cr-Commit-Position: refs/heads/master@{#459924}
Committed:
https://chromium.googlesource.com/chromium/src/+/13c19c95a727c5dbb003bcc089e6...
==========

[commit-bot: I haz the power, 2017-03-27 23:09:04]

Committed patchset #9 (id:130001) as
https://chromium.googlesource.com/chromium/src/+/13c19c95a727c5dbb003bcc089e6...

[Adam Rice, 2017-03-28 03:52:12]

On 2017/03/27 18:53:55, qyearsley wrote:
> http/tests/streams/: domenic@, ricea@

lgtm

external/wpt/streams failures are expected for now until I land an update to the
latest version of the standard.
http/tests/streams failures are maybe caused by them being copies of the ones in
external/wpt/streams? I will remove the duplicate tests.

[domenic, 2017-03-28 04:02:56]

On 2017/03/28 at 03:52:12, ricea wrote:
> On 2017/03/27 18:53:55, qyearsley wrote:
> > http/tests/streams/: domenic@, ricea@
> 
> lgtm
> 
> external/wpt/streams failures are expected for now until I land an update to
the latest version of the standard.
> http/tests/streams failures are maybe caused by them being copies of the ones
in external/wpt/streams? I will remove the duplicate tests.

In particular they were fixed in the external copies as part of the initial
import into WPT at https://github.com/w3c/web-platform-tests/pull/4559 plus a
follow-up at https://github.com/w3c/web-platform-tests/pull/4589. Deleting our
copies is the way to go.

[qyearsley, 2017-03-28 16:51:04]

On 2017/03/28 at 03:52:12, ricea wrote:
> On 2017/03/27 18:53:55, qyearsley wrote:
> > http/tests/streams/: domenic@, ricea@
> 
> lgtm
> 
> external/wpt/streams failures are expected for now until I land an update to
the latest version of the standard.
> http/tests/streams failures are maybe caused by them being copies of the ones
in external/wpt/streams? I will remove the duplicate tests.

I think that "harness_status.message = 7 duplicate test names" means that there
were duplicate subtest description strings inside of one test; having duplicates
of whole test files should be fine.

[imcheng, 2017-03-28 18:06:56]

A revert of this CL (patchset #9 id:130001) has been created in
https://codereview.chromium.org/2782653002/ by imcheng@chromium.org.

The reason for reverting is: Broke WebKit Linux Trusty (dbg)

Test failure:
virtual/wheelscrolllatching/fast/events/wheel/wheel-scroll-latching-on-scrollbar.html.

[qyearsley, 2017-03-28 18:45:35]

On 2017/03/28 at 18:06:56, imcheng wrote:
> A revert of this CL (patchset #9 id:130001) has been created in
https://codereview.chromium.org/2782653002/ by imcheng@chromium.org.
> 
> The reason for reverting is: Broke WebKit Linux Trusty (dbg)
> 
> Test failure:
>
virtual/wheelscrolllatching/fast/events/wheel/wheel-scroll-latching-on-scrollbar.html.

Update: the revert was not applied cleanly; I made a CL to add an entry to
TestExpectations: https://codereview.chromium.org/2777243003