Report w3c test result count for WPT

Report w3c test result count for WPT

Some expected.txt have over 1000 tests mixing PASS/FAIL.
This CL changes testharnssreport to output counter for ease to recognize how 
each WPT case gets better/worse for cases over 50 tests.

BUG=683066
Review-Url: https://codereview.chromium.org/2658793002
Cr-Commit-Position: refs/heads/master@{#450922}
Committed: https://chromium.googlesource.com/chromium/src/+/7c15aa049a9c754139c3320bae7ab69c93919e68

[yoichio, 2017-01-26 09:20:06]

The CQ bit was checked by yoichio@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-01-26 09:20:22]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-01-26 09:27:03]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-01-26 09:27:04]

Dry run: Try jobs failed on following builders:
  ios-simulator on master.tryserver.chromium.mac (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.mac/builders/ios-simulator/bui...)
  ios-simulator-xcode-clang on master.tryserver.chromium.mac (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.mac/builders/ios-simulator-xco...)

[yoichio, 2017-01-26 09:27:25]

Description was changed from

==========
culc testharnss


init

BUG=
==========

to

==========
Report w3c test result count for WPT

Some expected.txt have over 1000 tests mixing PASS/FAIL.
This CL changes testharnssreport to output counter for ease to recognize how
each
 WPT case gets better/worse.

BUG=683066
==========

[yoichio, 2017-01-26 09:27:25]

yoichio@chromium.org changed reviewers:
+ tkent@chromium.org

[tkent, 2017-01-26 22:09:44]

https://codereview.chromium.org/2658793002/diff/20001/third_party/WebKit/Layo...
File third_party/WebKit/LayoutTests/resources/testharnessreport.js (right):

https://codereview.chromium.org/2658793002/diff/20001/third_party/WebKit/Layo...
third_party/WebKit/LayoutTests/resources/testharnessreport.js:188: } else if
(output_document.URL.indexOf("http://web-platform.test") >= 0){
I don't think this condition makes much sense.  The new stat line would be
helpful for all of testharness.js tests, not only external/wpt/.

https://codereview.chromium.org/2658793002/diff/20001/third_party/WebKit/Layo...
third_party/WebKit/LayoutTests/resources/testharnessreport.js:190: let
resultCounter = [0,0,0,0];
[0,0,0,0] -> [0, 0, 0, 0]

https://codereview.chromium.org/2658793002/diff/20001/third_party/WebKit/Layo...
third_party/WebKit/LayoutTests/resources/testharnessreport.js:197: resultStr +=
`Found ${tests.length} tests.` +
I recommend to show the stats only if
 - non-0 FAIL/TIMEOUT/NOTRUN counts, and
  (If we do so, I think the change of testharness_results.py is unnecessary.)
 - tests.length is larger than a large number.
  For example, tests.length >= 50 ?
  If the number of tests are small, I don't think the stats line is very
helpful.
 - Or show the stats line only for specific URLs such as
web-platform.test/selection/.

The current format is something like:
  Found 46 tests. 36 PASS 10 FAIL 0 TIMEOUT 0 NOTRUN

I'd like:
  Found 46 tests; PASS:36 FAIL:10 TIMEOUT:0 NOTRUN:0
or
  Found 46 tests; 36 PASS, 10 FAIL, 0 TIMEOUT, 0 NOTRUN.

https://codereview.chromium.org/2658793002/diff/20001/third_party/WebKit/Layo...
third_party/WebKit/LayoutTests/resources/testharnessreport.js:213: 
Do not add unnecessary blank lines.

[yoichio, 2017-01-27 07:15:51]

The CQ bit was checked by yoichio@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-01-27 07:16:05]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[yoichio, 2017-01-27 07:21:57]

https://codereview.chromium.org/2658793002/diff/20001/third_party/WebKit/Layo...
File third_party/WebKit/LayoutTests/resources/testharnessreport.js (right):

https://codereview.chromium.org/2658793002/diff/20001/third_party/WebKit/Layo...
third_party/WebKit/LayoutTests/resources/testharnessreport.js:188: } else if
(output_document.URL.indexOf("http://web-platform.test") >= 0){
On 2017/01/26 22:09:43, tkent wrote:
> I don't think this condition makes much sense.  The new stat line would be
> helpful for all of testharness.js tests, not only external/wpt/.
I would be helpful but it needs much rebasing -expected.txt.
I prefer step-by-step on each Layouttest directory.

https://codereview.chromium.org/2658793002/diff/20001/third_party/WebKit/Layo...
third_party/WebKit/LayoutTests/resources/testharnessreport.js:190: let
resultCounter = [0,0,0,0];
On 2017/01/26 22:09:43, tkent wrote:
> [0,0,0,0] -> [0, 0, 0, 0]

Done.

https://codereview.chromium.org/2658793002/diff/20001/third_party/WebKit/Layo...
third_party/WebKit/LayoutTests/resources/testharnessreport.js:197: resultStr +=
`Found ${tests.length} tests.` +
On 2017/01/26 22:09:43, tkent wrote:
> I recommend to show the stats only if
>  - non-0 FAIL/TIMEOUT/NOTRUN counts, and
>   (If we do so, I think the change of testharness_results.py is unnecessary.)
>  - tests.length is larger than a large number.
>   For example, tests.length >= 50 ?
>   If the number of tests are small, I don't think the stats line is very
> helpful.
>  - Or show the stats line only for specific URLs such as
> web-platform.test/selection/.
> 
> The current format is something like:
>   Found 46 tests. 36 PASS 10 FAIL 0 TIMEOUT 0 NOTRUN
> 
> I'd like:
>   Found 46 tests; PASS:36 FAIL:10 TIMEOUT:0 NOTRUN:0
> or
>   Found 46 tests; 36 PASS, 10 FAIL, 0 TIMEOUT, 0 NOTRUN.

Done.

https://codereview.chromium.org/2658793002/diff/20001/third_party/WebKit/Layo...
third_party/WebKit/LayoutTests/resources/testharnessreport.js:213: 
On 2017/01/26 22:09:43, tkent wrote:
> Do not add unnecessary blank lines.

Done.

[tkent, 2017-01-27 07:35:36]

lgtm

https://codereview.chromium.org/2658793002/diff/40001/third_party/WebKit/Layo...
File third_party/WebKit/LayoutTests/resources/testharnessreport.js (right):

https://codereview.chromium.org/2658793002/diff/40001/third_party/WebKit/Layo...
third_party/WebKit/LayoutTests/resources/testharnessreport.js:200: tests.length
>= 50 && (resultCounter[1] || resultCounter[2] || resultCounter[3])) {
(resultCounter[1] || resultCounter[2] || resultCounter[3])
 ->
tests.length != resultCounter[0]

[commit-bot: I haz the power, 2017-01-27 08:25:48]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-01-27 08:25:50]

Dry run: Try jobs failed on following builders:
  win_chromium_rel_ng on master.tryserver.chromium.win (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_...)

[yoichio, 2017-02-06 06:42:11]

The CQ bit was checked by yoichio@chromium.org

[yoichio, 2017-02-06 06:42:12]

The patchset sent to the CQ was uploaded after l-g-t-m from tkent@chromium.org
Link to the patchset: https://codereview.chromium.org/2658793002/#ps60001
(title: "update")

[commit-bot: I haz the power, 2017-02-06 06:42:29]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-02-06 08:13:21]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-02-06 08:13:22]

Try jobs failed on following builders:
  mac_chromium_rel_ng on master.tryserver.chromium.mac (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_...)

[yoichio, 2017-02-07 07:07:06]

The CQ bit was checked by yoichio@chromium.org

[yoichio, 2017-02-07 07:07:07]

The patchset sent to the CQ was uploaded after l-g-t-m from tkent@chromium.org
Link to the patchset: https://codereview.chromium.org/2658793002/#ps80001
(title: "update")

[commit-bot: I haz the power, 2017-02-07 07:07:15]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-02-07 08:23:01]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-02-07 08:23:02]

Try jobs failed on following builders:
  win_chromium_rel_ng on master.tryserver.chromium.win (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_...)

[yoichio, 2017-02-10 11:40:55]

Description was changed from

==========
Report w3c test result count for WPT

Some expected.txt have over 1000 tests mixing PASS/FAIL.
This CL changes testharnssreport to output counter for ease to recognize how
each
 WPT case gets better/worse.

BUG=683066
==========

to

==========
Report w3c test result count for WPT

Some expected.txt have over 1000 tests mixing PASS/FAIL.
This CL changes testharnssreport to output counter for ease to recognize how 
each WPT case gets better/worse for cases over 50 tests.

BUG=683066
==========

[yoichio, 2017-02-14 08:30:18]

The CQ bit was checked by yoichio@chromium.org

[yoichio, 2017-02-14 08:30:19]

The patchset sent to the CQ was uploaded after l-g-t-m from tkent@chromium.org
Link to the patchset: https://codereview.chromium.org/2658793002/#ps100001
(title: "rebase")

[commit-bot: I haz the power, 2017-02-14 08:30:37]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-02-14 08:33:42]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-02-14 08:33:43]

Try jobs failed on following builders:
  ios-device-xcode-clang on master.tryserver.chromium.mac (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.mac/builders/ios-device-xcode-...)

[yoichio, 2017-02-15 03:35:26]

The CQ bit was checked by yoichio@chromium.org

[yoichio, 2017-02-15 03:35:27]

The patchset sent to the CQ was uploaded after l-g-t-m from tkent@chromium.org
Link to the patchset: https://codereview.chromium.org/2658793002/#ps120001
(title: "rebase")

[commit-bot: I haz the power, 2017-02-15 03:36:37]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-02-15 05:04:14]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-02-15 05:04:16]

Try jobs failed on following builders:
  mac_chromium_rel_ng on master.tryserver.chromium.mac (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_...)

[yoichio, 2017-02-15 08:13:30]

The CQ bit was checked by yoichio@chromium.org

[yoichio, 2017-02-15 08:13:31]

The patchset sent to the CQ was uploaded after l-g-t-m from tkent@chromium.org
Link to the patchset: https://codereview.chromium.org/2658793002/#ps110083
(title: "update")

[commit-bot: I haz the power, 2017-02-15 08:13:57]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[tkent, 2017-02-15 08:21:36]

You may land manually, and watch layout test bots after landing

[commit-bot: I haz the power, 2017-02-15 09:27:46]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-02-15 09:27:47]

Try jobs failed on following builders:
  win_chromium_rel_ng on master.tryserver.chromium.win (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_...)

[yoichio, 2017-02-16 03:17:45]

The CQ bit was checked by yoichio@chromium.org

[yoichio, 2017-02-16 03:17:46]

The patchset sent to the CQ was uploaded after l-g-t-m from tkent@chromium.org
Link to the patchset: https://codereview.chromium.org/2658793002/#ps150001
(title: "rebase")

[commit-bot: I haz the power, 2017-02-16 03:19:11]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-02-16 05:21:39]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-02-16 05:21:41]

Try jobs failed on following builders:
  cast_shell_linux on master.tryserver.chromium.linux (JOB_TIMED_OUT, no build
URL)
  chromeos_amd64-generic_chromium_compile_only_ng on
master.tryserver.chromium.linux (JOB_TIMED_OUT, no build URL)
  chromium_presubmit on master.tryserver.chromium.linux (JOB_TIMED_OUT, no build
URL)
  linux_chromium_chromeos_rel_ng on master.tryserver.chromium.linux
(JOB_TIMED_OUT, no build URL)
  linux_chromium_compile_dbg_ng on master.tryserver.chromium.linux
(JOB_TIMED_OUT, no build URL)
  linux_chromium_rel_ng on master.tryserver.chromium.linux (JOB_TIMED_OUT, no
build URL)

[yoichio, 2017-02-16 07:14:36]

The CQ bit was checked by yoichio@chromium.org

[commit-bot: I haz the power, 2017-02-16 07:15:06]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-02-16 08:04:05]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-02-16 08:04:07]

Try jobs failed on following builders:
  chromium_presubmit on master.tryserver.chromium.linux (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/chromium_presub...)

[yoichio, 2017-02-16 08:40:24]

The CQ bit was checked by yoichio@chromium.org

[yoichio, 2017-02-16 08:40:25]

The patchset sent to the CQ was uploaded after l-g-t-m from tkent@chromium.org
Link to the patchset: https://codereview.chromium.org/2658793002/#ps170001
(title: "rebase")

[commit-bot: I haz the power, 2017-02-16 08:41:37]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-02-16 10:10:20]

CQ is committing da patch.

Bot data: {"patchset_id": 170001, "attempt_start_ts": 1487234424216620,
"parent_rev": "fc32eb29940183a79b03a38c0fce1e209a4de445", "commit_rev":
"7c15aa049a9c754139c3320bae7ab69c93919e68"}

[commit-bot: I haz the power, 2017-02-16 10:11:08]

Description was changed from

==========
Report w3c test result count for WPT

Some expected.txt have over 1000 tests mixing PASS/FAIL.
This CL changes testharnssreport to output counter for ease to recognize how 
each WPT case gets better/worse for cases over 50 tests.

BUG=683066
==========

to

==========
Report w3c test result count for WPT

Some expected.txt have over 1000 tests mixing PASS/FAIL.
This CL changes testharnssreport to output counter for ease to recognize how 
each WPT case gets better/worse for cases over 50 tests.

BUG=683066

Review-Url: https://codereview.chromium.org/2658793002
Cr-Commit-Position: refs/heads/master@{#450922}
Committed:
https://chromium.googlesource.com/chromium/src/+/7c15aa049a9c754139c3320bae7a...
==========

[commit-bot: I haz the power, 2017-02-16 10:11:09]

Committed patchset #10 (id:170001) as
https://chromium.googlesource.com/chromium/src/+/7c15aa049a9c754139c3320bae7a...

[tkent, 2017-02-17 03:09:03]

Congratulation on landing finally!