webkitpy: Rework the JSON output flags.

webkitpy: Rework the JSON output flags.

This does the following;

 * Makes --write-full-results-to and --isolated-script-test-output
   aliases for --json-test-results.

 * Adds --json-failing-test-results in case anyone depends on only
   getting failing results.

 * Moves us towards removing needing
   testing/scripts/run_isolated_script_test.py to run layout tests on
   swarming (one less wrapper script layer, yay!).

 * This fixes the output counts on BuildBot for Layout Tests (when not
   run on swarming).

---------------------------------------
 Total tests: 2160
  * Passed: 1172 (1172 expected, 0 unexpected)
* Failed: 956 (956 expected, 0 unexpected)
* Flaky: 32 (32 expected, 0 unexpected)
---------------------------------------

   Changes to;
---------------------------------------
Total tests: 61245
* Passed: 51187 (51156 expected, 31 unexpected)
* Failed: 10020 (10020 expected, 0 unexpected)
* Flaky: 38 (38 expected, 0 unexpected)
---------------------------------------

BUG=524758, 704394
Review-Url: https://codereview.chromium.org/2813173002
Cr-Commit-Position: refs/heads/master@{#464284}
Committed: https://chromium.googlesource.com/chromium/src/+/c035f184530b4a493f56a30f114e049e3d9c5cff

[mithro, 2017-04-12 09:45:47]

Description was changed from

==========
webkitpy: Rework the JSON output flags.

This does the following;

 * Makes --write-full-results-to and --isolated-script-test-output
  aliases for --json-test-results.

 * Adds --json-failing-test-results in case anyone depends on only
   getting failing results.

 * Moves us towards removing needing
   testing/scripts/run_isolated_script_test.py to run layout tests on
   swarming (one less wrapper script layer, yay!).

 * This fixes the output counts on BuildBot for Layout Tests (when not
   run on swarming).

---------------------------------------
 Total tests: 2160
  * Passed: 1172 (1172 expected, 0 unexpected)
* Failed: 956 (956 expected, 0 unexpected)
* Flaky: 32 (32 expected, 0 unexpected)
---------------------------------------

   Changes to;
---------------------------------------
Total tests: 61245
* Passed: 51187 (51156 expected, 31 unexpected)
* Failed: 10020 (10020 expected, 0 unexpected)
* Flaky: 38 (38 expected, 0 unexpected)
---------------------------------------

BUG=524758,704394
==========

to

==========
webkitpy: Rework the JSON output flags.

This does the following;

 * Makes --write-full-results-to and --isolated-script-test-output
   aliases for --json-test-results.

 * Adds --json-failing-test-results in case anyone depends on only
   getting failing results.

 * Moves us towards removing needing
   testing/scripts/run_isolated_script_test.py to run layout tests on
   swarming (one less wrapper script layer, yay!).

 * This fixes the output counts on BuildBot for Layout Tests (when not
   run on swarming).

---------------------------------------
 Total tests: 2160
  * Passed: 1172 (1172 expected, 0 unexpected)
* Failed: 956 (956 expected, 0 unexpected)
* Flaky: 32 (32 expected, 0 unexpected)
---------------------------------------

   Changes to;
---------------------------------------
Total tests: 61245
* Passed: 51187 (51156 expected, 31 unexpected)
* Failed: 10020 (10020 expected, 0 unexpected)
* Flaky: 38 (38 expected, 0 unexpected)
---------------------------------------

BUG=524758,704394
==========

[mithro, 2017-04-12 09:45:47]

tansell@chromium.org changed reviewers:
+ mcgreevy@chromium.org, qyearsley@chromium.org

[mithro, 2017-04-12 10:02:29]

Hi qyearsley,

When you landed https://chromium-review.googlesource.com/c/469108/ (the change
to remove most of the functionality of the stupid wrapper script) you
accidentally broke the test number output on the build bots (because you
effectively removed https://chromium-review.googlesource.com/c/457944/).

One fix would have been to reland my patch at
https://chromium-review.googlesource.com/c/458466/6 (which had to be reverted
because of the stupid wrapper script).

However, I think as we want to remove as many layers of wrapper scripts the
better solution is to make run-webkit-tests support the isolated script API
directly.

This change does that by making --json-test-results, --write-full-results-to and
--isolated-script-test-output aliases which write out the full test result file.
I also added a --json-failing-test-results incase someone still needs only the
failing results functionality.

About the only thing which run-webkit-tests still needs to support for starting
xvfb server which is sounds like you might be working on?

Tim 'mithro' Ansell

[mithro, 2017-04-12 12:01:53]

The CQ bit was checked by tansell@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-04-12 12:02:08]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-04-12 13:18:50]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-04-12 13:18:51]

Dry run: This issue passed the CQ dry run.

[qyearsley, 2017-04-12 19:28:15]

LGTM!

On 2017/04/12 at 10:02:29, tansell wrote:
> Hi qyearsley,
> 
> When you landed https://chromium-review.googlesource.com/c/469108/ (the change
to remove most of the functionality of the stupid wrapper script) you
accidentally broke the test number output on the build bots (because you
effectively removed https://chromium-review.googlesource.com/c/457944/).

Darn, sorry about this.

> One fix would have been to reland my patch at
https://chromium-review.googlesource.com/c/458466/6 (which had to be reverted
because of the stupid wrapper script).
> 
> However, I think as we want to remove as many layers of wrapper scripts the
better solution is to make run-webkit-tests support the isolated script API
directly.
> 
> This change does that by making --json-test-results, --write-full-results-to
and --isolated-script-test-output aliases which write out the full test result
file. I also added a --json-failing-test-results incase someone still needs only
the failing results functionality.
> 
> About the only thing which run-webkit-tests still needs to support for
starting xvfb server which is sounds like you might be working on?

Right :-)

https://codereview.chromium.org/2813173002/diff/20001/third_party/WebKit/Tool...
File third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py
(right):

https://codereview.chromium.org/2813173002/diff/20001/third_party/WebKit/Tool...
third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py:170:
help=('Path to write the JSON test results for *all* tests.')),
Nit: parentheses after help= could be removed

https://codereview.chromium.org/2813173002/diff/20001/third_party/WebKit/Tool...
third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py:174:
help=('Path to write the JSON test results for only *failing* tests.')),
I bet this could also be removed now, since I don't think it's required, but
removing sometime in the future in a separate CL SGTM

[mithro, 2017-04-13 02:37:54]

The CQ bit was checked by tansell@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-04-13 02:38:42]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[mithro, 2017-04-13 02:56:08]

Hi qyearsley,

Don't worry about the accidental revert, getting rid of that stupid wrapper
script is worth the pain.

Tim 'mithro' Ansell

https://codereview.chromium.org/2813173002/diff/20001/third_party/WebKit/Tool...
File third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py
(right):

https://codereview.chromium.org/2813173002/diff/20001/third_party/WebKit/Tool...
third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py:170:
help=('Path to write the JSON test results for *all* tests.')),
On 2017/04/12 19:28:15, qyearsley wrote:
> Nit: parentheses after help= could be removed

Done.

https://codereview.chromium.org/2813173002/diff/20001/third_party/WebKit/Tool...
third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py:174:
help=('Path to write the JSON test results for only *failing* tests.')),
On 2017/04/12 19:28:15, qyearsley wrote:
> I bet this could also be removed now, since I don't think it's required, but
> removing sometime in the future in a separate CL SGTM

Yeah, lets leave it for now.

[mithro, 2017-04-13 02:59:52]

The CQ bit was unchecked by tansell@chromium.org

[mithro, 2017-04-13 03:00:01]

The CQ bit was checked by tansell@chromium.org

[mithro, 2017-04-13 03:00:02]

The patchset sent to the CQ was uploaded after l-g-t-m from
qyearsley@chromium.org
Link to the patchset: https://codereview.chromium.org/2813173002/#ps40001
(title: "Fixing for review.")

[commit-bot: I haz the power, 2017-04-13 03:00:27]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-04-13 03:49:30]

CQ is committing da patch.

Bot data: {"patchset_id": 40001, "attempt_start_ts": 1492052401753880,
"parent_rev": "d6ad7119d64c9b6d62eaf315eab570cd476b2fcf", "commit_rev":
"c035f184530b4a493f56a30f114e049e3d9c5cff"}

[commit-bot: I haz the power, 2017-04-13 03:50:24]

Description was changed from

==========
webkitpy: Rework the JSON output flags.

This does the following;

 * Makes --write-full-results-to and --isolated-script-test-output
   aliases for --json-test-results.

 * Adds --json-failing-test-results in case anyone depends on only
   getting failing results.

 * Moves us towards removing needing
   testing/scripts/run_isolated_script_test.py to run layout tests on
   swarming (one less wrapper script layer, yay!).

 * This fixes the output counts on BuildBot for Layout Tests (when not
   run on swarming).

---------------------------------------
 Total tests: 2160
  * Passed: 1172 (1172 expected, 0 unexpected)
* Failed: 956 (956 expected, 0 unexpected)
* Flaky: 32 (32 expected, 0 unexpected)
---------------------------------------

   Changes to;
---------------------------------------
Total tests: 61245
* Passed: 51187 (51156 expected, 31 unexpected)
* Failed: 10020 (10020 expected, 0 unexpected)
* Flaky: 38 (38 expected, 0 unexpected)
---------------------------------------

BUG=524758,704394
==========

to

==========
webkitpy: Rework the JSON output flags.

This does the following;

 * Makes --write-full-results-to and --isolated-script-test-output
   aliases for --json-test-results.

 * Adds --json-failing-test-results in case anyone depends on only
   getting failing results.

 * Moves us towards removing needing
   testing/scripts/run_isolated_script_test.py to run layout tests on
   swarming (one less wrapper script layer, yay!).

 * This fixes the output counts on BuildBot for Layout Tests (when not
   run on swarming).

---------------------------------------
 Total tests: 2160
  * Passed: 1172 (1172 expected, 0 unexpected)
* Failed: 956 (956 expected, 0 unexpected)
* Flaky: 32 (32 expected, 0 unexpected)
---------------------------------------

   Changes to;
---------------------------------------
Total tests: 61245
* Passed: 51187 (51156 expected, 31 unexpected)
* Failed: 10020 (10020 expected, 0 unexpected)
* Flaky: 38 (38 expected, 0 unexpected)
---------------------------------------

BUG=524758,704394

Review-Url: https://codereview.chromium.org/2813173002
Cr-Commit-Position: refs/heads/master@{#464284}
Committed:
https://chromium.googlesource.com/chromium/src/+/c035f184530b4a493f56a30f114e...
==========

[commit-bot: I haz the power, 2017-04-13 03:50:25]

Committed patchset #3 (id:40001) as
https://chromium.googlesource.com/chromium/src/+/c035f184530b4a493f56a30f114e...

[mithro, 2017-04-19 06:10:51]

Patchset #4 (id:60001) has been deleted