Add compare_failures gardening utility.

Add compare_failures gardening utility.

R=karlklose@google.com

Committed: https://github.com/dart-lang/sdk/commit/a3b193db8033015479120bc8c1e1531b014910bb

[Johnni Winther, 2017-02-23 09:47:45]

johnniwinther@google.com changed reviewers:
+ karlklose@google.com

[karlklose, 2017-02-24 06:33:34]

LGTM

We should also add a README file in the directory to document the tools and how
to use them.

For the future I hope we could base our tools on an API that returns for each
build and each test
- configuration (e.g, dart2js-win-chrome)
- expectation
- actual result (maybe output)
- timing

https://codereview.chromium.org/2711733005/diff/1/tools/gardening/compare_fai...
File tools/gardening/compare_failures.dart (right):

https://codereview.chromium.org/2711733005/diff/1/tools/gardening/compare_fai...
tools/gardening/compare_failures.dart:84: inTiming = true;
'inTiming' -> 'parsingTimingBlock'?

https://codereview.chromium.org/2711733005/diff/1/tools/gardening/compare_fai...
tools/gardening/compare_failures.dart:205: /// Returns the [TestFailure]s for
failing tests did not time out.
'for failing tests *that* did not time out.'

https://codereview.chromium.org/2711733005/diff/1/tools/gardening/compare_fai...
tools/gardening/compare_failures.dart:253:
'$prefix/builders/$botName/builds/$buildNumber/steps/$stepName/logs/$suffix');
Long line. Consider putting this into a getter. It is also similar to
`toString`.

https://codereview.chromium.org/2711733005/diff/1/tools/gardening/compare_fai...
tools/gardening/compare_failures.dart:268: class TestId {
Ho about 'TestConfiguration'?

[jensj, 2017-02-24 11:36:58]

jensj@google.com changed reviewers:
+ jensj@google.com

[jensj, 2017-02-24 11:36:59]

DBC:

There's also https://github.com/sigmundch/bot_failures

and e.g.
bot_failure_summary vm-mac-debug-simdbc64-reload-be -b 5 --summarize

as I couldn't make this seem to run I don't know if it does something different
though...

[Johnni Winther, 2017-03-14 08:34:29]

Description was changed from

==========
Add compare_failures gardening utility.
==========

to

==========
Add compare_failures gardening utility.

R=karlklose@google.com

Committed:
https://github.com/dart-lang/sdk/commit/a3b193db8033015479120bc8c1e1531b014910bb
==========

[Johnni Winther, 2017-03-14 08:34:30]

Committed patchset #2 (id:20001) manually as
a3b193db8033015479120bc8c1e1531b014910bb (presubmit successful).

[Johnni Winther, 2017-03-15 09:49:54]

https://codereview.chromium.org/2711733005/diff/1/tools/gardening/compare_fai...
File tools/gardening/compare_failures.dart (right):

https://codereview.chromium.org/2711733005/diff/1/tools/gardening/compare_fai...
tools/gardening/compare_failures.dart:84: inTiming = true;
On 2017/02/24 06:33:34, karlklose wrote:
> 'inTiming' -> 'parsingTimingBlock'?

Done.

https://codereview.chromium.org/2711733005/diff/1/tools/gardening/compare_fai...
tools/gardening/compare_failures.dart:205: /// Returns the [TestFailure]s for
failing tests did not time out.
On 2017/02/24 06:33:34, karlklose wrote:
> 'for failing tests *that* did not time out.'

Done.

https://codereview.chromium.org/2711733005/diff/1/tools/gardening/compare_fai...
tools/gardening/compare_failures.dart:253:
'$prefix/builders/$botName/builds/$buildNumber/steps/$stepName/logs/$suffix');
On 2017/02/24 06:33:34, karlklose wrote:
> Long line. Consider putting this into a getter. It is also similar to
> `toString`.

Done.

https://codereview.chromium.org/2711733005/diff/1/tools/gardening/compare_fai...
tools/gardening/compare_failures.dart:268: class TestId {
On 2017/02/24 06:33:34, karlklose wrote:
> Ho about 'TestConfiguration'?

Done.