tools/test.dart: Add summary report of the number of tests with each expectation.

tools/test.dart: Add summary report of the number of tests with each expectation.


BUG=
TEST=tools/test.dart --report

Committed: https://code.google.com/p/dart/source/detail?r=2017

[Bill Hesse, 2011-12-01 16:31:13]

Produces the same numbers as test.py on standalone and language.  The numbers
are computed strangely, though.  We may want to change them to more informative.
 You may also dislike the architecture.

[Mads Ager (google), 2011-12-01 19:53:53]

LGTM

I agree with you that we should probably improve on this reporting. I don't like
that test marked as either crashing or failing are not taking into account in
the break down.

http://codereview.chromium.org/8771007/diff/2001/tools/testing/dart/test_suit...
File tools/testing/dart/test_suite.dart (right):

http://codereview.chromium.org/8771007/diff/2001/tools/testing/dart/test_suit...
tools/testing/dart/test_suite.dart:210: if (configuration["report"]) {
You want to add this above in the CCTestSuite as well. Potentially extract this
into a helper now that getting the expectations is more than a line?

http://codereview.chromium.org/8771007/diff/2001/tools/testing/dart/test_suit...
tools/testing/dart/test_suite.dart:424: static void add(Set<String> expect) {
I'd spell out 'expectations'.

http://codereview.chromium.org/8771007/diff/2001/tools/testing/dart/test_suit...
tools/testing/dart/test_suite.dart:433: if (expect.contains(PASS) &&
expect.isSubsetOf([PASS])) {
I would prefer:

expect.contains(PASS) && expect.length == 1

PASS is in there and there is only that element.

Similarly below:

expect.containsAll([FAIL, OK]) && expect.length == 2

I don't have strong opinions on it though and will let you decide.

http://codereview.chromium.org/8771007/diff/2001/tools/testing/dart/test_suit...
tools/testing/dart/test_suite.dart:448: }
I see what you mean about the reporting being strange. How about cases such as:

CRASH, OK
PASS, OK
CRASH, FAIL
CRASH, FAIL, PASS

In particular I think we should add reporting of the last category of flaky
tests that either crash or fail (or even pass).