tools/test.dart: Add summary report of the number of tests with each expectation.

tools/testing/dart/test_suite.dart

Index: tools/testing/dart/test_suite.dart
diff --git a/tools/testing/dart/test_suite.dart b/tools/testing/dart/test_suite.dart
index 594c2d684d86f0271413516db8210d4014ff7b0c..3f07d7789e8273c49b483be0ea907ca99d00f4a4 100644
--- a/tools/testing/dart/test_suite.dart
+++ b/tools/testing/dart/test_suite.dart
@@ -207,7 +207,12 @@ class StandardTestSuite implements TestSuite {
             filename.substring(middle + 1, filename.length - 5);
       }
       Set<String> expectations = testExpectations.expectations(testName);
-
+      if (configuration["report"]) {

[Mads Ager (google), 2011/12/01 19:53:53]

You want to add this above in the CCTestSuite as well. Potentially extract this
into a helper now that getting the expectations is more than a line?

+        // Tests with multiple VMOptions are counted more than once.
+        for (var dummy in optionsFromFile["vmOptions"]) {
+          SummaryReport.add(expectations);
+        }
+      }
       if (expectations.contains(SKIP)) return;
 
       isNegative = isNegative ||
@@ -405,3 +410,56 @@ class TestUtils {
     return args;
   }
 }
+
+class SummaryReport {
+  static int total = 0;
+  static int skipped = 0;
+  static int noCrash = 0;
+  static int pass = 0;
+  static int failOk = 0;
+  static int fail = 0;
+  static int crash = 0;
+  static int timeout = 0;
+
+  static void add(Set<String> expect) {

[Mads Ager (google), 2011/12/01 19:53:53]

I'd spell out 'expectations'.

+    ++total;
+    if (expect.contains(SKIP)) {
+      ++skipped;
+    } else {
+      if (expect.contains(PASS) && expect.contains(FAIL) &&
+          !expect.contains(CRASH) && !expect.contains(OK)) {
+        ++noCrash;
+      }
+      if (expect.contains(PASS) && expect.isSubsetOf([PASS])) {

[Mads Ager (google), 2011/12/01 19:53:53]

I would prefer:

expect.contains(PASS) && expect.length == 1

PASS is in there and there is only that element.

Similarly below:

expect.containsAll([FAIL, OK]) && expect.length == 2

I don't have strong opinions on it though and will let you decide.

+        ++pass;
+      }
+      if (expect.containsAll([FAIL, OK]) && expect.isSubsetOf([FAIL, OK])) {
+        ++failOk;
+      }
+      if (expect.contains(FAIL) && expect.isSubsetOf([FAIL])) {
+        ++fail;
+      }
+      if (expect.contains(CRASH) && expect.isSubsetOf([CRASH])) {
+        ++crash;
+      }
+      if (expect.contains(TIMEOUT)) {
+        ++timeout;
+      }
+    }

[Mads Ager (google), 2011/12/01 19:53:53]

I see what you mean about the reporting being strange. How about cases such as:

CRASH, OK
PASS, OK
CRASH, FAIL
CRASH, FAIL, PASS

In particular I think we should add reporting of the last category of flaky
tests that either crash or fail (or even pass).

+  }
+
+  static void printReport() {
+    if (total == 0) return;
+    String report = """\
+Total: $total tests
+ * $skipped tests will be skipped
+ * $noCrash tests are expected to be flaky but not crash
+ * $pass tests are expected to pass
+ * $failOk tests are expected to fail that we won't fix
+ * $fail tests are expected to fail that we should fix
+ * $crash tests are expected to crash that we should fix
+ * $timeout tests are allowed to timeout\
+""";
+    print(report);
+  }
+}