gm self-test: make sure we report failures in individual rendering modes

gm/tests/outputs/pipe-playback-failure/output-expected/json-summary.txt

Index: gm/tests/outputs/pipe-playback-failure/output-expected/json-summary.txt
===================================================================
--- gm/tests/outputs/pipe-playback-failure/output-expected/json-summary.txt	(revision 0)
+++ gm/tests/outputs/pipe-playback-failure/output-expected/json-summary.txt	(working copy)
@@ -1,6 +1,10 @@
 {
    "actual-results" : {
-      "failed" : null,
+      "failed" : {
+         "comparison/selftest1-pipe" : {

[epoger, 2013/03/14 22:36:27]

So now we report cross-drawing-mode results in the JSON summary, but *only* in
case of failure.  Seem reasonable?

What I like about this is that our CompareGMs step will, as planned, be able to
just look for anything in this "failed" section to report whether the GM results
were OK or not.

[borenet, 2013/03/15 20:15:53]

This effectively treats the built-in comparisons, which I had thought of as
assertions, as additional GMs.  Will we also write out images for these on
failure?  While that wasn't really considered before, it might be helpful in
diagnosing problems.

[epoger, 2013/03/15 22:05:55]

I hadn't thought about that either way.  Seems like it would be useful (when we
add the option of writing out failed images only).

+            "checksum" : 4259036727585789440
+         }
+      },
       "failure-ignored" : null,
       "no-comparison" : null,
       "succeeded" : {
@@ -20,6 +24,10 @@
       "8888/selftest1" : {
          "checksums" : [ 14022967492765711532 ],
          "ignore-failure" : false
+      },
+      "comparison/selftest1-pipe" : {
+         "checksums" : [ 14022967492765711532 ],

[epoger, 2013/03/14 22:36:27]

Regardless of whether multiple checksums were allowed for 8888/selftest1 , only
one expected checksum will show up here: the actual results we got for
8888/selftest1 in this run.

[borenet, 2013/03/15 20:15:53]

I'm a bit confused by this.  Is the expectation for comparison/selftest1-pipe
defined as the actual result for 8888/selftest1 for THIS run?  This breaks the
assumption that we know all of our expectations BEFORE running gm, even though
it kind of makes sense in a this-is-how-it-currently-works kind of way.  This
may not be a bad thing, as long as it's well-documented.  What's the expected
behavior when the expectations file that we give to gm contains an expectation
for comparison/selftest1-pipe?  What if it's different from 8888/selftest1?

Alternatively, the expectation for comparison/selftest1-pipe could be made to be
the same as the pre-gm expectation for 8888/selftest1.  This would cause *both*
tests to fail when the rendering changes but the two draw the same new image and
would cause only comparison/selftest1-pipe in the case that it's designed to
test.  In that case we wouldn't really need to include comparison/* in the
expecations at all, which is consistent with what we do now.

[epoger, 2013/03/15 22:05:55]

Yes, this could get confusing, or we could end up with an explosion of
expectations...

Presently:
1. The expectation for comparison/selftest1-pipe is defined as the actual result
for 8888/selftest1 for THIS run.
2. Any expectations for comparison/selftest1-pipe (in the JSON expected
checksums file) will be IGNORED, as if they were for 8888/nonexistent_test.

These behaviors are motivated by:
A. Our belief that all rendering modes should result in exactly the same bitmap.
B. Our desire to not have a ridiculously large expectations file, with identical
expectations for every drawing mode of each test.
If we truly believe that results should be the same across all drawing modes at
all times, then I think the present behavior is better.  That way, even if we
don't have expectations for the test, we will fail if there is a difference
across drawing modes.

But I'm certainly open to change on this...

[borenet, 2013/03/18 12:26:37]

Should this be an error?  "Invalid expectation" or "Unknown test"?
This sounds fine with me, as long as it's made very clear where the extra
expectations are coming from.

[epoger, 2013/03/21 16:08:32]

Ideally, yes.  I think it's a long-term issue, not a short-term one... filed as
https://code.google.com/p/skia/issues/detail?id=1177 ('gm: report error if any
expected gms did not run')

+         "ignore-failure" : false
       }
    }
 }