gm: report ExpectationsMismatch and RenderModeMismatch results separately

gm: report ExpectationsMismatch and RenderModeMismatch results separately
Committed: https://code.google.com/p/skia/source/detail?r=8535

[epoger, 2013-04-04 16:42:56]

https://codereview.chromium.org/13650002/diff/1/gm/gmmain.cpp
File gm/gmmain.cpp (right):

https://codereview.chromium.org/13650002/diff/1/gm/gmmain.cpp#newcode1
gm/gmmain.cpp:1: /*
patchset 1 adds simulatePipePlaybackFailure flag, and a gm self-test that
demonstrates the current result: it is reported as ImageMismatch, just as if it
mismatched expectations.

[epoger, 2013-04-04 16:50:49]

https://codereview.chromium.org/13650002/diff/3001/gm/gm_error.h
File gm/gm_error.h (right):

https://codereview.chromium.org/13650002/diff/3001/gm/gm_error.h#newcode29
gm/gm_error.h:29: kExpectationsMismatch_ErrorType,
patchset 2 renames kImageMismatch to kExpectationsMismatch (making semantic room
for kRenderModeMismatch), and updates the expected self-test results
accordingly.

[epoger, 2013-04-04 16:55:57]

https://codereview.chromium.org/13650002/diff/15003/gm/gm_error.h
File gm/gm_error.h (right):

https://codereview.chromium.org/13650002/diff/15003/gm/gm_error.h#newcode29
gm/gm_error.h:29: kRenderModeMismatch_ErrorType,
patchset 3 adds a new kRenderModeMismatch_ErrorType.  It's not triggered
anywhere yet, but I had to update the expected self-test results for all of the
"RenderModeMismatch=0" reports.

https://codereview.chromium.org/13650002/diff/15003/gm/tests/outputs/pipe-pla...
File gm/tests/outputs/pipe-playback-failure/output-expected/stdout (right):

https://codereview.chromium.org/13650002/diff/15003/gm/tests/outputs/pipe-pla...
gm/tests/outputs/pipe-playback-failure/output-expected/stdout:10: GM: [*] 1
ExpectationsMismatch: comparison/selftest1-pipe
note that comparison/selftest1-pipe is reported as an ExpectationsMismatch, when
we really want it to be reported as a RenderModeMismatch.  That change comes
next...

[epoger, 2013-04-04 17:29:44]

This is a step towards the addition of a --noFailOnMismatch option to gm, as
part of https://goto.google.com/ChecksumRemainingSteps .

Ready for review at patchset 4!

https://codereview.chromium.org/13650002/diff/18002/gm/gmmain.cpp
File gm/gmmain.cpp (right):

https://codereview.chromium.org/13650002/diff/18002/gm/gmmain.cpp#newcode713
gm/gmmain.cpp:713: errors.add(kRenderModeMismatch_ErrorType);
patchset 4 records the error as RenderModeMismatch if it was running a
renderMode with a non-blank renderModeDescriptor!

https://codereview.chromium.org/13650002/diff/18002/gm/tests/outputs/pipe-pla...
File gm/tests/outputs/pipe-playback-failure/output-expected/json-summary.txt
(right):

https://codereview.chromium.org/13650002/diff/18002/gm/tests/outputs/pipe-pla...
gm/tests/outputs/pipe-playback-failure/output-expected/json-summary.txt:3:
"failed" : {
Note that renderMode results are reported in the JSON summary, but *only* for
failures.  (We don't want to clutter the JSON summary with a billion as-expected
results, given that we know the cross-renderMode results are supposed to be
identical.)

https://codereview.chromium.org/13650002/diff/18002/gm/tests/outputs/pipe-pla...
gm/tests/outputs/pipe-playback-failure/output-expected/json-summary.txt:29:
"checksums" : [ 14022967492765711532 ],
Currently, the cross-renderMode comparisons are run in 8888 config only.  So the
expected results will always be the same as those...

https://codereview.chromium.org/13650002/diff/18002/gm/tests/outputs/pipe-pla...
File gm/tests/outputs/pipe-playback-failure/output-expected/stdout (right):

https://codereview.chromium.org/13650002/diff/18002/gm/tests/outputs/pipe-pla...
gm/tests/outputs/pipe-playback-failure/output-expected/stdout:9: GM: [*] 1
RenderModeMismatch: comparison/selftest1-pipe
now reported as a RenderMoreMismatch, not an ExpectationsMismatch!

[borenet, 2013-04-04 18:05:45]

Nice, this LGTM

https://codereview.chromium.org/13650002/diff/18002/gm/gmmain.cpp
File gm/gmmain.cpp (right):

https://codereview.chromium.org/13650002/diff/18002/gm/gmmain.cpp#newcode1001
gm/gmmain.cpp:1001: const SkBitmap& referenceBitmap, bool simulateFailure) {
I guess we had this discussion before: neither of us likes this, but it's the
cleanest way to self-test this.

https://codereview.chromium.org/13650002/diff/18002/gm/tests/outputs/pipe-pla...
File gm/tests/outputs/pipe-playback-failure/output-expected/json-summary.txt
(right):

https://codereview.chromium.org/13650002/diff/18002/gm/tests/outputs/pipe-pla...
gm/tests/outputs/pipe-playback-failure/output-expected/json-summary.txt:3:
"failed" : {
On 2013/04/04 17:29:44, epoger wrote:
> Note that renderMode results are reported in the JSON summary, but *only* for
> failures.  (We don't want to clutter the JSON summary with a billion
as-expected
> results, given that we know the cross-renderMode results are supposed to be
> identical.)

SGTM

[epoger, 2013-04-04 18:06:48]

https://codereview.chromium.org/13650002/diff/18002/gm/gmmain.cpp
File gm/gmmain.cpp (right):

https://codereview.chromium.org/13650002/diff/18002/gm/gmmain.cpp#newcode1001
gm/gmmain.cpp:1001: const SkBitmap& referenceBitmap, bool simulateFailure) {
On 2013/04/04 18:05:45, borenet wrote:
> I guess we had this discussion before: neither of us likes this, but it's the
> cleanest way to self-test this.

Yup.  It's the least bad option we could come up with. :-)

[epoger, 2013-04-04 18:11:43]

Committed patchset #4 manually as r8535 (presubmit successful).