gm self-tests: stop comparing stdout/stderr/command_line, just look at JSON summary

gm self-tests: stop comparing stdout/stderr/command_line, just look at JSON summary
BUG=skia:1568

[epoger, 2013-08-28 19:43:04]

Any thoughts on my comments below?

https://codereview.chromium.org/23725003/diff/1/gm/tests/outputs/add-config-p...
File gm/tests/outputs/add-config-pdf/output-expected/stdout (left):

https://codereview.chromium.org/23725003/diff/1/gm/tests/outputs/add-config-p...
gm/tests/outputs/add-config-pdf/output-expected/stdout:9: GM: ... over  3
configs ["8888", "565", "pdf"]
I'm disappointed we will no longer check this "header" info (which configs/modes
were run, and the "Ran 10 tests" line).  That seemed like a nice way to make
sure we were aware of any gm behavior changes.

Like, what if someone accidentally removes tilegrid from the list of default
modes to run, and later the tilegrid mode gets broken?  How would we find out? 
(The buildbots would not report any failures in any modes...)

https://codereview.chromium.org/23725003/diff/1/gm/tests/run.sh
File gm/tests/run.sh (left):

https://codereview.chromium.org/23725003/diff/1/gm/tests/run.sh#oldcode99
gm/tests/run.sh:99: grep ^GM: $ACTUAL_OUTPUT_DIR/stdout
>$ACTUAL_OUTPUT_DIR/stdout-tmp
Now that we aren't filtering stdout for these "GM: " lines, I can get rid of the
gm_fprintf() function at
https://code.google.com/p/skia/source/browse/trunk/gm/gm_expectations.cpp?r=1...
.  But I'll save that for later...

[bsalomon, 2013-08-28 20:41:59]

On 2013/08/28 19:43:04, epoger wrote:
> Any thoughts on my comments below?
> 
>
https://codereview.chromium.org/23725003/diff/1/gm/tests/outputs/add-config-p...
> File gm/tests/outputs/add-config-pdf/output-expected/stdout (left):
> 
>
https://codereview.chromium.org/23725003/diff/1/gm/tests/outputs/add-config-p...
> gm/tests/outputs/add-config-pdf/output-expected/stdout:9: GM: ... over  3
> configs ["8888", "565", "pdf"]
> I'm disappointed we will no longer check this "header" info (which
configs/modes
> were run, and the "Ran 10 tests" line).  That seemed like a nice way to make
> sure we were aware of any gm behavior changes.
> 
> Like, what if someone accidentally removes tilegrid from the list of default
> modes to run, and later the tilegrid mode gets broken?  How would we find out?

> (The buildbots would not report any failures in any modes...)
> 
> https://codereview.chromium.org/23725003/diff/1/gm/tests/run.sh
> File gm/tests/run.sh (left):
> 
> https://codereview.chromium.org/23725003/diff/1/gm/tests/run.sh#oldcode99
> gm/tests/run.sh:99: grep ^GM: $ACTUAL_OUTPUT_DIR/stdout
> >$ACTUAL_OUTPUT_DIR/stdout-tmp
> Now that we aren't filtering stdout for these "GM: " lines, I can get rid of
the
> gm_fprintf() function at
>
https://code.google.com/p/skia/source/browse/trunk/gm/gm_expectations.cpp?r=1...
> .  But I'll save that for later...

Sorry been tied up away from mail most of the day. My general thought is I have
no problem self-testing in order to validate that the tool did what we expect.
But I would like it if we did that by dumping a report in a separate
machine-readable format (e.g. a json, csv, or xml file) rather than using the
stdout and stderr output that a dev sees. It's nice to be able to decide to
reformat the print outs to make them more usable to developers without busting
the test system.

[epoger, 2013-08-29 16:03:25]

On 2013/08/28 20:41:59, bsalomon wrote:
> My general thought is I have
> no problem self-testing in order to validate that the tool did what we expect.
> But I would like it if we did that by dumping a report in a separate
> machine-readable format (e.g. a json, csv, or xml file) rather than using the
> stdout and stderr output that a dev sees. It's nice to be able to decide to
> reformat the print outs to make them more usable to developers without busting
> the test system.

Seems fair enough.

In the case of GM, we already have the JSON summary that the tool writes out. 
Do you think it might be a good idea to add a stanza to that JSON summary
containing this information in some form?

GM: Ran 1 GMs	
GM: ... over  3 configs ["8888", "565", "pdf"]	
GM: ...  and  7 modes   ["pipe", "pipe cross-process", "pipe cross-process,
shared address", "replay", "rtree", "serialize", "tilegrid"]	
GM: ... so there should be a total of 10 tests.	
GM: Ran 10 tests: NoGpuContext=0 IntentionallySkipped=0 RenderModeMismatch=0
ExpectationsMismatch=0 MissingExpectations=1 WritingReferenceImage=0

[bsalomon, 2013-08-29 17:49:27]

On 2013/08/29 16:03:25, epoger wrote:
> On 2013/08/28 20:41:59, bsalomon wrote:
> > My general thought is I have
> > no problem self-testing in order to validate that the tool did what we
expect.
> > But I would like it if we did that by dumping a report in a separate
> > machine-readable format (e.g. a json, csv, or xml file) rather than using
the
> > stdout and stderr output that a dev sees. It's nice to be able to decide to
> > reformat the print outs to make them more usable to developers without
busting
> > the test system.
> 
> Seems fair enough.
> 
> In the case of GM, we already have the JSON summary that the tool writes out. 
> Do you think it might be a good idea to add a stanza to that JSON summary
> containing this information in some form?
> 
> GM: Ran 1 GMs	
> GM: ... over  3 configs ["8888", "565", "pdf"]	
> GM: ...  and  7 modes   ["pipe", "pipe cross-process", "pipe cross-process,
> shared address", "replay", "rtree", "serialize", "tilegrid"]	
> GM: ... so there should be a total of 10 tests.	
> GM: Ran 10 tests: NoGpuContext=0 IntentionallySkipped=0 RenderModeMismatch=0
> ExpectationsMismatch=0 MissingExpectations=1 WritingReferenceImage=0

That makes sense to me.