gm: record IntentionallySkipped tests

gm: record IntentionallySkipped tests
Committed: https://code.google.com/p/skia/source/detail?r=8586

[epoger, 2013-04-04 19:42:18]

https://codereview.chromium.org/13663003/diff/1/gm/tests/run.sh
File gm/tests/run.sh (right):

https://codereview.chromium.org/13663003/diff/1/gm/tests/run.sh#newcode163
gm/tests/run.sh:163: gm_test "--hierarchy --match selftest1 selftest2 $CONFIGS"
"$GM_OUTPUTS/intentionally-skipped-tests"
patchset 1 just adds a test case that runs both selftest1 and selftest2 gms. 
For now, none of the renderModes are intentionally skipped in either gm... but
that will change in the next patchset.

[epoger, 2013-04-04 19:59:11]

https://codereview.chromium.org/13663003/diff/4001/gm/selftest.cpp
File gm/selftest.cpp (right):

https://codereview.chromium.org/13663003/diff/4001/gm/selftest.cpp#newcode58
gm/selftest.cpp:58: return new SelfTestGM("selftest2", kTranslucentBlue, 
skiagm::GM::kSkipPipe_Flag);
in patchset 2, selftest2 now skips piped rendering modes...

https://codereview.chromium.org/13663003/diff/4001/gm/tests/outputs/intention...
File gm/tests/outputs/intentionally-skipped-tests/output-expected/stdout
(right):

https://codereview.chromium.org/13663003/diff/4001/gm/tests/outputs/intention...
gm/tests/outputs/intentionally-skipped-tests/output-expected/stdout:7: GM: Ran
15 tests: NoGpuContext=0 RenderModeMismatch=0 ExpectationsMismatch=0
MissingExpectations=4 WritingReferenceImage=0
... note that the number of tests went down (and no longer matches the expected
18), but there is no explanation of which tests were skipped or why.

[epoger, 2013-04-09 15:20:40]

https://codereview.chromium.org/13663003/diff/12001/gm/gm_error.h
File gm/gm_error.h (right):

https://codereview.chromium.org/13663003/diff/12001/gm/gm_error.h#newcode29
gm/gm_error.h:29: kIntentionallySkipped_ErrorType,
patchset 3 adds kIntentionallySkipped_ErrorType, but not any code to ever return
this error type.

https://codereview.chromium.org/13663003/diff/12001/gm/gmmain.cpp
File gm/gmmain.cpp (right):

https://codereview.chromium.org/13663003/diff/12001/gm/gmmain.cpp#newcode182
gm/gmmain.cpp:182:
fIgnorableErrorCombination.add(kIntentionallySkipped_ErrorType);
kIntentionallySkipped_ErrorType is not cause for concern...

https://codereview.chromium.org/13663003/diff/12001/gm/tests/outputs/compared...
File
gm/tests/outputs/compared-against-different-pixels-images/output-expected/stdout
(right):

https://codereview.chromium.org/13663003/diff/12001/gm/tests/outputs/compared...
gm/tests/outputs/compared-against-different-pixels-images/output-expected/stdout:7:
GM: Ran 9 tests: NoGpuContext=0 IntentionallySkipped=0 RenderModeMismatch=0
ExpectationsMismatch=2 MissingExpectations=0 WritingReferenceImage=0
all of the expected stdout now includes "IntentionallySkipped=0"

[epoger, 2013-04-09 17:57:57]

https://codereview.chromium.org/13663003/diff/17003/gm/gmmain.cpp
File gm/gmmain.cpp (right):

https://codereview.chromium.org/13663003/diff/17003/gm/gmmain.cpp#newcode1010
gm/gmmain.cpp:1010: RecordTestResults(kIntentionallySkipped_ErrorType, name,
patchset 4 now looks for cases where we are intentionally skipping tests (due to
gmFlags such as kSkipPipe_Flag) and reports them as such.

https://codereview.chromium.org/13663003/diff/17003/gm/gmmain.cpp#newcode1487
gm/gmmain.cpp:1487: SkScalar recordScale = SkScalarInvert(replayScale);
many of the changed lines around here are just re-indenting because of the
gmFlags check...

https://codereview.chromium.org/13663003/diff/17003/gm/tests/outputs/intentio...
File gm/tests/outputs/intentionally-skipped-tests/output-expected/stdout
(right):

https://codereview.chromium.org/13663003/diff/17003/gm/tests/outputs/intentio...
gm/tests/outputs/intentionally-skipped-tests/output-expected/stdout:9: GM: [ ] 3
IntentionallySkipped: comparison/selftest2-pipe comparison/selftest2-pipe
cross-process comparison/selftest2-pipe cross-process, shared address
now we report the 3 intentionally skipped pipe tests!

[epoger, 2013-04-09 18:38:51]

Ready to review at patchset 5!

This resolves some TODOs that I added in
https://codereview.chromium.org/12851022/#msg5

https://codereview.chromium.org/13663003/diff/23004/gm/gmmain.cpp
File gm/gmmain.cpp (right):

https://codereview.chromium.org/13663003/diff/23004/gm/gmmain.cpp#newcode1813
gm/gmmain.cpp:1813: if (expectedNumberOfTests != gmmain.fTestsRun) {
Patchset 5 adds a check that will return error if we are missing any of the
expected tests.

For now, this is disabled... I want to turn that on in a separate CL, so we can
quickly revert the check if it causes problems on the bots.

[borenet, 2013-04-09 19:47:53]

LGTM at patch set 6!

https://codereview.chromium.org/13663003/diff/23004/gm/gmmain.cpp
File gm/gmmain.cpp (right):

https://codereview.chromium.org/13663003/diff/23004/gm/gmmain.cpp#newcode1813
gm/gmmain.cpp:1813: if (expectedNumberOfTests != gmmain.fTestsRun) {
On 2013/04/09 18:38:52, epoger wrote:
> Patchset 5 adds a check that will return error if we are missing any of the
> expected tests.
> 
> For now, this is disabled... I want to turn that on in a separate CL, so we
can
> quickly revert the check if it causes problems on the bots.

SGTM

[epoger, 2013-04-10 12:22:40]

Committed patchset #6 manually as r8586 (presubmit successful).