rebaseline.py: --tests and --configs are now FILTERS within json results

rebaseline.py: --tests and --configs are now FILTERS within json results

Before this change, if --tests were specified, all results for those tests
would be rebaselined without regard for the json result summary.

After this change, --tests and --configs are used to *filter* the list of
tests to rebaseline (in case you want to rebaseline aaclip tests, because you
know someone else is going to rebaseline the blur tests).

In other words...
After this change, specifying --tests or --configs will only *reduce* the 
number of tests to be rebaselined.

[epoger, 2013-06-06 19:18:23]

I had been tracking the development of this CL at
https://codereview.chromium.org/16306010 , but that issue stopped working
properly for me... moved over here.

Almost ready for review... I still want to add the --add-new option first.

[epoger, 2013-06-06 20:02:17]

https://codereview.chromium.org/15789010/diff/3001/tools/rebaseline.py
File tools/rebaseline.py (right):

https://codereview.chromium.org/15789010/diff/3001/tools/rebaseline.py#newcod...
tools/rebaseline.py:237: parser.add_argument('--add-new', action='store_true',
Patchset 2 adds a new --add-new option, but it doesn't actually change the
behavior yet.

https://codereview.chromium.org/15789010/diff/3001/tools/tests/rebaseline/out...
File tools/tests/rebaseline/output/add-new-tests/output-expected/stdout (right):

https://codereview.chromium.org/15789010/diff/3001/tools/tests/rebaseline/out...
tools/tests/rebaseline/output/add-new-tests/output-expected/stdout:12: #
aaclip_565.png
For now, this test rebaselines the FAILING aaclip test results, but not the NEW
ones.

[epoger, 2013-06-06 20:19:02]

Ready for review at patchset 3!

https://codereview.chromium.org/15789010/diff/15001/tools/rebaseline.py
File tools/rebaseline.py (right):

https://codereview.chromium.org/15789010/diff/15001/tools/rebaseline.py#newco...
tools/rebaseline.py:174: if self._add_new:
Patchset 3 adds implementation of the --add-new option

https://codereview.chromium.org/15789010/diff/15001/tools/tests/rebaseline/ou...
File tools/tests/rebaseline/output/add-new-tests/output-expected/stdout (right):

https://codereview.chromium.org/15789010/diff/15001/tools/tests/rebaseline/ou...
tools/tests/rebaseline/output/add-new-tests/output-expected/stdout:24: #
aaclip_pdf.png
... and the self-test results confirm the new functionality.

[Stephen White, 2013-06-07 14:30:08]

LGTM

[epoger, 2013-06-07 14:34:50]

Thanks, Stephen.

Since most Chapel Hillians are out of the office today, and I will be out on
Monday, I will wait until Tuesday morning to commit this change (so I can be
around to deal with any difficulties that might come up).

[epoger, 2013-06-07 15:15:00]

https://codereview.chromium.org/15789010/diff/21001/tools/rebaseline.py
File tools/rebaseline.py (right):

https://codereview.chromium.org/15789010/diff/21001/tools/rebaseline.py#newco...
tools/rebaseline.py:98: self._missing_json_is_fatal = False
Made one more change in patchset 4:

If we are running without specifying particular subdirs (i.e. platforms), and we
are unable to read the actual results for one platform, keep going and do the
rest of the platforms.

But if the caller specified a particular set of subdirs, exit if we fail to
rebaseline any of those subdirs.

[epoger, 2013-06-07 15:51:52]

Rats, I just realized that it's too early in the transition process to commit
this change.

The problem is: the actual-results.json file generated by the Android bots does
not contain valid information about whether tests passed or failed.  That's
because the Android bots, unlike the Mac/Windows/Linux ones, don't have access
to the expected images when GM is run.  (We don't want to copy all those image
files down to the Android device; it takes too long.)

See https://code.google.com/p/skia/issues/detail?id=1300 ('buildbots: copy
expected image CHECKSUMS down to Android devices before GenerateGMs step')

The good news is: the actual-results.json file generated on all bots DOES
contain valid bitmap hash values for all the actually-generated images.  So we
CAN rely on those hash values to download actual images from Google Storage (as
noted in Step 1 of https://goto.google.com/ChecksumTransitionDetail )

So here's my plan:

1. I will create new CLs to commit the other parts of this change (adding
--add-new option, renaming --dry_run to --dry-run, etc).

2. As for the "filter" behavior change... we can evaluate that again once we
have completed Step 4 in https://goto.google.com/ChecksumTransitionDetail

Comments welcome.