rebaseline.py: --tests and --configs are now FILTERS within json results

rebaseline.py: --tests and --configs are now FILTERS within json results

Before this change, if --tests were specified, all results for those tests
would be rebaselined without regard for the json result summary.

After this change, --tests and --configs are used to *filter* the list of
tests to rebaseline (in case you want to rebaseline aaclip tests, because you
know someone else is going to rebaseline the blur tests).

In other words...
After this change, specifying --tests or --configs will only *reduce* the 
number of tests to be rebaselined.

[epoger, 2013-06-06 17:53:29]

https://codereview.chromium.org/16306010/diff/1/tools/rebaseline.py
File tools/rebaseline.py (right):

https://codereview.chromium.org/16306010/diff/1/tools/rebaseline.py#newcode181
tools/rebaseline.py:181: self._DownloadFile(source_url=url,
dest_filename=outfilename)
Patchset 1 makes the script fail early if any downloads fail.  (Once we are only
downloading images that the JSON summary tells us need rebaselining, downloads
should never fail!)

[epoger, 2013-06-06 17:54:30]

https://codereview.chromium.org/16306010/diff/2001/tools/rebaseline.py
File tools/rebaseline.py (right):

https://codereview.chromium.org/16306010/diff/2001/tools/rebaseline.py#newcod...
tools/rebaseline.py:251: parser.add_argument('--dry-run', action='store_true',
Patchset 2 renames the command-line arguments: use dashes instead of underscores
for multiword option names.

[epoger, 2013-06-06 18:29:52]

https://codereview.chromium.org/16306010/diff/11001/tools/rebaseline.py
File tools/rebaseline.py (right):

https://codereview.chromium.org/16306010/diff/11001/tools/rebaseline.py#newco...
tools/rebaseline.py:205: json_url = '/'.join([self._json_base_url,
Patchset 3 deletes the code path that iterated over the tests/configs specified
by the caller without regard for the JSON file.

I haven't added the test/config filtering yet...

https://codereview.chromium.org/16306010/diff/11001/tools/tests/rebaseline/ou...
File tools/tests/rebaseline/output/all/output-expected/stdout (right):

https://codereview.chromium.org/16306010/diff/11001/tools/tests/rebaseline/ou...
tools/tests/rebaseline/output/all/output-expected/stdout:1: # Getting files to
rebaseline from JSON summary URL
file:tools/tests/rebaseline/input/json1/base-android-galaxy-nexus/Test-Android-GalaxyNexus-SGX540-Arm7-Debug/base-android-galaxy-nexus/actual-results.json
...
I have manually inspected this file and confirmed that these are the expected
results (with no test/config filters applied)

https://codereview.chromium.org/16306010/diff/11001/tools/tests/run.sh
File tools/tests/run.sh (right):

https://codereview.chromium.org/16306010/diff/11001/tools/tests/run.sh#newcod...
tools/tests/run.sh:200: rebaseline_test "$BASELINE_OPTIONS"
"$REBASELINE_OUTPUT/all"
Removed the old tests, and so far created just one that does not exercise
test/config filtering.

[epoger, 2013-06-06 18:38:26]

https://codereview.chromium.org/16306010/diff/3006/tools/tests/rebaseline/out...
File tools/tests/rebaseline/output/all/output-expected/stdout (right):

https://codereview.chromium.org/16306010/diff/3006/tools/tests/rebaseline/out...
tools/tests/rebaseline/output/all/output-expected/stdout:2: #
Patchset 4 just adds some blank lines to the output (only in dry-run mode),
making it easier to read.

[epoger, 2013-06-06 18:46:26]

https://codereview.chromium.org/16306010/diff/26001/tools/tests/run.sh
File tools/tests/run.sh (right):

https://codereview.chromium.org/16306010/diff/26001/tools/tests/run.sh#newcod...
tools/tests/run.sh:201: rebaseline_test "$BASELINE_OPTIONS --configs 8888 pdf"
"$REBASELINE_OUTPUT/configs"
Patchset 5 adds new tests to exercise filtering by configs and tests... but for
now, since there is no filtering behavior yet, the results of all 3 tests are
identical.

$ sum tools/tests/rebaseline/output/all/output-expected/stdout 
47755     4
$ sum tools/tests/rebaseline/output/configs/output-expected/stdout 
47755     4
$ sum tools/tests/rebaseline/output/tests/output-expected/stdout 
47755     4

[epoger, 2013-06-06 19:22:09]

This codereview issue somehow got in a bad state... moved my code over to
https://codereview.chromium.org/15789010/

[Stephen White, 2013-06-06 20:41:52]

https://codereview.chromium.org/16306010/diff/20002/tools/rebaseline.py
File tools/rebaseline.py (right):

https://codereview.chromium.org/16306010/diff/20002/tools/rebaseline.py#newco...
tools/rebaseline.py:29: # Make sure that the 'gm' dir is in the PYTHONPATH, but
add ir at the *end*
Nit: ir -> it?

https://codereview.chromium.org/16306010/diff/20002/tools/rebaseline.py#newco...
tools/rebaseline.py:214: json_url = '/'.join([self._json_base_url,
Is there a way to force an override of the json file, and just grab what was
explicitly requested directly, as it used to work? Otherwise if the json
generator failed for some reason, you won't be able to rebaseline until the
upstream script runs (I know, I'm being pessimistic).

Rebaselining a single test on all configs (which is a common workflow) doesn't
take too long, even in using the old brute force method. It'd be nice to have
that as a fallback.

[epoger, 2013-06-07 14:15:21]

https://codereview.chromium.org/16306010/diff/20002/tools/rebaseline.py
File tools/rebaseline.py (right):

https://codereview.chromium.org/16306010/diff/20002/tools/rebaseline.py#newco...
tools/rebaseline.py:29: # Make sure that the 'gm' dir is in the PYTHONPATH, but
add ir at the *end*
On 2013/06/06 20:41:52, Stephen White wrote:
> Nit: ir -> it?

Stephen- this is already fixed in the new home for this codereview,
https://codereview.chromium.org/15789010/ .  See my note from 
https://codereview.chromium.org/16306010/#msg6 :

"This codereview issue somehow got in a bad state... moved my code over to
https://codereview.chromium.org/15789010/ "

So, please take a look at the other codereview instead.

https://codereview.chromium.org/16306010/diff/20002/tools/rebaseline.py#newco...
tools/rebaseline.py:214: json_url = '/'.join([self._json_base_url,
On 2013/06/06 20:41:52, Stephen White wrote:
> Is there a way to force an override of the json file, and just grab what was
> explicitly requested directly, as it used to work? Otherwise if the json
> generator failed for some reason, you won't be able to rebaseline until the
> upstream script runs (I know, I'm being pessimistic).

Pessimism is underrated.

I am a bit anxious about relying fully on the json, too.  But I think that's
what we have to do, because before long we will NEED the json file to tell us
the hash digest of the actual image result that we should download.  See step 2
in https://goto.google.com/ChecksumTransitionDetail .

The good news is, the advantages of the checksum-based system will outweigh the
generalized pessimistic fears that you and I share.

In fact, as of Step 6, the rebaseline script will ONLY copy checksums around, it
won't have to download images at all (except for viewing pixel diffs as needed
to approve the rebaseline).

Let me know if you think this is problematic... if so, maybe we can come up with
a backup system.  But it would be tricky.