Add script to verify that image files exist for every actual_result checksum

Add script to verify that image files exist for every actual_result checksum

Should help with https://code.google.com/p/skia/issues/detail?id=1410
(SkipBuildbotRuns)

R=epoger@google.com

Committed: https://code.google.com/p/skia/source/detail?r=10017

[epoger, 2013-07-10 19:30:44]

looks good at a high level, some quibbles below

https://codereview.chromium.org/18743006/diff/1/tools/verify_images_for_gm_re...
File tools/verify_images_for_gm_results.py (right):

https://codereview.chromium.org/18743006/diff/1/tools/verify_images_for_gm_re...
tools/verify_images_for_gm_results.py:8: """ find_missing_images: Look through
skia-autogen, searching for all checksums
Delete no-longer-accurate script name

https://codereview.chromium.org/18743006/diff/1/tools/verify_images_for_gm_re...
tools/verify_images_for_gm_results.py:19: AUTOGEN_URL =
'http://skia-autogen.googlecode.com/svn/gm-actual'
optional: use argparse with defaults (like
https://code.google.com/p/skia/source/browse/trunk/tools/rebaseline.py ) to make
these overridable at the command line

https://codereview.chromium.org/18743006/diff/1/tools/verify_images_for_gm_re...
tools/verify_images_for_gm_results.py:23: def FileNameToResultType(filename):
please add a brief description of each function

https://codereview.chromium.org/18743006/diff/1/tools/verify_images_for_gm_re...
tools/verify_images_for_gm_results.py:25: for suffix in ['.png', '_8888',
'_565', '_gpu', '_gpudebug', '_msaa4']:
I don't understand why we need this list of suffixes (which seems fragile).  I
think it makes more sense to build the gs_url like we do in
https://code.google.com/p/skia/source/browse/trunk/tools/rebaseline_imagefile...
, and just build up a list of complete gs_urls as we go (rather than
"checksums", which are actually partial GS paths in this case)

https://codereview.chromium.org/18743006/diff/1/tools/verify_images_for_gm_re...
tools/verify_images_for_gm_results.py:104: print 'Missing files:'
please print the number of missing files

[borenet, 2013-07-10 20:36:45]

Uploaded patch set 2.

https://codereview.chromium.org/18743006/diff/1/tools/verify_images_for_gm_re...
File tools/verify_images_for_gm_results.py (right):

https://codereview.chromium.org/18743006/diff/1/tools/verify_images_for_gm_re...
tools/verify_images_for_gm_results.py:8: """ find_missing_images: Look through
skia-autogen, searching for all checksums
On 2013/07/10 19:30:44, epoger wrote:
> Delete no-longer-accurate script name

Done.

https://codereview.chromium.org/18743006/diff/1/tools/verify_images_for_gm_re...
tools/verify_images_for_gm_results.py:19: AUTOGEN_URL =
'http://skia-autogen.googlecode.com/svn/gm-actual'
On 2013/07/10 19:30:44, epoger wrote:
> optional: use argparse with defaults (like
> https://code.google.com/p/skia/source/browse/trunk/tools/rebaseline.py ) to
make
> these overridable at the command line

Mind if I save this for the future?  I'd like to just move on.

https://codereview.chromium.org/18743006/diff/1/tools/verify_images_for_gm_re...
tools/verify_images_for_gm_results.py:23: def FileNameToResultType(filename):
On 2013/07/10 19:30:44, epoger wrote:
> please add a brief description of each function

Done.

https://codereview.chromium.org/18743006/diff/1/tools/verify_images_for_gm_re...
tools/verify_images_for_gm_results.py:25: for suffix in ['.png', '_8888',
'_565', '_gpu', '_gpudebug', '_msaa4']:
On 2013/07/10 19:30:44, epoger wrote:
> I don't understand why we need this list of suffixes (which seems fragile).  I
> think it makes more sense to build the gs_url like we do in
>
https://code.google.com/p/skia/source/browse/trunk/tools/rebaseline_imagefile...
> , and just build up a list of complete gs_urls as we go (rather than
> "checksums", which are actually partial GS paths in this case)

Done.

https://codereview.chromium.org/18743006/diff/1/tools/verify_images_for_gm_re...
tools/verify_images_for_gm_results.py:104: print 'Missing files:'
On 2013/07/10 19:30:44, epoger wrote:
> please print the number of missing files

Done.

[epoger, 2013-07-11 14:14:25]

LGTM

Thanks for doing this!

https://codereview.chromium.org/18743006/diff/1/tools/verify_images_for_gm_re...
File tools/verify_images_for_gm_results.py (right):

https://codereview.chromium.org/18743006/diff/1/tools/verify_images_for_gm_re...
tools/verify_images_for_gm_results.py:19: AUTOGEN_URL =
'http://skia-autogen.googlecode.com/svn/gm-actual'
On 2013/07/10 20:36:45, borenet wrote:
> On 2013/07/10 19:30:44, epoger wrote:
> > optional: use argparse with defaults (like
> > https://code.google.com/p/skia/source/browse/trunk/tools/rebaseline.py ) to
> make
> > these overridable at the command line
> 
> Mind if I save this for the future?  I'd like to just move on.

No problem.

https://codereview.chromium.org/18743006/diff/5001/tools/verify_images_for_gm...
File tools/verify_images_for_gm_results.py (right):

https://codereview.chromium.org/18743006/diff/5001/tools/verify_images_for_gm...
tools/verify_images_for_gm_results.py:1: #!/usr/bin/python
Some high-level, long-term thoughts:

A. This script could be the basis for a tool to "groom" the images stored in
Google Storage (deleting any that aren't referred to anymore, so that it does
not grow without bound).


B. In the long term (not necessary right away), I think this script could
operate more efficiently by being implemented this way:

1. Iterate over all the actual-results.json (and maybe expected-results.json)
files, building up a checksums_needed dictionary like this:

{
  'aaclip': set(['111111', '2222222']),
  'imageblur': set('333333', '444444']),
  ...
}

2. Do a "gsutil ls" of each test directory in Google Storage ('aaclip', then
'imageblur', etc.; this is more efficient than doing a "gsutil ls" of each file)
to build up a checksums_available dictionary like this:

{
  'aaclip': set(['2222222']),
  'imageblur': set('333333', '444444', '555555']),
  ...
}

3. Diff the two dictionaries:

missing_checksums = checksums_needed - checksums_available
checksums_we_can_delete = checksums_available - checksums_needed

https://codereview.chromium.org/18743006/diff/5001/tools/verify_images_for_gm...
tools/verify_images_for_gm_results.py:44: gs_urls: dict; stores Google Storage
URLs as keys and lists of the JSON files
optional: I find that a small example of a valid dictionary is very helpful in
these cases

[borenet, 2013-07-11 18:05:28]

Committed patchset #4 manually as r10017 (presubmit successful).