rebaseline_server: download actual-results.json files from GCS instead of SVN

gm/rebaseline_server/download_actuals.py

 #!/usr/bin/python
 
 """
 Copyright 2014 Google Inc.
 
 Use of this source code is governed by a BSD-style license that can be
 found in the LICENSE file.
 
 Download actual GM results for a particular builder.
 """
@@ skipping 87 matching lines @@
       results_of_this_type = actual_results_dict[result_type]
       if not results_of_this_type:
         continue
       for image_name in sorted(results_of_this_type.keys()):
         (test, config) = self._image_filename_re.match(image_name).groups()
         (hash_type, hash_digest) = results_of_this_type[image_name]
         source_url = gm_json.CreateGmActualUrl(
             test_name=test, hash_type=hash_type, hash_digest=hash_digest,
             gm_actuals_root_url=self._gm_actuals_root_url)
         dest_path = os.path.join(dest_dir, config, test + '.png')
-        # TODO(epoger): To speed this up, we should only download files that
-        # we don't already have on local disk.

[epoger, 2014/06/03 20:11:03]

This comment turns out to be misleading... download_actuals.py writes out files
with NON-checksum-based filenames, so we can't easily tell whether the images we
have already downloaded need to be overwritten.

         copy_contents(source_url=source_url, dest_path=dest_path,
                       create_subdirs_if_needed=True)
 
 
 def create_filepath_url(filepath):
   """ Returns a file:/// URL pointing at the given filepath on local disk.
 
   For now, this is only used by unittests, but I anticipate it being useful
   in production, as a way for developers to run rebaseline_server over locally
   generated images.
@@ skipping 35 matching lines @@
   """
   if create_subdirs_if_needed:
     dest_dir = os.path.dirname(dest_path)
     if not os.path.exists(dest_dir):
       os.makedirs(dest_dir)
   with contextlib.closing(urllib.urlopen(source_url)) as source_handle:
     with open(dest_path, 'wb') as dest_handle:
       shutil.copyfileobj(fsrc=source_handle, fdst=dest_handle)
 
 
+def gcs_download_file(source_bucket, source_path, dest_path,
+                      create_subdirs_if_needed=False):
+  """ Downloads a single file from Google Cloud Storage to local disk.
+
+  EPOGER: move into a gcs utility module?

[epoger, 2014/06/03 20:11:03]

I think I should move gcs_download_file() and gcs_list_bucket_contents() into a
new gcs_utils.py module before committing this.  Does that sound good to you?

[borenet, 2014/06/03 20:27:47]

Yes, that SGTM, but what's the difference then between "gcs_utils" and
"gs_utils"?  Should one supercede the other?  This is another module which would
be nice to share between buildbot and skia...

[epoger, 2014/06/03 20:39:04]

In this case I went with "gcs_utils.py" because (a) the public API documentation
refers to Google Cloud Storage, and (b) it seemed helpful to be clear that this
is distinct from the gs_utils.py file in buildbot and not using the gsutil
binary.  But I'm happy calling it just about anything.

In buildbot-land, I think we should try to update gs_utils.py to use these API
calls instead of the gsutil binary.  (Ideally, the callers to it won't have to
change at all, unless they want to.)

I agree that it would be good for the two modules to be shared somehow.  I'll
add a TODO when I create the new module.

+
+  Args:
+    source_bucket: GCS bucket to download the file from
+    source_path: full path (Posix-style) within that bucket
+    dest_path: full path (local-OS-style) on local disk to copy the file to
+    create_subdirs_if_needed: boolean; whether to create subdirectories as
+        needed to create dest_path
+  """
+  source_http_url = posixpath.join(
+      'http://storage.googleapis.com', source_bucket, source_path)
+  copy_contents(source_url=source_http_url, dest_path=dest_path,
+                create_subdirs_if_needed=create_subdirs_if_needed)
+
+
 def gcs_list_bucket_contents(bucket, subdir=None):
   """ Returns files in the Google Cloud Storage bucket as a (dirs, files) tuple.
 
   Uses the API documented at
   https://developers.google.com/storage/docs/json_api/v1/objects/list
 
+  EPOGER: move into a gcs utility module?
+
   Args:
     bucket: name of the Google Storage bucket
     subdir: directory within the bucket to list, or None for root directory
   """
   # The GCS command relies on the subdir name (if any) ending with a slash.
   if subdir and not subdir.endswith('/'):
     subdir += '/'
   subdir_length = len(subdir) if subdir else 0
 
   storage = build_service('storage', 'v1')
@@ skipping 11 matching lines @@
     dir_basename = dir_fullpath[subdir_length:]
     dirs.append(dir_basename[:-1])  # strip trailing slash
   files = []
   for file_properties in results.get('items', []):
     file_fullpath = file_properties['name']
     file_basename = file_fullpath[subdir_length:]
     files.append(file_basename)
   return (dirs, files)
 
 
+def get_builders_list(summaries_bucket=GM_SUMMARIES_BUCKET):
+  """ Returns the list of builders we have actual results for.
+
+  Args:
+    summaries_bucket: Google Cloud Storage bucket containing the summary
+        JSON files
+  """
+  dirs, _ = gcs_list_bucket_contents(bucket=GM_SUMMARIES_BUCKET)
+  return dirs
+
+
 def main():
   parser = optparse.OptionParser()
   required_params = []
   parser.add_option('--actuals-base-url',
                     action='store', type='string',
                     default=DEFAULT_ACTUALS_BASE_URL,
                     help=('Base URL from which to read files containing JSON '
                           'summaries of actual GM results; defaults to '
                           '"%default".'))
   required_params.append('builder')
@@ skipping 15 matching lines @@
   parser.add_option('--json-filename',
                     action='store', type='string',
                     default=DEFAULT_JSON_FILENAME,
                     help=('JSON summary filename to read for each builder; '
                           'defaults to "%default".'))
   parser.add_option('--list-builders', action='store_true',
                     help=('List all available builders.'))
   (params, remaining_args) = parser.parse_args()
 
   if params.list_builders:
-    dirs, _ = gcs_list_bucket_contents(bucket=GM_SUMMARIES_BUCKET)
-    print '\n'.join(dirs)
+    print '\n'.join(get_builders_list())
     return
 
   # Make sure all required options were set,
   # and that there were no items left over in the command line.
   for required_param in required_params:
     if not getattr(params, required_param):
       raise Exception('required option \'%s\' was not set' % required_param)
   if len(remaining_args) is not 0:
     raise Exception('extra items specified in the command line: %s' %
                     remaining_args)
 
   downloader = Download(actuals_base_url=params.actuals_base_url)
   downloader.fetch(builder_name=params.builder,
                    dest_dir=params.dest_dir)
 
 
 
 if __name__ == '__main__':
   main()