new tool: download all GM images for a given builder, ready for skdiff

gm/rebaseline_server/url_or_path.py

+#!/usr/bin/python

[rmistry, 2014/01/22 22:00:30]

The file name is confusing to me, how about something like
'url_filepath_utils.py'.
How about creating two subclasses UrlUtils and FilepathUtils (or better names)
and a factory that instantiates and returns the right one to the caller based on
is_url == True. That would avoid the many 'if is_url'. All these classes could
live in the same module.

[epoger, 2014/01/23 05:00:53]

Good thoughts; let's discuss live.

One more option to consider: throw away all of this module except for
create_filepath_url(), and use URLs everywhere.  If we ever want to access a
file on local disk, call create_filepath_url() first thing to get the filepath
in URL form.

I think that would work, perhaps with a performance penalty (copying from one
file to another on local disk will require a trip through urllib).

[rmistry, 2014/01/23 12:48:23]

Feel free to grab me anytime today.

[epoger, 2014/01/23 15:34:19]

Discussed live: I will go with file:/// URLs, and get rid of most of this file.

[epoger, 2014/01/24 02:00:46]

Done.  I use file:/// URLs for the unittest.  I kept create_filepath_url() in
the "real code" (not the unittest), in case we want to enable reading results
from local disk (which we would like the tool to be able to do).

+
+"""
+Copyright 2014 Google Inc.
+
+Use of this source code is governed by a BSD-style license that can be
+found in the LICENSE file.
+
+Utility module for handling paths that may be either URL paths or local
+filepaths.
+
+TODO(epoger): Move this module to some common place so we can use it more widely
+"""
+
+# System-level imports
+import contextlib
+import os
+import posixpath
+import shutil
+import urllib
+import urllib2
+import urlparse
+
+def is_url(path):
+  """ Returns true if path looks like a URL. """
+  scheme = urlparse.urlparse(path).scheme

[rmistry, 2014/01/22 22:00:30]

return True if urlparse.urlparse(path).scheme else False
or
return urlparse.urlparse(path).scheme != None (assuming scheme is None for file
paths).

[epoger, 2014/01/23 05:00:53]

Done.

+  if scheme:
+    return True
+  else:
+    return False
+
+def join(first, *remaining):
+  """ Joins two or more URL or pathname components.  If the first component
+  looks like a valid URL, we assemble a URL; otherwise, we assemble it as
+  a pathname to a local file.
+
+  Args:
+    first: string; first component of the URL/path
+    remaining: variable number of strings; remaining URL/path components
+
+  Returns:
+    The complete URL or filename path.
+  """
+  if is_url(first):
+    return posixpath.join(first, *remaining)
+  else:
+    return os.path.join(first, *remaining)
+
+def create_filepath_url(filepath):
+  """ Returns a file:/// URL pointing at the given filepath on local disk.
+
+  Args:
+    filepath: string; path to a file on local disk (may be absolute or relative,
+        and the file does not need to exist)
+
+  Returns:
+    A file:/// URL pointing at the file.  Regardless of whether filepath was
+        specified as a relative or absolute path, the URL will contain an
+        absolute path to the file.
+
+  Raises:
+    An Exception, if filepath is already a URL.
+  """
+  if is_url(filepath):
+    raise Exception('"%s" is already a URL' % filepath)
+  return urlparse.urljoin(
+      'file:', urllib.pathname2url(os.path.abspath(filepath)))
+
+def read_as_string(path):
+  """ Reads in the full contents of the URL or filepath 'path', returning those
+  contents as a single string.
+
+  Args:
+    path: string; complete URL or filepath (if filepath, may be absolute or
+        relative)
+
+  Returns:
+    Complete contents of the URL or filepath as a string.
+
+  Raises:
+    Some subclass of Exception if unable to read the contents (URL/file not
+    found, etc.)
+  """
+  if is_url(path):
+    try:
+      return urllib2.urlopen(path).read()
+    except Exception, e:
+      e.msg += ' for URL ' + path
+      raise
+  else:
+    return open(path, 'r').read()
+
+def copy_contents(source_path, dest_path, create_subdirs_if_needed=False):
+  """ Copies the full contents of the URL or filepath 'source_path' into
+  filepath 'dest_path'.
+
+  Args:
+    source_path: string; complete URL or filepath to read from (if filepath,
+        may be absolute or relative)
+    dest_path: string; complete filepath to write to (may be absolute or
+        relative)
+    create_subdirs_if_needed: boolean; whether to create subdirectories as
+        needed to create dest_path
+
+  Raises:
+    Some subclass of Exception if unable to read source_path (URL/file not
+    found, etc.) or write dest_path.
+  """
+  if create_subdirs_if_needed:
+    dest_dir = os.path.dirname(dest_path)
+    if not os.path.exists(dest_dir):
+      os.makedirs(dest_dir)
+  if is_url(source_path):
+    with contextlib.closing(urllib.urlopen(source_path)) as source_handle:
+      with open(dest_path, 'wb') as dest_handle:
+        shutil.copyfileobj(fsrc=source_handle, fdst=dest_handle)
+  else:
+    shutil.copyfile(source_path, dest_path)