Scripts to download files from google storage based on sha1 sums

third_party/gsutil/ReleaseNotes.txt

Index: third_party/gsutil/ReleaseNotes.txt
diff --git a/third_party/gsutil/ReleaseNotes.txt b/third_party/gsutil/ReleaseNotes.txt
new file mode 100644
index 0000000000000000000000000000000000000000..49b56aa770aa2501e77dbc62b70a4b0bb52e7a07
--- /dev/null
+++ b/third_party/gsutil/ReleaseNotes.txt
@@ -0,0 +1,581 @@
+Release 3.21 (release-date: 2012-12-10)
+
+New Features
+------------
+
+  - Added the ability for the cp command to continue even if there is an
+    error. This can be activated with the -c flag.
+
+  - Added support for specifying src args for gsutil cp on stdin (-I option)
+
+
+Bug Fixes
+---------
+
+  - Fixed gsutil test cp, which assumed it was run from gsutil install dir.
+
+  - Mods so we send generation subresource only when user requested
+    version parsing (-v option for cp and cat commands).
+
+
+Other Changes
+-------------
+
+  - Updated docs about using setmeta with versioning enabled.
+
+
+================================================================================
+
+
+Release 3.20 (release-date: 2012-11-30)
+
+New Features
+------------
+
+  - Added a noclobber (-n) setting for the cp command. Existing objects/files
+    will not be overwritten when using this setting.
+
+
+Bug Fixes
+---------
+
+  - Fixed off-by-one error when reporting bytes transferred.
+
+
+Other Changes
+-------------
+
+  - Improved versioning support for the remove command.
+
+  - Improved test runner support.
+
+
+================================================================================
+
+
+Release 3.19 (release-date: 2012-11-26)
+
+New Features
+------------
+  - Added support for object versions.
+
+  - Added support for storage classes (including Durable Reduced Availability).
+
+
+Bug Fixes
+---------
+  - Fixed problem where cp -q prevented resumable uploads from being performed.
+
+  - Made setwebcfg and setcors tests robust wrt XML formatting variation.
+
+
+Other Changes
+-------------
+
+  - Incorporated vapier@ mods to make version command not fail if CHECKSUM file
+    missing.
+
+  - Refactored gsutil such that most functionality exists in boto.
+
+  - Updated gsutil help dev instructions for how to check out source.
+
+
+================================================================================
+
+
+Release 3.18 (release-date: 2012-09-19)
+
+Bug Fixes
+---------
+
+  - Fixed resumable upload boundary condition when handling POST request
+    when server already has complete file, which resulted in an infinite
+    loop that consumed 100% of the CPU.
+
+  - Fixed one more place that outputted progress info when gsutil cp -q
+    specified (during streaming uploads).
+
+
+Other Changes
+-------------
+
+  - Updated help text for "gsutil help setmeta" and "gsutil help metadata", to
+    clarify and deepen parts of the documentation.
+
+
+================================================================================
+
+
+Release 3.17 (release-date: 2012-08-17)
+
+Bug Fixes
+---------
+
+  - Fixed race condition when multiple threads attempt to get an OAuth2 refresh
+    token concurrently.
+
+
+Other Changes
+-------------
+
+  - Implemented simplified syntax for setmeta command. The old syntax still
+    works but is now deprecated.
+
+  - Added help to gsutil cp -z option, to describe how to change where temp
+    files are written.
+
+
+================================================================================
+
+
+Release 3.16 (release-date: 2012-08-13)
+
+Bug Fixes
+---------
+
+  - Added info to built-in help for setmeta command, to explain the syntax
+    needed when running from Windows.
+
+================================================================================
+
+
+Release 3.15 (release-date: 2012-08-12)
+
+New Features
+------------
+  - Implemented gsutil setmeta command.
+
+  - Made gsutil understand bucket subdir conventions used by various tools
+    (like GCS Manager and CloudBerry) so if you cp or mv to a subdir you
+    created with one of those tools it will work as expected.
+
+  - Added support for Windows drive letter-prefaced paths when using Storage
+    URIs.
+
+
+Bug Fixes
+---------
+
+  - Fixed performance bug when downloading a large object with Content-
+    Encoding:gzip, where decompression attempted to load the entire object
+    in memory. Also added "Uncompressing" log output if file is larger than
+    50M, to make it clear the download hasn't stalled.
+
+  - Fixed naming bug when performing gsutil mv from a bucket subdir to
+    and existing bucket subdir.
+
+  - Fixed bug that caused cross-provider copies into Google Cloud Storage to
+    fail.
+
+  - Made change needed to make resumable transfer progress messages not print
+    when running gsutil cp -q.
+
+  - Fixed copy/paste error in config file documentation for
+    https_validate_certificates option.
+
+  - Various typo fixes.
+
+Other Changes
+-------------
+
+  - Changed gsutil to unset http_proxy environment variable if it's set,
+    because it confuses boto. (Proxies should instead be configured via the
+    boto config file.)
+
+================================================================================
+
+
+Release 3.14 (release-date: 2012-07-28)
+
+New Features
+------------
+  - Added cp -q option, to support quiet operation from cron jobs.
+
+  - Made config command restore backed up file if there was a failure or user
+    hits ^C.
+
+Bug Fixes
+---------
+
+  - Fixed bug where gsutil cp -R from a source directory didn't generate
+    correct destination path.
+
+  - Fixed file handle leak in gsutil cp -z
+
+  - Fixed bug that caused cp -a option not to work when copying in the cloud.
+
+  - Fixed bug that caused '/-' to be appended to object name for streaming
+    uploads.
+
+  - Revert incorrect line I changed in previous CL, that attempted to
+    get fp from src_key object. The real fix that's needed is described in
+    http://code.google.com/p/gsutil/issues/detail?id=73.
+
+Other Changes
+-------------
+
+  - Changed logging to print "Copying..." and Content-Type on same line;
+    refactored content type and log handling.
+
+
+================================================================================
+
+
+Release 3.13 (release-date: 2012-07-19)
+
+Bug Fixes
+---------
+
+  - Included the fix to make 'gsutil config' honor BOTO_CONFIG environment
+    variable (which was intended to be included in Release 3.12)
+
+
+================================================================================
+
+
+Release 3.11 (release-date: 2012-06-28)
+
+New Features
+------------
+
+  - Added support for configuring website buckets.
+
+Bug Fixes
+---------
+
+  - Fixed bug that caused simultaneous resumable downloads of the same source
+    object to use the same tracker file.
+
+  - Changed language code spec pointer from Wikipedia to loc.gov (for
+    Content-Language header).
+
+
+================================================================================
+
+
+Release 3.10 (release-date: 2012-06-19)
+
+New Features
+------------
+
+  - Added support for setting and listing Content-Language header.
+
+
+Bug Fixes
+---------
+
+  - Fixed bug that caused getacl/setacl commands to get a character encoding
+    exception when ACL content contained content not representable in ISO-8859-1
+    character set.
+
+  - Fixed gsutil update not to fail under Windows exclusive file locking.
+
+  - Fixed gsutil ls -L to continue past 403 errors.
+
+  - Updated gsutil tests and also help dev with instructions on how to run
+    boto tests, based on recent test refactoring done to in boto library.
+
+  - Cleaned up parts of cp help text.
+
+
+================================================================================
+
+
+Release 3.9 (release-date: 2012-05-24)
+
+Bug Fixes
+---------
+
+  - Fixed bug that caused extra "file:/" to be included in pathnames with
+    doing gsutil cp -R on Windows.
+
+
+================================================================================
+
+
+Release 3.8 (release-date: 2012-05-20)
+
+Bug Fixes
+---------
+
+  - Fixed problem with non-ASCII filename characters not setting encoding before
+    attempting to hash for generating resumable transfer filename.
+
+
+================================================================================
+
+
+Release 3.7 (release-date: 2012-05-11)
+
+Bug Fixes
+---------
+
+  - Fixed handling of HTTPS tunneling through a proxy.
+
+
+================================================================================
+
+
+Release 3.6 (release-date: 2012-05-09)
+
+Bug Fixes
+---------
+
+  - Fixed bug that caused wildcards spanning directories not to work.
+  - Fixed bug that gsutil cp -z not to find available tmp space correctly
+    under Windows.
+
+
+================================================================================
+
+
+Release 3.5 (release-date: 2012-04-30)
+
+Performance Improvement
+-----------------------
+
+  - Change by Evan Worley to calculate MD5s incrementally during uploads and
+    downloads. This reduces overall transfer time substantially for large
+    objects.
+
+Bug Fixes
+---------
+
+  - Fixed bug where uploading and moving multiple files to a bucket subdirectory
+    didn't work as intended.
+    (http://code.google.com/p/gsutil/issues/detail?id=93).
+  - Fixed bug where gsutil cp -r sourcedir didn't copy to specified subdir
+    if there is only one file in sourcedir.
+  - Fixed bug where tracker file included a timestamp that caused it not to
+    be recognized across sessions.
+  - Fixed bug where gs://bucket/*/dir wildcard matches too many objects.
+  - Fixed documentation errors in help associated with ACLs and projects.
+  - Changed GCS ACL parsing to be case-insensitive.
+  - Changed ls to print error and exit with non-0 status when wildcard matches
+    nothing, to be more consistent with UNIX shell behavior.
+
+
+================================================================================
+
+
+Release 3.4 (release-date: 2012-04-06)
+
+Bug Fixes
+---------
+
+  - Fixed problem where resumable uploads/downloads of objects with very long
+    names would generate tracking files with names that exceeded local file
+    system limits, making it impossible to complete resumable transfers for
+    those objects. Solution was to build the tracking file name from a fixed
+    prefix, SHA1 hash of the long filename, epoch timestamp and last 16
+    chars of the long filename, which is guarantee to be a predicable and
+    reasonable length.
+
+  - Fixed minor bug in output from 'gsutil help dev' which advised executing
+    an inconsequential test script (test_util.py).
+
+
+================================================================================
+
+
+Release 3.3 (release-date: 2012-04-03)
+
+Bug Fixes
+---------
+
+  - Fixed problem where gsutil ver and debug flags crashed when used
+    with newly generated boto config files.
+
+  - Fixed gsutil bug in windows path handling, and make checksumming work
+    across platforms.
+
+  - Fixed enablelogging to translate -b URI param to plain bucket name in REST
+    API request.
+
+
+================================================================================
+
+
+Release 3.2 (release-date: 2012-03-30)
+
+Bug Fixes
+---------
+
+  - Fixed problem where gsutil didn't convert between OS-specific directory
+    separators when copying individually-named files (issue 87).
+
+  - Fixed problem where gsutil ls -R didn't work right if there was a key
+    with a leading path (like /foo/bar/baz)
+
+
+================================================================================
+
+
+Release 3.1 (release-date: 2012-03-20)
+
+Bug Fixes
+---------
+
+  - Removed erroneous setting of Content-Encoding when a gzip file is uploaded
+    (vs running gsutil cp -z, when Content-Encoding should be set). This
+    error caused users to get gsutil.tar.gz file uncompressed by the user
+    agent (like wget) while downloading, making the file appear to be of the
+    wrong size/content.
+
+  - Fixed handling of gsutil help for Windows (previous code depended on
+    termios and fcntl libs, which are Linux/MacOS-specific).
+
+
+================================================================================
+
+
+Release 3.0 (release-date: 2012-03-20)
+
+
+Important Notes
+---------------
+
+  - Backwards-incompatible wildcard change
+
+      The '*' wildcard now only matches objects within a bucket directory. If
+      you have scripts that depend on being able to match spanning multiple
+      directories you need to use '**' instead. For example, the command:
+
+        gsutil cp gs://bucket/*.txt
+
+      will now only match .txt files in the top-level directory.
+
+        gsutil cp gs://bucket/**.txt
+
+      will match across all directories.
+
+  - gsutil ls now lists one directory at a time. If you want to list all objects
+    in a bucket, you can use:
+
+        gsutil ls gs://bucket/**
+
+    or:
+
+        gsutil ls -R gs://bucket
+
+
+New Features
+------------
+
+  - Built-in help for all commands and many additional topics. Try
+    "gsutil help" for a list of available commands and topics.
+
+  - A new hierarchical file tree abstraction layer, which makes the flat bucket
+    name space look like a hierarchical file tree. This makes several things
+    possible:
+    - copying data to/from bucket sub-directories (see “gsutil help cp”).
+    - distributing large uploads/downloads across many machines
+      (see “gsutil help cp”)
+    - renaming bucket sub-directories (see “gsutil help mv”).
+    - listing individual bucket sub-directories and for listing directories
+      recursively (see “gsutil help ls”).
+    - setting ACLs for objects in a sub-directory (see “gsutil help setacl”).
+
+  - Support for per-directory (*) and recursive (**) wildcards. Essentially,
+    ** works the way * did in previous gsutil releases, and * now behaves
+    consistently with how it works in command interpreters (like bash). The
+    ability to specify directory-only wildcards also enables a number of use
+    cases, such as distributing large uploads/downloads by wildcarded name. See
+    "gsutil help wildcards" for details.
+
+  - Support for Cross-Origin Resource Sharing (CORS) configuration. See "gsutil
+    help cors" for details.
+
+  - Support for multi-threading and recursive operation for setacl command
+    (see “gsutil help setacl”).
+
+  - Ability to use the UNIX 'file' command to do content type recognition as
+    an alternative to filename extensions. 
+
+  - Introduction of new end-to-end test suite.
+
+  - The gsutil version command now computes a checksum of the code, to detect
+    corruption and local modification when assisting with technical support.
+
+  - The gsutil update command is no longer beta/experimental, and now also
+    supports updating from named URIs (for early/test releases).
+
+  - Changed gsutil ls -L to also print Content-Disposition header.
+
+
+Bug Fixes
+---------
+
+  - The gsutil cp -t option previously didn't work as documented, and instead
+    Content-Type was always detected based on filename extension. Content-Type
+    detection is now the default, the -t option is deprecated (to be removed in
+    the future), and specifying a -h Content-Type header now correctly overrides
+    the filename extension based handling. For details see "gsutil help
+    metadata".
+
+  - Fixed bug that caused multi-threaded mv command not to percolate failures
+    during the cp phase to the rm phase, which could under some circumstances
+    cause data that was not copied to be deleted.
+
+  - Fixed bug that caused gsutil to use GET for ls -L requests. It now uses HEAD
+    for ls -L requests, which is more efficient and faster.
+
+  - Fixed bug that caused gsutil not to preserve metadata during
+    copy-in-the-cloud.
+
+  - Fixed bug that prevented setacl command from allowing DisplayName's in ACLs.
+
+  - Fixed bug that caused gsutil/boto to suppress consecutive slashes in path
+    names.
+
+  - Fixed spec-non-compliant URI construction for resumable uploads.
+
+  - Fixed bug that caused rm -f not to work.
+
+  - Fixed UnicodeEncodeError that happened when redirecting gsutil ls output
+    to a file with non-ASCII object names.
+
+
+Other Changes
+-------------
+
+  - UserAgent sent in HTTP requests now includes gsutil version number and OS
+    name.
+
+  - Starting with this release users are able to get individual named releases
+    from version-named objects: gs://pub/gsutil_<version>.tar.gz
+    and gs://pub/gsutil_<version>.zip. The version-less counterparts
+    (gs://pub/gsutil.tar.gz and gs://pub/gsutil.zip) will contain the latest
+    release. Also, the gs://pub bucket is now publicly readable (so, anyone
+    can list its contents).
+
+
+================================================================================
+
+Release 2.0 (release-date: 2012-01-13)
+
+
+New Features
+------------
+
+  - Support for for two new installation modes: enterprise and RPM.
+    Customers can now install gsutil one of three ways:
+
+    - Individual user mode (previously the only available mode): unpacking from
+      a gzipped tarball (gs://pub/gsutil.tar.gz) or zip file
+      (gs://pub/gsutil.zip) and running the gsutil command in place in the
+      unpacked gsutil directory.
+
+    - Enterprise mode (new): unpacking as above, and then running the setup.py
+      script in the unpacked gsutil directory. This allows a systems
+      administrator to install gsutil in a central location, using the Python
+      distutils facility. This mode is supported only on Linux and MacOS.
+
+    - RPM mode (new). A RedHat RPM can be built from the gsutil.spec.in file
+      in the unpacked gsutil directory, allowing it to be installed as part of
+      a RedHat build.
+
+  - Note: v2.0 is the first numbered gsutil release. Previous releases
+    were given timestamps for versions. Numbered releases enable downstream
+    package builds (like RPMs) to define dependencies more easily.
+    This is also the first version where we began including release notes.