depot_tools: Remove third_party/gsutil

third_party/gsutil/ReleaseNotes.txt

-Release 3.25 (release-date: 2013-02-21)
-
-
-Bug Fixes
----------
-
-  - Fixed two version-specific URI bugs:
-
-    1. gsutil cp -r gs://bucket1 gs://bucket2 would create objects in bucket2
-       with names corresponding to version-specific URIs in bucket1 (e.g.,
-       gs://bucket2/obj#1361417568482000, where the "#1361417568482000" part was
-       part of the object name, not the object's generation).
-
-       This problem similarly caused gsutil cp -r gs://bucket1 ./dir to create
-       files names corresponding to version-specific URIs in bucket1.
-
-    2. gsutil rm -a gs://bucket/obj would attempt to delete the same object
-       twice, getting a NoSuchKey error on the second attempt.
-
-
-================================================================================
-
-
-Release 3.24 (release-date: 2013-02-19)
-
-
-Bug Fixes
----------
-
-  - Fixed bug that caused attempt to dupe-encode a unicode filename.
-
-
-Other Changes
----------
-
-  - Refactored retry logic from setmeta and chacl to use @Retry decorator.
-
-  - Moved @Retry decorator to third_party.
-
-  - Fixed flaky tests.
-
-
-================================================================================
-
-
-Release 3.23 (release-date: 2013-02-16)
-
-
-Bug Fixes
----------
-
-  - Make version-specific URI parsing more robust. This fixes a bug where
-    listing buckets in certain cases would result in the error
-    'BucketStorageUri' object has no attribute 'version_specific_uri'
-
-
-================================================================================
-
-
-Release 3.22 (release-date: 2013-02-15)
-
-
-New Features
-------------
-
-  - Implemented new chacl command, which makes it easy to add and remove bucket
-    and object ACL grants without having to edit XML (like the older setacl
-    command).
-
-  - Implemented new "daisy-chain" copying mode, which allows cross-provider
-    copies to run without buffering to local disk, and to use resumable uploads.
-    This copying mode also allows copying between locations and between storage
-    classes, using the new gsutil cp -D option. (Daisy-chain copying is the
-    default when copying between providers, but must be explicitly requested for
-    the other cases to keep costs and performance expectations clear.)
-
-  - Implemented new perfdiag command to run a diagnostic test against
-    a bucket, collect system information, and report results. Useful
-    when working with Google Cloud Storage team to resolve questions
-    about performance.
-
-  - Added SIGQUIT (^\) handler, to allow breakpointing a running gsutil.
-
-
-Bug Fixes
----------
-
-  - Fixed bug where gsutil setwebcfg signature didn't match with
-    HMAC authentication.
-
-  - Fixed ASCII codec decode error when constructing tracker filename
-    from non-7bit ASCII intput filename.
-
-  - Changed boto auth plugin framework to allow multiple plugins
-    supporting requested capability, which fixes gsutil exception
-    that used to happen where a GCE user had a service account
-    configured and then ran gsutil config.
-
-  - Changed Command.Apply method to be resilient to name expansion
-    exceptions. Before this change, if an exception was raised
-    during iteration of NameExpansionResult, the parent process
-    would immediately stop execution, causing the
-    _EOF_NAME_EXPANSION_RESULT to never be sent to child processes.
-    This resulted in the process hanging forever.
-
-  - Fixed various bugs for gsutil running on Windows:
-    - Fixed various places from a hard-coded '/' to os.sep.
-    - Fixed a bug in the cp command where it was using the destination
-      URI's .delim property instead of the source URI.
-    - Fixed a bug in the cp command's _SrcDstSame function by
-      simplifying it to use os.path.normpath.
-    - Fixed windows bug in tests/util.py _NormalizeURI function.
-    - Fixed ZeroDivisionError sometimes happening during unit tests
-      on Windows.
-
-  - Fixed gsutil rm bug that caused exit status 1 when encountered
-    non-existent URI.
-
-  - Fixed support for gsutil cp file -.
-
-  - Added preconditions and retry logic to setmeta command, to
-    enforce concurrency control.
-
-  - Fixed bug in copying subdirs to subdirs.
-
-  - Fixed cases where boto debug_level caused too much or too little
-    logging:
-    - resumable and one-shot uploads weren't showing response headers
-      when connection.debug > 0.
-    - payload was showing up in debug output when connection.debug
-      < 4 for streaming uploads.
-
-  - Removed XML parsing from setacl. The previous implementation
-    relied on loose XML handling, which could truncate what it sends
-    to the service, allowing invalid XML to be specified by the
-    user. Instead now the ACL XML is passed verbatim and we rely
-    on server-side schema enforcement.
-
-  - Added user-agent header to resumable uploads.
-
-  - Fixed reporting bits/s when it was really bytes/s.
-
-  - Changed so we now pass headers with API version & project ID
-    to create_bucket().
-
-  - Made "gsutil rm -r gs://bucket/folder" remove xyz_$folder$ object
-    (which is created by various GUI tools).
-
-  - Fixed bug where gsutil binary was shipped with protection 750
-    instead of 755.
-
-
-Other Changes
----------
-
-  - Reworked versioned object handling:
-    - Removed need for commands to specify -v option to parse
-      versions. Versioned URIs are now uniformly handled by all
-      commands.
-    - Refactored StorageUri parsing that had been split across
-      storage_uri and conveience; made versioned URIs render with
-      version string so StorageUri is round-trippable (boto change).
-    - Implemented gsutil cp -v option for printing the version-specific
-      URI that was just created.
-    - Added error detail for attempt to delete non-empty versioned
-      bucket. Also added versioning state to ls -L -b gs://bucket
-      output.
-    - Changed URI parsing to use pre-compiled regex's.
-    - Other bug fixes.
-
-  - Rewrote/deepened/improved various parts of built-in help:
-    - Updated 'gsutil help dev'.
-    - Fixed help command handling when terminal does not have the
-      number of rows set.
-    - Rewrote versioning help.
-    - Added gsutil help text for common 403 AccountProblem error.
-    - Added text to 'gsutil help dev' about legal agreement needed
-      with code submissions.
-    - Fixed various other typos.
-    - Updated doc for cp command regarding metadata not being
-      preserved when copying between providers.
-    - Fixed gsutil ls command documentation typo for the -L option.
-    - Added HTTP scheme to doc/examples for gsutil setcors command.
-    - Changed minimum version in documentation from 2.5 to 2.6 since
-      gsutil no longer works in Python 2.5.
-    - Cleaned up/clarify/deepen various other parts of gsutil
-      built-in documentation.
-
-  - Numerous improvements to testing infrastructure:
-    - Completely refactored infrastructure, allowing deeper testing
-      and more readable test code, and enabling better debugging
-      output when tests fail.
-    - Moved gslib/test_*.py unit tests to gslib/tests module.
-    - Made all tests (unit and integration, per-command and modules
-      (like naming) run from single gsutil test command.
-    - Moved TempDir functions from GsUtilIntegrationTestCase to
-      GsUtilTestCase.
-    - Made test runner message show the test function being run.
-    - Added file path support to ObjectToURI function.
-    - Disabled the test command if running on Python 2.6 and unittest2
-      is not available instead of breaking all of gsutil.
-    - Changed to pass GCS V2 API and project_id from boto config
-      if necessary in integration_testcase#CreateBucket().
-    - Fixed unit tests by using a GS-specific mocking class to
-      override the S3 provider.
-    - Added friendlier error message if test path munging fails.
-    - Fixed bug where gsutil test only cleaned up first few test files.
-    - Implemented setacl integration tests.
-    - Implemented StorageUri parsing unit tests.
-    - Implemented test for gsutil cp -D.
-    - Implemented setacl integration tests.
-    - Implemented tests for reading and seeking past end of file.
-    - Implemented and tests for it in new tests module.
-    - Changed cp tests that don't specify a Content-Type to check
-      for new binary/octet-stream default instead of server-detected
-      mime type.
-
-  - Changed gsutil mv to allow moving local files/dirs to the cloud.
-    Previously this was disallowed in the belief we should be
-    conservative about deleting data from local disk but there are
-    legitimate use cases for moving data from a local dir to the
-    cloud, it's clear to the user this would remove data from the
-    local disk, and allowing it makes the tool behavior more
-    consistent with what users would expect.
-
-  - Changed gsutil update command to insist on is_secure and
-    https_validate_certificates.
-
-  - Fixed release no longer to include extraneous boto dirs in
-    top-level of gsutil distribution (like bin/ and docs/).
-
-  - Changed resumable upload threshold from 1 MB to 2 MB.
-
-  - Removed leftover cloudauth and cloudreader dirs. Sample code
-    now lives at https://github.com/GoogleCloudPlatform.
-
-  - Updated copyright notice on code files.
-
-
-================================================================================
-
-
-Release 3.21 (release-date: 2012-12-10)
-
-New Features
-------------
-
-  - Added the ability for the cp command to continue even if there is an
-    error. This can be activated with the -c flag.
-
-  - Added support for specifying src args for gsutil cp on stdin (-I option)
-
-
-Bug Fixes
----------
-
-  - Fixed gsutil test cp, which assumed it was run from gsutil install dir.
-
-  - Mods so we send generation subresource only when user requested
-    version parsing (-v option for cp and cat commands).
-
-
-Other Changes
--------------
-
-  - Updated docs about using setmeta with versioning enabled.
-
-  - Changed GCS endpoint in boto to storage.googleapis.com.
-
-
-================================================================================
-
-
-Release 3.20 (release-date: 2012-11-30)
-
-New Features
-------------
-
-  - Added a noclobber (-n) setting for the cp command. Existing objects/files
-    will not be overwritten when using this setting.
-
-
-Bug Fixes
----------
-
-  - Fixed off-by-one error when reporting bytes transferred.
-
-
-Other Changes
--------------
-
-  - Improved versioning support for the remove command.
-
-  - Improved test runner support.
-
-
-================================================================================
-
-
-Release 3.19 (release-date: 2012-11-26)
-
-New Features
-------------
-  - Added support for object versions.
-
-  - Added support for storage classes (including Durable Reduced Availability).
-
-
-Bug Fixes
----------
-  - Fixed problem where cp -q prevented resumable uploads from being performed.
-
-  - Made setwebcfg and setcors tests robust wrt XML formatting variation.
-
-
-Other Changes
--------------
-
-  - Incorporated vapier@ mods to make version command not fail if CHECKSUM file
-    missing.
-
-  - Refactored gsutil such that most functionality exists in boto.
-
-  - Updated gsutil help dev instructions for how to check out source.
-
-
-================================================================================
-
-
-Release 3.18 (release-date: 2012-09-19)
-
-Bug Fixes
----------
-
-  - Fixed resumable upload boundary condition when handling POST request
-    when server already has complete file, which resulted in an infinite
-    loop that consumed 100% of the CPU.
-
-  - Fixed one more place that outputted progress info when gsutil cp -q
-    specified (during streaming uploads).
-
-
-Other Changes
--------------
-
-  - Updated help text for "gsutil help setmeta" and "gsutil help metadata", to
-    clarify and deepen parts of the documentation.
-
-
-================================================================================
-
-
-Release 3.17 (release-date: 2012-08-17)
-
-Bug Fixes
----------
-
-  - Fixed race condition when multiple threads attempt to get an OAuth2 refresh
-    token concurrently.
-
-
-Other Changes
--------------
-
-  - Implemented simplified syntax for setmeta command. The old syntax still
-    works but is now deprecated.
-
-  - Added help to gsutil cp -z option, to describe how to change where temp
-    files are written.
-
-
-================================================================================
-
-
-Release 3.16 (release-date: 2012-08-13)
-
-Bug Fixes
----------
-
-  - Added info to built-in help for setmeta command, to explain the syntax
-    needed when running from Windows.
-
-================================================================================
-
-
-Release 3.15 (release-date: 2012-08-12)
-
-New Features
-------------
-  - Implemented gsutil setmeta command.
-
-  - Made gsutil understand bucket subdir conventions used by various tools
-    (like GCS Manager and CloudBerry) so if you cp or mv to a subdir you
-    created with one of those tools it will work as expected.
-
-  - Added support for Windows drive letter-prefaced paths when using Storage
-    URIs.
-
-
-Bug Fixes
----------
-
-  - Fixed performance bug when downloading a large object with Content-
-    Encoding:gzip, where decompression attempted to load the entire object
-    in memory. Also added "Uncompressing" log output if file is larger than
-    50M, to make it clear the download hasn't stalled.
-
-  - Fixed naming bug when performing gsutil mv from a bucket subdir to
-    and existing bucket subdir.
-
-  - Fixed bug that caused cross-provider copies into Google Cloud Storage to
-    fail.
-
-  - Made change needed to make resumable transfer progress messages not print
-    when running gsutil cp -q.
-
-  - Fixed copy/paste error in config file documentation for
-    https_validate_certificates option.
-
-  - Various typo fixes.
-
-Other Changes
--------------
-
-  - Changed gsutil to unset http_proxy environment variable if it's set,
-    because it confuses boto. (Proxies should instead be configured via the
-    boto config file.)
-
-================================================================================
-
-
-Release 3.14 (release-date: 2012-07-28)
-
-New Features
-------------
-  - Added cp -q option, to support quiet operation from cron jobs.
-
-  - Made config command restore backed up file if there was a failure or user
-    hits ^C.
-
-Bug Fixes
----------
-
-  - Fixed bug where gsutil cp -R from a source directory didn't generate
-    correct destination path.
-
-  - Fixed file handle leak in gsutil cp -z
-
-  - Fixed bug that caused cp -a option not to work when copying in the cloud.
-
-  - Fixed bug that caused '/-' to be appended to object name for streaming
-    uploads.
-
-  - Revert incorrect line I changed in previous CL, that attempted to
-    get fp from src_key object. The real fix that's needed is described in
-    http://code.google.com/p/gsutil/issues/detail?id=73.
-
-Other Changes
--------------
-
-  - Changed logging to print "Copying..." and Content-Type on same line;
-    refactored content type and log handling.
-
-
-================================================================================
-
-
-Release 3.13 (release-date: 2012-07-19)
-
-Bug Fixes
----------
-
-  - Included the fix to make 'gsutil config' honor BOTO_CONFIG environment
-    variable (which was intended to be included in Release 3.12)
-
-
-================================================================================
-
-
-Release 3.11 (release-date: 2012-06-28)
-
-New Features
-------------
-
-  - Added support for configuring website buckets.
-
-Bug Fixes
----------
-
-  - Fixed bug that caused simultaneous resumable downloads of the same source
-    object to use the same tracker file.
-
-  - Changed language code spec pointer from Wikipedia to loc.gov (for
-    Content-Language header).
-
-
-================================================================================
-
-
-Release 3.10 (release-date: 2012-06-19)
-
-New Features
-------------
-
-  - Added support for setting and listing Content-Language header.
-
-
-Bug Fixes
----------
-
-  - Fixed bug that caused getacl/setacl commands to get a character encoding
-    exception when ACL content contained content not representable in ISO-8859-1
-    character set.
-
-  - Fixed gsutil update not to fail under Windows exclusive file locking.
-
-  - Fixed gsutil ls -L to continue past 403 errors.
-
-  - Updated gsutil tests and also help dev with instructions on how to run
-    boto tests, based on recent test refactoring done to in boto library.
-
-  - Cleaned up parts of cp help text.
-
-
-================================================================================
-
-
-Release 3.9 (release-date: 2012-05-24)
-
-Bug Fixes
----------
-
-  - Fixed bug that caused extra "file:/" to be included in pathnames with
-    doing gsutil cp -R on Windows.
-
-
-================================================================================
-
-
-Release 3.8 (release-date: 2012-05-20)
-
-Bug Fixes
----------
-
-  - Fixed problem with non-ASCII filename characters not setting encoding before
-    attempting to hash for generating resumable transfer filename.
-
-
-================================================================================
-
-
-Release 3.7 (release-date: 2012-05-11)
-
-Bug Fixes
----------
-
-  - Fixed handling of HTTPS tunneling through a proxy.
-
-
-================================================================================
-
-
-Release 3.6 (release-date: 2012-05-09)
-
-Bug Fixes
----------
-
-  - Fixed bug that caused wildcards spanning directories not to work.
-  - Fixed bug that gsutil cp -z not to find available tmp space correctly
-    under Windows.
-
-
-================================================================================
-
-
-Release 3.5 (release-date: 2012-04-30)
-
-Performance Improvement
------------------------
-
-  - Change by Evan Worley to calculate MD5s incrementally during uploads and
-    downloads. This reduces overall transfer time substantially for large
-    objects.
-
-Bug Fixes
----------
-
-  - Fixed bug where uploading and moving multiple files to a bucket subdirectory
-    didn't work as intended.
-    (http://code.google.com/p/gsutil/issues/detail?id=93).
-  - Fixed bug where gsutil cp -r sourcedir didn't copy to specified subdir
-    if there is only one file in sourcedir.
-  - Fixed bug where tracker file included a timestamp that caused it not to
-    be recognized across sessions.
-  - Fixed bug where gs://bucket/*/dir wildcard matches too many objects.
-  - Fixed documentation errors in help associated with ACLs and projects.
-  - Changed GCS ACL parsing to be case-insensitive.
-  - Changed ls to print error and exit with non-0 status when wildcard matches
-    nothing, to be more consistent with UNIX shell behavior.
-
-
-================================================================================
-
-
-Release 3.4 (release-date: 2012-04-06)
-
-Bug Fixes
----------
-
-  - Fixed problem where resumable uploads/downloads of objects with very long
-    names would generate tracking files with names that exceeded local file
-    system limits, making it impossible to complete resumable transfers for
-    those objects. Solution was to build the tracking file name from a fixed
-    prefix, SHA1 hash of the long filename, epoch timestamp and last 16
-    chars of the long filename, which is guarantee to be a predicable and
-    reasonable length.
-
-  - Fixed minor bug in output from 'gsutil help dev' which advised executing
-    an inconsequential test script (test_util.py).
-
-
-================================================================================
-
-
-Release 3.3 (release-date: 2012-04-03)
-
-Bug Fixes
----------
-
-  - Fixed problem where gsutil ver and debug flags crashed when used
-    with newly generated boto config files.
-
-  - Fixed gsutil bug in windows path handling, and make checksumming work
-    across platforms.
-
-  - Fixed enablelogging to translate -b URI param to plain bucket name in REST
-    API request.
-
-
-================================================================================
-
-
-Release 3.2 (release-date: 2012-03-30)
-
-Bug Fixes
----------
-
-  - Fixed problem where gsutil didn't convert between OS-specific directory
-    separators when copying individually-named files (issue 87).
-
-  - Fixed problem where gsutil ls -R didn't work right if there was a key
-    with a leading path (like /foo/bar/baz)
-
-
-================================================================================
-
-
-Release 3.1 (release-date: 2012-03-20)
-
-Bug Fixes
----------
-
-  - Removed erroneous setting of Content-Encoding when a gzip file is uploaded
-    (vs running gsutil cp -z, when Content-Encoding should be set). This
-    error caused users to get gsutil.tar.gz file uncompressed by the user
-    agent (like wget) while downloading, making the file appear to be of the
-    wrong size/content.
-
-  - Fixed handling of gsutil help for Windows (previous code depended on
-    termios and fcntl libs, which are Linux/MacOS-specific).
-
-
-================================================================================
-
-
-Release 3.0 (release-date: 2012-03-20)
-
-
-Important Notes
----------------
-
-  - Backwards-incompatible wildcard change
-
-      The '*' wildcard now only matches objects within a bucket directory. If
-      you have scripts that depend on being able to match spanning multiple
-      directories you need to use '**' instead. For example, the command:
-
-        gsutil cp gs://bucket/*.txt
-
-      will now only match .txt files in the top-level directory.
-
-        gsutil cp gs://bucket/**.txt
-
-      will match across all directories.
-
-  - gsutil ls now lists one directory at a time. If you want to list all objects
-    in a bucket, you can use:
-
-        gsutil ls gs://bucket/**
-
-    or:
-
-        gsutil ls -R gs://bucket
-
-
-New Features
-------------
-
-  - Built-in help for all commands and many additional topics. Try
-    "gsutil help" for a list of available commands and topics.
-
-  - A new hierarchical file tree abstraction layer, which makes the flat bucket
-    name space look like a hierarchical file tree. This makes several things
-    possible:
-    - copying data to/from bucket sub-directories (see “gsutil help cp”).
-    - distributing large uploads/downloads across many machines
-      (see “gsutil help cp”)
-    - renaming bucket sub-directories (see “gsutil help mv”).
-    - listing individual bucket sub-directories and for listing directories
-      recursively (see “gsutil help ls”).
-    - setting ACLs for objects in a sub-directory (see “gsutil help setacl”).
-
-  - Support for per-directory (*) and recursive (**) wildcards. Essentially,
-    ** works the way * did in previous gsutil releases, and * now behaves
-    consistently with how it works in command interpreters (like bash). The
-    ability to specify directory-only wildcards also enables a number of use
-    cases, such as distributing large uploads/downloads by wildcarded name. See
-    "gsutil help wildcards" for details.
-
-  - Support for Cross-Origin Resource Sharing (CORS) configuration. See "gsutil
-    help cors" for details.
-
-  - Support for multi-threading and recursive operation for setacl command
-    (see “gsutil help setacl”).
-
-  - Ability to use the UNIX 'file' command to do content type recognition as
-    an alternative to filename extensions. 
-
-  - Introduction of new end-to-end test suite.
-
-  - The gsutil version command now computes a checksum of the code, to detect
-    corruption and local modification when assisting with technical support.
-
-  - The gsutil update command is no longer beta/experimental, and now also
-    supports updating from named URIs (for early/test releases).
-
-  - Changed gsutil ls -L to also print Content-Disposition header.
-
-
-Bug Fixes
----------
-
-  - The gsutil cp -t option previously didn't work as documented, and instead
-    Content-Type was always detected based on filename extension. Content-Type
-    detection is now the default, the -t option is deprecated (to be removed in
-    the future), and specifying a -h Content-Type header now correctly overrides
-    the filename extension based handling. For details see "gsutil help
-    metadata".
-
-  - Fixed bug that caused multi-threaded mv command not to percolate failures
-    during the cp phase to the rm phase, which could under some circumstances
-    cause data that was not copied to be deleted.
-
-  - Fixed bug that caused gsutil to use GET for ls -L requests. It now uses HEAD
-    for ls -L requests, which is more efficient and faster.
-
-  - Fixed bug that caused gsutil not to preserve metadata during
-    copy-in-the-cloud.
-
-  - Fixed bug that prevented setacl command from allowing DisplayName's in ACLs.
-
-  - Fixed bug that caused gsutil/boto to suppress consecutive slashes in path
-    names.
-
-  - Fixed spec-non-compliant URI construction for resumable uploads.
-
-  - Fixed bug that caused rm -f not to work.
-
-  - Fixed UnicodeEncodeError that happened when redirecting gsutil ls output
-    to a file with non-ASCII object names.
-
-
-Other Changes
--------------
-
-  - UserAgent sent in HTTP requests now includes gsutil version number and OS
-    name.
-
-  - Starting with this release users are able to get individual named releases
-    from version-named objects: gs://pub/gsutil_<version>.tar.gz
-    and gs://pub/gsutil_<version>.zip. The version-less counterparts
-    (gs://pub/gsutil.tar.gz and gs://pub/gsutil.zip) will contain the latest
-    release. Also, the gs://pub bucket is now publicly readable (so, anyone
-    can list its contents).
-
-
-================================================================================
-
-Release 2.0 (release-date: 2012-01-13)
-
-
-New Features
-------------
-
-  - Support for for two new installation modes: enterprise and RPM.
-    Customers can now install gsutil one of three ways:
-
-    - Individual user mode (previously the only available mode): unpacking from
-      a gzipped tarball (gs://pub/gsutil.tar.gz) or zip file
-      (gs://pub/gsutil.zip) and running the gsutil command in place in the
-      unpacked gsutil directory.
-
-    - Enterprise mode (new): unpacking as above, and then running the setup.py
-      script in the unpacked gsutil directory. This allows a systems
-      administrator to install gsutil in a central location, using the Python
-      distutils facility. This mode is supported only on Linux and MacOS.
-
-    - RPM mode (new). A RedHat RPM can be built from the gsutil.spec.in file
-      in the unpacked gsutil directory, allowing it to be installed as part of
-      a RedHat build.
-
-  - Note: v2.0 is the first numbered gsutil release. Previous releases
-    were given timestamps for versions. Numbered releases enable downstream
-    package builds (like RPMs) to define dependencies more easily.
-    This is also the first version where we began including release notes.