OLD | NEW |
(Empty) | |
| 1 Release 4.13 (release date: 2015-06-03) |
| 2 ======================================= |
| 3 New Features |
| 4 ------------ |
| 5 - Added -U flag to cp and rsync commands to allow skipping of unsupported |
| 6 object types. |
| 7 - Added support for Google Developer Shell credentials. |
| 8 |
| 9 Bug Fixes |
| 10 --------- |
| 11 - Precondition headers (x-goog-if-...) are now respected for the setmeta |
| 12 command. |
| 13 - Fixed an index out of range error that could occur with an empty |
| 14 parallel composite upload tracker file. |
| 15 - The stat command outputs errors to stderr instead of stdout. |
| 16 - Fixed two possible sources of ResumableUploadStartOverException from |
| 17 httplib2 and oauth2client. |
| 18 - Fixed a bug in the compose command where a missing source object resulted |
| 19 in an error message claiming the destination object was missing. |
| 20 |
| 21 Other Changes |
| 22 ------------- |
| 23 - Added a help section on throttling gsutil. |
| 24 - Resumable uploads will now start over if a PUT to the upload ID returns |
| 25 an HTTP 404. Previously this behavior applied only to an HTTP 410. |
| 26 - XML API resumable uploads now retry on HTTP 429 errors, matching the |
| 27 behavior of JSON API resumable uploads. |
| 28 - Improved response to process kill signals, reducing the likelihood of |
| 29 leaving orphaned child processes and temporary files. |
| 30 - Bucket lifecycle configuration now works for S3. |
| 31 - Removed the deprecated setmeta -n option. |
| 32 |
| 33 |
| 34 Release 4.12 (release date: 2015-04-20) |
| 35 ======================================= |
| 36 New Features |
| 37 ------------ |
| 38 - Added support for JSON-format service account private key files. |
| 39 - Added support for the Rewrite API (JSON API only). This is used for |
| 40 all copies within the Google Cloud and supports copying objects across |
| 41 storage classes and/or locations. |
| 42 |
| 43 Bug Fixes |
| 44 --------- |
| 45 - Fixed a bug that could cause downloads to have a hash mismatch (and deletion |
| 46 of the corrupted file) when resumed across process breaks via a tracker file. |
| 47 |
| 48 Other Changes |
| 49 ------------- |
| 50 - Updated documentation and examples for several topics including |
| 51 acl, cp, dev, signurl, stat, and wildcards. |
| 52 |
| 53 |
| 54 Release 4.11 (release date: 2015-03-10) |
| 55 ======================================= |
| 56 New Features |
| 57 ------------ |
| 58 - Added Nearline storage class support to the mb command. |
| 59 |
| 60 Bug Fixes |
| 61 --------- |
| 62 - Fixed a bug for streaming uploads that could occasionally cause a HTTP 410 |
| 63 from the service or a hash mismatch (and deletion of the corrutped file). |
| 64 - Fixed an OverflowError that occurred when uploading files > 4GiB on a 32-bit |
| 65 operating system. |
| 66 |
| 67 Other Changes |
| 68 ------------- |
| 69 - Added documentation around using the Content-MD5 header to extend integrity |
| 70 checking to include checksums computed by a client-side content pipeline. |
| 71 |
| 72 |
| 73 Release 4.10 (release date: 2015-03-03) |
| 74 ======================================= |
| 75 Bug Fixes |
| 76 --------- |
| 77 - Fixed a bug that could cause undetected data corruption (preserving incorrect |
| 78 data) if a streaming upload encountered a service error on non-8KiB-aligned |
| 79 boundary. |
| 80 - Fixed a bug that caused downloads to be truncated if the connection broke, |
| 81 resulting in a hash mismatch (and deletion of the corrupted file) for that |
| 82 download. |
| 83 - Fixed a format string arguments error that occurred if a download exhausted |
| 84 all retries. |
| 85 |
| 86 Other Changes |
| 87 ------------- |
| 88 - The lifecycle command now accepts JSON input in the form of |
| 89 "{ "lifecycle": { "rule" ..." in addition to "{ "rule": ...". |
| 90 - Improved access token expiry logic for GCE credentials. |
| 91 |
| 92 |
| 93 Release 4.9 (release date: 2015-02-13) |
| 94 ======================================= |
| 95 New Features |
| 96 ------------ |
| 97 - When using the JSON API, the ch acl/defacl subcommand now supports |
| 98 project groups via the -p flag. For details, see "gsutil help acl ch". |
| 99 |
| 100 Bug Fixes |
| 101 --------- |
| 102 - Fixed a bug that caused daisy-chain copies (including cross-provider |
| 103 copies) for files large than 100MiB to fail. |
| 104 - Fixed a bug that caused streaming uploads than ran for longer than |
| 105 an hour to fail with HTTP 400s. |
| 106 - Fixed a bug where perfdiag would not properly clean up its test files. |
| 107 - Fixed a bug where using ls with the XML API could mistakenly report bucket |
| 108 configuration as present. |
| 109 |
| 110 Other Changes |
| 111 ------------- |
| 112 - Updated documentation for metadata, retries, security, and subdirs. |
| 113 - Tracker files are no longer written for small downloads. |
| 114 |
| 115 |
| 116 Release 4.8 (release date: 2015-01-23) |
| 117 ======================================= |
| 118 New Features |
| 119 ------------ |
| 120 - gsutil now supports HTTP proxy configuration via the http_proxy, |
| 121 https_proxy, or HTTPS_PROXY environment variables. This configuration |
| 122 is used only if proxy configuration is not present in the .boto |
| 123 configuration file. |
| 124 - gsutil rsync now supports regex-based source and destination URL |
| 125 exclusion via the -x flag. |
| 126 - The rm command now supports arguments on stdin via the -I flag. |
| 127 |
| 128 Bug Fixes |
| 129 --------- |
| 130 - Fixed a bug where perfdiag would fail if netstat was not available. |
| 131 - Fixed a bug where temporary ca_certs files were not being cleaned up. |
| 132 - Fixed a bug in rsync to unnecessarily remove or write objects, in some |
| 133 cases leaving the destination in a non-synchronized state. |
| 134 caused rsync to unnecessarily remove or rewrite objects. |
| 135 - Fixed a bug where rsync temporary listing files were not being |
| 136 cleaned up when the rsync process was killed. |
| 137 - Fixed a bug where rsync would remove destination URLs if listing the |
| 138 source encountered a non-retryable failure (for example, if the source |
| 139 did not exist). |
| 140 - Fixed a bug where mv would fail for some Unicode filenames. |
| 141 - Fixed a bug where mv would remove the source URL after skipping the |
| 142 destination URL. |
| 143 - Fixed a bug that caused daisy chain uploads to hang if the download thread |
| 144 raised an exception. |
| 145 - Fixed a bug where acl ch would return a zero exit code even if it failed. |
| 146 - Fixed a bug that sometimes caused the progress display to render multiple |
| 147 times at the end of an upload or download. |
| 148 |
| 149 Other Changes |
| 150 ------------- |
| 151 - Resumable uploads of files using the JSON API now send their data in a |
| 152 single request, making separate HTTP calls only when resuming is necessary. |
| 153 - The test command now runs tests in parallel by default, and test |
| 154 parallelism on Windows is now supported. |
| 155 - All non-streaming downloads are now resumable (and retryable) by default, |
| 156 regardless of size. |
| 157 - Canned ACLs and canned default object ACLs are now supported in the JSON |
| 158 API (previously they would fall back to using the XML API). |
| 159 - Google Compute Engine service account credential tokens are now cached, |
| 160 avoiding unnecessary refreshes. |
| 161 - Improved detection of the Google Compute Engine metadata server, |
| 162 particularly when using the -m flag for multiprocessing. |
| 163 - Added new help sections about filename encoding and security/privacy |
| 164 considerations. |
| 165 - Download progress is now displayed for small XML API downloads. |
| 166 |
| 167 |
| 168 Release 4.7 (release date: 2014-11-17) |
| 169 ======================================= |
| 170 New Features |
| 171 ------------ |
| 172 - Tab completion now works on gs:// URLs (for Cloud SDK installs only). |
| 173 To install via Cloud SDK, see https://cloud.google.com/sdk/#Quick_Start |
| 174 - Streaming uploads (with source URL "-") using the JSON API now buffer |
| 175 in-memory up to 100MB, allowing large streams to be retried in the event |
| 176 of network flakiness or service errors. |
| 177 - Resumable uploads that receive a fatal service error (typically a 410) |
| 178 are now automatically retried from the beginning. |
| 179 |
| 180 Bug Fixes |
| 181 --------- |
| 182 - Fixed an apitools bug that impacted upload performance and caused |
| 183 "Retrying from byte __ after exception" to print after every 100MiB. |
| 184 - Fixed _$folder$ placeholder object detection on versioned buckets. |
| 185 - Removed an unnecessary credential check on load which increased |
| 186 startup time by over one second in some cases. |
| 187 - SignURL now properly retries when checking if the signed object is |
| 188 readable. |
| 189 - Files with both Content-Encoding and Content-Type gzip are now properly |
| 190 removed when hash validation fails (only one of the two should be set). |
| 191 - The x-goog-if-generation-match:0 header now works properly with the XML API. |
| 192 - Fixed a bug that caused "defacl ch" on a bucket with a private ACL to fail. |
| 193 - The rm command now properly supports precondition headers. |
| 194 - Fixed a bug that caused large streaming uploads to fail with the message |
| 195 "Failure: length too large" when using the JSON API. |
| 196 - Fixed a bug that caused JSON lifecycle configurations with createdBefore |
| 197 conditions to fail with a DecodeError. |
| 198 |
| 199 Other Changes |
| 200 ------------- |
| 201 - Byte counts now display accurate abbreviations of binary sizing. For example, |
| 202 messages previously labeled MB are now properly labeled MiB to indicate |
| 203 2**20 bytes. Only the labeling changed - the actual sizes were always binary. |
| 204 - Improved Cloud SDK integration, including improved error messages |
| 205 and instructions. |
| 206 - The num_retries .boto configuration value now applies to all requests |
| 207 (previously it was ignored for JSON API requests). |
| 208 - rsync now works with non-existent destination subdirectories. |
| 209 - Raised the default resumable upload threshold from 2MB to 8MB to |
| 210 improve performance. |
| 211 - Benign retry messages now print only when debug mode is enabled via the |
| 212 top-level -d flag. |
| 213 - The top-level -q flag now suppresses suggestions to use the -m flag. |
| 214 - Command synopsis is now output when the wrong number of arguments are |
| 215 provided. |
| 216 - Removed dependency on google-api-python-client module, added dependencies on |
| 217 oauth2client and six modules. |
| 218 |
| 219 Release 4.6 (release date: 2014-09-08) |
| 220 ======================================= |
| 221 |
| 222 Bug Fixes |
| 223 --------- |
| 224 - Fixed a TypeError bug that occurred in perfdiag write throughput tests. |
| 225 - Fixed an rsync bug that caused invalid symlinks to abort the transfer |
| 226 even when -e option was specified. |
| 227 - Fixed a perfdiag assumption that ipaddrlist was populated. |
| 228 - Fixed an AttributeError when setting an invalid canned ACL with defacl set. |
| 229 - Fixed a bug where non-resumable uploads would include payload in debug output |
| 230 when for running in debug mode (-D). |
| 231 |
| 232 Other Changes |
| 233 ------------- |
| 234 - Added the proxy_rdns configuration variable for clients that |
| 235 do DNS lookups via a proxy. |
| 236 - Added the state_dir configuration variable for choosing the location of |
| 237 gsutil's internal state files, including resumable transfer tracker files. |
| 238 resumable_tracker_dir configuration variable is now deprecated. |
| 239 - Added DNS, connection latency, and proxy use information to perfdiag |
| 240 command. |
| 241 - perfdiag command will not perform DNS lookups if they are disabled in |
| 242 boto config. |
| 243 - perfdiag command will now only attempt to delete uploaded objects when |
| 244 running write tests. |
| 245 - Added code coverage support to test command. |
| 246 - rsync -d now succeeds on a 404 for a to-be-deleted object (for example, when |
| 247 the object was already deleted by an external process). |
| 248 |
| 249 Release 4.5 (release date: 2014-08-14) |
| 250 ======================================= |
| 251 |
| 252 Bug Fixes |
| 253 --------- |
| 254 - Fixed a bug that caused resumable uploads to restart if gsutil was |
| 255 terminated with CTRL-C. |
| 256 - Fixed a bug in defacl ch command that caused a failure when updating |
| 257 an existing default object ACL entry. |
| 258 - Fixed an invalid literal bug during rsync file listing. |
| 259 - Made several improvements to JSON upload stability, including fixing a bug |
| 260 which occasionally caused resumable upload hashes not to catch up properly. |
| 261 - All JSON calls now have socket timeouts, eliminating hangs under |
| 262 flaky network conditions. |
| 263 - Fixed a bug where acl ch -g AllAuthenticatedUsers would instead add |
| 264 AllUsers. |
| 265 - Fixed a bug that caused object custom metadata not to be preserved when |
| 266 copying in the cloud. |
| 267 - Fixed a bug where du -s did not properly elide subdirectories. |
| 268 |
| 269 Other Changes |
| 270 ------------- |
| 271 - Parallel composite uploads are now disabled by default until crcmod is |
| 272 available in major Linux distributions. To re-enable the setting from |
| 273 prior versions, in the [GSUtil] section of your .boto config file, add: |
| 274 parallel_composite_upload_threshold=150M |
| 275 - Non-wildcarded URLs for existing objects now use Get before trying List |
| 276 (as in gsutil3), and thus are not subject to eventual listing consistency. |
| 277 - gsutil -D now redacts proxy configuration values in the output. |
| 278 |
| 279 Release 4.4 (release date: 2014-07-17) |
| 280 ======================================= |
| 281 |
| 282 New Features |
| 283 ------------ |
| 284 - Added the hash command, which can calculate hashes of local files. |
| 285 gsutil already calculates hashes for integrity checking, but this allows |
| 286 the user to separately calculate the MD5 and CRC32c hashes of a local file. |
| 287 |
| 288 Bug Fixes |
| 289 --------- |
| 290 - Many improvements to JSON API media transfers, including far |
| 291 more robust retry logic. |
| 292 - Fixed "File changed during upload: EOF..." errors on XML resumable uploads. |
| 293 - Fixed rsync command to read and write index files in binary mode. |
| 294 - Fix potential TypeError in _CheckAndHandleCredentialException. |
| 295 - Fixed possible data corruption when using JSON API uploads for |
| 296 small files with lines starting with "From:", which would cause |
| 297 integrity checks to fail. |
| 298 - Fixed gsutil cp to skip directory placeholders when downloading, avoiding |
| 299 "directory exists where the file needs to be created" errors. |
| 300 - Fixed daisy chain cp/rsync for files >= 100MB. |
| 301 - Fixed a bug in JSON proxy support where the proxy info was sometimes unused. |
| 302 - Fixed a bug where an acl get on a private default object ACL returned an |
| 303 error instead of a blank ACL. |
| 304 - Fixed a JSON API issue with large HTTP responses on 32-bit systems. |
| 305 |
| 306 Other Changes |
| 307 ------------- |
| 308 - Improved object listing performance when using the XML API. |
| 309 - Improved various error messages. |
| 310 - Improved progress display during media transfer. |
| 311 - Switched to truncated exponential backoff for retries. |
| 312 - Improved OS-specific ulimit checks. |
| 313 - Added some information such as OS and Cloud SDK wrapping to gsutil version, |
| 314 and changed the output format to be more uniform. |
| 315 - Daisy chain cp/rsync now supports resumable uploads. |
| 316 - Improved proxy support for proxy username and passwords. |
| 317 - x-amz headers are now supported for cp, rsync, and setmeta. x-amz-meta |
| 318 headers continue to be supported as well. |
| 319 |
| 320 Release 4.3 (release date: 2014-06-10) |
| 321 ======================================= |
| 322 |
| 323 Bug Fixes |
| 324 --------- |
| 325 - Fix acl/defacl ch changing the role of an existing group. |
| 326 - Fix unicode and 404 errors when using manifests. |
| 327 - Fix parallelism configuration bug that limited gsutil rsync to two threads |
| 328 and could lead to rsync hangs. "gsutil -m rsync" runs much faster, and rsync |
| 329 uploads of large local files are now faster via parallel composite upload. |
| 330 Parallel composite uploads of large files are also faster. |
| 331 - Fix rsync bug with parallel composite uploads. |
| 332 - Fix TypeError that could occur when running the cp command with no |
| 333 credentials. |
| 334 |
| 335 Other Changes |
| 336 ------------- |
| 337 - Progress indicators for -m cp/rsync commands are now more readable. |
| 338 - Added command being run to gsutil -d/-D output. |
| 339 - Lowered default parallelism for 'gsutil -m test' and added hang detection. |
| 340 |
| 341 Release 4.2 (release date: 2014-06-05) |
| 342 ======================================= |
| 343 |
| 344 New Features |
| 345 ------------ |
| 346 - Added parallel test execution support to test command, ex: "gsutil -m test" |
| 347 |
| 348 Bug Fixes |
| 349 --------- |
| 350 - Fix failure during retry of an XML download. |
| 351 - Moved to boto release 2.29.1 fixing boto authentication erroneously |
| 352 reporting OAuth2 credentials as invalid. |
| 353 - Fix parallel composite uploads when using only a single process and thread. |
| 354 - Fix an invalid seek during daisy chain operation that affected file copy |
| 355 from Google Cloud Storage -> S3 for files greater than 8KB in size. |
| 356 - Fix "gsutil acl ch" with AllUsers or AllAuthenticatedUsers groups. |
| 357 - Fix some copy errors writing new lines to the manifest file. |
| 358 - Fix "gsutil test" return code to properly be 0 on success. |
| 359 |
| 360 Other Changes |
| 361 ------------- |
| 362 - "gsutil cp -z" now ignores whitespace in the extension list. |
| 363 |
| 364 Release 4.1 (release date: 2014-05-28) |
| 365 ======================================= |
| 366 |
| 367 Bug Fixes |
| 368 --------- |
| 369 - Fixed a bug in parallel composite uploads where uploads with |
| 370 existing components would fail. |
| 371 - Moved gcs-oauth2-boto-plugin to version 1.5, fixing a bug in the PyPi gsutil |
| 372 distribution that would cause gsutil to unnecessarily attempt to query |
| 373 the Google Compute Engine metadata service. |
| 374 |
| 375 Other Changes |
| 376 ------------- |
| 377 - Parallel composite uploads no longer specify an if-not-match precondition |
| 378 when uploading component parts. |
| 379 - Parallel composite uploads no longer calculate a CRC32c hash prior to |
| 380 uploading component parts (these are still validated by an MD5 hash). |
| 381 - Removed apitools dependency on gflags. |
| 382 |
| 383 Release 4.0 (release date: 2014-05-27) |
| 384 ======================================= |
| 385 |
| 386 Major New Gsutil Version - Backwards-Incompatible Changes |
| 387 ------------------------------ |
| 388 - The Google Cloud Storage configuration data supported by the acl, cors, |
| 389 and lifecycle commands now uses the JSON format instead of the older XML |
| 390 format. gsutil 4.0 will fail and provide conversion instructions if an XML |
| 391 configuration file is provided as an argument for a gs:// URL. |
| 392 - gsutil no longer accepts arbitrary headers via the global -h flag. |
| 393 Documented headers for gsutil commands are still supported; for the |
| 394 full list of supported headers, see "gsutil help command_opts". |
| 395 - The compose command will now default the destination object's |
| 396 Content-Type to the Content-Type of the first source object if none |
| 397 is provided via the -h global flag. |
| 398 - The long-deprecated -t and -q options have been removed from the cp command. |
| 399 - The perfdiag command no longer supports adding a host header. |
| 400 - Having OAuth2 User Account credentials and OAuth2 Service Account |
| 401 credentials configured simultaneously will now fail with an error message |
| 402 to avoid confusion. Also, a single invalid credential will fail with an |
| 403 error message. See "gsutil help creds" for details. |
| 404 - Bucket relocate scripts have been removed. |
| 405 - Downloading object names ending with '/' is no longer supported to avoid |
| 406 problems this caused for directores using the Google Cloud Console. |
| 407 - rm -r now implies rm -ra (removing all object versions recursively). |
| 408 - All commands using the global -m option or a force option (such as |
| 409 rm -f or cp -c) will now return a non-zero exit code if there are any |
| 410 failures during the operation. |
| 411 - MD5 and CRC32c values are now represented in base64 encoding instead |
| 412 of hex encoding (this includes manifest files). |
| 413 |
| 414 New Features |
| 415 ------------ |
| 416 - The Google Cloud Storage JSON API (v1) is now the default API used |
| 417 by gsutil for all commands targeting gs:// URLs. The JSON API is more |
| 418 bandwidth efficient than the older XML API when transferring metadata |
| 419 and does not require separate calls to preserve object ACLs when copying. |
| 420 The XML API will automatically be used when accessing s3:// URLs. |
| 421 - The Google Cloud Storage XML API can be used in lieu of the JSON API |
| 422 by setting 'prefer_api = xml' in the GSUtil section of your boto config file. |
| 423 - Added the rsync command that can synchronize cloud and local directories. |
| 424 - Added the signurl command that can generate Google Cloud Storage signed URLs. |
| 425 - The perfdiag command now supports a listing latency test. |
| 426 - The rb command now supports a -f flag allowing it to continue when errors |
| 427 are encountered. |
| 428 - The test command now supports a -s flag that runs tests against S3. |
| 429 |
| 430 Other Changes |
| 431 ------------- |
| 432 - All python files not under a third_party directory are now pylint-clean, |
| 433 with the exception of TODO-format and a handful of warnings in root-level |
| 434 files. As part of the de-linting process, many edge-case bugs were |
| 435 identified and fixed. |
| 436 - The ls command now operates depth-first (as in Unix ls) instead |
| 437 of breadth-first. |
| 438 - Daisy-chain copying does not currently support resumable uploads. |
| 439 - Several compatibility improvements for Windows and S3. |
| 440 |
| 441 |
| 442 Release 3.42 (release-date: 2014-01-15) |
| 443 ======================================= |
| 444 |
| 445 Other Changes |
| 446 ------------- |
| 447 |
| 448 - Fixed potential bug with update command on CentOS. |
| 449 |
| 450 |
| 451 Release 3.41 (release-date: 2014-01-14) |
| 452 ======================================= |
| 453 |
| 454 Other Changes |
| 455 ------------- |
| 456 |
| 457 - Changes to protect security of resumable upload IDs. |
| 458 |
| 459 |
| 460 Release 3.40: Skipped |
| 461 |
| 462 |
| 463 Release 3.39: Skipped |
| 464 |
| 465 |
| 466 Release 3.38 (release-date: 2013-11-25) |
| 467 |
| 468 Bug Fixes |
| 469 --------- |
| 470 |
| 471 - Fix to include version number in user-agent string. |
| 472 - Fix bug wherein -m flag or parallel uploads caused crash on systems without |
| 473 /dev/shm. |
| 474 - Fix SSL errors and invalid results with perfdiag -c and -k rthru test. |
| 475 - Fixed cases where parallel composite uploads could leave orphaned components. |
| 476 - Fix bug attempting to stat objects you don't have auth to read. |
| 477 - Fixed bug breaking defacl's -d option. |
| 478 |
| 479 |
| 480 Other Changes |
| 481 ------------- |
| 482 |
| 483 - Fixed gsutil config doc. |
| 484 - Fixed references to old command names; fix defacl ch example. |
| 485 - Improved error messages for deprecated command aliases. |
| 486 - Updated gsutil support info. |
| 487 |
| 488 |
| 489 New Features |
| 490 ------------ |
| 491 |
| 492 - Enabled -R flag for recursion with setmeta command. |
| 493 |
| 494 |
| 495 Release 3.37 (release-date: 2013-09-25) |
| 496 ======================================= |
| 497 |
| 498 Bug Fixes |
| 499 --------- |
| 500 |
| 501 - Fix parsing of -R for "acl ch" and chacl commands. |
| 502 - Fixed import statement of unittest2 which caused installations using Python |
| 503 2.6 without unittest2 installed to fail when starting up gsutil. |
| 504 |
| 505 |
| 506 Other Changes |
| 507 ------------- |
| 508 |
| 509 - Fixed tests so they pass on Windows and package installs. |
| 510 - Add a root logging handler manually instead of relying on basicConfig. |
| 511 - Fix apiclient import statement. |
| 512 - Exponential backoff for access token requests. |
| 513 - Fix flakiness in test TearDown to account for eventual consistency of object |
| 514 listings. |
| 515 |
| 516 |
| 517 Release 3.36 (release-date: 2013-09-18) |
| 518 ======================================= |
| 519 |
| 520 Bug Fixes |
| 521 --------- |
| 522 |
| 523 - Fix bug when a 400 or 403 exception has no detail. |
| 524 - Fix bugs with config -e and config -o. |
| 525 |
| 526 |
| 527 Other Changes |
| 528 ------------- |
| 529 |
| 530 - Clarify stat command documentation regarding trailing slashes. |
| 531 - Add Generation and Metageneration to gsutil stat output. |
| 532 |
| 533 |
| 534 New Features |
| 535 ------------ |
| 536 |
| 537 - Set config values from command line with -o. |
| 538 |
| 539 |
| 540 Release 3.35 (release-date: 2013-09-09) |
| 541 ======================================= |
| 542 |
| 543 Bug Fixes |
| 544 --------- |
| 545 |
| 546 - Fix streaming upload to S3 and provide more useful stack traces multi-threaded
failures. |
| 547 - Fix race condition in test_rm. |
| 548 - Fix retry decorator during test bucket cleanup. |
| 549 - Fixed cat bug that caused version to be ignored in URIs. |
| 550 - Don't decode -p or -h values other than x-goog-meta-. Fixes ability to use str
ing project names. |
| 551 - Update bucket_relocate.sh to work on GCE. |
| 552 - Fix recursive uploading from subdirectories with unexpanded wildcard as source
URI. |
| 553 - Make gsutil error text include <Message> content. |
| 554 - Change shebang line back to python because this doesn't work on some systems. |
| 555 - Fix hash_algs differences in perfdiag. |
| 556 - Update Python version check and shebang line. |
| 557 - Enforce project_id entry in config command; provide friendly error if missing
proj ID. |
| 558 - Use transcoding-invariant headers when available in gs.Key. |
| 559 - Make gsutil cp not fail if unable to check versioning config on dest bucket. |
| 560 - Make gsutil detect when config fails because of proxy and prompt for proxy con
fig. |
| 561 - Avoid checking metageneration attribute when long-listing S3 objects. |
| 562 - Exclude the no-op auth handler as indicating credentials are configured. |
| 563 |
| 564 |
| 565 New Features |
| 566 ------------ |
| 567 |
| 568 - Implemented gsutil stat command. |
| 569 |
| 570 |
| 571 Other Changes |
| 572 ------------- |
| 573 |
| 574 - Consolidate config-related commands. |
| 575 - Changed rm -r gs://bucket to delete bucket at end. |
| 576 - Various doc cleanup and improvement. |
| 577 - Warn user before updating to major new version. Also fixed minor version compa
rison bug, and added tests. |
| 578 - Change max component count to 1024. |
| 579 - Add retry-decorator as a submodule. |
| 580 - Explicitly state control chars to avoid in gsutil naming documentation. |
| 581 - Make config command recommend project strings. |
| 582 - Made long listing format a little better looking. |
| 583 - Allow --help flag for subcommands. |
| 584 - Implement help for subcommands and add OPTIONS sections for subcommands. |
| 585 - Add more detailed error message to notification watchbucket command. |
| 586 - Add notification URL configuration for notification tests. |
| 587 - Refactor to use upstream retry_decorator as external dependency. |
| 588 - Distribute cacerts file with gsutil. |
| 589 - Updated gsutil help to point to Google Cloud Console instead of older APIs con
sole. |
| 590 - Make gsutil pass bundled cacerts.txt to oauth2client; stop checking SHA1 of ce
rts, now that we no longer depend on boto distribution. |
| 591 - Move all TTY checks to a common util function and mock it for update tests. |
| 592 - Fix duplicate entry created in .gitmodules. |
| 593 - Fix unit test breakage because VERSION file is old. |
| 594 - Fix test using ? glob with ObjectToURI. |
| 595 - Fix update tests that fail for package installs. |
| 596 - Change bucket delete teardown to try more times. |
| 597 - Fix tests that perform operations on bucket listings. |
| 598 - Keep package install set to True unless VERSION file doesn't exist. |
| 599 - Fix handling of non-numeric version strings in update test. |
| 600 |
| 601 |
| 602 Release 3.34 (release-date: 2013-07-18) |
| 603 ======================================= |
| 604 |
| 605 Bug Fixes |
| 606 --------- |
| 607 |
| 608 - Fixed a bug where the no-op authentication handler was being loaded after |
| 609 other authentication plugins, causing the no-op handler to be chosen instead |
| 610 of other valid credentials. |
| 611 |
| 612 |
| 613 Release 3.33 (release-date: 2013-07-16) |
| 614 ======================================= |
| 615 |
| 616 Bug Fixes |
| 617 --------- |
| 618 |
| 619 - Added .git* to MANIFEST.in excludes and fixed cp doc typo. This was needed to |
| 620 overcome problem caused by accidental inclusion of .git* files in release, |
| 621 which caused the update command no longer to allow updates (since starting |
| 622 in 3.32 it checks whether the user has any extraneous files in the gsutil |
| 623 directory before updating) |
| 624 |
| 625 |
| 626 Release 3.32 (release-date: 2013-07-16) |
| 627 ======================================= |
| 628 |
| 629 New Features |
| 630 ------------ |
| 631 |
| 632 - Added support for getting and setting lifecycle configuration for buckets. |
| 633 - Implemented Parallel Composite Uploads. |
| 634 - Added a new du command that displays object size, similar to Linux du. |
| 635 |
| 636 |
| 637 Bug Fixes |
| 638 --------- |
| 639 |
| 640 - Fixed a bug when using ls -R on objects with trailing slashes. Closes #93. |
| 641 - Fixed so won't crash in perfdiag when nslookup is missing or gethostbyname |
| 642 fails. |
| 643 - Smartly compare version strings during autoupdate check. |
| 644 - Made header handling for upload case-insensitive. |
| 645 - Re-enabled software update check for users with no credentials configured. |
| 646 - Fixed incorrectly-generated password editing comment in service account |
| 647 config. Fixes #146. |
| 648 |
| 649 |
| 650 Other Changes |
| 651 ------------- |
| 652 |
| 653 - Improved flow when encounter auth failure for GCE service account with no |
| 654 configured storage scopes: |
| 655 1. Changed HasConfiguredCredentials() logic not to include |
| 656 has_auth_plugins as part of the evaluated expression, since that will |
| 657 always evaluate to true under GCE (since GCE configures its internal |
| 658 service account plugin under /etc/boto.cfg). |
| 659 2. Changed ConfigureNoOpAuthIfNeeded logic so we configure no-op auth |
| 660 plugin even if there is a config_file list, since GCE always configures |
| 661 /etc/boto.cfg, even if user has no storage scopes configured. |
| 662 3. Additional changes: |
| 663 a. Removed assertion of oauth access token cache check log from |
| 664 test_Doption.py, which may not be true sometimes (e.g., if user is |
| 665 using HMAC creds). |
| 666 b. Removed remnants of CONFIG_REQUIRED left over from earlier CL. |
| 667 c. Merged dupe _ConfigureNoOpAuthIfNeeded functions from two code files, |
| 668 moved to util.py. |
| 669 - Fixed confusing gsutil rm "Omitting" message. |
| 670 - Wrapped long gsutil update message. |
| 671 - Silenced additional possible perfdiag errors. |
| 672 - Improved perfdiag performance by only generating one chunk of random file. |
| 673 - Changed to swallow broken pipe errors when piping gsutil to other programs. |
| 674 - Made DotfulBucketNameNotUnderTld error message more user friendly. |
| 675 - Extracted function for building ACL error text from main try/except loop, |
| 676 for better readability. |
| 677 - Disallowed gsutil update when user data present in gsutil dir. |
| 678 - Plumbed accept-encoding into HEAD requests in ls -L command. |
| 679 - Updated README and moved ReleaseNotes.txt to CHANGES.md. |
| 680 - Updated crcmod docs with link to Windows installer. |
| 681 - Updated documentation regarding gzip content-encoding. |
| 682 - Removed StorageUri parse check for lone ':' (interferes with using filenames |
| 683 containing ':') |
| 684 - Added tests for gsutil update check and fixed bug for bad file contents. |
| 685 - Set accept-encoding and handle gzip on-the-fly encoding. |
| 686 |
| 687 |
| 688 Release 3.31 (release-date: 2013-06-10) |
| 689 ======================================= |
| 690 |
| 691 New Features |
| 692 ------------ |
| 693 |
| 694 - Implemented consumption of manifest files for cp. |
| 695 - Add ETag to ls -l and make ls -b variants more efficient. |
| 696 - Expand the manifest path to allow for tildes in paths. |
| 697 - Added bucket_relocate.sh script to gsutil. |
| 698 |
| 699 |
| 700 Bug Fixes |
| 701 --------- |
| 702 |
| 703 - Fix gsutil cp -R to copy all versioned objects. |
| 704 - Fixed bug where gsutil cp -D -n caused precondition failure. |
| 705 - Fixed gsutil daisy-chain copy to allow preserving ACLs when copying within |
| 706 same provider. |
| 707 - Fix identification of non-MD5 ETags. |
| 708 - Fixed bugs where gsutil -q cp and gsutil cp -q sometimes weren't quiet. |
| 709 - Fixed unicode error when constructing tracker filename from non-ASCII |
| 710 Unicode filename. |
| 711 - Fixed that noclobber would not resume partial resumable downloads. |
| 712 - Fixed bug when running gsutil cp -Dp by user other than object owner. |
| 713 - Properly encode metageneration and etag in ls output with -a and -e. |
| 714 - Update resumable threshold stated in gsutil help prod. |
| 715 |
| 716 |
| 717 Other Changes |
| 718 ------------- |
| 719 |
| 720 - ls -Lb no longer shows total # files/total size of bucket, so that ls -Lb |
| 721 instead provides an efficient way to view just the metadata for large buckets. |
| 722 - Catch and ignore EEXIST error when creating gsutil tracker dir. |
| 723 - Add note to gsutil update doc about auto-update checks being disabled with |
| 724 gsutil -q option. |
| 725 - Disable hashing and increase buffer size in perfdiag. |
| 726 - Added better error messages for service account auth. |
| 727 - Make perfdiag behave more like normal gsutil, with multi-threading option. |
| 728 - Changed so auto-update check/prompt aren't made if gsutil -q specified. |
| 729 - Changed gsutil mb command to clarify that EU means European Union. |
| 730 - Added doc warnings about losing version ordering if using gsutil -m cp |
| 731 between versioned buckets; removed trailing whitespace. |
| 732 - Added to gsutil cp -L doc to describe how to build a reliable script for |
| 733 copying many objects. |
| 734 |
| 735 |
| 736 Release 3.30 (release-date: 2013-06-10) |
| 737 ======================================= |
| 738 |
| 739 - Abandoned. |
| 740 |
| 741 |
| 742 Release 3.29 (release-date: 2013-05-13) |
| 743 ======================================= |
| 744 |
| 745 Bug Fixes |
| 746 --------- |
| 747 |
| 748 - Fixed incorrect package installation detection that resulted in not being |
| 749 able to run the update command while running gsutil from a symlink. |
| 750 |
| 751 Other Changes |
| 752 ------------- |
| 753 |
| 754 - Added a test for debug mode (gsutil -D) output. |
| 755 - List numbering and title case fixes in additional help pages. |
| 756 - Removed dateutil module dependency from cp command test. |
| 757 - Updated documentation to clarify that public-read objects are cached for 1 |
| 758 hour by default. |
| 759 - Added a filter to suppress "module was already imported" warnings that were |
| 760 sometimes printed while running gsutil on Google Compute Engine instances. |
| 761 |
| 762 |
| 763 Release 3.28 (release-date: 2013-05-07) |
| 764 ======================================= |
| 765 |
| 766 New Features |
| 767 ------------ |
| 768 |
| 769 - Added support for new Object Change Notifications feature. |
| 770 |
| 771 Bug Fixes |
| 772 --------- |
| 773 |
| 774 - Fixed problem where gsutil update command didn’t take default action. |
| 775 - Fixed a problem with the update command sometimes triggering an additional |
| 776 update command. |
| 777 |
| 778 Other Changes |
| 779 ------------- |
| 780 |
| 781 - Add packaging information to version output. |
| 782 - Removed fancy_urllib, since it is no longer used. |
| 783 - Changed num_retries default for resumable downloads to 6. |
| 784 - Don’t check for newer software version if gs_host is specified in boto |
| 785 config file. |
| 786 - Modified oauth2client logging behavior to be consistent with gsutil. |
| 787 - Added gs_port configuration option. |
| 788 - Skip update tests when SSL is disabled. |
| 789 |
| 790 |
| 791 Release 3.27 (release-date: 2013-04-25) |
| 792 ======================================= |
| 793 |
| 794 New Features |
| 795 ------------ |
| 796 |
| 797 - Added a human readable option (-h) to ls command. |
| 798 - Changed WildcardIterator not to materialize list of all matching files from |
| 799 directory listing (so works faster when walking over large directories) |
| 800 - Added -f option to setacl command to allow command to continue after errors |
| 801 encountered. |
| 802 - Add manifest log support for the cp command. |
| 803 - Added never option for check_hashes_config; fixed bug that assumes an ETag |
| 804 is always returned from server. |
| 805 - Made gsutil provide friendlier error message if attempting non-public data |
| 806 access with missing credentials. |
| 807 - Set 70 second default socket timeout for httplib. |
| 808 - Add ability to run a single test class or function with the test command. |
| 809 |
| 810 Bug Fixes |
| 811 --------- |
| 812 |
| 813 - Don't check for updates if the user has no credentials configured. This |
| 814 fixes a bug for users without credentials trying to use gsutil for first |
| 815 time. |
| 816 - Fixed case where chacl command incorrectly recognized an email address as a |
| 817 domain. |
| 818 - Fix setmeta command for S3 objects. |
| 819 - Fixed bug where wildcarded dest URI attempted string op on Key object. |
| 820 - Fixed case where gsutil -q outputted progress output when doing a streaming |
| 821 upload. |
| 822 - Error handling for out of space during downloads. |
| 823 - Include ISO 8601-required "Z" at end of timestamp string for gsutil ls -l, |
| 824 to be spec-compliant. |
| 825 - Removed deprecated setmeta syntax and fixed unicode issues. |
| 826 - Changed update command not to suggest running sudo if running under Cygwin. |
| 827 - Removed references to deprecated gs-discussion forum from gsutil built-in |
| 828 help. |
| 829 - Add literal quotes around CORS config example URL in gsutil setcors help to |
| 830 avoid having example URL turn into an HREF in auto-generated doc. |
| 831 |
| 832 Other Changes |
| 833 ------------- |
| 834 |
| 835 - Added proper setup.py to make gsutil installable via PyPi. |
| 836 - Added warning to gsutil built-in help that delete operations cannot be |
| 837 undone. |
| 838 - Replaced gsutil's OAuth2 client implementation with oauth2client. |
| 839 - Updates to perfdiag. |
| 840 - Updated config help about currently supported settings. |
| 841 - Fixes to setup.py and modified version command. |
| 842 - Move gslib/commands/cred_types.py to gslib, so only Command subclasses live |
| 843 in gslib/commands. |
| 844 - Updated gsutil setmeta help no longer to warn that setmeta with versioning |
| 845 enabled creates a new object. |
| 846 |
| 847 |
| 848 Release 3.26 (release-date: 2013-03-25) |
| 849 ======================================= |
| 850 |
| 851 New Features |
| 852 ------------ |
| 853 |
| 854 - Added support for object composition. |
| 855 - Added support for external service accounts. |
| 856 - Changed gsutil to check for available updates periodically (only while |
| 857 stdin, stderr, stdout are connected to a TTY, so as not to interfere with |
| 858 cron jobs). |
| 859 - Added chdefacl command. |
| 860 - Made gsutil built-in help available under |
| 861 https://developers.google.com/storage/docs/gsutil |
| 862 - Add a command suggestion when the command name is not found. |
| 863 - Added byte suffix parsing to the -s parameter of perfdiag. |
| 864 - Added --help support to subcommands. Fixes #96. |
| 865 - Updated perfdiag command to track availability and record TCP settings. |
| 866 - Added metadata parameter to perfdiag command. |
| 867 - Added support for specifying byte range to cat command. |
| 868 - Output more bucket metadata on ls -Lb. |
| 869 - Implemented gsutil -q (global quiet) option (fixes issue #130). Also changed |
| 870 gsutil to output all progress indicators using logging levels. Also changed |
| 871 help command not to output bold escape sequences and not use PAGER if stdout |
| 872 is not a tty, which also fixes bug that caused gsutil help test to fail. |
| 873 - Plumbed https_validate_certificates through to OAuth2 plugin handler, |
| 874 allowing control over cert validation for OAuth2 requests |
| 875 - Fixed ISO 639.1 ref in config command help text |
| 876 |
| 877 Bug Fixes |
| 878 --------- |
| 879 |
| 880 - Fixed bug where gsutil cp -D didn't preserve metadata |
| 881 - Fixed problem where gsutil -m is hard to interrupt (partial fix for issue |
| 882 #99 - only for Linux/MacOS; problem still exists for Windows). |
| 883 - Fixed broken reference to boto_lib_dir in update command. |
| 884 - Made changing ACL not retry on 400 error. |
| 885 - Fixed name expansion bug for case where uri_strs is itself an iterator |
| 886 (issue #131); implemented additional naming unit test for this case. |
| 887 - Fixed flaky gsutil rm test |
| 888 - Fixed a bug in the chacl command that made it so you couldn't delete the |
| 889 AllAuthenticatedUsers group from an ACL. |
| 890 |
| 891 Other Changes |
| 892 ------------- |
| 893 |
| 894 - Refactored gsutil main function into gslib, with gsutil being a thin |
| 895 wrapper. |
| 896 - Added a test for the update command. |
| 897 - Renamed gsutil meta_generation params to metageneration, for consistency |
| 898 with GCS docs. |
| 899 - Removed .pyc files from tarball/zipfile. |
| 900 - Added new root certs to cacerts.txt, to provide additional flexibility |
| 901 in the future. |
| 902 |
| 903 |
| 904 Release 3.25 (release-date: 2013-02-21) |
| 905 ======================================= |
| 906 |
| 907 Bug Fixes |
| 908 --------- |
| 909 |
| 910 - Fixed two version-specific URI bugs: |
| 911 1. gsutil cp -r gs://bucket1 gs://bucket2 would create objects in bucket2 |
| 912 with names corresponding to version-specific URIs in bucket1 (e.g., |
| 913 gs://bucket2/obj#1361417568482000, where the "#1361417568482000" part was |
| 914 part of the object name, not the object's generation). |
| 915 |
| 916 This problem similarly caused gsutil cp -r gs://bucket1 ./dir to create |
| 917 files names corresponding to version-specific URIs in bucket1. |
| 918 2. gsutil rm -a gs://bucket/obj would attempt to delete the same object |
| 919 twice, getting a NoSuchKey error on the second attempt. |
| 920 |
| 921 |
| 922 Release 3.24 (release-date: 2013-02-19) |
| 923 ======================================= |
| 924 |
| 925 Bug Fixes |
| 926 --------- |
| 927 |
| 928 - Fixed bug that caused attempt to dupe-encode a unicode filename. |
| 929 |
| 930 Other Changes |
| 931 ------------- |
| 932 |
| 933 - Refactored retry logic from setmeta and chacl to use @Retry decorator. |
| 934 - Moved @Retry decorator to third_party. |
| 935 - Fixed flaky tests. |
| 936 |
| 937 |
| 938 Release 3.23 (release-date: 2013-02-16) |
| 939 ======================================= |
| 940 |
| 941 Bug Fixes |
| 942 --------- |
| 943 |
| 944 - Make version-specific URI parsing more robust. This fixes a bug where |
| 945 listing buckets in certain cases would result in the error |
| 946 'BucketStorageUri' object has no attribute 'version_specific_uri' |
| 947 |
| 948 |
| 949 Release 3.22 (release-date: 2013-02-15) |
| 950 ======================================= |
| 951 |
| 952 New Features |
| 953 ------------ |
| 954 |
| 955 - Implemented new chacl command, which makes it easy to add and remove bucket |
| 956 and object ACL grants without having to edit XML (like the older setacl |
| 957 command). |
| 958 - Implemented new "daisy-chain" copying mode, which allows cross-provider |
| 959 copies to run without buffering to local disk, and to use resumable uploads. |
| 960 This copying mode also allows copying between locations and between storage |
| 961 classes, using the new gsutil cp -D option. (Daisy-chain copying is the |
| 962 default when copying between providers, but must be explicitly requested for |
| 963 the other cases to keep costs and performance expectations clear.) |
| 964 - Implemented new perfdiag command to run a diagnostic test against |
| 965 a bucket, collect system information, and report results. Useful |
| 966 when working with Google Cloud Storage team to resolve questions |
| 967 about performance. |
| 968 - Added SIGQUIT (^\) handler, to allow breakpointing a running gsutil. |
| 969 |
| 970 Bug Fixes |
| 971 --------- |
| 972 |
| 973 - Fixed bug where gsutil setwebcfg signature didn't match with |
| 974 HMAC authentication. |
| 975 - Fixed ASCII codec decode error when constructing tracker filename |
| 976 from non-7bit ASCII input filename. |
| 977 - Changed boto auth plugin framework to allow multiple plugins |
| 978 supporting requested capability, which fixes gsutil exception |
| 979 that used to happen where a GCE user had a service account |
| 980 configured and then ran gsutil config. |
| 981 - Changed Command.Apply method to be resilient to name expansion |
| 982 exceptions. Before this change, if an exception was raised |
| 983 during iteration of NameExpansionResult, the parent process |
| 984 would immediately stop execution, causing the |
| 985 _EOF_NAME_EXPANSION_RESULT to never be sent to child processes. |
| 986 This resulted in the process hanging forever. |
| 987 - Fixed various bugs for gsutil running on Windows: |
| 988 - Fixed various places from a hard-coded '/' to os.sep. |
| 989 - Fixed a bug in the cp command where it was using the destination |
| 990 URI's .delim property instead of the source URI. |
| 991 - Fixed a bug in the cp command's _SrcDstSame function by |
| 992 simplifying it to use os.path.normpath. |
| 993 - Fixed windows bug in tests/util.py _NormalizeURI function. |
| 994 - Fixed ZeroDivisionError sometimes happening during unit tests |
| 995 on Windows. |
| 996 |
| 997 - Fixed gsutil rm bug that caused exit status 1 when encountered |
| 998 non-existent URI. |
| 999 - Fixed support for gsutil cp file -. |
| 1000 - Added preconditions and retry logic to setmeta command, to |
| 1001 enforce concurrency control. |
| 1002 - Fixed bug in copying subdirs to subdirs. |
| 1003 - Fixed cases where boto debug_level caused too much or too little |
| 1004 logging: |
| 1005 - resumable and one-shot uploads weren't showing response headers |
| 1006 when connection.debug > 0. |
| 1007 - payload was showing up in debug output when connection.debug |
| 1008 < 4 for streaming uploads. |
| 1009 |
| 1010 - Removed XML parsing from setacl. The previous implementation |
| 1011 relied on loose XML handling, which could truncate what it sends |
| 1012 to the service, allowing invalid XML to be specified by the |
| 1013 user. Instead now the ACL XML is passed verbatim and we rely |
| 1014 on server-side schema enforcement. |
| 1015 - Added user-agent header to resumable uploads. |
| 1016 - Fixed reporting bits/s when it was really bytes/s. |
| 1017 - Changed so we now pass headers with API version & project ID |
| 1018 to create_bucket(). |
| 1019 - Made "gsutil rm -r gs://bucket/folder" remove xyz_$folder$ object |
| 1020 (which is created by various GUI tools). |
| 1021 - Fixed bug where gsutil binary was shipped with protection 750 |
| 1022 instead of 755. |
| 1023 |
| 1024 Other Changes |
| 1025 ------------- |
| 1026 |
| 1027 - Reworked versioned object handling: |
| 1028 - Removed need for commands to specify -v option to parse |
| 1029 versions. Versioned URIs are now uniformly handled by all |
| 1030 commands. |
| 1031 - Refactored StorageUri parsing that had been split across |
| 1032 storage_uri and convenience; made versioned URIs render with |
| 1033 version string so StorageUri is round-trippable (boto change). |
| 1034 - Implemented gsutil cp -v option for printing the version-specific |
| 1035 URI that was just created. |
| 1036 - Added error detail for attempt to delete non-empty versioned |
| 1037 bucket. Also added versioning state to ls -L -b gs://bucket |
| 1038 output. |
| 1039 - Changed URI parsing to use pre-compiled regex's. |
| 1040 - Other bug fixes. |
| 1041 |
| 1042 - Rewrote/deepened/improved various parts of built-in help: |
| 1043 - Updated 'gsutil help dev'. |
| 1044 - Fixed help command handling when terminal does not have the |
| 1045 number of rows set. |
| 1046 - Rewrote versioning help. |
| 1047 - Added gsutil help text for common 403 AccountProblem error. |
| 1048 - Added text to 'gsutil help dev' about legal agreement needed |
| 1049 with code submissions. |
| 1050 - Fixed various other typos. |
| 1051 - Updated doc for cp command regarding metadata not being |
| 1052 preserved when copying between providers. |
| 1053 - Fixed gsutil ls command documentation typo for the -L option. |
| 1054 - Added HTTP scheme to doc/examples for gsutil setcors command. |
| 1055 - Changed minimum version in documentation from 2.5 to 2.6 since |
| 1056 gsutil no longer works in Python 2.5. |
| 1057 - Cleaned up/clarify/deepen various other parts of gsutil |
| 1058 built-in documentation. |
| 1059 |
| 1060 - Numerous improvements to testing infrastructure: |
| 1061 - Completely refactored infrastructure, allowing deeper testing |
| 1062 and more readable test code, and enabling better debugging |
| 1063 output when tests fail. |
| 1064 - Moved gslib/test_*.py unit tests to gslib/tests module. |
| 1065 - Made all tests (unit and integration, per-command and modules |
| 1066 (like naming) run from single gsutil test command. |
| 1067 - Moved TempDir functions from GsUtilIntegrationTestCase to |
| 1068 GsUtilTestCase. |
| 1069 - Made test runner message show the test function being run. |
| 1070 - Added file path support to ObjectToURI function. |
| 1071 - Disabled the test command if running on Python 2.6 and unittest2 |
| 1072 is not available instead of breaking all of gsutil. |
| 1073 - Changed to pass GCS V2 API and project_id from boto config |
| 1074 if necessary in integration_testcase#CreateBucket(). |
| 1075 - Fixed unit tests by using a GS-specific mocking class to |
| 1076 override the S3 provider. |
| 1077 - Added friendlier error message if test path munging fails. |
| 1078 - Fixed bug where gsutil test only cleaned up first few test files. |
| 1079 - Implemented setacl integration tests. |
| 1080 - Implemented StorageUri parsing unit tests. |
| 1081 - Implemented test for gsutil cp -D. |
| 1082 - Implemented setacl integration tests. |
| 1083 - Implemented tests for reading and seeking past end of file. |
| 1084 - Implemented and tests for it in new tests module. |
| 1085 - Changed cp tests that don't specify a Content-Type to check |
| 1086 for new binary/octet-stream default instead of server-detected |
| 1087 mime type. |
| 1088 |
| 1089 - Changed gsutil mv to allow moving local files/dirs to the cloud. |
| 1090 Previously this was disallowed in the belief we should be |
| 1091 conservative about deleting data from local disk but there are |
| 1092 legitimate use cases for moving data from a local dir to the |
| 1093 cloud, it's clear to the user this would remove data from the |
| 1094 local disk, and allowing it makes the tool behavior more |
| 1095 consistent with what users would expect. |
| 1096 - Changed gsutil update command to insist on is_secure and |
| 1097 https_validate_certificates. |
| 1098 - Fixed release no longer to include extraneous boto dirs in |
| 1099 top-level of gsutil distribution (like bin/ and docs/). |
| 1100 - Changed resumable upload threshold from 1 MB to 2 MB. |
| 1101 - Removed leftover cloudauth and cloudreader dirs. Sample code |
| 1102 now lives at https://github.com/GoogleCloudPlatform. |
| 1103 - Updated copyright notice on code files. |
| 1104 |
| 1105 |
| 1106 Release 3.21 (release-date: 2012-12-10) |
| 1107 ======================================= |
| 1108 |
| 1109 New Features |
| 1110 ------------ |
| 1111 |
| 1112 - Added the ability for the cp command to continue even if there is an |
| 1113 error. This can be activated with the -c flag. |
| 1114 - Added support for specifying src args for gsutil cp on stdin (-I option) |
| 1115 |
| 1116 Bug Fixes |
| 1117 --------- |
| 1118 |
| 1119 - Fixed gsutil test cp, which assumed it was run from gsutil install dir. |
| 1120 - Mods so we send generation subresource only when user requested |
| 1121 version parsing (-v option for cp and cat commands). |
| 1122 |
| 1123 Other Changes |
| 1124 ------------- |
| 1125 |
| 1126 - Updated docs about using setmeta with versioning enabled. |
| 1127 - Changed GCS endpoint in boto to storage.googleapis.com. |
| 1128 |
| 1129 |
| 1130 Release 3.20 (release-date: 2012-11-30) |
| 1131 ======================================= |
| 1132 |
| 1133 New Features |
| 1134 ------------ |
| 1135 |
| 1136 - Added a noclobber (-n) setting for the cp command. Existing objects/files |
| 1137 will not be overwritten when using this setting. |
| 1138 |
| 1139 Bug Fixes |
| 1140 --------- |
| 1141 |
| 1142 - Fixed off-by-one error when reporting bytes transferred. |
| 1143 |
| 1144 Other Changes |
| 1145 ------------- |
| 1146 |
| 1147 - Improved versioning support for the remove command. |
| 1148 - Improved test runner support. |
| 1149 |
| 1150 |
| 1151 Release 3.19 (release-date: 2012-11-26) |
| 1152 ======================================= |
| 1153 |
| 1154 New Features |
| 1155 ------------ |
| 1156 |
| 1157 - Added support for object versions. |
| 1158 - Added support for storage classes (including Durable Reduced Availability). |
| 1159 |
| 1160 Bug Fixes |
| 1161 --------- |
| 1162 - Fixed problem where cp -q prevented resumable uploads from being performed. |
| 1163 - Made setwebcfg and setcors tests robust wrt XML formatting variation. |
| 1164 |
| 1165 Other Changes |
| 1166 ------------- |
| 1167 |
| 1168 - Incorporated vapier@ mods to make version command not fail if CHECKSUM file |
| 1169 missing. |
| 1170 - Refactored gsutil such that most functionality exists in boto. |
| 1171 - Updated gsutil help dev instructions for how to check out source. |
| 1172 |
| 1173 |
| 1174 Release 3.18 (release-date: 2012-09-19) |
| 1175 ======================================= |
| 1176 |
| 1177 Bug Fixes |
| 1178 --------- |
| 1179 |
| 1180 - Fixed resumable upload boundary condition when handling POST request |
| 1181 when server already has complete file, which resulted in an infinite |
| 1182 loop that consumed 100% of the CPU. |
| 1183 - Fixed one more place that outputted progress info when gsutil cp -q |
| 1184 specified (during streaming uploads). |
| 1185 |
| 1186 Other Changes |
| 1187 ------------- |
| 1188 |
| 1189 - Updated help text for "gsutil help setmeta" and "gsutil help metadata", to |
| 1190 clarify and deepen parts of the documentation. |
| 1191 |
| 1192 |
| 1193 Release 3.17 (release-date: 2012-08-17) |
| 1194 ======================================= |
| 1195 |
| 1196 Bug Fixes |
| 1197 --------- |
| 1198 |
| 1199 - Fixed race condition when multiple threads attempt to get an OAuth2 refresh |
| 1200 token concurrently. |
| 1201 |
| 1202 Other Changes |
| 1203 ------------- |
| 1204 |
| 1205 - Implemented simplified syntax for setmeta command. The old syntax still |
| 1206 works but is now deprecated. |
| 1207 - Added help to gsutil cp -z option, to describe how to change where temp |
| 1208 files are written. |
| 1209 |
| 1210 |
| 1211 Release 3.16 (release-date: 2012-08-13) |
| 1212 ======================================= |
| 1213 |
| 1214 Bug Fixes |
| 1215 --------- |
| 1216 |
| 1217 - Added info to built-in help for setmeta command, to explain the syntax |
| 1218 needed when running from Windows. |
| 1219 |
| 1220 |
| 1221 Release 3.15 (release-date: 2012-08-12) |
| 1222 ======================================= |
| 1223 |
| 1224 New Features |
| 1225 ------------ |
| 1226 |
| 1227 - Implemented gsutil setmeta command. |
| 1228 - Made gsutil understand bucket subdir conventions used by various tools |
| 1229 (like GCS Manager and CloudBerry) so if you cp or mv to a subdir you |
| 1230 created with one of those tools it will work as expected. |
| 1231 - Added support for Windows drive letter-prefaced paths when using Storage |
| 1232 URIs. |
| 1233 |
| 1234 Bug Fixes |
| 1235 --------- |
| 1236 |
| 1237 - Fixed performance bug when downloading a large object with Content- |
| 1238 Encoding:gzip, where decompression attempted to load the entire object |
| 1239 in memory. Also added "Uncompressing" log output if file is larger than |
| 1240 50M, to make it clear the download hasn't stalled. |
| 1241 - Fixed naming bug when performing gsutil mv from a bucket subdir to |
| 1242 and existing bucket subdir. |
| 1243 - Fixed bug that caused cross-provider copies into Google Cloud Storage to |
| 1244 fail. |
| 1245 - Made change needed to make resumable transfer progress messages not print |
| 1246 when running gsutil cp -q. |
| 1247 - Fixed copy/paste error in config file documentation for |
| 1248 https_validate_certificates option. |
| 1249 - Various typo fixes. |
| 1250 |
| 1251 Other Changes |
| 1252 ------------- |
| 1253 |
| 1254 - Changed gsutil to unset http_proxy environment variable if it's set, |
| 1255 because it confuses boto. (Proxies should instead be configured via the |
| 1256 boto config file.) |
| 1257 |
| 1258 |
| 1259 Release 3.14 (release-date: 2012-07-28) |
| 1260 ======================================= |
| 1261 |
| 1262 New Features |
| 1263 ------------ |
| 1264 |
| 1265 - Added cp -q option, to support quiet operation from cron jobs. |
| 1266 - Made config command restore backed up file if there was a failure or user |
| 1267 hits ^C. |
| 1268 |
| 1269 Bug Fixes |
| 1270 --------- |
| 1271 |
| 1272 - Fixed bug where gsutil cp -R from a source directory didn't generate |
| 1273 correct destination path. |
| 1274 - Fixed file handle leak in gsutil cp -z |
| 1275 - Fixed bug that caused cp -a option not to work when copying in the cloud. |
| 1276 - Fixed bug that caused '/-' to be appended to object name for streaming |
| 1277 uploads. |
| 1278 - Revert incorrect line I changed in previous CL, that attempted to |
| 1279 get fp from src_key object. The real fix that's needed is described in |
| 1280 https://github.com/GoogleCloudPlatform/gsutil/issues/72. |
| 1281 |
| 1282 Other Changes |
| 1283 ------------- |
| 1284 |
| 1285 - Changed logging to print "Copying..." and Content-Type on same line; |
| 1286 refactored content type and log handling. |
| 1287 |
| 1288 |
| 1289 Release 3.13 (release-date: 2012-07-19) |
| 1290 ======================================= |
| 1291 |
| 1292 Bug Fixes |
| 1293 --------- |
| 1294 |
| 1295 - Included the fix to make 'gsutil config' honor BOTO_CONFIG environment |
| 1296 variable (which was intended to be included in Release 3.12) |
| 1297 |
| 1298 |
| 1299 Release 3.11 (release-date: 2012-06-28) |
| 1300 ======================================= |
| 1301 |
| 1302 New Features |
| 1303 ------------ |
| 1304 |
| 1305 - Added support for configuring website buckets. |
| 1306 |
| 1307 Bug Fixes |
| 1308 --------- |
| 1309 |
| 1310 - Fixed bug that caused simultaneous resumable downloads of the same source |
| 1311 object to use the same tracker file. |
| 1312 - Changed language code spec pointer from Wikipedia to loc.gov (for |
| 1313 Content-Language header). |
| 1314 |
| 1315 |
| 1316 Release 3.10 (release-date: 2012-06-19) |
| 1317 ======================================= |
| 1318 |
| 1319 New Features |
| 1320 ------------ |
| 1321 |
| 1322 - Added support for setting and listing Content-Language header. |
| 1323 |
| 1324 Bug Fixes |
| 1325 --------- |
| 1326 |
| 1327 - Fixed bug that caused getacl/setacl commands to get a character encoding |
| 1328 exception when ACL content contained content not representable in ISO-8859-1 |
| 1329 character set. |
| 1330 - Fixed gsutil update not to fail under Windows exclusive file locking. |
| 1331 - Fixed gsutil ls -L to continue past 403 errors. |
| 1332 - Updated gsutil tests and also help dev with instructions on how to run |
| 1333 boto tests, based on recent test refactoring done to in boto library. |
| 1334 - Cleaned up parts of cp help text. |
| 1335 |
| 1336 |
| 1337 Release 3.9 (release-date: 2012-05-24) |
| 1338 ====================================== |
| 1339 |
| 1340 Bug Fixes |
| 1341 --------- |
| 1342 |
| 1343 - Fixed bug that caused extra "file:/" to be included in pathnames with |
| 1344 doing gsutil cp -R on Windows. |
| 1345 |
| 1346 |
| 1347 Release 3.8 (release-date: 2012-05-20) |
| 1348 ====================================== |
| 1349 |
| 1350 Bug Fixes |
| 1351 --------- |
| 1352 |
| 1353 - Fixed problem with non-ASCII filename characters not setting encoding before |
| 1354 attempting to hash for generating resumable transfer filename. |
| 1355 |
| 1356 |
| 1357 Release 3.7 (release-date: 2012-05-11) |
| 1358 ====================================== |
| 1359 |
| 1360 Bug Fixes |
| 1361 --------- |
| 1362 |
| 1363 - Fixed handling of HTTPS tunneling through a proxy. |
| 1364 |
| 1365 |
| 1366 Release 3.6 (release-date: 2012-05-09) |
| 1367 ====================================== |
| 1368 |
| 1369 Bug Fixes |
| 1370 --------- |
| 1371 |
| 1372 - Fixed bug that caused wildcards spanning directories not to work. |
| 1373 - Fixed bug that gsutil cp -z not to find available tmp space correctly |
| 1374 under Windows. |
| 1375 |
| 1376 |
| 1377 Release 3.5 (release-date: 2012-04-30) |
| 1378 ====================================== |
| 1379 |
| 1380 Performance Improvement |
| 1381 ----------------------- |
| 1382 |
| 1383 - Change by Evan Worley to calculate MD5s incrementally during uploads and |
| 1384 downloads. This reduces overall transfer time substantially for large |
| 1385 objects. |
| 1386 |
| 1387 Bug Fixes |
| 1388 --------- |
| 1389 |
| 1390 - Fixed bug where uploading and moving multiple files to a bucket subdirectory |
| 1391 didn't work as intended. |
| 1392 (https://github.com/GoogleCloudPlatform/gsutil/issues/92). |
| 1393 - Fixed bug where gsutil cp -r sourcedir didn't copy to specified subdir |
| 1394 if there is only one file in sourcedir. |
| 1395 - Fixed bug where tracker file included a timestamp that caused it not to |
| 1396 be recognized across sessions. |
| 1397 - Fixed bug where gs://bucket/*/dir wildcard matches too many objects. |
| 1398 - Fixed documentation errors in help associated with ACLs and projects. |
| 1399 - Changed GCS ACL parsing to be case-insensitive. |
| 1400 - Changed ls to print error and exit with non-0 status when wildcard matches |
| 1401 nothing, to be more consistent with UNIX shell behavior. |
| 1402 |
| 1403 |
| 1404 Release 3.4 (release-date: 2012-04-06) |
| 1405 ====================================== |
| 1406 |
| 1407 Bug Fixes |
| 1408 --------- |
| 1409 |
| 1410 - Fixed problem where resumable uploads/downloads of objects with very long |
| 1411 names would generate tracking files with names that exceeded local file |
| 1412 system limits, making it impossible to complete resumable transfers for |
| 1413 those objects. Solution was to build the tracking file name from a fixed |
| 1414 prefix, SHA1 hash of the long filename, epoch timestamp and last 16 |
| 1415 chars of the long filename, which is guarantee to be a predictable and |
| 1416 reasonable length. |
| 1417 - Fixed minor bug in output from 'gsutil help dev' which advised executing |
| 1418 an inconsequential test script (test_util.py). |
| 1419 |
| 1420 |
| 1421 Release 3.3 (release-date: 2012-04-03) |
| 1422 ====================================== |
| 1423 |
| 1424 Bug Fixes |
| 1425 --------- |
| 1426 |
| 1427 - Fixed problem where gsutil ver and debug flags crashed when used |
| 1428 with newly generated boto config files. |
| 1429 - Fixed gsutil bug in windows path handling, and make checksumming work |
| 1430 across platforms. |
| 1431 - Fixed enablelogging to translate -b URI param to plain bucket name in REST |
| 1432 API request. |
| 1433 |
| 1434 |
| 1435 Release 3.2 (release-date: 2012-03-30) |
| 1436 ====================================== |
| 1437 |
| 1438 Bug Fixes |
| 1439 --------- |
| 1440 |
| 1441 - Fixed problem where gsutil didn't convert between OS-specific directory |
| 1442 separators when copying individually-named files (issue 87). |
| 1443 - Fixed problem where gsutil ls -R didn't work right if there was a key |
| 1444 with a leading path (like /foo/bar/baz) |
| 1445 |
| 1446 |
| 1447 Release 3.1 (release-date: 2012-03-20) |
| 1448 ====================================== |
| 1449 |
| 1450 Bug Fixes |
| 1451 --------- |
| 1452 |
| 1453 - Removed erroneous setting of Content-Encoding when a gzip file is uploaded |
| 1454 (vs running gsutil cp -z, when Content-Encoding should be set). This |
| 1455 error caused users to get gsutil.tar.gz file uncompressed by the user |
| 1456 agent (like wget) while downloading, making the file appear to be of the |
| 1457 wrong size/content. |
| 1458 - Fixed handling of gsutil help for Windows (previous code depended on |
| 1459 termios and fcntl libs, which are Linux/MacOS-specific). |
| 1460 |
| 1461 |
| 1462 Release 3.0 (release-date: 2012-03-20) |
| 1463 ====================================== |
| 1464 |
| 1465 Important Notes |
| 1466 --------------- |
| 1467 |
| 1468 - Backwards-incompatible wildcard change: |
| 1469 The '*' wildcard now only matches objects within a bucket directory. If |
| 1470 you have scripts that depend on being able to match spanning multiple |
| 1471 directories you need to use '**' instead. For example, the command: |
| 1472 |
| 1473 gsutil cp gs://bucket/*.txt |
| 1474 |
| 1475 will now only match .txt files in the top-level directory. |
| 1476 |
| 1477 gsutil cp gs://bucket/**.txt |
| 1478 |
| 1479 will match across all directories. |
| 1480 - gsutil ls now lists one directory at a time. If you want to list all objects |
| 1481 in a bucket, you can use: |
| 1482 |
| 1483 gsutil ls gs://bucket/** |
| 1484 |
| 1485 or: |
| 1486 |
| 1487 gsutil ls -R gs://bucket |
| 1488 |
| 1489 New Features |
| 1490 ------------ |
| 1491 |
| 1492 - Built-in help for all commands and many additional topics. Try |
| 1493 "gsutil help" for a list of available commands and topics. |
| 1494 - A new hierarchical file tree abstraction layer, which makes the flat bucket |
| 1495 name space look like a hierarchical file tree. This makes several things |
| 1496 possible: |
| 1497 - copying data to/from bucket sub-directories (see “gsutil help cp”). |
| 1498 - distributing large uploads/downloads across many machines |
| 1499 (see “gsutil help cp”) |
| 1500 - renaming bucket sub-directories (see “gsutil help mv”). |
| 1501 - listing individual bucket sub-directories and for listing directories |
| 1502 recursively (see “gsutil help ls”). |
| 1503 - setting ACLs for objects in a sub-directory (see “gsutil help setacl”). |
| 1504 |
| 1505 - Support for per-directory (*) and recursive (**) wildcards. Essentially, |
| 1506 ** works the way * did in previous gsutil releases, and * now behaves |
| 1507 consistently with how it works in command interpreters (like bash). The |
| 1508 ability to specify directory-only wildcards also enables a number of use |
| 1509 cases, such as distributing large uploads/downloads by wildcarded name. See |
| 1510 "gsutil help wildcards" for details. |
| 1511 - Support for Cross-Origin Resource Sharing (CORS) configuration. See "gsutil |
| 1512 help cors" for details. |
| 1513 - Support for multi-threading and recursive operation for setacl command |
| 1514 (see “gsutil help setacl”). |
| 1515 - Ability to use the UNIX 'file' command to do content type recognition as |
| 1516 an alternative to filename extensions. |
| 1517 - Introduction of new end-to-end test suite. |
| 1518 - The gsutil version command now computes a checksum of the code, to detect |
| 1519 corruption and local modification when assisting with technical support. |
| 1520 - The gsutil update command is no longer beta/experimental, and now also |
| 1521 supports updating from named URIs (for early/test releases). |
| 1522 - Changed gsutil ls -L to also print Content-Disposition header. |
| 1523 |
| 1524 Bug Fixes |
| 1525 --------- |
| 1526 |
| 1527 - The gsutil cp -t option previously didn't work as documented, and instead |
| 1528 Content-Type was always detected based on filename extension. Content-Type |
| 1529 detection is now the default, the -t option is deprecated (to be removed in |
| 1530 the future), and specifying a -h Content-Type header now correctly overrides |
| 1531 the filename extension based handling. For details see "gsutil help |
| 1532 metadata". |
| 1533 - Fixed bug that caused multi-threaded mv command not to percolate failures |
| 1534 during the cp phase to the rm phase, which could under some circumstances |
| 1535 cause data that was not copied to be deleted. |
| 1536 - Fixed bug that caused gsutil to use GET for ls -L requests. It now uses HEAD |
| 1537 for ls -L requests, which is more efficient and faster. |
| 1538 - Fixed bug that caused gsutil not to preserve metadata during |
| 1539 copy-in-the-cloud. |
| 1540 - Fixed bug that prevented setacl command from allowing DisplayName's in ACLs. |
| 1541 - Fixed bug that caused gsutil/boto to suppress consecutive slashes in path |
| 1542 names. |
| 1543 - Fixed spec-non-compliant URI construction for resumable uploads. |
| 1544 - Fixed bug that caused rm -f not to work. |
| 1545 - Fixed UnicodeEncodeError that happened when redirecting gsutil ls output |
| 1546 to a file with non-ASCII object names. |
| 1547 |
| 1548 Other Changes |
| 1549 ------------- |
| 1550 |
| 1551 - UserAgent sent in HTTP requests now includes gsutil version number and OS |
| 1552 name. |
| 1553 - Starting with this release users are able to get individual named releases |
| 1554 from version-named objects: gs://pub/gsutil_<version>.tar.gz |
| 1555 and gs://pub/gsutil_<version>.zip. The version-less counterparts |
| 1556 (gs://pub/gsutil.tar.gz and gs://pub/gsutil.zip) will contain the latest |
| 1557 release. Also, the gs://pub bucket is now publicly readable (so, anyone |
| 1558 can list its contents). |
| 1559 |
| 1560 |
| 1561 Release 2.0 (release-date: 2012-01-13) |
| 1562 ====================================== |
| 1563 |
| 1564 New Features |
| 1565 ------------ |
| 1566 |
| 1567 - Support for for two new installation modes: enterprise and RPM. |
| 1568 Customers can now install gsutil one of three ways: |
| 1569 - Individual user mode (previously the only available mode): unpacking from |
| 1570 a gzipped tarball (gs://pub/gsutil.tar.gz) or zip file |
| 1571 (gs://pub/gsutil.zip) and running the gsutil command in place in the |
| 1572 unpacked gsutil directory. |
| 1573 - Enterprise mode (new): unpacking as above, and then running the setup.py |
| 1574 script in the unpacked gsutil directory. This allows a systems |
| 1575 administrator to install gsutil in a central location, using the Python |
| 1576 distutils facility. This mode is supported only on Linux and MacOS. |
| 1577 - RPM mode (new). A RedHat RPM can be built from the gsutil.spec.in file |
| 1578 in the unpacked gsutil directory, allowing it to be installed as part of |
| 1579 a RedHat build. |
| 1580 |
| 1581 - Note: v2.0 is the first numbered gsutil release. Previous releases |
| 1582 were given timestamps for versions. Numbered releases enable downstream |
| 1583 package builds (like RPMs) to define dependencies more easily. |
| 1584 This is also the first version where we began including release notes. |
OLD | NEW |