| OLD | NEW |
| 1 # -*- coding: utf-8 -*- | 1 # -*- coding: utf-8 -*- |
| 2 # Copyright 2012 Google Inc. All Rights Reserved. | 2 # Copyright 2012 Google Inc. All Rights Reserved. |
| 3 # | 3 # |
| 4 # Licensed under the Apache License, Version 2.0 (the "License"); | 4 # Licensed under the Apache License, Version 2.0 (the "License"); |
| 5 # you may not use this file except in compliance with the License. | 5 # you may not use this file except in compliance with the License. |
| 6 # You may obtain a copy of the License at | 6 # You may obtain a copy of the License at |
| 7 # | 7 # |
| 8 # http://www.apache.org/licenses/LICENSE-2.0 | 8 # http://www.apache.org/licenses/LICENSE-2.0 |
| 9 # | 9 # |
| 10 # Unless required by applicable law or agreed to in writing, software | 10 # Unless required by applicable law or agreed to in writing, software |
| (...skipping 15 matching lines...) Expand all Loading... |
| 26 additional documentation to help users understand how gsutil handles | 26 additional documentation to help users understand how gsutil handles |
| 27 subdirectories differently than most GUI / web-based tools (e.g., why | 27 subdirectories differently than most GUI / web-based tools (e.g., why |
| 28 those other tools create "dir_$folder$" objects), and also to explain cost and | 28 those other tools create "dir_$folder$" objects), and also to explain cost and |
| 29 performance implications of the gsutil approach, for those interested in such | 29 performance implications of the gsutil approach, for those interested in such |
| 30 details. | 30 details. |
| 31 | 31 |
| 32 gsutil provides the illusion of a hierarchical file tree atop the "flat" | 32 gsutil provides the illusion of a hierarchical file tree atop the "flat" |
| 33 name space supported by the Google Cloud Storage service. To the service, | 33 name space supported by the Google Cloud Storage service. To the service, |
| 34 the object gs://your-bucket/abc/def/ghi.txt is just an object that happens to | 34 the object gs://your-bucket/abc/def/ghi.txt is just an object that happens to |
| 35 have "/" characters in its name. There are no "abc" or "abc/def" directories; | 35 have "/" characters in its name. There are no "abc" or "abc/def" directories; |
| 36 just a single object with the given name. This | 36 just a single object with the given name. This diagram: |
| 37 `diagram <https://cloud.google.com/storage/images/gsutil-subdirectories-thumb.
png>` | 37 https://cloud.google.com/storage/images/gsutil-subdirectories-thumb.png |
| 38 illustrates how gsutil provides a hierarchical view of objects in a bucket. | 38 illustrates how gsutil provides a hierarchical view of objects in a bucket. |
| 39 | 39 |
| 40 gsutil achieves the hierarchical file tree illusion by applying a variety of | 40 gsutil achieves the hierarchical file tree illusion by applying a variety of |
| 41 rules, to try to make naming work the way users would expect. For example, in | 41 rules, to try to make naming work the way users would expect. For example, in |
| 42 order to determine whether to treat a destination URI as an object name or the | 42 order to determine whether to treat a destination URL as an object name or the |
| 43 root of a directory under which objects should be copied gsutil uses these | 43 root of a directory under which objects should be copied gsutil uses these |
| 44 rules: | 44 rules: |
| 45 | 45 |
| 46 1. If the destination object ends with a "/" gsutil treats it as a directory. | 46 1. If the destination object ends with a "/" gsutil treats it as a directory. |
| 47 For example, if you run the command: | 47 For example, if you run the command: |
| 48 | 48 |
| 49 gsutil cp your-file gs://your-bucket/abc/ | 49 gsutil cp your-file gs://your-bucket/abc/ |
| 50 | 50 |
| 51 gsutil will create the object gs://your-bucket/abc/file. | 51 gsutil will create the object gs://your-bucket/abc/your-file. |
| 52 | 52 |
| 53 2. If the destination object is XYZ and an object exists called XYZ_$folder$ | 53 2. If the destination object is XYZ and an object exists called XYZ_$folder$ |
| 54 gsutil treats XYZ as a directory. For example, if you run the command: | 54 gsutil treats XYZ as a directory. For example, if you run the command: |
| 55 | 55 |
| 56 gsutil cp your-file gs://your-bucket/abc | 56 gsutil cp your-file gs://your-bucket/abc |
| 57 | 57 |
| 58 and there exists an object called abc_$folder$, gsutil will create the | 58 and there exists an object called abc_$folder$, gsutil will create the |
| 59 object gs://your-bucket/abc/file. | 59 object gs://your-bucket/abc/your-file. |
| 60 | 60 |
| 61 3. If you attempt to copy multiple source files to a destination URI, gsutil | 61 3. If you attempt to copy multiple source files to a destination URL, gsutil |
| 62 treats the destination URI as a directory. For example, if you run | 62 treats the destination URL as a directory. For example, if you run |
| 63 the command: | 63 the command: |
| 64 | 64 |
| 65 gsutil cp -r your-dir gs://your-bucket/abc | 65 gsutil cp -r your-dir gs://your-bucket/abc |
| 66 | 66 |
| 67 gsutil will create objects like gs://your-bucket/abc/your-dir/file1, etc. | 67 gsutil will create objects like gs://your-bucket/abc/your-dir/file1, etc. |
| 68 (assuming file1 is a file under the source directory). | 68 (assuming file1 is a file under the source directory your-dir). |
| 69 | 69 |
| 70 4. If none of the above rules applies, gsutil performs a bucket listing to | 70 4. If none of the above rules applies, gsutil performs a bucket listing to |
| 71 determine if the target of the operation is a prefix match to the | 71 determine if the target of the operation is a prefix match to the |
| 72 specified string. For example, if you run the command: | 72 specified string. For example, if you run the command: |
| 73 | 73 |
| 74 gsutil cp your-file gs://your-bucket/abc | 74 gsutil cp your-file gs://your-bucket/abc |
| 75 | 75 |
| 76 gsutil will make a bucket listing request for the named bucket, using | 76 gsutil will make a bucket listing request for the named bucket, using |
| 77 delimiter="/" and prefix="abc". It will then examine the bucket listing | 77 delimiter="/" and prefix="abc". It will then examine the bucket listing |
| 78 results and determine whether there are objects in the bucket whose path | 78 results and determine whether there are objects in the bucket whose path |
| 79 starts with gs://your-bucket/abc/, to determine whether to treat the target | 79 starts with gs://your-bucket/abc/, to determine whether to treat the target |
| 80 as an object name or a directory name. In turn this impacts the name of the | 80 as an object name or a directory name. In turn this impacts the name of the |
| 81 object you create: If the above check indicates there is an "abc" directory | 81 object you create: If the above check indicates there is an "abc" directory |
| 82 you will end up with the object gs://your-bucket/abc/file; otherwise you | 82 you will end up with the object gs://your-bucket/abc/your-file; otherwise |
| 83 will end up with the object gs://your-bucket/abc. (See | 83 you will end up with the object gs://your-bucket/abc. (See |
| 84 "HOW NAMES ARE CONSTRUCTED" under "gsutil help cp" for more details.) | 84 "HOW NAMES ARE CONSTRUCTED" under "gsutil help cp" for more details.) |
| 85 | 85 |
| 86 This rule-based approach stands in contrast to the way many tools work, which | 86 This rule-based approach stands in contrast to the way many tools work, which |
| 87 create objects to mark the existence of folders (such as "dir_$folder$"). | 87 create objects to mark the existence of folders (such as "dir_$folder$"). |
| 88 gsutil understands several conventions used by such tools but does not | 88 gsutil understands several conventions used by such tools but does not |
| 89 require such marker objects to implement naming behavior consistent with | 89 require such marker objects to implement naming behavior consistent with |
| 90 UNIX commands. | 90 UNIX commands. |
| 91 | 91 |
| 92 A downside of the gsutil approach is it requires an extra bucket listing | 92 A downside of the gsutil approach is it requires an extra bucket listing |
| 93 before performing the needed cp or mv command. However those listings are | 93 before performing the needed cp or mv command. However those listings are |
| (...skipping 12 matching lines...) Expand all Loading... |
| 106 help_spec = HelpProvider.HelpSpec( | 106 help_spec = HelpProvider.HelpSpec( |
| 107 help_name='subdirs', | 107 help_name='subdirs', |
| 108 help_name_aliases=[ | 108 help_name_aliases=[ |
| 109 'dirs', 'directory', 'directories', 'folder', 'folders', 'hierarchy', | 109 'dirs', 'directory', 'directories', 'folder', 'folders', 'hierarchy', |
| 110 'subdir', 'subdirectory', 'subdirectories'], | 110 'subdir', 'subdirectory', 'subdirectories'], |
| 111 help_type='additional_help', | 111 help_type='additional_help', |
| 112 help_one_line_summary='How Subdirectories Work', | 112 help_one_line_summary='How Subdirectories Work', |
| 113 help_text=_DETAILED_HELP_TEXT, | 113 help_text=_DETAILED_HELP_TEXT, |
| 114 subcommand_help_text={}, | 114 subcommand_help_text={}, |
| 115 ) | 115 ) |
| OLD | NEW |