OLD | NEW |
1 # -*- coding: utf-8 -*- | 1 # -*- coding: utf-8 -*- |
2 # Copyright 2012 Google Inc. All Rights Reserved. | 2 # Copyright 2012 Google Inc. All Rights Reserved. |
3 # | 3 # |
4 # Licensed under the Apache License, Version 2.0 (the "License"); | 4 # Licensed under the Apache License, Version 2.0 (the "License"); |
5 # you may not use this file except in compliance with the License. | 5 # you may not use this file except in compliance with the License. |
6 # You may obtain a copy of the License at | 6 # You may obtain a copy of the License at |
7 # | 7 # |
8 # http://www.apache.org/licenses/LICENSE-2.0 | 8 # http://www.apache.org/licenses/LICENSE-2.0 |
9 # | 9 # |
10 # Unless required by applicable law or agreed to in writing, software | 10 # Unless required by applicable law or agreed to in writing, software |
(...skipping 15 matching lines...) Expand all Loading... |
26 additional documentation to help users understand how gsutil handles | 26 additional documentation to help users understand how gsutil handles |
27 subdirectories differently than most GUI / web-based tools (e.g., why | 27 subdirectories differently than most GUI / web-based tools (e.g., why |
28 those other tools create "dir_$folder$" objects), and also to explain cost and | 28 those other tools create "dir_$folder$" objects), and also to explain cost and |
29 performance implications of the gsutil approach, for those interested in such | 29 performance implications of the gsutil approach, for those interested in such |
30 details. | 30 details. |
31 | 31 |
32 gsutil provides the illusion of a hierarchical file tree atop the "flat" | 32 gsutil provides the illusion of a hierarchical file tree atop the "flat" |
33 name space supported by the Google Cloud Storage service. To the service, | 33 name space supported by the Google Cloud Storage service. To the service, |
34 the object gs://your-bucket/abc/def/ghi.txt is just an object that happens to | 34 the object gs://your-bucket/abc/def/ghi.txt is just an object that happens to |
35 have "/" characters in its name. There are no "abc" or "abc/def" directories; | 35 have "/" characters in its name. There are no "abc" or "abc/def" directories; |
36 just a single object with the given name. This | 36 just a single object with the given name. This diagram: |
37 `diagram <https://cloud.google.com/storage/images/gsutil-subdirectories-thumb.
png>` | 37 https://cloud.google.com/storage/images/gsutil-subdirectories-thumb.png |
38 illustrates how gsutil provides a hierarchical view of objects in a bucket. | 38 illustrates how gsutil provides a hierarchical view of objects in a bucket. |
39 | 39 |
40 gsutil achieves the hierarchical file tree illusion by applying a variety of | 40 gsutil achieves the hierarchical file tree illusion by applying a variety of |
41 rules, to try to make naming work the way users would expect. For example, in | 41 rules, to try to make naming work the way users would expect. For example, in |
42 order to determine whether to treat a destination URI as an object name or the | 42 order to determine whether to treat a destination URL as an object name or the |
43 root of a directory under which objects should be copied gsutil uses these | 43 root of a directory under which objects should be copied gsutil uses these |
44 rules: | 44 rules: |
45 | 45 |
46 1. If the destination object ends with a "/" gsutil treats it as a directory. | 46 1. If the destination object ends with a "/" gsutil treats it as a directory. |
47 For example, if you run the command: | 47 For example, if you run the command: |
48 | 48 |
49 gsutil cp your-file gs://your-bucket/abc/ | 49 gsutil cp your-file gs://your-bucket/abc/ |
50 | 50 |
51 gsutil will create the object gs://your-bucket/abc/file. | 51 gsutil will create the object gs://your-bucket/abc/your-file. |
52 | 52 |
53 2. If the destination object is XYZ and an object exists called XYZ_$folder$ | 53 2. If the destination object is XYZ and an object exists called XYZ_$folder$ |
54 gsutil treats XYZ as a directory. For example, if you run the command: | 54 gsutil treats XYZ as a directory. For example, if you run the command: |
55 | 55 |
56 gsutil cp your-file gs://your-bucket/abc | 56 gsutil cp your-file gs://your-bucket/abc |
57 | 57 |
58 and there exists an object called abc_$folder$, gsutil will create the | 58 and there exists an object called abc_$folder$, gsutil will create the |
59 object gs://your-bucket/abc/file. | 59 object gs://your-bucket/abc/your-file. |
60 | 60 |
61 3. If you attempt to copy multiple source files to a destination URI, gsutil | 61 3. If you attempt to copy multiple source files to a destination URL, gsutil |
62 treats the destination URI as a directory. For example, if you run | 62 treats the destination URL as a directory. For example, if you run |
63 the command: | 63 the command: |
64 | 64 |
65 gsutil cp -r your-dir gs://your-bucket/abc | 65 gsutil cp -r your-dir gs://your-bucket/abc |
66 | 66 |
67 gsutil will create objects like gs://your-bucket/abc/your-dir/file1, etc. | 67 gsutil will create objects like gs://your-bucket/abc/your-dir/file1, etc. |
68 (assuming file1 is a file under the source directory). | 68 (assuming file1 is a file under the source directory your-dir). |
69 | 69 |
70 4. If none of the above rules applies, gsutil performs a bucket listing to | 70 4. If none of the above rules applies, gsutil performs a bucket listing to |
71 determine if the target of the operation is a prefix match to the | 71 determine if the target of the operation is a prefix match to the |
72 specified string. For example, if you run the command: | 72 specified string. For example, if you run the command: |
73 | 73 |
74 gsutil cp your-file gs://your-bucket/abc | 74 gsutil cp your-file gs://your-bucket/abc |
75 | 75 |
76 gsutil will make a bucket listing request for the named bucket, using | 76 gsutil will make a bucket listing request for the named bucket, using |
77 delimiter="/" and prefix="abc". It will then examine the bucket listing | 77 delimiter="/" and prefix="abc". It will then examine the bucket listing |
78 results and determine whether there are objects in the bucket whose path | 78 results and determine whether there are objects in the bucket whose path |
79 starts with gs://your-bucket/abc/, to determine whether to treat the target | 79 starts with gs://your-bucket/abc/, to determine whether to treat the target |
80 as an object name or a directory name. In turn this impacts the name of the | 80 as an object name or a directory name. In turn this impacts the name of the |
81 object you create: If the above check indicates there is an "abc" directory | 81 object you create: If the above check indicates there is an "abc" directory |
82 you will end up with the object gs://your-bucket/abc/file; otherwise you | 82 you will end up with the object gs://your-bucket/abc/your-file; otherwise |
83 will end up with the object gs://your-bucket/abc. (See | 83 you will end up with the object gs://your-bucket/abc. (See |
84 "HOW NAMES ARE CONSTRUCTED" under "gsutil help cp" for more details.) | 84 "HOW NAMES ARE CONSTRUCTED" under "gsutil help cp" for more details.) |
85 | 85 |
86 This rule-based approach stands in contrast to the way many tools work, which | 86 This rule-based approach stands in contrast to the way many tools work, which |
87 create objects to mark the existence of folders (such as "dir_$folder$"). | 87 create objects to mark the existence of folders (such as "dir_$folder$"). |
88 gsutil understands several conventions used by such tools but does not | 88 gsutil understands several conventions used by such tools but does not |
89 require such marker objects to implement naming behavior consistent with | 89 require such marker objects to implement naming behavior consistent with |
90 UNIX commands. | 90 UNIX commands. |
91 | 91 |
92 A downside of the gsutil approach is it requires an extra bucket listing | 92 A downside of the gsutil approach is it requires an extra bucket listing |
93 before performing the needed cp or mv command. However those listings are | 93 before performing the needed cp or mv command. However those listings are |
(...skipping 12 matching lines...) Expand all Loading... |
106 help_spec = HelpProvider.HelpSpec( | 106 help_spec = HelpProvider.HelpSpec( |
107 help_name='subdirs', | 107 help_name='subdirs', |
108 help_name_aliases=[ | 108 help_name_aliases=[ |
109 'dirs', 'directory', 'directories', 'folder', 'folders', 'hierarchy', | 109 'dirs', 'directory', 'directories', 'folder', 'folders', 'hierarchy', |
110 'subdir', 'subdirectory', 'subdirectories'], | 110 'subdir', 'subdirectory', 'subdirectories'], |
111 help_type='additional_help', | 111 help_type='additional_help', |
112 help_one_line_summary='How Subdirectories Work', | 112 help_one_line_summary='How Subdirectories Work', |
113 help_text=_DETAILED_HELP_TEXT, | 113 help_text=_DETAILED_HELP_TEXT, |
114 subcommand_help_text={}, | 114 subcommand_help_text={}, |
115 ) | 115 ) |
OLD | NEW |