Chromium Code Reviews
chromiumcodereview-hr@appspot.gserviceaccount.com (chromiumcodereview-hr) | Please choose your nickname with Settings | Help | Chromium Project | Gerrit Changes | Sign out
(76)

Side by Side Diff: third_party/gsutil/gslib/addlhelp/subdirs.py

Issue 1380943003: Roll version of gsutil to 4.15. (Closed) Base URL: https://github.com/catapult-project/catapult.git@master
Patch Set: rebase Created 5 years ago
Use n/p to move between diff chunks; N/P to move between comments. Draft comments are only viewable by you.
Jump to:
View unified diff | Download patch
OLDNEW
1 # -*- coding: utf-8 -*- 1 # -*- coding: utf-8 -*-
2 # Copyright 2012 Google Inc. All Rights Reserved. 2 # Copyright 2012 Google Inc. All Rights Reserved.
3 # 3 #
4 # Licensed under the Apache License, Version 2.0 (the "License"); 4 # Licensed under the Apache License, Version 2.0 (the "License");
5 # you may not use this file except in compliance with the License. 5 # you may not use this file except in compliance with the License.
6 # You may obtain a copy of the License at 6 # You may obtain a copy of the License at
7 # 7 #
8 # http://www.apache.org/licenses/LICENSE-2.0 8 # http://www.apache.org/licenses/LICENSE-2.0
9 # 9 #
10 # Unless required by applicable law or agreed to in writing, software 10 # Unless required by applicable law or agreed to in writing, software
(...skipping 15 matching lines...) Expand all
26 additional documentation to help users understand how gsutil handles 26 additional documentation to help users understand how gsutil handles
27 subdirectories differently than most GUI / web-based tools (e.g., why 27 subdirectories differently than most GUI / web-based tools (e.g., why
28 those other tools create "dir_$folder$" objects), and also to explain cost and 28 those other tools create "dir_$folder$" objects), and also to explain cost and
29 performance implications of the gsutil approach, for those interested in such 29 performance implications of the gsutil approach, for those interested in such
30 details. 30 details.
31 31
32 gsutil provides the illusion of a hierarchical file tree atop the "flat" 32 gsutil provides the illusion of a hierarchical file tree atop the "flat"
33 name space supported by the Google Cloud Storage service. To the service, 33 name space supported by the Google Cloud Storage service. To the service,
34 the object gs://your-bucket/abc/def/ghi.txt is just an object that happens to 34 the object gs://your-bucket/abc/def/ghi.txt is just an object that happens to
35 have "/" characters in its name. There are no "abc" or "abc/def" directories; 35 have "/" characters in its name. There are no "abc" or "abc/def" directories;
36 just a single object with the given name. This 36 just a single object with the given name. This diagram:
37 `diagram <https://cloud.google.com/storage/images/gsutil-subdirectories-thumb. png>` 37 https://cloud.google.com/storage/images/gsutil-subdirectories-thumb.png
38 illustrates how gsutil provides a hierarchical view of objects in a bucket. 38 illustrates how gsutil provides a hierarchical view of objects in a bucket.
39 39
40 gsutil achieves the hierarchical file tree illusion by applying a variety of 40 gsutil achieves the hierarchical file tree illusion by applying a variety of
41 rules, to try to make naming work the way users would expect. For example, in 41 rules, to try to make naming work the way users would expect. For example, in
42 order to determine whether to treat a destination URI as an object name or the 42 order to determine whether to treat a destination URL as an object name or the
43 root of a directory under which objects should be copied gsutil uses these 43 root of a directory under which objects should be copied gsutil uses these
44 rules: 44 rules:
45 45
46 1. If the destination object ends with a "/" gsutil treats it as a directory. 46 1. If the destination object ends with a "/" gsutil treats it as a directory.
47 For example, if you run the command: 47 For example, if you run the command:
48 48
49 gsutil cp your-file gs://your-bucket/abc/ 49 gsutil cp your-file gs://your-bucket/abc/
50 50
51 gsutil will create the object gs://your-bucket/abc/file. 51 gsutil will create the object gs://your-bucket/abc/your-file.
52 52
53 2. If the destination object is XYZ and an object exists called XYZ_$folder$ 53 2. If the destination object is XYZ and an object exists called XYZ_$folder$
54 gsutil treats XYZ as a directory. For example, if you run the command: 54 gsutil treats XYZ as a directory. For example, if you run the command:
55 55
56 gsutil cp your-file gs://your-bucket/abc 56 gsutil cp your-file gs://your-bucket/abc
57 57
58 and there exists an object called abc_$folder$, gsutil will create the 58 and there exists an object called abc_$folder$, gsutil will create the
59 object gs://your-bucket/abc/file. 59 object gs://your-bucket/abc/your-file.
60 60
61 3. If you attempt to copy multiple source files to a destination URI, gsutil 61 3. If you attempt to copy multiple source files to a destination URL, gsutil
62 treats the destination URI as a directory. For example, if you run 62 treats the destination URL as a directory. For example, if you run
63 the command: 63 the command:
64 64
65 gsutil cp -r your-dir gs://your-bucket/abc 65 gsutil cp -r your-dir gs://your-bucket/abc
66 66
67 gsutil will create objects like gs://your-bucket/abc/your-dir/file1, etc. 67 gsutil will create objects like gs://your-bucket/abc/your-dir/file1, etc.
68 (assuming file1 is a file under the source directory). 68 (assuming file1 is a file under the source directory your-dir).
69 69
70 4. If none of the above rules applies, gsutil performs a bucket listing to 70 4. If none of the above rules applies, gsutil performs a bucket listing to
71 determine if the target of the operation is a prefix match to the 71 determine if the target of the operation is a prefix match to the
72 specified string. For example, if you run the command: 72 specified string. For example, if you run the command:
73 73
74 gsutil cp your-file gs://your-bucket/abc 74 gsutil cp your-file gs://your-bucket/abc
75 75
76 gsutil will make a bucket listing request for the named bucket, using 76 gsutil will make a bucket listing request for the named bucket, using
77 delimiter="/" and prefix="abc". It will then examine the bucket listing 77 delimiter="/" and prefix="abc". It will then examine the bucket listing
78 results and determine whether there are objects in the bucket whose path 78 results and determine whether there are objects in the bucket whose path
79 starts with gs://your-bucket/abc/, to determine whether to treat the target 79 starts with gs://your-bucket/abc/, to determine whether to treat the target
80 as an object name or a directory name. In turn this impacts the name of the 80 as an object name or a directory name. In turn this impacts the name of the
81 object you create: If the above check indicates there is an "abc" directory 81 object you create: If the above check indicates there is an "abc" directory
82 you will end up with the object gs://your-bucket/abc/file; otherwise you 82 you will end up with the object gs://your-bucket/abc/your-file; otherwise
83 will end up with the object gs://your-bucket/abc. (See 83 you will end up with the object gs://your-bucket/abc. (See
84 "HOW NAMES ARE CONSTRUCTED" under "gsutil help cp" for more details.) 84 "HOW NAMES ARE CONSTRUCTED" under "gsutil help cp" for more details.)
85 85
86 This rule-based approach stands in contrast to the way many tools work, which 86 This rule-based approach stands in contrast to the way many tools work, which
87 create objects to mark the existence of folders (such as "dir_$folder$"). 87 create objects to mark the existence of folders (such as "dir_$folder$").
88 gsutil understands several conventions used by such tools but does not 88 gsutil understands several conventions used by such tools but does not
89 require such marker objects to implement naming behavior consistent with 89 require such marker objects to implement naming behavior consistent with
90 UNIX commands. 90 UNIX commands.
91 91
92 A downside of the gsutil approach is it requires an extra bucket listing 92 A downside of the gsutil approach is it requires an extra bucket listing
93 before performing the needed cp or mv command. However those listings are 93 before performing the needed cp or mv command. However those listings are
(...skipping 12 matching lines...) Expand all
106 help_spec = HelpProvider.HelpSpec( 106 help_spec = HelpProvider.HelpSpec(
107 help_name='subdirs', 107 help_name='subdirs',
108 help_name_aliases=[ 108 help_name_aliases=[
109 'dirs', 'directory', 'directories', 'folder', 'folders', 'hierarchy', 109 'dirs', 'directory', 'directories', 'folder', 'folders', 'hierarchy',
110 'subdir', 'subdirectory', 'subdirectories'], 110 'subdir', 'subdirectory', 'subdirectories'],
111 help_type='additional_help', 111 help_type='additional_help',
112 help_one_line_summary='How Subdirectories Work', 112 help_one_line_summary='How Subdirectories Work',
113 help_text=_DETAILED_HELP_TEXT, 113 help_text=_DETAILED_HELP_TEXT,
114 subcommand_help_text={}, 114 subcommand_help_text={},
115 ) 115 )
OLDNEW
« no previous file with comments | « third_party/gsutil/gslib/addlhelp/security.py ('k') | third_party/gsutil/gslib/addlhelp/throttling.py » ('j') | no next file with comments »

Powered by Google App Engine
This is Rietveld 408576698