OLD | NEW |
| (Empty) |
1 # Copyright 2012 Google Inc. All Rights Reserved. | |
2 # | |
3 # Licensed under the Apache License, Version 2.0 (the "License"); | |
4 # you may not use this file except in compliance with the License. | |
5 # You may obtain a copy of the License at | |
6 # | |
7 # http://www.apache.org/licenses/LICENSE-2.0 | |
8 # | |
9 # Unless required by applicable law or agreed to in writing, software | |
10 # distributed under the License is distributed on an "AS IS" BASIS, | |
11 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | |
12 # See the License for the specific language governing permissions and | |
13 # limitations under the License. | |
14 | |
15 from gslib.help_provider import HELP_NAME | |
16 from gslib.help_provider import HELP_NAME_ALIASES | |
17 from gslib.help_provider import HELP_ONE_LINE_SUMMARY | |
18 from gslib.help_provider import HelpProvider | |
19 from gslib.help_provider import HELP_TEXT | |
20 from gslib.help_provider import HelpType | |
21 from gslib.help_provider import HELP_TYPE | |
22 | |
23 _detailed_help_text = (""" | |
24 <B>OVERVIEW</B> | |
25 Versioning-enabled buckets maintain an archive of objects, providing a way to | |
26 un-delete data that you accidentally deleted, or to retrieve older versions of | |
27 your data. You can turn versioning on or off for a bucket at any time. Turning | |
28 versioning off leaves existing object versions in place, and simply causes the | |
29 bucket to stop accumulating new object versions. In this case, if you upload | |
30 to an existing object the current version is overwritten instead of creating | |
31 a new version. | |
32 | |
33 Regardless of whether you have enabled versioning on a bucket, every object | |
34 has two associated positive integer fields: | |
35 - the generation, which is updated when the content of an object is | |
36 overwritten. | |
37 - the meta-generation, which identifies the metadata generation. It starts | |
38 at 1; is updated every time the metadata (e.g., ACL or Content-Type) for a | |
39 given content generation is updated; and gets reset when the generation | |
40 number changes. | |
41 | |
42 Of these two integers, only the generation is used when working with versioned | |
43 data. Both generation and meta-generation can be used with concurrency control | |
44 (discussed in a later section). | |
45 | |
46 To work with object versioning in gsutil, you can use a flavor of storage URIs | |
47 that that embed the object generation, which we refer to as version-specific U
RIs. | |
48 For example, the version-less object URI: | |
49 | |
50 gs://bucket/object | |
51 | |
52 might have have two versions, with these version-specific URIs: | |
53 | |
54 gs://bucket/object#1360383693690000 | |
55 gs://bucket/object#1360383802725000 | |
56 | |
57 The following sections discuss how to work with versioning and concurrency | |
58 control. | |
59 | |
60 | |
61 <B>OBJECT VERSIONING</B> | |
62 You can view, enable, and disable object versioning on a bucket using | |
63 the getversioning and setversioning commands. For example: | |
64 | |
65 gsutil setversioning on gs://bucket | |
66 | |
67 will enable versioning for the named bucket. See 'gsutil help getversioning' | |
68 and 'gsutil help setversioning' for additional details. | |
69 | |
70 To see all object versions in a versioning-enabled bucket along with | |
71 their generation.meta-generation information, use gsutil ls -a: | |
72 | |
73 gsutil ls -a gs://bucket | |
74 | |
75 You can also specify particular objects for which you want to find the | |
76 version-specific URI(s), or you can use wildcards: | |
77 | |
78 gsutil ls -a gs://bucket/object1 gs://bucket/images/*.jpg | |
79 | |
80 The generation values form a monotonically increasing sequence as you create | |
81 additional object versions. Because of this, the latest object version is | |
82 always the last one listed in the gsutil ls output for a particular object. | |
83 For example, if a bucket contains these three versions of gs://bucket/object: | |
84 | |
85 gs://bucket/object#1360035307075000 | |
86 gs://bucket/object#1360101007329000 | |
87 gs://bucket/object#1360102216114000 | |
88 | |
89 then gs://bucket/object#1360102216114000 is the latest version and | |
90 gs://bucket/object#1360035307075000 is the oldest available version. | |
91 | |
92 If you specify version-less URIs with gsutil, you will operate on the | |
93 latest not-deleted version of an object, for example: | |
94 | |
95 gsutil cp gs://bucket/object ./dir | |
96 | |
97 or | |
98 | |
99 gsutil rm gs://bucket/object | |
100 | |
101 To operate on a specific object version, use a version-specific URI. | |
102 For example, suppose the output of the above gsutil ls -a command is: | |
103 | |
104 gs://bucket/object#1360035307075000 | |
105 gs://bucket/object#1360101007329000 | |
106 | |
107 In this case, the command: | |
108 | |
109 gsutil cp gs://bucket/object#1360035307075000 ./dir | |
110 | |
111 will retrieve the second most recent version of the object. | |
112 | |
113 Note that version-specific URIs cannot be the target of the gsutil cp | |
114 command (trying to do so will result in an error), because writing to a | |
115 versioned object always creates a new version. | |
116 | |
117 If an object has been deleted, it will not show up in a normal gsutil ls | |
118 listing (i.e., ls without the -a option). You can restore a deleted object by | |
119 running gsutil ls -a to find the available versions, and then copying one of | |
120 the version-specific URIs to the version-less URI, for example: | |
121 | |
122 gsutil cp gs://bucket/object#1360101007329000 gs://bucket/object | |
123 | |
124 Note that when you do this it creates a new object version, which will incur | |
125 additional charges. You can get rid of the extra copy by deleting the older | |
126 version-specfic object: | |
127 | |
128 gsutil rm gs://bucket/object#1360101007329000 | |
129 | |
130 Or you can combine the two steps by using the gsutil mv command: | |
131 | |
132 gsutil mv gs://bucket/object#1360101007329000 gs://bucket/object | |
133 | |
134 If you want to remove all versions of an object use the gsutil rm -a option: | |
135 | |
136 gsutil rm -a gs://bucket/object | |
137 | |
138 Note that there is no limit to the number of older versions of an object you | |
139 will create if you continue to upload to the same object in a versioning- | |
140 enabled bucket. It is your responsibility to delete versions beyond the ones | |
141 you want to retain. | |
142 | |
143 | |
144 <B>CONCURRENCY CONTROL</B> | |
145 If you are building an application using Google Cloud Storage, you may need to | |
146 be careful about concurrency control. Normally gsutil itself isn't used for | |
147 this purpose, but it's possible to write scripts around gsutil that perform | |
148 concurrency control. | |
149 | |
150 For example, suppose you want to implement a "rolling update" system using | |
151 gsutil, where a periodic job computes some data and uploads it to the cloud. | |
152 On each run, the job starts with the data that it computed from last run, and | |
153 computes a new value. To make this system robust, you need to have multiple | |
154 machines on which the job can run, which raises the possibility that two | |
155 simultaneous runs could attempt to update an object at the same time. This | |
156 leads to the following potential race condition: | |
157 - job 1 computes the new value to be written | |
158 - job 2 computes the new value to be written | |
159 - job 2 writes the new value | |
160 - job 1 writes the new value | |
161 | |
162 In this case, the value that job 1 read is no longer current by the time | |
163 it goes to write the updated object, and writing at this point would result | |
164 in stale (or, depending on the application, corrupt) data. | |
165 | |
166 To prevent this, you can find the version-specific name of the object that was | |
167 created, and then use the information contained in that URI to specify an | |
168 x-goog-if-generation-match header on a subsequent gsutil cp command. You can | |
169 do this in two steps. First, use the gsutil cp -v option at upload time to get | |
170 the version-specific name of the object that was created, for example: | |
171 | |
172 gsutil cp -v file gs://bucket/object | |
173 | |
174 might output: | |
175 | |
176 Created: gs://bucket/object#1360432179236000 | |
177 | |
178 You can extract the generation value from this object and then construct a | |
179 subsequent gsutil command like this: | |
180 | |
181 gsutil -h x-goog-if-generation-match:1360432179236000 cp newfile \\ | |
182 gs://bucket/object | |
183 | |
184 This command requests Google Cloud Storage to attempt to upload newfile | |
185 but to fail the request if the generation of newfile that is live at the | |
186 time of the upload does not match that specified. | |
187 | |
188 If the command you use updates object metadata, you will need to find the | |
189 current meta_generation for an object. To do this, use the gsutil ls -a and | |
190 -l options. For example, the command: | |
191 | |
192 gsutil ls -l -a gs://bucket/object | |
193 | |
194 will output something like: | |
195 | |
196 64 2013-02-12T19:59:13 gs://bucket/object#1360699153986000 meta_generat
ion=3 | |
197 1521 2013-02-13T02:04:08 gs://bucket/object#1360721048778000 meta_generat
ion=2 | |
198 | |
199 Given this information, you could use the following command to request setting | |
200 the ACL on the older version of the object, such that the command will fail | |
201 unless that is the current version of the data+metadata: | |
202 | |
203 gsutil -h x-goog-if-generation-match:1360699153986000 -h \\ | |
204 x-goog-if-metageneration-match:3 setacl public-read \\ | |
205 gs://bucket/object#1360699153986000 | |
206 | |
207 Without adding these headers, the update would simply overwrite the existing | |
208 ACL. Note that in contrast, the gsutil chacl command uses these headers | |
209 automatically, because it performs a read-modify-write cycle in order to edit | |
210 ACLs. | |
211 | |
212 If you want to experiment with how generations and metagenerations work, try | |
213 the following. First, upload an object; then use gsutil ls -l -a to list all | |
214 versions of the object, along with each version's meta_generation; then re- | |
215 upload the object and repeat the gsutil ls -l -a. You should see two object | |
216 versions, each with meta_generation=1. Now try setting the ACL, and rerun the | |
217 gsutil ls -l -a. You should see the most recent object generation now has | |
218 meta_generation=2. | |
219 | |
220 | |
221 <B>FOR MORE INFORMATION</B> | |
222 For more details on how to use versioning and preconditions, see | |
223 https://developers.google.com/storage/docs/object-versioning | |
224 """) | |
225 | |
226 | |
227 class CommandOptions(HelpProvider): | |
228 """Additional help about object versioning.""" | |
229 | |
230 help_spec = { | |
231 # Name of command or auxiliary help info for which this help applies. | |
232 HELP_NAME : 'versioning', | |
233 # List of help name aliases. | |
234 HELP_NAME_ALIASES : ['concurrency', 'concurrency control', 'versioning', | |
235 'versions'], | |
236 # Type of help: | |
237 HELP_TYPE : HelpType.ADDITIONAL_HELP, | |
238 # One line summary of this help. | |
239 HELP_ONE_LINE_SUMMARY : 'Working with object versions; concurrency control', | |
240 # The full help text. | |
241 HELP_TEXT : _detailed_help_text, | |
242 } | |
OLD | NEW |