Chromium Code Reviews
chromiumcodereview-hr@appspot.gserviceaccount.com (chromiumcodereview-hr) | Please choose your nickname with Settings | Help | Chromium Project | Gerrit Changes | Sign out
(33)

Issue 807463005: Add support for tar.gz archive files to download from download_from_google_storage (Closed)

Created:
5 years, 11 months ago by ricow1
Modified:
5 years, 5 months ago
Reviewers:
Ryan Tseng, azarchs, hinoka
CC:
chromium-reviews, cmp-cc_chromium.org, dpranke+depot_tools_chromium.org, iannucci+depot_tools_chromium.org
Target Ref:
refs/heads/master
Visibility:
Public.

Description

Add support for tar.gz archive files to download from download_from_google_storage Also, add support for directly setting the public read flag on uploaded files. The support for tar.gz allows the uploaded bundle to be a single tar.gz file instead of a lot of individual files using the directory support already in. The benefit here is that it is much easier to update the dependency. Simply clean out the existing files, copy in the new ones, create a tar.gz file, with the same name as the directory + 'tar.gz'. If the directory name and file name does not match up we will not clean up existing artifacts on download (i.e., there can be left over files after extracting). I am doing this because I am moving a bunch of the dart dependencies to gcs, and a lot of our dependencies is much easier to manage with this in. If you don't like this, I can simply wrap the download script in another python script and do the logic there, but this may be handy for other people as well. Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=295872

Patch Set 1 #

Total comments: 10

Patch Set 2 : #

Total comments: 4

Patch Set 3 : #

Total comments: 18

Patch Set 4 : #

Patch Set 5 : #

Total comments: 7

Patch Set 6 : add some tests #

Total comments: 8

Patch Set 7 : address comments #

Total comments: 20

Patch Set 8 : fixes #

Total comments: 1
Unified diffs Side-by-side diffs Delta from patch set Stats (+224 lines, -11 lines) Patch
M download_from_google_storage.py View 1 2 3 4 5 6 9 chunks +55 lines, -6 lines 0 comments Download
M tests/download_from_google_storage_unittests.py View 1 2 3 4 5 6 7 8 chunks +106 lines, -5 lines 0 comments Download
M tests/upload_to_google_storage_unittests.py View 1 2 3 4 5 6 7 3 chunks +26 lines, -0 lines 0 comments Download
M upload_to_google_storage.py View 1 2 3 4 5 6 7 3 chunks +37 lines, -0 lines 1 comment Download

Messages

Total messages: 34 (7 generated)
ricow1
Please let me know if this is something you want in and I will update ...
5 years, 11 months ago (2015-01-19 16:02:28 UTC) #2
hinoka
I like having tar.gz support for this script. Thanks for the patch. A couple of ...
5 years, 11 months ago (2015-01-20 19:28:52 UTC) #4
ricow1
please take another look. Regarding testing: I have updated the existing tests to take the ...
5 years, 11 months ago (2015-01-22 15:46:04 UTC) #5
ricow1
ping
5 years, 10 months ago (2015-02-02 08:55:45 UTC) #6
ricow1
another friendly ping, this is blocking us on git migration. I can still just do ...
5 years, 10 months ago (2015-02-05 14:42:47 UTC) #7
hinoka
https://codereview.chromium.org/807463005/diff/20001/download_from_google_storage.py File download_from_google_storage.py (right): https://codereview.chromium.org/807463005/diff/20001/download_from_google_storage.py#newcode244 download_from_google_storage.py:244: or not output_filename.endswith('tar.gz')): '.tar.gz' to match with expectations on ...
5 years, 10 months ago (2015-02-05 18:44:17 UTC) #8
ricow1
https://codereview.chromium.org/807463005/diff/20001/download_from_google_storage.py File download_from_google_storage.py (right): https://codereview.chromium.org/807463005/diff/20001/download_from_google_storage.py#newcode244 download_from_google_storage.py:244: or not output_filename.endswith('tar.gz')): On 2015/02/05 18:44:16, hinoka wrote: > ...
5 years, 10 months ago (2015-02-06 11:52:29 UTC) #9
hinoka
Looks good in general, time for tests. Particularly, I'm not entirely sure about the path ...
5 years, 10 months ago (2015-02-06 18:54:35 UTC) #10
ricow1
ptal, re testing, see my previous comment: "Regarding testing: I have updated the existing tests ...
5 years, 10 months ago (2015-02-10 11:15:45 UTC) #11
hinoka
Re: testing For the _downloader_worker_thread() portion I think its sufficient to just add code coverage. ...
5 years, 10 months ago (2015-02-11 00:16:57 UTC) #12
ricow1
better late than never, here is an updated version with some tests. https://codereview.chromium.org/807463005/diff/80001/download_from_google_storage.py File download_from_google_storage.py ...
5 years, 6 months ago (2015-06-19 09:31:27 UTC) #13
ricow1
PTAL
5 years, 6 months ago (2015-06-22 06:56:02 UTC) #14
ricow1
ping
5 years, 6 months ago (2015-06-24 13:01:35 UTC) #15
hinoka
mostly good, last comments https://codereview.chromium.org/807463005/diff/100001/download_from_google_storage.py File download_from_google_storage.py (right): https://codereview.chromium.org/807463005/diff/100001/download_from_google_storage.py#newcode251 download_from_google_storage.py:251: tar = tarfile.open(output_filename, 'r:gz') with ...
5 years, 6 months ago (2015-06-24 18:54:38 UTC) #16
ricow1
address comments
5 years, 6 months ago (2015-06-25 06:22:56 UTC) #17
ricow1
https://codereview.chromium.org/807463005/diff/100001/download_from_google_storage.py File download_from_google_storage.py (right): https://codereview.chromium.org/807463005/diff/100001/download_from_google_storage.py#newcode251 download_from_google_storage.py:251: tar = tarfile.open(output_filename, 'r:gz') On 2015/06/24 18:54:38, hinoka wrote: ...
5 years, 6 months ago (2015-06-25 06:24:05 UTC) #18
Ryan Tseng
lgtm % comments https://codereview.chromium.org/807463005/diff/120001/tests/download_from_google_storage_unittests.py File tests/download_from_google_storage_unittests.py (right): https://codereview.chromium.org/807463005/diff/120001/tests/download_from_google_storage_unittests.py#newcode61 tests/download_from_google_storage_unittests.py:61: 2 newlines (top level block) https://codereview.chromium.org/807463005/diff/120001/tests/download_from_google_storage_unittests.py#newcode75 ...
5 years, 6 months ago (2015-06-25 17:19:15 UTC) #19
ricow1
fixes
5 years, 6 months ago (2015-06-26 06:17:43 UTC) #20
ricow1
https://codereview.chromium.org/807463005/diff/120001/tests/download_from_google_storage_unittests.py File tests/download_from_google_storage_unittests.py (right): https://codereview.chromium.org/807463005/diff/120001/tests/download_from_google_storage_unittests.py#newcode61 tests/download_from_google_storage_unittests.py:61: On 2015/06/25 17:19:14, Ryan Tseng wrote: > 2 newlines ...
5 years, 6 months ago (2015-06-26 06:18:05 UTC) #21
commit-bot: I haz the power
CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/patch-status/807463005/140001
5 years, 6 months ago (2015-06-26 06:18:23 UTC) #24
commit-bot: I haz the power
Try jobs failed on following builders: depot_tools_presubmit on tryserver.chromium.linux (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.linux/builders/depot_tools_presubmit/builds/101)
5 years, 6 months ago (2015-06-26 06:23:08 UTC) #26
ricow1
Ryan: looking at the owners file, you probably need to lgtm this with your @chromium.org ...
5 years, 6 months ago (2015-06-26 06:25:11 UTC) #27
hinoka
lgtm
5 years, 5 months ago (2015-06-29 20:30:50 UTC) #28
commit-bot: I haz the power
CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/patch-status/807463005/140001
5 years, 5 months ago (2015-06-30 06:11:04 UTC) #30
commit-bot: I haz the power
Committed patchset #8 (id:140001) as http://src.chromium.org/viewvc/chrome?view=rev&revision=295872
5 years, 5 months ago (2015-06-30 06:13:18 UTC) #31
azarchs
https://codereview.chromium.org/807463005/diff/140001/upload_to_google_storage.py File upload_to_google_storage.py (right): https://codereview.chromium.org/807463005/diff/140001/upload_to_google_storage.py#newcode242 upload_to_google_storage.py:242: help='Archive directory as a tar.gz file') This conflicts with ...
5 years, 5 months ago (2015-06-30 14:49:37 UTC) #33
boliu
5 years, 5 months ago (2015-06-30 15:06:32 UTC) #34
Message was sent while issue was closed.
A revert of this CL (patchset #8 id:140001) has been created in
https://codereview.chromium.org/1209033006/ by boliu@chromium.org.

The reason for reverting is: This conflicts with (and duplicates?) the -z option
below and has broken one of the chrome android downstream bots.

Safer to revert than trying to fix.

Powered by Google App Engine
This is Rietveld 408576698