Add a way for git cache to bootstrap off Google Storage.

Add a way for git cache to bootstrap off Google Storage.

This is based off the svn tarball bootstrap script in gclient_scm.py
Also included is a new command to create a new zarchive/zipball.

BUG=261741
Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=261002

[Ryan Tseng, 2014-04-01 01:22:22]

This is what I have today.  I've tested it on my local machine and it works. 
Have not tested it on a windows machine and i'm 80% sure it won't work.

[iannucci, 2014-04-01 05:47:11]

https://codereview.chromium.org/220423002/diff/40001/git_cache.py
File git_cache.py (right):

https://codereview.chromium.org/220423002/diff/40001/git_cache.py#newcode161
git_cache.py:161: """Create and uploads a bootstrap tarball."""
let's just assert this is on !win32 w/ a friendly message

https://codereview.chromium.org/220423002/diff/40001/git_cache.py#newcode164
git_cache.py:164: CMDpopulate(parser, args)
you'll want to do a full gc repack

https://codereview.chromium.org/220423002/diff/40001/git_cache.py#newcode167
git_cache.py:167: gen_number = subprocess.check_output(['git',
'number']).strip()
git number master

https://codereview.chromium.org/220423002/diff/40001/git_cache.py#newcode172
git_cache.py:172: url = args[0]
I don't think these lines are needed b/c CMDpopulate

https://codereview.chromium.org/220423002/diff/40001/git_cache.py#newcode229
git_cache.py:229: path_folders = os.environ.get('PATH').split(':')
os.pathsep

https://codereview.chromium.org/220423002/diff/40001/git_cache.py#newcode288
git_cache.py:288: gclient_utils.rmtree(tempdir)
this whole chunk of code begs to be refactored into a ZipTool class with a zip
and an unzip method...

[Ryan Tseng, 2014-04-01 20:18:09]

ptal!

https://codereview.chromium.org/220423002/diff/40001/git_cache.py
File git_cache.py (right):

https://codereview.chromium.org/220423002/diff/40001/git_cache.py#newcode161
git_cache.py:161: """Create and uploads a bootstrap tarball."""
On 2014/04/01 05:47:11, iannucci wrote:
> let's just assert this is on !win32 w/ a friendly message

Done.

https://codereview.chromium.org/220423002/diff/40001/git_cache.py#newcode164
git_cache.py:164: CMDpopulate(parser, args)
On 2014/04/01 05:47:11, iannucci wrote:
> you'll want to do a full gc repack

Done.

https://codereview.chromium.org/220423002/diff/40001/git_cache.py#newcode167
git_cache.py:167: gen_number = subprocess.check_output(['git',
'number']).strip()
On 2014/04/01 05:47:11, iannucci wrote:
> git number master

Done.

https://codereview.chromium.org/220423002/diff/40001/git_cache.py#newcode172
git_cache.py:172: url = args[0]
On 2014/04/01 05:47:11, iannucci wrote:
> I don't think these lines are needed b/c CMDpopulate

Done.

https://codereview.chromium.org/220423002/diff/40001/git_cache.py#newcode229
git_cache.py:229: path_folders = os.environ.get('PATH').split(':')
On 2014/04/01 05:47:11, iannucci wrote:
> os.pathsep

Done.

https://codereview.chromium.org/220423002/diff/40001/git_cache.py#newcode288
git_cache.py:288: gclient_utils.rmtree(tempdir)
On 2014/04/01 05:47:11, iannucci wrote:
> this whole chunk of code begs to be refactored into a ZipTool class with a zip
> and an unzip method...

A whole new class to house 3 lines for zip and 5 lines for unzip seems
excessive... Maybe you mean for "Download from this gs location and unzip to
there".  But even then I feel like we'd be making classes for the sake of making
classes.

If it was used in one other place I'd agree with the need to refactor into a
generic "ziptool" that searches for 7z, unzip, and falls back to a python
implementation if it can't find either.  But I think thats unnecessarily complex
if this serves the need.

[agable, 2014-04-01 23:44:52]

LGTM after you've addressed comments.

https://codereview.chromium.org/220423002/diff/60001/git_cache.py
File git_cache.py (right):

https://codereview.chromium.org/220423002/diff/60001/git_cache.py#newcode17
git_cache.py:17: import download_from_google_storage
from download_from_google_storage import Gsutil? Easier to change that later if
Gsutil gets refactored out of that file.

https://codereview.chromium.org/220423002/diff/60001/git_cache.py#newcode168
git_cache.py:168: args.append('--no_bootstrap')
Would prefer to manipulate a copy of args, so that we don't add 'no-bootstrap'
here, send it to populate... and then parse our own 'no-bootstrap' on line 172
but ignore it.

https://codereview.chromium.org/220423002/diff/60001/git_cache.py#newcode226
git_cache.py:226: # Mimics the "which" utility.
Do a real docstring.

https://codereview.chromium.org/220423002/diff/60001/git_cache.py#newcode238
git_cache.py:238: # Bootstrap the repo from Google Stroage if there is a
pre-checked out
Do a real docstring.

https://codereview.chromium.org/220423002/diff/60001/git_cache.py#newcode246
git_cache.py:246: print 'Install 7z from http://www.7-zip.org/download.html if
you want '
...if you want git cache to be able to bootstrap...

https://codereview.chromium.org/220423002/diff/60001/git_cache.py#newcode252
git_cache.py:252: print 'Install unzip if you want to create a git cache to
boostrap '
...if you want git cache to be able to bootstrap...

https://codereview.chromium.org/220423002/diff/60001/git_cache.py#newcode283
git_cache.py:283: code = subprocess.call(cmd)
retcode

[Ryan Tseng, 2014-04-02 00:32:19]

Going to commit this now.

It'll probably break on Windows, but only on Windows bots that no one looks at
right now.

https://codereview.chromium.org/220423002/diff/60001/git_cache.py
File git_cache.py (right):

https://codereview.chromium.org/220423002/diff/60001/git_cache.py#newcode17
git_cache.py:17: import download_from_google_storage
On 2014/04/01 23:44:52, agable wrote:
> from download_from_google_storage import Gsutil? Easier to change that later
if
> Gsutil gets refactored out of that file.

Done.

https://codereview.chromium.org/220423002/diff/60001/git_cache.py#newcode168
git_cache.py:168: args.append('--no_bootstrap')
On 2014/04/01 23:44:52, agable wrote:
> Would prefer to manipulate a copy of args, so that we don't add 'no-bootstrap'
> here, send it to populate... and then parse our own 'no-bootstrap' on line 172
> but ignore it.

Done.

https://codereview.chromium.org/220423002/diff/60001/git_cache.py#newcode226
git_cache.py:226: # Mimics the "which" utility.
On 2014/04/01 23:44:52, agable wrote:
> Do a real docstring.

Done.

https://codereview.chromium.org/220423002/diff/60001/git_cache.py#newcode238
git_cache.py:238: # Bootstrap the repo from Google Stroage if there is a
pre-checked out
On 2014/04/01 23:44:52, agable wrote:
> Do a real docstring.

Done.

https://codereview.chromium.org/220423002/diff/60001/git_cache.py#newcode246
git_cache.py:246: print 'Install 7z from http://www.7-zip.org/download.html if
you want '
On 2014/04/01 23:44:52, agable wrote:
> ...if you want git cache to be able to bootstrap...

Done.

https://codereview.chromium.org/220423002/diff/60001/git_cache.py#newcode252
git_cache.py:252: print 'Install unzip if you want to create a git cache to
boostrap '
On 2014/04/01 23:44:52, agable wrote:
> ...if you want git cache to be able to bootstrap...

Done.

https://codereview.chromium.org/220423002/diff/60001/git_cache.py#newcode283
git_cache.py:283: code = subprocess.call(cmd)
On 2014/04/01 23:44:52, agable wrote:
> retcode

Done.

[Ryan Tseng, 2014-04-02 00:32:31]

The CQ bit was checked by hinoka@google.com

[commit-bot: I haz the power, 2014-04-02 00:32:38]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/hinoka@google.com/220423002/100001

[commit-bot: I haz the power, 2014-04-02 00:36:25]

Change committed as 261002