Add 'fetch' command to gclient with support for --jobs.

Rework gclient 'recurse' command to use a WorkQueue.

Support --jobs in 'fetch' and 'recurse' commands. 

BUG=115840
TEST=

[M-A Ruel, 2012-02-27 16:52:48]

https://chromiumcodereview.appspot.com/9463044/diff/1/gclient.py
File gclient.py (right):

https://chromiumcodereview.appspot.com/9463044/diff/1/gclient.py#newcode1174
gclient.py:1174: parser.remove_option('--jobs')
It's useful and safe to do them in parallel. I'd keep --jobs.

https://chromiumcodereview.appspot.com/9463044/diff/1/gclient.py#newcode1175
gclient.py:1175: (options, parsed_args) = parser.parse_args(args)
Why parse the arguments here? Just forward the modified args directly to
CMDrecurse?

https://chromiumcodereview.appspot.com/9463044/diff/1/gclient.py#newcode1179
gclient.py:1179: args.extend('-s git git fetch origin'.split(' '))
Let's do
args = ['-s', 'git', 'git', 'fetch'] + args
so origin is not specified by default, so people with multiple remote have all
the remote fetched by default and that specifying a remote work properly, e.g.
gclient fetch origin
works intuitively.

[davidbarr, 2012-02-28 00:16:03]

On 2012/02/27 16:52:48, Marc-Antoine Ruel wrote:
> https://chromiumcodereview.appspot.com/9463044/diff/1/gclient.py
> File gclient.py (right):
> 
> https://chromiumcodereview.appspot.com/9463044/diff/1/gclient.py#newcode1174
> gclient.py:1174: parser.remove_option('--jobs')
> It's useful and safe to do them in parallel. I'd keep --jobs.

There is currently no support for --jobs in 'gclient recurse'.
Also, there is the question of how to combine the output from parallel fetches.

> https://chromiumcodereview.appspot.com/9463044/diff/1/gclient.py#newcode1175
> gclient.py:1175: (options, parsed_args) = parser.parse_args(args)
> Why parse the arguments here? Just forward the modified args directly to
> CMDrecurse?

I think the desired template is:
gclient fetch [options] [args] => gclient recurse [options] -s git git fetch
[args]

> https://chromiumcodereview.appspot.com/9463044/diff/1/gclient.py#newcode1179
> gclient.py:1179: args.extend('-s git git fetch origin'.split(' '))
> Let's do
> args = ['-s', 'git', 'git', 'fetch'] + args
> so origin is not specified by default, so people with multiple remote have all
> the remote fetched by default and that specifying a remote work properly, e.g.
> gclient fetch origin
> works intuitively.

I agree with this change. However, note that git fetch:
Defaults to the upstream remote of the current branch.
If there is no such remote, falls back to origin.
To fetch all remotes the --all flag or a remotes group may be required.

[M-A Ruel, 2012-02-28 14:49:53]

On 2012/02/28 00:16:03, davidbarr wrote:
> On 2012/02/27 16:52:48, Marc-Antoine Ruel wrote:
> > https://chromiumcodereview.appspot.com/9463044/diff/1/gclient.py
> > File gclient.py (right):
> > 
> > https://chromiumcodereview.appspot.com/9463044/diff/1/gclient.py#newcode1174
> > gclient.py:1174: parser.remove_option('--jobs')
> > It's useful and safe to do them in parallel. I'd keep --jobs.
> 
> There is currently no support for --jobs in 'gclient recurse'.

I think it's worth to add it back. There is no reason to not run them in
parallel.

> Also, there is the question of how to combine the output from parallel
fetches.

gclient handles that automatically. You don't need to do anything. Just enable
--jobs support and the rest will be auto-magical.


> > https://chromiumcodereview.appspot.com/9463044/diff/1/gclient.py#newcode1175
> > gclient.py:1175: (options, parsed_args) = parser.parse_args(args)
> > Why parse the arguments here? Just forward the modified args directly to
> > CMDrecurse?
> 
> I think the desired template is:
> gclient fetch [options] [args] => gclient recurse [options] -s git git fetch
> [args]

You are right. See below.

> > https://chromiumcodereview.appspot.com/9463044/diff/1/gclient.py#newcode1179
> > gclient.py:1179: args.extend('-s git git fetch origin'.split(' '))
> > Let's do
> > args = ['-s', 'git', 'git', 'fetch'] + args
> > so origin is not specified by default, so people with multiple remote have
all
> > the remote fetched by default and that specifying a remote work properly,
e.g.
> > gclient fetch origin
> > works intuitively.
> 
> I agree with this change. However, note that git fetch:
> Defaults to the upstream remote of the current branch.
> If there is no such remote, falls back to origin.
> To fetch all remotes the --all flag or a remotes group may be required.

I had forgot that --all wasn't the default, duh. Bah, I don't have much opinion.
I'm fine with the current state.

M-A

[davidbarr, 2012-02-29 03:58:21]

I'm pretty happy with this as a starting point for the new fetch command.
It is still the case that the recurse command does not use a work queue, so
--jobs has no effect.

The following syntax is supported but not explicitly described:
gclient fetch [gclient options] -- [git options] [git args]

To fetch the curent upstream remote:
gclient fetch

To fetch specifically origin:
gclient fetch origin

To fetch all remotes:
gclient fetch -- --all

[davidbarr, 2012-03-01 01:15:29]

These changes probably ought to be in two patches. However, I want to illustrate
how gclient recurse might be adapted to a workqueue. Supporting --jobs makes
gclient fetch much more valuable.

Compare the results with and without --jobs=8 from Sydney.

$ time gclient fetch [-j8]
git fetch:  98% (67/68), done.                                              

real	0m8.873s
user	0m0.949s
sys	0m1.693s

$ time gclient fetch -j1
git fetch:  98% (67/68), done.

real	0m42.886s
user	0m0.911s
sys	0m1.422s

This is kind of moot anyway with local mirror:

$ git config --global url.git://chromium/.insteadof http://git.chromium.org/
$ time gclient fetch -j1
git fetch:  98% (67/68), done.

real	0m6.825s
user	0m0.734s
sys	0m1.155s

[M-A Ruel, 2012-03-01 01:33:33]

On 2012/03/01 01:15:29, davidbarr wrote:
> These changes probably ought to be in two patches.

I agree, can you revert to patchset 3 then lgtm?

[commit-bot: I haz the power, 2012-03-01 02:11:26]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/davidbarr@chromium.org/9463044/6002

[commit-bot: I haz the power, 2012-03-01 02:18:33]

Change committed as 124334