gclient: Add '--no-bootstrap' sync option.

gclient: Add '--no-bootstrap' sync option.

Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=294714

[USE eero AT chromium.org, 2015-04-01 11:34:16]

e.hakkinen@samsung.com changed reviewers:
+ hinoka@chromium.org, iannucci@chromium.org, jochen@chromium.org

[USE eero AT chromium.org, 2015-04-01 11:35:08]

Site local git mirrors can be much faster than remote Google Storage. This does
not change the default, as without local git mirrors bootstrapping helps a lot.

[hinoka, 2015-04-02 06:33:36]

I'm not particularly against this, but I'd like to know what situation does not
bootstrapping from Google Storage help.

Also this only applies to gclient specs with "cache_dir" set, which I don't know
if we've suggested anyone do just yet.

https://codereview.chromium.org/1050853002/diff/1/gclient.py
File gclient.py (right):

https://codereview.chromium.org/1050853002/diff/1/gclient.py#newcode2029
gclient.py:2029: help='Don\'t bootstrap from Google Storage')
nit: period

[USE eero AT chromium.org, 2015-04-02 08:00:45]

On 2015/04/02 06:33:36, hinoka wrote:
> I'm not particularly against this, but I'd like to know what situation does
not
> bootstrapping from Google Storage help.
> 
> Also this only applies to gclient specs with "cache_dir" set, which I don't
know
> if we've suggested anyone do just yet.

For various reasons, I have to gclient sync from scratch very often (i.e. to
create completely new checkout). For that, cache_dir helps a lot (and it works
for me).

I also often have to do that quite often on new hosts (or similar). To speed
that up, I add the following section to ~/.gitconfig:
 [url "http://hostname.localdomain/chromium/mirror/"]
   insteadOf = https://boringssl.googlesource.com/
   insteadOf = https://chromium.googlesource.com/
   insteadOf = https://pdfium.googlesource.com/

That way git fetches from a local mirror instead of from Google servers. That
makes fetches super fast as they come from internal network. That way full
fetches (from local mirrors) are even order of magnitude faster bootstrapping
from Google Storage. So in this particular case, bootstrapping from Google
Storage does not help.

[USE eero AT chromium.org, 2015-04-02 08:00:56]

https://codereview.chromium.org/1050853002/diff/1/gclient.py
File gclient.py (right):

https://codereview.chromium.org/1050853002/diff/1/gclient.py#newcode2029
gclient.py:2029: help='Don\'t bootstrap from Google Storage')
On 2015/04/02 06:33:35, hinoka wrote:
> nit: period

Done.

I copied the help message from git_cache.py but it seems that gclient uses
periods in help message while git cache populate does not.

[hinoka, 2015-04-03 02:06:39]

Google Storage bootstrapping not only downloads a bootstrapped zip from GS, but
also bypasses delta decompression, which takes a really long time on blink. 
Maybe you're not seeing the effects of that because you're cloning from a local
mirror that isn't sending a super compressed pack file?

Anyways this change looks small and valid. lgtm.

[USE eero AT chromium.org, 2015-04-07 07:34:49]

The CQ bit was checked by e.hakkinen@samsung.com

[commit-bot: I haz the power, 2015-04-07 07:34:56]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1050853002/20001

[commit-bot: I haz the power, 2015-04-07 07:38:17]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2015-04-07 07:38:19]

Presubmit check for 1050853002-20001 failed and returned exit status 1.

Running presubmit commit checks ...
Checking out rietveld...
Running save-description-on-failure.sh
Running push-basic.sh
Running upstream.sh
Running submit-from-new-dir.sh
Running abandon.sh
Running submodule-merge-test.sh
Running upload-local-tracking-branch.sh
Running hooks.sh
Running post-dcommit-hook-test.sh
Running upload-stale.sh
Running patch.sh
Running basic.sh

** Presubmit ERRORS **
tests/gclient_smoketest.py (53.45s) failed
Ffatal: reference is not a tree: 04cf0053efd2f0d001f5920a1d4dd71f86e2229b
EE..............................................
======================================================================
ERROR: testBlinkDEPSChangeUsingGit (__main__.BlinkDEPSTransitionSmokeTest)
Like testBlinkDEPSChangeUsingGclient, but move the main project using
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/gclient_smoketest.py", line 1630, in testBlinkDEPSChangeUsingGit
    cwd=self.checkout_path)
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/subprocess2.py", line
484, in check_call
    check_call_out(args, **kwargs)
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/subprocess2.py", line
478, in check_call_out
    returncode, args, kwargs.get('cwd'), out[0], out[1])
CalledProcessError: Command git checkout -q
04cf0053efd2f0d001f5920a1d4dd71f86e2229b returned non-zero exit status 128 in
/tmp/trialMz18wq/__main__.BlinkDEPSTransitionSmokeTest.testBlinkDEPSChangeUsingGit/src

======================================================================
ERROR: testBlinkLocalBranchesArePreserved
(__main__.BlinkDEPSTransitionSmokeTest)
Checks that the state of local git branches are effectively preserved
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/gclient_smoketest.py", line 1656, in
testBlinkLocalBranchesArePreserved
    self.CheckStatusPreMergePoint()
  File "tests/gclient_smoketest.py", line 1558, in CheckStatusPreMergePoint
    self.blink), self.blink_git_url)
  File "/b/infra_internal/commit_queue/workdir/tools/depot_tools/scm.py", line
121, in Capture
    cwd=cwd, stderr=subprocess2.PIPE, env=env, **kwargs)
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/subprocess2.py", line
515, in check_output
    return check_call_out(args, stdout=PIPE, **kwargs)[0]
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/subprocess2.py", line
475, in check_call_out
    out, returncode = communicate(args, **kwargs)
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/subprocess2.py", line
449, in communicate
    proc = Popen(args, **kwargs)
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/subprocess2.py", line
253, in __init__
    % (str(e), kwargs.get('cwd'), args[0]))
OSError: Execution failed with error: [Errno 2] No such file or directory:
'/tmp/trialMz18wq/__main__.BlinkDEPSTransitionSmokeTest.testBlinkLocalBranchesArePreserved/src/third_party/WebKit'.
Check that
/tmp/trialMz18wq/__main__.BlinkDEPSTransitionSmokeTest.testBlinkLocalBranchesArePreserved/src/third_party/WebKit
or git exist and have execution permission.

======================================================================
FAIL: testBlinkDEPSChangeUsingGclient (__main__.BlinkDEPSTransitionSmokeTest)
Checks that {src,blink} repos are consistent when syncing going back and
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/gclient_smoketest.py", line 1601, in
testBlinkDEPSChangeUsingGclient
    self.assertEqual(res[2], 0, 'DEPS change sync failed.')
AssertionError: DEPS change sync failed.

----------------------------------------------------------------------
Ran 49 tests in 53.242s

FAILED (failures=1, errors=2)


Presubmit checks took 189.8s to calculate.

[USE eero AT chromium.org, 2015-04-07 08:34:29]

The CQ bit was checked by e.hakkinen@samsung.com

[commit-bot: I haz the power, 2015-04-07 08:34:44]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1050853002/20001

[commit-bot: I haz the power, 2015-04-07 08:37:46]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2015-04-07 08:37:47]

Presubmit check for 1050853002-20001 failed and returned exit status 1.

Running presubmit commit checks ...
Checking out rietveld...
Running save-description-on-failure.sh
Running push-basic.sh
Running upstream.sh
Running submit-from-new-dir.sh
Running abandon.sh
Running submodule-merge-test.sh
Running upload-local-tracking-branch.sh
Running hooks.sh
Running post-dcommit-hook-test.sh
Running upload-stale.sh
Running patch.sh
Running basic.sh

** Presubmit ERRORS **
tests/gclient_smoketest.py (51.98s) failed
Ffatal: reference is not a tree: 79c33e9e88d023b9286ea602bd823817170c026e
EE..............................................
======================================================================
ERROR: testBlinkDEPSChangeUsingGit (__main__.BlinkDEPSTransitionSmokeTest)
Like testBlinkDEPSChangeUsingGclient, but move the main project using
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/gclient_smoketest.py", line 1630, in testBlinkDEPSChangeUsingGit
    cwd=self.checkout_path)
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/subprocess2.py", line
484, in check_call
    check_call_out(args, **kwargs)
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/subprocess2.py", line
478, in check_call_out
    returncode, args, kwargs.get('cwd'), out[0], out[1])
CalledProcessError: Command git checkout -q
79c33e9e88d023b9286ea602bd823817170c026e returned non-zero exit status 128 in
/tmp/trialiZYxRR/__main__.BlinkDEPSTransitionSmokeTest.testBlinkDEPSChangeUsingGit/src

======================================================================
ERROR: testBlinkLocalBranchesArePreserved
(__main__.BlinkDEPSTransitionSmokeTest)
Checks that the state of local git branches are effectively preserved
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/gclient_smoketest.py", line 1656, in
testBlinkLocalBranchesArePreserved
    self.CheckStatusPreMergePoint()
  File "tests/gclient_smoketest.py", line 1558, in CheckStatusPreMergePoint
    self.blink), self.blink_git_url)
  File "/b/infra_internal/commit_queue/workdir/tools/depot_tools/scm.py", line
121, in Capture
    cwd=cwd, stderr=subprocess2.PIPE, env=env, **kwargs)
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/subprocess2.py", line
515, in check_output
    return check_call_out(args, stdout=PIPE, **kwargs)[0]
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/subprocess2.py", line
475, in check_call_out
    out, returncode = communicate(args, **kwargs)
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/subprocess2.py", line
449, in communicate
    proc = Popen(args, **kwargs)
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/subprocess2.py", line
253, in __init__
    % (str(e), kwargs.get('cwd'), args[0]))
OSError: Execution failed with error: [Errno 2] No such file or directory:
'/tmp/trialiZYxRR/__main__.BlinkDEPSTransitionSmokeTest.testBlinkLocalBranchesArePreserved/src/third_party/WebKit'.
Check that
/tmp/trialiZYxRR/__main__.BlinkDEPSTransitionSmokeTest.testBlinkLocalBranchesArePreserved/src/third_party/WebKit
or git exist and have execution permission.

======================================================================
FAIL: testBlinkDEPSChangeUsingGclient (__main__.BlinkDEPSTransitionSmokeTest)
Checks that {src,blink} repos are consistent when syncing going back and
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/gclient_smoketest.py", line 1601, in
testBlinkDEPSChangeUsingGclient
    self.assertEqual(res[2], 0, 'DEPS change sync failed.')
AssertionError: DEPS change sync failed.

----------------------------------------------------------------------
Ran 49 tests in 51.835s

FAILED (failures=1, errors=2)


Presubmit checks took 176.6s to calculate.

[USE eero AT chromium.org, 2015-04-08 07:57:13]

The CQ bit was checked by e.hakkinen@samsung.com

[commit-bot: I haz the power, 2015-04-08 07:57:30]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1050853002/20001

[commit-bot: I haz the power, 2015-04-08 08:00:38]

Committed patchset #2 (id:20001) as
http://src.chromium.org/viewvc/chrome?view=rev&revision=294714