Reland "Make gclient ready for the Blink (DEPS to main project)"

Reland "Make gclient ready for the Blink (DEPS to main project)"

Reland crrev.com/743083002, which was reverted in crrev.com/796053002
due to some test flakiness, probably related with an old version of Git on
the bots. Relanding now that the infra has been updated to Trusty (plus
adding some de-flake precautions).

Original CL Description:
Make gclient ready for the Blink (DEPS to main project) transition

This CL makes gclient understand correctly whether a git project is
being moved from DEPS to an upper project and vice-versa.
The driving use case for this is the upcoming Blink merge, where
third_party/Webkit will be removed from DEPS (and .gitignore) and will
become part of the main project.

At present state, gclient leaves the .git folder around when a project
is removed from DEPS, and that causes many problems.

Furthermore this CL solves the performance problem of bisecting across
the merge point. The subproject's (Blink) .git/ folder is moved to a
backup location (in the main checkout root) and is restored when moving
backwards, avoiding a re-fetch when bisecting across the merge point.

BUG=431469
Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=294082

[Primiano Tucci (use gerrit), 2015-02-09 21:32:04]

primiano@chromium.org changed reviewers:
+ maruel@chromium.org

[Primiano Tucci (use gerrit), 2015-02-09 21:32:04]

M-A, with your benediction I'd like to re-land this (for the upcoming
chrome+blink merge).
It fell off my radar after I kept it on hold waiting for the infra update to
trusty).

Note: PS1 is identical to the original CL.
I made some minimal changes in PS2 to try do deal with the flakiness.

https://codereview.chromium.org/910913003/diff/1/testing_support/fake_repos.py
File testing_support/fake_repos.py (right):

https://codereview.chromium.org/910913003/diff/1/testing_support/fake_repos.p...
testing_support/fake_repos.py:389: self.populateGit()
Moving the git commits operations before starting the git daemon, in order to
reduce potential sources of flakiness.
Rationale: the flake experienced in
https://codereview.chromium.org/753543006/#msg18 shows that the "gclient sync"
command didn't pull the commit that was made in fake repo.
The way the test works today is by doing commits directly on the repo using the
git CLI (not pushing them) *after* the server has been started.
I wonder whether doing this, i.e. making raw commits under the git daemon feet, 
causes a race (e.g., if the git server keeps some cache of the refs).
Starting the server *after* the git repo has been populated seems more reliable
to me.

[M-A Ruel, 2015-02-11 16:34:47]

lgtm, thanks for uploading both patchsets for easier comparison, it's
appreciated.

[Primiano Tucci (use gerrit), 2015-02-11 16:37:55]

The CQ bit was checked by primiano@chromium.org

[commit-bot: I haz the power, 2015-02-11 16:38:09]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/910913003/20001

[commit-bot: I haz the power, 2015-02-11 16:41:20]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2015-02-11 16:41:21]

Presubmit check for 910913003-20001 failed and returned exit status 1.

Running presubmit commit checks ...
Checking out rietveld...
Running save-description-on-failure.sh
Running push-basic.sh
Running upstream.sh
Running submit-from-new-dir.sh
Running abandon.sh
Running submodule-merge-test.sh
Running upload-local-tracking-branch.sh
Running hooks.sh
Running post-dcommit-hook-test.sh
Running upload-stale.sh
Running patch.sh
Running basic.sh

** Presubmit ERRORS **
tests/scm_unittest.py (1.11s) failed
.....EEEEE.........
======================================================================
ERROR: testGetGetSha1ForSvnRev (__main__.RealGitSvnTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/scm_unittest.py", line 195, in setUp
    self.enabled = self.FAKE_REPOS.set_up_git() and self.FAKE_REPOS.set_up_svn()
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/testing_support/fake_repos.py",
line 383, in set_up_git
    self.check_port_is_free(self.git_port)
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/testing_support/fake_repos.py",
line 427, in check_port_is_free
    assert False, '%d shouldn\'t be bound' % port
AssertionError: 20000 shouldn't be bound

======================================================================
ERROR: testGetGitSvnHeadRev (__main__.RealGitSvnTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/scm_unittest.py", line 195, in setUp
    self.enabled = self.FAKE_REPOS.set_up_git() and self.FAKE_REPOS.set_up_svn()
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/testing_support/fake_repos.py",
line 362, in set_up_git
    assert self.git_pid_file == None
AssertionError

======================================================================
ERROR: testIsGitSvn (__main__.RealGitSvnTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/scm_unittest.py", line 195, in setUp
    self.enabled = self.FAKE_REPOS.set_up_git() and self.FAKE_REPOS.set_up_svn()
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/testing_support/fake_repos.py",
line 362, in set_up_git
    assert self.git_pid_file == None
AssertionError

======================================================================
ERROR: testParseGitSvnSha1 (__main__.RealGitSvnTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/scm_unittest.py", line 195, in setUp
    self.enabled = self.FAKE_REPOS.set_up_git() and self.FAKE_REPOS.set_up_svn()
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/testing_support/fake_repos.py",
line 362, in set_up_git
    assert self.git_pid_file == None
AssertionError

======================================================================
ERROR: testIsValidRevision (__main__.RealGitTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/scm_unittest.py", line 176, in setUp
    self.enabled = self.FAKE_REPOS.set_up_git()
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/testing_support/fake_repos.py",
line 362, in set_up_git
    assert self.git_pid_file == None
AssertionError

----------------------------------------------------------------------
Ran 19 tests in 0.871s

FAILED (errors=5)


Presubmit checks took 185.8s to calculate.

[Primiano Tucci (use gerrit), 2015-02-17 10:58:18]

The CQ bit was checked by primiano@chromium.org

[commit-bot: I haz the power, 2015-02-17 10:58:35]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/910913003/20001

[commit-bot: I haz the power, 2015-02-17 11:01:41]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2015-02-17 11:01:43]

Presubmit check for 910913003-20001 failed and returned exit status 1.

Running presubmit commit checks ...
Checking out rietveld...
Running save-description-on-failure.sh
Running push-basic.sh
Running upstream.sh
Running submit-from-new-dir.sh
Running abandon.sh
Running submodule-merge-test.sh
Running upload-local-tracking-branch.sh
Running hooks.sh
Running post-dcommit-hook-test.sh
Running upload-stale.sh
Running patch.sh
Running basic.sh

** Presubmit ERRORS **
tests/scm_unittest.py (1.10s) failed
.....EEEEE.........
======================================================================
ERROR: testGetGetSha1ForSvnRev (__main__.RealGitSvnTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/scm_unittest.py", line 195, in setUp
    self.enabled = self.FAKE_REPOS.set_up_git() and self.FAKE_REPOS.set_up_svn()
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/testing_support/fake_repos.py",
line 383, in set_up_git
    self.check_port_is_free(self.git_port)
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/testing_support/fake_repos.py",
line 427, in check_port_is_free
    assert False, '%d shouldn\'t be bound' % port
AssertionError: 20000 shouldn't be bound

======================================================================
ERROR: testGetGitSvnHeadRev (__main__.RealGitSvnTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/scm_unittest.py", line 195, in setUp
    self.enabled = self.FAKE_REPOS.set_up_git() and self.FAKE_REPOS.set_up_svn()
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/testing_support/fake_repos.py",
line 362, in set_up_git
    assert self.git_pid_file == None
AssertionError

======================================================================
ERROR: testIsGitSvn (__main__.RealGitSvnTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/scm_unittest.py", line 195, in setUp
    self.enabled = self.FAKE_REPOS.set_up_git() and self.FAKE_REPOS.set_up_svn()
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/testing_support/fake_repos.py",
line 362, in set_up_git
    assert self.git_pid_file == None
AssertionError

======================================================================
ERROR: testParseGitSvnSha1 (__main__.RealGitSvnTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/scm_unittest.py", line 195, in setUp
    self.enabled = self.FAKE_REPOS.set_up_git() and self.FAKE_REPOS.set_up_svn()
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/testing_support/fake_repos.py",
line 362, in set_up_git
    assert self.git_pid_file == None
AssertionError

======================================================================
ERROR: testIsValidRevision (__main__.RealGitTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/scm_unittest.py", line 176, in setUp
    self.enabled = self.FAKE_REPOS.set_up_git()
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/testing_support/fake_repos.py",
line 362, in set_up_git
    assert self.git_pid_file == None
AssertionError

----------------------------------------------------------------------
Ran 19 tests in 0.846s

FAILED (errors=5)


push-basic.sh failed
Command
/b/infra_internal/commit_queue/workdir/tools/depot_tools/tests/push-basic.sh
returned non-zero exit status 1 in
/b/infra_internal/commit_queue/workdir/tools/depot_tools/tests
Setting up test upstream git repo...
Setting up test git repo...
TESTING: git-cl upload wants a server
TESTING: git-cl status has no issue
TESTING: upload succeeds (needs a server running on localhost)
TESTING: git-cl status now knows the issue
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--    
0
100    77    0     0  100    77      0   1532 --:--:-- --:--:-- --:--:--  1540
TESTING: Base URL contains branch name
TESTING: git-cl land ok
Using 50% similarity for rename/copy detection. Override with --similarity.
Loaded authentication cookies from /home/chrome-bot/.codereview_upload_cookies
Running presubmit commit checks ...

Presubmit checks passed.
Description:
foo-quux

Review URL: http://localhost:10000/5629499534213120
Closing issue (you may be prompted for your codereview password)...
Traceback (most recent call last):
  File "/b/infra_internal/commit_queue/workdir/tools/depot_tools/git_cl.py",
line 3053, in <module>
    sys.exit(main(sys.argv[1:]))
  File "/b/infra_internal/commit_queue/workdir/tools/depot_tools/git_cl.py",
line 3039, in main
    return dispatcher.execute(OptionParser(), argv)
  File "/b/infra_internal/commit_queue/workdir/tools/depot_tools/subcommand.py",
line 245, in execute
    return command(parser, args[1:])
  File "/b/infra_internal/commit_queue/workdir/tools/depot_tools/git_cl.py",
line 2450, in CMDland
    return SendUpstream(parser, args, 'land')
  File "/b/infra_internal/commit_queue/workdir/tools/depot_tools/git_cl.py",
line 2309, in SendUpstream
    cl.CloseIssue()
  File "/b/infra_internal/commit_queue/workdir/tools/depot_tools/git_cl.py",
line 955, in CloseIssue
    return self.RpcServer().close_issue(self.GetIssue())
  File "/b/infra_internal/commit_queue/workdir/tools/depot_tools/rietveld.py",
line 86, in close_issue
    self.post("/%d/close" % issue, [('xsrf_token', self.xsrf_token())])
  File "/b/infra_internal/commit_queue/workdir/tools/depot_tools/rietveld.py",
line 394, in post
    return self._send(request_path, payload=body, content_type=ctype, **kwargs)
  File "/b/infra_internal/commit_queue/workdir/tools/depot_tools/rietveld.py",
line 420, in _send
    result = self.rpc_server.Send(request_path, **kwargs)
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/third_party/upload.py",
line 460, in Send
    f = self.opener.open(req, timeout=70)
  File "/usr/lib/python2.7/urllib2.py", line 410, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 523, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 448, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 531, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 403: Forbidden
FAILURE: git-cl land ok


Presubmit checks took 179.8s to calculate.

[Primiano Tucci (use gerrit), 2015-02-17 11:09:38]

New patchsets have been uploaded after l-g-t-m from maruel@chromium.org

[Primiano Tucci (use gerrit), 2015-02-17 11:11:23]

Sigh, I wish I could understand why presubmit checks (or running the individual
test) runs reliably fine on my machine and fails on the CQ.
I am reverting the deflake attempt to populateGit on fake_repos.py.
Apologies for the spam.

[Primiano Tucci (use gerrit), 2015-02-17 11:11:28]

The CQ bit was checked by primiano@chromium.org

[commit-bot: I haz the power, 2015-02-17 11:12:34]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/910913003/40001

[commit-bot: I haz the power, 2015-02-17 11:15:43]

Committed patchset #3 (id:40001) as
http://src.chromium.org/viewvc/chrome?view=rev&revision=294082