Make gclient ready for the Blink (DEPS to main project) transition

Make gclient ready for the Blink (DEPS to main project) transition

This CL makes gclient understand correctly whether a git project is
being moved from DEPS to an upper project and vice-versa.
The driving use case for this is the upcoming Blink merge, where
third_party/Webkit will be removed from DEPS (and .gitignore) and will 
become part of the main project.

At present state, gclient leaves the .git folder around when a project
is removed from DEPS, and that causes many problems. 

Furthermore this CL solves the performance problem of bisecting across
the merge point. The subproject's (Blink) .git/ folder is moved to a
backup location (in the main checkout root) and is restored when moving
backwards, avoiding a re-fetch when bisecting across the merge point. 

BUG=431469
Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=293329

[Primiano Tucci (use gerrit), 2014-11-21 16:39:04]

Patchset #3 (id:40001) has been deleted

[Primiano Tucci (use gerrit), 2014-11-21 16:41:07]

Patchset #3 (id:60001) has been deleted

[Primiano Tucci (use gerrit), 2014-11-21 20:50:19]

Patchset #3 (id:80001) has been deleted

[Primiano Tucci (use gerrit), 2014-11-22 02:16:45]

Patchset #4 (id:120001) has been deleted

[Primiano Tucci (use gerrit), 2014-11-22 02:23:48]

primiano@chromium.org changed reviewers:
+ iannucci@chromium.org, maruel@chromium.org

[Primiano Tucci (use gerrit), 2014-11-22 02:23:48]

Gentlemen, I need your eyes and your souls for this CL.
The title should explain what it is about.

You can find more accurate details behind the choice and the potential
side-effects of this cl in the linked bug (crbug.com/431469). 

If you don't feel you are the right reviewers feel free to reroute to the
appropriate folks.

[jochen (gone - plz use gerrit), 2014-11-25 16:53:07]

jochen@chromium.org changed reviewers:
+ jochen@chromium.org

[jochen (gone - plz use gerrit), 2014-11-25 16:53:08]

lgtm from my point of view

Ian, M-A, any objections?

[M-A Ruel, 2014-11-25 17:08:17]

lgtm

[M-A Ruel, 2014-11-25 17:08:32]

https://codereview.chromium.org/743083002/diff/140001/gclient.py
File gclient.py (right):

https://codereview.chromium.org/743083002/diff/140001/gclient.py#newcode1597
gclient.py:1597: 'level checkout. The git folder containing all the local '
fix 80 cols here

[Primiano Tucci (use gerrit), 2014-11-25 17:22:32]

On 2014/11/25 17:08:32, M-A Ruel wrote:
> https://codereview.chromium.org/743083002/diff/140001/gclient.py
> File gclient.py (right):
> 
> https://codereview.chromium.org/743083002/diff/140001/gclient.py#newcode1597
> gclient.py:1597: 'level checkout. The git folder containing all the local '
> fix 80 cols here

Just to be clear: I am not going to commit this anytime soon. Besides the nits,
I want to see this working and not blowing up everything in the next days on my
Linux Mac and Windows boxed before ticking the damn box.

However, if anybody in the meanwhile can spot something anything even marginally
suspicious please shout!
I definitely don't want to become popular as the guy who blew up ~1000
commiters' repos in one go :)

[Primiano Tucci (use gerrit), 2014-12-09 16:26:46]

I did some testing in the last weeks using that locally on both Windows and
Linux (Nico tried it on mac) and it seems ready to go.

[Primiano Tucci (use gerrit), 2014-12-09 16:26:51]

The CQ bit was checked by primiano@chromium.org

[commit-bot: I haz the power, 2014-12-09 16:27:32]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/743083002/160001

[commit-bot: I haz the power, 2014-12-09 16:29:30]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2014-12-09 16:29:31]

Presubmit check for 743083002-160001 failed and returned exit status 1.

Running presubmit commit checks ...
Checking out rietveld...
Running save-description-on-failure.sh
Running push-basic.sh
Running upstream.sh
Running submit-from-new-dir.sh
Running abandon.sh
Running submodule-merge-test.sh
Running upload-local-tracking-branch.sh
Running hooks.sh
Running post-dcommit-hook-test.sh
Running upload-stale.sh
Running patch.sh
Running basic.sh

** Presubmit ERRORS **
Pylint (111 files) (67.82s) failed
/b/depot_tools/third_party/logilab/astroid/modutils.py:520: UserWarning: Module
six was already imported from /b/depot_tools/third_party/six/__init__.py, but
/usr/local/lib/python2.7/dist-packages is being added to sys.path
  import pkg_resources
Problem importing module .svn: No module named .svn
Problem importing module .svn: No module named .svn
************* Module gclient
W:1574,13: Using possibly undefined loop variable 'scm_class'
(undefined-loop-variable)
W:1577,15: Using possibly undefined loop variable 'scm_class'
(undefined-loop-variable)
W:1578,29: Using possibly undefined loop variable 'scm_class'
(undefined-loop-variable)
W:1581,17: Using possibly undefined loop variable 'scm_class'
(undefined-loop-variable)
W:1594,16: Using possibly undefined loop variable 'scm_class'
(undefined-loop-variable)
************* Module testing_support.fake_repos
W:831, 0: Method 'populateSvn' is abstract in class 'FakeReposBase' but is not
overridden (abstract-method)


tests/scm_unittest.py (22.74s) failed
..['CleanupDir', 'IsDirectoryVersioned']
F.................
======================================================================
FAIL: testMembersChanged (__main__.GitWrapperTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/third_party/pymox/mox.py",
line 1618, in new_method
    func(self, *args, **kwargs)
  File "tests/scm_unittest.py", line 105, in testMembersChanged
    self.compareMembers(scm.GIT, members)
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/testing_support/super_mox.py",
line 78, in compareMembers
    self.assertEqual(actual_members, expected_members)
AssertionError: Lists differ: ['ApplyEnvVars', 'AssertVersio... !=
['ApplyEnvVars', 'AssertVersio...

First differing element 4:
CleanupDir
FetchUpstreamTuple

First list contains 2 additional elements.
First extra element 27:
ShortBranchName

  ['ApplyEnvVars',
   'AssertVersion',
   'Capture',
   'CaptureStatus',
-  'CleanupDir',
   'FetchUpstreamTuple',
   'GenerateDiff',
   'GetBlessedSha1ForSvnRev',
   'GetBranch',
   'GetBranchRef',
   'GetCheckoutRoot',
   'GetDifferentFiles',
   'GetEmail',
   'GetGitDir',
   'GetGitSvnHeadRev',
   'GetPatchName',
   'GetSVNBranch',
   'GetSha1ForSvnRev',
   'GetUpstreamBranch',
-  'IsDirectoryVersioned',
   'IsGitSvn',
   'IsInsideWorkTree',
   'IsValidRevision',
   'IsWorkTreeDirty',
   'MatchSvnGlob',
   'ParseGitSvnSha1',
   'RefToRemoteRef',
   'ShortBranchName',
   'current_version']

----------------------------------------------------------------------
Ran 20 tests in 22.451s

FAILED (failures=1)


tests/gclient_scm_test.py (5.91s) failed
.................FF................................
======================================================================
FAIL: testUpdateConflict (__main__.ManagedGitWrapperTestCaseMox)
----------------------------------------------------------------------
Traceback (most recent call last):
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/third_party/pymox/mox.py",
line 1618, in new_method
    func(self, *args, **kwargs)
  File "tests/gclient_scm_test.py", line 1415, in testUpdateConflict
    scm.update(options, None, [])
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/gclient_scm.py", line
401, in update
    if os.path.exists(backup_dir) and not os.path.exists(target_dir):
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/third_party/pymox/mox.py",
line 771, in __call__
    expected_method = self._VerifyMethodCall()
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/third_party/pymox/mox.py",
line 827, in _VerifyMethodCall
    raise UnexpectedMethodCallError(self, expected)
UnexpectedMethodCallError: Unexpected method call.  unexpected:-  expected:+
- Stub for <function exists at 0x7f9c1aa4ced8>.__call__('/tmp/old_fake.git') ->
None
?                                                             ----    ----     
^^^

+ Stub for <function exists at 0x7f9c1aa4ced8>.__call__('/tmp/fake') -> True
?                                                                       ^^^


======================================================================
FAIL: testUpdateNoDotGit (__main__.ManagedGitWrapperTestCaseMox)
----------------------------------------------------------------------
Traceback (most recent call last):
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/third_party/pymox/mox.py",
line 1618, in new_method
    func(self, *args, **kwargs)
  File "tests/gclient_scm_test.py", line 1379, in testUpdateNoDotGit
    scm.update(options, None, [])
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/gclient_scm.py", line
401, in update
    if os.path.exists(backup_dir) and not os.path.exists(target_dir):
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/third_party/pymox/mox.py",
line 771, in __call__
    expected_method = self._VerifyMethodCall()
  File
"/b/infra_internal/commit_queue/workdir/tools/depot_tools/third_party/pymox/mox.py",
line 827, in _VerifyMethodCall
    raise UnexpectedMethodCallError(self, expected)
UnexpectedMethodCallError: Unexpected method call.  unexpected:-  expected:+
- Stub for <function exists at 0x7f9c1aa4ced8>.__call__('/tmp/old_fake.git') ->
None
?                                                             ----    ----     
^^^

+ Stub for <function exists at 0x7f9c1aa4ced8>.__call__('/tmp/fake') -> True
?                                                                       ^^^


----------------------------------------------------------------------
Ran 51 tests in 5.742s

FAILED (failures=2)


Presubmit checks took 111.7s to calculate.

[Primiano Tucci (use gerrit), 2014-12-10 15:51:02]

The CQ bit was checked by primiano@chromium.org

[commit-bot: I haz the power, 2014-12-10 15:51:32]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/743083002/180001

[commit-bot: I haz the power, 2014-12-10 15:53:44]

Committed patchset #6 (id:180001) as
http://src.chromium.org/viewvc/chrome?view=rev&revision=293329

[M-A Ruel, 2014-12-11 16:31:25]

On 2014/12/10 15:53:44, I haz the power (commit-bot) wrote:
> Committed patchset #6 (id:180001) as
> http://src.chromium.org/viewvc/chrome?view=rev&revision=293329

Test BlinkDEPSTransitionSmokeTest is flaky;
https://codereview.chromium.org/753543006/.

[Primiano Tucci (use gerrit), 2014-12-11 16:35:10]

> Test BlinkDEPSTransitionSmokeTest is flaky;
> https://codereview.chromium.org/753543006/.

Uhm, sigh. Looking...

[Primiano Tucci (use gerrit), 2014-12-11 17:22:36]

A revert of this CL (patchset #6 id:180001) has been created in
https://codereview.chromium.org/796053002/ by primiano@chromium.org.

The reason for reverting is: The test intoruced is flake (see
crrev.com/753543006/).
Need to look into that with more patience as it reproduces only on precise.
.