Re-land: Added virtualenv for depot_tools

Re-land: Added virtualenv for depot_tools

Original CL: https://chromiumcodereview.appspot.com/1200843003/

R=pgervais@chromium.org
BUG=503067
TEST=tested on Mac, Windows and Linux bots

Committed: https://src.chromium.org/viewvc/chrome?view=rev&revision=295842

[Sergiy Byelozyorov, 2015-06-24 13:35:16]

Patchset #4 (id:60001) has been deleted

[Sergiy Byelozyorov, 2015-06-24 13:35:30]

PTAL

[Sergiy Byelozyorov, 2015-06-24 15:15:20]

Patchset #4 (id:80001) has been deleted

[Sergiy Byelozyorov, 2015-06-24 15:17:59]

I've added another patchset that suppresses all output from the bootstrap.py
script to avoid breaking tools that parse output from certain gclient commands,
e.g. gclient revinfo. The deps will therefore be updated silently in the
background without the users's knowledge.

Once this CL lands (and doesn't get reverted), I'll backport all changes to
bootstrap.py in infra.

[Sergiy Byelozyorov, 2015-06-24 15:19:44]

Patchset #4 (id:100001) has been deleted

[pgervais, 2015-06-24 16:31:33]

https://chromiumcodereview.appspot.com/1205873002/diff/90009/update_depot_tools
File update_depot_tools (right):

https://chromiumcodereview.appspot.com/1205873002/diff/90009/update_depot_too...
update_depot_tools:159:
VIRTUALENV_REVISION="4243b272823228dde5d18a7400c404ce52fb4cea"
Virtualenv is vendored in infra.git/bootstrap/virtualenv, we should do the same
here (put it in third_party instead if you want).

We really don't want to depend on yet another network connection on all slaves.

[Sergiy Byelozyorov, 2015-06-25 14:42:17]

https://codereview.chromium.org/1205873002/diff/90009/update_depot_tools
File update_depot_tools (right):

https://codereview.chromium.org/1205873002/diff/90009/update_depot_tools#newc...
update_depot_tools:159:
VIRTUALENV_REVISION="4243b272823228dde5d18a7400c404ce52fb4cea"
On 2015/06/24 16:31:33, pgervais wrote:
> Virtualenv is vendored in infra.git/bootstrap/virtualenv, we should do the
same
> here (put it in third_party instead if you want).
> 
> We really don't want to depend on yet another network connection on all
slaves.

It's checked out with DEPS. I thought 'vendored' means 'copied'. We can't use
DEPS here since gclient is not set up yet. Also there are other commands using
network here... why adding another one makes it worse?

[pgervais, 2015-06-25 17:12:02]

https://chromiumcodereview.appspot.com/1205873002/diff/90009/update_depot_tools
File update_depot_tools (right):

https://chromiumcodereview.appspot.com/1205873002/diff/90009/update_depot_too...
update_depot_tools:159:
VIRTUALENV_REVISION="4243b272823228dde5d18a7400c404ce52fb4cea"
On 2015/06/25 14:42:17, Sergiy Byelozyorov wrote:
> On 2015/06/24 16:31:33, pgervais wrote:
> > Virtualenv is vendored in infra.git/bootstrap/virtualenv, we should do the
> same
> > here (put it in third_party instead if you want).
> > 
> > We really don't want to depend on yet another network connection on all
> slaves.
> 
> It's checked out with DEPS. I thought 'vendored' means 'copied'. We can't use
> DEPS here since gclient is not set up yet. Also there are other commands using
> network here... why adding another one makes it worse?

Yes 'vendored' means copied. The idea behind Glyco is to vendor everything (in
depot_tools) anyway.

I ack the fact that this file uses the network already, but it's also a question
of atomicity. If git fetch fails, no big deal. If git fetch succeeds but
downloading virtualenv fails, we end up in  a weird situation. BTW with the
current code we have no way to update virtualenv. We'll need some more logic to
fix it, which can also break, etc. The simpler the better.

[Sergiy Byelozyorov, 2015-06-25 17:34:51]

https://chromiumcodereview.appspot.com/1205873002/diff/90009/update_depot_tools
File update_depot_tools (right):

https://chromiumcodereview.appspot.com/1205873002/diff/90009/update_depot_too...
update_depot_tools:159:
VIRTUALENV_REVISION="4243b272823228dde5d18a7400c404ce52fb4cea"
On 2015/06/25 17:12:02, pgervais wrote:
> On 2015/06/25 14:42:17, Sergiy Byelozyorov wrote:
> > On 2015/06/24 16:31:33, pgervais wrote:
> > > Virtualenv is vendored in infra.git/bootstrap/virtualenv, we should do the
> > same
> > > here (put it in third_party instead if you want).
> > > 
> > > We really don't want to depend on yet another network connection on all
> > slaves.
> > 
> > It's checked out with DEPS. I thought 'vendored' means 'copied'. We can't
use
> > DEPS here since gclient is not set up yet. Also there are other commands
using
> > network here... why adding another one makes it worse?
> 
> Yes 'vendored' means copied. The idea behind Glyco is to vendor everything (in
> depot_tools) anyway.
> 
> I ack the fact that this file uses the network already, but it's also a
question
> of atomicity. If git fetch fails, no big deal. If git fetch succeeds but
> downloading virtualenv fails, we end up in  a weird situation. BTW with the
> current code we have no way to update virtualenv. We'll need some more logic
to
> fix it, which can also break, etc. The simpler the better.

Done.

[pgervais, 2015-06-25 17:54:10]

On 2015/06/25 17:34:51, Sergiy Byelozyorov wrote:
>
https://chromiumcodereview.appspot.com/1205873002/diff/90009/update_depot_tools
> File update_depot_tools (right):
> 
>
https://chromiumcodereview.appspot.com/1205873002/diff/90009/update_depot_too...
> update_depot_tools:159:
> VIRTUALENV_REVISION="4243b272823228dde5d18a7400c404ce52fb4cea"
> On 2015/06/25 17:12:02, pgervais wrote:
> > On 2015/06/25 14:42:17, Sergiy Byelozyorov wrote:
> > > On 2015/06/24 16:31:33, pgervais wrote:
> > > > Virtualenv is vendored in infra.git/bootstrap/virtualenv, we should do
the
> > > same
> > > > here (put it in third_party instead if you want).
> > > > 
> > > > We really don't want to depend on yet another network connection on all
> > > slaves.
> > > 
> > > It's checked out with DEPS. I thought 'vendored' means 'copied'. We can't
> use
> > > DEPS here since gclient is not set up yet. Also there are other commands
> using
> > > network here... why adding another one makes it worse?
> > 
> > Yes 'vendored' means copied. The idea behind Glyco is to vendor everything
(in
> > depot_tools) anyway.
> > 
> > I ack the fact that this file uses the network already, but it's also a
> question
> > of atomicity. If git fetch fails, no big deal. If git fetch succeeds but
> > downloading virtualenv fails, we end up in  a weird situation. BTW with the
> > current code we have no way to update virtualenv. We'll need some more logic
> to
> > fix it, which can also break, etc. The simpler the better.
> 
> Done.

lgtm then

[Sergiy Byelozyorov, 2015-06-25 17:58:57]

The CQ bit was checked by sergiyb@chromium.org

[commit-bot: I haz the power, 2015-06-25 17:59:24]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1205873002/130001

[Sergiy Byelozyorov, 2015-06-25 18:01:06]

The CQ bit was unchecked by sergiyb@chromium.org

[Sergiy Byelozyorov, 2015-06-25 18:05:00]

The CQ bit was checked by sergiyb@chromium.org

[Sergiy Byelozyorov, 2015-06-25 18:05:00]

The patchset sent to the CQ was uploaded after l-g-t-m from
pgervais@chromium.org
Link to the patchset: https://codereview.chromium.org/1205873002/#ps150001
(title: "Restore cd commands")

[commit-bot: I haz the power, 2015-06-25 18:05:23]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1205873002/150001

[Sergiy Byelozyorov, 2015-06-25 18:06:31]

The CQ bit was unchecked by sergiyb@chromium.org

[Sergiy Byelozyorov, 2015-06-25 18:08:55]

Committed patchset #6 (id:150001) manually as 295842 (presubmit successful).

[Sergiy Byelozyorov, 2015-11-09 15:09:38]

On 2015/06/25 18:08:55, Sergiy Byelozyorov wrote:
> Committed patchset #6 (id:150001) manually as 295842 (presubmit successful).

Revert for this is https://codereview.chromium.org/1437483002/.