On linux, fetch should install the build deps before running gyp

On linux, fetch should install the build deps before running gyp

BUG=232291
R=dpranke@chromium.org

[Dirk Pranke, 2013-04-17 16:27:05]

The first version of fetch actually did almost exactly this, but the infra guys
have concerns around combining the checkout and installing build deps.

I'm cc'ing Aaron on this so he can weigh in on how far away we are from having
their preferred solution for this.

[agable, 2013-04-17 17:22:38]

Ok, so apparently this is a thing that people want (see also
https://codereview.chromium.org/14087013/).

However, this is a thing that I would push pretty strongly against, especially
hardcoded like this.

Fetch is meant to be a general tool. We only have two recipes for it right now
-- Blink and Chromium -- but many more are coming: Android, iOS, toolkit,
quickoffice, build tools, etc... pretty much one for every project around here.
Hardcoding something like this that is very specific to a single one of those
projects into fetch is a bad idea. It shouldn't have any knowledge of the
project it is checking out.

It is arguable whether instructions like this should be encoded in the recipe.
Since the recipe knows things about the project (i.e. what kind of checkout it
is, and what URLs it lives at), it could conceivably encode extra instructions
like "make sure you run this script in the middle." But again, that's a
direction I don't think we want to go. Fetch is for fetching -- not installing
or bootstrapping or building. We plan to have other tools to perform those other
actions, configured using the same recipes as fetch uses.

So yeah. I don't think this change should go in, since it goes against the
project-agnostic nature of fetch. The CL I linked above accomplishes nearly the
same thing, though I have smaller concerns there (read my comments on that CL if
you're interested).

I'd be more than happy to talk about other ways to make sure build deps are
installed. Thanks for putting up with my wall of text :)

[jochen (gone - plz use gerrit), 2013-04-17 18:07:02]

my main concern is that if gclient sync fails at the end, you possibly already
waited for hours, downloaded gigabytes of data, and fetch just aborts.

That's however a general problem with fetch. e.g. git svn fetch fails a lot for
me, because the svn connection times out while git-svn matches the revisions
against hashes, and when I fetch blink, fetch.py aborts in the middle...

[agable, 2013-04-17 18:19:56]

Agreed, fetch needs better error handling. I'd even like it to have retry logic
for the idempotent steps.

Robbie and I now think that the gclient portion should just run sync with
--nohooks anyway, in keeping with the "fetch is for fetching" philosophy since
the hooks do all sorts of other stuff.

[Dirk Pranke, 2013-04-17 19:14:51]

Chase and I had a longish conversation about this last week, and I recall us
more-or-less agreeing on something at the end, but of course now I don't
remember what it was that we agreed to.

Basically, we want the bots to run something fetch-like, and we don't want the
bots to run install-build-deps, which means that the bots have a need for
something different than what devs might want.

That said, my initial goal for "gclient init" was as much as possible to get a
developer configured and ready to develop in a single command. If there had been
a way to include fetching depot_tools as part of that, I would've done that. I
definitely think my use case includes running install-build-deps and/or
toolchain as part of the single command. It also involves running the hooks; to
me it would be very weird to run a sync but not run the hooks by default.

If that means that my command isn't "fetch" any more, then I think we need to
refactor things so that we can share the pieces we need to share. Maybe we need
an "init" that is fetch + install-build-deps + sync + run-hooks ?

I would also rather converge on an answer to this sooner rather than later. This
(falling over w/o the install-build-deps on linux) is a commonly recurring
problem, mostly because we tell people to run fetch on a new config and don't
give them a documented alternative. I want to solve this problem ASAP, not wait
for us to come up with the ideal decomposition of scripts before we solve it.
It's not that hard to change things even after they roll out, and I'm not too
worried about breaking fetch backward compatibility.

[janx, 2013-04-17 21:59:17]

I also think that fetch should run `gclient` with `--nohooks` by default (see
#14087013), but there could be a higher-level tool for lazy developers that
fetches, installs build dependencies, and then runs the hooks in one go.

Dirk I think that my `cr clone` is similar to your `gclient init`, i.e. it sets
everything up from scratch on a new machine so that developers can be good to go
in one command. It even fetches depot_tools if necessary, adds them to the PATH
permanently, configures ninja etc, but it is linux-only because it's bash and
assumes a ~/.bashrc file: https://github.com/jankeromnes/cr/blob/master/cr (look
for the do_clone function) Are there plans to do similar tooling in gclient?