Add runhooks step to AOSP bot to pick up essential build dependencies (like gn).

Add runhooks step to AOSP bot to pick up essential build dependencies (like gn).

Needed for https://chromiumcodereview.appspot.com/62803003

R=brettw@chromium.org, mkosiba@chromium.org
BUG=

Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=236353

[brettw, 2013-11-18 23:48:52]

LGTM (note: I don't know this code)

[mkosiba (inactive), 2013-11-19 00:48:29]

this won't work unfortunately. The problem is that because we don't check out
all the third_party DEPS the regular gyp_chromium hook will fail (gyp:
/usr/local/google/code/chromium_tools/build/slave/fake_slave/build/src/third_party/webrtc/tools/tools.gyp
not found). We'd need to have a way of invoking gyp_webview instead of
gyp_chromium in runhooks.

[mkosiba (inactive), 2013-11-19 05:47:20]

Also, after having looked at the gn wiki I'm curious why would you need to add
this to the AOSP bot. The whole reason we have this bot is to generate the
Android.mk files using the custom GYP backend. Using gn on this bot would
completely defeat its purpose.

[brettw, 2013-11-19 07:32:29]

On 2013/11/19 05:47:20, mkosiba wrote:
> Also, after having looked at the gn wiki I'm curious why would you need to add
> this to the AOSP bot. The whole reason we have this bot is to generate the
> Android.mk files using the custom GYP backend. Using gn on this bot would
> completely defeat its purpose.

GN will be used to generate GYP files for the forseeable future.

[mkosiba (inactive), 2013-11-19 16:50:12]

On 2013/11/19 07:32:29, brettw wrote:
> On 2013/11/19 05:47:20, mkosiba wrote:
> > Also, after having looked at the gn wiki I'm curious why would you need to
add
> > this to the AOSP bot. The whole reason we have this bot is to generate the
> > Android.mk files using the custom GYP backend. Using gn on this bot would
> > completely defeat its purpose.
> 
> GN will be used to generate GYP files for the forseeable future.

ooh.. ok. From reading the wiki I thought it would generate ninja files
directly.

In that case the simplest fix for this particular bot is to explicitly call the
hook. We already do that for a couple of other things.

[brettw, 2013-11-19 21:24:47]

I'm OK with manually calling the hook since I'm blocked on getting this hooked
up.

However, I'm worried in general about the state of this thing. It's very
unfortunate to have to have a separate dependency system for this one build that
normal engineers don't even know about and can't easily be updated. I had to ask
several people to even figure out what was going on here.

Can this bot be made to be more like the rest of the builders? The technical
reasons behind the current state aren't clear to me.

[iannucci, 2013-11-19 21:45:20]

On 2013/11/19 21:24:47, brettw wrote:
> I'm OK with manually calling the hook since I'm blocked on getting this hooked
> up.
> 
> However, I'm worried in general about the state of this thing. It's very
> unfortunate to have to have a separate dependency system for this one build
that
> normal engineers don't even know about and can't easily be updated. I had to
ask
> several people to even figure out what was going on here.
> 
> Can this bot be made to be more like the rest of the builders? The technical
> reasons behind the current state aren't clear to me.

So, the basic issue is summarized as:
  * this bot checks out a subset of the source to avoid being slow (I think?)
  * gclient runhooks will always invoke gyp_chromium which is not what we want
here (we want gyp_webview).
  * gyp_chromium fails due to not having a 'full' checkout (and even if it did
run, it would be 2 minutes of slowness for nothing useful).

What if we had gclient's gyp_chromium hook invoke a wrapper script which e.g.
inspects the environment to determine which gyp_* script to run. Then this bot
could just set the environment variable...

(FTR: I hate environment variables since they tend to lead to
action-at-a-distance which becomes very difficult to debug over time).

[mkosiba (inactive), 2013-11-19 22:16:51]

On 2013/11/19 21:45:20, iannucci wrote:
> On 2013/11/19 21:24:47, brettw wrote:
> > I'm OK with manually calling the hook since I'm blocked on getting this
hooked
> > up.
> > 
> > However, I'm worried in general about the state of this thing. It's very
> > unfortunate to have to have a separate dependency system for this one build
> that
> > normal engineers don't even know about and can't easily be updated. I had to
> ask
> > several people to even figure out what was going on here.

Sorry about that. I think I sent out a PSA about this bot to chromium-dev. I
could
add a description somewhere but I don't know of any good location for that.

> > 
> > Can this bot be made to be more like the rest of the builders? The technical
> > reasons behind the current state aren't clear to me.

The reason for the bot is to keep Chromium engineers from breaking the webview
build. The webview build is special because:
 - we can't take all third_party deps for license reasons,
 - we have to use the Android special make-based build system.

So.. no, unfortunately the bot will have to stay special.

> 
> So, the basic issue is summarized as:
>   * this bot checks out a subset of the source to avoid being slow (I think?)

see above.

>   * gclient runhooks will always invoke gyp_chromium which is not what we want
> here (we want gyp_webview).
>   * gyp_chromium fails due to not having a 'full' checkout (and even if it did
> run, it would be 2 minutes of slowness for nothing useful).
> 
> What if we had gclient's gyp_chromium hook invoke a wrapper script which e.g.
> inspects the environment to determine which gyp_* script to run. Then this bot
> could just set the environment variable...

I personally dislike the runhooks running gyp for me (since I invoke GYP via
a set of scripts that pass in different params/env vars depending on what
I want to build in the checkout). If we're adding env variables could we just
have the gyp_chromium hook look at the environment and not do anything if
NO_GYP_ON_HOOKS is set?

> 
> (FTR: I hate environment variables since they tend to lead to
> action-at-a-distance which becomes very difficult to debug over time).

[mkosiba (inactive), 2013-11-19 23:18:20]

On 2013/11/19 22:16:51, mkosiba wrote:
 > Sorry about that. I think I sent out a PSA about this bot to chromium-dev.

I just checked and the PSA didn't go to chromium-dev. I'd prefer to stick
the description in a wiki though.

[iannucci, 2013-11-19 23:58:00]

On 2013/11/19 23:18:20, mkosiba wrote:
> On 2013/11/19 22:16:51, mkosiba wrote:
>  > Sorry about that. I think I sent out a PSA about this bot to chromium-dev.
> 
> I just checked and the PSA didn't go to chromium-dev. I'd prefer to stick
> the description in a wiki though.

CL to add env var to gyp_chromium is here:
https://codereview.chromium.org/76403005

[iannucci, 2013-11-20 22:56:08]

TALP!

[mkosiba (inactive), 2013-11-20 23:41:12]

MTGL LGTM

[commit-bot: I haz the power, 2013-11-21 00:09:16]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/iannucci@chromium.org/69173006/140001

[commit-bot: I haz the power, 2013-11-21 00:09:47]

Change committed as 236353