android: Make buildbot able to use a custom manifest.

android: Make buildbot able to use a custom manifest.

In order to support rolling to a new version of AOSP to run gyp tests
(to pick up android build system changes), introduce logic in the
buildbot script to compare the current manifest being used with the one
in the gyp tree and resync/compile android if it changes. Check in an
initial version of this manifest which is pinned to a recent revision of
AOSP's master branch.

BUG=gyp:324
R=jbudorick@chromium.org, torne@chromium.org

Committed: https://code.google.com/p/gyp/source/detail?r=1968

[Primiano Tucci (use gerrit), 2014-08-21 08:10:44]

Torne: PTAL
This ran overnight for ~50 times, apparently with no error.
Patchset 1 is identical to your crrev.com/426783004
Patchset 2 has my changes on top.

[Torne, 2014-08-21 10:44:07]

Hi Brad, John,

Can you take a look at this? This makes the adb connection much more robust,
which is required to use a more recent android (seems like adb/emulator is more
flaky there :/ )

[Torne, 2014-08-21 11:34:22]

LGTM but it'd be good to get another set of eyes since it's based on my original
patch.

[jbudorick, 2014-08-22 05:10:48]

lgtm w/ nit

https://codereview.chromium.org/493743002/diff/40001/buildbot/buildbot_run.py
File buildbot/buildbot_run.py (right):

https://codereview.chromium.org/493743002/diff/40001/buildbot/buildbot_run.py...
buildbot/buildbot_run.py:104: CallSubProcess(['git', 'config', '--global',
'user.name', 'trybot'])
Ha, I think I wound up trying to upload a CL as "trybot" because of this.

https://codereview.chromium.org/493743002/diff/40001/buildbot/buildbot_run.py...
buildbot/buildbot_run.py:143: # If taskset is available, use it to force adbd to
run only on one core, as,
yay adb

https://codereview.chromium.org/493743002/diff/40001/buildbot/buildbot_run.py...
buildbot/buildbot_run.py:155: '%s && emulator -no-window' % _ANDROID_SETUP],
I think I asked this on torne's review, but is this now in the path? Was this
because emulator became a prebuilt?

https://codereview.chromium.org/493743002/diff/40001/test/lib/TestGyp.py
File test/lib/TestGyp.py (right):

https://codereview.chromium.org/493743002/diff/40001/test/lib/TestGyp.py#newc...
test/lib/TestGyp.py:557: def _call_adb(self, command, timeout=15, retry=3):
I should've done this to begin with. Thanks for doing it.

nit: "retry" implies that it's the number of times the command will be retried
on failure (i.e., what our scripts do in the main repo), but it's actually just
the number of tries here.

https://codereview.chromium.org/493743002/diff/40001/test/lib/TestGyp.py#newc...
test/lib/TestGyp.py:595: else:
This only retries on timeouts, not failures. Is that what we want to do?

[Primiano Tucci (use gerrit), 2014-08-22 11:18:16]

https://codereview.chromium.org/493743002/diff/40001/buildbot/buildbot_run.py
File buildbot/buildbot_run.py (right):

https://codereview.chromium.org/493743002/diff/40001/buildbot/buildbot_run.py...
buildbot/buildbot_run.py:104: CallSubProcess(['git', 'config', '--global',
'user.name', 'trybot'])
On 2014/08/22 05:10:48, jbudorick wrote:
> Ha, I think I wound up trying to upload a CL as "trybot" because of this.

Actually turns out that we don't need this at all using just a small trick (see
the change in  CallSubProcess). If either stdin or stdout are not tty, repo will
not bother checking for git config.

https://codereview.chromium.org/493743002/diff/40001/buildbot/buildbot_run.py...
buildbot/buildbot_run.py:143: # If taskset is available, use it to force adbd to
run only on one core, as,
On 2014/08/22 05:10:48, jbudorick wrote:
> yay adb

Amazing, isn't it?

https://codereview.chromium.org/493743002/diff/40001/buildbot/buildbot_run.py...
buildbot/buildbot_run.py:155: '%s && emulator -no-window' % _ANDROID_SETUP],
On 2014/08/22 05:10:48, jbudorick wrote:
> I think I asked this on torne's review, but is this now in the path? Was this
> because emulator became a prebuilt?

Correct. envsetup / lunch make it so that the correct (w.r.t. lunch
configuration) emulator prebuilt is in the PATH.

[Torne, 2014-08-22 11:30:30]

LGTM. I'll run this through the gyp-android tryserver and then land it unless
Brad has any objection.

[Torne, 2014-08-22 16:11:57]

Woo, the tryserver is happy. I'm going to land this to unblock making progress
on gyp-android test coverage; if anyone has comments/objections we'll address
them in a followup CL.

[Torne, 2014-08-22 16:12:31]

Committed patchset #4 manually as 1968 (presubmit successful).

[bradn, 2014-08-22 16:13:32]

lgtm

https://codereview.chromium.org/493743002/diff/80001/buildbot/buildbot_run.py
File buildbot/buildbot_run.py (right):

https://codereview.chromium.org/493743002/diff/80001/buildbot/buildbot_run.py...
buildbot/buildbot_run.py:132: CallSubProcess(['/bin/bash', '-c',
Rather than going thru /bin/bash you can add the argument shell=True

https://codereview.chromium.org/493743002/diff/80001/buildbot/buildbot_run.py...
buildbot/buildbot_run.py:142: CallSubProcess(['/bin/bash', '-c',
same

https://codereview.chromium.org/493743002/diff/80001/buildbot/buildbot_run.py...
buildbot/buildbot_run.py:147: ['/bin/bash', '-c',
same

https://codereview.chromium.org/493743002/diff/80001/test/lib/TestGyp.py
File test/lib/TestGyp.py (right):

https://codereview.chromium.org/493743002/diff/80001/test/lib/TestGyp.py#newc...
test/lib/TestGyp.py:583: deadline = time.time() + timeout
I assume there's no need for exponential backoff?

[bradn, 2014-08-22 16:14:01]

Oops, fired too late.
Oh well, nothing worth another CL.