Add gclient hook to download SDK extras on bots.

build/download_sdk_extras.py

+#!/usr/bin/env python
+# Copyright (c) 2014 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+"""Script to download sdk/extras packages on the bots from google storage."""
+
+import os
+import subprocess
+import sys
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'android'))
+from pylib import constants
+
+GSUTIL_PATH = os.path.join(os.path.dirname(__file__), os.pardir, os.pardir,
+    os.pardir, os.pardir, os.pardir, 'third_party', 'gsutil', 'gsutil')
+SDK_EXTRAS_BUCKET = 'gs://chrome-sdk-extras'

[nyquist, 2014/11/25 20:08:26]

Does this already have the exact same content (except from samples and docs) as
https://chromium.googlesource.com/android_tools.git/+/master/sdk/extras/ ?

[navabi, 2014/11/25 20:40:49]

Yes.

+SDK_EXTRAS_PATH = os.path.join(constants.ANDROID_SDK_ROOT, 'extras')

[nyquist, 2014/11/25 20:08:26]

It feels like there should be a .gitignore in
https://chromium.googlesource.com/android_tools.git that ignores sdk/extras. Is
that in place?

[navabi, 2014/11/25 20:40:49]

Yes. It is in place, in that the head of the android_tools dir has sdk/extras/*
in the .gitignore. I still need to roll DEPS to check out the newly pushed
revision before this lands.

[nyquist, 2014/11/26 03:17:29]

So you also cleaned out that folder? Or are you saying there's a .gitignore in
there, but the repo also contains files in the location the .gitignore refers
to?

+
+
+def GetCmdOutputAndStatus(cmd_lst):
+  process = subprocess.Popen(cmd_lst, stdout=subprocess.PIPE)
+  stdout, _ = process.communicate()
+  return stdout, process.returncode
+
+def is_buildbot_checkout():

[nyquist, 2014/11/25 20:08:26]

this is a little bit odd to me, but I don't have a good suggestion.
Does all buildbots set BUILDBOT_BUILDERNAME by the way?
Maybe checking os.environ.get('BUILDBOT_BUILDERNAME', '') or something?

It might be fine as is though.

[navabi, 2014/11/25 20:40:49]

I'll consider it. But even if we checked some other way, we still want to check
if these paths exist. And then at that point, checking for anything else would
not be necessary, except maybe it would be more clear we are checking if we are
on a bot ? Even then, a comment would do.

+  if not os.path.exists(GSUTIL_PATH) or not os.path.exists(SDK_EXTRAS_PATH):
+    return False
+  stdout, rc = GetCmdOutputAndStatus([GSUTIL_PATH, 'ls', SDK_EXTRAS_BUCKET])
+  # If successfully read bucket, then this must be a bot with permissions
+  return not rc
+
+def main(args):
+  if is_buildbot_checkout():
+    success = True
+    for arg in args[1:]:
+      package_bucket = '%s/%s' % (SDK_EXTRAS_BUCKET, arg)
+      cmd_lst = [GSUTIL_PATH, '-m', 'rsync', '-r', package_bucket,

[nyquist, 2014/11/25 20:08:26]

would you want '-d' here to delete the files that do not exist anymore on the
server?

[navabi, 2014/11/25 20:40:49]

No. That would remove other packages the way we currently have it. i.e. if
locally you have:
sdk/extras/google/package1
sdk/extras/google/package2

And you do an rsync -d of a bucket that looks like this:
sdk/extras/google/package1 right now we just do an rsync -d into sdk/packages
which would mean google/package2 would get deleted.

To use the -d, we would need to either split the name of package name
"google_google_play_services_21.0.0" to come up with the directory to download
to (i.e. google/google_play_services/), or we would need to do a sequence of
gsutil ls calls to determine that directory, which would take up more time
because gsutil calls are expensive.

I did name the package such that the name of the directory is clear just to
avoid doing a gsutil ls to find the directory name, but I'm not sure we need to
delete things that don't exist on the bot. I could be convinced otherwise
though.

+                 SDK_EXTRAS_PATH]
+      stdout, rc = GetCmdOutputAndStatus(cmd_lst)
+      success = (rc == 0) and success
+    if not success:
+      return 1
+  return 0
+
+
+if __name__ == '__main__':
+  sys.exit(main(sys.argv))