Sysroot: Add build_and_upload.py

build/linux/sysroot_scripts/build_and_upload.py

+#!/usr/bin/env python
+# Copyright 2016 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+"""Automates running BuildPackageLists, BuildSysroot, and
+UploadSysroot for each supported arch of each sysroot creator.
+"""
+
+import hashlib
+import json
+import multiprocessing
+import os
+import random
+import re
+import string
+import subprocess
+import sys
+
+def rand_string_32():
+  return ''.join(random.choice(string.hexdigits) for _ in range(32))
+
+def run_script(args):
+  fnull = open(os.devnull, 'w')
+  subprocess.check_call(args, stdout=fnull, stderr=fnull)
+
+def sha1sumfile(filename):
+  sha1 = hashlib.sha1()
+  with open(filename, 'rb') as f:
+    while True:
+      data = f.read(65536)
+      if not data:
+        break
+      sha1.update(data)
+  return sha1.hexdigest()
+
+def build_and_upload(script_path, distro, release, arch):
+  revision = rand_string_32()

[Tom (Use chromium acct), 2016/12/12 18:49:57]

The revision can really be any string, but maybe it should be an increasing
integer?  Something like distro_release_arch_integer?  Or just the commit hash
on ToT?

I probably don't want to leave it random because that means if you run the
script multiple times, you'd be wasting google storage space.

[Sam Clegg, 2016/12/12 19:18:00]

The revision we've been using previously has been the git SHA of the version of
chrome that was used to build in the image.

In theory it should possible to checkout the same chrome revisions and generate
the exact same tarballs.

Perhaps the best options is use `git rev-parse HEAD`.  In theory one should only
upload from the clean checkout and a revision that is on master, but I don't
think writing such as check is worth it.  Only certain people have upload access
anyway.

[Tom (Use chromium acct), 2016/12/12 21:19:34]

Done.

+  run_script([script_path, 'UpdatePackageLists%s' % arch])
+  run_script([script_path, 'BuildSysroot%s' % arch])
+  run_script([script_path, 'UploadSysroot%s' % arch, revision])
+
+  tarball = '%s_%s_%s_sysroot.tgz' % (distro, release, arch.lower())
+  tgz_path = os.path.join(os.path.dirname(os.path.realpath(__file__)),
+                          "..", "..", "..", "out", "sysroot-build",
+                          release, tarball)
+  sha1sum = sha1sumfile(tgz_path)
+  sysroot_dir = '%s_%s_%s-sysroot' % (distro, release, arch.lower())
+
+  sysroot_metadata = {
+      'Revision': revision,
+      'Tarball': tarball,
+      'Sha1Sum': sha1sum,
+      'SysrootDir': sysroot_dir
+  }
+  with open('sysroot.%s.%s' % (release, arch.lower()), 'w') as f:
+    f.write(json.dumps(sysroot_metadata, sort_keys=True, indent=4))
+    f.write('\n')
+    f.close()
+
+def get_proc_output(args):
+  fnull = open(os.devnull, 'w')
+  p = subprocess.Popen(args, stderr=fnull, stdout=subprocess.PIPE)
+  (stdout, _) = p.communicate()
+  return stdout.strip()

[Sam Clegg, 2016/12/12 19:18:00]

You probably don't want to throw away stderr do you?

How about just using `subprocess.check_output().strip()` here?

[Tom (Use chromium acct), 2016/12/12 21:19:34]

Done.

+
+def build_and_upload_all():
+  script_dir = os.path.dirname(os.path.realpath(__file__))
+  procs = []
+  for filename in os.listdir(script_dir):
+    if (not os.path.isfile(os.path.join(script_dir, filename)) or
+        not re.search(r'sysroot-creator-.+\.sh', filename)):
+      continue
+    script_path = os.path.join(script_dir, filename)
+    distro = get_proc_output([script_path, 'PrintDistro'])
+    release = get_proc_output([script_path, 'PrintRelease'])
+    architectures = get_proc_output([script_path, 'PrintArchitectures'])
+    for arch in architectures.split('\n'):
+      proc = multiprocessing.Process(target=build_and_upload,
+                                     args=(script_path, distro, release, arch))
+      procs.append(("%s %s (%s)" % (distro, release, arch), proc))
+      proc.start()
+  for (_, proc) in procs:
+    proc.join()

[Sam Clegg, 2016/12/12 19:17:59]

Is multiprocessing really helpful here?  I imagine a lot of the stuff is going
to be network bound, and since this is rare operation I think maybe just doing
things synchronously (and showing the output as it happens) might be better.

[Tom (Use chromium acct), 2016/12/12 21:19:34]

It is helpful.  I created this test script:

#! /bin/bash
set -o errexit
./sysroot-creator-precise.sh UpdatePackageListsAll >/dev/null 2>/dev/null
./sysroot-creator-precise.sh BuildSysrootAll >/dev/null 2>/dev/null
./sysroot-creator-precise.sh UploadSysrootAll test_revision_please_delete
>/dev/null 2>/dev/null
./sysroot-creator-trusty.sh UpdatePackageListsAll >/dev/null 2>/dev/null
./sysroot-creator-trusty.sh BuildSysrootAll >/dev/null 2>/dev/null
./sysroot-creator-trusty.sh UploadSysrootAll test_revision_please_delete
>/dev/null 2>/dev/null
./sysroot-creator-wheezy.sh UpdatePackageListsAll >/dev/null 2>/dev/null
./sysroot-creator-wheezy.sh BuildSysrootAll >/dev/null 2>/dev/null
./sysroot-creator-wheezy.sh UploadSysrootAll test_revision_please_delete
>/dev/null 2>/dev/null
./sysroot-creator-jessie.sh UpdatePackageListsAll >/dev/null 2>/dev/null
./sysroot-creator-jessie.sh BuildSysrootAll >/dev/null 2>/dev/null
./sysroot-creator-jessie.sh UploadSysrootAll test_revision_please_delete
>/dev/null 2>/dev/null

I deleted sysroot-build and ran it, and it took 17m42s to finish.
For build_and_upload.py, I also deleted sysroot-build and ran it, but it only
took 2m20s.

If any script fails, a stack trace will be printed which shows the script and
args that failed and you can run that manually to see the output.  Maybe in the
sysroot creation summary, we can just give a message saying to run the failing
scripts manually to see the output (or even save the log to a file)?  The
intended use-case for this script is just before you're ready to upload, which
is why I didn't add an option to disable upload.  I expect if you're making
package changes, you'll be running just the script/step you need manually anyway

+
+  print "SYSROOT CREATION SUMMARY"
+  for (name, proc) in procs:
+    status = "FAILURE" if proc.exitcode else "SUCCESS"
+    print "%s sysroot creation\t%s" % (name, status)
+
+if __name__ == '__main__':
+  build_and_upload_all()