Add a virtualenv-based python bootstrapping service to infra.

bootstrap/README.md

Index: bootstrap/README.md
diff --git a/bootstrap/README.md b/bootstrap/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..868bf81986c625640162febb91ea71e224be4018
--- /dev/null
+++ b/bootstrap/README.md
@@ -0,0 +1,136 @@
+# Bootstrapping the Chromium Infra Repo
+
+The infra/infra repo uses python [wheel files][1], [virtualenv][2] and [pip][3]

[agable, 2014/07/10 17:12:56]

FWIW, I prefer putting [asdf][1]-style footnotes at the bottom of the paragraph
or section, not the bottom of the whole document.

[iannucci, 2014/07/11 02:14:44]

Done.

+to manage dependencies.  The process for bootstrapping these is contained
+entirely within the bootstrap directory.
+
+
+## `bootstrap.py` (a.k.a. "I just want a working infra repo!")
+Run `./bootstrap/bootstrap.py --deps_file bootstrap/deps.pst`. This will create

[agable, 2014/07/10 17:12:56]

Same comment re: python -m.

[iannucci, 2014/07/11 02:14:44]

Nack!

+a virtualenv called `{repo_root}/ENV` with all the deps contained in
+`bootstrap/deps.pst`.  You must be online, or must already have the wheels for

[agable, 2014/07/10 17:12:56]

nit: normalize number of spaces after periods to always be 1.

[iannucci, 2014/07/11 02:14:44]

Done.

+your system cached in `{repo_root}/.wheelcache`.
+
+If you already have an ENV, `bootstrap.py` will check the manifest in ENV to
+see if it matches `deps.pst` (e.g. the diff is zero). If it's not, then ENV

[agable, 2014/07/10 17:12:56]

I think you mean i.e., not e.g.

[iannucci, 2014/07/11 02:14:44]

So I do... And I was being so careful....

+will be re-created _from scratch_.
+
+Running `{repo_root}/run.py` in any capacity will implicitly call `bootstrap.py`

[agable, 2014/07/10 17:12:56]

I don't see it doing this...

[iannucci, 2014/07/11 02:14:44]

Done.

+unless `--no_bootstrap` is passed (which will make run.py use whatever ENV is

[agable, 2014/07/10 17:12:56]

run.py doesn't implement --no_bootstrap.

[iannucci, 2014/07/11 02:14:44]

Acknowledged.

+currently present).
+
+
+## `deps.pst`

[agable, 2014/07/10 17:12:56]

Is ".pst" a standard extension for PyAST files, or did you just make that up?

[iannucci, 2014/07/11 02:14:44]

Changing to .pyl

+This file is a python dictionary containing the exact versions of all python
+module dependencies. These versions are the standard upstream package versions
+(e.g. '0.8.0'), plus the commit hash or sha1.{ext} of an ingested source bundle
+(see `inject_source.py`).
+
+The format of this file is `{'package_name': <values>}`. This file is a python
+[ast literal][4], so comments are allowed and encouraged.
+
+Values are:
+  * version: The pip version of the module
+  * build: An integer representing which build of this version/hash. If you
+      modify the _way_ that a requirement is built, but not the source hash, you
+      can bump the build number to get a new pinned dependency.
+
+And either:
+  * rev: The revision or sha1 of the source for this module.
+    * The repo is
+      'git+https://chromium.googlesource.com/infra/third_party/{package_name}'

[agable, 2014/07/10 17:12:56]

What happens when spread this system to additional repositories. Will their
dependencies live in chromium/tools/third_party? What about infra_internal,
which may rely on both third_party repos from chromium.googlesource and
chrome-internal.googlesource?

[iannucci, 2014/07/11 02:14:44]

Chatted in person, but essentially, the only time this would happen is if we had
an internal library which we wanted to package as a wheel. This won't happen for
a while, and we can extend the deps.pyl format at that point.

+  * gs: '{sha1}.{ext}' indicates file
+    'gs://chrome-infra-wheelhouse/sources/{sha1}.{ext}'. The sha1 will be
+    checked against the content of the file.
+
+And optionally:
+  * implicit: A boolean indicating that this dep should only be installed as a
+      dependency of some other dep. For example, you want package A, which
+      depends on package Z, but you don't really care about Z. You should mark
+      Z as 'implicit' to allow it to be pinned correctly, but not to
+      deliberately install it.
+

[agable, 2014/07/10 17:12:56]

nit: two newlines

[iannucci, 2014/07/11 02:14:44]

Done.

+## `ingest_source.py`
+Some python modules don't have functional python repos (i.e. ones that pip
+can natively clone+build), and thus ship their source in tarballs. To ingest
+such a tarball into the infra google storage bucket, use
+  `ingest_source.py /path/to/archive`.
+This will print the value for the 'gs' key for a deps.pst entry.
+
+
+## `build_deps.py` / rolling deps

[agable, 2014/07/10 17:12:56]

This file doesn't actually do a good job of describing the relationship between
ingest_source and build_deps and other tools. While you describe each tool well,
you don't describe workflows. AFAIK, the following workflows need to be covered:
* Adding a pure-python dependency
* Adding a pure-python dependency that isn't a well-formed package
* Adding a compiled dependency
* Rolling forward all of the above
* Forking + patching all of the above

[iannucci, 2014/07/11 02:14:44]

Done. Sort of. :D

+Any time a new dependency/version is introduced into `deps.pst`, you must run
+`build_deps.py`. If the dependency is a pure-python dependency (i.e. no compiled
+extensions), you only need to run it once on CPython 2.7. You can tell that it's
+a pure python module by looking at the name of the wheel file. For example:
+
+  requests-2.3.0-py2.py3-none-any.whl
+
+Is compatible with Python 2 and Python 3 (py2.py3) any python ABI (none), and
+any OS platform (any).
+
+If the module does contain compiled extensions, you must run `build_deps.py` on
+the following systems (all with CPython 2.7):
+  * OS X 10.9 - `x86_64`
+  * Windows 7 - `x86_64`
+  * Linux     - `x86_64`
+
+  TODO(iannucci): Add job to build wheels on all appropriate systems.
+
+Once a wheel is sucessfully built, it is uploaded to

[Vadim Sh., 2014/07/10 18:18:24]

Who has write access to the bucket?

+`gs://chrome-python-wheelhouse/wheels` if it is not there already.
+
+Running `build_deps.py` will only attempt to build dependencies which are
+missing for the current platform.
+
+`build_deps.py` assumes that it can find `gsutil` on PATH, so go ahead and

[Vadim Sh., 2014/07/10 18:18:24]

Does it have minimum version requirement for gsutil?

+install it appropriately for whichever platform you're on.
+
+
+## custom builds
+Sometimes building a wheel is a bit trickier than `pip wheel {repo}@{hash}`. In
+order to support this, add a script named `custom_builds/{name}.py`. This module
+should have a function defined like:
+
+  `def Build(source_path, wheelhouse_path)`
+
+Where `source_path` is a string path to the checked-out / unpacked source code,
+and `wheelhouse_path` is a string path where `build_deps.py` expects to find
+a .whl file after Build completes.
+
+
+## rolling the version of wheel
+Since wheel is a package needed to build the wheels, it has a slightly different
+treatment. To roll wheel, bump the version in deps.pst, and then run
+  `bootstrap_wheel_wheel.sh`

[Vadim Sh., 2014/07/10 18:18:24]

funny name :)

+To build and upload the wheel for 'wheel' pinned at the version in deps.pst.

[agable, 2014/07/10 17:12:56]

nit: uncapitalize To

[iannucci, 2014/07/11 02:14:44]

Done.

+
+Once you do that, `build_deps.py` will continue working as expected.
+
+
+## Building deps on Windows
+TODO(iannucci): actually implement this
+
+Windows builds require a slightly more care when building, due to the
+complexities of getting a compile environment. To this effect, `build_deps.py`
+relies on the `depot_tools/win_toolchain` functionality to get a hermetic
+windows compiler toolchain. This should not be an issue for chromium devs
+working on windows, since they should already have this installed by compiling
+chromium, but it's something to be aware of.
+
+
+## modified (non-upstream) deps
+If it is necessary to roll a patched version of a library, we should branch it
+in the infra googlesource mirror. This branch should be named `{version}-cr`,
+and will build packages whose version is `{version}.{cr_version}` (e.g. modify
+setup.py on this branch to add an additional component to the version field).
+
+For example, given the package `jane` at version `2.1.3`, we would create
+a branch `2.1.3-cr`. On this branch we would commit any changes necessary to
+`2.1.3`, and would adjust the version number in the builds to be e.g. `2.1.3.0`.
+
+
+1: http://legacy.python.org/dev/peps/pep-0427/
+2: https://github.com/pypa/virtualenv
+3: https://github.com/pypa/pip
+4: https://docs.python.org/2/library/ast.html#ast.literal_eval