vpython: Add README and environment protobuf.

vpython/README.md

Index: vpython/README.md
diff --git a/vpython/README.md b/vpython/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..d527145f6a1bf79119028b6f9d87f56e9e27f6f4
--- /dev/null
+++ b/vpython/README.md
@@ -0,0 +1,178 @@
+## vpython - simple and easy VirtualEnv Python
+
+`vpython` is a tool, written in Go, which enables the simple and easy invocation
+of Python code in [VirtualEnv](https://virtualenv.pypa.io/en/stable/)
+environments.
+
+For the standard case, employing `vpython` is as simple as:
+
+* Add `vpython` to `PATH`.
+* Write an enviornment specification naming packages.
+* Change tool invocation from `python` to `vpython`.

[iannucci, 2017/02/20 21:18:22]

(set up cipd server, upload packages to cipd)? (right?)

[dnj, 2017/02/22 06:39:10]

Done.

+
+Using `vpython` offers several benefits to standard Python, especially when

[iannucci, 2017/02/20 21:18:20]

I think there's a missing section above this which actually explains what using
vpython looks like in practice. This section is basically sales copy but doesn't
explain anything about what vpython actually IS.

vpython allows your individual python scripts to express their own package-level
dependencies with immutable references to specific versions. When invoking such
a script via vpython, it downloads these dependencies and prepares an immutable
VirtualEnv containing them. It then invokes the script as usual, using this
virtual env. `vpython` has two levels of caching:
  * SHA-verified dependency-level caching
  * Immutable 'dependency set' virtual env caching
The first caches individual dependency packages in a content-addressed-cache to
avoid unnecessary re-downloads. The second caches full VirtualEnvs which
encapsulate a script's entire set of dependencies (to avoid the overhead of
preparing a fresh virtualenv when re-invoking the same script repeatedly, or
when invoking multiple scripts that share the same set of dependencies).

ish

[dnj, 2017/02/22 06:39:10]

Done.

+vendoring packages. Notably, with `vpython`:

[iannucci, 2017/02/20 21:18:29]

I wouldn't contrast vpython v "standard python" it makes it sound like vpython
is a different interpreter or implementation of python.

[dnj, 2017/02/22 06:39:11]

Done.

+
+* It is trivially enable hermetic Python everywhere.

[iannucci, 2017/02/20 21:18:26]

it trivially enables

[dnj, 2017/02/22 06:39:11]

Done.

+* No `sys.path` manipulation is needed to load vendored or imported packages.
+* Any tool can trivially define which package(s) it needs, without requiring
+  coordination or cooperation from other tools.

[iannucci, 2017/02/20 21:18:20]

well... you need to upload the new packages to cipd first

[dnj, 2017/02/22 06:39:10]

noted

+* Adding new Python dependencies to a project is non-invasive and immediate.
+* Package downloading and deployment are baked into `vpython` and built on
+  fast and secure Google Cloud Platform technologies.
+* No more custom bootstraps. Several projects and tools, including multiple
+  within the infra code base, have bootstrap scripts that vendor packages or
+  mimic a VirtualEnv. These are at best repetitive and, at worst, buggy and
+  insecure.
+* Depenencies are explicitly stated, not assumed.
+
+### Why VirtualEnv?
+
+VirtualEnv offers several benefits over system Python. Primarily, it is the
+community-supported mechanism to vendor Python packages. This means that we're
+using packages the way they are written and tested to be used, no hacks needed.

[iannucci, 2017/02/20 21:18:24]

Is this written targeting external devs, or infra folk? I would recommend having
this README target the community, and any comments directed at infra (current or
historical) be captured in a different document.

[dnj, 2017/02/22 06:39:11]

It's a paraphrase of the design doc, really.

+
+By using the same environemnt everywhere, Python invocations become
+reproducible. A tool run on a developer's system will load the same versions
+of the same libraries as it will on a production system. A production system
+will no longer fail because it is missing a package, or because it has the
+wrong version.
+
+The classic mechanism for vendoring, `sys.path` manipulation, is nuanced, buggy,

[iannucci, 2017/02/20 21:18:27]

I wouldn't really call this 'classic', unless you're meaning 'in chrome infra',
in which case it was more of a historical mistake.

[dnj, 2017/02/22 06:39:10]

Acknowledged.

+and unsupported by the Python community. It is difficult to get right on all
+platforms in all environments for all packages. A notorious example of this is
+`protobuf` and other domain-bound packages, which actively fight `sys.path`
+inclusion. Using VirtualEnv means that any compliant Python package can
+trivially be included into a project.
+
+### Why CIPD?
+
+[CIPD](https://github.com/luci/luci-go/tree/master/cipd) is Chrome's
+infrastructure package deployment system. It is simple, accessible, fast, and
+backed by resilient systems such as Google Storage and AppEngine.

[iannucci, 2017/02/20 21:18:20]

again, buzzwords about implementation details... this sentence would be better
to give a brief summary:

CIPD is a cross-platform service+tool for securely fetching and deploying
immutable 'packages' (~= zip files) into the local file system. Unlike 'package
managers' it avoids platform-specific assumptions, executable 'hooks', or the
complexities of dependency resolution. `vpython` uses this as a mechanism for
housing+deploying wheels.

[dnj, 2017/02/22 06:39:11]

Done.

+
+Unlike `pip`, a CIPD package is defined by its content, enabling precise package
+matching instead of fuzzy version matching (e.g., `numpy >= 1.2`, and
+`numpy == 1.2` both can match multiple `numpy` packages in `pip`).
+
+CIPD also supports ACLs, enabling privileged Python projects to easily vendor
+sensitive packages.
+
+### Why wheels?
+
+A Python [wheel](https://www.python.org/dev/peps/pep-0427/) is a simple binary

[iannucci, 2017/02/20 21:18:22]

See? This is a good description. This is relatively buzzword free and
informative.

[dnj, 2017/02/22 06:39:10]

A wheel is a bespoke mechanism that synergizes with a specific operating system
to accelerate deployment over deprecated pip packages.

+distrubition of Python code. A wheel can be generic (pure Python) or system-
+and architecture-bound (e.g., 64-bit Mac OSX).
+
+Wheels are prefered over eggs because they come packaged with compiled binaries.
+This makes their deployment simple (unpack via `pip`) and reduces system
+requirements and variation, since local compilation is not needed.
+
+The increased management burden of maintaining separate wheels for the same
+package, one for each architecture, is handled naturally by CIPD, removing the
+only real pain point.
+
+## Wheel Guidance
+
+This section contains recommendations for building or uploading wheel CIPD
+packages, including platform-specific guidance.
+
+CIPD wheel packages are CIPD packages that contain Python wheels. A given CIPD
+package can contain multiple wheels for multiple platforms, but should only
+contain one version of any given package for any given architecture/platform.
+
+For example, you can bundle a Windows, Linux, and Mac OSX version of `numpy` and
+`coverage` in the same CIPD package, but you should not bundle `numpy==1.11` and

[iannucci, 2017/02/20 21:18:28]

oh... isn't this wasteful? Why not use the existing cipd convention with
${platform} and ${arch}? This means that we'd need to fetch all platforms' data
to install on a given platform?

[dnj, 2017/02/22 06:39:10]

One can, but one is not required to. Your suggestion will also work. Up to the
dev!

+`numpy==1.12` in the same package.
+
+The reason for this is that `vpython` identifies which wheels to install by
+scanning the contents of the CIPD package, and if multiple versions appear,
+there is no clear guidance about which should be used.
+
+### Mac OSX
+
+Use the `m` ABI suffix and the `macosx_...` platform. `vpython` installs wheels
+with the `--force` flag, so slight binary incompatibilities (e.g., specific OSX
+versions) can be glossed over.
+
+    coverage-4.3.4-cp27-cp27m-macosx_10_10_x86_64.whl

[iannucci, 2017/02/20 21:18:19]

I'd like it if we could add additional template parameters (e.g. ${abi} or
${os_ver}) to cover these differences. I think we already use ${os_ver} in
puppet for python packages.

[dnj, 2017/02/22 06:39:10]

Acknowledged. Let's solve this later.

+
+### Linux
+
+Use wheels with the `mu` ABI suffix and the `manylinux1` platform. For example:
+
+    coverage-4.3.4-cp27-cp27mu-manylinux1_x86_64.whl
+
+### Windows
+
+

[iannucci, 2017/02/20 21:18:19]

No particular guidance, or just forgot to fill this in?

[dnj, 2017/02/22 06:39:11]

oops, I hadn't actually played with Windows when I wrote this :P Fixed.

+
+## Setup and Invocation
+
+`vpython` can be invoked by replacing `python` in the command-line with
+`vpython`.
+
+`vpython` works with a default Python environment out of the box. To add
+vendored packges, you need to define an enviornment specification file that
+describes which wheels to install.
+
+An enviornment specification file is a text protobuf defined as `Spec`
+[here](./api/env/spec.proto). An example is:
+
+```
+# Any 2.7 interpreter will do.
+python_version: "2.7"
+
+# Include "numpy" for the current architecture.
+wheel {
+  path: "infra/python/wheels/numpy/${platform}-${arch}"
+  version: "version:1.11.0"
+}
+
+# Include "coverage" for the current architecture.
+wheel {
+  path: "infra/python/wheels/coverage/${platform}-${arch}"
+  version: "version:4.1"
+}
+```
+
+This specification can be supplied in one of three ways:
+
+* Explicitly, as a command-line option to `vpython`.

[iannucci, 2017/02/20 21:18:25]

name the option?

[dnj, 2017/02/22 06:39:10]

Done.

+* Implicitly, as a file alongside your entry point. For example, if you are
+  running `test_runner.py`, `vpython` will look for `test_runner.py.vpython`
+  next to it and load the environment from there.
+* Implicitly, inined in your main file. `vpython` will scan the main entry point
+  for sentinel text and, if present, load the specification from that.

[iannucci, 2017/02/20 21:18:23]

can't it also get it from the environment too?

[dnj, 2017/02/22 06:39:10]

Oh yeah. Didn't exist when I wrote this :P

+
+### Migration
+
+#### Command-line.
+
+`vpython` is a natural replacement for `python` in the command line:
+
+```sh
+python ./foo/bar/baz.py -d --flag value arg arg whatever
+```
+
+Becomes:
+```sh
+vpython ./foo/bar/baz.py -d --flag value arg arg whatever
+```
+
+The `vpython` tool accepts its own command-line arguments. In this case, use
+a `--` seprator to differentiate between `vpython` options and `python` options:
+
+```sh
+vpython -spec /path/to/spec.vpython -- ./foo/bar/baz.py
+```
+
+#### Shebang (POSIX)
+
+If your script uses implicit specification (file or inline), replacing `python`
+with `vpython` in your shebang line will automatically work.
+
+```sh
+#!/usr/bin/env vpython
+```
+