vpython: Add README and environment protobuf.

vpython: Add README and environment protobuf.

Add the vpython tool README and environment protobuf. The README details
the tool and its layout and usage.

BUG=chromium:691818
TEST=None

Review-Url: https://codereview.chromium.org/2703463006
Committed: https://github.com/luci/luci-go/commit/49cc23e49067b4303d84e86a81e85d25bb27f3ee

[dnj, 2017-02-17 23:17:04]

dnj@chromium.org changed reviewers:
+ iannucci@chromium.org

[dnj, 2017-02-17 23:17:05]

PTAL! This is part of a larger "vpython" CL set for the tool. More information
can be found in the bug, or in the linked design doc:
https://docs.google.com/document/d/1GKN7rtW6-006RbG_Ryh6vr4TpUxSxJlPSCKFLq2nO...

Total CL set is:
https://codereview.chromium.org/2703463006/ (README, protobufs)
https://codereview.chromium.org/2703793003/ (filesystem)
https://codereview.chromium.org/2700273002/ (wheel)
https://codereview.chromium.org/2705623003/ (environment spec)
https://codereview.chromium.org/2701073002/ (Python interpreter management)
https://codereview.chromium.org/2699063004/ (main VirtualEnv package)
https://codereview.chromium.org/2699223002/ (CIPD support)
https://codereview.chromium.org/2702873002/ (package entry point)
https://codereview.chromium.org/2705593003/ (main application)
https://chromium-review.googlesource.com/c/444040/ (Infra entry point)

[iannucci, 2017-02-20 21:18:32]

lgtm w/ comments

https://codereview.chromium.org/2703463006/diff/1/vpython/README.md
File vpython/README.md (right):

https://codereview.chromium.org/2703463006/diff/1/vpython/README.md#newcode11
vpython/README.md:11: * Change tool invocation from `python` to `vpython`.
(set up cipd server, upload packages to cipd)? (right?)

https://codereview.chromium.org/2703463006/diff/1/vpython/README.md#newcode13
vpython/README.md:13: Using `vpython` offers several benefits to standard
Python, especially when
I think there's a missing section above this which actually explains what using
vpython looks like in practice. This section is basically sales copy but doesn't
explain anything about what vpython actually IS.

vpython allows your individual python scripts to express their own package-level
dependencies with immutable references to specific versions. When invoking such
a script via vpython, it downloads these dependencies and prepares an immutable
VirtualEnv containing them. It then invokes the script as usual, using this
virtual env. `vpython` has two levels of caching:
  * SHA-verified dependency-level caching
  * Immutable 'dependency set' virtual env caching
The first caches individual dependency packages in a content-addressed-cache to
avoid unnecessary re-downloads. The second caches full VirtualEnvs which
encapsulate a script's entire set of dependencies (to avoid the overhead of
preparing a fresh virtualenv when re-invoking the same script repeatedly, or
when invoking multiple scripts that share the same set of dependencies).

ish

https://codereview.chromium.org/2703463006/diff/1/vpython/README.md#newcode14
vpython/README.md:14: vendoring packages. Notably, with `vpython`:
I wouldn't contrast vpython v "standard python" it makes it sound like vpython
is a different interpreter or implementation of python.

https://codereview.chromium.org/2703463006/diff/1/vpython/README.md#newcode16
vpython/README.md:16: * It is trivially enable hermetic Python everywhere.
it trivially enables

https://codereview.chromium.org/2703463006/diff/1/vpython/README.md#newcode19
vpython/README.md:19: coordination or cooperation from other tools.
well... you need to upload the new packages to cipd first

https://codereview.chromium.org/2703463006/diff/1/vpython/README.md#newcode33
vpython/README.md:33: using packages the way they are written and tested to be
used, no hacks needed.
Is this written targeting external devs, or infra folk? I would recommend having
this README target the community, and any comments directed at infra (current or
historical) be captured in a different document.

https://codereview.chromium.org/2703463006/diff/1/vpython/README.md#newcode41
vpython/README.md:41: The classic mechanism for vendoring, `sys.path`
manipulation, is nuanced, buggy,
I wouldn't really call this 'classic', unless you're meaning 'in chrome infra',
in which case it was more of a historical mistake.

https://codereview.chromium.org/2703463006/diff/1/vpython/README.md#newcode52
vpython/README.md:52: backed by resilient systems such as Google Storage and
AppEngine.
again, buzzwords about implementation details... this sentence would be better
to give a brief summary:

CIPD is a cross-platform service+tool for securely fetching and deploying
immutable 'packages' (~= zip files) into the local file system. Unlike 'package
managers' it avoids platform-specific assumptions, executable 'hooks', or the
complexities of dependency resolution. `vpython` uses this as a mechanism for
housing+deploying wheels.

https://codereview.chromium.org/2703463006/diff/1/vpython/README.md#newcode63
vpython/README.md:63: A Python
[wheel](https://www.python.org/dev/peps/pep-0427/) is a simple binary
See? This is a good description. This is relatively buzzword free and
informative.

https://codereview.chromium.org/2703463006/diff/1/vpython/README.md#newcode85
vpython/README.md:85: `coverage` in the same CIPD package, but you should not
bundle `numpy==1.11` and
oh... isn't this wasteful? Why not use the existing cipd convention with
${platform} and ${arch}? This means that we'd need to fetch all platforms' data
to install on a given platform?

https://codereview.chromium.org/2703463006/diff/1/vpython/README.md#newcode98
vpython/README.md:98: coverage-4.3.4-cp27-cp27m-macosx_10_10_x86_64.whl
I'd like it if we could add additional template parameters (e.g. ${abi} or
${os_ver}) to cover these differences. I think we already use ${os_ver} in
puppet for python packages.

https://codereview.chromium.org/2703463006/diff/1/vpython/README.md#newcode108
vpython/README.md:108: 
No particular guidance, or just forgot to fill this in?

https://codereview.chromium.org/2703463006/diff/1/vpython/README.md#newcode141
vpython/README.md:141: * Explicitly, as a command-line option to `vpython`.
name the option?

https://codereview.chromium.org/2703463006/diff/1/vpython/README.md#newcode146
vpython/README.md:146: for sentinel text and, if present, load the specification
from that.
can't it also get it from the environment too?

https://codereview.chromium.org/2703463006/diff/1/vpython/api/env/spec.proto
File vpython/api/env/spec.proto (right):

https://codereview.chromium.org/2703463006/diff/1/vpython/api/env/spec.proto#...
vpython/api/env/spec.proto:19: //   "2.7.12", that will suffice.
And similarly '2.6' could get '2.7'?

This may lead to some issues though... e.g.
https://hg.python.org/cpython/raw-file/v2.7.9/Misc/NEWS which verifies https
certs by default. This could cause some code to go from working to !working
without any change to the dependencies, which is a bit meh :/

It's probably good to start this way though.

https://codereview.chromium.org/2703463006/diff/1/vpython/api/env/spec.proto#...
vpython/api/env/spec.proto:21: // If empty, the default Python interpreter
("python") will be used.
can we require that they specify at least a major version? If `python` ever
becomes `python -> python3`, we'd be in trouble.

https://codereview.chromium.org/2703463006/diff/1/vpython/api/env/spec.proto#...
vpython/api/env/spec.proto:35: //   ref/tag.
"key:value" is a tag, "thing" is a ref

I would just say, "any recognized CIPD version (i.e. id, tag, or ref)".

[dnj, 2017-02-22 06:39:12]

https://codereview.chromium.org/2703463006/diff/1/vpython/README.md
File vpython/README.md (right):

https://codereview.chromium.org/2703463006/diff/1/vpython/README.md#newcode11
vpython/README.md:11: * Change tool invocation from `python` to `vpython`.
On 2017/02/20 21:18:22, iannucci wrote:
> (set up cipd server, upload packages to cipd)? (right?)

Done.

https://codereview.chromium.org/2703463006/diff/1/vpython/README.md#newcode13
vpython/README.md:13: Using `vpython` offers several benefits to standard
Python, especially when
On 2017/02/20 21:18:20, iannucci wrote:
> I think there's a missing section above this which actually explains what
using
> vpython looks like in practice. This section is basically sales copy but
doesn't
> explain anything about what vpython actually IS.
> 
> vpython allows your individual python scripts to express their own
package-level
> dependencies with immutable references to specific versions. When invoking
such
> a script via vpython, it downloads these dependencies and prepares an
immutable
> VirtualEnv containing them. It then invokes the script as usual, using this
> virtual env. `vpython` has two levels of caching:
>   * SHA-verified dependency-level caching
>   * Immutable 'dependency set' virtual env caching
> The first caches individual dependency packages in a content-addressed-cache
to
> avoid unnecessary re-downloads. The second caches full VirtualEnvs which
> encapsulate a script's entire set of dependencies (to avoid the overhead of
> preparing a fresh virtualenv when re-invoking the same script repeatedly, or
> when invoking multiple scripts that share the same set of dependencies).
> 
> ish

Done.

https://codereview.chromium.org/2703463006/diff/1/vpython/README.md#newcode14
vpython/README.md:14: vendoring packages. Notably, with `vpython`:
On 2017/02/20 21:18:29, iannucci wrote:
> I wouldn't contrast vpython v "standard python" it makes it sound like vpython
> is a different interpreter or implementation of python.

Done.

https://codereview.chromium.org/2703463006/diff/1/vpython/README.md#newcode16
vpython/README.md:16: * It is trivially enable hermetic Python everywhere.
On 2017/02/20 21:18:26, iannucci wrote:
> it trivially enables

Done.

https://codereview.chromium.org/2703463006/diff/1/vpython/README.md#newcode19
vpython/README.md:19: coordination or cooperation from other tools.
On 2017/02/20 21:18:20, iannucci wrote:
> well... you need to upload the new packages to cipd first

noted

https://codereview.chromium.org/2703463006/diff/1/vpython/README.md#newcode33
vpython/README.md:33: using packages the way they are written and tested to be
used, no hacks needed.
On 2017/02/20 21:18:24, iannucci wrote:
> Is this written targeting external devs, or infra folk? I would recommend
having
> this README target the community, and any comments directed at infra (current
or
> historical) be captured in a different document.

It's a paraphrase of the design doc, really.

https://codereview.chromium.org/2703463006/diff/1/vpython/README.md#newcode41
vpython/README.md:41: The classic mechanism for vendoring, `sys.path`
manipulation, is nuanced, buggy,
On 2017/02/20 21:18:27, iannucci wrote:
> I wouldn't really call this 'classic', unless you're meaning 'in chrome
infra',
> in which case it was more of a historical mistake.

Acknowledged.

https://codereview.chromium.org/2703463006/diff/1/vpython/README.md#newcode52
vpython/README.md:52: backed by resilient systems such as Google Storage and
AppEngine.
On 2017/02/20 21:18:20, iannucci wrote:
> again, buzzwords about implementation details... this sentence would be better
> to give a brief summary:
> 
> CIPD is a cross-platform service+tool for securely fetching and deploying
> immutable 'packages' (~= zip files) into the local file system. Unlike
'package
> managers' it avoids platform-specific assumptions, executable 'hooks', or the
> complexities of dependency resolution. `vpython` uses this as a mechanism for
> housing+deploying wheels.

Done.

https://codereview.chromium.org/2703463006/diff/1/vpython/README.md#newcode63
vpython/README.md:63: A Python
[wheel](https://www.python.org/dev/peps/pep-0427/) is a simple binary
On 2017/02/20 21:18:22, iannucci wrote:
> See? This is a good description. This is relatively buzzword free and
> informative.

A wheel is a bespoke mechanism that synergizes with a specific operating system
to accelerate deployment over deprecated pip packages.

https://codereview.chromium.org/2703463006/diff/1/vpython/README.md#newcode85
vpython/README.md:85: `coverage` in the same CIPD package, but you should not
bundle `numpy==1.11` and
On 2017/02/20 21:18:28, iannucci wrote:
> oh... isn't this wasteful? Why not use the existing cipd convention with
> ${platform} and ${arch}? This means that we'd need to fetch all platforms'
data
> to install on a given platform?

One can, but one is not required to. Your suggestion will also work. Up to the
dev!

https://codereview.chromium.org/2703463006/diff/1/vpython/README.md#newcode98
vpython/README.md:98: coverage-4.3.4-cp27-cp27m-macosx_10_10_x86_64.whl
On 2017/02/20 21:18:19, iannucci wrote:
> I'd like it if we could add additional template parameters (e.g. ${abi} or
> ${os_ver}) to cover these differences. I think we already use ${os_ver} in
> puppet for python packages.

Acknowledged. Let's solve this later.

https://codereview.chromium.org/2703463006/diff/1/vpython/README.md#newcode108
vpython/README.md:108: 
On 2017/02/20 21:18:19, iannucci wrote:
> No particular guidance, or just forgot to fill this in?

oops, I hadn't actually played with Windows when I wrote this :P Fixed.

https://codereview.chromium.org/2703463006/diff/1/vpython/README.md#newcode141
vpython/README.md:141: * Explicitly, as a command-line option to `vpython`.
On 2017/02/20 21:18:25, iannucci wrote:
> name the option?

Done.

https://codereview.chromium.org/2703463006/diff/1/vpython/README.md#newcode146
vpython/README.md:146: for sentinel text and, if present, load the specification
from that.
On 2017/02/20 21:18:23, iannucci wrote:
> can't it also get it from the environment too?

Oh yeah. Didn't exist when I wrote this :P

https://codereview.chromium.org/2703463006/diff/1/vpython/api/env/spec.proto
File vpython/api/env/spec.proto (right):

https://codereview.chromium.org/2703463006/diff/1/vpython/api/env/spec.proto#...
vpython/api/env/spec.proto:19: //   "2.7.12", that will suffice.
On 2017/02/20 21:18:31, iannucci wrote:
> And similarly '2.6' could get '2.7'?
> 
> This may lead to some issues though... e.g.
> https://hg.python.org/cpython/raw-file/v2.7.9/Misc/NEWS which verifies https
> certs by default. This could cause some code to go from working to !working
> without any change to the dependencies, which is a bit meh :/
> 
> It's probably good to start this way though.

Done. Yeah, in all likelihood we can just say "2" unless otherwise needed.

https://codereview.chromium.org/2703463006/diff/1/vpython/api/env/spec.proto#...
vpython/api/env/spec.proto:21: // If empty, the default Python interpreter
("python") will be used.
On 2017/02/20 21:18:31, iannucci wrote:
> can we require that they specify at least a major version? If `python` ever
> becomes `python -> python3`, we'd be in trouble.

Would we? Lots of packages are compatible with both.

I wouldn't mind actually not requiring this. It's trivial enough to specify, and
also trivial enough to go back and fix if a problem arises.

https://codereview.chromium.org/2703463006/diff/1/vpython/api/env/spec.proto#...
vpython/api/env/spec.proto:35: //   ref/tag.
On 2017/02/20 21:18:30, iannucci wrote:
> "key:value" is a tag, "thing" is a ref
> 
> I would just say, "any recognized CIPD version (i.e. id, tag, or ref)".

Done.

[dnj, 2017-02-22 06:39:19]

The CQ bit was checked by dnj@chromium.org

[dnj, 2017-02-22 06:39:19]

The patchset sent to the CQ was uploaded after l-g-t-m from
iannucci@chromium.org
Link to the patchset: https://codereview.chromium.org/2703463006/#ps20001
(title: "comments")

[commit-bot: I haz the power, 2017-02-22 06:39:30]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-02-22 06:47:46]

CQ is committing da patch.

Bot data: {"patchset_id": 20001, "attempt_start_ts": 1487745559048490,
"parent_rev": "6cff766c3aa0c293361346bba1b084572863a595", "commit_rev":
"49cc23e49067b4303d84e86a81e85d25bb27f3ee"}

[commit-bot: I haz the power, 2017-02-22 06:47:49]

Description was changed from

==========
vpython: Add README and environment protobuf.

Add the vpython tool README and environment protobuf. The README details
the tool and its layout and usage.

BUG=chromium:691818
TEST=None
==========

to

==========
vpython: Add README and environment protobuf.

Add the vpython tool README and environment protobuf. The README details
the tool and its layout and usage.

BUG=chromium:691818
TEST=None

Review-Url: https://codereview.chromium.org/2703463006
Committed:
https://github.com/luci/luci-go/commit/49cc23e49067b4303d84e86a81e85d25bb27f3ee
==========

[commit-bot: I haz the power, 2017-02-22 06:47:50]

Committed patchset #2 (id:20001) as
https://github.com/luci/luci-go/commit/49cc23e49067b4303d84e86a81e85d25bb27f3ee