vpython: Add CIPD support.

vpython: Add CIPD support.

Add a VirtualEnv PackageLoader implementation built on CIPD. This will
be installed by the main "vpython" executable and used exclusively in
the Chrome Operations build of "vpython".

The CIPD installation is baked into "vpython", and uses (in-memory)
ensure file format to resolve packages by name.

BUG=chromium:691818
TEST=None

Review-Url: https://codereview.chromium.org/2699223002
Committed: https://github.com/luci/luci-go/commit/ddf90529577a925f0c6720bc71238f18a0184611

[dnj, 2017-02-17 23:14:09]

Description was changed from

==========
Updated patchset dependency

vpython: Add CIPD support.

Add a VirtualEnv PackageLoader implementation built on CIPD. This will
be installed by the main "vpython" executable and used exclusively in
the Chrome Operations build of "vpython".

The CIPD installation is baked into "vpython", and uses (in-memory)
ensure file format to resolve packages by name.

BUG=chromium:691818
TEST=None
==========

to

==========
vpython: Add CIPD support.

Add a VirtualEnv PackageLoader implementation built on CIPD. This will
be installed by the main "vpython" executable and used exclusively in
the Chrome Operations build of "vpython".

The CIPD installation is baked into "vpython", and uses (in-memory)
ensure file format to resolve packages by name.

BUG=chromium:691818
TEST=None
==========

[dnj, 2017-02-17 23:17:47]

dnj@chromium.org changed reviewers:
+ iannucci@chromium.org

[dnj, 2017-02-17 23:17:48]

PTAL! This is part of a larger "vpython" CL set for the tool. More information
can be found in the bug, or in the linked design doc:
https://docs.google.com/document/d/1GKN7rtW6-006RbG_Ryh6vr4TpUxSxJlPSCKFLq2nO...

Total CL set is:
https://codereview.chromium.org/2703463006/ (README, protobufs)
https://codereview.chromium.org/2703793003/ (filesystem)
https://codereview.chromium.org/2700273002/ (wheel)
https://codereview.chromium.org/2705623003/ (environment spec)
https://codereview.chromium.org/2701073002/ (Python interpreter management)
https://codereview.chromium.org/2699063004/ (main VirtualEnv package)
https://codereview.chromium.org/2699223002/ (CIPD support)
https://codereview.chromium.org/2702873002/ (package entry point)
https://codereview.chromium.org/2705593003/ (main application)
https://chromium-review.googlesource.com/c/444040/ (Infra entry point)

[iannucci, 2017-02-23 01:06:56]

lgtm

https://codereview.chromium.org/2699223002/diff/1/vpython/cipd/cipd.go
File vpython/cipd/cipd.go (right):

https://codereview.chromium.org/2699223002/diff/1/vpython/cipd/cipd.go#newcode38
vpython/cipd/cipd.go:38: // The resulting packages slice will be updated
in-place with the resolves
resolved

https://codereview.chromium.org/2699223002/diff/1/vpython/cipd/cipd.go#newcode64
vpython/cipd/cipd.go:64: ef, err := ensure.ParseFile(&ensureFile)
could just make an ensure.File directly rather than going through the parser.

https://codereview.chromium.org/2699223002/diff/1/vpython/cipd/cipd.go#newcode79
vpython/cipd/cipd.go:79: packages[i].Path = pkg.PackageName
why is it packages[i].Path? Shouldn't it be Name or something like that?

https://codereview.chromium.org/2699223002/diff/1/vpython/cipd/cipd.go#newcod...
vpython/cipd/cipd.go:148: func validatePackage(pkg *env.Spec_Package) error {
tbh... Ensure will do this better anyway:
https://github.com/luci/luci-go/blob/master/cipd/client/cipd/client.go#L1255

[dnj, 2017-03-11 16:47:24]

https://codereview.chromium.org/2699223002/diff/1/vpython/cipd/cipd.go
File vpython/cipd/cipd.go (right):

https://codereview.chromium.org/2699223002/diff/1/vpython/cipd/cipd.go#newcode38
vpython/cipd/cipd.go:38: // The resulting packages slice will be updated
in-place with the resolves
On 2017/02/23 01:06:56, iannucci wrote:
> resolved

Done.

https://codereview.chromium.org/2699223002/diff/1/vpython/cipd/cipd.go#newcode64
vpython/cipd/cipd.go:64: ef, err := ensure.ParseFile(&ensureFile)
On 2017/02/23 01:06:56, iannucci wrote:
> could just make an ensure.File directly rather than going through the parser.

Oh neat! I don't think this was exported the first time I wrote this.

https://codereview.chromium.org/2699223002/diff/1/vpython/cipd/cipd.go#newcode79
vpython/cipd/cipd.go:79: packages[i].Path = pkg.PackageName
On 2017/02/23 01:06:56, iannucci wrote:
> why is it packages[i].Path? Shouldn't it be Name or something like that?

In the protobuf, "package" is an intentionally generic "Path" / "Version"
combination. The way this is constructed, someone could create a PackageLoader
using a non-CIPD service, in which case "Path" could map to someone's Google
Storage bucket URL, or be a full download URL, and the protobuf fields are
applicable there, too.

I didn't want to need to implement first-class support for each possible method
in the proto (e.g., oneof), as that would make the text form more annoying and
also imply the full set of imagined capabilities are available.

Maybe I'll change to "Name" though; it's just as generic and also fits better.

https://codereview.chromium.org/2699223002/diff/1/vpython/cipd/cipd.go#newcod...
vpython/cipd/cipd.go:148: func validatePackage(pkg *env.Spec_Package) error {
On 2017/02/23 01:06:56, iannucci wrote:
> tbh... Ensure will do this better anyway:
> https://github.com/luci/luci-go/blob/master/cipd/client/cipd/client.go#L1255

Done.

[dnj, 2017-03-11 16:53:52]

The CQ bit was checked by dnj@chromium.org

[dnj, 2017-03-11 16:53:52]

The patchset sent to the CQ was uploaded after l-g-t-m from
iannucci@chromium.org
Link to the patchset: https://codereview.chromium.org/2699223002/#ps40001
(title: "comments, path => name")

[commit-bot: I haz the power, 2017-03-11 16:54:04]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-03-11 17:00:21]

CQ is committing da patch.

Bot data: {"patchset_id": 40001, "attempt_start_ts": 1489251232352650,
"parent_rev": "af075cbfb85dba33b2fc91881e530700e3d74e46", "commit_rev":
"ddf90529577a925f0c6720bc71238f18a0184611"}

[commit-bot: I haz the power, 2017-03-11 17:00:25]

Description was changed from

==========
vpython: Add CIPD support.

Add a VirtualEnv PackageLoader implementation built on CIPD. This will
be installed by the main "vpython" executable and used exclusively in
the Chrome Operations build of "vpython".

The CIPD installation is baked into "vpython", and uses (in-memory)
ensure file format to resolve packages by name.

BUG=chromium:691818
TEST=None
==========

to

==========
vpython: Add CIPD support.

Add a VirtualEnv PackageLoader implementation built on CIPD. This will
be installed by the main "vpython" executable and used exclusively in
the Chrome Operations build of "vpython".

The CIPD installation is baked into "vpython", and uses (in-memory)
ensure file format to resolve packages by name.

BUG=chromium:691818
TEST=None

Review-Url: https://codereview.chromium.org/2699223002
Committed:
https://github.com/luci/luci-go/commit/ddf90529577a925f0c6720bc71238f18a0184611
==========

[commit-bot: I haz the power, 2017-03-11 17:00:26]

Committed patchset #3 (id:40001) as
https://github.com/luci/luci-go/commit/ddf90529577a925f0c6720bc71238f18a0184611