Add CIPD binary installation to DEPS runhooks.

Add CIPD binary installation to DEPS runhooks.

Adds a bootstrap script to DEPS runhooks to download a `cipd` client
and install a list of packages underneath the <infra>/bin/ directory.
The client version and package list are selected based on the host
architecture.

BUG=None
TEST=local
  - Ran locally several times.

Committed: https://chromium.googlesource.com/infra/infra/+/29ea8cd046513577b2d988bc9412401a07b6399a

[dnj, 2015-09-24 03:26:33]

Patchset #1 (id:1) has been deleted

[dnj, 2015-09-24 03:27:57]

Patchset #1 (id:20001) has been deleted

[dnj, 2015-09-24 03:28:28]

dnj@chromium.org changed reviewers:
+ iannucci@chromium.org, vadimsh@chromium.org

[dnj, 2015-09-24 03:28:28]

PTAL, does need some comments, will add tomorrow.

[Vadim Sh., 2015-09-24 16:50:32]

looks good

https://codereview.chromium.org/1364983002/diff/60001/bootstrap/install_cipd_...
File bootstrap/install_cipd_packages.py (right):

https://codereview.chromium.org/1364983002/diff/60001/bootstrap/install_cipd_...
bootstrap/install_cipd_packages.py:40: 'cipd_package_version':
'99439270562c887c887b7538ed634ed3a3c0dc83',
since you are using resolve_cipd_instance_id you can specify version via tag
here, it is more "readable", e.g.
"git_revision:ed808139130f0ea8fd52fc64fd9ec47d150a2e49"

Such version string will also look the same for clients for other platforms
(since they all are built from same revision).

If you prefer to keep raw instance_id, then document how you got it, e.g. 
"./cipd resolve infra/tools/cipd/linux-amd64
-version=git_revision:ed808139130f0ea8fd52fc64fd9ec47d150a2e49"

Raw instance_id only make sense if you also modify resolve_cipd_instance_id to
return it unchanged without making a backend call (since it "resolves" to
itself). Raw instance_id (and only it) matches '^[0-9a-f]{40}$'. Using raw
instance ID everywhere has an advantage that client will make no backend calls
whatsoever if it has all packages installed already.

https://codereview.chromium.org/1364983002/diff/60001/bootstrap/install_cipd_...
bootstrap/install_cipd_packages.py:73: def write_file(path, data):
this is Linux only :( On Windows os.rename when 'path' exists will fail.

https://codereview.chromium.org/1364983002/diff/60001/bootstrap/install_cipd_...
bootstrap/install_cipd_packages.py:251: return 200, urllib2.urlopen(req,
timeout=60).read()
This thing doesn't validate SSL cert :( 

Consider using something like
https://chrome-internal.googlesource.com/infra/infra_internal/+/master/ccompu...

Or moving the bootstrap script to infra.tools.* and using 'requests' library
from venv we have there.

[dnj, 2015-09-24 17:19:35]

https://codereview.chromium.org/1364983002/diff/60001/bootstrap/install_cipd_...
File bootstrap/install_cipd_packages.py (right):

https://codereview.chromium.org/1364983002/diff/60001/bootstrap/install_cipd_...
bootstrap/install_cipd_packages.py:40: 'cipd_package_version':
'99439270562c887c887b7538ed634ed3a3c0dc83',
On 2015/09/24 16:50:32, Vadim Sh. wrote:
> since you are using resolve_cipd_instance_id you can specify version via tag
> here, it is more "readable", e.g.
> "git_revision:ed808139130f0ea8fd52fc64fd9ec47d150a2e49"
> 
> Such version string will also look the same for clients for other platforms
> (since they all are built from same revision).
> 
> If you prefer to keep raw instance_id, then document how you got it, e.g. 
> "./cipd resolve infra/tools/cipd/linux-amd64
> -version=git_revision:ed808139130f0ea8fd52fc64fd9ec47d150a2e49"
> 
> Raw instance_id only make sense if you also modify resolve_cipd_instance_id to
> return it unchanged without making a backend call (since it "resolves" to
> itself). Raw instance_id (and only it) matches '^[0-9a-f]{40}$'. Using raw
> instance ID everywhere has an advantage that client will make no backend calls
> whatsoever if it has all packages installed already.

Ah okay. I'll still use the raw instance ID to avoid the round-trip, but will
add the note and check.

https://codereview.chromium.org/1364983002/diff/60001/bootstrap/install_cipd_...
bootstrap/install_cipd_packages.py:73: def write_file(path, data):
On 2015/09/24 16:50:32, Vadim Sh. wrote:
> this is Linux only :( On Windows os.rename when 'path' exists will fail.

Oh good point. I don't really see a need to make this an atomic write here, so
I'll just go with a regular open in 'rb' mode.

https://codereview.chromium.org/1364983002/diff/60001/bootstrap/install_cipd_...
bootstrap/install_cipd_packages.py:251: return 200, urllib2.urlopen(req,
timeout=60).read()
On 2015/09/24 16:50:32, Vadim Sh. wrote:
> This thing doesn't validate SSL cert :( 
> 
> Consider using something like
>
https://chrome-internal.googlesource.com/infra/infra_internal/+/master/ccompu...
> 
> Or moving the bootstrap script to infra.tools.* and using 'requests' library
> from venv we have there.

grosssssssssssss, forgot about that. Done.

I had considered moving it into a proper tools package, but I was concerned
because some of the other bootstrapping code is required for that package to
work and I didn't want a DEPS file order to be the only thing standing in the
way of the bootstrap breaking.

[Vadim Sh., 2015-09-24 18:09:58]

lgtm with nit

https://codereview.chromium.org/1364983002/diff/80001/bootstrap/install_cipd_...
File bootstrap/install_cipd_packages.py (right):

https://codereview.chromium.org/1364983002/diff/80001/bootstrap/install_cipd_...
bootstrap/install_cipd_packages.py:50: # $ cipd resolve \
nit: "cipd resolve infra/tools/cipd/ -version=..."

'/' at the end instructs it to do the package listing first, here's the output:

$ ./cipd resolve infra/tools/cipd/
-version=git_revision:ed808139130f0ea8fd52fc64fd9ec47d150a2e49
Packages:
  infra/tools/cipd/linux-386:ae3d3163c1498da269cee7d9047708ed6e54ebd4
  infra/tools/cipd/linux-amd64:99439270562c887c887b7538ed634ed3a3c0dc83
  infra/tools/cipd/mac-amd64:68adbfc8694142d17bbd16f5b57a08d32ea3cd0f
  infra/tools/cipd/windows-386:e6ebf6cada9363c96aebd438b764a19246cbcf55
  infra/tools/cipd/windows-amd64:e998e9022b58e0781f7b57b7dae8232c2e52864f

[dnj, 2015-09-24 18:14:35]

ping iannucci@, any comments?

https://codereview.chromium.org/1364983002/diff/80001/bootstrap/install_cipd_...
File bootstrap/install_cipd_packages.py (right):

https://codereview.chromium.org/1364983002/diff/80001/bootstrap/install_cipd_...
bootstrap/install_cipd_packages.py:50: # $ cipd resolve \
On 2015/09/24 18:09:57, Vadim Sh. wrote:
> nit: "cipd resolve infra/tools/cipd/ -version=..."
> 
> '/' at the end instructs it to do the package listing first, here's the
output:
> 
> $ ./cipd resolve infra/tools/cipd/
> -version=git_revision:ed808139130f0ea8fd52fc64fd9ec47d150a2e49
> Packages:
>   infra/tools/cipd/linux-386:ae3d3163c1498da269cee7d9047708ed6e54ebd4
>   infra/tools/cipd/linux-amd64:99439270562c887c887b7538ed634ed3a3c0dc83
>   infra/tools/cipd/mac-amd64:68adbfc8694142d17bbd16f5b57a08d32ea3cd0f
>   infra/tools/cipd/windows-386:e6ebf6cada9363c96aebd438b764a19246cbcf55
>   infra/tools/cipd/windows-amd64:e998e9022b58e0781f7b57b7dae8232c2e52864f
> 

Done.

[Vadim Sh., 2015-09-24 18:20:29]

https://codereview.chromium.org/1364983002/diff/100001/bootstrap/install_cipd...
File bootstrap/install_cipd_packages.py (right):

https://codereview.chromium.org/1364983002/diff/100001/bootstrap/install_cipd...
bootstrap/install_cipd_packages.py:349: cipd.ensure(os.path.join(CIPD_LIST_DIR,
l), root)
actually that will probably not work as you expect.

"cipd ensure -list=..." makes sure contents of the directory matches exactly to
what's in the list. In particular, it will remove packages that are not in
ensure.txt list.

[dnj, 2015-09-24 18:23:05]

https://codereview.chromium.org/1364983002/diff/100001/bootstrap/install_cipd...
File bootstrap/install_cipd_packages.py (right):

https://codereview.chromium.org/1364983002/diff/100001/bootstrap/install_cipd...
bootstrap/install_cipd_packages.py:349: cipd.ensure(os.path.join(CIPD_LIST_DIR,
l), root)
On 2015/09/24 18:20:29, Vadim Sh. wrote:
> actually that will probably not work as you expect.
> 
> "cipd ensure -list=..." makes sure contents of the directory matches exactly
to
> what's in the list. In particular, it will remove packages that are not in
> ensure.txt list.

Oh I see, so one list file per CIPD root? Currently I'm dropping the `cipd`
client into the same directory and it seems fine, though. Does it only wrangle
with symlinks? Should I drop the client somewhere else anyway?

[Vadim Sh., 2015-09-24 18:30:25]

On 2015/09/24 18:23:05, dnj wrote:
>
https://codereview.chromium.org/1364983002/diff/100001/bootstrap/install_cipd...
> File bootstrap/install_cipd_packages.py (right):
> 
>
https://codereview.chromium.org/1364983002/diff/100001/bootstrap/install_cipd...
> bootstrap/install_cipd_packages.py:349:
cipd.ensure(os.path.join(CIPD_LIST_DIR,
> l), root)
> On 2015/09/24 18:20:29, Vadim Sh. wrote:
> > actually that will probably not work as you expect.
> > 
> > "cipd ensure -list=..." makes sure contents of the directory matches exactly
> to
> > what's in the list. In particular, it will remove packages that are not in
> > ensure.txt list.
> 
> Oh I see, so one list file per CIPD root? Currently I'm dropping the `cipd`
> client into the same directory and it seems fine, though. Does it only wrangle
> with symlinks? Should I drop the client somewhere else anyway?

When cipd installs a package into a directory it creates <dir>/.cipd/* service
directory with stuff. In particular it tracks what's installed there. It will
not touch any "untracker" files like cipd client itself, but it will remove any
previously installed packages if they are not in latest "ensure" list. "ensure"
is for specifying the final state. I'm adding "cipd install" subcommand
(https://codereview.chromium.org/1358533003/), but I think 'ensure' is better
for various scripts since its "stateless" in some sense.

tl;dr convert 'cipd_install_lists' from list of files to a single file, should
be good enough for now. If we ever need to share package lists between different
platform we'll come up with something.

[iannucci, 2015-09-24 18:39:48]

lgtm

https://codereview.chromium.org/1364983002/diff/100001/bootstrap/cipd/cipd_li...
File bootstrap/cipd/cipd_linux_amd64.txt (right):

https://codereview.chromium.org/1364983002/diff/100001/bootstrap/cipd/cipd_li...
bootstrap/cipd/cipd_linux_amd64.txt:2: infra/tools/protoc/linux-amd64     
413616a6197cf9030d3b7df5956dfdcbf0388a3b
it would be nice if there was some basic documentation about how to maintain
this file. It's not obvious where the hash comes from, or how to find a newer
version of it.

Also out of curiosity, what happens when someone does check this out on !linux?

In particular, `fetch infra_internal`, which is the currently recommended way to
hack on all things infra (including works-on-windows things such as recipes),
will run this.

[dnj, 2015-09-24 19:20:20]

Patchset #5 (id:120001) has been deleted

[dnj, 2015-09-24 19:20:43]

> tl;dr convert 'cipd_install_lists' from list of files to a single file, should
> be good enough for now. If we ever need to share package lists between
different
> platform we'll come up with something.

Got it, done.

[dnj, 2015-09-24 19:20:51]

https://codereview.chromium.org/1364983002/diff/100001/bootstrap/cipd/cipd_li...
File bootstrap/cipd/cipd_linux_amd64.txt (right):

https://codereview.chromium.org/1364983002/diff/100001/bootstrap/cipd/cipd_li...
bootstrap/cipd/cipd_linux_amd64.txt:2: infra/tools/protoc/linux-amd64     
413616a6197cf9030d3b7df5956dfdcbf0388a3b
On 2015/09/24 18:39:48, iannucci wrote:
> it would be nice if there was some basic documentation about how to maintain
> this file. It's not obvious where the hash comes from, or how to find a newer
> version of it.

I'll check in a README.md to document how I built it.

> Also out of curiosity, what happens when someone does check this out on
!linux?
> 
> In particular, `fetch infra_internal`, which is the currently recommended way
to
> hack on all things infra (including works-on-windows things such as recipes),
> will run this.

The `cipd` client and packages lists are platform-specific. If a platform isn't
listed, `install_cipd_packages.py` no-ops with return code 0. Currently there is
no Windows package configuration, so Windows is a no-op. If we add it in the
future, we'd have a separate `cipd` client and package list for Windows.

[Vadim Sh., 2015-09-24 19:28:30]

lgtm with nit

https://codereview.chromium.org/1364983002/diff/140001/bootstrap/cipd/doc/inf...
File bootstrap/cipd/doc/infra/tools/protoc/linux-amd64/README.md (right):

https://codereview.chromium.org/1364983002/diff/140001/bootstrap/cipd/doc/inf...
bootstrap/cipd/doc/infra/tools/protoc/linux-amd64/README.md:33: $ cipd create \
nit: slap a git_commit:... tag too with SHA1 of protobuf.git, just to keep track
of what's what.

Something like:
cipd create \
  -name "infra/tools/protoc/linux-amd64" \
  -in $ROOT/PREFIX/bin \
  -tag "git_commit:`git rev-parse HEAD`"

[dnj, 2015-09-24 20:29:56]

Cool, thanks for the reviews!

https://codereview.chromium.org/1364983002/diff/140001/bootstrap/cipd/doc/inf...
File bootstrap/cipd/doc/infra/tools/protoc/linux-amd64/README.md (right):

https://codereview.chromium.org/1364983002/diff/140001/bootstrap/cipd/doc/inf...
bootstrap/cipd/doc/infra/tools/protoc/linux-amd64/README.md:33: $ cipd create \
On 2015/09/24 19:28:30, Vadim Sh. wrote:
> nit: slap a git_commit:... tag too with SHA1 of protobuf.git, just to keep
track
> of what's what.
> 
> Something like:
> cipd create \
>   -name "infra/tools/protoc/linux-amd64" \
>   -in $ROOT/PREFIX/bin \
>   -tag "git_commit:`git rev-parse HEAD`"

Done.

[dnj (Google), 2015-09-25 15:26:33]

dnj@google.com changed reviewers:
+ dnj@google.com

[dnj (Google), 2015-09-25 15:26:34]

PTAL one last time (I hope).

This time I fully used `protoc` as it's installed here in a LUCI CL. Some
assumptions that I made were not correct, notably that all I needed was the
`protoc` binary. No! It has external ".proto" files that it needs to include for
its standard structures, notably "Timestamp" and "Duration". Fortunately it's
sane enough to search for them relative to its binary (rather than configured
installation path, *cough*gcc*cough*), so I restructured a bit.

Previous:
<infra>/bin/protoc

Now:
<infra>/cipd_package_root/bin/protoc
<infra>/cipd_package_root/include/.../*.proto

Basically forming a sysroot in <infra> instead of just <bin>. Updated the
`protoc` CIPD package to expose the sysroot, too.

Updated instructions and whatnot, just want to vet the scheme by someone before
committing since this is a medium-size change.

[Vadim Sh., 2015-09-25 18:26:35]

https://codereview.chromium.org/1364983002/diff/180001/.gitignore
File .gitignore (right):

https://codereview.chromium.org/1364983002/diff/180001/.gitignore#newcode11
.gitignore:11: /cipd_package_root/
just /cipd/ ?

cipd_package_root is long..

note also that most of our infra/tools/* packages install in root directly, not
in bin/*. So /cipd will look a bit weird:
/cipd/bin/protoc
/cipd/include/...
/cipd/cipd
/cipd/authutil
/cipd/cloudtail
...

Maybe move protoc out of bin/? Perhaps by providing --exec-prefix or different
--includedir to 'configure'?

[dnj, 2015-09-25 18:39:14]

https://codereview.chromium.org/1364983002/diff/180001/.gitignore
File .gitignore (right):

https://codereview.chromium.org/1364983002/diff/180001/.gitignore#newcode11
.gitignore:11: /cipd_package_root/
On 2015/09/25 18:26:35, Vadim Sh. wrote:
> just /cipd/ ?
> 
> cipd_package_root is long..

Okay.

> note also that most of our infra/tools/* packages install in root directly,
not
> in bin/*. So /cipd will look a bit weird:
> /cipd/bin/protoc
> /cipd/include/...
> /cipd/cipd
> /cipd/authutil
> /cipd/cloudtail
> ...
> 
> Maybe move protoc out of bin/? Perhaps by providing --exec-prefix or different
> --includedir to 'configure'?

The include directory searching is hard-coded into "protoc" logic :( It
searches:
- <binary>/include
- <binary>/../include

I could go with <binary>/include (so still stop-level), but I was trying to be
somewhat forward-thinking in case other utilities needed similar system-level
directories. Not sure if that's too important, though. WDYT?

[Vadim Sh., 2015-09-25 19:28:08]

On 2015/09/25 18:39:14, dnj wrote:
> https://codereview.chromium.org/1364983002/diff/180001/.gitignore
> File .gitignore (right):
> 
> https://codereview.chromium.org/1364983002/diff/180001/.gitignore#newcode11
> .gitignore:11: /cipd_package_root/
> On 2015/09/25 18:26:35, Vadim Sh. wrote:
> > just /cipd/ ?
> > 
> > cipd_package_root is long..
> 
> Okay.

Also drop some cipd/README.md in there, explaining what it is what DEPS hook
built it.

> 
> > note also that most of our infra/tools/* packages install in root directly,
> not
> > in bin/*. So /cipd will look a bit weird:
> > /cipd/bin/protoc
> > /cipd/include/...
> > /cipd/cipd
> > /cipd/authutil
> > /cipd/cloudtail
> > ...
> > 
> > Maybe move protoc out of bin/? Perhaps by providing --exec-prefix or
different
> > --includedir to 'configure'?
> 
> The include directory searching is hard-coded into "protoc" logic :( It
> searches:
> - <binary>/include
> - <binary>/../include
> 
> I could go with <binary>/include (so still stop-level), but I was trying to be
> somewhat forward-thinking in case other utilities needed similar system-level
> directories. Not sure if that's too important, though. WDYT?

I initially planned for "separate sysroot" approach, but somehow all packages
ended up using <root>/exe_name instead of <root>/bin/exe_name :) And now its
PITA to change...

My current rationalization for this: let's model it as a depot_tools-like
toolkit. It is a single directory added to PATH. It has all binaries in root,
and all other stuff in subdirectories. If something doesn't work in this model,
we can use symlinks.... e.g. <root>/roots/protoc/bin/protoc,
<root>/roots/protoc/include/..., <root>/protoc -> symlink to
<root>/roots/protoc/bin/protoc. (CIPD does support symlinks, though not on
Windows, screw windows).

"Real" solution would be to teach CIPD to install in non-root directory, e.g.
cipd_linux_amd64.txt could have:

/ infra/tools/protoc <version>
/bin infra/tools/authutil <version>
/bin infra/tools/cloudtail <version>

But it requires some work...

[dnj (Google), 2015-09-25 23:37:16]

Patchset #8 (id:200001) has been deleted

[dnj (Google), 2015-09-25 23:39:04]

> > Okay.
> 
> Also drop some cipd/README.md in there, explaining what it is what DEPS hook
> built it.

Done

> My current rationalization for this: let's model it as a depot_tools-like
> toolkit. It is a single directory added to PATH. It has all binaries in root,
> and all other stuff in subdirectories. If something doesn't work in this
model,
> we can use symlinks.... e.g. <root>/roots/protoc/bin/protoc,
> <root>/roots/protoc/include/..., <root>/protoc -> symlink to
> <root>/roots/protoc/bin/protoc. (CIPD does support symlinks, though not on
> Windows, screw windows).
> 
> "Real" solution would be to teach CIPD to install in non-root directory, e.g.
> cipd_linux_amd64.txt could have:
> 
> / infra/tools/protoc <version>
> /bin infra/tools/authutil <version>
> /bin infra/tools/cloudtail <version>
> 
> But it requires some work...

Okay - I'll use the "include in same directory as `protoc` binary" approach.
I've repacked with this layout, updated instructions, revised directory layout.
Currently hooks on Linux/amd64 will drop:

/cipd/cipd
/cipd/.cipd_tag_whatever
/cipd/README.md
/cipd/protoc
/cipd/include/google/protobuf/...

[Vadim Sh., 2015-09-25 23:46:33]

https://codereview.chromium.org/1364983002/diff/220001/bootstrap/cipd/doc/DEP...
File bootstrap/cipd/doc/DEPS.README.md (right):

https://codereview.chromium.org/1364983002/diff/220001/bootstrap/cipd/doc/DEP...
bootstrap/cipd/doc/DEPS.README.md:3: The contents of this directory, including
this file, are automatically managed
you can put it in cipd/README.md directly. cipd client won't touch it when
installing or removing stuff.

https://codereview.chromium.org/1364983002/diff/220001/bootstrap/install_cipd...
File bootstrap/install_cipd_packages.py (right):

https://codereview.chromium.org/1364983002/diff/220001/bootstrap/install_cipd...
bootstrap/install_cipd_packages.py:367: install_readme(root)
yeah, I don't understand what is the advantage of this compared to just keeping
cipd/README.md in the repo. Unless you want to auto-generate some content in it.

[dnj (Google), 2015-09-26 01:10:45]

https://codereview.chromium.org/1364983002/diff/220001/bootstrap/cipd/doc/DEP...
File bootstrap/cipd/doc/DEPS.README.md (right):

https://codereview.chromium.org/1364983002/diff/220001/bootstrap/cipd/doc/DEP...
bootstrap/cipd/doc/DEPS.README.md:3: The contents of this directory, including
this file, are automatically managed
On 2015/09/25 23:46:33, Vadim Sh. wrote:
> you can put it in cipd/README.md directly. cipd client won't touch it when
> installing or removing stuff.

Hmm, I suppose I can.

https://codereview.chromium.org/1364983002/diff/220001/bootstrap/install_cipd...
File bootstrap/install_cipd_packages.py (right):

https://codereview.chromium.org/1364983002/diff/220001/bootstrap/install_cipd...
bootstrap/install_cipd_packages.py:367: install_readme(root)
On 2015/09/25 23:46:33, Vadim Sh. wrote:
> yeah, I don't understand what is the advantage of this compared to just
keeping
> cipd/README.md in the repo. Unless you want to auto-generate some content in
it.

Originally I was thinking I'd template in a date or something, but decided that
was overkill.

[Vadim Sh., 2015-09-26 01:16:37]

lgtm with nit

https://codereview.chromium.org/1364983002/diff/240001/cipd/README.md
File cipd/README.md (right):

https://codereview.chromium.org/1364983002/diff/240001/cipd/README.md#newcode3
cipd/README.md:3: The contents of this directory, including this file, are
automatically managed
"including this file" is no longer true

[dnj (Google), 2015-09-26 01:21:54]

The CQ bit was checked by dnj@google.com

[dnj (Google), 2015-09-26 01:21:55]

The patchset sent to the CQ was uploaded after l-g-t-m from
iannucci@chromium.org, vadimsh@chromium.org
Link to the patchset: https://codereview.chromium.org/1364983002/#ps260001
(title: "Technically correct, the best kind of correct! :)")

[commit-bot: I haz the power, 2015-09-26 01:22:03]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1364983002/260001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1364983002/260001

[commit-bot: I haz the power, 2015-09-26 01:25:46]

Committed patchset #10 (id:260001) as
https://chromium.googlesource.com/infra/infra/+/29ea8cd046513577b2d988bc94124...