NaCl: Pull in NaCl's integrated runtime (IRT) library via DEPS

NaCl: Pull in NaCl's integrated runtime (IRT) library via DEPS

This hooks up the download script that I added in an earlier commit.

This also tweaks download_nacl_irt.py to add a "nacl_" prefix to the
downloaded files.  This is in preparation for when the files are added
to a Gyp 'copies' rule, since these Gyp rules are not able to rename
files.  We want to make the files' purpose clear when they are copied
into an install directory along with non-NaCl-related files.

BUG=http://code.google.com/p/nativeclient/issues/detail?id=1595
TEST=run "gclient runhooks" in various situations:
 1) IRT binaries not present: files should get downloaded
 2) IRT binaries present: re-use existing downloaded files
 3) nacl_revision changed: re-download, and give error about hashes

Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=83267

[bradn, 2011-04-27 23:52:24]

LGTM

[Mark Mentovai, 2011-04-28 18:41:16]

http://codereview.chromium.org/6893080/diff/1/DEPS
File DEPS (right):

http://codereview.chromium.org/6893080/diff/1/DEPS#newcode366
DEPS:366: "action": ["python", "src/build/download_nacl_irt.py",
What’s this about?

[Mark Seaborn, 2011-04-28 19:01:24]

On 28 April 2011 11:41, <mark@chromium.org> wrote:

> http://codereview.chromium.org/6893080/diff/1/DEPS
> File DEPS (right):
>
> http://codereview.chromium.org/6893080/diff/1/DEPS#newcode366
> DEPS:366: "action": ["python", "src/build/download_nacl_irt.py",
> What’s this about?
>

It downloads binaries of a library which is built as Native Client untrusted
code.  The library requires the NaCl toolchain to build.  It gets built on
NaCl's buildbots, which upload the binaries to
http://commondatastorage.googleapis.com automatically.

On IM you suggested putting the binaries in SVN.  We can't do this manually
-- the library changes too frequently -- so we'd have to do it
automatically.  I'm not opposed to that, but I defer to Brad Nelson on this
topic because he set up the infrastructure for NaCl in which toolchain
snapshots are uploaded and downloaded.

Cheers,
Mark

[Mark Mentovai, 2011-04-28 19:07:03]

Mark Seaborn wrote:
> On 28 April 2011 11:41, <mark@chromium.org> wrote:
>>
>> http://codereview.chromium.org/6893080/diff/1/DEPS
>> File DEPS (right):
>>
>> http://codereview.chromium.org/6893080/diff/1/DEPS#newcode366
>> DEPS:366: "action": ["python", "src/build/download_nacl_irt.py",
>> What’s this about?
>
> It downloads binaries of a library which is built as Native Client untrusted
> code.  The library requires the NaCl toolchain to build.  It gets built on
> NaCl's buildbots, which upload the binaries to
> http://commondatastorage.googleapis.com automatically.
>
> On IM you suggested putting the binaries in SVN.  We can't do this manually
> -- the library changes too frequently -- so we'd have to do it
> automatically.  I'm not opposed to that, but I defer to Brad Nelson on this
> topic because he set up the infrastructure for NaCl in which toolchain
> snapshots are uploaded and downloaded.

I feel pretty strongly that we should put this stuff into Subversion.
As I mentioned on IM, the download script seems like it’s trying to
simulate version control, except it leaves open lots of questions,
like how long the data is retained for and how to do the equivalent of
“svn log”. I’m worried about build reproducibility and that kind of
thing.

For Chrome’s purposes, where it seems like we only care about these
two IRT library files, I’m of the mind that we should just check them
in, either to the repository that already holds NaCl stuff, or to a
new googlecode repository dedicated to the task. Using the NaCl
repository would be preferable in my opinion because the revision
numbers would match up with the revision numbers we’re already putting
into DEPS.

Regardless of the solution, this will need some sort of svn:ignore to
keep all of those ? lines from showing up in “svn stat” (or “gclient
status”).

[Mark Seaborn, 2011-04-29 02:19:01]

On 28 April 2011 12:06, Mark Mentovai <mark@chromium.org> wrote:

> Mark Seaborn wrote:
> > On 28 April 2011 11:41, <mark@chromium.org> wrote:
> >>
> >> http://codereview.chromium.org/6893080/diff/1/DEPS
> >> File DEPS (right):
> >>
> >> http://codereview.chromium.org/6893080/diff/1/DEPS#newcode366
> >> DEPS:366: "action": ["python", "src/build/download_nacl_irt.py",
> >> What’s this about?
> >
> > It downloads binaries of a library which is built as Native Client
> untrusted
> > code.  The library requires the NaCl toolchain to build.  It gets built
> on
> > NaCl's buildbots, which upload the binaries to
> > http://commondatastorage.googleapis.com automatically.
> >
> > On IM you suggested putting the binaries in SVN.  We can't do this
> manually
> > -- the library changes too frequently -- so we'd have to do it
> > automatically.  I'm not opposed to that, but I defer to Brad Nelson on
> this
> > topic because he set up the infrastructure for NaCl in which toolchain
> > snapshots are uploaded and downloaded.
>
> I feel pretty strongly that we should put this stuff into Subversion.
>

Brad N has replied separately about the practicalities of doing that.


As I mentioned on IM, the download script seems like it’s trying to
> simulate version control, except it leaves open lots of questions,
> like how long the data is retained for and how to do the equivalent of
> “svn log”. I’m worried about build reproducibility and that kind of
> thing.
>

I also worry about build reproducibility.  I think there are two halves to
this:

 1) Availability:  can we ensure that past data is accessible and hasn't
disappeared?
 2) Integrity:  when we retrieve past data, can be sure that it is the right
data?

The integrity part should be addressed by the hash checks I put into
download_nacl_irt.py.  At least, once the IRT binaries have been captured in
Chrome's DEPS, they cannot be changed on the Google Storage for Developers
site without the change being detected when they are downloaded.  (The files
could be changed before they're captured in DEPS but that's another matter.)

If you're worried about availability it's pretty simple to mirror
http://gsdview.appspot.com/nativeclient-archive2/irt/ with wget or similar.

In some ways this is a little better than SVN.  A reference to an SVN repo
in DEPS does not give an integrity guarantee:  you can't be sure that you'll
check out a tree that is byte-for-byte that same as what was pinned in
DEPS.  It's possible for third parties to mirror SVN repos (e.g. with
svnsync), but probably harder than mirroring flat files.


For Chrome’s purposes, where it seems like we only care about these
> two IRT library files, I’m of the mind that we should just check them
> in, either to the repository that already holds NaCl stuff, or to a
> new googlecode repository dedicated to the task. Using the NaCl
> repository would be preferable in my opinion because the revision
> numbers would match up with the revision numbers we’re already putting
> into DEPS.
>

We don't want to put these binaries in the NaCl source repo because they're
automatically built by NaCl's Buildbot on every commit.

Mark