Add gclient hook to download prebuilt mojo_shell

Add gclient hook to download prebuilt mojo_shell

This adds a gclient hook that downloads prebuilt mojo_shell binaries
from google storage (linux only for now) and gyp/gn flags to toggle
using these prebuilt binaries instead of ones built from source. This
also adds a helper script that uploads a locally built mojo_shell binary
to the appropriate place in google storage.  The versioning scheme is:

*) mojo/public/VERSION contains an identifier (committish currently) for
  the mojo public SDK. The mojo_shell binary must match the interfaces
  defined in this version of mojo/public/.

*) gs://mojo/shell/$VERSION/$PLATFORM.zip contains a prebuilt mojo_shell
  binary for a particular version. $PLATFORM for now must be linux-x64,
  but we can add more later as needed.

gclient runhooks downloads the shell binary referred to by
mojo/public/VERSION if it is available for the host platform to
mojo/public/tools/prebuilt/ and updates mojo/public/tools/prebuilt/VERSION
to tag the version downloaded if successful. On subsequent runs the hook
will check this against mojo/public/VERSION and avoid downloading again if
the binary is up to date.

This also adds variables to the GYP and GN builds that control if building
'mojo' builds a mojo_shell binary from source or copies the prebuilt binary
into the outdir.  They both default to false for now.

R=abarth@chromium.org

Committed: https://crrev.com/e098527822f3e25fb5b86c1441f7ce6e4c2cfdc8
Cr-Commit-Position: refs/heads/master@{#300600}

[jamesr, 2014-10-20 22:49:03]

Still working out some of the kinks but PTAL

https://codereview.chromium.org/667853002/diff/1/mojo/public/tools/download_s...
File mojo/public/tools/download_shell_binary.py (right):

https://codereview.chromium.org/667853002/diff/1/mojo/public/tools/download_s...
mojo/public/tools/download_shell_binary.py:35: subprocess.check_call(["gsutil",
"cp", gs_path, temp_zip_file.name])
this should use the gsutil binary in depot_tools - still trying to figure out
how to get that path

[jamesr, 2014-10-20 23:07:59]

OK, think it's using the correct gsutil binary now.

[jamesr, 2014-10-20 23:14:30]

Hmm, apparently that doesn't work.  Something somewhere must be mucking with the
python path to make the hooks work in general...

[jamesr, 2014-10-21 21:03:00]

OK, looking for 'gclient.py' is the ticket since many bots have a
non-depot_tools directory in the PATH with a file in it called 'gclient'

[abarth-chromium, 2014-10-21 21:24:29]

LGTM

https://codereview.chromium.org/667853002/diff/100001/mojo/public/tools/downl...
File mojo/public/tools/download_shell_binary.py (right):

https://codereview.chromium.org/667853002/diff/100001/mojo/public/tools/downl...
mojo/public/tools/download_shell_binary.py:19: sys.exit(0)
exit(1) ?

https://codereview.chromium.org/667853002/diff/100001/tools/find_depot_tools.py
File tools/find_depot_tools.py (right):

https://codereview.chromium.org/667853002/diff/100001/tools/find_depot_tools....
tools/find_depot_tools.py:24: return i
Why this change?

[jamesr, 2014-10-21 22:10:15]

https://codereview.chromium.org/667853002/diff/100001/mojo/public/tools/downl...
File mojo/public/tools/download_shell_binary.py (right):

https://codereview.chromium.org/667853002/diff/100001/mojo/public/tools/downl...
mojo/public/tools/download_shell_binary.py:19: sys.exit(0)
On 2014/10/21 21:24:29, abarth wrote:
> exit(1) ?

I think this is still a successful termination of the hook

https://codereview.chromium.org/667853002/diff/100001/tools/find_depot_tools.py
File tools/find_depot_tools.py (right):

https://codereview.chromium.org/667853002/diff/100001/tools/find_depot_tools....
tools/find_depot_tools.py:24: return i
On 2014/10/21 21:24:29, abarth wrote:
> Why this change?

Lingering effects of diagnostic changes - will revert

[abarth-chromium, 2014-10-21 22:12:38]

ok

[jamesr, 2014-10-21 22:14:46]

The CQ bit was checked by jamesr@chromium.org

[commit-bot: I haz the power, 2014-10-21 22:17:04]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/667853002/120001

[commit-bot: I haz the power, 2014-10-22 01:19:24]

Committed patchset #7 (id:120001)

[commit-bot: I haz the power, 2014-10-22 04:04:46]

Patchset 7 (id:??) landed as
https://crrev.com/e098527822f3e25fb5b86c1441f7ce6e4c2cfdc8
Cr-Commit-Position: refs/heads/master@{#300600}

[yurys, 2014-10-22 05:09:17]

yurys@chromium.org changed reviewers:
+ yurys@chromium.org

[yurys, 2014-10-22 05:09:17]

I'm getting the following error when running gclient sync. It's unclear from the
message which credentials I should check as the rest of the update process went
smoothly.


________ running '/usr/bin/python
src/mojo/public/tools/download_shell_binary.py' in '/sources/chromium'
Failure: No handler was ready to authenticate. 3 handlers were checked.
['OAuth2Auth', 'HmacAuthV1Handler', 'SSOAuth'] Check your credentials.
Traceback (most recent call last):
  File "src/mojo/public/tools/download_shell_binary.py", line 45, in <module>
    subprocess.check_call([gsutil_exe, "cp", gs_path, temp_zip_file.name])
  File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command
'['/sources/depot_tools/third_party/gsutil/gsutil', 'cp',
'gs://mojo/shell/00f7cdc52b6fcab8deda7c5e266c2db8ba60a28a/linux-x64.zip',
'/tmp/tmpnOx_6E']' returned non-zero exit status 1
Sending crash report ...
  args: ['src/mojo/public/tools/download_shell_binary.py']
  cwd: /sources/chromium
  exception: Command '['/sources/depot_tools/third_party/gsutil
  host: <...>
  stack:   File "src/mojo/public/tools/download_shell_binar
  user: <...>
  version: 2.7.6 (default, Mar 22 2014, 22:59:56) 
[GCC 4.8.2
A stack trace has been sent to the maintainers.
Error: Command /usr/bin/python src/mojo/public/tools/download_shell_binary.py
returned non-zero exit status 1 in /sources/chromium

[jamesr, 2014-10-22 05:22:45]

Something's amiss - disabling the hook in
https://codereview.chromium.org/669023002/ until I can investigate.

[loislo, 2014-10-22 05:22:50]

On 2014/10/22 05:09:17, yurys wrote:
> I'm getting the following error when running gclient sync. It's unclear from
the
> message which credentials I should check as the rest of the update process
went
> smoothly.
> 
> 
> ________ running '/usr/bin/python
> src/mojo/public/tools/download_shell_binary.py' in '/sources/chromium'
> Failure: No handler was ready to authenticate. 3 handlers were checked.
> ['OAuth2Auth', 'HmacAuthV1Handler', 'SSOAuth'] Check your credentials.
> Traceback (most recent call last):
>   File "src/mojo/public/tools/download_shell_binary.py", line 45, in <module>
>     subprocess.check_call([gsutil_exe, "cp", gs_path, temp_zip_file.name])
>   File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
>     raise CalledProcessError(retcode, cmd)
> subprocess.CalledProcessError: Command
> '['/sources/depot_tools/third_party/gsutil/gsutil', 'cp',
> 'gs://mojo/shell/00f7cdc52b6fcab8deda7c5e266c2db8ba60a28a/linux-x64.zip',
> '/tmp/tmpnOx_6E']' returned non-zero exit status 1
> Sending crash report ...
>   args: ['src/mojo/public/tools/download_shell_binary.py']
>   cwd: /sources/chromium
>   exception: Command '['/sources/depot_tools/third_party/gsutil
>   host: <...>
>   stack:   File "src/mojo/public/tools/download_shell_binar
>   user: <...>
>   version: 2.7.6 (default, Mar 22 2014, 22:59:56) 
> [GCC 4.8.2
> A stack trace has been sent to the maintainers.
> Error: Command /usr/bin/python src/mojo/public/tools/download_shell_binary.py
> returned non-zero exit status 1 in /sources/chromium

same problem on my box

[yurys, 2014-10-22 05:23:02]

A revert of this CL (patchset #7 id:120001) has been created in
https://codereview.chromium.org/670073002/ by yurys@chromium.org.

The reason for reverting is: gclient sync fails to execute
download_shell_binary.py.

[jamesr, 2014-10-22 05:23:24]

The thing that's downloaded isn't being used yet, so you can safely ignore the
error (until the hook is removed) and continue with your work.  Sorry about the
confusion.

[viettrungluu, 2014-10-22 05:50:31]

On 2014/10/22 05:22:45, jamesr wrote:
> Something's amiss - disabling the hook in
> https://codereview.chromium.org/669023002/ until I can investigate.

I'd suggest looking at download_from_google_storage.py, but I can't make any
sense of it:

* Its --no_auth option sets the boto file to /dev/null, which it then translates
to BOTO_CONFIG= (i.e., empty/unset), which should lead gsutil to use ~/.boto by
default.
* It has stuff for custom .boto file paths (as well as ~/.boto.depot_tools), but
it's not obvious to me if that would ever get used in the context of gclient
sync.

But maybe the real (?) problem is that the ACLs on the gs://mojo bucket are
wrong....

[jamesr, 2014-10-22 23:08:02]

The ACLs are right, the issue is that the gsutil wrapper we have in gsutil
attempts to authenticate before trying to hit the actual bucket and
authentication fails.  --bypass_prodaccess convinces our wrapper to do the right
thing.  Will land this script in the mojo repo and roll it in as part of the SDK