use gsutil to download clang et al binaries

use gsutil to download clang et al binaries

BUG=586146

[Mostyn Bramley-Moore, 2016-02-12 00:08:42]

The CQ bit was checked by mostynb@opera.com to run a CQ dry run

[commit-bot: I haz the power, 2016-02-12 00:10:44]

Dry run: CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1688183004/1
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1688183004/1

[hans, 2016-02-12 00:39:10]

hans@chromium.org changed reviewers:
+ hans@chromium.org

[hans, 2016-02-12 00:39:11]

https://codereview.chromium.org/1688183004/diff/1/tools/clang/scripts/update.py
File tools/clang/scripts/update.py (right):

https://codereview.chromium.org/1688183004/diff/1/tools/clang/scripts/update....
tools/clang/scripts/update.py:94: subprocess.check_call(cmd,
stderr=open('/dev/null', 'w'))
What does the error message look like for the user of update.py when gsutil
fails?

[Mostyn Bramley-Moore, 2016-02-12 00:43:33]

https://codereview.chromium.org/1688183004/diff/1/tools/clang/scripts/update.py
File tools/clang/scripts/update.py (right):

https://codereview.chromium.org/1688183004/diff/1/tools/clang/scripts/update....
tools/clang/scripts/update.py:94: subprocess.check_call(cmd,
stderr=open('/dev/null', 'w'))
On 2016/02/12 00:39:11, hans wrote:
> What does the error message look like for the user of update.py when gsutil
> fails?

This is what I get on a throttled/blocked machine, without a ~/.boto config
setup (inside a retry loop):

Downloading gs://chromium-browser-clang/Linux_x64/clang-259396-1.tgz 
Command '['python', '/root/depot_tools/gsutil.py', 'cp',
'gs://chromium-browser-clang/Linux_x64/clang-259396-1.tgz', '/tmp/tmpqRDSzF']'
returned non-zero exit status 1

[commit-bot: I haz the power, 2016-02-12 00:53:10]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-02-12 00:53:11]

Dry run: Try jobs failed on following builders:
  mac_chromium_rel_ng on tryserver.chromium.mac (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_...)

[hans, 2016-02-12 01:42:33]

https://codereview.chromium.org/1688183004/diff/1/tools/clang/scripts/update.py
File tools/clang/scripts/update.py (right):

https://codereview.chromium.org/1688183004/diff/1/tools/clang/scripts/update....
tools/clang/scripts/update.py:94: subprocess.check_call(cmd,
stderr=open('/dev/null', 'w'))
On 2016/02/12 00:43:33, Mostyn Bramley-Moore wrote:
> On 2016/02/12 00:39:11, hans wrote:
> > What does the error message look like for the user of update.py when gsutil
> > fails?
> 
> This is what I get on a throttled/blocked machine, without a ~/.boto config
> setup (inside a retry loop):
> 
> Downloading gs://chromium-browser-clang/Linux_x64/clang-259396-1.tgz 
> Command '['python', '/root/depot_tools/gsutil.py', 'cp',
> 'gs://chromium-browser-clang/Linux_x64/clang-259396-1.tgz', '/tmp/tmpqRDSzF']'
> returned non-zero exit status 1

That's pretty opaque. The current code will tell the user what went wrong.

https://codereview.chromium.org/1688183004/diff/1/tools/clang/scripts/update....
tools/clang/scripts/update.py:94: subprocess.check_call(cmd,
stderr=open('/dev/null', 'w'))
I assume opening /dev/null doesn't work on Windows?

https://codereview.chromium.org/1688183004/diff/1/tools/clang/scripts/update....
tools/clang/scripts/update.py:353: except Exception:
This exception handler is much broader than the previous one.

For example, if there is some error while unpacking the file, that error is now
no longer surfaced to the user.

[Mostyn Bramley-Moore, 2016-02-15 13:29:44]

Dropping this for now since we can't verify if it would help get our build
machines unblocked (it's not clear how we would authenticate without creating
our own google storage bucket).

Instead here's a CL which makes it possible for us to define local mirrors:
https://codereview.chromium.org/1693363002/

https://codereview.chromium.org/1688183004/diff/1/tools/clang/scripts/update.py
File tools/clang/scripts/update.py (right):

https://codereview.chromium.org/1688183004/diff/1/tools/clang/scripts/update....
tools/clang/scripts/update.py:94: subprocess.check_call(cmd,
stderr=open('/dev/null', 'w'))
On 2016/02/12 01:42:33, hans wrote:
> I assume opening /dev/null doesn't work on Windows?

Ahh, right- I copied this pattern from build/download_gold_plugin.py which is
not used on windows.

[Mostyn Bramley-Moore, 2016-02-15 13:29:58]

Description was changed from

==========
use gsutil to download clang et al binaries

BUG=586146
==========

to

==========
use gsutil to download clang et al binaries

BUG=586146
==========