Build and install LLVM gold plugin on Linux.

Build and install LLVM gold plugin on Linux.

Also blacklist /usr/{,local/}{include,lib} instead of /usr to make
things less confusing for people who check out Chrome under /usr.

The plugin is downloaded separately from the rest of the LLVM
package to avoid bloating the package with a large file that
most people don't need right now.

BUG=453195, 463361
R=thakis@chromium.org,hans@chromium.org
CC=dcheng@chromium.org

Committed: https://crrev.com/b1dea52802af355f47822b89fb68eb25fc124f55
Cr-Commit-Position: refs/heads/master@{#319317}

[Nico, 2015-03-03 05:45:07]

The /usr/ change looks like a good idea to me.

https://codereview.chromium.org/970053002/diff/1/tools/clang/scripts/package.sh
File tools/clang/scripts/package.sh (right):

https://codereview.chromium.org/970053002/diff/1/tools/clang/scripts/package....
tools/clang/scripts/package.sh:129: cp "${LLVM_LIB_DIR}/LLVMgold.${SO_EXT}"
$PDIR/lib
How large is this?

[pcc1, 2015-03-03 05:55:37]

https://codereview.chromium.org/970053002/diff/1/tools/clang/scripts/package.sh
File tools/clang/scripts/package.sh (right):

https://codereview.chromium.org/970053002/diff/1/tools/clang/scripts/package....
tools/clang/scripts/package.sh:129: cp "${LLVM_LIB_DIR}/LLVMgold.${SO_EXT}"
$PDIR/lib
On 2015/03/03 05:45:07, Nico wrote:
> How large is this?

32859400 bytes here, or 29884848 stripped.

[Nico, 2015-03-03 06:00:30]

What, that's pretty sizable, that'll increase the size of the tgz by like 100%,
right?

[pcc1, 2015-03-03 06:08:06]

On 2015/03/03 06:00:30, Nico wrote:
> What, that's pretty sizable, that'll increase the size of the tgz by like
100%,
> right?

It increases the size of the tgz by around 11MB (~50%).

Would that be a concern? There are probably things we can do to share the code
between the binaries.

[Nico, 2015-03-03 16:40:55]

On 2015/03/03 06:08:06, pcc1 wrote:
> On 2015/03/03 06:00:30, Nico wrote:
> > What, that's pretty sizable, that'll increase the size of the tgz by like
> 100%,
> > right?
> 
> It increases the size of the tgz by around 11MB (~50%).
> 
> Would that be a concern? There are probably things we can do to share the code
> between the binaries.

This plugin will only be used by you and a single bot for the foreseeable
future, right? If it's not too much trouble to prevent everyone downloading 11
MB they don't need, it's something worth considering.

The plugin needs to be at the same revision as clang, right?
Ideas:
* Make package.sh write the plugin into a separate tgz and download that
separately on demand (relatively easy to do, adds some long-term complexity)
* Since we build clang with symbols (for plugins), could the clang binary itself
be the gold plugin? I'm guessing that almost all of that 11 MB is duplicate LLVM
code, so this would probably get us the plugin for free (from a binary size
point of view)?

[pcc, 2015-03-03 19:02:56]

On 2015/03/03 16:40:55, Nico wrote:
> On 2015/03/03 06:08:06, pcc1 wrote:
> > On 2015/03/03 06:00:30, Nico wrote:
> > > What, that's pretty sizable, that'll increase the size of the tgz by like
> > 100%,
> > > right?
> > 
> > It increases the size of the tgz by around 11MB (~50%).
> > 
> > Would that be a concern? There are probably things we can do to share the
code
> > between the binaries.
> 
> This plugin will only be used by you and a single bot for the foreseeable
> future, right? If it's not too much trouble to prevent everyone downloading 11
> MB they don't need, it's something worth considering.
> 
> The plugin needs to be at the same revision as clang, right?
> Ideas:
> * Make package.sh write the plugin into a separate tgz and download that
> separately on demand (relatively easy to do, adds some long-term complexity)

How do you imagine that would work exactly? I guess the plugin would need to be
downloaded if someone uses use_lto=1. Is that information available at gclient
runhooks time? Even if it is, I guess it wouldn't work if someone uses multiple
configurations with different output_dirs. So we'd need to teach gyp to download
the package somehow?

> * Since we build clang with symbols (for plugins), could the clang binary
itself
> be the gold plugin? I'm guessing that almost all of that 11 MB is duplicate
LLVM
> code, so this would probably get us the plugin for free (from a binary size
> point of view)?

I'm not sure if the clang binary could be the gold plugin, but we might try
moving all the clang code into LLVMgold.so and having the clang binary be a
small stub that calls into that.

While on the subject of reducing file size, I noticed that we are currently
building all the LLVM targets. Do we need any targets other than X86 and ARM?

[pcc, 2015-03-03 21:17:12]

On 2015/03/03 19:02:56, pcc wrote:
> I'm not sure if the clang binary could be the gold plugin, but we might try
> moving all the clang code into LLVMgold.so and having the clang binary be a
> small stub that calls into that.

Well, relinking seemed to help (see patch set 2), we now consume only around
2.7MB more in the tgz.

But I guess this is on hold until we can un-revert the roll.

[Nico, 2015-03-03 21:22:03]

hahaha :-) worth a try, i think. (does this slow down regular compiles?)

[pcc, 2015-03-03 21:45:03]

On 2015/03/03 21:22:03, Nico wrote:
> hahaha :-) worth a try, i think. (does this slow down regular compiles?)

Looks like it does, by around 25%. :(

I'll try a few other things.

[pcc1, 2015-03-03 23:57:07]

On 2015/03/03 21:45:03, pcc wrote:
> On 2015/03/03 21:22:03, Nico wrote:
> > hahaha :-) worth a try, i think. (does this slow down regular compiles?)
> 
> Looks like it does, by around 25%. :(
> 
> I'll try a few other things.

Patch set 3 is a crazy hack that parses the output of "nm" for the binaries that
use LLVMgold.so. I measured a performance hit of around 8%. Maybe I'll try the
on-demand thing? This road is probably getting too complex.

[Nico, 2015-03-04 00:02:04]

> How do you imagine that would work exactly?

There's be some "download-plugin.py" script one would have to manually run, I
suppose. (That's how clang used to work for the longest time too.)

Once / if enough people use that, we can reconsider bundling.

[pcc, 2015-03-04 02:01:52]

On 2015/03/04 00:02:04, Nico wrote:
> > How do you imagine that would work exactly?
> 
> There's be some "download-plugin.py" script one would have to manually run, I
> suppose. (That's how clang used to work for the longest time too.)
> 
> Once / if enough people use that, we can reconsider bundling.

Sounds good. Done, PTAL.

[Nico, 2015-03-04 05:44:39]

lgtm!

https://codereview.chromium.org/970053002/diff/60001/build/download_gold_plug...
File build/download_gold_plugin.py (right):

https://codereview.chromium.org/970053002/diff/60001/build/download_gold_plug...
build/download_gold_plugin.py:6: """Script to download LLVM gold plugin from
google storage."""
nit: should this be in tools/clang/scripts? (if you prefer it here, that's fine
too)

[pcc1, 2015-03-05 19:34:37]

The CQ bit was checked by pcc@chromium.org

[pcc1, 2015-03-05 19:34:37]

The patchset sent to the CQ was uploaded after l-g-t-m from thakis@chromium.org
Link to the patchset: https://codereview.chromium.org/970053002/#ps80001 (title:
"Rebase, drop CLANG_SUB_REVISION change")

[commit-bot: I haz the power, 2015-03-05 19:35:44]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/970053002/80001

[commit-bot: I haz the power, 2015-03-05 20:33:17]

Committed patchset #5 (id:80001)

[commit-bot: I haz the power, 2015-03-05 20:34:25]

Patchset 5 (id:??) landed as
https://crrev.com/b1dea52802af355f47822b89fb68eb25fc124f55
Cr-Commit-Position: refs/heads/master@{#319317}