Clang toolchain: checkout LLD on all platforms but Darwin.

Ship LLD and llvm-ar with Clang toolchain on Linux.

Theoretically, that gives us a chance to switch from Gold plugin
to lld for all modes in Chrome.

Two bundled, potentially risky changes:

- LLD and Gold plugin are built with threads enabled
- LLD and Gold plugin are linked with LLD + ThinLTO instead of FullLTO + Gold plugin.

BUG=607968
Review-Url: https://codereview.chromium.org/2572833003
Cr-Commit-Position: refs/heads/master@{#456453}
Committed: https://chromium.googlesource.com/chromium/src/+/11bb25447c3170340828888298f90ae5ac7de746

[krasin1, 2016-12-13 19:57:34]

krasin@chromium.org changed reviewers:
+ thakis@chromium.org

[krasin1, 2016-12-13 19:58:20]

On 2016/12/13 19:57:34, krasin1 wrote:

FYI: I have tested this test with https://codereview.chromium.org/2576553002/

[krasin1, 2016-12-13 19:59:30]

The CQ bit was checked by krasin@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2016-12-13 20:00:32]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[Nico, 2016-12-13 20:02:23]

https://codereview.chromium.org/2572833003/diff/1/tools/clang/scripts/update.py
File tools/clang/scripts/update.py (left):

https://codereview.chromium.org/2572833003/diff/1/tools/clang/scripts/update....
tools/clang/scripts/update.py:460: if sys.platform == 'win32' or
use_head_revision:
Do we still need use_head_revision here now that we do this on Linux? I guess
the Mac tot bots currently build LLD but they probably shouldn't anyway.
use_head_revision is probably here 'cause you added it for the lld tot bot? (I
don't remember)

[krasin1, 2016-12-13 20:08:19]

https://codereview.chromium.org/2572833003/diff/1/tools/clang/scripts/update.py
File tools/clang/scripts/update.py (left):

https://codereview.chromium.org/2572833003/diff/1/tools/clang/scripts/update....
tools/clang/scripts/update.py:460: if sys.platform == 'win32' or
use_head_revision:
On 2016/12/13 20:02:23, Nico wrote:
> Do we still need use_head_revision here now that we do this on Linux? I guess
> the Mac tot bots currently build LLD but they probably shouldn't anyway.
> use_head_revision is probably here 'cause you added it for the lld tot bot? (I
> don't remember)

It was Peter who set up the LLD ToT bot, but you're right about the intention. I
can invert the condition to sys.platform != 'darwin'. Would that be better?

[Nico, 2016-12-13 20:12:55]

https://codereview.chromium.org/2572833003/diff/1/tools/clang/scripts/update.py
File tools/clang/scripts/update.py (left):

https://codereview.chromium.org/2572833003/diff/1/tools/clang/scripts/update....
tools/clang/scripts/update.py:460: if sys.platform == 'win32' or
use_head_revision:
On 2016/12/13 20:08:19, krasin1 wrote:
> On 2016/12/13 20:02:23, Nico wrote:
> > Do we still need use_head_revision here now that we do this on Linux? I
guess
> > the Mac tot bots currently build LLD but they probably shouldn't anyway.
> > use_head_revision is probably here 'cause you added it for the lld tot bot?
(I
> > don't remember)
> 
> It was Peter who set up the LLD ToT bot, but you're right about the intention.
I
> can invert the condition to sys.platform != 'darwin'. Would that be better?

I mean I think we can remove the use_head_revision bit now. I don't care if we
do `platform == win || platform == linux` or `platform != darwin`, those are
both fine.

(If we do do remove use_head_revision, we should probably have an else: branch
that deletes lld if it's there, to clean up the checkouts on the mac tot bots,
at least for a short while)

[commit-bot: I haz the power, 2016-12-13 21:00:12]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-12-13 21:00:13]

Dry run: This issue passed the CQ dry run.

[krasin1, 2017-02-07 21:36:22]

Description was changed from

==========
Clang toolchain: checkout LLD on Linux unconditionally.

This is a follow up to https://codereview.chromium.org/2569813003/

BUG=607968
==========

to

==========
Clang toolchain: checkout LLD on all platforms but Darwin

This is a follow up to https://codereview.chromium.org/2569813003/

BUG=607968
==========

[krasin1, 2017-02-08 02:33:05]

Description was changed from

==========
Clang toolchain: checkout LLD on all platforms but Darwin

This is a follow up to https://codereview.chromium.org/2569813003/

BUG=607968
==========

to

==========
Ship LLD and llvm-ar with Clang toolchain on Linux.

Theoretically, that gives us a chance to switch from Gold plugin
to lld for all modes in Chrome.

Two bundled, potentially risky changes:

- LLD and Gold plugin are built with threads enabled
- LLD and Gold plugin are linked with LLD + ThinLTO instead of FullLTO + Gold
plugin.

BUG=607968
==========

[krasin1, 2017-02-08 02:36:41]

Hi Nico,

I have extended the CL. Now it has everything we need in order to switch from
Gold plugin to LLD, and therefore to get rid of the requirement to set
GYP_DEFINES='buildtype=Official' when building with LTO to download the plugin.

Please, take a look.

Note: I am going to Hong Kong / Shenzhen for 2 weeks starting this Thursday and
will not have secure means of communication. If we don't cut it before the end
of Wednesday, we'll probably need to wait until I am back (unless someone picks
up the CL)

[krasin1, 2017-02-08 19:58:55]

On 2017/02/08 02:36:41, krasin1 wrote:
> Hi Nico,
> 
> I have extended the CL. Now it has everything we need in order to switch from
> Gold plugin to LLD, and therefore to get rid of the requirement to set
> GYP_DEFINES='buildtype=Official' when building with LTO to download the
plugin.
> 
> Please, take a look.
> 
> Note: I am going to Hong Kong / Shenzhen for 2 weeks starting this Thursday
and
> will not have secure means of communication. If we don't cut it before the end
> of Wednesday, we'll probably need to wait until I am back (unless someone
picks
> up the CL)

Friendly ping. Let me know, if you actually want to postpone landing this CL
until we roll Clang.

[Nico, 2017-02-08 20:02:00]

Risky is bad since the last roll was way too long ago and will already be super
bumpy probably.

The plan is to stop shipping the gold plugin eventually, right?

https://codereview.chromium.org/2572833003/diff/40001/tools/clang/scripts/pac...
File tools/clang/scripts/package.py (right):

https://codereview.chromium.org/2572833003/diff/40001/tools/clang/scripts/pac...
tools/clang/scripts/package.py:225: want.append('bin/llvm-ar')
do we need this? on windows we run `lld-link /lib` to get ar iirc

[krasin1, 2017-02-08 20:36:35]

On 2017/02/08 20:02:00, Nico wrote:
> Risky is bad since the last roll was way too long ago and will already be
super
> bumpy probably.
Sure. Let's postpone submitting this CL until the roll is done.

It might be a good idea to roll Clang, let it settle and then roll the same
version of Clang but with this CL applied (so, with lld and llvm-ar)

> 
> The plan is to stop shipping the gold plugin eventually, right?
Yes. To be specific, the plan is to:

1. Add lld and llvm-ar
2. Move bots to lld and llvm-ar
3. Remove LLVMgold.so from the distribution.

> 
>
https://codereview.chromium.org/2572833003/diff/40001/tools/clang/scripts/pac...
> File tools/clang/scripts/package.py (right):
> 
>
https://codereview.chromium.org/2572833003/diff/40001/tools/clang/scripts/pac...
> tools/clang/scripts/package.py:225: want.append('bin/llvm-ar')
> do we need this? on windows we run `lld-link /lib` to get ar iirc

[krasin1, 2017-02-08 20:36:41]

https://codereview.chromium.org/2572833003/diff/40001/tools/clang/scripts/pac...
File tools/clang/scripts/package.py (right):

https://codereview.chromium.org/2572833003/diff/40001/tools/clang/scripts/pac...
tools/clang/scripts/package.py:225: want.append('bin/llvm-ar')
On 2017/02/08 20:02:00, Nico wrote:
> do we need this? on windows we run `lld-link /lib` to get ar iirc

yes, we do. Right now, when we do an LTO build, ar is running with LLVMgold.so
as a plugin. I have tested that llvm-ar can do the job, but it seems to be the
only way to get rid of LLVMgold.so.

[Nico, 2017-02-08 21:16:23]

I mean, couldn't we run lld with the "run in ar mode" flag like we do on
windows?

On Wed, Feb 8, 2017 at 3:36 PM, <krasin@chromium.org> wrote:

>
> https://codereview.chromium.org/2572833003/diff/40001/
> tools/clang/scripts/package.py
> File tools/clang/scripts/package.py (right):
>
> https://codereview.chromium.org/2572833003/diff/40001/
> tools/clang/scripts/package.py#newcode225
> tools/clang/scripts/package.py:225: want.append('bin/llvm-ar')
> On 2017/02/08 20:02:00, Nico wrote:
> > do we need this? on windows we run `lld-link /lib` to get ar iirc
>
> yes, we do. Right now, when we do an LTO build, ar is running with
> LLVMgold.so as a plugin. I have tested that llvm-ar can do the job, but
> it seems to be the only way to get rid of LLVMgold.so.
>
> https://codereview.chromium.org/2572833003/
>

-- 
You received this message because you are subscribed to the Google Groups
"Chromium-reviews" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to chromium-reviews+unsubscribe@chromium.org.

[krasin1, 2017-02-08 21:18:55]

On 2017/02/08 21:16:23, Nico wrote:
> I mean, couldn't we run lld with the "run in ar mode" flag like we do on
> windows?
Ah!

I didn't get that from your first hint. Thank you for restating it.

I need to check it, but it might very well be true.

> 
> On Wed, Feb 8, 2017 at 3:36 PM, <mailto:krasin@chromium.org> wrote:
> 
> >
> > https://codereview.chromium.org/2572833003/diff/40001/
> > tools/clang/scripts/package.py
> > File tools/clang/scripts/package.py (right):
> >
> > https://codereview.chromium.org/2572833003/diff/40001/
> > tools/clang/scripts/package.py#newcode225
> > tools/clang/scripts/package.py:225: want.append('bin/llvm-ar')
> > On 2017/02/08 20:02:00, Nico wrote:
> > > do we need this? on windows we run `lld-link /lib` to get ar iirc
> >
> > yes, we do. Right now, when we do an LTO build, ar is running with
> > LLVMgold.so as a plugin. I have tested that llvm-ar can do the job, but
> > it seems to be the only way to get rid of LLVMgold.so.
> >
> > https://codereview.chromium.org/2572833003/
> >
> 
> -- 
> You received this message because you are subscribed to the Google Groups
> "Chromium-reviews" group.
> To unsubscribe from this group and stop receiving emails from it, send an
email
> to mailto:chromium-reviews+unsubscribe@chromium.org.

[pcc1, 2017-03-10 21:43:19]

On 2017/02/08 21:18:55, krasin1 wrote:
> On 2017/02/08 21:16:23, Nico wrote:
> > I mean, couldn't we run lld with the "run in ar mode" flag like we do on
> > windows?
> Ah!
> 
> I didn't get that from your first hint. Thank you for restating it.
> 
> I need to check it, but it might very well be true.

The "run in ar mode" flag is in fact a "run in lib.exe mode" flag. lib.exe has a
different command line interface to ar, and does not accept ELF files (because
of https://reviews.llvm.org/D27776). llvm-ar does not have a library interface
so I don't think we can easily reuse it from lld.

If we cared a lot about distribution size I suppose we could busybox a bunch of
LLVM binaries together, and you might recall that last year I sent you a
proposal/patch to do that. It's unclear whether we want to do that at this point
though.

[krasin1, 2017-03-13 17:19:26]

Let's submit the CL as is. Nico, any objections?

[Nico, 2017-03-13 17:36:49]

How big is llvm-ar?

As usual, the bar for putting stuff in the clang bundle is that we expect
~everyone to use it. I can see us using lld by default in all linux builds, but
it'll need someone to shepherd that through (making sure it works, collecting
perf numbers that show that lld is faster than gold in common build configs,
switching, dealing with fallout). Are you signing up for doing that?

(Also, looks like
https://build.chromium.org/p/chromium.fyi/builders/ClangToTLinuxLLD is red at
the moment.)

[krasin1, 2017-03-13 17:52:41]

On 2017/03/13 17:36:49, Nico wrote:
> How big is llvm-ar?
It's about as big as lld or Gold plugin. From what I remember, these binaries
together compress somewhat okay, if put together, but I don't have hard numbers
at hand.

What's the current status for Clang upload trybots? How do I run everything on
this change to see if everything works and what is the size of the resulting
archive?

> 
> As usual, the bar for putting stuff in the clang bundle is that we expect
> ~everyone to use it. I can see us using lld by default in all linux builds,
but
> it'll need someone to shepherd that through (making sure it works, collecting
> perf numbers that show that lld is faster than gold in common build configs,
> switching, dealing with fallout). Are you signing up for doing that?
Yes, for Linux only. Tracking bug is http://crbug.com/607968

But I want to state it in advance that I don't want to deal with
switch-it-all-at-once kind of situation.
My first focus will be to eliminate gold plugin (so, switching CFI / LTO builds,
including the official ones), and
only then to eliminate gold.

Does it sound good to you?

> 
> (Also, looks like
> https://build.chromium.org/p/chromium.fyi/builders/ClangToTLinuxLLD is red at
> the moment.)
There are other failures at the moment and also some bots are (again) offline
for more than two days, so the status is unknown: http://crbug.com/700965

[Nico, 2017-03-13 19:09:04]

On 2017/03/13 17:52:41, krasin1 wrote:
> On 2017/03/13 17:36:49, Nico wrote:
> > How big is llvm-ar?
> It's about as big as lld or Gold plugin. From what I remember, these binaries
> together compress somewhat okay, if put together, but I don't have hard
numbers
> at hand.
> 
> What's the current status for Clang upload trybots? How do I run everything on
> this change to see if everything works and what is the size of the resulting
> archive?
> 
> > 
> > As usual, the bar for putting stuff in the clang bundle is that we expect
> > ~everyone to use it. I can see us using lld by default in all linux builds,
> but
> > it'll need someone to shepherd that through (making sure it works,
collecting
> > perf numbers that show that lld is faster than gold in common build configs,
> > switching, dealing with fallout). Are you signing up for doing that?
> Yes, for Linux only. Tracking bug is http://crbug.com/607968
> 
> But I want to state it in advance that I don't want to deal with
> switch-it-all-at-once kind of situation.
> My first focus will be to eliminate gold plugin (so, switching CFI / LTO
builds,
> including the official ones), and
> only then to eliminate gold.
> 
> Does it sound good to you?

Not doing it all at once sounds fine as long as not too much time passes between
switching LTO and the rest. I don't want LTO to use lld (so that clang rolls now
can break lld) while most bots still use gold (so that it's possible we miss lld
bots on most bots).

If you agree with that, then lgtm.

> 
> > 
> > (Also, looks like
> > https://build.chromium.org/p/chromium.fyi/builders/ClangToTLinuxLLD is red
at
> > the moment.)
> There are other failures at the moment and also some bots are (again) offline
> for more than two days, so the status is unknown: http://crbug.com/700965

[krasin1, 2017-03-13 19:12:49]

On 2017/03/13 19:09:04, Nico wrote:
> On 2017/03/13 17:52:41, krasin1 wrote:
> > On 2017/03/13 17:36:49, Nico wrote:
> > > How big is llvm-ar?
> > It's about as big as lld or Gold plugin. From what I remember, these
binaries
> > together compress somewhat okay, if put together, but I don't have hard
> numbers
> > at hand.
> > 
> > What's the current status for Clang upload trybots? How do I run everything
on
> > this change to see if everything works and what is the size of the resulting
> > archive?
> > 
> > > 
> > > As usual, the bar for putting stuff in the clang bundle is that we expect
> > > ~everyone to use it. I can see us using lld by default in all linux
builds,
> > but
> > > it'll need someone to shepherd that through (making sure it works,
> collecting
> > > perf numbers that show that lld is faster than gold in common build
configs,
> > > switching, dealing with fallout). Are you signing up for doing that?
> > Yes, for Linux only. Tracking bug is http://crbug.com/607968
> > 
> > But I want to state it in advance that I don't want to deal with
> > switch-it-all-at-once kind of situation.
> > My first focus will be to eliminate gold plugin (so, switching CFI / LTO
> builds,
> > including the official ones), and
> > only then to eliminate gold.
> > 
> > Does it sound good to you?
> 
> Not doing it all at once sounds fine as long as not too much time passes
between
> switching LTO and the rest. I don't want LTO to use lld (so that clang rolls
now
> can break lld) while most bots still use gold (so that it's possible we miss
lld
> bots on most bots).
> 
> If you agree with that, then lgtm.
It's in my interests to make LTO / CFI bots as close to other bots as possible,
as it would reduce our maintenance pains. So, absolutely - we want to switch to
lld all the bots.
That will also let us to remove some parts of third_party/binutils build, which
is nice.

> 
> > 
> > > 
> > > (Also, looks like
> > > https://build.chromium.org/p/chromium.fyi/builders/ClangToTLinuxLLD is red
> at
> > > the moment.)
> > There are other failures at the moment and also some bots are (again)
offline
> > for more than two days, so the status is unknown: http://crbug.com/700965

[krasin1, 2017-03-13 19:12:54]

The CQ bit was checked by krasin@chromium.org

[commit-bot: I haz the power, 2017-03-13 19:13:44]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-03-13 20:12:05]

CQ is committing da patch.

Bot data: {"patchset_id": 60001, "attempt_start_ts": 1489432374768400,
"parent_rev": "8a995f1eb09b00382fc071d347d59cc87c2544b2", "commit_rev":
"11bb25447c3170340828888298f90ae5ac7de746"}

[commit-bot: I haz the power, 2017-03-13 20:12:50]

Description was changed from

==========
Ship LLD and llvm-ar with Clang toolchain on Linux.

Theoretically, that gives us a chance to switch from Gold plugin
to lld for all modes in Chrome.

Two bundled, potentially risky changes:

- LLD and Gold plugin are built with threads enabled
- LLD and Gold plugin are linked with LLD + ThinLTO instead of FullLTO + Gold
plugin.

BUG=607968
==========

to

==========
Ship LLD and llvm-ar with Clang toolchain on Linux.

Theoretically, that gives us a chance to switch from Gold plugin
to lld for all modes in Chrome.

Two bundled, potentially risky changes:

- LLD and Gold plugin are built with threads enabled
- LLD and Gold plugin are linked with LLD + ThinLTO instead of FullLTO + Gold
plugin.

BUG=607968

Review-Url: https://codereview.chromium.org/2572833003
Cr-Commit-Position: refs/heads/master@{#456453}
Committed:
https://chromium.googlesource.com/chromium/src/+/11bb25447c3170340828888298f9...
==========

[commit-bot: I haz the power, 2017-03-13 20:12:52]

Committed patchset #4 (id:60001) as
https://chromium.googlesource.com/chromium/src/+/11bb25447c3170340828888298f9...

[Reid Kleckner, 2017-03-14 00:28:57]

On 2017/03/10 21:43:19, pcc1 wrote:
> If we cared a lot about distribution size I suppose we could busybox a bunch
of
> LLVM binaries together, and you might recall that last year I sent you a
> proposal/patch to do that. It's unclear whether we want to do that at this
point
> though.

I'd rather improve support for -DLLVM_ENABLE_DYLIB=ON or whatever it's called
and get libLLVM.so and LLVM.dll working. I know there are a lot of challenges
there around performance and annotation of library interfaces, but it would
align much better with what packagers want.

[Nico, 2017-03-14 00:45:41]

The annotations are only needed on Windows. pcc tried libllvm.so a year ago
or so and it regressed build time by like 30%, so that's probably not an
option.

On Mar 13, 2017 8:28 PM, <rnk@chromium.org> wrote:

> On 2017/03/10 21:43:19, pcc1 wrote:
> > If we cared a lot about distribution size I suppose we could busybox a
> bunch
> of
> > LLVM binaries together, and you might recall that last year I sent you a
> > proposal/patch to do that. It's unclear whether we want to do that at
> this
> point
> > though.
>
> I'd rather improve support for -DLLVM_ENABLE_DYLIB=ON or whatever it's
> called
> and get libLLVM.so and LLVM.dll working. I know there are a lot of
> challenges
> there around performance and annotation of library interfaces, but it would
> align much better with what packagers want.
>
> https://codereview.chromium.org/2572833003/
>

-- 
You received this message because you are subscribed to the Google Groups
"Chromium-reviews" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to chromium-reviews+unsubscribe@chromium.org.