Adding le32-nacl build of pnacl tools.

Adding le32-nacl build of pnacl tools.

Adding command line option to enable pnacl-in-pnacl.
Adding command line option to accept extra arguments to follow cc + cxx.

With the bradnelson/pnacl-in-pnacl branches for bintuils + clang + llvm,
this allows pnacl builds of the toolchains.

Generalize CompilersForHost to select an AR and RANLIB.
Select tools from NACL_SDK_ROOT for le32-nacl,
following existing SDK + naclports convention.

Assume static linking of pnacl plugin with gold.
Add dependency on llvm from binutils when building for le32-nacl.
Explicitly link llvm plugin libraries into gold.

Skip symlinks for le32-nacl build.

Disable shared library builds when building for le32-nacl

BUG=https://code.google.com/p/naclports/issues/detail?id=130
TEST=manual + naclports
R=dschuff@chromium.org

Committed: https://src.chromium.org/viewvc/native_client?view=rev&revision=f8aa9dc1f2789cde7d4c070a87ded730e9d8618e

[Derek Schuff, 2015-01-24 00:15:38]

https://codereview.chromium.org/867403003/diff/20001/toolchain_build/toolchai...
File toolchain_build/toolchain_build_pnacl.py (right):

https://codereview.chromium.org/867403003/diff/20001/toolchain_build/toolchai...
toolchain_build/toolchain_build_pnacl.py:221: extra_cxx_args =
list(extra_cc_args)
is there some particular reason why all the linker flags have to go in cflags
can't go in LDFLAGS and/or LIBS? Is there some particular reason why all this
has to go in the compiler command and not in CFLAGS/CXXFLAGS?

https://codereview.chromium.org/867403003/diff/20001/toolchain_build/toolchai...
toolchain_build/toolchain_build_pnacl.py:527: static_llvm_deps =
['llvm_le32_nacl']
this can be H('llvm')

https://codereview.chromium.org/867403003/diff/20001/toolchain_build/toolchai...
toolchain_build/toolchain_build_pnacl.py:532: '-lLLVMgold', '-lLLVMCodeGen',
'-lLTO', '-lLLVMX86Disassembler',
It occurs to be that we could make our le32 clang a lot smaller by leaving out
some of these backends.

https://codereview.chromium.org/867403003/diff/20001/toolchain_build/toolchai...
toolchain_build/toolchain_build_pnacl.py:707: elif not options.gcc:
how about
elif not options.gcc and host != 'le32-nacl':

https://codereview.chromium.org/867403003/diff/20001/toolchain_build/toolchai...
toolchain_build/toolchain_build_pnacl.py:768: def HostToolsDirectToNacl(host,
options):
is this necessary?

https://codereview.chromium.org/867403003/diff/20001/toolchain_build/toolchai...
toolchain_build/toolchain_build_pnacl.py:1083: hosts.append('le32-nacl')
won't this cause it to be built on the toolchain bot as well? Maybe we need a
flag instead? (and when you build it off the toolchain bots, you don't need to
build it for linux also, do you?

https://codereview.chromium.org/867403003/diff/20001/toolchain_build/toolchai...
toolchain_build/toolchain_build_pnacl.py:1089:
packages.update(HostToolsDirectToNacl(host, args))
is the extra flag necessary?
And, are you actually intending to build nacl-clang for the native architectures
here too?

[bradn, 2015-01-25 07:42:36]

PTAL

Pared this back and moved more of the potentially changing build logic to
naclports.
Once the clang, llvm, binutils branches land, I'll definitely want to start
building on the toolchain waterfall.
I've already hit a case where there's skew between the SDK and the nacl tree
(Jan added a libgcc.a for pnacl).

https://codereview.chromium.org/867403003/diff/20001/toolchain_build/toolchai...
File toolchain_build/toolchain_build_pnacl.py (right):

https://codereview.chromium.org/867403003/diff/20001/toolchain_build/toolchai...
toolchain_build/toolchain_build_pnacl.py:221: extra_cxx_args =
list(extra_cc_args)
On 2015/01/24 00:15:37, Derek Schuff wrote:
> is there some particular reason why all the linker flags have to go in cflags
> can't go in LDFLAGS and/or LIBS? Is there some particular reason why all this
> has to go in the compiler command and not in CFLAGS/CXXFLAGS?

After a good bit of experimentation, it seems the autoconf build mixes and
matches when its willing to include the CFLAGS at in opportune times.
We may be able to do something better once Sam's change to allow main to be the
entry point goes thru, though, as then perhaps just nacl_io + glibc-compat would
be required in extra_cc_args.

I've gone ahead and added an optional parameter to add to this set externally,
that way the build doesn't need to be tied to the state of nacl_io / nacl_spawn.

https://codereview.chromium.org/867403003/diff/20001/toolchain_build/toolchai...
toolchain_build/toolchain_build_pnacl.py:527: static_llvm_deps =
['llvm_le32_nacl']
On 2015/01/24 00:15:37, Derek Schuff wrote:
> this can be H('llvm')

Done.

https://codereview.chromium.org/867403003/diff/20001/toolchain_build/toolchai...
toolchain_build/toolchain_build_pnacl.py:532: '-lLLVMgold', '-lLLVMCodeGen',
'-lLTO', '-lLLVMX86Disassembler',
On 2015/01/24 00:15:37, Derek Schuff wrote:
> It occurs to be that we could make our le32 clang a lot smaller by leaving out
> some of these backends.

Agreed, though I'm torn as I have longer term ambitions of targeting multiple
things. Ideally these might be load on demand .so

https://codereview.chromium.org/867403003/diff/20001/toolchain_build/toolchai...
toolchain_build/toolchain_build_pnacl.py:707: elif not options.gcc:
On 2015/01/24 00:15:37, Derek Schuff wrote:
> how about
> elif not options.gcc and host != 'le32-nacl':

Done.

https://codereview.chromium.org/867403003/diff/20001/toolchain_build/toolchai...
toolchain_build/toolchain_build_pnacl.py:768: def HostToolsDirectToNacl(host,
options):
On 2015/01/24 00:15:37, Derek Schuff wrote:
> is this necessary?

Nope. dropped

https://codereview.chromium.org/867403003/diff/20001/toolchain_build/toolchai...
toolchain_build/toolchain_build_pnacl.py:1083: hosts.append('le32-nacl')
On 2015/01/24 00:15:37, Derek Schuff wrote:
> won't this cause it to be built on the toolchain bot as well? Maybe we need a
> flag instead? (and when you build it off the toolchain bots, you don't need to
> build it for linux also, do you?

Changed to a separate command line option as we discussed.

https://codereview.chromium.org/867403003/diff/20001/toolchain_build/toolchai...
toolchain_build/toolchain_build_pnacl.py:1089:
packages.update(HostToolsDirectToNacl(host, args))
On 2015/01/24 00:15:37, Derek Schuff wrote:
> is the extra flag necessary?
> And, are you actually intending to build nacl-clang for the native
architectures
> here too?

Nope, dropped.

[Derek Schuff, 2015-01-26 17:13:54]

otherwise lgtm

https://codereview.chromium.org/867403003/diff/120001/toolchain_build/toolcha...
File toolchain_build/toolchain_build_pnacl.py (right):

https://codereview.chromium.org/867403003/diff/120001/toolchain_build/toolcha...
toolchain_build/toolchain_build_pnacl.py:182: extra_cc_args +=
[options.extra_cc_args]
should options.extra_cc_args already be a list?

https://codereview.chromium.org/867403003/diff/120001/toolchain_build/toolcha...
toolchain_build/toolchain_build_pnacl.py:604: cleanup_libs = []
maybe call this cleanup_static_libs

[bradn, 2015-01-26 19:32:21]

https://codereview.chromium.org/867403003/diff/120001/toolchain_build/toolcha...
File toolchain_build/toolchain_build_pnacl.py (right):

https://codereview.chromium.org/867403003/diff/120001/toolchain_build/toolcha...
toolchain_build/toolchain_build_pnacl.py:182: extra_cc_args +=
[options.extra_cc_args]
On 2015/01/26 17:13:54, Derek Schuff wrote:
> should options.extra_cc_args already be a list?

So I decided it was easier to treat this one as a string coming in, otherwise
there'd be the issue of spliting it on space and then combining it back
together.
The configure args need to be separate so that they can individually have spaces
for things like: 'LIBS=-lfoo -lbar', which actually needs to be passes in a
separte argv for configure.

https://codereview.chromium.org/867403003/diff/120001/toolchain_build/toolcha...
toolchain_build/toolchain_build_pnacl.py:604: cleanup_libs = []
On 2015/01/26 17:13:54, Derek Schuff wrote:
> maybe call this cleanup_static_libs

Done.

[bradn, 2015-01-26 22:02:15]

The CQ bit was checked by bradnelson@google.com

[commit-bot: I haz the power, 2015-01-26 22:02:40]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/867403003/180001

[bradn, 2015-01-27 00:38:31]

The CQ bit was unchecked by bradnelson@google.com

[bradn, 2015-01-27 00:39:06]

Committed patchset #10 (id:180001) manually as
f8aa9dc1f2789cde7d4c070a87ded730e9d8618e (presubmit successful).