Work on GN toolchain definitions and build

This adds preliminary support for Android toolchains. The logic is copied from the current GYP build.

Refactors the toolchain definitions by adding an include file to help defining GCC toolchains. This eliminates most of the duplication on Linux and Android. Adds make_global_settings for most Linux and Mac builds that should match the GYP build (this will require some more passes to get every place to be an exact match).

Fixes Android build issues in WebKit. Fixes the definition of is_android.

Renames the Linux toolchains to use the GN name of the CPU architecture so one can refer to them without conditionals. Remove conditionals from BUILDCONFIG.gn

Adds the gyp_header variable to the help (it wasn't getting inserted into the list of available variables).

R=thakis@chromium.org

Committed: https://src.chromium.org/viewvc/chrome?view=rev&revision=242674

[Nico, 2013-12-26 21:57:25]

A bot:
ERROR at //build/toolchain/mac/BUILD.gn:24:6: Undefined identifier
ld = cxx
     ^--
Generating gyp files from GN...


I guess after this is reviewed you'd have to push new binaries before landing
this?

[brettw, 2013-12-26 22:13:01]

On 2013/12/26 21:57:25, Nico wrote:
> A bot:
> ERROR at //build/toolchain/mac/BUILD.gn:24:6: Undefined identifier
> ld = cxx
>      ^--
> Generating gyp files from GN...
> 
> 
> I guess after this is reviewed you'd have to push new binaries before landing
> this?

Nope, this was just an undefined variable because I only defined "cc" and "cxx"
when running under clang. Should be fixed now.

[Nico, 2013-12-26 22:14:33]

Mostly lgtm (decide yourself which comments you'd like to address)

I wonder if it's maybe less work to change android to be less different instead
of adding all the compat scripts with "TODO: remove".


Some general advice about writing python scripts: It's almost always a good idea
to not have any toplevel statements other than classes and functions, and to put
main()-type code into

  def main(args):
    # code here

  if __name__ == '__main__':
    sys.exit(main(sys.argv[1:])

The reason for this is that python has only 2 namespaces: global and
per-function. If your main program is in the global namespace, all variables
used by the main program end up being global functions, and if one of your other
functions happens to access them by accident, it'll just read the global
variable instead of getting an "unknown variable" error at runtime.

(The scripts here are all tiny so it's ok for most of them, but it would've been
a good idea for the script I sent you a CL for not too long ago.)

https://codereview.chromium.org/117863003/diff/170001/build/config/BUILDCONFI...
File build/config/BUILDCONFIG.gn (right):

https://codereview.chromium.org/117863003/diff/170001/build/config/BUILDCONFI...
build/config/BUILDCONFIG.gn:430: host_toolchain =
"//build/toolchain/android:$build_cpu_arch"
Hm, wouldn't the host toolchain be whatever the host os is? (linux on linux, mac
on mac, and windows on windows?)

https://codereview.chromium.org/117863003/diff/170001/build/toolchain/android...
File build/toolchain/android/BUILD.gn (right):

https://codereview.chromium.org/117863003/diff/170001/build/toolchain/android...
build/toolchain/android/BUILD.gn:11:
rebase_path("//third_party/android_tools/ndk", ".", "")
Mostly unrelated: As it happens, I'm currently trying to get android to build on
mac (and eventually, it probably should build on windows too). As part of the
bringup I need to use a different ndk (the checked-in one contains linux
binaries in source control 9_9) – with gn, I'd have to change this in the .gn
file and couldn't set something like a GYP_DEFINE or (with the current
envsetup.sh) an env var, right?

https://codereview.chromium.org/117863003/diff/170001/build/toolchain/android...
File build/toolchain/android/default_goma_for_android.py (right):

https://codereview.chromium.org/117863003/diff/170001/build/toolchain/android...
build/toolchain/android/default_goma_for_android.py:17: print '""'
Why the ""? Is this supposed to be shell-escaped already? Or gn-escaped? (say in
the comment at the top)

https://codereview.chromium.org/117863003/diff/170001/build/toolchain/clang.gni
File build/toolchain/clang.gni (right):

https://codereview.chromium.org/117863003/diff/170001/build/toolchain/clang.g...
build/toolchain/clang.gni:12: # for GYP so must match the directory format of
GYP (rather than using
…for GYP, so it must…

https://codereview.chromium.org/117863003/diff/170001/build/toolchain/gcc_too...
File build/toolchain/gcc_toolchain.gni (right):

https://codereview.chromium.org/117863003/diff/170001/build/toolchain/gcc_too...
build/toolchain/gcc_toolchain.gni:24: # cflags_pch_c
?

https://codereview.chromium.org/117863003/diff/170001/build/toolchain/gcc_too...
build/toolchain/gcc_toolchain.gni:31: # cflags_pch_cc
?

https://codereview.chromium.org/117863003/diff/170001/build/toolchain/gcc_too...
build/toolchain/gcc_toolchain.gni:42: command = "if [ ! -e \$lib -o ! -e
\${lib}.TOC ]; then $ld -shared \$ldflags -o \$lib -Wl,-soname=\$soname
-Wl,--whole-archive \$in \$solibs -Wl,--no-whole-archive \$libs && { readelf -d
\${lib} | grep SONAME ; nm -gD -f p \${lib} | cut -f1-2 -d' '; } > \${lib}.TOC;
else $ld -shared \$ldflags -o \$lib -Wl,-soname=\$soname -Wl,--whole-archive
\$in \$solibs -Wl,--no-whole-archive \$libs && { readelf -d \${lib} | grep
SONAME ; nm -gD -f p \${lib} | cut -f1-2 -d' '; } > \${lib}.tmp && if ! cmp -s
\${lib}.tmp \${lib}.TOC; then mv \${lib}.tmp \${lib}.TOC ; fi; fi"
Does gn have multiline strings, or string concatenation? Can you make this less
unreadable somehow?

https://codereview.chromium.org/117863003/diff/170001/build/toolchain/gcc_too...
build/toolchain/gcc_toolchain.gni:50: #pool = "link_pool"
?

https://codereview.chromium.org/117863003/diff/170001/build/toolchain/goma.gni
File build/toolchain/goma.gni (right):

https://codereview.chromium.org/117863003/diff/170001/build/toolchain/goma.gn...
build/toolchain/goma.gni:11: # completely different logic for configuring Goma.
+1000, this is true for many things in the android build

https://codereview.chromium.org/117863003/diff/170001/build/toolchain/goma.gn...
build/toolchain/goma.gni:16: goma_dir = ""
It defaults to `/bin/echo -n ${HOME}/goma` on non-android. Is that just not here
yet since this is only for android?

https://codereview.chromium.org/117863003/diff/170001/build/toolchain/goma.gn...
build/toolchain/goma.gni:25: # The Android GYP build has some log that
automatically turns on goma when the
…some logic…

https://codereview.chromium.org/117863003/diff/170001/build/toolchain/linux/f...
File build/toolchain/linux/find_android_compilers.py (right):

https://codereview.chromium.org/117863003/diff/170001/build/toolchain/linux/f...
build/toolchain/linux/find_android_compilers.py:30: print '["' + cc[0] + '","' +
cxx[0] + '","' + which_gcc + '","' + which_gxx + '"]'
80 chars

https://codereview.chromium.org/117863003/diff/170001/tools/gn/secondary/thir...
File tools/gn/secondary/third_party/WebKit/Source/wtf/BUILD.gn (right):

https://codereview.chromium.org/117863003/diff/170001/tools/gn/secondary/thir...
tools/gn/secondary/third_party/WebKit/Source/wtf/BUILD.gn:261: ldflags = []
Is this intentional?

[brettw, 2013-12-26 23:01:29]

On 2013/12/26 22:14:33, Nico wrote:
> Mostly lgtm (decide yourself which comments you'd like to address)
> 
> I wonder if it's maybe less work to change android to be less different
instead
> of adding all the compat scripts with "TODO: remove".

I have to exactly match GYP's toolchains in all circumstances or GYP will lose
its mind. And I'm not fixing the GYP Android build now :(


>
https://codereview.chromium.org/117863003/diff/170001/build/config/BUILDCONFI...
> build/config/BUILDCONFIG.gn:430: host_toolchain =
> "//build/toolchain/android:$build_cpu_arch"
> Hm, wouldn't the host toolchain be whatever the host os is? (linux on linux,
mac
> on mac, and windows on windows?)

Yup, thanks!


>
https://codereview.chromium.org/117863003/diff/170001/build/toolchain/android...
> File build/toolchain/android/BUILD.gn (right):
> 
>
https://codereview.chromium.org/117863003/diff/170001/build/toolchain/android...
> build/toolchain/android/BUILD.gn:11:
> rebase_path("//third_party/android_tools/ndk", ".", "")
> Mostly unrelated: As it happens, I'm currently trying to get android to build
on
> mac (and eventually, it probably should build on windows too). As part of the
> bringup I need to use a different ndk (the checked-in one contains linux
> binaries in source control 9_9) – with gn, I'd have to change this in the .gn
> file and couldn't set something like a GYP_DEFINE or (with the current
> envsetup.sh) an env var, right?

It depends. For some settings I have GN parameters and GYP will convert
GYP_DEFINES to the GN params. If you're adding a new one or something I haven't
done yet, you'll have to add this conversion. We need to be sure that the
settings stay in sync.


>
https://codereview.chromium.org/117863003/diff/170001/build/toolchain/android...
> File build/toolchain/android/default_goma_for_android.py (right):
> 
>
https://codereview.chromium.org/117863003/diff/170001/build/toolchain/android...
> build/toolchain/android/default_goma_for_android.py:17: print '""'
> Why the ""? Is this supposed to be shell-escaped already? Or gn-escaped? (say
in
> the comment at the top)

The exec_script call says "value" as the last parameter which interprets the
result of the script as a GN rvalue. So this is returning an empty string.

>
https://codereview.chromium.org/117863003/diff/170001/build/toolchain/gcc_too...
> build/toolchain/gcc_toolchain.gni:31: # cflags_pch_cc
> ?

I haven't done precompiled headers yet. This is just copied from GYP's ninja
files so it's easy to tell what's not done yet.

>
https://codereview.chromium.org/117863003/diff/170001/build/toolchain/gcc_too...
> build/toolchain/gcc_toolchain.gni:42: command = "if [ ! -e \$lib -o ! -e
> \${lib}.TOC ]; then $ld -shared \$ldflags -o \$lib -Wl,-soname=\$soname
> -Wl,--whole-archive \$in \$solibs -Wl,--no-whole-archive \$libs && { readelf
-d
> \${lib} | grep SONAME ; nm -gD -f p \${lib} | cut -f1-2 -d' '; } >
\${lib}.TOC;
> else $ld -shared \$ldflags -o \$lib -Wl,-soname=\$soname -Wl,--whole-archive
> \$in \$solibs -Wl,--no-whole-archive \$libs && { readelf -d \${lib} | grep
> SONAME ; nm -gD -f p \${lib} | cut -f1-2 -d' '; } > \${lib}.tmp && if ! cmp -s
> \${lib}.tmp \${lib}.TOC; then mv \${lib}.tmp \${lib}.TOC ; fi; fi"
> Does gn have multiline strings, or string concatenation? Can you make this
less
> unreadable somehow?

I could use + on the strings. But I thought that made the file overall more
difficult to read while making individual commands only slightly easier to read.

>
https://codereview.chromium.org/117863003/diff/170001/build/toolchain/gcc_too...
> build/toolchain/gcc_toolchain.gni:50: #pool = "link_pool"
> ?

I need to hook up link pools still.


>
https://codereview.chromium.org/117863003/diff/170001/build/toolchain/goma.gn...
> build/toolchain/goma.gni:16: goma_dir = ""
> It defaults to `/bin/echo -n ${HOME}/goma` on non-android. Is that just not
here
> yet since this is only for android?

Good catch. I'll tweak this.

>
https://codereview.chromium.org/117863003/diff/170001/tools/gn/secondary/thir...
> tools/gn/secondary/third_party/WebKit/Source/wtf/BUILD.gn:261: ldflags = []
> Is this intentional?

Yes, otherwise the statement below fails when it tries to append to the list
(which doesn't exist in some cases). Nobody had tested this file on iOS before.

[brettw, 2013-12-28 06:49:39]

Committed patchset #10 manually as r242674 (presubmit successful).