First pass on blink compiling for GN build

First pass on blink compiling for GN build

This adds most of the blink bindings and code to the GN build. It runs but I have only spot-checked that
the outputs are identical to the GYP ones.

Moved the feature defines to config.gni. Added a string version of these defines (GN doesn't
have a nice way to do this in the format the scripts expect, so I also added a python script).
This is used in various actions.

Provide GN equivalents for the core and bindings gypis.

The bindings BUILD.gn file is a bit icky with rebase_path all over the place and it's difficult to
follow where the names are relative to. We read the file lists from gypis and the source isn't
really the way GN would prefer. I'm planning to add a function to GN to convert a path to a
source-absolute path which will clean up some of these calls (in a later pass).

R=nbarth@chromium.org, thakis@chromium.org

Committed: https://src.chromium.org/viewvc/blink?view=rev&revision=174920

[brettw, 2014-05-22 23:09:00]

New patch fixes the "always rebuilds" dependency issue.

[esprehn, 2014-05-22 23:59:02]

It'd be nice if you gave the patch a title that was a bit more specific about
what it made "work". :)

[brettw, 2014-05-23 00:00:33]

Clarified patch to be GN-specific

[eseidel, 2014-05-23 00:00:41]

https://codereview.chromium.org/298703005/diff/100001/Source/bindings/binding...
File Source/bindings/bindings.gni (right):

https://codereview.chromium.org/298703005/diff/100001/Source/bindings/binding...
Source/bindings/bindings.gni:32: bindings_core_generated_aggregate_files = [
I guess GN hasn't learned auto-sharding yet? :)

[brettw, 2014-05-23 21:13:07]

New snap up with modules and html/dom/rendering/css compiling.

[Nico, 2014-05-25 15:07:22]

lgtm as far as I can tell, but I didn't compare all gn and gyp files side by
side.

I'd like nbarth to take a look at the bindings stuff, as that's tricky and he
knows it well.

https://codereview.chromium.org/298703005/diff/120001/Source/BUILD.gn
File Source/BUILD.gn (right):

https://codereview.chromium.org/298703005/diff/120001/Source/BUILD.gn#newcode74
Source/BUILD.gn:74: config("npapi_stub") {
third_party/WebKit/Source/BUILD.gn seems like a strange place for this.

https://codereview.chromium.org/298703005/diff/120001/Source/bindings/BUILD.gn
File Source/bindings/BUILD.gn (right):

https://codereview.chromium.org/298703005/diff/120001/Source/bindings/BUILD.g...
Source/bindings/BUILD.gn:17: write_file(core_idl_files_list,
Does write_file() write only if the contents of the file change? (help doesn't
say, but seems likely?)

https://codereview.chromium.org/298703005/diff/120001/Source/bindings/BUILD.g...
Source/bindings/BUILD.gn:39: # *is* known at runtime, but we currently mimic the
GYP build.
Hm, I thought part of it is also because of other reasons. +nbarth to chime in.

https://codereview.chromium.org/298703005/diff/120001/Source/bindings/BUILD.g...
Source/bindings/BUILD.gn:40: static_interface_idl_files =
(`gn help static` finds nothing; i assume it has the "local to this file"
meaning)

https://codereview.chromium.org/298703005/diff/120001/Source/bindings/BUILD.g...
Source/bindings/BUILD.gn:113: ]
(grit has an --inputs feature where it returns a list of all .py files that make
up grit, and it collects this using some python reflection api. But this matches
what gyp does for bindings, so it's fine as is too.)

https://codereview.chromium.org/298703005/diff/120001/Source/build/gn_list_to...
File Source/build/gn_list_to_space_separated_string.py (right):

https://codereview.chromium.org/298703005/diff/120001/Source/build/gn_list_to...
Source/build/gn_list_to_space_separated_string.py:12: print result
print ' '.join(['"%s"' % s for s in sys.argv[1:]])

https://codereview.chromium.org/298703005/diff/120001/Source/build/scripts/sc...
File Source/build/scripts/scripts.gni (right):

https://codereview.chromium.org/298703005/diff/120001/Source/build/scripts/sc...
Source/build/scripts/scripts.gni:72: # use /usr/bin/clang once we require Xcode
4.x.
(this comment is outdated, "/usr/bin/gcc" is now just another way to say "clang"
on OS X. Not that it matters much.)

https://codereview.chromium.org/298703005/diff/120001/Source/core/BUILD.gn
File Source/core/BUILD.gn (right):

https://codereview.chromium.org/298703005/diff/120001/Source/core/BUILD.gn#ne...
Source/core/BUILD.gn:17: #  - optimize: max (check what this does on Android
before using).
(i think optimize:max is a windows-only thing)

https://codereview.chromium.org/298703005/diff/120001/Source/core/BUILD.gn#ne...
Source/core/BUILD.gn:109: # direct_dependent_configs private and only forward
some of them.
hmmm…

https://codereview.chromium.org/298703005/diff/120001/Source/core/BUILD.gn#ne...
Source/core/BUILD.gn:164: # Shard this taret into parts to work around linker
limitations.
typo taret

https://codereview.chromium.org/298703005/diff/120001/Source/core/BUILD.gn#ne...
Source/core/BUILD.gn:209: "rendering/RenderThemeChromiumFontProviderWin.cpp",
Hm, don't filename patterns do this?

https://codereview.chromium.org/298703005/diff/120001/Source/core/BUILD.gn#ne...
Source/core/BUILD.gn:220: # uninitialized variable.
We no longer use gcc 4.6 on android, maybe change to "we got" and "TODO: try
removing, now that we're on 4.8"

https://codereview.chromium.org/298703005/diff/120001/Source/core/BUILD.gn#ne...
Source/core/BUILD.gn:244: 
empty if

https://codereview.chromium.org/298703005/diff/120001/Source/core/BUILD.gn#ne...
Source/core/BUILD.gn:426: # generated_testing_idls
---------------------------------------------------------------
80 cols

[brettw, 2014-05-25 17:35:34]

A few responses (no new patch yet)...

https://codereview.chromium.org/298703005/diff/120001/Source/BUILD.gn
File Source/BUILD.gn (right):

https://codereview.chromium.org/298703005/diff/120001/Source/BUILD.gn#newcode74
Source/BUILD.gn:74: config("npapi_stub") {
On 2014/05/25 15:07:22, Nico wrote:
> third_party/WebKit/Source/BUILD.gn seems like a strange place for this.

I was just putting it in the root for WEbKit since that's what this stub is for.
I actually have NPAPI converted in another patch so this will be very short
lived.

https://codereview.chromium.org/298703005/diff/120001/Source/bindings/BUILD.gn
File Source/bindings/BUILD.gn (right):

https://codereview.chromium.org/298703005/diff/120001/Source/bindings/BUILD.g...
Source/bindings/BUILD.gn:17: write_file(core_idl_files_list,
On 2014/05/25 15:07:22, Nico wrote:
> Does write_file() write only if the contents of the file change? (help doesn't
> say, but seems likely?)

No, interesting catch! I was getting unnecessary rebuilds whenever I changed a
BUILD file, and this seems like the likely cause. I'll make a note to add this
feature.

https://codereview.chromium.org/298703005/diff/120001/Source/bindings/BUILD.g...
Source/bindings/BUILD.gn:40: static_interface_idl_files =
On 2014/05/25 15:07:22, Nico wrote:
> (`gn help static` finds nothing; i assume it has the "local to this file"
> meaning)

No, the name static is copied from GYP (these are the "static interfaces" I
guess?). Vars can't get out of a BUILD file so everything is static in that way.

https://codereview.chromium.org/298703005/diff/120001/Source/bindings/BUILD.g...
Source/bindings/BUILD.gn:113: ]
On 2014/05/25 15:07:22, Nico wrote:
> (grit has an --inputs feature where it returns a list of all .py files that
make
> up grit, and it collects this using some python reflection api. But this
matches
> what gyp does for bindings, so it's fine as is too.)

The right way to do this is to have it write a .d file so I don't have to spawn
all of these processes during GN time. When the build is complete, I believe the
vast majority of the time will be spent in grit.

https://codereview.chromium.org/298703005/diff/120001/Source/core/BUILD.gn
File Source/core/BUILD.gn (right):

https://codereview.chromium.org/298703005/diff/120001/Source/core/BUILD.gn#ne...
Source/core/BUILD.gn:209: "rendering/RenderThemeChromiumFontProviderWin.cpp",
On 2014/05/25 15:07:22, Nico wrote:
> Hm, don't filename patterns do this?

The patters only have the Chromium-style _win.cc. When there are only a handful
of files of a type I've not been adding new patterns since this has a
multiplicative effect on the number of checks for each source file.

[Nils Barth (inactive), 2014-05-26 08:58:36]

On 2014/05/25 15:07:22, Nico wrote:
> I'd like nbarth to take a look at the bindings stuff, as that's tricky and he
> knows it well.

Hi Brett,
An FYI: I'm currently making lots of changes to the bindings GYP, as part of
componentizing
core vs. modules. E.g.:
https://codereview.chromium.org/295403002/
https://codereview.chromium.org/300133002/
https://codereview.chromium.org/302433005/

I should be mostly finished by Wednesday, so you might want to hold off until
then.
Alternatively, I can sync GYP to GN if you'd prefer (and will likely need to do
so for minor changes
next week).

Will take a closer look at bindings.gn and comment in detail.

[Nils Barth (inactive), 2014-05-26 09:14:01]

AFAICT bindings LGTM.

Main ugliness is the frequent use of rebase_path;
the path handling is a pervasive part of GYP,
and looks like GN either doesn't have it or wants to make
it more explicit?
(Any more details or thoughts?)

Other than this, the GN seems clear and rather more legible and explicit than
GYP!

https://codereview.chromium.org/298703005/diff/120001/Source/bindings/BUILD.gn
File Source/bindings/BUILD.gn (right):

https://codereview.chromium.org/298703005/diff/120001/Source/bindings/BUILD.g...
Source/bindings/BUILD.gn:17: write_file(core_idl_files_list,
On 2014/05/25 17:35:35, brettw wrote:
> On 2014/05/25 15:07:22, Nico wrote:
> > Does write_file() write only if the contents of the file change? (help
doesn't
> > say, but seems likely?)
> 
> No, interesting catch! I was getting unnecessary rebuilds whenever I changed a
> BUILD file, and this seems like the likely cause. I'll make a note to add this
> feature.

Yup, unnecessary rebuilds are a big issue in bindings, and the main solution is
to not rewrite generated files (build-time or compile-time) if there are no
changes -- it's a key feature for us!

Currently we have to support this explicitly for compile-time rewriting
(b/c varies by build system: make doesn't like it),
but with GN I assume we can get rid of this!

https://codereview.chromium.org/298703005/diff/120001/Source/bindings/BUILD.g...
Source/bindings/BUILD.gn:39: # *is* known at runtime, but we currently mimic the
GYP build.
On 2014/05/25 15:07:22, Nico wrote:
> Hm, I thought part of it is also because of other reasons. +nbarth to chime
in.

AFAIK the *only* reason for passing at the command line is b/c the build dir
is not known at GYP time: otherwise we would know the path and could just list
it in the generated file.
(Technically GYP doesn't allow expansion of the build dir during variable
expansion,
but that's b/c of this assumption.)

BTW Brett, could you add FIXMEs for any GYP-isms that can be removed later?
Thanks!

https://codereview.chromium.org/298703005/diff/120001/Source/bindings/BUILD.g...
Source/bindings/BUILD.gn:174: "--write-file-only-if-changed=1",  # Always true
for Ninja.
Could you add a FIXME to remove (hardcode to True) once we switch to GN?

https://codereview.chromium.org/298703005/diff/120001/Source/bindings/BUILD.g...
Source/bindings/BUILD.gn:177: # Generated files must be passed at command line
Well, they *are* passed at the command line, but this isn't needed in GN
(so could we have a FIXME?)

https://codereview.chromium.org/298703005/diff/120001/Source/bindings/BUILD.g...
Source/bindings/BUILD.gn:219: # TODO(brettw) GYP adds a "-S before the script
name to skip "import site" to
This isn't a big deal, but it's a small performance gain.
Allowing command line options is probably necessary in other uses as well, no?

Something like interpreter_opts/interpreter_args ?

[brettw, 2014-05-26 17:08:35]

Background on rebase_path usage:

GN expects paths to be relative to the current buildfile or
source-repository-absolute ("//third_party/foo/bar"). This normally works fine.

In this case we're reading .gypis with paths relative to different directories
than the build files, which causes the problem. Assuming a pure-GN world, we
wouldn't have these and would put the file lists and build rules in the same
directory (or maybe even file), or list the paths we need to share as absolute,
and all of the rebase_paths go away (except in the args of actions).

Shorter term I plan on adding a function to convert a path to a source-absolute
one which will get rid of some of these calls, since the .gni file that imports
the gypi can do that, and all other files can treat the files like normal.

[brettw, 2014-05-26 17:58:02]

I think what would be easiest for me is if we landed this without taking
nbarth's refactoring into account. These build files aren't actually hooked up
to the build yet (pending getting a proper WebKit GN buildbot), so this
shouldn't cause any delays in landing refactorings. It sounds like the
refactorings and the new buildbots will both be done in a few days, at which
point we can update the GN build files and hook them up tot he build.

[Nils Barth (inactive), 2014-05-27 01:23:32]

On 2014/05/26 17:08:35, brettw wrote:
> Background on rebase_path usage:
<snip>

Thanks for the background!
So current path handling is essentially "act like GYP", but once switched to GN
we can refactor build to eliminate these; we can do any cleanup then.

> Shorter term I plan on adding a function to convert a path to a
source-absolute
> one which will get rid of some of these calls, since the .gni file that
imports
> the gypi can do that, and all other files can treat the files like normal.

Got it, sounds good!

[Nils Barth (inactive), 2014-05-27 01:24:11]

On 2014/05/26 17:58:02, brettw wrote:
> I think what would be easiest for me is if we landed this without taking
> nbarth's refactoring into account. These build files aren't actually hooked up
> to the build yet (pending getting a proper WebKit GN buildbot), so this
> shouldn't cause any delays in landing refactorings. It sounds like the
> refactorings and the new buildbots will both be done in a few days, at which
> point we can update the GN build files and hook them up tot he build.

Sounds good, thanks!

[brettw, 2014-05-27 23:42:18]

The CQ bit was checked by brettw@chromium.org

[brettw, 2014-05-27 23:42:20]

The CQ bit was unchecked by brettw@chromium.org

[brettw, 2014-05-27 23:53:03]

https://codereview.chromium.org/298703005/diff/120001/Source/bindings/BUILD.gn
File Source/bindings/BUILD.gn (right):

https://codereview.chromium.org/298703005/diff/120001/Source/bindings/BUILD.g...
Source/bindings/BUILD.gn:219: # TODO(brettw) GYP adds a "-S before the script
name to skip "import site" to
On 2014/05/26 09:14:01, Nils Barth wrote:
> This isn't a big deal, but it's a small performance gain.
> Allowing command line options is probably necessary in other uses as well, no?
> 
> Something like interpreter_opts/interpreter_args ?

I've never encountered another place where the command line option needs to be
passed to Python itself and not the script.

[brettw, 2014-05-27 23:53:08]

The CQ bit was checked by brettw@chromium.org

[commit-bot: I haz the power, 2014-05-27 23:53:22]

CQ is trying da patch. Follow status at
 https://chromium-status.appspot.com/cq/brettw@chromium.org/298703005/160001

[Nils Barth (inactive), 2014-05-28 00:52:36]

https://codereview.chromium.org/298703005/diff/120001/Source/bindings/BUILD.gn
File Source/bindings/BUILD.gn (right):

https://codereview.chromium.org/298703005/diff/120001/Source/bindings/BUILD.g...
Source/bindings/BUILD.gn:219: # TODO(brettw) GYP adds a "-S before the script
name to skip "import site" to
On 2014/05/27 23:53:04, brettw wrote:
> I've never encountered another place where the command line option needs to be
> passed to Python itself and not the script.

Other relatively common ones (as these things go) are -O and -OO for
optimization
(these don't work here, but otherwise I'd be using them as well).

However, understood that these are not a common use case:
we're only going to these lengths b/c this is on the critical path of build
time,
so tweaks are worth it.

So: "nice to have, but not deal-breaker".

[commit-bot: I haz the power, 2014-05-28 01:19:52]

FYI, CQ is re-trying this CL (attempt #1).
The failing builders are:
  blink_presubmit on tryserver.blink
(http://build.chromium.org/p/tryserver.blink/builders/blink_presubmit/builds/6298)
  linux_gpu_triggered_tests on tryserver.chromium.gpu
(http://build.chromium.org/p/tryserver.chromium.gpu/builders/linux_gpu_trigger...)

[commit-bot: I haz the power, 2014-05-28 01:20:32]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2014-05-28 01:20:32]

Try jobs failed on following builders:
  linux_gpu_triggered_tests on tryserver.chromium.gpu
(http://build.chromium.org/p/tryserver.chromium.gpu/builders/linux_gpu_trigger...)

[brettw, 2014-05-28 01:58:29]

Committed patchset #9 manually as r174920.