[gn] Consolidate exec_script calls to speed up generation

[gn] Consolidate exec_script calls to speed up generation

Calling out to python from GN to process gypi files is relatively
expensive with a 20-45ms fixed overhead for setup/teardown regardless
of what the script does. This makes runtime/vm/BUILD.gn take 1-1.5s
(per toolchain) to run as the template for libraries expands
out to 25 calls to gypi_to_gn.py, even though the actual time spent
processing the gypi files is negligible.

This replaces those repeated calls to gypi_to_gn.py with a call to a
custom script that process all of the gypi files needed for runtime/vm
and places the results into a single scope which can then be read from
in the template and replaces a few other scattered calls to
gypi_to_gn.py with a smaller number of calls that process multiple
gypi files and place the results into a single scope.

The end result is processing all of dart's GN files in a fuchsia build
takes ~250ms instead of >3 seconds.

R=zra@google.com

Committed: https://github.com/dart-lang/sdk/commit/ad86d6ed26e3b6eeeaa96b654881a49c4b04a561
Committed: https://github.com/dart-lang/sdk/commit/7a43c648c381be8c98b7a0710d239aa7d44100ae

[jamesr, 2016-11-10 21:15:15]

Description was changed from

==========
[gn] Consolidate exec_script calls to speed up generation

This the same approach as https://codereview.chromium.org/2471713003
except it doesn't rely on any GN changes.
==========

to

==========
[gn] Consolidate exec_script calls to speed up generation

Calling out to python from GN to process gypi files is relatively
expensive with a 20-45ms fixed overhead for setup/teardown regardless
of what the script does. This makes runtime/vm/BUILD.gn take 1-1.5s
(per toolchain) to run as the template for libraries expands
out to 25 calls to gypi_to_gn.py, even though the actual time spent
processing the gypi files is negligible.

This replaces those repeated calls to gypi_to_gn.py with a call to a
custom script that process all of the gypi files needed for runtime/vm
and places the results into a single scope which can then be read from
in the template and replaces a few other scattered calls to
gypi_to_gn.py with a smaller number of calls that process multiple
gypi files and place the results into a single scope.

The end result is processing all of dart's GN files in a fuchsia build
takes ~250ms instead of >3 seconds.
==========

[jamesr, 2016-11-10 21:15:26]

jamesr@chromium.org changed reviewers:
+ zra@google.com

[jamesr, 2016-11-10 21:17:26]

I verified this produces the same build by diffing the output of 'gn desc' on
all the labels under //dart/runtime in a fuchsia build with and without this
patch.

https://codereview.chromium.org/2472813002/diff/60001/runtime/vm/BUILD.gn
File runtime/vm/BUILD.gn (right):

https://codereview.chromium.org/2472813002/diff/60001/runtime/vm/BUILD.gn#new...
runtime/vm/BUILD.gn:281: processed_gypis.async_sdk_sources,
i sent a patch to GN to allow indexing into a scope with a string, which would
avoid having to repeat "async" here, but the GN maintainers didn't like that
direction

[zra, 2016-11-10 21:30:41]

Cool, thanks!

lgtm w/ comment and indentation fix.

https://codereview.chromium.org/2472813002/diff/60001/runtime/vm/BUILD.gn
File runtime/vm/BUILD.gn (right):

https://codereview.chromium.org/2472813002/diff/60001/runtime/vm/BUILD.gn#new...
runtime/vm/BUILD.gn:183: # This templates expects invoker.sources to be a list
of pairs of strings.
This comment was already out of date, but maybe now is a good time to update it.

https://codereview.chromium.org/2472813002/diff/60001/runtime/vm/BUILD.gn#new...
runtime/vm/BUILD.gn:281: processed_gypis.async_sdk_sources,
On 2016/11/10 21:17:26, jamesr wrote:
> i sent a patch to GN to allow indexing into a scope with a string, which would
> avoid having to repeat "async" here, but the GN maintainers didn't like that
> direction

Acknowledged.

https://codereview.chromium.org/2472813002/diff/60001/tools/process_gypis.py
File tools/process_gypis.py (right):

https://codereview.chromium.org/2472813002/diff/60001/tools/process_gypis.py#...
tools/process_gypis.py:15: file_string = open(path).read()
For python we follow the chromium style and use 2 space indent.

[jamesr, 2016-11-10 23:10:38]

Description was changed from

==========
[gn] Consolidate exec_script calls to speed up generation

Calling out to python from GN to process gypi files is relatively
expensive with a 20-45ms fixed overhead for setup/teardown regardless
of what the script does. This makes runtime/vm/BUILD.gn take 1-1.5s
(per toolchain) to run as the template for libraries expands
out to 25 calls to gypi_to_gn.py, even though the actual time spent
processing the gypi files is negligible.

This replaces those repeated calls to gypi_to_gn.py with a call to a
custom script that process all of the gypi files needed for runtime/vm
and places the results into a single scope which can then be read from
in the template and replaces a few other scattered calls to
gypi_to_gn.py with a smaller number of calls that process multiple
gypi files and place the results into a single scope.

The end result is processing all of dart's GN files in a fuchsia build
takes ~250ms instead of >3 seconds.
==========

to

==========
[gn] Consolidate exec_script calls to speed up generation

Calling out to python from GN to process gypi files is relatively
expensive with a 20-45ms fixed overhead for setup/teardown regardless
of what the script does. This makes runtime/vm/BUILD.gn take 1-1.5s
(per toolchain) to run as the template for libraries expands
out to 25 calls to gypi_to_gn.py, even though the actual time spent
processing the gypi files is negligible.

This replaces those repeated calls to gypi_to_gn.py with a call to a
custom script that process all of the gypi files needed for runtime/vm
and places the results into a single scope which can then be read from
in the template and replaces a few other scattered calls to
gypi_to_gn.py with a smaller number of calls that process multiple
gypi files and place the results into a single scope.

The end result is processing all of dart's GN files in a fuchsia build
takes ~250ms instead of >3 seconds.

R=zra@google.com

Committed:
https://github.com/dart-lang/sdk/commit/ad86d6ed26e3b6eeeaa96b654881a49c4b04a561
==========

[jamesr, 2016-11-10 23:10:39]

Committed patchset #5 (id:80001) manually as
ad86d6ed26e3b6eeeaa96b654881a49c4b04a561 (presubmit successful).

[jamesr, 2016-11-10 23:17:09]

Seems to redden the bots, revert here:
https://codereview.chromium.org/2492053002

[jamesr, 2016-11-11 00:23:24]

PTAL, fixed applied here (as the diff is small relative to the patch set that
landed).

https://codereview.chromium.org/2472813002/diff/100001/runtime/vm/BUILD.gn
File runtime/vm/BUILD.gn (right):

https://codereview.chromium.org/2472813002/diff/100001/runtime/vm/BUILD.gn#ne...
runtime/vm/BUILD.gn:421: libsources = rebase_path(library[1], ".", library[2])
This fixes the issues seen on bots with the targets generated by the
generate_patched_sdk() template. the file lists need to be rebased to either
../lib or ../bin.  I didn't see this in a fuchsia build as this target isn't
used there (why is that?), but I confirmed that in a standalone mac build with
this patch applied the 'gn desc' output is the same for all targets under
//dart/runtime/vm

[zra, 2016-11-11 03:23:51]

lgtm

https://codereview.chromium.org/2472813002/diff/100001/runtime/vm/BUILD.gn
File runtime/vm/BUILD.gn (right):

https://codereview.chromium.org/2472813002/diff/100001/runtime/vm/BUILD.gn#ne...
runtime/vm/BUILD.gn:421: libsources = rebase_path(library[1], ".", library[2])
On 2016/11/11 00:23:24, jamesr wrote:
> This fixes the issues seen on bots with the targets generated by the
> generate_patched_sdk() template. the file lists need to be rebased to either
> ../lib or ../bin.  I didn't see this in a fuchsia build as this target isn't
> used there (why is that?)

We don't yet do a full Dart SDK build as part of Fuchsia.

[jamesr, 2016-11-12 18:56:49]

Description was changed from

==========
[gn] Consolidate exec_script calls to speed up generation

Calling out to python from GN to process gypi files is relatively
expensive with a 20-45ms fixed overhead for setup/teardown regardless
of what the script does. This makes runtime/vm/BUILD.gn take 1-1.5s
(per toolchain) to run as the template for libraries expands
out to 25 calls to gypi_to_gn.py, even though the actual time spent
processing the gypi files is negligible.

This replaces those repeated calls to gypi_to_gn.py with a call to a
custom script that process all of the gypi files needed for runtime/vm
and places the results into a single scope which can then be read from
in the template and replaces a few other scattered calls to
gypi_to_gn.py with a smaller number of calls that process multiple
gypi files and place the results into a single scope.

The end result is processing all of dart's GN files in a fuchsia build
takes ~250ms instead of >3 seconds.

R=zra@google.com

Committed:
https://github.com/dart-lang/sdk/commit/ad86d6ed26e3b6eeeaa96b654881a49c4b04a561
==========

to

==========
[gn] Consolidate exec_script calls to speed up generation

Calling out to python from GN to process gypi files is relatively
expensive with a 20-45ms fixed overhead for setup/teardown regardless
of what the script does. This makes runtime/vm/BUILD.gn take 1-1.5s
(per toolchain) to run as the template for libraries expands
out to 25 calls to gypi_to_gn.py, even though the actual time spent
processing the gypi files is negligible.

This replaces those repeated calls to gypi_to_gn.py with a call to a
custom script that process all of the gypi files needed for runtime/vm
and places the results into a single scope which can then be read from
in the template and replaces a few other scattered calls to
gypi_to_gn.py with a smaller number of calls that process multiple
gypi files and place the results into a single scope.

The end result is processing all of dart's GN files in a fuchsia build
takes ~250ms instead of >3 seconds.

R=zra@google.com

Committed:
https://github.com/dart-lang/sdk/commit/ad86d6ed26e3b6eeeaa96b654881a49c4b04a561
Committed:
https://github.com/dart-lang/sdk/commit/7a43c648c381be8c98b7a0710d239aa7d44100ae
==========

[jamesr, 2016-11-12 18:56:51]

Committed patchset #6 (id:100001) manually as
7a43c648c381be8c98b7a0710d239aa7d44100ae (presubmit successful).