Scripts for unity/jumbo (default disabled) compilation

build/config/jumbo.gni

Index: build/config/jumbo.gni
diff --git a/build/config/jumbo.gni b/build/config/jumbo.gni
new file mode 100644
index 0000000000000000000000000000000000000000..3f0ded734598c3c462645476d42fea1999f71945
--- /dev/null
+++ b/build/config/jumbo.gni
@@ -0,0 +1,143 @@
+# Copyright 2017 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+import("//build/split_static_library.gni")  # When someone uses that target_type
+
+declare_args() {
+  # If true, use a jumbo build (files compiled together) to speed up
+  # compilation.

[Dirk Pranke, 2017/06/30 21:38:46]

We should document in more details somewhere what we mean by a "jumbo build" /
"files compiled together". I think the right thing to do would probably be to
link to an in-repo markdown page that talks about what this is, why we do it,
how to use this stuff, and common gotchas. I don't think we need to block this
CL on that page being written, though; we could add this in a follow-up.

[Daniel Bratell, 2017/07/03 09:21:17]

I will move some text from the current Google Doc to a markdown document in
another CL.

+  use_jumbo_build = false
+
+  # A target to exclude from jumbo builds, for optimal round trip time
+  # when frequently changing a single cpp file.
+  jumbo_build_excluded = ""
+
+  # How many files to group at most. Smaller numbers give more
+  # parallellism, higher numbers give less total CPU usage. Higher
+  # numbers also give longer single-file recompilation times.
+  #
+  # Recommendations:
+  # Higher numbers than 200 does not reduce wall clock compile times
+  # for 4 cores or less.
+  # 200 uses 8% less total CPU than 100 when compiling content and 10%
+  # less wall clock when compiling with 4 cores.

[brucedawson, 2017/06/30 18:49:15]

I would expect that we would want to tune this for eight-core or twelve-core
machines rather than four-core. But, I don't want to block this CL on that issue
because I know it takes a lot of testing to measure different values.

[Dirk Pranke, 2017/06/30 21:38:46]

I feel similarly :). 200 is fine for now.

[Daniel Bratell, 2017/07/03 09:21:17]

Looking forward to numbers from other computers. 4 cores is still the highest
number of cores on consumer processors. One good thing with high numbers is that
you can divide them down so if 200 works, 100, 50 and 25 probably compile as
well. 400 on the other hand may not.

I ran some experiments with 100 and I think 100 might be close enough to 200
that we could shift down to 100 with only 5-10 minutes longer full builds, but
I'm leaving the final tuning for post-landing.

+  jumbo_file_merge_limit = 200
+}
+
+# Use this to generate a target which merges sources if possible to
+# compile much faster.
+#
+# Special values.
+#
+#   target_type
+#      The kind of target to build. For example the string
+#      "static_library".

[Dirk Pranke, 2017/06/30 21:38:46]

Given the set_defaults() call on line 138, this means that this currently only
works for static_libraries and source_sets (and things like the component
template that reduce to that)? We should probably add an assert for that, or at
least add asserts that they're *not* passing in target_types that are known not
to work.

I'd like to eventually make this work for executables, shared_libraries, and the
other types of targets, but I think you probably would have to change the
approach a bit, maybe to define a possibly_jumbo version of each target type or
something (this is also part of the reason why I think this ultimately should be
built into GN).

[Daniel Bratell, 2017/07/03 09:21:17]

I have considered making one jumbo_source_set, one jumbo_static_library, one
jumbo_component, one jumbo_split_static_library and so on but since it would
make jumbo.gni a bit more verbose so I delayed it until later. Right now it
seems to work for everything I've tried, but I might not have tried executables.

+#
+#   always_build_jumbo
+#      If set and set to true, then use jumbo compile even when it is
+#      globally disabled. Otherwise it has no effect

[Dirk Pranke, 2017/06/30 21:38:46]

Nit: add periods after "no effect", here and on line 43.

[Daniel Bratell, 2017/07/03 09:21:17]

Done.

+#
+#   never_build_jumbo
+#      If set and set to true, then do not jumbo compile even if it is
+#      globally enabled. Otherwise it has no effect
+#
+#   jumbo_excluded_sources
+#      If set to a list of files, those files will not be merged with
+#      the rest. Can be used if files are problematic.

[Dirk Pranke, 2017/06/30 21:38:45]

s/Can be used if files are problematic/This can be necessary if merging the
files causes compilation issues and fixing the issues is impractical./

[Daniel Bratell, 2017/07/03 09:21:17]

Done.

+template("jumbo_target") {
+  use_jumbo_build_for_target = use_jumbo_build
+  if (defined(invoker.always_build_jumbo) && invoker.always_build_jumbo) {
+    use_jumbo_build_for_target = true
+  }
+  if (defined(invoker.never_build_jumbo) && invoker.never_build_jumbo) {
+    use_jumbo_build_for_target = false
+  }
+  if (target_name == jumbo_build_excluded) {
+    use_jumbo_build_for_target = false
+  }
+
+  excluded_sources = []
+  if (defined(invoker.jumbo_excluded_sources)) {
+    excluded_sources += invoker.jumbo_excluded_sources
+  }
+
+  invoker_sources = invoker.sources
+  gen_target_dir = get_path_info(invoker_sources[0], "gen_dir")
+  assert(excluded_sources != [] || true)  # Prevent "unused variable".
+  assert(gen_target_dir != "")  # Prevent "unused variable".
+
+  if (use_jumbo_build_for_target) {
+    jumbo_files = []
+
+    # Split the sources list into chunks that are not excessively large
+    files_per_chunk = jumbo_file_merge_limit
+    current_file_index = 0
+    next_chunk_start = 0
+    next_chunk_number = 1
+    foreach(source_file, invoker.sources) {
+      if (current_file_index == next_chunk_start) {
+        jumbo_files += [ "$gen_target_dir/" + target_name + "_jumbo_" +
+                         next_chunk_number + ".cc" ]
+        next_chunk_number += 1
+        next_chunk_start += files_per_chunk
+      }
+      current_file_index += 1
+    }
+
+    # Create an action that calls a script that merges all the source files.
+    action(target_name + "_jumbo") {

[Dirk Pranke, 2017/06/30 21:38:45]

I'd pull target_name + "_jumbo" into a variable so that you can reuse it here
and on line 119. I might also call it "__jumbo_merge" or
"__merge_for_jumbo_build" or something like that. We usually use two underscores
for helper targets in templates, and I don't think being particularly verbose is
too awful here.

[Daniel Bratell, 2017/07/03 09:21:17]

Done.

+      script = "//build/config/jumbo.py"
+      response_file_contents =
+          rebase_path(invoker.sources - excluded_sources, gen_target_dir)
+      outputs = jumbo_files
+      args = [ "--outputs" ] + rebase_path(outputs, root_build_dir) +
+             [ "--file-list={{response_file_name}}" ]

[Dirk Pranke, 2017/06/30 21:38:46]

I think you need to set inputs = invoker.sources - excluded_sources here, to
know when we need to re-merge?

[Daniel Bratell, 2017/07/03 09:21:17]

Not for the #include method since the contents of the files actually doesn't
matter. It would be needed if the files were physically merged into large files.
That is slower and more complicated but has a few benefits so we might end up
there eventually. For now I'm choosing the least complicated method until we
know that we need something more complicated. 

+    }
+  }
+
+  target_type = invoker.target_type
+  if (use_jumbo_build_for_target && target_type == "split_static_library") {
+    # Meaningless and also impossible if split_count > len(jumbo_files)
+    target_type = "static_library"
+
+    # Prevent "unused variable" warning.
+    assert(!defined(invoker.split_count) || invoker.split_count > 0)
+  }
+
+  # Perform the actual operation, either on the original sources or
+  # the sources post-jumbo merging.
+  target(target_type, target_name) {
+    deps = []
+    if (defined(invoker.deps)) {
+      deps += invoker.deps
+    }
+
+    # Take everything else not handled above from the invoker.
+    variables_to_not_forward = [ "deps" ]
+    if (use_jumbo_build_for_target) {
+      deps += [ ":" + target_name + "_jumbo" ]
+      variables_to_not_forward += [ "sources" ]
+      assert(jumbo_files != [])
+      sources = jumbo_files + excluded_sources
+
+      # Need to keep the headers in sources so that dependency checks
+      # work, and we need to keep Objective-C code since they
+      # cannot be merged into a cc file (FIXME).
+      foreach(source_file, invoker.sources) {
+        source_ext = get_path_info(source_file, "extension")
+        if (source_ext == "h" || source_ext == "mm") {
+          sources += [ source_file ]
+        }
+      }
+    }
+    forward_variables_from(invoker, "*", variables_to_not_forward)
+  }
+}
+
+set_defaults("jumbo_target") {
+  # This sets the default list of configs when the content_source_set target
+  # is defined. The default_compiler_configs comes from BUILDCONFIG.gn and
+  # is the list normally applied to static libraries and source sets.
+  configs = default_compiler_configs
+}