Scripts for unity/jumbo (default disabled) compilation

build/jumbo.gni

Index: build/jumbo.gni
diff --git a/build/jumbo.gni b/build/jumbo.gni
new file mode 100644
index 0000000000000000000000000000000000000000..7dc5dd24ac4b1fb4dc3654fb704e3b1edd86c781
--- /dev/null
+++ b/build/jumbo.gni
@@ -0,0 +1,124 @@
+# Copyright 2017 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+import("//build/split_static_library.gni")  # When someone uses that target_type
+
+declare_args() {
+  # If true, use a jumbo build (files compiled together) to speed up
+  # compilation.
+  use_blink_jumbo_build = false

[Daniel Bratell, 2017/06/28 14:31:00]

It is named "blink" here, but considering that the ambition is that this will be
used far outside blink, maybe "use_jumbo_build" is a better name?

[Dirk Pranke, 2017/06/29 23:09:46]

How opposed are you to using the 'unity' name rather than the 'jumbo' name?

I would rather use unity if that's okay, I think it's at least a bit more
understood although neither usage is common.

I would get rid of the 'blink' part either way though.

[Daniel Bratell, 2017/06/30 08:22:19]

I don't think it's good to call it unity because it is a word already used for
so many other things. In the current source it's used to talk about the Ubuntu
graphical environment/style and about audio filters and QUIC congestion control.
In the wider software world it is a 3D graphical engine and other things.

Using unity for this will make it harder for those people to discuss their own
code without confusion, and it will make it harder for us to talk about unity
builds without confusion. When I started talking about this I noticed that
people thought they knew what I was talking about and then tuned out. That
stopped happening when I changed to Jumbo.

I understand your argument, and I did try to call it unity initially because
that is the most accepted term for it, but it wasn't useful beyond the ~3 people
that happened to know about it in advance.
Will do!

+
+  # A target to exclude from jumbo builds, for optimal round trip time
+  # when frequently changing a single cpp file.
+  jumbo_build_excluded = ""
+}
+
+# Use this to generate a target which merges sources if possible to
+# compile much faster.
+#
+# Special values.
+#
+#   target_type
+#      The kind of target to build. For example the string
+#      "static_library".
+#
+#   jumbo_build_override
+#      If set to true, then use jumbo compile even when it is globally
+#      disabled. If set to false, then it is disabled even if jumbo is
+#      globally enabled.

[Dirk Pranke, 2017/06/29 23:09:46]

Overloading a single flag like this seems kinda awkward. WDYT about having two
different flags: always_build_jumbo and never_build_jumbo or something like
that?

[Daniel Bratell, 2017/06/30 08:22:19]

Yes, renaming this one would really be useful for readability. Do you think
about those flags being two boolean values? They sound like an annotation and
I'm not sure how to do that in gn?

[Daniel Bratell, 2017/06/30 10:09:53]

Done.

+#
+#   jumbo_excluded_sources
+#      If set to a list of files, those files will not be merged with
+#      the rest. Can be used if files are problematic.
+template("jumbo_target") {
+  use_jumbo_build = use_blink_jumbo_build
+  if (defined(invoker.jumbo_build_override)) {
+    use_jumbo_build = invoker.jumbo_build_override
+  }
+  if (target_name == jumbo_build_excluded) {
+    use_jumbo_build = false
+  }
+  excluded_sources = []
+  if (defined(invoker.jumbo_excluded_sources)) {
+    excluded_sources += invoker.jumbo_excluded_sources
+  }
+  invoker_sources = invoker.sources
+  gen_target_dir = get_path_info(invoker_sources[0], "gen_dir")
+  assert(excluded_sources != [] || true)  # Prevent "unused variable".
+  assert(gen_target_dir != "")  # Prevent "unused variable".
+
+  if (use_jumbo_build) {
+    jumbo_files = []
+
+    # Split the sources list into chunks that are not excessively large
+    files_per_chunk = 200

[Dirk Pranke, 2017/06/29 23:09:46]

Maybe we should make this configurable so we can tune it more easily, or perhaps
make this an input parameter to the template? When I was testing my version of
the jumbo build concept, I would see significant speedups on much smaller
targets, but if the chunk size was too large some targets wouldn't be
parallelized at all. Or maybe there should be a minimum number of chunks per
target, or something?

[Daniel Bratell, 2017/06/30 08:22:19]

I can make it a declare_args configurable variable with a default to make it
easier to tune, though for every different size, the sets of files grouped with
each other changes so just because 200 compiles, 199 or 201 may not. (100
probably will though since it's just halving).

[Daniel Bratell, 2017/06/30 10:09:53]

Done.

+    current_file_index = 0
+    next_chunk_start = 0
+    next_chunk_number = 1
+    foreach(source_file, invoker.sources) {
+      if (current_file_index == next_chunk_start) {
+        jumbo_files += [ "$gen_target_dir/" + target_name + "_jumbo_" +
+                         next_chunk_number + ".cc" ]
+        next_chunk_number += 1
+        next_chunk_start += files_per_chunk
+      }
+      current_file_index += 1
+    }
+
+    # Create an action that calls a script that merges all the source files.
+    action(target_name + "_jumbo") {
+      script = "//build/jumbo.py"
+      response_file_contents =
+          rebase_path(invoker.sources - excluded_sources, gen_target_dir)
+      outputs = jumbo_files
+      args = [ "--outputs" ] + rebase_path(outputs, root_build_dir) +
+             [ "--file-list={{response_file_name}}" ]
+    }
+  }
+
+  target_type = invoker.target_type
+  if (use_jumbo_build && target_type == "split_static_library") {
+    # Meaningless and also impossible if split_count > len(jumbo_files)
+    target_type = "static_library"
+
+    # Prevent "unused variable" warning.
+    assert(!defined(invoker.split_count) || invoker.split_count > 0)
+  }
+
+  # Perform the actual operation, either on the original sources or
+  # the sources post-jumbo merging.
+  target(target_type, target_name) {
+    deps = []
+    if (defined(invoker.deps)) {
+      deps += invoker.deps
+    }
+
+    # Take everything else not handled above from the invoker.
+    variables_to_not_forward = [ "deps" ]
+    if (use_jumbo_build) {
+      deps += [ ":" + target_name + "_jumbo" ]
+      variables_to_not_forward += [ "sources" ]
+      assert(jumbo_files != [])
+      sources = jumbo_files + excluded_sources
+
+      # Need to keep the headers in sources so that dependency checks
+      # work, and we need to keep Objective-C code since they
+      # cannot be merged into a cc file (FIXME).
+      foreach(source_file, invoker.sources) {
+        source_ext = get_path_info(source_file, "extension")
+        if (source_ext == "h" || source_ext == "mm") {

[Daniel Bratell, 2017/06/28 14:31:00]

I have plans for Objective-C eventually, but I don't want to make the system
more complex before everyone is familiar with the base functionality.

+          sources += [ source_file ]
+        }
+      }
+    }
+    forward_variables_from(invoker, "*", variables_to_not_forward)
+  }
+}
+
+set_defaults("jumbo_target") {
+  # This sets the default list of configs when the content_source_set target
+  # is defined. The default_compiler_configs comes from BUILDCONFIG.gn and
+  # is the list normally applied to static libraries and source sets.
+  configs = default_compiler_configs
+}