Rework how MB and GN handle concurrent links.

build/toolchain/toolchain.gni

Index: build/toolchain/toolchain.gni
diff --git a/build/toolchain/toolchain.gni b/build/toolchain/toolchain.gni
index 9f564794816655edd06d110aeb2ce14390f037bb..0b7aacb7e383ebb63a420b234c0ed9f6df9693cb 100644
--- a/build/toolchain/toolchain.gni
+++ b/build/toolchain/toolchain.gni
@@ -6,8 +6,18 @@
 # toolchain() rules themselves.
 
 import("//build/config/chrome_build.gni")
+import("//build/config/sanitizers/sanitizers.gni")
 
 declare_args() {
+  # Limit the number of concurrent links; we often want to run fewer
+  # links at once than we do compiles, because linking is memory-intensive.
+  # In particular, when using distributed compilation (like w/ goma),

[brucedawson, 2016/06/03 19:42:17]

This is true even without goma. On a Z840 we default to 48 concurrent compiles
but only 12 concurrent links, and even then we can end up consuming twice as
much RAM as the machine contains.

[Dirk Pranke, 2016/06/05 01:19:49]

Acknowledged.

+  # we may be running *a lot* of compiles in parallel but can't do that
+  # many links in parallel. The default to use varies by platform and
+  # by the amount of memory available, so we call out to a script to
+  # get the right value.
+  concurrent_links = -1
+
   # Enable Link Time Optimization in optimized builds (output programs run
   # faster, but linking is up to 5-20x slower).
   # Note: use target_os == "linux" rather than is_linux so that it does not
@@ -28,6 +38,15 @@ declare_args() {
   }
 }
 
+if (concurrent_links == -1) {
+  if (allow_posix_link_time_opt || is_cfi) {

[brucedawson, 2016/06/03 19:42:17]

Should we pass along more arguments to control the calculation? For instance, on
Windows Debug builds use significantly more memory than release builds.
Component builds and win_fastlink builds both use significantly less (and
component win_fastlink builds use very little).

LTCG on Windows also increases memory usage - about as much as debug
non-component builds.

I have scripts that can run a matrix of build tests while monitoring linker
memory if we want more inputs, or we could just pass in the information and
ignore it for now.

[brettw, 2016/06/03 21:31:29]

I'm OK writing a TODO with the information you wrote here.

[brucedawson, 2016/06/03 22:04:50]

Sounds good.

[Dirk Pranke, 2016/06/05 01:19:49]

Will add.

+    args = [ "--lto" ]
+  } else {
+    args = []
+  }
+  concurrent_links = exec_script("get_concurrent_links.py", args, "value")

[Dirk Pranke, 2016/06/03 05:44:43]

@brettw - am I right in thinking/remembering that this is only evaluated once
now, or is it once per toolchain or even once per import? 

If one of the latter two, do you have suggestions for how to rework this so that
I don't have to duplicate it in three toolchain BUILD files, or should I just
duplicate it?

[brettw, 2016/06/03 21:31:29]

This will be executed once for each toolchain that this file is imported in.

I believe this is only necessary in the default toolchain, which is the
toolchain where toolchains are defined.

We could keep the definition in this header and conditionally define it so that
it only exists in the default toolchain.

Theoretically, the build/toolchain/*/BUILD.gn files should only ever be opened
in the default toolchain. It might be cleaner to make a new .gni file and assert
that the current toolchain is the default toolchain in that file. Then it can
only be used from the toolchain definition BUILD files.

[Dirk Pranke, 2016/06/05 01:19:49]

Acknowledged.

+}
+
 # Subdirectory within root_out_dir for shared library files.
 # TODO(agrieve): GYP sets this to "lib" for Linux & Android, but this won't work
 #     in GN until support for loadable_module() is added.