Rework how MB and GN handle concurrent links.

build/toolchain/toolchain.gni

 # Copyright 2015 The Chromium Authors. All rights reserved.
 # Use of this source code is governed by a BSD-style license that can be
 # found in the LICENSE file.
 
 # Toolchain-related configuration that may be needed outside the context of the
 # toolchain() rules themselves.
 
 import("//build/config/chrome_build.gni")
+import("//build/config/sanitizers/sanitizers.gni")
 
 declare_args() {
+  # Limit the number of concurrent links; we often want to run fewer
+  # links at once than we do compiles, because linking is memory-intensive.
+  # In particular, when using distributed compilation (like w/ goma),

[brucedawson, 2016/06/03 19:42:17]

This is true even without goma. On a Z840 we default to 48 concurrent compiles
but only 12 concurrent links, and even then we can end up consuming twice as
much RAM as the machine contains.

[Dirk Pranke, 2016/06/05 01:19:49]

Acknowledged.

+  # we may be running *a lot* of compiles in parallel but can't do that
+  # many links in parallel. The default to use varies by platform and
+  # by the amount of memory available, so we call out to a script to
+  # get the right value.
+  concurrent_links = -1
+
   # Enable Link Time Optimization in optimized builds (output programs run
   # faster, but linking is up to 5-20x slower).
   # Note: use target_os == "linux" rather than is_linux so that it does not
   # apply to host_toolchain when target_os="android".
   allow_posix_link_time_opt =
       target_os == "linux" && !is_chromeos && target_cpu == "x64" &&
       is_chrome_branded && is_official_build
 
   # Set to true to use lld, the LLVM linker. This flag may be used on Windows
   # with the shipped LLVM toolchain, or on Linux with a self-built top-of-tree
   # LLVM toolchain (see llvm_force_head_revision in
   # build/config/compiler/BUILD.gn).
   use_lld = false
 
   if (is_clang) {
     # Clang compiler version. Clang files are placed at version-dependent paths.
     clang_version = "3.9.0"
   }
 }
 
+if (concurrent_links == -1) {
+  if (allow_posix_link_time_opt || is_cfi) {

[brucedawson, 2016/06/03 19:42:17]

Should we pass along more arguments to control the calculation? For instance, on
Windows Debug builds use significantly more memory than release builds.
Component builds and win_fastlink builds both use significantly less (and
component win_fastlink builds use very little).

LTCG on Windows also increases memory usage - about as much as debug
non-component builds.

I have scripts that can run a matrix of build tests while monitoring linker
memory if we want more inputs, or we could just pass in the information and
ignore it for now.

[brettw, 2016/06/03 21:31:29]

I'm OK writing a TODO with the information you wrote here.

[brucedawson, 2016/06/03 22:04:50]

Sounds good.

[Dirk Pranke, 2016/06/05 01:19:49]

Will add.

+    args = [ "--lto" ]
+  } else {
+    args = []
+  }
+  concurrent_links = exec_script("get_concurrent_links.py", args, "value")

[Dirk Pranke, 2016/06/03 05:44:43]

@brettw - am I right in thinking/remembering that this is only evaluated once
now, or is it once per toolchain or even once per import? 

If one of the latter two, do you have suggestions for how to rework this so that
I don't have to duplicate it in three toolchain BUILD files, or should I just
duplicate it?

[brettw, 2016/06/03 21:31:29]

This will be executed once for each toolchain that this file is imported in.

I believe this is only necessary in the default toolchain, which is the
toolchain where toolchains are defined.

We could keep the definition in this header and conditionally define it so that
it only exists in the default toolchain.

Theoretically, the build/toolchain/*/BUILD.gn files should only ever be opened
in the default toolchain. It might be cleaner to make a new .gni file and assert
that the current toolchain is the default toolchain in that file. Then it can
only be used from the toolchain definition BUILD files.

[Dirk Pranke, 2016/06/05 01:19:49]

Acknowledged.

+}
+
 # Subdirectory within root_out_dir for shared library files.
 # TODO(agrieve): GYP sets this to "lib" for Linux & Android, but this won't work
 #     in GN until support for loadable_module() is added.
 #     See: https://codereview.chromium.org/1236503002/
 shlib_subdir = "."
 
 # Root out dir for shared library files.
 root_shlib_dir = root_out_dir
 if (shlib_subdir != ".") {
   root_shlib_dir += "/$shlib_subdir"
@@ skipping 30 matching lines @@
 stamp_description = "STAMP {{output}}"
 copy_description = "COPY {{source}} {{output}}"
 if (host_os == "win") {
   stamp_command = "$python_path gyp-win-tool stamp {{output}}"
   copy_command =
       "$python_path gyp-win-tool recursive-mirror {{source}} {{output}}"
 } else {
   stamp_command = "touch {{output}}"
   copy_command = "ln -f {{source}} {{output}} 2>/dev/null || (rm -rf {{output}} && cp -af {{source}} {{output}})"
 }