First pass at a meta-build wrapper for the gyp->gn migration

tools/mb/docs/design_spec.md

+# The MB (Meta-Build wrapper) design spec
+
+[TOC]
+
+## Summary of Design Requirements
+
+MB is intended to address two major aspects of the GYP -> GN
+transition for Chromium and two minor aspects:
+
+1. "bot toggling" - make it so that we can easily flip a given bot
+   back and forth between GN and GYP.
+
+2. "bot configuration" - provide a single source of truth for all of
+   the different configurations (os/arch/gyp_define combinations) of
+   Chromium that are supported.
+
+3. "multi-build support" - make it easier for developers to build
+   multiple different build configs (gyp and gn, or linux and android,
+   for example) in a single checkout.
+
+4. "gn configs" - explore ways to manage sets of GN flags more easily
+   (i.e., all the flags needed for a ChromeCast build, or the Linux ASAN
+   bot). Eventually we may migrate this functionality into GN itself.
+
+MB must handle at least the `gen` and `analyze` steps on the bots, i.e.,
+we need to wrap both the `gyp_chromium` invocation to generate the
+Ninja files, and the `analyze` step that takes a list of modified files
+and a list of targets to build and returns which targets are affected by
+the files.
+
+MB may also provide a `build` wrapper in order to build multiple
+configurations in a single step for developer convenience.
+
+## Design
+
+MB is intended to be as simple as possible, and to defer as much work as
+possible to GN or GYP. It should live as a very simple Python wrapper
+that offers little in the way of surprises.
+
+### Command line
+
+It is structured as a single binary that supports a list of subcommands:
+
+* `mb gen //out/Release chromium_linux_rel`
+* `mb analyze -i input.json -o output.json chromium_linux_rel`
+
+### Configurations
+
+`mb` looks in the `//build/mb_conf.pyl` config file to determine whether
+to use GYP or GN for a particular build directory, and what set of flags
+(`GYP_DEFINES` or `gn args`) to use.
+
+`mb_conf.pyl` is structured as a file containing a single PYthon Literal
+expression, a dictionary with two main keys, `build_configs` and
+`mixins`.
+
+The `build_configs` key points to a dictionary of named build
+configurations. Each name should be either a bot name (e.g.,
+`chromium_linux_rel`) or an obvious developer configuration
+(`gyp_shared_debug`).
+
+There should be an entry in this dict for every supported configuration
+of Chromium, meaning every configuration we have a bot for, and every
+configuration commonly used by develpers but that we may not have a bot
+for.
+
+Each build_config entry then points to a list of "mixins" that will
+define what that build_config does. Each item in the list must be
+an entry in the dictionary value of the `mixins` key.
+
+Each entry is itself a diction that contains one or more of the
+following keys:
+
+  * `gyp_configs`: a list of the configurations to build, e.g.,
+    ['Release', 'Release_x64']; specifying this in conjunction
+    with type=gyp_one_config can reduce the time it takes to run gyp_chromium
+    a bit.
+  * `gyp_defines`: a string containing a list of GYP_DEFINES.
+  * `gn_args`: a string containing a list of values passed to gn --args.
+  * `mixins`: a list of other mixins that should be included.
+  * `type`: a string with either the value `gyp`, `gyp_one_config`, or `gn`;
+    setting this indicates which meta-build tool to use.
+
+When `mb gen` executes, it takes a build_config name, looks it up in the
+'build_configs' dict, and then does a left-to-right expansion of the
+mixins; gyp_defines and gn_args values are concatenated, and type and
+gyp_configs values override each other.
+
+For example, if you had:
+
+```
+{
+  'build_configs`: {
+    'linux_release_trybot': ['gyp_release', 'trybot'],
+    'gn_shared_debug': None,
+  }
+  'mixins': {
+    'bot': {
+      'gyp_defines': 'use_goma=1 dcheck_always_on=0',
+      'gn_args': 'use_goma=true dcheck_always_on=false',
+    },
+    'debug': {
+      'gn_args': 'is_debug=true',
+    },
+    'gn': {'type': 'gn'},
+    'gyp_release': {
+      'gyp_config': 'Release'
+      'mixins': ['release'],
+      'type': 'gyp',
+    },
+    'release': {
+      'gn_args': 'is_debug=false',
+    }
+    'shared': {
+      'gn_args': 'is_component_build=true',
+      'gyp_defines': 'component=shared_library',
+    },
+    'trybot': {
+      'gyp_defines': 'dcheck_always_on=1',
+      'gn_args': 'dcheck_always_on=true',
+    }
+  }
+}
+
+and you ran `mb gen //out/Release linux_release_trybot`, it would
+translate into a call to `gyp_chromium -G Release` with `GYP_DEFINES` set to
+`"use_goma=true dcheck_always_on=false dcheck_always_on=true"`.
+
+(From that you can see that mb is intentionally dumb and does not
+attempt to de-dup the flags, it lets gyp do that).
+
+### Handling the analyze step
+
+The way a GYP bot runs the "analyze" step is to pass gyp_chromium
+two filename paths.
+
+The first path is to a JSON file used as input; it contains a single
+object with a two fields:
+
+  * `files`: an array of the modified filenames to check (as
+    paths relative to the checkout root).
+  * `targets`: an array of the unqualified target names to check.
+
+The second path is a path where the analyze generator should write the
+result, also as a JSON object. This object may contain the following
+fields:
+
+  * `error`: this should only be present if something failed.
+  * `targets`: the subset of the input `targets` that depend on the
+    input `files`.
+  * `build_targets`: the minimal subset of targets needed to build all
+    of `targets` that were affected.
+  * `status`: one of three strings:
+    * `"Found dependency"` (build the `build_targets`)
+    * `"No dependency"` (i.e., no build needed)
+    * `"Found dependency (all)"` (build everything, in which case
+      `targets` and `build_targets` are not returned).
+
+mb replicates this command line interface, through `'mb analyze -i
+input.json -o output.json build_config`.
+
+It implements the equivalent functionality in GN by calling `'gn refs
+[list of files] --type=executable --all --as=output` and filtering the
+output to match the list of targets.
+
+### Multi-build support
+
+Today one can pass options to GYP builds in many different ways:
+
+* specifying args directly to `gyp_chromium` with the `-D` flag.
+* setting the `GYP_DEFINES` environment variable.
+* setting the `GYP_DEFINES` key in a file named `chromium.gyp_env` that
+  lives in the directory above `src`.
+* setting variables in a file in `~/.gyp/include.gypi`.
+
+Each of these approaches has advantages and disadvantages; the first two
+are flexible, and allow you to customize each build directory (except
+that you cannot have different flags for debug and release builds unless
+you invoke gyp twice and only build one config each time), but they
+require you to set something every time you run the command line (i.e.,
+extra typing). The third saves typing, but does not allow you to
+generate different build dirs in a single checkout; the fourth applies
+not only to every build dir, but every checkout.
+
+GN, on the other hand, only supports setting flags via the `--args`
+command line argument, or the `args.gn` file in the build directory;
+these are the equivalent of the first two GYP methods.
+
+Ideally we would have a mechanism that offered the best of both worlds:
+we could customize each build directory (including whether to use gyp or
+gn), and we could generate multiple directories at once.
+
+MB supports this by looking for a config file named `builds.pyl` in the
+directory above the checkout. The builds.pyl config file should contain
+a dictionary where each key indicates the path to a desired build
+directory, and each value specifies a particular build config (one of
+the valid `build_config` values in `//build/mb_conf.pyl`).
+
+For example, this:
+
+```
+src% cat ../build.pyl
+{
+  "//out/Release": ["linux_release_trybot"],
+  "//out/Debug.gn": ["gn_shared_debug"],
+}
+src% mbw gen
+src%
+```
+
+is the equivalent of:
+
+```
+src% GYP_DEFINES="use_goma=1 dcheck_always_on=0 dcheck_always_on=1" build/gyp_chromium -G config=Release -G output_dir=out
+src% gn gen //out/Debug.gn --args='use_goma=true dcheck_always_on=true dcheck_always_on=false'
+```
+
+## Detailed Design Requirements and Rationale
+
+This section is collection of semi-organized notes on why MB is the way
+it is
+
+### in-tree or out-of-tree
+
+The first issue is whether or not this should exist as a script in
+Chromium at all; an alternative would be to simply change the bot
+configurations to know whether to use GYP or GN, and which flags to
+pass.
+
+That would certainly work, but experience over the past two years
+suggests a few things:
+
+  * we should push as much logic as we can into the source repositories
+    so that they can be versioned and changed atomically with changes to
+    the product code; having to coordinate changes between src/ and
+    build/ is at best annoying and can lead to weird errors.
+  * the infra team would really like to move to providing
+    product-independent services (i.e., not have to do one thing for
+    Chromium, another for NaCl, a third for V8, etc.).
+  * we found that during the SVN->GIT migration the ability to flip bot
+    configurations between the two via changes to a file in chromium
+    was very useful.
+
+All of this suggests that the interface between bots and Chromium should
+be a simple one, hiding as much of the chromium logic as possible.
+
+### Do we need multi-build support?
+
+Strictly speaking, no. MB could be useful just as a bot->chromium
+interface. However, we do know that managing multiple build directories
+is often awkward for many Chromium devs, at least until they write their
+own wrappers. Many devs often also use multiple checkouts here, which
+has higher overhead and can lead to confusion.
+
+We also know that some people miss the equivalents of setting
+environment variables, chromium.gyp_env, etc., and would like an easier
+way to make sure that multiple build dirs can share settings (like
+use_goma and goma_dir, for example).
+
+### Why not have MB be smarter about de-duping flags?
+
+This just adds complexity to the MB implementation, and duplicates logic
+that GYP and GN already have to support anyway; in particular, it might
+require MB to know how to parse GYP and GN values. The belief is that
+if MB does *not* do this, it will lead to fewer surprises.
+
+It will not be hard to change this if need be.
+
+### Non-goals
+
+* MB is not intended to replace the 
+  [CR tool](https://code.google.com/p/chromium/wiki/CRUserManual). It
+  is only intended to replace the gyp_chromium part of `'gclient
+  runhooks'`. Hopefully the CR tool will learn about and integrate w/ MB
+  once MB becomes stable.
+
+* MB is not (yet?) intended to replace direct invocation of GN or GYP for
+  complicated build scenarios (aka ChromeOS), where multiple flags need
+  to be set to user-defined paths for specific toolchains (e.g., where
+  ChromeOS needs to specify specific board types and compilers).
+
+### Open issues
+
+* Some common flags (goma_dir being the obvious one) may need to be
+  specified via the user, and it's unclear how to integrate this with
+  the concept of build_configs.
+  
+  Right now, MB has hard-coded support for a few flags (i.e., you can
+  pass the --goma-dir flag, and it will know to expand "${goma_dir}" in
+  the string before calling out to the tool. We may want to generalize
+  this to a common key/value approach (perhaps then meeting the
+  ChromeOS non-goal, above), or we may want to keep this very strictly
+  limited for simplicity.
+
+* Do we need to add a `'mb build'` command?
+
+  Being able to update multiple build dirs after syncing is useful, but
+  for GN-based dirs, ninja will do it for us automatically. It's unclear
+  if enough people will have a mixture of GYP and GN builds to make this
+  worth it to add.
+
+* How do we integrate `mb` with `gclient runhooks`?
+
+  On the bots, we will disable gyp_chromium as part of runhooks (using
+  GYP_CHROMIUM_NO_ACTION=1), so that mb shows up as a separate step.
+
+  However, gyp_chromium is run by default as part of `gclient runhooks` 
+  and `gclient sync` for normal developers, and we probably don't want
+  to make everyone need to use GYP_CHROMIUM_NO_ACTION. Given that
+  `mb gen` with no arguments and no builds.pyl file is equivalent to 
+  `gyp_chromium`, we might want to just update the hook to call mb.
+  Are there downsides to doing so?