Chromium Code Reviews
chromiumcodereview-hr@appspot.gserviceaccount.com (chromiumcodereview-hr) | Please choose your nickname with Settings | Help | Chromium Project | Gerrit Changes | Sign out
(490)

Side by Side Diff: tools/mb/docs/user_guide.md

Issue 2299953002: [mb] Copy MB from Chromium repo (Closed)
Patch Set: Pin to V8's config and delete obsolete validation code Created 4 years, 3 months ago
Use n/p to move between diff chunks; N/P to move between comments. Draft comments are only viewable by you.
Jump to:
View unified diff | Download patch
« no previous file with comments | « tools/mb/docs/design_spec.md ('k') | tools/mb/mb » ('j') | no next file with comments »
Toggle Intra-line Diffs ('i') | Expand Comments ('e') | Collapse Comments ('c') | Show Comments Hide Comments ('s')
OLDNEW
(Empty)
1 # The MB (Meta-Build wrapper) user guide
2
3 [TOC]
4
5 ## Introduction
6
7 `mb` is a simple python wrapper around the GYP and GN meta-build tools to
8 be used as part of the GYP->GN migration.
9
10 It is intended to be used by bots to make it easier to manage the configuration
11 each bot builds (i.e., the configurations can be changed from chromium
12 commits), and to consolidate the list of all of the various configurations
13 that Chromium is built in.
14
15 Ideally this tool will no longer be needed after the migration is complete.
16
17 For more discussion of MB, see also [the design spec](design_spec.md).
18
19 ## MB subcommands
20
21 ### `mb analyze`
22
23 `mb analyze` is reponsible for determining what targets are affected by
24 a list of files (e.g., the list of files in a patch on a trybot):
25
26 ```
27 mb analyze -c chromium_linux_rel //out/Release input.json output.json
28 ```
29
30 Either the `-c/--config` flag or the `-m/--master` and `-b/--builder` flags
31 must be specified so that `mb` can figure out which config to use.
32
33 The first positional argument must be a GN-style "source-absolute" path
34 to the build directory.
35
36 The second positional argument is a (normal) path to a JSON file containing
37 a single object with the following fields:
38
39 * `files`: an array of the modified filenames to check (as paths relative to
40 the checkout root).
41 * `test_targets`: an array of (ninja) build targets that needed to run the
42 tests we wish to run. An empty array will be treated as if there are
43 no tests that will be run.
44 * `additional_compile_targets`: an array of (ninja) build targets that
45 reflect the stuff we might want to build *in addition to* the list
46 passed in `test_targets`. Targets in this list will be treated
47 specially, in the following way: if a given target is a "meta"
48 (GN: group, GYP: none) target like 'blink_tests' or
49 'chromium_builder_tests', or even the ninja-specific 'all' target,
50 then only the *dependencies* of the target that are affected by
51 the modified files will be rebuilt (not the target itself, which
52 might also cause unaffected dependencies to be rebuilt). An empty
53 list will be treated as if there are no additional targets to build.
54 Empty lists for both `test_targets` and `additional_compile_targets`
55 would cause no work to be done, so will result in an error.
56 * `targets`: a legacy field that resembled a union of `compile_targets`
57 and `test_targets`. Support for this field will be removed once the
58 bots have been updated to use compile_targets and test_targets instead.
59
60 The third positional argument is a (normal) path to where mb will write
61 the result, also as a JSON object. This object may contain the following
62 fields:
63
64 * `error`: this should only be present if something failed.
65 * `compile_targets`: the list of ninja targets that should be passed
66 directly to the corresponding ninja / compile.py invocation. This
67 list may contain entries that are *not* listed in the input (see
68 the description of `additional_compile_targets` above and
69 [design_spec.md](the design spec) for how this works).
70 * `invalid_targets`: a list of any targets that were passed in
71 either of the input lists that weren't actually found in the graph.
72 * `test_targets`: the subset of the input `test_targets` that are
73 potentially out of date, indicating that the matching test steps
74 should be re-run.
75 * `targets`: a legacy field that indicates the subset of the input `targets`
76 that depend on the input `files`.
77 * `build_targets`: a legacy field that indicates the minimal subset of
78 targets needed to build all of `targets` that were affected.
79 * `status`: a field containing one of three strings:
80
81 * `"Found dependency"` (build the `compile_targets`)
82 * `"No dependency"` (i.e., no build needed)
83 * `"Found dependency (all)"` (`test_targets` is returned as-is;
84 `compile_targets` should contain the union of `test_targets` and
85 `additional_compile_targets`. In this case the targets do not
86 need to be pruned).
87
88 See [design_spec.md](the design spec) for more details and examples; the
89 differences can be subtle. We won't even go into how the `targets` and
90 `build_targets` differ from each other or from `compile_targets` and
91 `test_targets`.
92
93 The `-b/--builder`, `-c/--config`, `-f/--config-file`, `-m/--master`,
94 `-q/--quiet`, and `-v/--verbose` flags work as documented for `mb gen`.
95
96 ### `mb audit`
97
98 `mb audit` is used to track the progress of the GYP->GN migration. You can
99 use it to check a single master, or all the masters we care about. See
100 `mb help audit` for more details (most people are not expected to care about
101 this).
102
103 ### `mb gen`
104
105 `mb gen` is responsible for generating the Ninja files by invoking either GYP
106 or GN as appropriate. It takes arguments to specify a build config and
107 a directory, then runs GYP or GN as appropriate:
108
109 ```
110 % mb gen -m tryserver.chromium.linux -b linux_rel //out/Release
111 % mb gen -c linux_rel_trybot //out/Release
112 ```
113
114 Either the `-c/--config` flag or the `-m/--master` and `-b/--builder` flags
115 must be specified so that `mb` can figure out which config to use. The
116 `--phase` flag must also be used with builders that have multiple
117 build/compile steps (and only with those builders).
118
119 By default, MB will look for a bot config file under `//ios/build/bots` (see
120 [design_spec.md](the design spec) for details of how the bot config files
121 work). If no matching one is found, will then look in
122 `//tools/mb/mb_config.pyl` to look up the config information, but you can
123 specify a custom config file using the `-f/--config-file` flag.
124
125 The path must be a GN-style "source-absolute" path (as above).
126
127 You can pass the `-n/--dryrun` flag to mb gen to see what will happen without
128 actually writing anything.
129
130 You can pass the `-q/--quiet` flag to get mb to be silent unless there is an
131 error, and pass the `-v/--verbose` flag to get mb to log all of the files
132 that are read and written, and all the commands that are run.
133
134 If the build config will use the Goma distributed-build system, you can pass
135 the path to your Goma client in the `-g/--goma-dir` flag, and it will be
136 incorporated into the appropriate flags for GYP or GN as needed.
137
138 If gen ends up using GYP, the path must have a valid GYP configuration as the
139 last component of the path (i.e., specify `//out/Release_x64`, not `//out`).
140 The gyp script defaults to `//build/gyp_chromium`, but can be overridden with
141 the `--gyp-script` flag, e.g. `--gyp-script=gypfiles/gyp_v8`.
142
143 ### `mb help`
144
145 Produces help output on the other subcommands
146
147 ### `mb lookup`
148
149 Prints what command will be run by `mb gen` (like `mb gen -n` but does
150 not require you to specify a path).
151
152 The `-b/--builder`, `-c/--config`, `-f/--config-file`, `-m/--master`,
153 `--phase`, `-q/--quiet`, and `-v/--verbose` flags work as documented for
154 `mb gen`.
155
156 ### `mb validate`
157
158 Does internal checking to make sure the config file is syntactically
159 valid and that all of the entries are used properly. It does not validate
160 that the flags make sense, or that the builder names are legal or
161 comprehensive, but it does complain about configs and mixins that aren't
162 used.
163
164 The `-f/--config-file` and `-q/--quiet` flags work as documented for
165 `mb gen`.
166
167 This is mostly useful as a presubmit check and for verifying changes to
168 the config file.
169
170 ## Isolates and Swarming
171
172 `mb gen` is also responsible for generating the `.isolate` and
173 `.isolated.gen.json` files needed to run test executables through swarming
174 in a GN build (in a GYP build, this is done as part of the compile step).
175
176 If you wish to generate the isolate files, pass `mb gen` the
177 `--swarming-targets-file` command line argument; that arg should be a path
178 to a file containing a list of ninja build targets to compute the runtime
179 dependencies for (on Windows, use the ninja target name, not the file, so
180 `base_unittests`, not `base_unittests.exe`).
181
182 MB will take this file, translate each build target to the matching GN
183 label (e.g., `base_unittests` -> `//base:base_unittests`, write that list
184 to a file called `runtime_deps` in the build directory, and pass that to
185 `gn gen $BUILD ... --runtime-deps-list-file=$BUILD/runtime_deps`.
186
187 Once GN has computed the lists of runtime dependencies, MB will then
188 look up the command line for each target (currently this is hard-coded
189 in [mb.py](https://code.google.com/p/chromium/codesearch?q=mb.py#chromium/src/to ols/mb/mb.py&q=mb.py%20GetIsolateCommand&sq=package:chromium&type=cs)), and writ e out the
190 matching `.isolate` and `.isolated.gen.json` files.
191
192 ## The `mb_config.pyl` config file
193
194 The `mb_config.pyl` config file is intended to enumerate all of the
195 supported build configurations for Chromium. Generally speaking, you
196 should never need to (or want to) build a configuration that isn't
197 listed here, and so by using the configs in this file you can avoid
198 having to juggle long lists of GYP_DEFINES and gn args by hand.
199
200 `mb_config.pyl` is structured as a file containing a single PYthon Literal
201 expression: a dictionary with three main keys, `masters`, `configs` and
202 `mixins`.
203
204 The `masters` key contains a nested series of dicts containing mappings
205 of master -> builder -> config . This allows us to isolate the buildbot
206 recipes from the actual details of the configs. The config should either
207 be a single string value representing a key in the `configs` dictionary,
208 or a list of strings, each of which is a key in the `configs` dictionary;
209 the latter case is for builders that do multiple compiles with different
210 arguments in a single build, and must *only* be used for such builders
211 (where a --phase argument must be supplied in each lookup or gen call).
212
213 The `configs` key points to a dictionary of named build configurations.
214
215 There should be an key in this dict for every supported configuration
216 of Chromium, meaning every configuration we have a bot for, and every
217 configuration commonly used by develpers but that we may not have a bot
218 for.
219
220 The value of each key is a list of "mixins" that will define what that
221 build_config does. Each item in the list must be an entry in the dictionary
222 value of the `mixins` key.
223
224 Each mixin value is itself a dictionary that contains one or more of the
225 following keys:
226
227 * `gyp_crosscompile`: a boolean; if true, GYP_CROSSCOMPILE=1 is set in
228 the environment and passed to GYP.
229 * `gyp_defines`: a string containing a list of GYP_DEFINES.
230 * `gn_args`: a string containing a list of values passed to gn --args.
231 * `mixins`: a list of other mixins that should be included.
232 * `type`: a string with either the value `gyp` or `gn`;
233 setting this indicates which meta-build tool to use.
234
235 When `mb gen` or `mb analyze` executes, it takes a config name, looks it
236 up in the 'configs' dict, and then does a left-to-right expansion of the
237 mixins; gyp_defines and gn_args values are concatenated, and the type values
238 override each other.
239
240 For example, if you had:
241
242 ```
243 {
244 'configs`: {
245 'linux_release_trybot': ['gyp_release', 'trybot'],
246 'gn_shared_debug': None,
247 }
248 'mixins': {
249 'bot': {
250 'gyp_defines': 'use_goma=1 dcheck_always_on=0',
251 'gn_args': 'use_goma=true dcheck_always_on=false',
252 },
253 'debug': {
254 'gn_args': 'is_debug=true',
255 },
256 'gn': {'type': 'gn'},
257 'gyp_release': {
258 'mixins': ['release'],
259 'type': 'gyp',
260 },
261 'release': {
262 'gn_args': 'is_debug=false',
263 }
264 'shared': {
265 'gn_args': 'is_component_build=true',
266 'gyp_defines': 'component=shared_library',
267 },
268 'trybot': {
269 'gyp_defines': 'dcheck_always_on=1',
270 'gn_args': 'dcheck_always_on=true',
271 }
272 }
273 }
274 ```
275
276 and you ran `mb gen -c linux_release_trybot //out/Release`, it would
277 translate into a call to `gyp_chromium -G Release` with `GYP_DEFINES` set to
278 `"use_goma=true dcheck_always_on=false dcheck_always_on=true"`.
279
280 (From that you can see that mb is intentionally dumb and does not
281 attempt to de-dup the flags, it lets gyp do that).
282
283 ## Debugging MB
284
285 By design, MB should be simple enough that very little can go wrong.
286
287 The most obvious issue is that you might see different commands being
288 run than you expect; running `'mb -v'` will print what it's doing and
289 run the commands; `'mb -n'` will print what it will do but *not* run
290 the commands.
291
292 If you hit weirder things than that, add some print statements to the
293 python script, send a question to gn-dev@chromium.org, or
294 [file a bug](https://crbug.com/new) with the label
295 'mb' and cc: dpranke@chromium.org.
296
297
OLDNEW
« no previous file with comments | « tools/mb/docs/design_spec.md ('k') | tools/mb/mb » ('j') | no next file with comments »

Powered by Google App Engine
This is Rietveld 408576698