Cross-repo recipe package system.

third_party/recipe_engine/doc/user_guide.md

-# Recipes
-
-Recipes are a domain-specific language (embedded in python) for specifying
-sequences of subprocess calls in a cross-platform and testable way.
-
-[TOC]
-
-## Background
-
-Chromium uses BuildBot for its builds.  It requires master restarts to change
-bot configs, which slows bot changes down.
-
-With Recipes, most build-related things happen in scripts that run on the
-slave, which means that the master does not need to be restarted in order
-to change something about a build configuration.
-
-Recipes also provide a way to unit test build scripts, by mocking commands and
-recording "expectations" of what will happen when the script runs under various
-conditions.  This makes it easy to verify that the scope of a change is limited.
-
-## Intro
-
-This README will seek to teach the ways of Recipes, so that you may do one or
-more of the following:
-
-  * Read them
-  * Make new recipes
-  * Fix bugs in recipes
-  * Create libraries (api modules) for others to use in their recipes.
-
-The document will build knowledge up in small steps using examples, and so it's
-probably best to read the whole doc through from top to bottom once before using
-it as a reference.
-
-## Small Beginnings
-
-**Recipes are a means to cause a series of commands to run on a machine.**
-
-All recipes take the form of a python file whose body looks like this:
-
-```python
-DEPS = ['step']
-
-def RunSteps(api):
-  api.step('Print Hello World', ['echo', 'hello', 'world'])
-```
-
-The `RunSteps` function is expected to take at least a single argument `api`
-(we'll get to that in more detail later), and run a series of steps by calling
-api functions.  All of these functions will eventually make calls to
-`api.step()`, which is the only way to actually get anything done on the
-machine.  Using python libraries with OS side-effects is prohibited to enable
-testing.
-
-For these examples we will work out of the
-[tools/build](https://chromium.googlesource.com/chromium/tools/build/)
-repository.
-
-Put this in a file under `scripts/slave/recipes/hello.py`. You can then
-run this recipe by calling
-
-    $ scripts/tools/run_recipe.py hello
-
-*** promo
-Note: every recipe execution (e.g. build on buildbot) emits
-a step log called `run_recipe` on the `setup_build` step which provides
-a precise invocation for `run_recipe.py` correlating exactly with the current
-recipe invocation. This is useful to locally repro a failing build without
-having to guess at the parameters to `run_recipe.py`.
-***
-
-## We should probably test as we go...
-
-**All recipes MUST have corresponding tests, which achieve 100% code coverage.**
-
-So, we have our recipe. Let's add a test to it.
-
-```python
-DEPS = ['step']
-
-def RunSteps(api):
-  api.step('Print Hello World', ['echo', 'hello', 'world'])
-
-def GenTests(api):
-  yield api.test('basic')
-```
-
-This causes a single test case to be generated, called 'basic', which has no
-input parameters.  As your recipe becomes more complex, you'll need to add more
-tests to make sure that you maintain 100% code coverage.
-
-In order to run the tests, run
-
-    $ scripts/slave/unittests/recipe_simulation_test.py train hello
-
-This will write the file `build/scripts/slave/recipes/hello.expected/basic.json`
-summarizing the actions of the recipe under the boring conditions
-specified by `api.test('basic')`.
-
-    [
-      {
-        "cmd": [
-          "echo",
-          "hello",
-          "world"
-        ],
-        "cwd": "[SLAVE_BUILD]",
-        "name": "Print Hello World"
-      }
-    ]
-
-## Let's do something useful
-
-### Properties are the primary input for your recipes
-
-In order to do something useful, we need to pull in parameters from the outside
-world. There's one primary source of input for recipes, which is `properties`.
-
-Properties are a relic from the days of BuildBot, though they have been
-dressed up a bit to be more like we'll want them in the future. If you're
-familiar with BuildBot, you'll probably know them as `factory_properties` and
-`build_properties`. The new `properties` object is a merging of these two, and
-is provided by the `properties` api module.
-
-```python
-from recipe_engine.recipe_api import Property
-
-DEPS = [
-  'step',
-]
-
-PROPERTIES = {
-  'target_of_admiration': Property(
-    kind=str, help="Who you love and adore.", default="Chrome Infra"),
-}
-
-def RunSteps(api, target_of_admiration):
-  verb = 'Hello, %s'
-  if target_of_admiration == 'DarthVader':
-    verb = 'Die in a fire, %s!'
-  api.step('Greet Admired Individual', ['echo', verb % target_of_admiration])
-
-def GenTests(api):
-  yield api.test('basic') + api.properties(target_of_admiration='Bob')
-  yield api.test('vader') + api.properties(target_of_admiration='DarthVader')
-  yield api.test('infra rocks')
-```
-
-Yes, elements of a test specification are combined with `+` and it's weird.
-
-To specify property values in a local run:
-
-    build/scripts/tools/run_recipe.py <recipe-name> opt=bob other=sally
-
-Or, more explicitly::
-
-    build/scripts/tools/run_recipe.py --properties-file <path/to/json>
-
-Where `<path/to/json>` is a file containing a valid json `object` (i.e.
-key:value pairs).
-
-### Modules
-
-There are all sorts of helper modules.  They are found in the `recipe_modules`
-directory alongside the `recipes` directory where the recipes go.
-
-Notice the `DEPS` line in the recipe. Any modules named by string in DEPS are
-'injected' into the `api` parameter that your recipe gets. If you leave them out
-of DEPS, you'll get an AttributeError when you try to access them. The modules
-are located primarily in `recipe_modules/`, and their name is their folder name.
-
-There are a whole bunch of modules which provide really helpful tools. You
-should go take a look at them. `scripts/tools/show_me_the_modules.py` is a
-pretty helpful tool. If you want to know more about properties, step and path, I
-would suggest starting with `show_me_the_modules.py`, and then delving into the
-helpful docstrings in those helpful modules.
-
-## Making Modules
-
-**Modules are for grouping functionality together and exposing it across
-recipes.**
-
-So now you feel like you're pretty good at recipes, but you want to share your
-echo functionality across a couple recipes which all start the same way. To do
-this, you need to add a module directory.
-
-```
-recipe_modules/
-  step/
-  properties/
-  path/
-  hello/
-    __init__.py  # (Required) Contains optional `DEPS = list([other modules])`
-    api.py       # (Required) Contains single required RecipeApi-derived class
-    config.py    # (Optional) Contains configuration for your api
-    *_config.py  # (Optional) These contain extensions to the configurations of
-                 #   your dependency APIs
-```
-
-First add an `__init__.py` with DEPS:
-
-```python
-# recipe_modules/hello/__init__.py
-from recipe_api import Property
-
-DEPS = ['properties', 'step']
-PROPERTIES = {
-  'target_of_admiration': Property(default=None),
-}
-```
-
-And your api.py should look something like:
-
-```python
-from slave import recipe_api
-
-class HelloApi(recipe_api.RecipeApi):
-  def __init__(self, target_of_admiration):
-    self._target = target_of_admiration
-
-  def greet(self, default_verb=None):
-    verb = default_verb or 'Hello %s'
-    if self._target == 'DarthVader':
-      verb = 'Die in a fire %s!'
-    self.m.step('Hello World',
-                ['echo', verb % self._target])
-```
-
-Note that all the DEPS get injected into `self.m`. This logic is handled outside
-of the object (i.e. not in `__init__`).
-
-> Because dependencies are injected after module initialization, *you do not
-> have access to injected modules in your APIs `__init__` method*!
-
-And now, our refactored recipe:
-
-```python
-DEPS = ['hello']
-
-def RunSteps(api):
-  api.hello.greet()
-
-def GenTests(api):
-  yield api.test('basic') + api.properties(target_of_admiration='Bob')
-  yield api.test('vader') + api.properties(target_of_admiration='DarthVader')
-```
-
-> NOTE: all of the modules are also require 100% code coverage, but you only
-> need coverage from SOME recipe.
-
-## So how do I really write those tests?
-
-The basic form of tests is:
-
-```python
-def GenTests(api):
-  yield api.test('testname') + # other stuff
-```
-
-Some modules define interfaces for specifying necessary step data; these are
-injected into `api` from `DEPS` similarly to how it works for `RunSteps`.  There
-are a few other methods available to `GenTests`'s `api`. Common ones include:
-
-  * `api.properties(buildername='foo_builder')` sets properties as we have seen.
-  * `api.platform('linux', 32)` sets the mock platform to 32-bit linux.
-  * `api.step_data('Hello World', retcode=1)` mocks the `'Hello World'` step
-  to have failed with exit code 1.
-
-By default all simulated steps succeed, the platform is 64-bit linux, and
-there are no properties.  The `api.properties.generic()` method populates some
-common properties for Chromium recipes.
-
-The `api` passed to GenTests is confusingly **NOT** the same as the recipe api.
-It's actually an instance of `recipe_test_api.py:RecipeTestApi()`. This is
-admittedly pretty weak, and it would be great to have the test api
-automatically created via modules. On the flip side, the test api is much less
-necessary than the recipe api, so this transformation has not been designed yet.
-
-## What is that config business?
-
-**Configs are a way for a module to expose it's "global" state in a reusable
-way.**
-
-A common problem in Building Things is that you end up with an inordinantly
-large matrix of configurations. Let's take chromium, for example. Here is a
-sample list of axes of configuration which chromium needs to build and test:
-
-  * BUILD_CONFIG
-  * HOST_PLATFORM
-  * HOST_ARCH
-  * HOST_BITS
-  * TARGET_PLATFORM
-  * TARGET_ARCH
-  * TARGET_BITS
-  * builder type (ninja? msvs? xcodebuild?)
-  * compiler
-  * ...
-
-Obviously there are a lot of combinations of those things, but only a relatively
-small number of *valid* combinations of those things. How can we represent all
-the valid states while still retaining our sanity?
-
-We begin by specifying a schema that configurations of the `hello` module
-will follow, and the config context based on it that we will add configuration
-items to.
-
-```python
-# recipe_modules/hello/config.py
-from slave.recipe_config import config_item_context, ConfigGroup
-from slave.recipe_config import SimpleConfig, StaticConfig, BadConf
-
-def BaseConfig(TARGET='Bob'):
-  # This is a schema for the 'config blobs' that the hello module deals with.
-  return ConfigGroup(
-    verb   = SimpleConfig(str),
-    # A config blob is not complete() until all required entries have a value.
-    tool   = SimpleConfig(str, required=True),
-    # Generally, your schema should take a series of CAPITAL args which will be
-    # set as StaticConfig data in the config blob.
-    TARGET = StaticConfig(str(TARGET)),
-  )
-
-config_ctx = config_item_context(BaseConfig)
-```
-
-The `BaseConfig` schema is expected to return a `ConfigGroup` instance of some
-sort. All the configs that you get out of this file will be a modified version
-of something returned by the schema method. The arguments should have sane
-defaults, and should be named in `ALL_CAPS` (this is to avoid argument name
-conflicts as we'll see later).
-
-`config_ctx` is the 'context' for all the config items in this file, and will
-magically become the `CONFIG_CTX` for the entire module.  Other modules may
-extend this context, which we will get to later.
-
-Finally let's define some config items themselves. A config item is a function
-decorated with the `config_ctx`, and takes a config blob as 'c'. The config item
-updates the config blob, perhaps conditionally. There are many features to
-`slave/recipe_config.py`. I would recommend reading the docstrings there
-for all the details.
-
-```python
-# Each of these functions is a 'config item' in the context of config_ctx.
-
-# is_root means that every config item will apply this item first.
-@config_ctx(is_root=True)
-def BASE(c):
-  if c.TARGET == 'DarthVader':
-    c.verb = 'Die in a fire, %s!'
-  else:
-    c.verb = 'Hello, %s'
-
-@config_ctx(group='tool'):  # items with the same group are mutually exclusive.
-def super_tool(c):
-  if c.TARGET != 'Charlie':
-    raise BadConf('Can only use super tool for Charlie!')
-  c.tool = 'unicorn.py'
-
-@config_ctx(group='tool'):
-def default_tool(c):
-  c.tool = 'echo'
-```
-
-Now that we have our config, let's use it.
-
-```python
-# recipe_modules/hello/api.py
-from slave import recipe_api
-
-class HelloApi(recipe_api.RecipeApi):
-  def __init__(self, target_of_admiration):
-    self._target = target_of_admiration
-
-  def get_config_defaults(self, _config_name):
-    return {'TARGET': self._target}
-
-  def greet(self):
-    self.m.step('Hello World', [
-        self.m.path.build(self.c.tool), self.c.verb % self.c.TARGET])
-```
-
-Note that `recipe_api.RecipeApi` contains all the plumbing for dealing with
-configs. If your module has a config, you can access its current value via
-`self.c`. The users of your module (read: recipes) will need to set this value
-in one way or another. Also note that c is a 'public' variable, which means that
-recipes have direct access to the configuration state by `api.<modname>.c`.
-
-```python
-# recipes/hello.py
-DEPS = ['hello']
-def RunSteps(api):
-  api.hello.set_config('default_tool')
-  api.hello.greet()  # Greets 'target_of_admiration' or 'Bob' with echo.
-
-def GenTests(api):
-  yield api.test('bob')
-  yield api.test('anya') + api.properties(target_of_admiration='anya')
-```
-
-Note the call to `set_config`. This method takes the configuration name
-specifed, finds it in the given module (`'hello'` in this case), and sets
-`api.hello.c` equal to the result of invoking the named config item
-(`'default_tool'`) with the default configuration (the result of calling
-`get_config_defaults`), merged over the static defaults specified by the schema.
-
-We can also call `set_config` differently to get different results:
-
-```python
-# recipes/rainbow_hello.py
-DEPS = ['hello']
-def RunSteps(api):
-  api.hello.set_config('super_tool', TARGET='Charlie')
-  api.hello.greet()  # Greets 'Charlie' with unicorn.py.
-
-def GenTests(api):
-  yield api.test('charlie')
-```
-
-```python
-# recipes/evil_hello.py
-DEPS = ['hello']
-def RunSteps(api):
-  api.hello.set_config('default_tool', TARGET='DarthVader')
-  api.hello.greet()  # Causes 'DarthVader' to despair with echo
-
-def GenTests(api):
-  yield api.test('darth')
-```
-
-`set_config()` also has one additional bit of magic. If a module (say,
-`chromium`), depends on some other modules (say, `gclient`), if you do
-`api.chromium.set_config('blink')`, it will apply the `'blink'` config item from
-the chromium module, but it will also attempt to apply the `'blink'` config for
-all the dependencies, too. This way, you can have the chromium module extend the
-gclient config context with a 'blink' config item, and then `set_configs` will
-stack across all the relevent contexts.  (This has since been recognized as a
-design mistake)
-
-`recipe_api.RecipeApi` also provides `make_config` and `apply_config`, which
-allow recipes more-direct access to the config items. However, `set_config()` is
-the most-preferred way to apply configurations.
-
-## What about getting data back from a step?
-
-Consider this recipe:
-
-```python
-DEPS = ['step', 'path']
-
-def RunSteps(api):
-  step_result = api.step('Determine blue moon',
-      [api.path['build'].join('is_blue_moon.sh')])
-
-  if step_result.retcode == 0:
-    api.step('HARLEM SHAKE!', [api.path['build'].join('do_the_harlem_shake.sh')])
-  else:
-    api.step('Boring', [api.path['build'].join('its_a_small_world.sh')])
-
-def GenTests(api):
-  yield api.test('harlem') + api.step_data('Determine blue moon', retcode=0)
-  yield api.test('boring') + api.step_data('Determine blue moon', retcode=1)
-```
-
-See how we use `step_result` to get the result of the last step? The item we get
-back is a `recipe_engine.main.StepData` instance (really, just a basic object
-with member data). The members of this object which are guaranteed to exist are:
-  * `retcode`: Pretty much what you think
-  * `step`: The actual step json which was sent to `annotator.py`. Not usually
-    useful for recipes, but it is used internally for the recipe tests
-    framework.
-  * `presentation`: An object representing how the step will show up on the
-    build page, including its exit status, links, and extra log text.  This is a
-    `recipe_engine.main.StepPresentation` object.
-    See also
-    [How to change step presentation](#how-to-change-step-presentation).
-
-This is pretty neat... However, it turns out that returncodes suck bigtime for
-communicating actual information. `api.json.output()` to the rescue!
-
-```python
-DEPS = ['step', 'path', 'step_history', 'json']
-
-def RunSteps(api):
-  step_result = api.step(
-      'run tests',
-      [api.path['build'].join('do_test_things.sh'), api.json.output()])
-  num_passed = step_result.json.output['num_passed']
-  if num_passed > 500:
-    api.step('victory', [api.path['build'].join('do_a_dance.sh')])
-  elif num_passed > 200:
-    api.step('not defeated', [api.path['build'].join('woohoo.sh')])
-  else:
-    api.step('deads!', [api.path['build'].join('you_r_deads.sh')])
-
-def GenTests(api):
-  yield (api.test('winning') +
-         api.step_data('run tests', api.json.output({'num_passed': 791}))
-  yield (api.test('not_dead_yet') +
-         api.step_data('run tests', api.json.output({'num_passed': 302}))
-  yield (api.test('noooooo') +
-         api.step_data('run tests', api.json.output({'num_passed': 10})))
-```
-
-### How does THAT work!?
-
-`api.json.output()` returns a `recipe_api.Placeholder` which is meant to be
-added into a step command list. When the step runs, the placeholder gets
-rendered into some strings (in this case, like '/tmp/some392ra8'). When the step
-finishes, the Placeholder adds data to the `StepData` object for the step which
-just ran, namespaced by the module name (in this case, the 'json' module decided
-to add an 'output' attribute to the `step_history` item). I'd encourage you to
-take a peek at the implementation of the json module to see how this is
-implemented.
-
-### Example: write to standard input of a step
-
-```python
-api.step(..., stdin=api.raw_io.input('test input'))
-```
-
-Also see [raw_io's
-example.py](https://chromium.googlesource.com/chromium/tools/build.git/+/master/scripts/slave/recipe_modules/raw_io/example.py).
-
-### Example: read standard output of a step as json
-
-```python
-step_result = api.step(..., stdout=api.json.output())
-data = step_result.stdout
-# data is a parsed JSON value, such as dict
-```
-
-Also see [json's
-example.py](https://chromium.googlesource.com/chromium/tools/build.git/+/master/scripts/slave/recipe_modules/json/example.py).
-
-### Example: write to standard input of a step as json
-
-```python
-data = {'value': 1}
-api.step(..., stdin=api.json.input(data))
-```
-
-Also see [json's
-example.py](https://chromium.googlesource.com/chromium/tools/build.git/+/master/scripts/slave/recipe_modules/json/example.py).
-
-### Example: simulated step output
-
-This example specifies the standard output that should be returned when
-a step is executed in simulation mode. This is typically used for
-specifying default test data in the recipe or recipe module and removes
-the need to specify too much test data for each test in GenTests:
-
-```python
-api.step(..., step_test_data=api.raw_io.output('test data'))
-```
-
-### Example: simulated step output for a test case
-
-```python
-yield (
-    api.test('my_test') +
-    api.step_data(
-        'step_name',
-        output=api.raw_io.output('test data')))
-```
-
-## How to change step presentation?
-
-`step_result.presentation` allows modifying the appearance of a step:
-
-### Logging
-
-```python
-step_result.presentation.logs['mylog'] = ['line1', 'line2']
-```
-
-Creates an extra log "mylog" under the step.
-
-### Setting properties
-
-`api.properties` are immutable, but you can change and add new
-properties at the buildbot level.
-
-```python
-step_result.presentation.properties['newprop'] = 1
-```
-
-### Example: step text
-
-This modifies the text displayed next to a step name:
-
-```python
-step_result = api.step(...)
-step_result.presentation.step_text = 'Dynamic step result text'
-```
-
-* `presentaton.logs` allows creating extra logs of a step run. Example:
-  ```python
-  step_result.presentation.logs['mylog'] = ['line1', 'line2']
-  ```
-* presentation.properties allows changing and adding new properties at the
-  buildbot level. Example:
-  ```python
-  step_result.presentation.properties['newprop'] = 1
-  ```
-
-## How do I know what modules to use?
-
-Use `scripts/tools/show_me_the_modules.py`. It's super effective!
-
-## How do I run those tests you were talking about?
-
-To test all the recipes/apis, use
-`scripts/slave/unittests/recipe_simulation_test.py`.  To set new expectations
-`scripts/slave/unittests/recipe_simulation_test.py train`.
-
-## Where's the docs on `*.py`?
-
-Check the docstrings in `*.py`. `<trollface text="Problem?"/>`
-
-In addition, most recipe modules have an `example.py` file which exercises most
-of the code in the module for both test coverage and example purposes.
-
-If you want to know what keys a step dictionary can take, take a look at
-`third_party/recipe_engine/main.py`.
-