Add support for LUCI_CONTEXT.

Add support for LUCI_CONTEXT.

R=iannucci@chromium.org
BUG=731854

[Vadim Sh., 2017-07-29 00:40:55]

This is unfinished sketch. Please let me know what you think. Is it too deep
integration? Did you have something else in mind?

https://codereview.chromium.org/2985323002/diff/1/recipe_engine/step_runner.py
File recipe_engine/step_runner.py (right):

https://codereview.chromium.org/2985323002/diff/1/recipe_engine/step_runner.p...
recipe_engine/step_runner.py:340: with luci_ctx.replace(luci_context) as path:
no changes here except lines 340-343
Rietveld is too dumb to realize.

[iannucci, 2017-08-04 18:55:51]

I think this level of integration looks right to me.

https://codereview.chromium.org/2985323002/diff/1/recipe_engine/recipe_api.py
File recipe_engine/recipe_api.py (right):

https://codereview.chromium.org/2985323002/diff/1/recipe_engine/recipe_api.py...
recipe_engine/recipe_api.py:270: 'luci_context',  # it's ambient and used only
for advanced stuff
This blacklist is used for excluding the entry from the expectations; I don't
think we want that.

Additionally, you may want to tweak
https://github.com/luci/recipes-py/blob/master/recipe_engine/step_runner.py#L290
to prevent it from dumping this to stdout (e.g. see the little preamble thingy
here:
https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Ftryserver.chromium.linux...
).

Can you also add a test (e.g. to
https://github.com/luci/recipes-py/blob/master/recipe_engine/unittests/run_te...)
which runs some luci_context-using recipe for real and asserts that it doesn't
show up in stdout? E.g. put some value like "super_secret_token" in it and
assert that it's not in stdout. Regressing this would be bad.

https://codereview.chromium.org/2985323002/diff/1/recipe_engine/step_runner.py
File recipe_engine/step_runner.py (right):

https://codereview.chromium.org/2985323002/diff/1/recipe_engine/step_runner.p...
recipe_engine/step_runner.py:321: luci_context: full value of LUCI_CONTEXT dict.
Maybe "entire intended value of the LUCI_CONTEXT dict for this step. e.g. if
this is empty, then the step will be run without any LUCI_CONTEXT parameters at
all"

https://codereview.chromium.org/2985323002/diff/1/recipe_engine/step_runner.p...
recipe_engine/step_runner.py:341: if path:
nit: I would rename path to luci_ctx_path or something, since this with
statement holds a bunch of lines where 'path' may be ambiguous

https://codereview.chromium.org/2985323002/diff/1/recipe_modules/context/api.py
File recipe_modules/context/api.py (right):

https://codereview.chromium.org/2985323002/diff/1/recipe_modules/context/api....
recipe_modules/context/api.py:78: * luci_context (dict) - LUCI_CONTEXT
overrides.
"see below for more info"

https://codereview.chromium.org/2985323002/diff/1/recipe_modules/context/api....
recipe_modules/context/api.py:131: there's no equivalent of %(PATH)s trick.
we're going to delete the %(PATH) trick anyway, so I would remove that bit.

I would also mention that the values must be either dict (to replace that value)
or None (to remove that value).

https://codereview.chromium.org/2985323002/diff/1/recipe_modules/context/api....
recipe_modules/context/api.py:208: new[k] = copy.deepcopy(v)
can we assert that v is serializable to json? Otherwise we'll get a weird error
from somewhere deeper in the stack.

https://codereview.chromium.org/2985323002/diff/1/recipe_modules/context/api....
recipe_modules/context/api.py:267: Do not dump the entirety of LUCI_CONTEXT into
logs, it may contain secrets.
s/may contain/contains

make them scared :)

https://codereview.chromium.org/2985323002/diff/1/recipe_modules/context/exam...
File recipe_modules/context/examples/full.expected/basic.json (right):

https://codereview.chromium.org/2985323002/diff/1/recipe_modules/context/exam...
recipe_modules/context/examples/full.expected/basic.json:128: },
the blacklist prevents the luci_config stuff from showing up here; I don't think
that's what we want.

https://codereview.chromium.org/2985323002/diff/1/recipe_modules/context/test...
File recipe_modules/context/test_api.py (right):

https://codereview.chromium.org/2985323002/diff/1/recipe_modules/context/test...
recipe_modules/context/test_api.py:11: """Completely mocks LUCI_CONTEXT
values."""
I would say something like "Sets the initial LUCI_CONTEXT values for the test."
I would also do **luci_ctx_values instead of ctx.