Move ENV into custom deps, allow it to be random.

Move ENV into custom deps, allow it to be random.

When a custom deps directory is specified, install the bootstrap
environment into that directory rather than into the main script
directory. This makes parallel invocations of recipe engine (e.g., for
recipe testing) more independent.

Add a special "--deps-path=-" option to instruct the recipe engine to
create a temporary deps directory, operate out of it, and clean it up at
termination. This will make using hermetic recipe test spaces much more
accessible to various testing scripts.

BUG=chromium:666404
TEST=local
  - Ran with "--use-bootstrap", still installs ENV locally.
  - Ran with "--use-bootstrap" and "--deps-dir /tmp/fake", works,
    installs ENV there.
  - Ran with "--use-bootstrap" and "--deps-dir=-", works, cleans up
    afterwards.
  - Ran without "--use-bootstrap" everything seems to be working.

Committed: https://github.com/luci/recipes-py/commit/67fe1f3ccee14c273b3d7d6235280497b4413cba

[dnj, 2016-11-17 19:25:37]

dnj@chromium.org changed reviewers:
+ iannucci@chromium.org, martiniss@chromium.org

[dnj, 2016-11-17 19:25:37]

PTAL. I'm hoping that by isolating recipe runs into temporary spaces, we can
have more hermetic testing.

[iannucci, 2016-11-17 21:15:04]

lgtm

https://codereview.chromium.org/2511213002/diff/1/recipes.py
File recipes.py (right):

https://codereview.chromium.org/2511213002/diff/1/recipes.py#newcode584
recipes.py:584: logging.debug('Using custom deps path: %s', args.deps_path)
let's make this warning?

https://codereview.chromium.org/2511213002/diff/1/recipes.py#newcode592
recipes.py:592: env_path = os.path.join(args.deps_path, 'ENV')
let's just make it .ENV

https://codereview.chromium.org/2511213002/diff/1/recipes.py#newcode624
recipes.py:624: shutil.rmtree(temp_deps_dir)
can we log the warning instead of throwing if this fails

[dnj, 2016-11-17 21:27:57]

https://codereview.chromium.org/2511213002/diff/1/recipes.py
File recipes.py (right):

https://codereview.chromium.org/2511213002/diff/1/recipes.py#newcode584
recipes.py:584: logging.debug('Using custom deps path: %s', args.deps_path)
On 2016/11/17 21:15:04, iannucci wrote:
> let's make this warning?

Done.

https://codereview.chromium.org/2511213002/diff/1/recipes.py#newcode592
recipes.py:592: env_path = os.path.join(args.deps_path, 'ENV')
On 2016/11/17 21:15:04, iannucci wrote:
> let's just make it .ENV

Done.

https://codereview.chromium.org/2511213002/diff/1/recipes.py#newcode624
recipes.py:624: shutil.rmtree(temp_deps_dir)
On 2016/11/17 21:15:04, iannucci wrote:
> can we log the warning instead of throwing if this fails

Done.

[dnj, 2016-11-17 21:28:01]

The CQ bit was checked by dnj@chromium.org

[dnj, 2016-11-17 21:28:02]

The patchset sent to the CQ was uploaded after l-g-t-m from
iannucci@chromium.org
Link to the patchset: https://codereview.chromium.org/2511213002/#ps20001
(title: "Comments.")

[commit-bot: I haz the power, 2016-11-17 21:28:10]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-11-17 21:31:46]

Description was changed from

==========
Move ENV into custom deps, allow it to be random.

When a custom deps directory is specified, install the bootstrap
environment into that directory rather than into the main script
directory. This makes parallel invocations of recipe engine (e.g., for
recipe testing) more independent.

Add a special "--deps-path=-" option to instruct the recipe engine to
create a temporary deps directory, operate out of it, and clean it up at
termination. This will make using hermetic recipe test spaces much more
accessible to various testing scripts.

BUG=chromium:666404
TEST=local
  - Ran with "--use-bootstrap", still installs ENV locally.
  - Ran with "--use-bootstrap" and "--deps-dir /tmp/fake", works,
    installs ENV there.
  - Ran with "--use-bootstrap" and "--deps-dir=-", works, cleans up
    afterwards.
  - Ran without "--use-bootstrap" everything seems to be working.
==========

to

==========
Move ENV into custom deps, allow it to be random.

When a custom deps directory is specified, install the bootstrap
environment into that directory rather than into the main script
directory. This makes parallel invocations of recipe engine (e.g., for
recipe testing) more independent.

Add a special "--deps-path=-" option to instruct the recipe engine to
create a temporary deps directory, operate out of it, and clean it up at
termination. This will make using hermetic recipe test spaces much more
accessible to various testing scripts.

BUG=chromium:666404
TEST=local
  - Ran with "--use-bootstrap", still installs ENV locally.
  - Ran with "--use-bootstrap" and "--deps-dir /tmp/fake", works,
    installs ENV there.
  - Ran with "--use-bootstrap" and "--deps-dir=-", works, cleans up
    afterwards.
  - Ran without "--use-bootstrap" everything seems to be working.

Committed:
https://github.com/luci/recipes-py/commit/67fe1f3ccee14c273b3d7d6235280497b44...
==========

[commit-bot: I haz the power, 2016-11-17 21:31:47]

Committed patchset #2 (id:20001) as
https://github.com/luci/recipes-py/commit/67fe1f3ccee14c273b3d7d6235280497b44...

[Michael Moss, 2016-12-02 17:08:57]

mmoss@chromium.org changed reviewers:
+ mmoss@chromium.org

[Michael Moss, 2016-12-02 17:08:57]

https://codereview.chromium.org/2511213002/diff/1/recipes.py
File recipes.py (right):

https://codereview.chromium.org/2511213002/diff/1/recipes.py#newcode584
recipes.py:584: logging.debug('Using custom deps path: %s', args.deps_path)
On 2016/11/17 21:15:04, iannucci wrote:
> let's make this warning?

Am I doing something wrong, or is expected that I see this warning every time I
run recipe_simulation_test.py nowadays?

[dnj, 2016-12-02 17:20:00]

On 2016/12/02 17:08:57, Michael Moss wrote:
> https://codereview.chromium.org/2511213002/diff/1/recipes.py
> File recipes.py (right):
> 
> https://codereview.chromium.org/2511213002/diff/1/recipes.py#newcode584
> recipes.py:584: logging.debug('Using custom deps path: %s', args.deps_path)
> On 2016/11/17 21:15:04, iannucci wrote:
> > let's make this warning?
> 
> Am I doing something wrong, or is expected that I see this warning every time
I
> run recipe_simulation_test.py nowadays?

Is it a bad warning? Log level warning is just b/c the default log level is
warning, and we want it to be displayed. If I could do "info +force" that'd be
better, but *shrug*

[Michael Moss, 2016-12-02 17:28:48]

On 2016/12/02 17:20:00, dnj wrote:
> On 2016/12/02 17:08:57, Michael Moss wrote:
> > https://codereview.chromium.org/2511213002/diff/1/recipes.py
> > File recipes.py (right):
> > 
> > https://codereview.chromium.org/2511213002/diff/1/recipes.py#newcode584
> > recipes.py:584: logging.debug('Using custom deps path: %s', args.deps_path)
> > On 2016/11/17 21:15:04, iannucci wrote:
> > > let's make this warning?
> > 
> > Am I doing something wrong, or is expected that I see this warning every
time
> I
> > run recipe_simulation_test.py nowadays?
> 
> Is it a bad warning? Log level warning is just b/c the default log level is
> warning, and we want it to be displayed. If I could do "info +force" that'd be
> better, but *shrug*

It's bad in that I have no idea if I should care (hence, "am I doing something
wrong"), or what to do about it if I should. If it's something that is always
expected to happen with that tool, it seems odd and confusing to warn about it.

[dnj, 2016-12-02 17:39:54]

On 2016/12/02 17:28:48, Michael Moss wrote:
> On 2016/12/02 17:20:00, dnj wrote:
> > On 2016/12/02 17:08:57, Michael Moss wrote:
> > > https://codereview.chromium.org/2511213002/diff/1/recipes.py
> > > File recipes.py (right):
> > > 
> > > https://codereview.chromium.org/2511213002/diff/1/recipes.py#newcode584
> > > recipes.py:584: logging.debug('Using custom deps path: %s',
args.deps_path)
> > > On 2016/11/17 21:15:04, iannucci wrote:
> > > > let's make this warning?
> > > 
> > > Am I doing something wrong, or is expected that I see this warning every
> time
> > I
> > > run recipe_simulation_test.py nowadays?
> > 
> > Is it a bad warning? Log level warning is just b/c the default log level is
> > warning, and we want it to be displayed. If I could do "info +force" that'd
be
> > better, but *shrug*
> 
> It's bad in that I have no idea if I should care (hence, "am I doing something
> wrong"), or what to do about it if I should. If it's something that is always
> expected to happen with that tool, it seems odd and confusing to warn about
it.

Really it's just useful info that is being printed. I don't know a good way to
print useful info without either changing the overall logging level or using a
logging level that is known to print by default. I could use "print", but that
doesn't behave nicely with the logging framework.

[Michael Moss, 2016-12-02 17:43:08]

On 2016/12/02 17:39:54, dnj wrote:
> On 2016/12/02 17:28:48, Michael Moss wrote:
> > On 2016/12/02 17:20:00, dnj wrote:
> > > On 2016/12/02 17:08:57, Michael Moss wrote:
> > > > https://codereview.chromium.org/2511213002/diff/1/recipes.py
> > > > File recipes.py (right):
> > > > 
> > > > https://codereview.chromium.org/2511213002/diff/1/recipes.py#newcode584
> > > > recipes.py:584: logging.debug('Using custom deps path: %s',
> args.deps_path)
> > > > On 2016/11/17 21:15:04, iannucci wrote:
> > > > > let's make this warning?
> > > > 
> > > > Am I doing something wrong, or is expected that I see this warning every
> > time
> > > I
> > > > run recipe_simulation_test.py nowadays?
> > > 
> > > Is it a bad warning? Log level warning is just b/c the default log level
is
> > > warning, and we want it to be displayed. If I could do "info +force"
that'd
> be
> > > better, but *shrug*
> > 
> > It's bad in that I have no idea if I should care (hence, "am I doing
something
> > wrong"), or what to do about it if I should. If it's something that is
always
> > expected to happen with that tool, it seems odd and confusing to warn about
> it.
> 
> Really it's just useful info that is being printed. I don't know a good way to
> print useful info without either changing the overall logging level or using a
> logging level that is known to print by default. I could use "print", but that
> doesn't behave nicely with the logging framework.

I guess I just disagree that it's all that useful, at least as a warning, since
it is apparently not (ever?) a problem. I probably would have preferred the
original logging.debug() message, but, eh.

[dnj, 2016-12-02 21:37:48]

On 2016/12/02 17:43:08, Michael Moss wrote:
> On 2016/12/02 17:39:54, dnj wrote:
> > On 2016/12/02 17:28:48, Michael Moss wrote:
> > > On 2016/12/02 17:20:00, dnj wrote:
> > > > On 2016/12/02 17:08:57, Michael Moss wrote:
> > > > > https://codereview.chromium.org/2511213002/diff/1/recipes.py
> > > > > File recipes.py (right):
> > > > > 
> > > > >
https://codereview.chromium.org/2511213002/diff/1/recipes.py#newcode584
> > > > > recipes.py:584: logging.debug('Using custom deps path: %s',
> > args.deps_path)
> > > > > On 2016/11/17 21:15:04, iannucci wrote:
> > > > > > let's make this warning?
> > > > > 
> > > > > Am I doing something wrong, or is expected that I see this warning
every
> > > time
> > > > I
> > > > > run recipe_simulation_test.py nowadays?
> > > > 
> > > > Is it a bad warning? Log level warning is just b/c the default log level
> is
> > > > warning, and we want it to be displayed. If I could do "info +force"
> that'd
> > be
> > > > better, but *shrug*
> > > 
> > > It's bad in that I have no idea if I should care (hence, "am I doing
> something
> > > wrong"), or what to do about it if I should. If it's something that is
> always
> > > expected to happen with that tool, it seems odd and confusing to warn
about
> > it.
> > 
> > Really it's just useful info that is being printed. I don't know a good way
to
> > print useful info without either changing the overall logging level or using
a
> > logging level that is known to print by default. I could use "print", but
that
> > doesn't behave nicely with the logging framework.
> 
> I guess I just disagree that it's all that useful, at least as a warning,
since
> it is apparently not (ever?) a problem. I probably would have preferred the
> original logging.debug() message, but, eh.

It's not meant as a warning. The log level was chosen to make it render, not to
represent its severity.

It is useful on bots for sure to see where they are building out of. Locally,
yeah probably less useful. Not sure what the right thing to do here is.

[Michael Moss, 2016-12-02 21:51:33]

> It's not meant as a warning. The log level was chosen to make it render, not
to
> represent its severity.

Yeah, I understand the reasoning now, though, ya know, it still does shout
"WARNING" in the output. Maybe I just have an unusually hard time disassociating
that from something I should actually care about. Or I'm just too damn literal.
 
> It is useful on bots for sure to see where they are building out of. Locally,
> yeah probably less useful. Not sure what the right thing to do here is.

Is there a reason bots can't run at 'info' or 'debug' log level?

[dnj, 2016-12-02 23:42:03]

> > It is useful on bots for sure to see where they are building out of.
Locally,
> > yeah probably less useful. Not sure what the right thing to do here is.
> 
> Is there a reason bots can't run at 'info' or 'debug' log level?

Those levels both show more log messages than we want to show in a standard
BuildBot run.