Add master_cleaner tool.

Add master_cleaner tool.

Add the master_cleaner tool, which aims to clean production master
directories by removing unused builder directories and old "twistd.log"
files.

BUG=chromium:586543
TEST=None

Committed: https://chromium.googlesource.com/infra/infra/+/793ded1c26d30e59c68c83bc6a9fd7bdb327a312

[dnj (Google), 2016-06-10 17:09:20]

dnj@google.com changed reviewers:
+ dnj@google.com, friedman@chromium.org, nodir@chromium.org

[dnj (Google), 2016-06-10 17:09:21]

PTAL, need basic sanity check.

[nodir, 2016-06-10 17:50:36]

https://codereview.chromium.org/2059833002/diff/1/infra/tools/master_cleaner/...
File infra/tools/master_cleaner/__main__.py (right):

https://codereview.chromium.org/2059833002/diff/1/infra/tools/master_cleaner/...
infra/tools/master_cleaner/__main__.py:2: # Copyright 2015 Google Inc. All
Rights Reserved.
2016

https://codereview.chromium.org/2059833002/diff/1/infra/tools/master_cleaner/...
infra/tools/master_cleaner/__main__.py:41: cmd, cwd, stdout, stderr)
include exit code

https://codereview.chromium.org/2059833002/diff/1/infra/tools/master_cleaner/...
infra/tools/master_cleaner/__main__.py:48: description='Cleanup directories on
BuildBot master systems.')
__doc__

https://codereview.chromium.org/2059833002/diff/1/infra/tools/master_cleaner/...
infra/tools/master_cleaner/__main__.py:75: # directories.
two negations (filter and aren't), hard to understand

https://codereview.chromium.org/2059833002/diff/1/infra/tools/master_cleaner/...
infra/tools/master_cleaner/__main__.py:124: age =
datetime.datetime.fromtimestamp(os.path.getctime(path))
I think this should use twistd_log_files[mid][1]
otherwise you sort by second value in tuple, but binary-search by something else
also less io

https://codereview.chromium.org/2059833002/diff/1/infra/tools/master_cleaner/...
infra/tools/master_cleaner/__main__.py:124: age =
datetime.datetime.fromtimestamp(os.path.getctime(path))
it is not age (the older the file, the bigger the age), but rather birth date
(the older the file, the lower the birth date): direction is reversed

https://codereview.chromium.org/2059833002/diff/1/infra/tools/master_cleaner/...
infra/tools/master_cleaner/__main__.py:179: 'mastername':
os.path.split(master_dir)[1],
this should be -1

https://codereview.chromium.org/2059833002/diff/1/infra/tools/master_cleaner/...
infra/tools/master_cleaner/__main__.py:184:
result['builddirs'].add(bcfg['builddir'])
builddir is not always there, e.g. 
  ./scripts/tools/dump_master_cfg.py masters/master.tryserver.infra/ -
does not contain builddirs
i think it should be bcfg.get('builddir', bcfg['name'])

otherwise you may nuke a lot of build dirs

https://codereview.chromium.org/2059833002/diff/1/infra/tools/master_cleaner/...
infra/tools/master_cleaner/__main__.py:228: for master in
sorted(set(opts.master)):
why set? you sort it anyway

https://codereview.chromium.org/2059833002/diff/1/infra_libs/time_functions/p...
File infra_libs/time_functions/parser.py (right):

https://codereview.chromium.org/2059833002/diff/1/infra_libs/time_functions/p...
infra_libs/time_functions/parser.py:29: tokens comprising the timedelta (e.g.,
10d,2h is 10 days, 2 hours).
yay go duration format

https://codereview.chromium.org/2059833002/diff/1/infra_libs/time_functions/p...
infra_libs/time_functions/parser.py:47: def timedelta_type(v):
consider changing the name so it is obvious that this is about argparse

[dnj (Google), 2016-06-10 18:00:52]

Thanks for the review!

https://codereview.chromium.org/2059833002/diff/1/infra/tools/master_cleaner/...
File infra/tools/master_cleaner/__main__.py (right):

https://codereview.chromium.org/2059833002/diff/1/infra/tools/master_cleaner/...
infra/tools/master_cleaner/__main__.py:41: cmd, cwd, stdout, stderr)
On 2016/06/10 17:50:36, nodir wrote:
> include exit code

Done.

https://codereview.chromium.org/2059833002/diff/1/infra/tools/master_cleaner/...
infra/tools/master_cleaner/__main__.py:48: description='Cleanup directories on
BuildBot master systems.')
On 2016/06/10 17:50:36, nodir wrote:
> __doc__

Done.

https://codereview.chromium.org/2059833002/diff/1/infra/tools/master_cleaner/...
infra/tools/master_cleaner/__main__.py:75: # directories.
On 2016/06/10 17:50:36, nodir wrote:
> two negations (filter and aren't), hard to understand

Done.

https://codereview.chromium.org/2059833002/diff/1/infra/tools/master_cleaner/...
infra/tools/master_cleaner/__main__.py:124: age =
datetime.datetime.fromtimestamp(os.path.getctime(path))
On 2016/06/10 17:50:36, nodir wrote:
> I think this should use twistd_log_files[mid][1]
> otherwise you sort by second value in tuple, but binary-search by something
else
> also less io

I think it's correct now.

https://codereview.chromium.org/2059833002/diff/1/infra/tools/master_cleaner/...
infra/tools/master_cleaner/__main__.py:179: 'mastername':
os.path.split(master_dir)[1],
On 2016/06/10 17:50:36, nodir wrote:
> this should be -1

Done.

https://codereview.chromium.org/2059833002/diff/1/infra/tools/master_cleaner/...
infra/tools/master_cleaner/__main__.py:184:
result['builddirs'].add(bcfg['builddir'])
On 2016/06/10 17:50:36, nodir wrote:
> builddir is not always there, e.g. 
>   ./scripts/tools/dump_master_cfg.py masters/master.tryserver.infra/ -
> does not contain builddirs
> i think it should be bcfg.get('builddir', bcfg['name'])
> 
> otherwise you may nuke a lot of build dirs

Done.

https://codereview.chromium.org/2059833002/diff/1/infra/tools/master_cleaner/...
infra/tools/master_cleaner/__main__.py:228: for master in
sorted(set(opts.master)):
On 2016/06/10 17:50:36, nodir wrote:
> why set? you sort it anyway

Just in case some person specifies the same master twice.

https://codereview.chromium.org/2059833002/diff/1/infra_libs/time_functions/p...
File infra_libs/time_functions/parser.py (right):

https://codereview.chromium.org/2059833002/diff/1/infra_libs/time_functions/p...
infra_libs/time_functions/parser.py:29: tokens comprising the timedelta (e.g.,
10d,2h is 10 days, 2 hours).
On 2016/06/10 17:50:36, nodir wrote:
> yay go duration format

Acknowledged.

https://codereview.chromium.org/2059833002/diff/1/infra_libs/time_functions/p...
infra_libs/time_functions/parser.py:47: def timedelta_type(v):
On 2016/06/10 17:50:36, nodir wrote:
> consider changing the name so it is obvious that this is about argparse

Done.

[nodir, 2016-06-10 18:05:48]

lgtm

[dnj (Google), 2016-06-10 18:06:52]

The CQ bit was checked by dnj@google.com

[commit-bot: I haz the power, 2016-06-10 18:06:57]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/2059833002/20001

[commit-bot: I haz the power, 2016-06-10 18:29:15]

Description was changed from

==========
Add master_cleaner tool.

Add the master_cleaner tool, which aims to clean production master
directories by removing unused builder directories and old "twistd.log"
files.

BUG=chromium:586543
TEST=None
==========

to

==========
Add master_cleaner tool.

Add the master_cleaner tool, which aims to clean production master
directories by removing unused builder directories and old "twistd.log"
files.

BUG=chromium:586543
TEST=None

Committed:
https://chromium.googlesource.com/infra/infra/+/793ded1c26d30e59c68c83bc6a9fd...
==========

[commit-bot: I haz the power, 2016-06-10 18:29:16]

Committed patchset #2 (id:20001) as
https://chromium.googlesource.com/infra/infra/+/793ded1c26d30e59c68c83bc6a9fd...