Add master_manager_launch script which launches master_manager scripts for each master on a host.

infra/tools/master_manager_launcher/__main__.py

Index: infra/tools/master_manager_launcher/__main__.py
diff --git a/infra/tools/master_manager_launcher/__main__.py b/infra/tools/master_manager_launcher/__main__.py
new file mode 100644
index 0000000000000000000000000000000000000000..c4023d41f59600c214bced23d577842bd2d42153
--- /dev/null
+++ b/infra/tools/master_manager_launcher/__main__.py
@@ -0,0 +1,123 @@
+#!/usr/bin/python

[agable, 2015/05/06 23:29:30]

infra/tools is meant to hold things for people to run one-off. infra/services is
meant to hold things that will be invoked in an automated fashion. Since I'm
pretty sure the whole point of this is be run automatically, shouldn't this live
in services/?

[ghost stip (do not use), 2015/05/07 07:07:21]

Done.

+# Copyright 2015 Google Inc. All Rights Reserved.
+# pylint: disable=F0401
+
+"""Launch a master_manager script for every master on a host."""
+
+import argparse
+import json
+import logging
+import operator
+import os
+import socket
+import subprocess
+import sys
+
+from infra.libs import logs
+from infra.libs.service_utils import daemon
+from infra.services.master_lifecycle import buildbot_state
+from infra.tools.master_manager_launcher import desired_state_parser
+
+
+SCRIPT_DIR = os.path.abspath(os.path.dirname(__file__))
+RUNPY = os.path.abspath(os.path.join(
+  SCRIPT_DIR, os.pardir, os.pardir, os.pardir, 'run.py'))
+
+
+def parse_args():  # pragma: no cover
+  parser = argparse.ArgumentParser(
+      description='Launches master_manager for every master on a host. NOTE: '
+                  'does not perform any action unless --prod is set.')
+  parser.add_argument('build_dir', nargs='?',
+      help='location of the tools/build directory')
+  parser.add_argument('--hostname',
+      default=socket.getfqdn(),
+      help='override local hostname (currently %(default)s)')
+  parser.add_argument('--json-location',
+      default=os.path.join(SCRIPT_DIR, 'desired_master_state.json'),

[agable, 2015/05/06 23:29:30]

Seems like an odd default. Why would the json be in this directory?

[ghost stip (do not use), 2015/05/07 07:07:21]

made it just the current dir.

+      help='desired master state configuration (default: %(default)s)')
+  parser.add_argument('--command-timeout',
+      help='apply a timeout in seconds to each master_manager process')
+  parser.add_argument('--verify', action='store_true',
+      help='verify the desired master state JSON is valid, then exit')
+  parser.add_argument('--prod', action='store_true',
+      help='actually perform actions instead of doing a dry run')
+  logs.add_argparse_options(parser)
+
+  args = parser.parse_args()
+  logs.process_argparse_options(args)
+
+  if not args.verify:
+    if not args.build_dir:
+      parser.error('A build/ directory must be specified.')
+
+  return args
+
+
+def synthesize_master_manager_cli(master_dict, hostname, prod=False):

[agable, 2015/05/06 23:29:30]

It's not a CLI, it's a command.

[ghost stip (do not use), 2015/05/07 07:07:21]

Done.

+  """Find the current desired state and synthesize a command for the master."""
+  state = desired_state_parser.get_master_state(master_dict['states'])
+  cli = [
+      RUNPY,
+      'infra.tools.master_manager',
+      master_dict['fulldir'],
+      str(state['desired_state']),
+      str(state['transition_time_utc']),
+      '--hostname', hostname,
+      '--enable-gclient-sync',
+      '--verbose',
+  ]
+
+  if prod:
+    cli.append('--prod')
+
+  return cli
+
+
+def log_triggered_ignored(triggered, ignored, hostname, logger):

[agable, 2015/05/06 23:29:30]

When/how is this covered?

[ghost stip (do not use), 2015/05/07 06:50:57]

actually, the whole file doesn't have a test (since I moved most of the logic to
desired_state_parser.py, which *is* tested.

I've moved the pragma: no cover to show humans this fact. do we have a better
way to signify 'whole file is untested'?

[agable, 2015/05/07 17:59:31]

Nope, there's not a better way to show "the whole file is uncovered".
Unfortnately, putting "#pragma: no cover" on the first line doesn't apply to the
entire file :(

+  """Outputs for humans which masters will be managed and which won't."""
+  if ignored:
+    logger.info(
+        '%d masters on host %s left unmanaged (no desired state section):\n%s',
+        len(ignored), hostname, '\n'.join(ignored))
+
+  triggered_master_string = '.'
+  if triggered:
+    triggered_master_string = ':\n'
+  triggered_master_string += '\n'.join(m['dirname'] for m in triggered)
+  logger.info(
+      '%d masters managed for host %s%s',
+      len(triggered), hostname, triggered_master_string)
+
+
+def main():  # pragma: no cover
+  args = parse_args()
+
+  desired_state = desired_state_parser.load_desired_state_file(
+      args.json_location)
+
+  logger = logging.getLogger(__name__)

[agable, 2015/05/06 23:29:30]

This shouldn't be necessary for the top-level __main__ script, only for
libraries. __main__ can just set up the logging argparse args, and then use
'logging.foo('a string')' iirc.

[iannucci, 2015/05/06 23:48:05]

Additionally, logger is actually one of the (very few) things which should be a
module-level global.

[ghost stip (do not use), 2015/05/07 06:50:57]

Done.

+
+  if desired_state is None:

[agable, 2015/05/06 23:29:30]

Why is not having a desired state an error? The log_triggered_ignored function
above just thinks the same error (afaict) is only worthy of an info-level log.

[ghost stip (do not use), 2015/05/07 06:50:57]

there's a difference between 'I've haven't written anything about this master,
ignore it' versus 'you wrote a configuration file that is invalid.'
load_desired_state_file() now raises an exception on a bad config, so these two
lines go away

+    return 1
+  if args.verify:
+    return 0  # File checks out, no need to continue.
+
+  triggered, ignored = desired_state_parser.get_masters_for_host(
+      desired_state, args.build_dir, args.hostname)
+  log_triggered_ignored(triggered, ignored, args.hostname, logger)
+
+  commands = [
+      synthesize_master_manager_cli(m, args.hostname, prod=args.prod)
+      for m in triggered]

[agable, 2015/05/06 23:29:30]

nit: outdent closing brace on next line (like you did for
synthesize_master_command)

[ghost stip (do not use), 2015/05/07 07:07:21]

Done.

+
+  if args.command_timeout:
+    commands = [daemon.add_timeout(c, args.command_timeout) for c in commands]
+
+  for command in commands:
+    logger.info('running %s' % command)
+    subprocess.call(command)

[agable, 2015/05/06 23:29:30]

Yeah we'll definitely need parallelism here.

[ghost stip (do not use), 2015/05/07 06:50:57]

Acknowledged.

+
+
+if __name__ == '__main__':  # pragma: no cover
+  sys.exit(main())