Have service_manager start a cloudtail attached to each service it starts.

infra_libs/experiments.py

+# Copyright 2015 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+"""Tools for gradually enabling a feature on a deterministic set of machines.
+
+Add a flag to your program to control the percentage of machines that a new
+feature should be enabled on::
+
+    def add_argparse_options(self, parser):
+      parser.add_argument('--myfeature-percent', type=int, default=0)
+
+    def main(self, opts):
+      if experiments.is_active_for_host('myfeature', opts.myfeature_percent):
+        # do myfeature
+"""
+
+import hashlib
+import logging
+import socket
+
+
+def _is_active(labels, percent):
+  h = hashlib.md5()
+  for label, value in labels.iteritems():

[Vadim Sh., 2015/09/23 17:40:22]

sorted(labels.iteritems())

[dsansome, 2015/09/24 04:34:13]

Done.

+    h.update(label)
+    h.update(value)
+
+  digest = h.digest()
+
+  print hash(digest) % 100

[Vadim Sh., 2015/09/23 17:40:22]

remove :)

[dsansome, 2015/09/24 04:34:13]

Oops!

+  return (hash(digest) % 100) < percent
+
+
+def is_active_for_host(experiment_name, percent):
+  ret = _is_active({
+      'name': experiment_name,
+      'host': socket.getfqdn(),

[Vadim Sh., 2015/09/23 17:40:21]

This thing can return nonsense (e.g. 1.1.1.1.1.1.arp.something-something) or
throw exceptions if it is called before network is fully up. Just saying :) I
think our upstart config sets up a dependency on eth0.

[dsansome, 2015/09/24 04:34:13]

Nonsense is fine - it's just being hashed anyway.
ts_mon.config also does a socket.getfqdn before this, so if that one worked this
one will too :)

+  }, percent)
+
+  if ret:
+    logging.info('Experiment "%s" is active', experiment_name)
+
+  return ret