Add concept of bot config and test spec database (BotConfigAndTestDB).

scripts/slave/recipe_modules/chromium_tests/bot_config_and_test_db.py

Index: scripts/slave/recipe_modules/chromium_tests/bot_config_and_test_db.py
diff --git a/scripts/slave/recipe_modules/chromium_tests/bot_config_and_test_db.py b/scripts/slave/recipe_modules/chromium_tests/bot_config_and_test_db.py
new file mode 100644
index 0000000000000000000000000000000000000000..fa8967a2e5b51d1186dab27bf4045563a332b6d0
--- /dev/null
+++ b/scripts/slave/recipe_modules/chromium_tests/bot_config_and_test_db.py
@@ -0,0 +1,56 @@
+# Copyright 2016 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+import collections

[Sergey Berezin, 2016/01/08 18:54:13]

Not used, remove.

[Ken Russell (switch to Gerrit), 2016/01/08 23:02:44]

Done.

+
+class BotConfigAndTestDB(object):

[Sergiy Byelozyorov, 2016/01/08 10:35:57]

IMHO, this should be named BuilderConfigAndTestDB, because 'bot' is typically a
word for a specific machine on swarming ('slave' is used for machines used by
Buildbot). For example, get_bot_config below actually returns a builder config,
which also why it is passed 'buildername' parameter. Probably what was meant
here is a short version of 'trybot' that is used for builders used by CQ, but
since this module is used by both waterfall and trybots, I suggest using
'builder' term.

[Ken Russell (switch to Gerrit), 2016/01/08 23:02:44]

Sergiy and I talked about this during a Hangout this morning and I think
'builder' would be more confusing than 'bot'. 'builder' has a specific meaning
in this code (as opposed to 'tester'), but 'bot' is already used throughout the
code to describe both kinds of machines. We are in agreement to leave the naming
as is.

[Sergey Berezin, 2016/01/08 23:31:59]

Acknowledged.

+  """An immutable database of bot configurations and test specifications.

[Sergey Berezin, 2016/01/08 18:54:13]

nit: s/bot/builder/ in the docstring as well - agreed with Sergiy's comment on
terminology.

[Ken Russell (switch to Gerrit), 2016/01/08 23:02:44]

Please see above -- I think the term 'bot' should be used here, not 'builder'.

+  Holds the data for potentially multiple waterfalls (masternames). Most
+  queries against this database are made with (mastername, buildername)
+  pairs.
+  """
+
+  def __init__(self):
+    # Indexed by mastername. Each entry contains a master_dict and a
+    # test_spec.
+    self._db = {}
+    pass

[Paweł Hajdan Jr., 2016/01/08 16:00:42]

nit: This line shouldn't be needed now.

[Ken Russell (switch to Gerrit), 2016/01/08 23:02:44]

Done.

+
+  def _add_master_dict_and_test_spec(self, mastername, master_dict, test_spec):
+    """Only used during construction in chromium_tests.prepare_checkout. Do not
+    call this externally.
+

[Sergey Berezin, 2016/01/08 18:54:13]

nit: remove the empty line.

[Ken Russell (switch to Gerrit), 2016/01/08 23:02:44]

Done.

+    """
+    # TODO(kbr): currently the master_dicts that are created by
+    # get_master_dict_with_dynamic_tests are over-specialized to a
+    # particular builder -- the "enable_swarming" flag paradoxically comes
+    # from that builder, rather than from each individual builder and/or
+    # the parent builder. This needs to be fixed so that there's exactly
+    # one master_dict per waterfall.
+    assert mastername not in self._db, (
+        'Illegal attempt to add multiple master dictionaries for waterfall %s' %
+        (mastername))
+    self._db[mastername] = { 'master_dict': master_dict,
+                             'test_spec': test_spec }
+
+  def get_bot_config(self, mastername, buildername):
+    return self._db[mastername]['master_dict'].get('builders', {}).get(

[Sergey Berezin, 2016/01/08 18:54:13]

nit: if mastername is not in the _db, this will raise an exception. It's fine if
we expect the tests to catch all these errors. But otherwise I'd use
_db.get(mastername, {'master_dict': {}}).

[Ken Russell (switch to Gerrit), 2016/01/08 23:02:44]

There should be strong guarantees that a dictionary for this mastername should
already be in the database. I think it's better to raise an exception here if
the lookup fails, because the consequence is that compile targets or tests
silently disappear if there's a bug, rather than the recipe failing visibly.

[Sergey Berezin, 2016/01/08 23:31:59]

Ack. Agreed, let's fail with exception here.

+        buildername)
+
+  def get_master_settings(self, mastername):
+    return self._db[mastername]['master_dict'].get('settings', {})

[Sergey Berezin, 2016/01/08 18:54:13]

nit: same as above.

[Ken Russell (switch to Gerrit), 2016/01/08 23:02:44]

Same reply.

+
+  def bot_configs_matching_parent_buildername(
+      self, mastername, parent_buildername):
+    """A generator of all the (buildername, bot_config) tuples whose
+    parent_buildername is the passed one on the given master.
+

[Sergey Berezin, 2016/01/08 18:54:13]

nit: remove the empty line.

[Ken Russell (switch to Gerrit), 2016/01/08 23:02:45]

Done.

+    """
+    for buildername, bot_config in self._db[mastername]['master_dict'].get(
+        'builders', {}).iteritems():
+      if bot_config.get('parent_buildername') == parent_buildername:
+        yield buildername, bot_config
+
+  def get_test_spec(self, mastername, buildername):
+    return self._db[mastername]['test_spec'].get(buildername, {})

[Sergey Berezin, 2016/01/08 18:54:13]

nit: same as above.

[Ken Russell (switch to Gerrit), 2016/01/08 23:02:44]

Same reply.