submit_try: Obtain the list of trybots from the checked-in slaves.cfg

submit_try: Obtain the list of trybots from the checked-in slaves.cfg

This should drastically speed up the script.

BUG=

Committed: http://code.google.com/p/skia/source/detail?r=13071

[borenet, 2014-01-13 21:00:14]

This should be independent of https://codereview.chromium.org/136823002/ and
should make its testing less painful...

https://codereview.chromium.org/136683006/diff/1/tools/buildbot_globals.py
File tools/buildbot_globals.py (right):

https://codereview.chromium.org/136683006/diff/1/tools/buildbot_globals.py#ne...
tools/buildbot_globals.py:49: def retrieve_from_googlesource(url):
This function could really go into a utility module, since it's not strictly
related to buildbot global variables.

[borenet, 2014-01-14 16:20:17]

Friendly ping

[epoger, 2014-01-14 16:26:58]

On 2014/01/14 16:20:17, borenet wrote:
> Friendly ping

I figured I would let Ravi take first crack at this one, rather than jumping on
it right away

[rmistry, 2014-01-14 17:39:54]

LGTM

https://codereview.chromium.org/136683006/diff/1/tools/submit_try
File tools/submit_try (right):

https://codereview.chromium.org/136683006/diff/1/tools/submit_try#newcode38
tools/submit_try:38: ALL_ALIASES = [ALL_BUILDERS, COMPILE_BUILDERS, REGEX,
CQ_BUILDERS]
Why rearrange this? it was sorted before.

https://codereview.chromium.org/136683006/diff/1/tools/submit_try#newcode105
tools/submit_try:105: 'master/slaves.cfg')
Can we make this a top-level constant.

[borenet, 2014-01-14 18:00:05]

https://codereview.chromium.org/136683006/diff/1/tools/submit_try
File tools/submit_try (right):

https://codereview.chromium.org/136683006/diff/1/tools/submit_try#newcode38
tools/submit_try:38: ALL_ALIASES = [ALL_BUILDERS, COMPILE_BUILDERS, REGEX,
CQ_BUILDERS]
On 2014/01/14 17:39:54, rmistry wrote:
> Why rearrange this? it was sorted before.

See the format in the help string below.  I wanted CQ builders to come last so
that I didn't have to use a format dictionary.

https://codereview.chromium.org/136683006/diff/1/tools/submit_try#newcode105
tools/submit_try:105: 'master/slaves.cfg')
On 2014/01/14 17:39:54, rmistry wrote:
> Can we make this a top-level constant.

Done.

[commit-bot: I haz the power, 2014-01-14 18:40:41]

CQ is trying da patch. Follow status at
https://skia-tree-status.appspot.com/cq/borenet@google.com/136683006/90001

[commit-bot: I haz the power, 2014-01-14 18:40:43]

Failed to apply patch for tools/submit_try:
While running patch -p1 --forward --force --no-backup-if-mismatch;
  patching file tools/submit_try
  Hunk #1 FAILED at 35.
  Hunk #2 succeeded at 103 (offset 2 lines).
  Hunk #3 succeeded at 178 (offset 2 lines).
  Hunk #4 succeeded at 188 (offset 2 lines).
  Hunk #5 succeeded at 217 (offset 2 lines).
  Hunk #6 succeeded at 312 (offset 4 lines).
  1 out of 6 hunks FAILED -- saving rejects to file tools/submit_try.rej

Patch:       tools/submit_try
Index: tools/submit_try
diff --git a/tools/submit_try b/tools/submit_try
index
586fb10dbbb326c506313830fa7b1cadf61d758e..e2348915ec5027991f078888180be16d05f056fc
100755
--- a/tools/submit_try
+++ b/tools/submit_try
@@ -35,11 +35,10 @@ CQ_BUILDERS = 'cq'
 # Alias which can be used to specify a regex to choose builders.
 REGEX = 'regex'
 
-ALL_ALIASES = [ALL_BUILDERS, COMPILE_BUILDERS, CQ_BUILDERS, REGEX]
+ALL_ALIASES = [ALL_BUILDERS, COMPILE_BUILDERS, REGEX, CQ_BUILDERS]
 
-# Contact information for the build master.
-SKIA_BUILD_MASTER_HOST = str(buildbot_globals.Get('public_master_host'))
-SKIA_BUILD_MASTER_PORT = str(buildbot_globals.Get('public_external_port'))
+SLAVES_CFG_URL = ('https://skia.googlesource.com/buildbot/+/master/'
+                  'master/slaves.cfg')
 
 # All try builders have this suffix.
 TRYBOT_SUFFIX = '-Trybot'
@@ -101,27 +100,34 @@ def GetTryRepo():
                   'defined in the %s file.' % codereview_settings_file)
 
 
-def RetrieveTrybotList(json_filename):
-  """ Retrieve the list of known trybots from the build master, stripping
-  TRYBOT_SUFFIX from the name. """
-  trybots = []
-  connection = httplib.HTTPConnection(SKIA_BUILD_MASTER_HOST,  
-                                      SKIA_BUILD_MASTER_PORT)
-  connection.request('GET', '/json/%s' % json_filename)
-  response = connection.getresponse()  
-  builders = json.load(response)
+def RetrieveTrybotList():
+  """Retrieve the list of known trybots from the checked-in buildbot
+  configuration."""
+  # Retrieve the slaves.cfg file from the repository.
+  slaves_cfg_text = buildbot_globals.retrieve_from_googlesource(SLAVES_CFG_URL)
 
-  for builder in builders:
-    if builder.endswith(TRYBOT_SUFFIX):
-      trybots.append(builder[:-len(TRYBOT_SUFFIX)])
-  return trybots
+  # Execute the slaves.cfg file to obtain the list of slaves.
+  vars = {}
+  exec(slaves_cfg_text, vars)
+  slaves_cfg = vars['slaves']
 
+  # Pull the list of known builders from the slaves list.
+  trybots = set()
+  for slave in slaves_cfg:
+    for builder in slave['builder']:
+      if not builder.endswith(TRYBOT_SUFFIX):
+        trybots.add(builder)
 
-def ValidateArgs(argv, trybots, is_svn=True):
+  return list(trybots), vars['cq_trybots']
+
+
+def ValidateArgs(argv, trybots, cq_trybots, is_svn=True):
   """ Parse and validate command-line arguments. If the arguments are valid,
   returns a tuple of (<changelist name>, <list of trybots>).
 
-  trybots: A list of the known try builders.
+  trybots: list of strings; A list of the known try builders.
+  cq_trybots: list of strings; Trybots who get run by the commit queue.
+  is_svn: bool; whether or not we're in an svn checkout.
   """
 
   class CollectedArgs(object):
@@ -169,7 +175,9 @@ submit_try %s--bot <buildername> [<buildername> ...]
     if arg == '-h' or arg == '--help':
       Error()
     elif arg == '-l' or arg == '--list_bots':
-      format_args = ['\n  '.join(sorted(trybots))] + ALL_ALIASES
+      format_args = ['\n  '.join(sorted(trybots))] + \
+                    ALL_ALIASES + \
+                    ['\n    '.join(sorted(cq_trybots))]
       print (
 """
 submit_try: Available builders:\n  %s
@@ -177,8 +185,8 @@ submit_try: Available builders:\n  %s
 Can also use the following aliases to run on groups of builders-
   %s: Will run against all trybots.
   %s: Will run against all compile trybots.
-  %s: Will run against the same trybots as the commit queue.
   %s: You will be prompted to enter a regex to select builders with.
+  %s: Will run against the same trybots as the commit queue:\n    %s
 
 """ % tuple(format_args))
       sys.exit(0)
@@ -206,7 +214,7 @@ Can also use the following aliases to run on groups of
builders-
             elif bot == COMPILE_BUILDERS:
               using_bots = [t for t in trybots if t.startswith('Build')]
             elif bot == CQ_BUILDERS:
-              using_bots = RetrieveTrybotList(json_filename='cqtrybots')
+              using_bots = cq_trybots
             elif bot == REGEX:
               while True:
                 regex = raw_input("Enter your trybot regex: ")
@@ -299,13 +307,14 @@ def SubmitTryRequest(args, is_svn=True):
 
 def main():
   # Retrieve the list of active try builders from the build master.
-  trybots = RetrieveTrybotList(json_filename='trybots')
+  trybots, cq_trybots = RetrieveTrybotList()
 
   # Determine if we're in an SVN checkout.
   is_svn = os.path.isdir('.svn')
 
   # Parse and validate the command-line arguments.
-  args = ValidateArgs(sys.argv[1:], trybots=trybots, is_svn=is_svn)
+  args = ValidateArgs(sys.argv[1:], trybots=trybots, cq_trybots=cq_trybots,
+                      is_svn=is_svn)
 
   # Submit the try request.
   SubmitTryRequest(args, is_svn=is_svn)

[commit-bot: I haz the power, 2014-01-14 18:43:09]

CQ is trying da patch. Follow status at
https://skia-tree-status.appspot.com/cq/borenet@google.com/136683006/150001

[commit-bot: I haz the power, 2014-01-14 19:18:47]

Change committed as 13071