CT Perf recipe to run benchmarks on the top 1k sites using swarming

scripts/slave/recipes/perf/ct_top1k_rr_perf.py

Index: scripts/slave/recipes/perf/ct_top1k_rr_perf.py
diff --git a/scripts/slave/recipes/perf/ct_top1k_rr_perf.py b/scripts/slave/recipes/perf/ct_top1k_rr_perf.py
new file mode 100644
index 0000000000000000000000000000000000000000..e8282e4d725939881a21901815f94848ffa9232d
--- /dev/null
+++ b/scripts/slave/recipes/perf/ct_top1k_rr_perf.py
@@ -0,0 +1,202 @@
+# Copyright 2015 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+import os

[M-A Ruel, 2015/11/06 18:56:06]

You are not supposed to import anything in recipes.

[rmistry, 2015/11/09 16:23:50]

Done.

+import shutil
+import tempfile
+import time
+
+
+DEPS = [
+  'archive',
+  'bot_update',
+  'chromium',
+  'file',
+  'gclient',
+  'gsutil',
+  'isolate',
+  'path',
+  'platform',
+  'properties',
+  'python',
+  'step',
+  'swarming',
+  'swarming_client',
+  'zip',
+]
+
+
+CT_BUCKET = 'cluster-telemetry'
+CT_PAGE_TYPE = '1k'
+CT_BINARY = 'run_chromium_perf_swarming'
+CT_ISOLATE_TEMPLATE = 'ct_top1k.isolate.tmpl'
+
+# Number of slaves to shard CT runs to.

[M-A Ruel, 2015/11/06 18:56:06]

s/slaves/bots/

[rmistry, 2015/11/09 16:23:49]

I used "slaves" here to be consistent with CT's nomenclature where it calls its
workers/bots "slaves". The GS directory it downloads artifacts from is also
called "slave{1..100}". I can use slaves->bots everywhere here if you prefer,
but the GS directory name will still be "slave{1..100}".

[M-A Ruel, 2015/11/09 16:51:21]

There's two points of view:
- the word slave was maybe used because it implied buildbot slave?
- if it's moving to swarming bots and it effectively represent swarming bots,
long term switching to bots is a better choice.

I don't mind too much.

[rmistry, 2015/11/09 19:37:24]

The word slave was used in CT independently of buildbot. It was used because in
the original architecture there was 1 master and 100 slaves. I will leave
"slave" here for now because changing it will require changing lots of
documentation.

+# TODO(rmistry): Change the below to 100 when ready to run the full top 1k.
+CT_NUM_SLAVES = 2
+
+
+def _DownloadAndExtractBinary(api):
+  """Downloads the binary from the revision passed to the recipe."""
+  build_archive_url = api.properties['parent_build_archive_url']
+  api.archive.download_and_unzip_build(
+      step_name='Download and Extract Binary',
+      target='Release',
+      build_url=None,  # This is a required parameter, but has no effect.
+      build_archive_url=build_archive_url)
+
+
+# TODO(rmistry): What priority can I give the below tasks??
+def RunSteps(api):
+  # Figure out which benchmark to use.
+  buildername = api.properties['buildername']
+  if 'Repaint' in buildername:
+    benchmark = 'repaint'
+  elif 'RR' in buildername:
+    benchmark = 'rasterize_and_record_micro'
+  else:
+    raise Exception('Do not recognise the buildername %s.' % buildername)
+
+  # Checkout chromium and swarming.
+  api.chromium.set_config('chromium')
+  api.gclient.set_config('chromium')
+  api.bot_update.ensure_checkout(force=True)
+  api.swarming_client.checkout()
+
+  # Download the prebuilt chromium binary.
+  _DownloadAndExtractBinary(api)
+
+  # Path to the chromium src directory.
+  chromium_src_dir = str(api.path['checkout'])
+  # Path to where artifacts should be downloaded from Google Storage.
+  downloads_dir = os.path.join(chromium_src_dir, 'content', 'test', 'ct')

[M-A Ruel, 2015/11/06 18:56:06]

Why not put this in tmp too?

[rmistry, 2015/11/09 16:23:49]

I initially went that route but it made the isolate template file messy. I then
opted into storing things in the chromium checkout because it seemed much easier
for the isolate file to find artifacts when they were relative to the chromium
checkout.

+  # Path where swarming artifacts (isolate file, json output) will be stored.
+  swarming_temp_dir = str(api.path.mkdtemp('swarming-temp-dir'))

[M-A Ruel, 2015/11/06 18:56:06]

Use self.m.path['tmp_base'] instead. That's what _gtest_collect_step() does.

[rmistry, 2015/11/09 16:23:49]

Done.

+
+  # Download Cluster Telemetry binary.
+  ct_binary_path = os.path.join(downloads_dir, CT_BINARY)
+  api.gsutil.download(
+      bucket=CT_BUCKET,
+      source=os.path.join('swarming', 'binaries', CT_BINARY),
+      dest=ct_binary_path)
+
+  # Record how long the step took in swarming tasks.
+  swarming_start_time = time.time()
+
+  for slave_num in range(1, CT_NUM_SLAVES + 1):
+    slave_dir = os.path.join(downloads_dir, 'slave%s' % slave_num)
+    os.makedirs(slave_dir)
+
+    # Download page sets.
+    page_sets_dir = os.path.join(slave_dir, 'page_sets')

[M-A Ruel, 2015/11/06 18:56:06]

I don't understand why you need to download files CT_NUM_SLAVES times.

[rmistry, 2015/11/09 16:23:49]

The purpose of this recipe is to run benchmarks on the top 1000 web pages.
This is done by sharding 10 web pages to each swarming bot. slave1 will process
the top 1-10 web pages. slave2 will process 11-20 webpages and so on.
Sharding these web pages involves sending each bot the pageset file of the web
pages and the archive of the web pages. Below you will see that these artifacts
are downloaded from a GS directory that is unique for each slave (contains
slave1,slave2,...).
Note: this follows the current architecture that CT uses
(https://skia.org/dev/testing/ct#system_diagram) but is attempting to use
swarming bots instead of the machines from CT's pool.

+    os.makedirs(page_sets_dir)
+    api.gsutil.download(
+        bucket=CT_BUCKET,
+        source=os.path.join('swarming', 'page_sets', CT_PAGE_TYPE,
+                            'slave%s' % slave_num, '*'),
+        dest=page_sets_dir)
+
+    # Download archives.
+    wpr_dir = os.path.join(page_sets_dir, 'data')
+    os.makedirs(wpr_dir)
+    api.gsutil.download(
+        bucket=CT_BUCKET,
+        source=os.path.join('swarming', 'webpage_archives', CT_PAGE_TYPE,
+                            'slave%s' % slave_num, '*'),
+        dest=wpr_dir)
+
+    # TODO(rmistry): Remove the entire below section after crrev.com/1410353007
+    #                is submitted.
+    api.file.copy(
+        'copy %s' % CT_ISOLATE_TEMPLATE,
+        '/repos/chromium/src/chrome/%s' % CT_ISOLATE_TEMPLATE,
+        os.path.join(chromium_src_dir, 'chrome', CT_ISOLATE_TEMPLATE))
+    for f in ['run_ct_top1k.py', 'path_util.py']:
+      api.file.copy(
+          'copy %s' % f,
+          '/repos/chromium/src/content/test/ct/%s' % f,
+          os.path.join(chromium_src_dir, 'content', 'test', 'ct', f))
+
+    # Create this slave's isolate file from the CT_ISOLATE_TEMPLATE.
+    isolate_dir = os.path.join(chromium_src_dir, 'chrome')
+    isolate_template_path = os.path.join(isolate_dir, CT_ISOLATE_TEMPLATE)
+
+    generated_isolate_path = os.path.join(isolate_dir, 'ct_top1k.isolate')
+    with open(generated_isolate_path, 'wt') as fout:
+      with open(isolate_template_path, 'rt') as fin:

[M-A Ruel, 2015/11/06 18:56:06]

use b instead of t, we don't want any CR character.

[rmistry, 2015/11/09 16:23:49]

Agreed. Done.

+        for line in fin:
+          fout.write(line.replace('[[SLAVE_NUM]]', str(slave_num))
+                         .replace('[[MASTER]]', api.properties['mastername'])
+                         .replace('[[BUILDER]]', api.properties['buildername'])
+                         .replace('[[GIT_HASH]]',
+                                  api.properties['git_revision'])
+                         .replace('[[BENCHMARK]]', benchmark))
+      fout.close()

[M-A Ruel, 2015/11/06 18:56:06]

When using with open() ..., you shouldn't close the handle.

[rmistry, 2015/11/09 16:23:49]

Done.

+
+    # Archive everything on the isolate server.
+    isolated_path = os.path.join(
+        swarming_temp_dir, 'ct-1k-task-%s.isolated' % slave_num)
+    isolate_args = [
+      'archive',
+      '--isolate', generated_isolate_path,
+      '--isolated', isolated_path,
+      '--config-variable', 'OS', 'linux',
+      '--isolate-server', api.isolate.isolate_server,
+      # TODO(rmistry): Why do I need PRODUCT_DIR ? fails without it. It also
+      #                requires bitmaptools in PRODUCT_DIR.
+      '--path-variable', 'PRODUCT_DIR', tempfile.gettempdir(),

[M-A Ruel, 2015/11/06 18:56:06]

Can you add more details about what fails?

[rmistry, 2015/11/09 16:23:49]

If I do not specify --path-variable then it fails with:
IsolateError: Variable "PRODUCT_DIR" was not found in {'EXECUTABLE_SUFFIX': ''}.
Did you forget to specify --path-variable?

Also if I do not create a dummy /tmp/bitmaptools then it fails with:
MappingError: Input file /tmp/bitmaptools doesn't exist

I believe this is due to
https://code.google.com/p/chromium/codesearch#chromium/src/tools/telemetry/te...
and I do not know how to make it pass without my above hack.

[M-A Ruel, 2015/11/09 16:51:21]

Do you need this file at all? If not, we should fix that.

[rmistry, 2015/11/09 19:37:24]

+nednyugen
Ned, is this file required? if yes then how do I specify a path to it?

+    ]
+    api.python(
+        'archiving isolate for slave%s' % slave_num,
+        os.path.join(str(api.swarming_client.path), 'isolate.py'),
+        isolate_args)
+
+    # Trigger swarming task.
+    task_name = 'ct-1k-task-%s' % slave_num
+    json_output = os.path.join(
+        swarming_temp_dir, 'ct-1k-task-%s.json' % slave_num)
+    swarming_trigger_args = [
+      'trigger',
+      '--task-name', task_name,
+      isolated_path,
+      '--swarming', api.swarming.swarming_server,
+      '--dimension', 'os', 'Ubuntu',
+      '--dimension', 'gpu', '10de',
+      '--isolate-server', api.isolate.isolate_server,
+      '--dump-json', json_output
+    ]
+    api.python(
+        'triggering task for slave%s' % slave_num,
+        os.path.join(str(api.swarming_client.path), 'swarming.py'),
+        swarming_trigger_args)
+
+    # We have triggered this slave's swarming task. Cleanup slave artifacts.
+    shutil.rmtree(slave_dir)
+
+  # Now collect all tasks.
+  for slave_num in range(1, CT_NUM_SLAVES + 1):
+    json_output = os.path.join(
+        swarming_temp_dir, 'ct-1k-task-%s.json' % slave_num)
+    swarming_collect_args = [
+      'collect',
+      '--swarming', api.swarming.swarming_server,
+      '--json', json_output
+    ]
+    api.python(
+        'collecting task for slave%s' % slave_num,
+        os.path.join(str(api.swarming_client.path), 'swarming.py'),
+        swarming_collect_args)
+
+  # TODO(rmistry): Delete me!

[M-A Ruel, 2015/11/06 18:56:06]

:)

[rmistry, 2015/11/09 16:23:49]

Deleted.

+  print 'my outputs!!!!'
+  print downloads_dir
+  print isolate_template_path
+  print generated_isolate_path
+
+
+  # Cleanup the temporary swarming dir.
+  shutil.rmtree(swarming_temp_dir)

[M-A Ruel, 2015/11/06 18:56:06]

git gs "def rmtree"
gave me
recipe_modules/file/api.py:140:  def rmtree(self, name, path, **kwargs):

[rmistry, 2015/11/09 16:23:49]

Done.

+
+  print ('Running isolating, triggering and collecting swarming tasks took a '
+         'total of %s seconds') % (time.time() - swarming_start_time)