CT Skia recipe to run DM on the top 10k SKPs using swarming

scripts/slave/recipes/skia/ct_10k_skps_dm.py

Index: scripts/slave/recipes/skia/ct_10k_skps_dm.py
diff --git a/scripts/slave/recipes/skia/ct_10k_skps_dm.py b/scripts/slave/recipes/skia/ct_10k_skps_dm.py
new file mode 100644
index 0000000000000000000000000000000000000000..b317eb52e1befc54eb609f5e14d355f474d4224f
--- /dev/null
+++ b/scripts/slave/recipes/skia/ct_10k_skps_dm.py
@@ -0,0 +1,127 @@
+# Copyright 2015 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+from common.skia import global_constants
+
+
+DEPS = [
+  'ct_swarming',
+  'file',
+  'gclient',
+  'path',
+  'properties',
+  'step',
+  'skia',
+  'swarming',
+  'swarming_client',
+  'time',
+]
+
+
+CT_PAGE_TYPE = '10k'
+CT_DM_ISOLATE = 'ct_dm.isolate'
+
+# Number of slaves to shard CT runs to.
+DEFAULT_CT_NUM_SLAVES = 100
+
+# The SKP repository to use.
+DEFAULT_SKPS_CHROMIUM_BUILD = '310ea93-42bd6bf'
+
+
+def RunSteps(api):
+  # Checkout Skia and Chromium.
+  gclient_cfg = api.gclient.make_config()
+  src = gclient_cfg.solutions.add()
+  src.name = 'src'
+  src.url = 'https://chromium.googlesource.com/chromium/src.git'
+  skia = gclient_cfg.solutions.add()
+  skia.name = 'skia'
+  skia.url = global_constants.SKIA_REPO
+  gclient_cfg.got_revision_mapping['skia'] = 'got_revision'
+  api.gclient.checkout(gclient_config=gclient_cfg)
+
+  # Checkout Swarming scripts.
+  api.swarming_client.checkout()
+  # Ensure swarming_client is compatible with what recipes expect.
+  api.swarming.check_client_version()
+
+  chromium_checkout = api.path['checkout']
+  skia_checkout = api.path['slave_build'].join('skia')
+  # Set api.path['checkout'] to Skia not to Chromium.
+  api.path['checkout'] = skia_checkout

[borenet, 2015/11/24 19:34:28]

This seems dangerous, but I can't think of an example where it would cause
problems.  Why is this needed?  I think if you specify the Skia checkout first,
api.path['checkout'] may be set to the Skia checkout without needing to override
it.

[rmistry, 2015/11/24 19:58:41]

Specified Skia checkout first and removed this.

Why is this needed is more complicated, here are some words:
I ran into a lot of problems trying to make this work. This because both skia
and different chromium recipes rely to api.path['checkout'] to point to
different places. Chromium recipes assume that checkout points to Chromium dir,
Skia assumes that checkout points to Skia. The ideal solution for me would be
that skia/api.py created a api.path['skia'] similar to what some other recipes
do. But that will likely need changes in many places.

+
+  # Build DM in Debug mode.
+  flavor = api.skia.get_flavor({})
+  flavor.compile('dm')

[borenet, 2015/11/24 19:34:28]

It's probably better to just run the compile command directly, rather than
trying to use what the Skia recipe module does.

[rmistry, 2015/11/24 19:58:41]

Compile directly from where? do you mean api.skia.flavor.compile ? If so then I
could not do that because flavor was initialized in gen_steps (which is not
called) and so was self.builder_cfg.

[borenet, 2015/11/24 20:06:40]

I mean just directly run "make dm" instead of trying to use the Skia recipe
module's flavor utils, which aren't needed here.  That way, you don't even need
the Skia DEP.

[rmistry, 2015/11/25 14:11:21]

Oh, makes sense. Done. Much better now, I could remove the skia_checkout var
too.

+
+  # TODO(rmistry): Remove the following after crrev.com/1469823003 is submitted.
+  api.file.copy(
+      'copy %s' % CT_DM_ISOLATE,
+      '/repos/chromium/src/chrome/%s' % CT_DM_ISOLATE,
+      chromium_checkout.join('chrome', CT_DM_ISOLATE))
+  for f in ['run_ct_dm.py']:

[borenet, 2015/11/24 19:34:28]

Why is the loop needed?

[rmistry, 2015/11/24 19:58:41]

This section is temporary and will be removed. The loop is not needed right now
but it makes adding new files easy if necessary.

[borenet, 2015/11/24 20:06:40]

Acknowledged.

+    api.file.copy(
+        'copy %s' % f,
+        '/repos/chromium/src/content/test/ct/%s' % f,
+        chromium_checkout.join('content', 'test', 'ct', f))
+
+  # Record how long the step took in swarming tasks.
+  swarming_start_time = api.time.time()
+
+  skps_chromium_build = api.properties.get(
+      'skps_chromium_build', DEFAULT_SKPS_CHROMIUM_BUILD)

[borenet, 2015/11/24 19:34:28]

How will this be specified?

[rmistry, 2015/11/24 19:58:41]

Will always probably be hardcoded here, made it set from properties for local
testing.

[borenet, 2015/11/24 20:06:40]

Acknowledged.

+  ct_num_slaves = api.properties.get('ct_num_slaves', DEFAULT_CT_NUM_SLAVES)
+
+  for slave_num in range(1, ct_num_slaves + 1):
+    # Download SKPs.
+    api.ct_swarming.download_skps(CT_PAGE_TYPE, slave_num, skps_chromium_build)
+
+    # Create this slave's isolated.gen.json file to use for batcharchiving.
+    isolate_dir = chromium_checkout.join('chrome')
+    isolate_path = isolate_dir.join(CT_DM_ISOLATE)
+    extra_variables = {
+        'SLAVE_NUM': str(slave_num),
+    }
+    api.ct_swarming.create_isolated_gen_json(
+        isolate_path, isolate_dir, 'linux', slave_num, extra_variables)
+
+  # Batcharchive everything on the isolate server for efficiency.
+  api.ct_swarming.batcharchive(ct_num_slaves)
+  swarm_hashes = (
+      api.step.active_result.presentation.properties['swarm_hashes']).values()
+
+  # Trigger all swarming tasks.
+  tasks = api.ct_swarming.trigger_swarming_tasks(
+      swarm_hashes, task_name_prefix='ct-10k-dm',
+      dimensions={'os': 'Ubuntu', 'gpu': '10de'})
+
+  # Now collect all tasks.
+  api.ct_swarming.collect_swarming_tasks(tasks)
+
+  print ('Running isolating, triggering and collecting swarming tasks took a '
+         'total of %s seconds') % (api.time.time() - swarming_start_time)
+
+
+def GenTests(api):
+  parent_got_swarming_client_revision = '12345'
+  ct_num_slaves = 5
+
+  yield(
+    api.test('CT_DM_10k_SKPs') +
+    api.properties(
+        buildername='CT-DM-10k-SKPs',
+        parent_got_swarming_client_revision=parent_got_swarming_client_revision,
+        ct_num_slaves=ct_num_slaves,
+    )
+  )
+
+  yield(
+    api.test('CT_DM_10k_SKPs_slave3_failure') +
+    api.step_data('ct-10k-dm-3 on Ubuntu', retcode=1) +
+    api.properties(
+        buildername='CT-DM-10k-SKPs',
+        parent_got_swarming_client_revision=parent_got_swarming_client_revision,
+        ct_num_slaves=ct_num_slaves,
+    )
+  )