Add swarming task for upload_dm_results

infra/bots/recipes/upload_dm_results.py

Index: infra/bots/recipes/upload_dm_results.py
diff --git a/infra/bots/recipes/upload_dm_results.py b/infra/bots/recipes/upload_dm_results.py
new file mode 100644
index 0000000000000000000000000000000000000000..27cab8bd04c66e0cf25dcf9ace27a92663adf0cf
--- /dev/null
+++ b/infra/bots/recipes/upload_dm_results.py
@@ -0,0 +1,97 @@
+# Copyright 2016 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+
+# Recipe for uploading DM results.
+
+
+DEPS = [
+  'recipe_engine/json',
+  'recipe_engine/path',
+  'recipe_engine/properties',
+  'recipe_engine/shutil',
+  'recipe_engine/step',
+  'recipe_engine/time',
+]
+
+
+DM_JSON = 'dm.json'
+GS_BUCKET = 'gs://chromium-skia-gm'
+VERBOSE_LOG = 'verbose.log'
+
+
+def RunSteps(api):
+  builder_name = api.properties['buildername']
+  revision = api.properties['revision']
+  issue = str(api.properties.get('issue', ''))
+  patchset = str(api.properties.get('patchset', ''))
+
+  results_dir = api.path['cwd'].join('dm')
+
+  # Validate the JSON file.
+  json_file = results_dir.join(DM_JSON)
+  api.json.read('validate dm.json', json_file)
+
+  # Move dm.json and verbose.log to their own directory.
+  log_file = results_dir.join(VERBOSE_LOG)
+  tmp_dir = api.path['cwd'].join('tmp_upload')
+  api.shutil.makedirs('tmp dir', tmp_dir, infra_step=True)
+  api.shutil.copy('copy dm.json', json_file, tmp_dir)
+  api.shutil.copy('copy verbose.log', log_file, tmp_dir)
+  api.shutil.remove('rm old dm.json', json_file)
+  api.shutil.remove('rm old verbose.log', log_file)
+
+  # Upload the images.
+  # TODO(borenet): Are the permissions correct? We want only Google-readable.

[borenet, 2016/09/23 15:56:39]

It *seems* to be correct, eg.

Old upload:
http://chromium-skia-gm.commondatastorage.googleapis.com/dm-images-v1/0010007...

New upload:
http://chromium-skia-gm.commondatastorage.googleapis.com/dm-images-v1-test/00...

The old upload does an individual upload per file with multiple worker threads.
It uploads each file to a temporary location, in case the upload gets
interrupted, with the pre-defined "private" ACL. Then, it moves the file to its
intended location and then sets the fine-grained Google-readable ACL on the
file. AFAICT the newer gsutil still doesn't support uploading a directory at a
time with a fine-grained ACL. I guess in this case we're inheriting the bucket
defaults.  IMO the simplest way forward is to create a new bucket which has the
right fine-grained ACLs as bucket defaults, so our uploader doesn't need to deal
with that.

[stephana, 2016/09/26 16:11:24]

We should set this to the new bucket before we land. I am testing right now that
make sure that the default permissions are correct and work for Gold. But +100
for not setting the ACLs in the scripts and set the right default permissions
instead. 

[borenet, 2016/09/26 17:35:38]

Changed to use the new bucket, and the default permissions there look correct to
me.

+  image_dest_path = '/'.join((GS_BUCKET, 'dm-images-v1-test'))
+  api.step(
+      'upload images',
+      cmd=['gsutil', 'cp', results_dir.join('*'), image_dest_path],
+   )
+
+  # Upload the JSON summary and verbose.log.
+  now = api.time.utcnow()
+  # TODO(borenet): These used to also be keyed by build number, which is no
+  # longer a concept. What should we use instead?

[borenet, 2016/09/23 15:56:39]

For nanobench results we don't use a build number; the builder name and revision
are enough to identify the results we want, and we add a timestamp to prevent
collisions for any duplicates.  Would that work here as well?

[stephana, 2016/09/26 16:11:24]

Per our offline conversation we should inject a timestamp (ms since the epoch)
into the path, to make sure we don't overwrite runs within the same hour.
Otherwise this should be fine. 

[borenet, 2016/09/26 17:35:38]

Added the timestamp.

+  summary_dest_path = '/'.join([
+      'dm-json-v1-test',
+      str(now.year ).zfill(4),
+      str(now.month).zfill(2),
+      str(now.day  ).zfill(2),
+      str(now.hour ).zfill(2),
+      revision,
+      builder_name])
+
+  # Trybot results are further siloed by issue/patchset.
+  if builder_name.endswith('-Trybot'):
+    if not (issue and patchset):  # pragma: nocover
+      raise Exception('issue and patchset properties are required for trybots.')
+    summary_dest_path = '/'.join(('trybot', summary_dest_path, issue, patchset))
+
+  summary_dest_path = '/'.join((GS_BUCKET, summary_dest_path))
+
+  api.step(
+      'upload JSON and logs',
+      cmd=['gsutil', 'cp', '-Z', tmp_dir.join('*'), summary_dest_path],
+  )
+
+
+def GenTests(api):
+  builder = 'Test-Ubuntu-GCC-GCE-CPU-AVX2-x86_64-Debug'
+  yield (
+    api.test('normal_bot') +
+    api.properties(buildername=builder,
+                   revision='abc123',
+                   path_config='kitchen')
+  )
+
+  builder = 'Test-Ubuntu-GCC-GCE-CPU-AVX2-x86_64-Debug-Trybot'
+  yield (
+    api.test('trybot') +
+    api.properties(buildername=builder,
+                   revision='abc123',
+                   path_config='kitchen',
+                   issue='12345',
+                   patchset='1002')
+  )