Chromium Code Reviews
chromiumcodereview-hr@appspot.gserviceaccount.com (chromiumcodereview-hr) | Please choose your nickname with Settings | Help | Chromium Project | Gerrit Changes | Sign out
(159)

Side by Side Diff: slave/skia_slave_scripts/utils/sync_bucket_subdir.py

Issue 304393002: Adds uploading picture benchmark JSON data (Closed) Base URL: https://skia.googlesource.com/buildbot.git@master
Patch Set: Fix broken upload code Created 6 years, 6 months ago
Use n/p to move between diff chunks; N/P to move between comments. Draft comments are only viewable by you.
Jump to:
View unified diff | Download patch
« no previous file with comments | « slave/skia_slave_scripts/upload_bench_results.py ('k') | no next file » | no next file with comments »
Toggle Intra-line Diffs ('i') | Expand Comments ('e') | Collapse Comments ('c') | Show Comments Hide Comments ('s')
OLDNEW
1 #!/usr/bin/env python 1 #!/usr/bin/env python
2 # Copyright (c) 2012 The Chromium Authors. All rights reserved. 2 # Copyright (c) 2012 The Chromium Authors. All rights reserved.
3 # Use of this source code is governed by a BSD-style license that can be 3 # Use of this source code is governed by a BSD-style license that can be
4 # found in the LICENSE file. 4 # found in the LICENSE file.
5 5
6 """Synchronize multiple files with a Google Storage Bucket and subdir. 6 """Synchronize multiple files with a Google Storage Bucket and subdir.
7 For a lack of a better word 'synchronization' was used, warnings: 7 For a lack of a better word 'synchronization' was used, warnings:
8 1) Downloads files that are not found locally 8 1) Downloads files that are not found locally
9 2) Uploads files that are not found in the bucket 9 2) Uploads files that are not found in the bucket
10 3) Does NOT check version, date or content of the file. If the file is reported 10 3) Does NOT check version, date or content of the file. If the file is reported
(...skipping 12 matching lines...) Expand all
23 import re 23 import re
24 24
25 25
26 DEFAULT_PERFDATA_GS_BASE = 'gs://chromium-skia-gm' 26 DEFAULT_PERFDATA_GS_BASE = 'gs://chromium-skia-gm'
27 KNOWN_FILENAMES = r'^bench_([0-9a-fr]*)_data.*' 27 KNOWN_FILENAMES = r'^bench_([0-9a-fr]*)_data.*'
28 IGNORE_UPLOAD_FILENAMES = ('.DS_Store') 28 IGNORE_UPLOAD_FILENAMES = ('.DS_Store')
29 29
30 30
31 def SyncBucketSubdir(directory, dest_gsbase=DEFAULT_PERFDATA_GS_BASE, subdir='', 31 def SyncBucketSubdir(directory, dest_gsbase=DEFAULT_PERFDATA_GS_BASE, subdir='',
32 do_upload=True, do_download=True, filenames_filter=KNOWN_FILENAMES, 32 do_upload=True, do_download=True, filenames_filter=KNOWN_FILENAMES,
33 exclude_json=False,
33 min_download_revision=0): 34 min_download_revision=0):
34 """ synchronizes a local directory with a cloud one 35 """ synchronizes a local directory with a cloud one
35 36
36 dir: directory to synchronize 37 dir: directory to synchronize
37 dest_gsbase: gs:// bucket to synchronize 38 dest_gsbase: gs:// bucket to synchronize
38 subdir: optional subdirectory within the bucket, multiple directory levels 39 subdir: optional subdirectory within the bucket, multiple directory levels
39 are supported, using Unix relative path syntax ("outer/innner") 40 are supported, using Unix relative path syntax ("outer/innner")
40 do_upload: True to perform upload, False otherwise 41 do_upload: True to perform upload, False otherwise
41 do_download: True to perform download, False otherwise 42 do_download: True to perform download, False otherwise
42 filenames_filter: is a regular expression used to match known file names, 43 filenames_filter: is a regular expression used to match known file names,
(...skipping 35 matching lines...) Expand 10 before | Expand all | Expand 10 after
78 79
79 # Uploads only files not present on the cloud storage 80 # Uploads only files not present on the cloud storage
80 if do_upload: 81 if do_upload:
81 to_upload = local_files.difference(cloud_files) 82 to_upload = local_files.difference(cloud_files)
82 for file_name in to_upload: 83 for file_name in to_upload:
83 if file_name not in IGNORE_UPLOAD_FILENAMES: 84 if file_name not in IGNORE_UPLOAD_FILENAMES:
84 match = re.search(filenames_filter, file_name) 85 match = re.search(filenames_filter, file_name)
85 if not match: 86 if not match:
86 # Ignore other files, rather than raising an exception 87 # Ignore other files, rather than raising an exception
87 continue 88 continue
89 # Ignore JSON files if the flag is set.
90 if exclude_json and file_name.endswith('.json'):
91 continue
88 # Ignore force builds without a revision number. 92 # Ignore force builds without a revision number.
89 if match.group(1) != '': 93 if match.group(1) != '':
90 upload_to_bucket.upload_to_bucket(os.path.join(directory, file_name), 94 upload_to_bucket.upload_to_bucket(os.path.join(directory, file_name),
91 dest_gsbase, 95 dest_gsbase,
92 subdir) 96 subdir)
93 return 0 97 return 0
OLDNEW
« no previous file with comments | « slave/skia_slave_scripts/upload_bench_results.py ('k') | no next file » | no next file with comments »

Powered by Google App Engine
This is Rietveld 408576698