OLD | NEW |
1 Cluster Telemetry | 1 Cluster Telemetry |
2 ================= | 2 ================= |
3 | 3 |
4 ### Contents ### | 4 ### Contents ### |
5 | 5 |
6 * [Overview](#overview) | 6 * [Overview](#overview) |
7 * [Framework Usage](#framework_usage) | 7 * [Framework Usage](#framework_usage) |
8 * [System Architecture](#system_architecture) | 8 * [System Architecture](#system_architecture) |
9 + [System Diagram](#system_diagram) | 9 + [System Diagram](#system_diagram) |
10 + [Detailed explanation of steps](#detailed_explanation) | 10 + [Detailed explanation of steps](#detailed_explanation) |
11 * [Code](#code) | 11 * [Code](#code) |
12 * [Contact Us](#contact_us) | 12 * [Contact Us](#contact_us) |
13 | 13 |
14 <a name="overview"></a> | 14 <a name="overview"></a> |
15 Overview | 15 Overview |
16 -------- | 16 -------- |
17 | 17 |
18 Cluster Telemetry allows you to run [telemetry](https://www.chromium.org/develop
ers/telemetry)'s benchmarks, lua scripts and other tasks using multiple reposito
ry patches through Alexa's [top 1 million](http://s3.amazonaws.com/alexa-static/
top-1m.csv.zip) web pages. | 18 Cluster Telemetry allows you to run [telemetry](https://www.chromium.org/develop
ers/telemetry)'s benchmarks, lua scripts and other tasks using multiple reposito
ry patches through Alexa's [top 1 million](http://s3.amazonaws.com/alexa-static/
top-1m.csv.zip) web pages. |
19 Developers can use the framework to measure the performance of their patch again
st the top subset of the internet on both Desktop and Android. | 19 Developers can use the framework to measure the performance of their patch again
st the top subset of the internet on both Desktop and Android. |
20 | 20 |
21 SKP files are a binary format for the draw commands Chromium sends to Skia for r
asterization. The goal of the project started off with wanting to collect a larg
e repository of 10k SKP files. This repository, after incremental changes in app
roaches, has since grown to ~900k and now supports running all telemetry benchma
rks. The top level feature request of this project was [skia:1268](https://code.
google.com/p/skia/issues/detail?id=1268). | 21 SKP files are a binary format for the draw commands Chromium sends to Skia for r
asterization. The goal of the project started off with wanting to collect a larg
e repository of 10k SKP files. This repository, after incremental changes in app
roaches, has since grown to ~910k and now supports running all telemetry benchma
rks. The top level feature request of this project was [skia:1268](https://code.
google.com/p/skia/issues/detail?id=1268). |
22 | 22 |
23 A web application has been created on App Engine that automates the process of c
apturing new archives and running telemetry benchmarks at a click of a button; r
esults are emailed to the requester and the web application contains complete hi
story of runs with links to results. You can run telemetry benchmarks at http://
ct.skia.org. | 23 A web application has been created on Google Compute Engine that automates the p
rocess of capturing new archives and running telemetry benchmarks at a click of
a button; results are emailed to the requester and the web application contains
complete history of runs with links to results. You can run telemetry benchmarks
at http://ct.skia.org. |
24 | 24 |
25 The framework also contains the ability to run lua scripts on the SKP repository
to scrape web pages. It only takes a few minutes to run a lua scrapping script
on ~900k SKP files. | 25 The framework also contains the ability to run lua scripts on the SKP repository
to scrape web pages. It only takes a few minutes to run a lua scraping script o
n ~900k SKP files. |
26 | 26 |
27 These are the different parts of the framework: | 27 Most users will use these two features: |
28 | 28 |
29 * Chromium Perf Tryserver. Documentation [here](https://www.chromium.org/develop
ers/cluster-telemetry). Webpage [here](http://skia-tree-status.appspot.com/skia-
telemetry/chromium_try). | 29 * Chromium Perf. Documentation [here](https://www.chromium.org/developers/cluste
r-telemetry). Webpage [here](https://ct.skia.org/chromium_perf/). |
30 * Skia Correctness Tryserver. Documentation [here](http://goto/skiatryclustertel
emetry). Webpage [here](http://skia-tree-status.appspot.com/skia-telemetry/skia_
try). | 30 * Run Lua Scripts. Documentation about lua bindings is [here](https://skia.org/u
ser/special/lua). Webpage [here](https://ct.skia.org/lua_script/). |
31 * Run Lua Scripts. Documentation about lua bindings is [here](https://skia.org/u
ser/special/lua). Webpage [here](http://skia-tree-status.appspot.com/skia-teleme
try/lua_script). | |
32 | 31 |
33 | 32 |
34 <a name="framework_usage"></a> | 33 <a name="framework_usage"></a> |
35 Framework Usage | 34 Framework Usage |
36 --------------- | 35 --------------- |
37 | 36 |
38 The Chromium Perf tryserver in CT has been used to gather perf data over the top
10k web pages for the following Chromium projects: | 37 The Chromium Perf tryserver in CT has been used to gather perf data over the top
10k web pages for the following Chromium projects: |
39 | 38 |
40 * Slimming paint | 39 * Slimming paint |
41 * Performance data for layer squashing and compositing overlap map | 40 * Performance data for layer squashing and compositing overlap map |
(...skipping 11 matching lines...) Expand all Loading... |
53 | 52 |
54 * [Loading measurement: alexa top 1,000](https://docs.google.com/a/chromium.org/
document/d/1ca_Q7xePmCRqaYnHe7vkpCmKNFNLdDXvzgtUPt9iG8w/edit) | 53 * [Loading measurement: alexa top 1,000](https://docs.google.com/a/chromium.org/
document/d/1ca_Q7xePmCRqaYnHe7vkpCmKNFNLdDXvzgtUPt9iG8w/edit) |
55 * [Loading measurement: alexa top million](https://docs.google.com/a/google.com/
document/d/1hDDUUNE5OUV8eCjtOj7Ow6EZ2DSBCTjQirnA3Rp5pOg/edit) | 54 * [Loading measurement: alexa top million](https://docs.google.com/a/google.com/
document/d/1hDDUUNE5OUV8eCjtOj7Ow6EZ2DSBCTjQirnA3Rp5pOg/edit) |
56 * [Loading measurement: alexa top million netsim](https://docs.google.com/a/goog
le.com/document/d/1cpLSSYpqi4SprkJcVxbS7af6avKM0qc-imxvkexmCZs/edit) | 55 * [Loading measurement: alexa top million netsim](https://docs.google.com/a/goog
le.com/document/d/1cpLSSYpqi4SprkJcVxbS7af6avKM0qc-imxvkexmCZs/edit) |
57 * [Perf profile - alexa top million sites](https://docs.google.com/a/google.com/
document/d/1di__87watociuZj_dm22Cn72UM2xsZBXixbl8TCFQmw/edit) | 56 * [Perf profile - alexa top million sites](https://docs.google.com/a/google.com/
document/d/1di__87watociuZj_dm22Cn72UM2xsZBXixbl8TCFQmw/edit) |
58 | 57 |
59 The framework has also been used to run multiple lua scripts to scrape the SKP r
epositories for the the following: | 58 The framework has also been used to run multiple lua scripts to scrape the SKP r
epositories for the the following: |
60 chars-vs-glyphs, bitmap transform types, gradient color counter, 3 color gradien
t checks, etc. | 59 chars-vs-glyphs, bitmap transform types, gradient color counter, 3 color gradien
t checks, etc. |
61 This has been very useful for the Skia team to help determine which parts of the
library to optimize and focus on. | 60 This has been very useful for the Skia team to help determine which parts of the
library to optimize and focus on. |
62 | 61 |
63 All runs are recorded [here](http://skia-tree-status.appspot.com/skia-telemetry/
all_tasks). | 62 All runs are recorded [here](https://ct.skia.org/history/). |
64 | 63 |
65 | 64 |
66 <a name="system_architecture"></a> | 65 <a name="system_architecture"></a> |
67 System Architecture | 66 System Architecture |
68 ------------------ | 67 ------------------- |
69 | 68 |
70 <a name="system_diagram"></a> | 69 <a name="system_diagram"></a> |
71 ### System Diagram | 70 ### System Diagram |
72 | 71 |
73 ![CT System Diagram](ct-system.png) | 72 ![CT System Diagram](ct-system-diagram.svg) |
74 | 73 |
75 | 74 |
76 <a name="detailed_explanation"></a> | 75 <a name="detailed_explanation"></a> |
77 ### Detailed explanation of steps | 76 ### Detailed explanation of steps |
78 | 77 |
79 1. User submits an Admin task (rebuild chrome, recreate pagesets, recreate webpa
ge archives), Lua script task, or Telemetry benchmark task using the App Engine
web application [here](http://ct.skia.org). | 78 1. User submits a Lua script task, a Performance task, or an Admin task (rebuild
chrome, recreate pagesets, recreate webpage archives, capture SKPs) using the G
CE web application [here](http://ct.skia.org). |
80 | 79 |
81 2. Each task is exposed by the web application in JSON. The CT master polls the
web application and picks up new tasks. | 80 2. Each task is exposed by the web application in JSON. The CT master polls the
web application and picks up new tasks. |
82 | 81 |
83 3. The master pushes new tasks to all the workers using the master scripts [here
](https://skia.googlesource.com/buildbot/+/master/ct/go/master_scripts/) (in a n
ew process so that the poller is not blocked). The master scripts then check to
see when the workers are done with the requested task. | 82 3. The master pushes new tasks to all the workers using the master scripts [here
](https://skia.googlesource.com/buildbot/+/master/ct/go/master_scripts/). The ma
ster scripts then check to see when the workers are done with the requested task
. |
84 | 83 |
85 4. The workers execute the task using the worker scripts [here](https://skia.goo
glesource.com/buildbot/+/master/ct/go/worker_scripts/). All generated artifacts
(CSV files, logs, SKP files, archives, etc) are then stored locally and copied t
o Google Storage. | 84 4. The workers execute the task using the worker scripts [here](https://skia.goo
glesource.com/buildbot/+/master/ct/go/worker_scripts/). All generated artifacts
(CSV files, logs, SKP files, archives, etc) are then stored locally and copied t
o Google Storage. |
86 | 85 |
87 5. The master scripts periodically check the workers to see when they are done w
ith the requested task. Once the workers are done the generated artifacts are th
en read from Google Storage and consolidated (if required). | 86 5. The master scripts periodically check the workers to see when they are done w
ith the requested task. Once the workers are done the generated artifacts are th
en read from Google Storage and consolidated (if required). |
88 | 87 |
89 6. The master scripts then email results of the task to the user who requested i
t. The master scripts also update the status of the task to completed on App Eng
ine. | 88 6. The master scripts then email results of the task to the user who requested i
t. The master scripts also update the status of the task to completed on App Eng
ine. |
90 | 89 |
91 | 90 |
92 <a name="code"></a> | 91 <a name="code"></a> |
93 Code | 92 Code |
94 ---- | 93 ---- |
95 | 94 |
96 Cluster Telemetry is primarily written in Go with a few python scripts. The fram
ework lives in [master/ct](https://skia.googlesource.com/buildbot/+/master/ct) a
nd the appengine code lives in [master/appengine_scripts](https://skia.googlesou
rce.com/buildbot/+/master/appengine_scripts/skia-tree-status). | 95 Cluster Telemetry is primarily written in Go with a few python scripts. The fram
ework lives in [master/ct](https://skia.googlesource.com/buildbot/+/master/ct). |
97 | 96 |
98 <a name="contact_us"></a> | 97 <a name="contact_us"></a> |
99 Contact Us | 98 Contact Us |
100 ---------- | 99 ---------- |
101 | 100 |
102 If you have questions, please email <cluster-telemetry@chromium.org> or contact
rmistry@ directly. | 101 If you have questions, please email <cluster-telemetry@chromium.org> or contact
rmistry@ directly. |
OLD | NEW |