| OLD | NEW |
| 1 # Perf Regression Sheriffing (go/perfregression-sheriff) | 1 # Perf Regression Sheriffing (go/perfregression-sheriff) |
| 2 | 2 |
| 3 The perf regression sheriff tracks performance regressions in Chrome's | 3 The perf regression sheriff tracks performance regressions in Chrome's |
| 4 continuous integration tests. Note that a [new rotation](perf_bot_sheriffing.md) | 4 continuous integration tests. Note that a [new rotation](perf_bot_sheriffing.md) |
| 5 has been created to ensure the builds and tests stay green, so the perf | 5 has been created to ensure the builds and tests stay green, so the perf |
| 6 regression sheriff role is now entirely focused on performance. | 6 regression sheriff role is now entirely focused on performance. |
| 7 | 7 |
| 8 ## Key Responsibilities | 8 ## Key Responsibilities |
| 9 | 9 |
| 10 * [Triage Regressions on the Perf Dashboard](#triage) | 10 * [Triage Regressions on the Perf Dashboard](#triage) |
| 11 * [Triaging Data Stoppage Alerts](#datastoppage) | 11 * [Triaging Data Stoppage Alerts](#datastoppage) |
| 12 * [Follow up on Performance Regressions](#followup) | 12 * [Follow up on Performance Regressions](#followup) |
| 13 * [Give Feedback on our Infrastructure](#feedback) | 13 * [Give Feedback on our Infrastructure](#feedback) |
| 14 | 14 |
| 15 ###<a name="triage"></a> Triage Regressions on the Perf Dashboard | 15 ##<a name="triage"></a> Triage Regressions on the Perf Dashboard |
| 16 | 16 |
| 17 Open the perf dashboard [alerts page](https://chromeperf.appspot.com/alerts). | 17 Open the perf dashboard [alerts page](https://chromeperf.appspot.com/alerts). |
| 18 | 18 |
| 19 In the upper right corner, **sign in with your Chromium account**. Signing in is | 19 In the upper right corner, **sign in with your Chromium account**. Signing in is |
| 20 important in order to be able to kick off bisect jobs, and see data from | 20 important in order to be able to kick off bisect jobs, and see data from |
| 21 internal waterfalls. | 21 internal waterfalls. |
| 22 | 22 |
| 23 Pick up **Chromium Perf Sheriff** from "Select an item ▼" drop down menu. There | 23 Pick up **Chromium Perf Sheriff** from "Select an item ▼" drop down menu. There |
| 24 are two tables of alerts that may be shown: | 24 are two tables of alerts that may be shown: |
| 25 | 25 |
| (...skipping 40 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... |
| 66 tooltip on the graph. If you see any likely culprits, cc the authors on the | 66 tooltip on the graph. If you see any likely culprits, cc the authors on the |
| 67 bug. | 67 bug. |
| 68 5. **Optionally, kick off more bisects**. The perf dashboard will automatically | 68 5. **Optionally, kick off more bisects**. The perf dashboard will automatically |
| 69 kick off a bisect for each bug you file. But if you think the regression is | 69 kick off a bisect for each bug you file. But if you think the regression is |
| 70 much clearer on one platform, or a specific page of a page set, or you want | 70 much clearer on one platform, or a specific page of a page set, or you want |
| 71 to see a broader revision range feel free to click on the alert on that graph | 71 to see a broader revision range feel free to click on the alert on that graph |
| 72 and kick off a bisect for it. There should be capacity to kick off as many | 72 and kick off a bisect for it. There should be capacity to kick off as many |
| 73 bisects as you feel are necessary to investigate; [give feedback](#feedback) | 73 bisects as you feel are necessary to investigate; [give feedback](#feedback) |
| 74 below if you feel that is not the case. | 74 below if you feel that is not the case. |
| 75 | 75 |
| 76 ###<a name="datastoppage"></a> Triaging data stoppage alerts | 76 ##<a name="datastoppage"></a> Triaging data stoppage alerts |
| 77 | 77 |
| 78 Data stoppage alerts are listed on the | 78 Data stoppage alerts are listed on the |
| 79 [perf dashboard alerts page](https://chromeperf.appspot.com/alerts). Whenever | 79 [perf dashboard alerts page](https://chromeperf.appspot.com/alerts). Whenever |
| 80 the dashboard is monitoring a metric, and that metric stops sending data, an | 80 the dashboard is monitoring a metric, and that metric stops sending data, an |
| 81 alert is fired. Some of these alerts are expected: | 81 alert is fired. Some of these alerts are expected: |
| 82 | 82 |
| 83 * When a telemetry benchmark is disabled, we get a data stoppage alert. | 83 * When a telemetry benchmark is disabled, we get a data stoppage alert. |
| 84 Check the [code for the benchmark](https://code.google.com/p/chromium/codes
earch#chromium/src/tools/perf/benchmarks/) | 84 Check the [code for the benchmark](https://code.google.com/p/chromium/codes
earch#chromium/src/tools/perf/benchmarks/) |
| 85 to see if it has been disabled, and if so associate the alert with the | 85 to see if it has been disabled, and if so associate the alert with the |
| 86 bug for the disable. | 86 bug for the disable. |
| (...skipping 14 matching lines...) Expand all Loading... |
| 101 bug as well. Check for any changes to the test in the revision range. | 101 bug as well. Check for any changes to the test in the revision range. |
| 102 * Go to the buildbot status page of the bot which should be running the test. | 102 * Go to the buildbot status page of the bot which should be running the test. |
| 103 Is it running the test? If not, note that in the bug. | 103 Is it running the test? If not, note that in the bug. |
| 104 * If it is running the test and the test is failing, diagnose as a test | 104 * If it is running the test and the test is failing, diagnose as a test |
| 105 failure. | 105 failure. |
| 106 * If it is running the test and the test is passing, check the `json.output` | 106 * If it is running the test and the test is passing, check the `json.output` |
| 107 link on the buildbot status page for the test. This is the data the test | 107 link on the buildbot status page for the test. This is the data the test |
| 108 sent to the perf dashboard. Are there null values? Sometimes it lists a | 108 sent to the perf dashboard. Are there null values? Sometimes it lists a |
| 109 reason as well. Please put your finding in the bug. | 109 reason as well. Please put your finding in the bug. |
| 110 | 110 |
| 111 ###<a name="followup"></a> Follow up on Performance Regressions | 111 ##<a name="followup"></a> Follow up on Performance Regressions |
| 112 | 112 |
| 113 During your shift, you should try to follow up on each of the bugs you filed. | 113 During your shift, you should try to follow up on each of the bugs you filed. |
| 114 Once you've triaged all the alerts, check to see if the bisects have come back, | 114 Once you've triaged all the alerts, check to see if the bisects have come back, |
| 115 or if they failed. If the results came back, and a culprit was found, follow up | 115 or if they failed. If the results came back, and a culprit was found, follow up |
| 116 with the CL author. If the bisects failed to update the bug with results, please | 116 with the CL author. If the bisects failed to update the bug with results, please |
| 117 file a bug on it (see [feedback](#feedback) links below). | 117 file a bug on it (see [feedback](#feedback) links below). |
| 118 | 118 |
| 119 Also during your shift, please spend any spare time driving down bugs from the | 119 Also during your shift, please spend any spare time driving down bugs from the |
| 120 [regression backlog](http://go/triage-backlog). Treat these bugs as you would | 120 [regression backlog](http://go/triage-backlog). Treat these bugs as you would |
| 121 your own - investigate the regressions, find out what the next step should be, | 121 your own - investigate the regressions, find out what the next step should be, |
| 122 and then move the bug along. As the backlog only contains bugs that haven't been | 122 and then move the bug along. As the backlog only contains bugs that haven't been |
| 123 modified in some time, you should be able to end your shift with an empty | 123 modified in some time, you should be able to end your shift with an empty |
| 124 backlog. | 124 backlog. |
| 125 | 125 |
| 126 After your shift, please try to follow up on the bugs you filed weekly. Kick off | 126 After your shift, please try to follow up on the bugs you filed weekly. Kick off |
| 127 new bisects if the previous ones failed, and if the bisect picks a likely | 127 new bisects if the previous ones failed, and if the bisect picks a likely |
| 128 culprit follow up to ensure the CL author addresses the problem. If you are | 128 culprit follow up to ensure the CL author addresses the problem. If you are |
| 129 certain that a specific CL caused a performance regression, and the author does | 129 certain that a specific CL caused a performance regression, and the author does |
| 130 not have an immediate plan to address the problem, please revert the CL. | 130 not have an immediate plan to address the problem, please revert the CL. |
| 131 | 131 |
| 132 ###<a name="feedback"></a> Give Feedback on our Infrastructure | 132 ##<a name="feedback"></a> Give Feedback on our Infrastructure |
| 133 | 133 |
| 134 Perf regression sheriffs have their eyes on the perf dashboard and bisects | 134 Perf regression sheriffs have their eyes on the perf dashboard and bisects |
| 135 more than anyone else, and their feedback is invaluable for making sure these | 135 more than anyone else, and their feedback is invaluable for making sure these |
| 136 tools are accurate and improving them. Please file bugs and feature requests | 136 tools are accurate and improving them. Please file bugs and feature requests |
| 137 as you see them: | 137 as you see them: |
| 138 | 138 |
| 139 * **Perf Dashboard**: Please use the red "Report Issue" link in the navbar. | 139 * **Perf Dashboard**: Please use the red "Report Issue" link in the navbar. |
| 140 * **Perf Bisect/Trybots**: If a bisect is identifying the wrong CL as culprit | 140 * **Perf Bisect/Trybots**: If a bisect is identifying the wrong CL as culprit |
| 141 or missing a clear culprit, or not reproducing what appears to be a clear | 141 or missing a clear culprit, or not reproducing what appears to be a clear |
| 142 regression, please link the comment the bisect bot posted on the bug at | 142 regression, please link the comment the bisect bot posted on the bug at |
| 143 [go/bad-bisects](https://docs.google.com/spreadsheets/d/13PYIlRGE8eZzsrSocA3SR
2LEHdzc8n9ORUoOE2vtO6I/edit#gid=0). | 143 [go/bad-bisects](https://docs.google.com/spreadsheets/d/13PYIlRGE8eZzsrSocA3SR
2LEHdzc8n9ORUoOE2vtO6I/edit#gid=0). |
| 144 The team triages these regularly. If you spot a really clear bug (bisect | 144 The team triages these regularly. If you spot a really clear bug (bisect |
| 145 job red, bugs not being updated with bisect results) please file it in | 145 job red, bugs not being updated with bisect results) please file it in |
| 146 crbug with label `Cr-Tests-AutoBisect`. | 146 crbug with label `Cr-Tests-AutoBisect`. |
| 147 * **Noisy Tests**: Please file a bug in crbug with label `Cr-Tests-Telemetry` | 147 * **Noisy Tests**: Please file a bug in crbug with label `Cr-Tests-Telemetry` |
| 148 and [cc the owner](http://go/perf-owners). | 148 and [cc the owner](http://go/perf-owners). |
| OLD | NEW |