Added documentation for running perf bisects and tryjobs.

docs/speed/bisects.md

+# Bisecting Performance Regressions
+
+[TOC]
+
+## What are performance bisects?
+
+The perf tests on chromium's continuous build are very long-running, so we
+cannot run them on every revision. Further, separate repositories like v8
+and skia sometimes roll multiple performance-sensitive changes into chromium
+at once. For these reasons, we need a tool that can bisect the root cause of
+performance regressions over a CL range, descending into third_party
+repositories as necessary. This is what the performance bisect bots do.
+
+The team is currently working on a new version of performance biscect called
+[pinpoint](https://docs.google.com/document/d/1FKPRNU2kbPJ15p6XHO0itCjYtfvCpGt2IHblriTX1tg/edit)
+
+## Starting a perf bisect
+
+Performance bisects are tightly integrated with the
+[Chrome Performance Dashboard](https://chromeperf.appspot.com/alerts) and
+[monorail](https://bugs.chromium.org/p/chromium/issues/list). Users kick off
+perf bisects on the perf dashboard and view results in monorail.
+
+You can kick off a perf bisect anywhere you see a performance graph on the perf
+dashboard (except for some tests which don't bisect, because they do not run on
+the [chromium.perf waterfall](https://build.chromium.org/p/chromium.perf/waterfall)).
+
+### To get to a graph, use one of the following methods:
+
+  * From a perf sheriff-filed bug, follow the link in #1 that looks like
+    `https://chromeperf.appspot.com/group_report?bug_id=XXXXXX`. Check the
+    boxes next to alerts in the table to display graphs.
+  * From the [alerts page](https://chromeperf.appspot.com/alerts), check the
+    box next to an alert and click the `Graph` button.
+  * From the [report page](https://chromeperf.appspot.com/report), use the menu
+    to navigate to the graph you want.
+
+### To kick off a bisect from the graph:
+
+![Bisecting on a performance graph](images/bisect_graph.png)
+![The bisect dialog](images/bisect_dialog.png)
+
+  1. Click on a data point in the graph.
+  2. In the tooltip that shows up, click the `BISECT` button.
+  3. Make sure to enter a Bug ID in the dialog that comes up.
+  4. Click the `START BISECT` button.
+
+### What are all the boxes in the form?
+
+  * **Bisect bot**: The name of the configuration in the perf lab to bisect on.
+    This has been prefilled to match the bot that generated the graph as
+    closely as possible.
+  * **Metric**: The metric of the performance test to bisect. This defaults to
+    the metric shown on the graph. It shows a list of other related metrics
+    (for example, if average page load time increased, the drop down will show
+    a list of individual pages which were measured).
+  * **Story filter**: This is a flag specific to
+    [telemetry](https://github.com/catapult-project/catapult/blob/master/telemetry/README.md).
+    It tells telemetry to only run a specific test case, instead of running all
+    the test cases in the suite. This dramatically reduces bisect time for
+    large test suites. The dashboard will prefill this box based on the graph
+    you clicked on. If you suspect that test cases in the benchmark are not
+    independent, you can try bisecting with this box cleared.
+  * **Bug ID**: The bug number in monorail. It's very important to fill in
+    this field, as this is where bisect results will be posted.
+  * **Earlier revision**: The chromium commit pos to start bisecting from. This
+    is prefilled by the dashboard to the start of the revision range for the
+    point you clicked on. You can set it to an earlier commit position to
+    bisect a larger range.
+  * **Later revision**: The chromium commit pos to bisect to. This is prefilled
+    by the dashboard to the end of the revision range for the point you clicked
+    on. You can set it to a later commit pos to bisect a larger range.
+  * **Launch on staging bots**: This is an internal feature, which allows the
+    bisect team to launch a bisect on a test configuration. You likely don't
+    want to check this box unless instructed by the bisect team.
+  * **Bisect mode**: use "mean" to bisect the mean value of the performance
+    test. See below for "return_code".
+
+## Bisecting test failures
+
+The perf bisect bots can also be used to bisect performance test failures.
+See details in [Triaging Data Stoppage Alerts](triaging_data_stoppage_alerts.md).
+
+## Interpreting the results
+
+The bisect bot will output a comment on the bug you input into the dialog when
+bisection is complete. See the
+[Understanding the Bisect Results](addressing_performance_regressions.md#Understanding-the-bisect-results)
+section of the Addressing Performance Regressions doc for details on how to
+interpret the results.
+
+## Getting more debugging data
+
+The bisect outputs some additional data which might be useful for really tough
+regressions or confusing results.
+
+### Traces
+
+Chrome traces are generated by most bisects and uploaded to cloud storage, but
+they're not very visible in the UI. We plan to address this in
+[pinpoint](https://docs.google.com/document/d/1FKPRNU2kbPJ15p6XHO0itCjYtfvCpGt2IHblriTX1tg/edit),
+but in the short term here are the steps to get the traces from a bisect:
+
+  1. The bisect comment should have a "Debug Info" link that looks like this:
+     `https://chromeperf.appspot.com/buildbucket_job_status/8980436717323504240`
+     Click it.
+  2. In the debug info, you should see a "Buildbot link" that looks like this:
+     `https://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus7_perf_bisect/builds/4097`
+     Click it.
+  3. There will be several steps on the buildbot status page named "Bisecting
+     Revision". Each has an annotation like "Revision: chromium@474894" so you
+     can tell which revision it ran. Pick the commit position you want the
+     trace from (usually the one at your CL and the one immediately before).
+     Click the arrow by "> Nested step(s) for: Bisecting revision..." on those
+     steps.
+  4. In the nested steps, you'll see several steps titled "Bisecting
+     revision.Performance Test X of Y". These are the actual perf test runs.
+     Click the "stdout" link for one of these steps.
+  5. In the output, do a text search for "View generated trace files online"
+     and you'll see a link to a trace that looks like this:
+     `https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/trace-file-id_0-2017-05-05_05-41-49-83206.html`
+
+Here are some screenshots showing what to click on:
+
+![Finding the Bisecting Revision Step](images/bisecting_revision_step.png)
+![Getting to the stdout](images/bisect_stdout.png)