Add testing section of docs

site/dev/testing/trybots.md

+Skia Trybots

[borenet, 2015/02/24 14:13:46]

Are we talking about buildbots or trybots here?  I generally think of trybots as
a subset of buildbots.

+============
+
+Overview
+--------
+
+Like the Chromium team, the Skia team uses [buildbot](http://trac.buildbot.net/)
+to run continuous builds and tests.
+
+Here is a link to our main buildbot page, from which you can also access our
+Android, Compile, and FYI test consoles\:
+  
+  Externally-facing: http://build.chromium.org/p/client.skia/console
+  
+  Internally-facing: http://chromegw.corp.google.com/i/client.skia/console
+                     http://chromegw.corp.google.com/i/client.skia.internal/console

[borenet, 2015/02/24 14:13:46]

While these links are still useful for tracking down issues, I think it's time
to push status.skia.org as the primary view for the buildbots.

+                     \(only visible internally\)
+
+Architecture
+------------
+
+The buildbot system consists of these elements: \(see
+http://buildbot.net/buildbot/docs/current/manual/introduction.html#system-architecture
+for more detail\) 
+
+* buildbot master
+    
+    * watches for new commits to land in the Skia repository 
+      \(https://skia.googlesource.com/skia\) 
+    * whenever a new commit lands, it tells buildbot slaves to start building and 
+      testing the latest revision 
+    * serves up status pages whenever anybody requests them
+
+* buildslave \(or "buildbot slave"\)
+    
+    * a machine that builds and runs code as directed by the buildbot master

[borenet, 2015/02/24 14:13:45]

More precisely, a process running on a machine.  The machine itself is the
buildslave host, and it may be running more than one buildslave process.

+    * one or more builders run on each buildslave
+
+* builder
+    
+    * one of perhaps many build configurations for a particular buildslave \(for
+      example, we have Debug and Release builds for each platform\)
+    * each builder maintains its own local checkout of the Skia repo
+    * only one builder is running at any given time on any single buildslave; otherwise, 
+       different builders could interfere with each other's performance numbers

[borenet, 2015/02/24 14:13:45]

Can we move this above buildslaves (maybe to the very top) since everything is
builder-centric?  I think we could also simplify the description to something
like:

* One repeatable build and/or test configuration on a given platform.

And can we add a section for "build"?

* build

    * One run of a particular build/test configuration, at a particular code
revision

+
+
+Source Code Revisions

[borenet, 2015/02/24 14:13:46]

I don't think this section is needed.  At most (if only because it bites us
whenever we land a cherry-pick on a branch), we could make a note that the Skia
code and the buildbot code are versioned independently of each other and are
only guaranteed to work properly together when both are at tip-of-tree.

+---------------------
+
+Because of the distributed nature of the buildbot, there may be several
+different revisions of the buildbot code in play at any one time:
+
+1. Current head of "master" branch \(or close to it\): checked out on slaves, run in
+   response to directives coming from the master
+2. The head of the "master" branch the last time we restarted the master.  Behavior
+   of the master-side buildbot Python code is governed by this revision.
+3. The revision currently checked out on the master, between \#1 and \#2. Behavior of
+   master-served HTML and run-on-the-client Javascript is governed by this
+   revision.
+4. Revision of the HTML/JS running on each client.  Should be close to
+    master_checkedout_revision but may lag behind \(especially if the client has a
+    really long refresh time set\).
+
+
+Console View

[borenet, 2015/02/24 14:13:46]

Again, I think this is useful information, but I'd prefer we focus on the new UI
and treat the old console as something rarely used.

+------------
+
+The buildbot master process watches for new commits to land in the Skia
+repository \(https://skia.googlesource.com/skia\).  Whenever a new commit is
+made, the buildbot master tells slaves to start building and testing the latest
+revision.
+
+If the new commits land faster than the buildbots have time to cycle through all
+buildbot types, then not every revision will be tested individually.  Instead,
+if the commits are on the same branch, they will be batched into a single build.
+
+The console view shows a table with builders on the X-axis and commits on the
+Y-axis.  The cells are colored in according to the status of the build for each
+commit: green indicates success, red indicates failure, orange indicates
+repeated failure, yellow indicates an in-progress build, and white indicates
+that no build has started yet for a given revision.
+
+For more detail, you can click on an individual cell to get a summary of the
+steps which ran for that build.  You can also click one of the colored bars at
+the top of each column to see a summary of recent builds for a given builder.
+
+
+Submitting a Try from Rietveld

[borenet, 2015/02/24 14:13:45]

This seems out of place here.  Maybe it belongs in a "how to create and land a
code change" doc instead?

+------------------------------
+
+The easiest way to submit a try is from the Rietveld web UI.  To submit a try
+this way, simply click the "try more bots" link underneath the desired patch
+set, and a menu will pop up so that you can choose one or more trybots to run.
+
+