Add alerting for CTFE V2.

Add alerting for CTFE V2. Currently configured for ct-staging for testing. We can change that to the non-staging instance once that is set up.

Configure prober to hit all HTML pages and all "get" JSON endpoints. Add some specific checks as new body testers. Also check that there is at least one valid chromium build and at least one valid SKP repository.

Refactor functions that get the oldest pending task. Create a new package "task_types" and move supportedTasks there (as a function to allow modifying the returned tasks).

Add metrics collection as a goroutine in CTFE. Report stats on pending tasks.

Set the graphite_server flag in ctfe.service.

BUG=skia:4074
Committed: https://skia.googlesource.com/buildbot/+/b82e9e56fcd5a62091bda7c00bd3be8d7e762c8b

[dogben, 2015-08-26 13:37:58]

Patchset #1 (id:1) has been deleted

[dogben, 2015-08-26 13:38:51]

benjaminwagner@google.com changed reviewers:
+ jcgregorio@google.com, rmistry@google.com

[jcgregorio, 2015-08-26 14:40:45]

LGTM modulo the nit with wrapping lines in Go.

https://codereview.chromium.org/1307333002/diff/20001/ct/go/ctfe/main.go
File ct/go/ctfe/main.go (right):

https://codereview.chromium.org/1307333002/diff/20001/ct/go/ctfe/main.go#newc...
ct/go/ctfe/main.go:118: pendingTasksGauge :=
metrics.GetOrRegisterGauge("num-pending-tasks",
Don't wrap long lines. Here and below.

[dogben, 2015-08-26 15:24:01]

https://codereview.chromium.org/1307333002/diff/20001/ct/go/ctfe/main.go
File ct/go/ctfe/main.go (right):

https://codereview.chromium.org/1307333002/diff/20001/ct/go/ctfe/main.go#newc...
ct/go/ctfe/main.go:118: pendingTasksGauge :=
metrics.GetOrRegisterGauge("num-pending-tasks",
On 2015/08/26 14:40:44, jcgregorio wrote:
> Don't wrap long lines. Here and below.

Done.

[rmistry, 2015-08-26 20:53:27]

lgtm

https://codereview.chromium.org/1307333002/diff/40001/go/common/common.go
File go/common/common.go (left):

https://codereview.chromium.org/1307333002/diff/40001/go/common/common.go#old...
go/common/common.go:59: glog.Warningf("No metrics server speicifed.")
Thank you. This bothered me this morning.

[dogben, 2015-08-26 21:19:39]

The CQ bit was checked by benjaminwagner@google.com

[dogben, 2015-08-26 21:19:40]

The patchset sent to the CQ was uploaded after l-g-t-m from
jcgregorio@google.com
Link to the patchset: https://codereview.chromium.org/1307333002/#ps40001
(title: "Reformat")

[commit-bot: I haz the power, 2015-08-26 21:19:48]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1307333002/40001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1307333002/40001

[commit-bot: I haz the power, 2015-08-26 21:20:39]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2015-08-26 21:20:39]

Try jobs failed on following builders:
  Infra-PerCommit-Trybot on client.skia.fyi (JOB_FAILED,
http://build.chromium.org/p/client.skia.fyi/builders/Infra-PerCommit-Trybot/b...)

[dogben, 2015-08-26 21:42:35]

The CQ bit was checked by benjaminwagner@google.com

[dogben, 2015-08-26 21:42:35]

The patchset sent to the CQ was uploaded after l-g-t-m from
jcgregorio@google.com, rmistry@google.com
Link to the patchset: https://codereview.chromium.org/1307333002/#ps60001
(title: "Rebase.")

[commit-bot: I haz the power, 2015-08-26 21:42:47]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1307333002/60001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1307333002/60001

[commit-bot: I haz the power, 2015-08-26 21:44:17]

Committed patchset #3 (id:60001) as
https://skia.googlesource.com/buildbot/+/b82e9e56fcd5a62091bda7c00bd3be8d7e76...