Make Buildbot masters send ts_mon metrics, and add standard HTTP server metrics.

Make Buildbot masters send ts_mon metrics, and add standard HTTP server metrics.

BUG=620610
Committed: https://chromium.googlesource.com/chromium/tools/build/+/301489eb072d508054ef27bb4c6027246bd1f3d2

[dsansome, 2016-06-28 11:26:45]

dsansome@chromium.org changed reviewers:
+ agable@chromium.org, stip@chromium.org

[dsansome, 2016-06-28 11:29:06]

Also +chrishall,friedman FYI

[agable, 2016-06-28 16:26:24]

High-level comment: doing this as a changesource poller is odd. Are there other
alternative ways to build this in to the master that you considered? In
particular, buildbot has the whole StatusReceiver architecture which I think
could both be leveraged to record a ton of relevant metrics (see
scripts/master/status_logger.py), and I think could be used to flush them as
well.

[agable, 2016-06-28 22:23:05]

And regarding your question about SVN DEPS (which should probably be removed
from the CL description): https://codereview.chromium.org/1644863004

[dsansome, 2016-06-29 01:35:49]

Description was changed from

==========
Make Buildbot masters send ts_mon metrics, and add standard HTTP server metrics.

I'm not sure why we didn't do this before - too scared to make buildbot depend
on infra_libs?  If this works I'd like to move mastermon's stuff in here too.

Also why are there still svn URLs in DEPS?  I thought we migrated everything
to git now?

BUG=620610
==========

to

==========
Make Buildbot masters send ts_mon metrics, and add standard HTTP server metrics.

BUG=620610
==========

[dsansome, 2016-06-29 03:26:59]

On 2016/06/28 22:23:05, agable wrote:
> High-level comment: doing this as a changesource poller is odd. Are there
other
> alternative ways to build this in to the master that you considered? In
> particular, buildbot has the whole StatusReceiver architecture which I think
> could both be leveraged to record a ton of relevant metrics (see
> scripts/master/status_logger.py), and I think could be used to flush them as
> well.

Sure, a changesource seemed like a good idea because it did the loop for me, but
I agree it doesn't really fit.  Changed to a StatusReceiverMultiService.

> And regarding your question about SVN DEPS (which should probably be removed
> from the CL description): https://codereview.chromium.org/1644863004

Thanks.  How come that wasn't submitted?

[agable, 2016-06-29 20:17:59]

https://codereview.chromium.org/2103073002/diff/20001/.DEPS.git
File .DEPS.git (right):

https://codereview.chromium.org/2103073002/diff/20001/.DEPS.git#newcode28
.DEPS.git:28: 'infra':
This either doesn't work, or is very risky/confusing. infra.git already has a
dependency on build/; turning that into a cyclic dependency is ill-advised, and
something that we have actively fought against in the past.

https://codereview.chromium.org/2103073002/diff/20001/scripts/master/monitori...
File scripts/master/monitoring_status_receiver.py (right):

https://codereview.chromium.org/2103073002/diff/20001/scripts/master/monitori...
scripts/master/monitoring_status_receiver.py:26: self.loop.start(60, now=False)
Nice, I like this a lot better than using a changesource.

[Sergey Berezin, 2016-07-22 01:41:48]

sergeyberezin@chromium.org changed reviewers:
+ sergeyberezin@chromium.org

[Sergey Berezin, 2016-07-22 01:41:49]

drive-by:

https://codereview.chromium.org/2103073002/diff/20001/.DEPS.git
File .DEPS.git (right):

https://codereview.chromium.org/2103073002/diff/20001/.DEPS.git#newcode28
.DEPS.git:28: 'infra':
On 2016/06/29 20:17:59, agable wrote:
> This either doesn't work, or is very risky/confusing. infra.git already has a
> dependency on build/; turning that into a cyclic dependency is ill-advised,
and
> something that we have actively fought against in the past.

Right. It probably "works" locally because you used build.git DEPS'ed from
infra.git (?), but in the wild, where build.git is checked out separately, it'll
likely make gclient explode. Let's use vendoring with glyco, or maybe DEPSing
https://chromium.googlesource.com/infra/infra/infra_libs .

But - fair warning: the above repo will soon change drastically to become an
actual python package.

[dsansome, 2016-08-08 06:51:17]

I've updated this to DEPS infra/infra/packages/infra_libs.git into third_party. 
PTAL!

[agable, 2016-08-08 16:43:53]

I think this LGTM; get confirmation from sergeyberezin as well. Even though it's
not using infra_libs as a pip package, I think this is the right way to do
code-reuse into build/.

[Sergey Berezin, 2016-08-08 17:18:31]

LGTM + a couple of comments. Thanks!

I was going to DEPS infra_libs into build this morning, but you did a better job
faster :-)

https://codereview.chromium.org/2103073002/diff/40001/DEPS
File DEPS (right):

https://codereview.chromium.org/2103073002/diff/40001/DEPS#newcode15
DEPS:15:
'https://chromium.googlesource.com/infra/infra/packages/infra_libs.git',
I'd also pin the dependency, to prevent breaking all the fleet on upstream
changes. I don't believe we run build tests on infra changes, even though build
is in infra's dependencies.

https://codereview.chromium.org/2103073002/diff/40001/environment.cfg.py
File environment.cfg.py (right):

https://codereview.chromium.org/2103073002/diff/40001/environment.cfg.py#newc...
environment.cfg.py:43: 'infra_libs',
nit: sort alphabetically (I hope the order is not important?)

[dsansome, 2016-08-09 04:29:04]

https://codereview.chromium.org/2103073002/diff/40001/DEPS
File DEPS (right):

https://codereview.chromium.org/2103073002/diff/40001/DEPS#newcode15
DEPS:15:
'https://chromium.googlesource.com/infra/infra/packages/infra_libs.git',
On 2016/08/08 17:18:31, Sergey Berezin wrote:
> I'd also pin the dependency, to prevent breaking all the fleet on upstream
> changes. I don't believe we run build tests on infra changes, even though
build
> is in infra's dependencies.

Good idea, done.

https://codereview.chromium.org/2103073002/diff/40001/environment.cfg.py
File environment.cfg.py (right):

https://codereview.chromium.org/2103073002/diff/40001/environment.cfg.py#newc...
environment.cfg.py:43: 'infra_libs',
On 2016/08/08 17:18:31, Sergey Berezin wrote:
> nit: sort alphabetically (I hope the order is not important?)

I guess site-packages needs to go at the end, but yeah I hope order shouldn't
matter apart from that.
Done.  #YOLO.

[dsansome, 2016-08-09 04:33:26]

The CQ bit was checked by dsansome@chromium.org

[dsansome, 2016-08-09 04:33:26]

The patchset sent to the CQ was uploaded after l-g-t-m from
sergeyberezin@chromium.org, agable@chromium.org
Link to the patchset: https://codereview.chromium.org/2103073002/#ps60001
(title: "Pin infra_libs and sort some pythonpath entries")

[commit-bot: I haz the power, 2016-08-09 04:53:26]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-08-09 05:01:37]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-08-09 05:01:38]

Try jobs failed on following builders:
  Build Presubmit on luci.infra.try (JOB_FAILED,
https://luci-milo.appspot.com/swarming/task/30867b40484b0310)

[dsansome, 2016-08-09 06:43:13]

This was getting a bit big so I've split out the infra_libs vendoring into
https://codereview.chromium.org/2228733003/.

[Sergey Berezin, 2016-08-09 21:34:14]

lgtm

[dsansome, 2016-08-10 05:06:28]

The CQ bit was checked by dsansome@chromium.org

[dsansome, 2016-08-10 05:06:29]

The patchset sent to the CQ was uploaded after l-g-t-m from agable@chromium.org
Link to the patchset: https://codereview.chromium.org/2103073002/#ps80001
(title: "Rebase on https://codereview.chromium.org/2228733003/")

[commit-bot: I haz the power, 2016-08-10 05:06:38]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-08-10 05:14:25]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-08-10 05:14:26]

Try jobs failed on following builders:
  Build Presubmit on luci.infra.try (JOB_FAILED,
https://luci-milo.appspot.com/swarming/task/308badafbcdb9d10)

[dsansome, 2016-08-11 04:18:13]

The CQ bit was checked by dsansome@chromium.org

[dsansome, 2016-08-11 04:18:13]

The patchset sent to the CQ was uploaded after l-g-t-m from
sergeyberezin@chromium.org, agable@chromium.org
Link to the patchset: https://codereview.chromium.org/2103073002/#ps120001
(title: "Oops")

[commit-bot: I haz the power, 2016-08-11 04:18:21]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-08-11 04:22:37]

Description was changed from

==========
Make Buildbot masters send ts_mon metrics, and add standard HTTP server metrics.

BUG=620610
==========

to

==========
Make Buildbot masters send ts_mon metrics, and add standard HTTP server metrics.

BUG=620610

Committed:
https://chromium.googlesource.com/chromium/tools/build/+/301489eb072d508054ef...
==========

[commit-bot: I haz the power, 2016-08-11 04:22:38]

Committed patchset #7 (id:120001) as
https://chromium.googlesource.com/chromium/tools/build/+/301489eb072d508054ef...

[dsansome, 2016-08-11 05:05:53]

A revert of this CL (patchset #7 id:120001) has been created in
https://codereview.chromium.org/2234773005/ by dsansome@chromium.org.

The reason for reverting is: This broke
/b/build/scripts/tools/dump_master_cfg.py.