Kill builds that produce too much output

Kill builds that produce too much output

This gums up the buildbot master

The limit is set to 50 MB so far (per step, but note that annotated builds have a step called "steps" or "slave_steps" which contains output of other steps).

BUG=382648

[cmp, 2014-06-10 15:51:23]

Thank you for taking this on.  This type of solution is exactly what we
discussed the last time we saw this issue.  I think this will help us avoid many
problems in this class.

Some comments inline.

Update the description to include the limit and mention explicitly that this
only stops the build if a particular step exceeds the limit.

This change will be more successful, actually, if you limit it to take effect
only on tryserver.chromium.  How can you do that?

Verify that 50 MB is the right limit.  Should it be higher because there are
already steps that produce that much output?  Should it be lower because there
are no steps that produce more than N MB (where N < 50) in output?  Is there a
specific N where the tryserver will fall over?

Plan to send a PSA to alert chromium-dev signaling that this is coming.

https://codereview.chromium.org/325963006/diff/1/third_party/buildbot_slave_8...
File third_party/buildbot_slave_8_4/buildslave/runprocess.py (right):

https://codereview.chromium.org/325963006/diff/1/third_party/buildbot_slave_8...
third_party/buildbot_slave_8_4/buildslave/runprocess.py:215: OUTPUT_LIMIT =
50*1024*1024
Add an inline comment ("Limit is 50 MB.")

Did you pick 50 MB based on reviewing the output in the tryserver directory?  If
you land this and any steps already produce that much output, we will have a bad
day.

And if no steps produce more than 20 MB today, let's lower this to 20 MB.  Or to
whatever amount may happen to be less than 50 MB.

[Paweł Hajdan Jr., 2014-06-10 17:32:27]

On 2014/06/10 15:51:23, cmp wrote:
> Update the description to include the limit and mention explicitly that this
> only stops the build if a particular step exceeds the limit.

Done.

> This change will be more successful, actually, if you limit it to take effect
> only on tryserver.chromium.  How can you do that?

It's not obvious how to do that, and even if we found a way, it seems a bit
hacky. I see a point, and then there is another one - reducing differences
between the tryserver and the main waterfall. With that condition something
could pass on the main waterfall and then break all TS builds.

> Verify that 50 MB is the right limit.  Should it be higher because there are
> already steps that produce that much output?

I'll double-check that.

> Should it be lower because there
> are no steps that produce more than N MB (where N < 50) in output?

I've seen 30 MB cases which look legitimate.

> Is there a
> specific N where the tryserver will fall over?

Not entirely sure, we discussed some ideas during today's standup meeting. It
seems to me at 100 MB it's already too late, but maybe if we catch it there and
kill the build (what this CL is doing) it'd be fine.

In other words, Sergey Berezin pointed out it's likely the rate of log sending
as opposed to just the log size.

> Plan to send a PSA to alert chromium-dev signaling that this is coming.

OK

>
https://codereview.chromium.org/325963006/diff/1/third_party/buildbot_slave_8...
> File third_party/buildbot_slave_8_4/buildslave/runprocess.py (right):
> 
>
https://codereview.chromium.org/325963006/diff/1/third_party/buildbot_slave_8...
> third_party/buildbot_slave_8_4/buildslave/runprocess.py:215: OUTPUT_LIMIT =
> 50*1024*1024
> Add an inline comment ("Limit is 50 MB.")
> 
> Did you pick 50 MB based on reviewing the output in the tryserver directory? 
If
> you land this and any steps already produce that much output, we will have a
bad
> day.
> 
> And if no steps produce more than 20 MB today, let's lower this to 20 MB.  Or
to
> whatever amount may happen to be less than 50 MB.

[cmp, 2014-06-10 18:12:49]

>> This change will be more successful, actually, if you limit it to take effect
>> only on tryserver.chromium.  How can you do that?
>
> It's not obvious how to do that, and even if we found a way, it seems a bit
> hacky. I see a point, and then there is another one - reducing differences
> between the tryserver and the main waterfall. With that condition something
> could pass on the main waterfall and then break all TS builds.

I think we're fully in "hacky" in this point.  If we don't, the complexity of
making this change goes up greatly.  For instance, you will need to scan all
masters on all master machines before proceeding, and PSA all groups to make
this change.  Please figure out how to do this.

As for the concern about the Chromium tree allowing something the tryservers
wouldn't, we can include Chromium masters in the list of masters that have this
check, too.

[ghost stip (do not use), 2014-06-11 18:16:08]

Note: annotator 'steps' stdio logs can be unbounded. A previously big offender
was android_dbg_triggered_tests which would emit logcat logs directly into stdio
(and thus both the annotator and step log). Thanks to the fix in
https://crbug.com/362787 this has been resolved, and
https://build.chromium.org/p/tryserver.chromium/builders/android_dbg_triggere...
is about 14 megs.

[cmp, 2015-01-29 01:01:54]

cmp@chromium.org changed reviewers:
- cmp@chromium.org