Truncate huge output snippets in the test launcher before printing them

Truncate huge output snippets in the test launcher before printing them

This avoids flooding the logs with lots of data that gums up
the infrastructure.

The summary contains the full snippet for debugging.

BUG=382648
R=sky@chromium.org

Committed: https://src.chromium.org/viewvc/chrome?view=rev&revision=276335

[sky, 2014-06-10 16:25:14]

LGTM

https://codereview.chromium.org/324893004/diff/1/base/test/launcher/test_laun...
File base/test/launcher/test_launcher.cc (right):

https://codereview.chromium.org/324893004/diff/1/base/test/launcher/test_laun...
base/test/launcher/test_launcher.cc:459: std::vector<std::string> snippet_lines;
This initially confused me. I would have thought you wanted the first 50 lines,
but rather you're showing the last. How about a comment?

[Sergey Berezin, 2014-06-10 16:25:53]

I'm not sure I understand the design assumptions here. The outage was caused by
the rate at which the logs were dumped, not the amount of logs per se. Do we
have a guarantee of some maximum number of snippets per second?

Also, would simply losing the log data be acceptable to solve this problem? How
do we diagnose that something is a problem with the test?

An outage like that is also notoriously difficult to diagnose at the moment. In
particular, there is no alert that I know of about the rate of log dumping
exceeding certain threshold, or which build / test is responsible for it.

I would vote for throttling the logs instead, without truncating them, and
effectively slowing down the test if it spits out too much logs. We can define
the exact policy for throttling (perhaps a "grace period" of, say, 100K of logs
at full speed followed by throttling, following by alert if exceeded 1MB while
throttled - just an example.)

Also, is this the best place to catch such logs? Are there tests that don't use
this launcher and can bypass it? If there are, is there another more common
place we can catch these logs and throttle them?

[Paweł Hajdan Jr., 2014-06-11 10:29:17]

Committed patchset #1 manually as r276335 (presubmit successful).

[Paweł Hajdan Jr., 2014-06-11 10:39:03]

On 2014/06/10 16:25:53, Sergey Berezin wrote:
> I'm not sure I understand the design assumptions here. The outage was caused
by
> the rate at which the logs were dumped, not the amount of logs per se.

Thanks for taking a look, Sergey.

I'm not convinced about the cause of the outage you've mentioned above. Do you
have data or any kind of evidence to back that claim?

> Do we have a guarantee of some maximum number of snippets per second?

No.

> Also, would simply losing the log data be acceptable to solve this problem?
How
> do we diagnose that something is a problem with the test?

Full snippet is preserved in the summary and uploaded to Google Storage by
runtest.py .

> An outage like that is also notoriously difficult to diagnose at the moment.
In
> particular, there is no alert that I know of about the rate of log dumping
> exceeding certain threshold, or which build / test is responsible for it.

Agreed. This is more of an infrastructure problem though - it seems the test
launcher
is not the piece we'd like to handle this. There are many other steps/scripts
that can flood the logs.

> I would vote for throttling the logs instead, without truncating them, and
> effectively slowing down the test if it spits out too much logs. We can define
> the exact policy for throttling (perhaps a "grace period" of, say, 100K of
logs
> at full speed followed by throttling, following by alert if exceeded 1MB while
> throttled - just an example.)

Specific recommendations are welcome. Please make sure to back them with
evidence
and explain why the test launcher would be the best place to implement such
logic
(that's not really obvious to me, given other scripts can flood the logs just as
well).

> Also, is this the best place to catch such logs?

I'm afraid at any other place we'd need to just kill the build. Here we have a
chance
to handle big log output gracefully.

This is not intended to be bulletproof
(https://codereview.chromium.org/325963006/ is,
and it's an infrastructure-side change).

> Are there tests that don't use this launcher and can bypass it?

Yes, e.g. gtest on Android or iOS, Telemetry...

> If there are, is there another more common
> place we can catch these logs and throttle them?

Please see https://codereview.chromium.org/325963006/ .