Enable compression of any perf logs greater than 10 MB.

Enable compression of any perf logs greater than 10 MB.

This should drastically improve master1's disk space issues.

BUG=584700
Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=299510

[bpastene, 2016-03-26 00:17:25]

bpastene@chromium.org changed reviewers:
+ dba@chromium.org, dtu@chromium.org, nednguyen@google.com,
phajdan.jr@chromium.org, sullivan@chromium.org

[bpastene, 2016-03-26 00:17:25]

ptal at this spicy change

+ perf folks for their opinion
+ sys-admin folks for their advice
+ phajdan@ for his experience with this on the tryservers

This has the potential to bog down master1 even further due to the additional
cpu cost of compressing logs as they arrive. However, with such a large
threshold, this will hopefully only happen for a very small amount of files.

Note that all the main tryservers have been running with compression enabled at
a mere 4 KB limit for 6 months now, and seem to be doing alright.

[nednguyen, 2016-03-26 00:24:38]

On 2016/03/26 00:17:25, bpastene wrote:
> ptal at this spicy change
> 
> + perf folks for their opinion
> + sys-admin folks for their advice
> + phajdan@ for his experience with this on the tryservers
> 
> This has the potential to bog down master1 even further due to the additional
> cpu cost of compressing logs as they arrive. However, with such a large
> threshold, this will hopefully only happen for a very small amount of files.
> 
> Note that all the main tryservers have been running with compression enabled
at
> a mere 4 KB limit for 6 months now, and seem to be doing alright.

Does this mean that for big perf logs, sherrifs will need to download the file,
then unzip it?

[bpastene, 2016-03-26 00:31:44]

On 2016/03/26 00:24:38, nednguyen wrote:
> Does this mean that for big perf logs, sherrifs will need to download the
file,
> then unzip it?

Nope, it'll work and look just like it does now. The master will automatically
do the unzipping for you and serve the plaintext as if it was never compressed.
Any of the 'stdio' links on
https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium...
are compressed, for example, but you'd never know it.

However, there will be a slight overhead in cpu usage & time for the master
machine in unzipping the files before sending it to your browser, and that's
what I'm worried about.

[nednguyen, 2016-03-26 00:41:16]

On 2016/03/26 00:31:44, bpastene wrote:
> On 2016/03/26 00:24:38, nednguyen wrote:
> > Does this mean that for big perf logs, sherrifs will need to download the
> file,
> > then unzip it?
> 
> Nope, it'll work and look just like it does now. The master will automatically
> do the unzipping for you and serve the plaintext as if it was never
compressed.
> Any of the 'stdio' links on
>
https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium...
> are compressed, for example, but you'd never know it.
> 
> However, there will be a slight overhead in cpu usage & time for the master
> machine in unzipping the files before sending it to your browser, and that's
> what I'm worried about.

I think only Chrome engs access these logs, so assuming qps is  a lot less than
1, we don't have to worry too much about the cpu usage overhead?

lgtm from my point of view

[sullivan, 2016-03-28 14:47:45]

lgtm

[bpastene, 2016-03-28 17:07:28]

The CQ bit was checked by bpastene@chromium.org

[commit-bot: I haz the power, 2016-03-28 17:07:31]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1831373004/1
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1831373004/1

[commit-bot: I haz the power, 2016-03-28 17:11:09]

Description was changed from

==========
Enable compression of any perf logs greater than 10 MB.

This should drastically improve master1's disk space issues.

BUG=584700
==========

to

==========
Enable compression of any perf logs greater than 10 MB.

This should drastically improve master1's disk space issues.

BUG=584700

Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=299510
==========

[commit-bot: I haz the power, 2016-03-28 17:11:09]

Committed patchset #1 (id:1) as
http://src.chromium.org/viewvc/chrome?view=rev&revision=299510