Updating memory.top_10_mobile benchmark to stop ignoring max, std, and count.

Updating memory.top_10_mobile benchmark to stop ignoring max, std, and count.

We are going to be using these metrics for the system health dashboard.

BUG=catapult:#3141
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.perf:linux_perf_cq;master.tryserver.chromium.perf:mac_retina_perf_cq;master.tryserver.chromium.perf:winx64_10_perf_cq

[jessimb, 2017-01-24 22:32:37]

Description was changed from

==========
Updating memory.top_10_mobile benchmark to stop ignoring max, std, and count.

BUG=catapult:#3141
==========

to

==========
Updating memory.top_10_mobile benchmark to stop ignoring max, std, and count.

BUG=catapult:#3141
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.perf:linux_perf_cq;master.tryserver.chromium.perf:mac_retina_perf_cq;master.tryserver.chromium.perf:winx64_10_perf_cq
==========

[jessimb, 2017-01-24 22:37:12]

jessimb@chromium.org changed reviewers:
+ eakuefner@chromium.org, sullivan@chromium.org

[jessimb, 2017-01-24 22:37:12]

PTAL.

We are turning max, count, and std back on for memory.top_10_mobile. This is for
the speed_releasing table/overall effort.

[sullivan, 2017-01-24 22:46:48]

Is this meant for submission? Or just for load testing/

[eakuefner, 2017-01-24 22:58:36]

It's meant partially as a load test, but partially so that the health table MVP
can work. If this succeeds I think we should keep it in.

lgtm, but please wait for Annie to confirm.

[sullivan, 2017-01-24 23:13:46]

lgtm

Please update the CL description to say we're going to use this for system
health dashboard, otherwise it's pretty confusing to the benchmark owners why we
added this back in.

Adding perezju as FYI

[jessimb, 2017-01-24 23:29:58]

Description was changed from

==========
Updating memory.top_10_mobile benchmark to stop ignoring max, std, and count.

BUG=catapult:#3141
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.perf:linux_perf_cq;master.tryserver.chromium.perf:mac_retina_perf_cq;master.tryserver.chromium.perf:winx64_10_perf_cq
==========

to

==========
Updating memory.top_10_mobile benchmark to stop ignoring max, std, and count.

We are going to be using these metrics for the system health dashboard.

BUG=catapult:#3141
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.perf:linux_perf_cq;master.tryserver.chromium.perf:mac_retina_perf_cq;master.tryserver.chromium.perf:winx64_10_perf_cq
==========

[perezju, 2017-01-25 00:10:58]

perezju@chromium.org changed reviewers:
+ perezju@chromium.org

[perezju, 2017-01-25 00:10:58]

Please let me think a bit about this before landing. Will comment back tomorrow.

[perezju, 2017-01-26 22:08:58]

On 2017/01/25 00:10:58, perezju wrote:
> Please let me think a bit about this before landing. Will comment back
tomorrow.

Sorry for the delay, swamped by many other things.

So, after thinking about it, I don't think this would do the right thing at all.

First, because the benchmarks only take a single memory dump per story, both the
_max and the _avg metrics always report the exact same value; and then taking
the average (across stories) of these individual "_max" values would be
meaningless (well, it will just be the regular average that we already have).

Now, the more meaningful value for "std" you already have in the dashboard
stored as "error" in the Row entity [1] (which was computed by Telemetry using
pooled stdev to account for data coming from different distributions, so this is
statistically sound and meaningful).

Could it be possible, on that same spot where you get the std [2], also compute
the max from the  list_of_scalar_values and somehow store it in the Row?

I would say there is no much need to worry about the "count", we added it there
mostly as a signal to figure if there were any issues with the benchmark. But
today it is much more stable, and we can glean that kind of information
elsewhere.

[1]:
https://github.com/catapult-project/catapult/blob/master/dashboard/dashboard/...
[2]:
https://github.com/catapult-project/catapult/blob/master/dashboard/dashboard/...

[eakuefner, 2017-01-27 22:16:31]

I think we landed offline on closing this CL because it's no longer needed,
right?

[jessimb, 2017-01-27 23:18:50]

On 2017/01/27 at 22:16:31, eakuefner wrote:
> I think we landed offline on closing this CL because it's no longer needed,
right?

Yes. Closing now.