[memory-infra] Document memory benchmarks

[memory-infra] Document memory benchmarks

Add documentation for memory benchmarks, system health in particular, on what
they are, where are they tracked, how to run them, how to understand results.

BUG=catapult:#3035

Review-Url: https://codereview.chromium.org/2595613002
Cr-Commit-Position: refs/heads/master@{#442278}
Committed: https://chromium.googlesource.com/chromium/src/+/8f6b9c067322040ebd328b225ea5bc7c8f6ce9f0

[perezju, 2016-12-20 17:22:09]

perezju@chromium.org changed reviewers:
+ primiano@chromium.org

[perezju, 2016-12-20 17:22:10]

A first draft for documentation of memory benchmarks, sending out for early
feedback.

Also, couldn't find where/how to upload images like e.g. here:
https://chromium.googlesource.com/chromium/src/+show/master/docs/memory-infra...

[perezju, 2017-01-06 14:01:13]

Ping. For when you have some time :)

[Primiano Tucci (use gerrit), 2017-01-06 17:29:33]

Thanks a lot. LGTM, we can always improve them later.
As regards images, I created a while ago gs://chromium-docs.appspot.com
so just use gstuil to upload there. I marked both committers@chromium.org and
chrome-team@ editors, so you should be able to upload anything.
By default objects are world readable.

https://codereview.chromium.org/2595613002/diff/20001/docs/memory-infra/memor...
File docs/memory-infra/memory_benchmarks.md (right):

https://codereview.chromium.org/2595613002/diff/20001/docs/memory-infra/memor...
docs/memory-infra/memory_benchmarks.md:9: ## Benchmarks
Before this I'd add a very small recap of the form:

## Glossary:
User-story: a set of pages and actions on them (e.g., open google homepage, type
"foo", click search, scroll down)
Metric: a data aggregation process that takes a chrome trace in input (produced
by a TBM telemetry run) and produces a set of summary numbers in output (e.g.
total memory while foreground = X)
Benchmark: a combination of (one or more) user-stories and (one or more)
metrics.

https://codereview.chromium.org/2595613002/diff/20001/docs/memory-infra/memor...
docs/memory-infra/memory_benchmarks.md:21: Other benchmarks maintained by the
memory-infra team are:
This one I'd put in an appendix, like:
"There are other benchmarks currently maintained, see appendix X"

Statistically most chromium developers are hit by system health. The other
benchmarks here are just distracting here.

https://codereview.chromium.org/2595613002/diff/20001/docs/memory-infra/memor...
docs/memory-infra/memory_benchmarks.md:41: ## What is measured
Before getting here, I think there should be a section here named something
like:
"What are system health benchmarks?"
In part you describe these below in lines 80-100 but it's a bit too fragmented.
This is a recurring question and should be on the top.

So here I'd say something along those lines:
- System health is an effort to unify top-level benchmarks (as opposite to
microbenchmarks and regression tests) to capture representative user stories
(...). Today they are grouped into the following groups:
-load
-browse
-background
(essentially move here the lines 80-100)

Another important point to clarify here is: when do they sample and what do they
do w.r.t trigger gc?

https://codereview.chromium.org/2595613002/diff/20001/docs/memory-infra/memor...
docs/memory-infra/memory_benchmarks.md:43: These benchmarks are run continuously
on the [chromium.perf][] waterfall,
Move these 4 lines up when you say "the most important benchmarks are"

https://codereview.chromium.org/2595613002/diff/20001/docs/memory-infra/memor...
docs/memory-infra/memory_benchmarks.md:52: To view data from one of the
benchmarks you should select:
This should be a section called:
# How to interpret the output emitted by memory metrics?
maybe you can steal the equation picture from
https://docs.google.com/presentation/d/1iy_EVuCn8fuzXDqKTmmZeXeX91c2PKwsARCGP...

[perezju, 2017-01-09 17:01:46]

I've some some re-ordering and a bit of re-writing of sections. Do have a quick
pass if you want to provide further feedback, otherwise I'll land this probably
tomorrow and we can start iterating from there.

https://codereview.chromium.org/2595613002/diff/20001/docs/memory-infra/memor...
File docs/memory-infra/memory_benchmarks.md (right):

https://codereview.chromium.org/2595613002/diff/20001/docs/memory-infra/memor...
docs/memory-infra/memory_benchmarks.md:9: ## Benchmarks
On 2017/01/06 17:29:33, Primiano Tucci (back but slow) wrote:
> Before this I'd add a very small recap of the form:
> 
> ## Glossary:
> User-story: a set of pages and actions on them (e.g., open google homepage,
type
> "foo", click search, scroll down)
> Metric: a data aggregation process that takes a chrome trace in input
(produced
> by a TBM telemetry run) and produces a set of summary numbers in output (e.g.
> total memory while foreground = X)
> Benchmark: a combination of (one or more) user-stories and (one or more)
> metrics.
>    

Done.

https://codereview.chromium.org/2595613002/diff/20001/docs/memory-infra/memor...
docs/memory-infra/memory_benchmarks.md:21: Other benchmarks maintained by the
memory-infra team are:
On 2017/01/06 17:29:33, Primiano Tucci (back but slow) wrote:
> This one I'd put in an appendix, like:
> "There are other benchmarks currently maintained, see appendix X"
> 
> Statistically most chromium developers are hit by system health. The other
> benchmarks here are just distracting here.

Done.

https://codereview.chromium.org/2595613002/diff/20001/docs/memory-infra/memor...
docs/memory-infra/memory_benchmarks.md:41: ## What is measured
On 2017/01/06 17:29:33, Primiano Tucci (back but slow) wrote:
> Before getting here, I think there should be a section here named something
> like:
> "What are system health benchmarks?"
> In part you describe these below in lines 80-100 but it's a bit too
fragmented.
> This is a recurring question and should be on the top.
> 
> So here I'd say something along those lines:
> - System health is an effort to unify top-level benchmarks (as opposite to
> microbenchmarks and regression tests) to capture representative user stories
> (...). Today they are grouped into the following groups:
> -load
> -browse
> -background
> (essentially move here the lines 80-100)
> 
> Another important point to clarify here is: when do they sample and what do
they
> do w.r.t trigger gc?
> 

Done.

https://codereview.chromium.org/2595613002/diff/20001/docs/memory-infra/memor...
docs/memory-infra/memory_benchmarks.md:43: These benchmarks are run continuously
on the [chromium.perf][] waterfall,
On 2017/01/06 17:29:33, Primiano Tucci (back but slow) wrote:
> Move these 4 lines up when you say "the most important benchmarks are"

Done.

https://codereview.chromium.org/2595613002/diff/20001/docs/memory-infra/memor...
docs/memory-infra/memory_benchmarks.md:52: To view data from one of the
benchmarks you should select:
On 2017/01/06 17:29:33, Primiano Tucci (back but slow) wrote:
> This should be a section called:
> # How to interpret the output emitted by memory metrics?
> maybe you can steal the equation picture from
>
https://docs.google.com/presentation/d/1iy_EVuCn8fuzXDqKTmmZeXeX91c2PKwsARCGP...

This is now a section "Understanding memory metrics" below.

[perezju, 2017-01-09 17:03:40]

Description was changed from

==========
System Health Docs (DRAFT)

BUG=catapult:#3035
==========

to

==========
[memory-infra] Document memory benchmarks

Add documentation for memory benchmarks, system health in particular, on what
they are, where are they tracked, how to run them, how to understand results.

BUG=catapult:#3035
==========

[Primiano Tucci (use gerrit), 2017-01-09 17:04:22]

great LGTM. let's land it and iterate eventually. looks in a good shape now.
thanks

[Primiano Tucci (use gerrit), 2017-01-09 17:04:25]

The CQ bit was checked by primiano@chromium.org

[commit-bot: I haz the power, 2017-01-09 17:05:15]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-01-09 17:17:28]

CQ is committing da patch.

Bot data: {"patchset_id": 60001, "attempt_start_ts": 1483981465687140,
"parent_rev": "093561e5b2476f8466aabc8f063575cd67c43c5d", "commit_rev":
"8f6b9c067322040ebd328b225ea5bc7c8f6ce9f0"}

[commit-bot: I haz the power, 2017-01-09 17:17:57]

Description was changed from

==========
[memory-infra] Document memory benchmarks

Add documentation for memory benchmarks, system health in particular, on what
they are, where are they tracked, how to run them, how to understand results.

BUG=catapult:#3035
==========

to

==========
[memory-infra] Document memory benchmarks

Add documentation for memory benchmarks, system health in particular, on what
they are, where are they tracked, how to run them, how to understand results.

BUG=catapult:#3035

Review-Url: https://codereview.chromium.org/2595613002
Cr-Commit-Position: refs/heads/master@{#442278}
Committed:
https://chromium.googlesource.com/chromium/src/+/8f6b9c067322040ebd328b225ea5...
==========

[commit-bot: I haz the power, 2017-01-09 17:18:00]

Committed patchset #4 (id:60001) as
https://chromium.googlesource.com/chromium/src/+/8f6b9c067322040ebd328b225ea5...