[memory-infra] Document memory benchmarks

docs/memory-infra/memory_benchmarks.md

+# Memory Benchmarks
+
+This document describes benchmarks available to track Chrome's and
+WebView's memory usage, where they live, what they measure, how to run them,
+and on how to diagnose regressions.
+
+[TOC]
+
+## Benchmarks

[Primiano Tucci (use gerrit), 2017/01/06 17:29:33]

Before this I'd add a very small recap of the form:

## Glossary:
User-story: a set of pages and actions on them (e.g., open google homepage, type
"foo", click search, scroll down)
Metric: a data aggregation process that takes a chrome trace in input (produced
by a TBM telemetry run) and produces a set of summary numbers in output (e.g.
total memory while foreground = X)
Benchmark: a combination of (one or more) user-stories and (one or more)
metrics.
   

[perezju, 2017/01/09 17:01:46]

Done.

+
+Benchmarks are implemented in [Telemetry][] and measurements are gathered via
+[memory-infra][] metrics.
+
+The most important benchmarks are:
+
+*   [system_health.memory_mobile][system_health] - collection of
+    user scenarios on mobile Android devices.
+*   [system_health.memory_desktop][system_health] - collection of
+    user scenarios on desktop platforms.
+
+Other benchmarks maintained by the memory-infra team are:

[Primiano Tucci (use gerrit), 2017/01/06 17:29:33]

This one I'd put in an appendix, like:
"There are other benchmarks currently maintained, see appendix X"

Statistically most chromium developers are hit by system health. The other
benchmarks here are just distracting here.

[perezju, 2017/01/09 17:01:46]

Done.

+
+*   [memory.dual_browser_test][memory_py] - cycle between doing Google searches
+    on a webview-based browser (a stand-in for the Google Search app) and
+    loading pages on Chrome. Runs on Android devices only.
+*   [memory.long_running_dual_browser_test][memory_py] - same as above, but the
+    test is run for 60 iterations keeping both browsers alive for the whole
+    duration of the test. Last-resort net to catch memory leaks not apparent on
+    shorter tests.
+*   [memory.top_10_mobile][memory_py] - cycle between loading a page on Chrome,
+    pushing the browser to the background, and then back to the foreground.
+    *To be deprecated in favor of system_health.memory_mobile.*
+*   [memory.top_10_mobile_stress][memory_py] - same as above, but keeps a single
+    instance of the browser open for 5 repetitions. *To be deprecated.*
+
+[Telemetry]: https://github.com/catapult-project/catapult/blob/master/telemetry/README.md
+[memory-infra]: /memory-infra/README.md
+[system_health]: https://chromium.googlesource.com/chromium/src/+/master/tools/perf/page_sets/system_health/
+[memory_py]: https://chromium.googlesource.com/chromium/src/+/master/tools/perf/benchmarks/memory.py
+
+## What is measured

[Primiano Tucci (use gerrit), 2017/01/06 17:29:33]

Before getting here, I think there should be a section here named something
like:
"What are system health benchmarks?"
In part you describe these below in lines 80-100 but it's a bit too fragmented.
This is a recurring question and should be on the top.

So here I'd say something along those lines:
- System health is an effort to unify top-level benchmarks (as opposite to
microbenchmarks and regression tests) to capture representative user stories
(...). Today they are grouped into the following groups:
-load
-browse
-background
(essentially move here the lines 80-100)

Another important point to clarify here is: when do they sample and what do they
do w.r.t trigger gc?

[perezju, 2017/01/09 17:01:46]

Done.

+
+These benchmarks are run continuously on the [chromium.perf][] waterfall,

[Primiano Tucci (use gerrit), 2017/01/06 17:29:33]

Move these 4 lines up when you say "the most important benchmarks are"

[perezju, 2017/01/09 17:01:46]

Done.

+collecting and reporting results on the
+[Chrome Performance Dashboard][chromeperf].
+
+[chromium.perf]: https://build.chromium.org/p/chromium.perf/waterfall
+[chromeperf]: https://chromeperf.appspot.com/report
+
+![Chrome Performance Dashboard](cfac7NMvNCW.png)
+
+To view data from one of the benchmarks you should select:

[Primiano Tucci (use gerrit), 2017/01/06 17:29:33]

This should be a section called:
# How to interpret the output emitted by memory metrics?
maybe you can steal the equation picture from
https://docs.google.com/presentation/d/1iy_EVuCn8fuzXDqKTmmZeXeX91c2PKwsARCGP...

[perezju, 2017/01/09 17:01:46]

This is now a section "Understanding memory metrics" below.

+
+*   **Test suite:** One of the [benchmarks](#Benchmarks) named above.
+*   **Bot:** A particular platform or device configuration. (Sign in to also
+    see internal bots.)
+*   **Subtest (1):** A *metric name* of the form
+    `memory:{browser}:{processes}:{source}:{component}:{kind}_avg`
+    where:
+    *   **browser:** One of `chrome` or `webview`.
+    *   **processess:** One of `browser_process`, `renderer_processess`,
+        `gpu_process`, or `all_processess`.
+    *   **source:** One of `reported_by_chrome` or `reported_by_os`
+    *   **component:** May be a Chrome component, e.g. `skia` or `sqlite`;
+        details about a specific component, e.g. `v8:heap`; or a class of memory
+        as seen by the OS, e.g. `system_memory:native_heap` or `gpu_memory`.
+    *   **kind:** The kind of memory being reported. For metrics reported by
+        Chrome this usually is `effective_size` (others are `locked_size`
+        and `allocated_objects_size`); for metrics by the OS this usually is
+        `proportional_resident_size` (others are `peak_resident_size` and
+        `private_dirty_size`).
+*   **Subtest (2):** The name of a *story group*.
+    *   On system health benchmarks it consists of the first two parts,
+        i.e. `{action}:{category}`, of the story name (see below).
+    *   On dual browser benchmarks it is either `on_chrome` or `on_webview`
+        indicating which of the two browsers was on foreground at the time of
+        measurement.
+    *   On top 10 mobile benchmarks it is either `foreground` or `background`
+        indicating the state of the browser at the time of measurement.
+*   **Subtest (3):** The name of a *story*.
+    *   On system health benchmarks these are of the form
+        `{action}:{category}:{site}`. Categories are things like `news`,
+        `social`, `tools`, etc. Sites are identified by a short name like
+        `cnn`, `facebook`, `gmail`, etc. The action identifies interactions
+        performed with the page:
+        *   `load:*` stories just navigate to a URL, wait for the page to
+            load, and take a memory measurement.
+        *   `browse:*` stories navigate to a URL, interact with the page (e.g.
+            scroll, click on elements, navigate to subpages, navigate back),
+            and take a single memory measurement in the end.
+        *   `background:*` stories navigate to a URL, may interact with the
+            page, push the browser to the background, and then take a memory
+            measurement.
+        *   `long_running:*` stories interact with a page for a longer period
+            of time (~ 5 mins each) before taking a memory measurement.
+        *   `blank:about:blank` takes a single memory measurement on
+            **about:blank**.
+
+## How to run the benchmarks
+
+Benchmarks may be run on a local platform/device or remotely on a try job.
+
+### How to run locally
+
+After building, e.g. `ChromePublic.apk`, you can run a specific system health
+story with the command:
+
+```
+$SRC/tools/perf/run_benchmark run system_health.memory_mobile \
+    --browser android-chromium --story-filter load:search:google
+```
+
+This will run the story with a default of 3 repetitions and produce a
+`results.html` file comparing results from this and any previous benchmark
+runs.
+
+![Example results.html file](zV8XAYb7Opc.png)
+
+Other available options are:
+
+*   `--pageset-repeat [n]` - override the default number of repetitions
+*   `--output-format html --output-format json` - select *both* output formats
+     to get individual [trace files][memory-infra] in addition to the
+     `results.html` file.
+*   `--reset-results` - clear results from any previous benchmark runs in the
+    `results.html` file.
+*   `--results-label [label]` - give meaningful names to your benchmark runs
+    to make it easier to compare them.
+
+To test WebView make sure to [replace the existing webview][webview_install]
+on the device and select `--browser android-webview`.
+
+[webview_install]: https://www.chromium.org/developers/how-tos/build-instructions-android-webview
+
+### How to run a try job
+
+Given a patch on a chromium checkout, try jobs provide a convenient way to
+evaluate its memory implications on devices or platforms which
+may not be immediately available to developers.
+
+To start a try job [upload a CL][contributing] and run the command, e.g.:
+
+```
+$SRC/tools/perf/run_benchmark try android-nexus5 system_health.memory_mobile
+```
+
+This will run all of the system health stories for you, and conveniently
+provide a `results.html` file comparing measurements with/without your patch.
+
+To see the full list of available try bots run the command:
+
+```
+$SRC/tools/perf/run_benchmark try list
+```
+
+[contributing]: https://www.chromium.org/developers/contributing-code