[telemetry] Implement first version of timeline based measurement

[telemetry] Implement first version of timeline based measurement

TimelineBasedMeasurement lets us compute a variety of different metrics for pages that do complex sequences of interactions. Whereas traditional telemetry measurements focus on reporting a single type of value for many pages, this lets a page decide what types of metrics it emits based on the behaviors its page actually performs.

The contract is that the page emits
console.time/timeEnd calls of a certain format describing when it is doing
something worth noting, and what kind fo interaction it is doing. For instance,
if a drawer animation is runnign, it would emit:
  MeasurementRequest.Drawer/smoothness

This tells the timelineBasedMeasurement that the timeline data generated during this time
should be analyzed for smoothness information, with results benig reported as things like
Drawer.frame_time.

Depends on https://codereview.chromium.org/177093013

BUG=345922
NOTRY=True

Committed: https://src.chromium.org/viewvc/chrome?view=rev&revision=255434

[nduca, 2014-02-22 03:08:04]

ptal, we shoudl be able to combine this with your smoothness cleanup to enable a
full benchmark i think

[tonyg, 2014-02-22 14:55:19]

https://codereview.chromium.org/165673008/diff/1/tools/perf/measurements/meta...
File tools/perf/measurements/meta_measurement.py (right):

https://codereview.chromium.org/165673008/diff/1/tools/perf/measurements/meta...
tools/perf/measurements/meta_measurement.py:1: # Copyright 2014 The Chromium
Authors. All rights reserved.
The term "meta" gave me the wrong initial expectation about what the goal of
this measurement is.

What about something like app_driven_measurement.py?

https://codereview.chromium.org/165673008/diff/1/tools/perf/measurements/meta...
tools/perf/measurements/meta_measurement.py:85:
tab.browser.StartTracing(timeline_module.MINIMAL_TRACE_CATEGORIES)
Thinking aloud here, but I'm concerned about the trend towards keeping tracing
on for all of our benchmarking (something we don't do for many benchmarks
including page cyclers and something we know has non-negligible overhead).

Conceptually we want two types of tests: regression tests that measure full
scenarios with no instrumentation and then profiling tests that report more
details. The first category is the ground truth for detecting regressions and
improvements, but is not very actionable data. The second category provides the
actionable data that can be trusted in the context of the first.

What this means practically, is that we should preserve a mode where
responsiveness, smoothness and efficiency numbers come from the [OS,
framegrabbers, tools external to the app under test, etc] without having to
question whether trace event or other instrumentation overheads are skewing
things.

Both types of benchmarks are crucial. We just shouldn't lull ourselves into
thinking either one of them works for both use-cases.

https://codereview.chromium.org/165673008/diff/1/tools/telemetry/unittest_dat...
File tools/telemetry/unittest_data/measurement_request_enabled_page.html
(right):

https://codereview.chromium.org/165673008/diff/1/tools/telemetry/unittest_dat...
tools/telemetry/unittest_data/measurement_request_enabled_page.html:35:
console.time('MetricRequest.smoothness/CenterAnimation');
Is it possible to use the standard User Timing marks and measures instead of
console?

[nednguyen, 2014-02-22 15:11:02]

https://codereview.chromium.org/165673008/diff/1/tools/perf/measurements/meta...
File tools/perf/measurements/meta_measurement.py (right):

https://codereview.chromium.org/165673008/diff/1/tools/perf/measurements/meta...
tools/perf/measurements/meta_measurement.py:30: self.end = event.end
nit: Maybe use self.bounds instead?

[nduca, 2014-02-22 20:38:18]

Hi Tony - one thing I messed up here is not giving you nearly enough context on
this patch sequence. Too much going on, huh! I think a chat might be worthwhile
if you're up for it. Ned and I have gotten this [maybe hairbrained] idea in our
heads that we can factor much of telemetry into a few more primitive parts:
- running actions on a page, such that either the page or the actions mark the
common timeline
- running various performance probes which all insert their data into the common
timeline
- analyzing that common timeline at the various marked areas to produce numbers
for the final benchmark

The key thing is that many of the things you call zero-overhead could be framed
as create timelines, too. Page cycler could, for instance, be a thing that turns
on nothing but produces a timeline with a single event: the load event. Or, page
cycler could produce a timeline plus the window.performance.navigationTiming
records, report the laod timing but keep the rest of the timleine around for
diagnostic purposes.

Speed index, too: when the video camera or screen grabber turns on it could
produce a timeline with a process in it for "Camera" that has individual events
for each frame, with the actual frame content being the argument of the event.

Someday, we might be able to get the webperf wg to agree to frame events in a
performance timeline.

So the more foundational idea here is "what if all our perf data produced a
timeline"? With that, we can clean up how we factor the analysis problem.

The issue of how to represent the overhead of the different ways to create a
timeline is always there: tracing can be low overhead with the right categories.
We've exhaustively verified tracing for use in computing smoothness [miletus],
latency [yufeng] and silk numbers [epenner]. There  you have to be careful about
the right categories so its important that people use it with care.

With some of that rant in mind, the thing that might be interesting to chat
about is how to represent all our current performance data sources in telemetry
behind an api for recording that would meet your good criteria of
easy-to-reason-about overhead. E.g., what if we had:
   browser.StartTracing(ZERO_OVERHEAD | 5_PERCENT_OVERHEAD  |
30_PERCENT_OVERHEAD | EVERYTHING)
Where we decide that zero overhead includes monsoon, no tracing,
window.performance.timeline & load time data, but not frame grabber. 5 percent
might have minimal tracing. etc.

Or, we could go listing out categories and leaving it to folks to choose:
  browser.StartTracing(thingsToEnable)


</braindump>

> What about something like app_driven_measurement.py?
I think meta is wrong, too, but I'm conflicted about changing it with various
other dependent patches in flight. Lemme mull this over a bit.

TimelineBasedMeasurement maybe. The part of the app marking the timeline vs the
actions I hope to make go away in the future.... python-based pagesets will
help.

[nduca, 2014-02-25 05:29:17]

Hey Tony, took a pass on this to have clearer intrusiveness controls, better
name. Various other polish, but thems the big changes. Lmk if you wanna chat
tomorrow about this. I gave you a wall o' text to absorb. :)

[nednguyen, 2014-02-26 00:59:55]

On 2014/02/25 05:29:17, nduca wrote:
> Hey Tony, took a pass on this to have clearer intrusiveness controls, better
> name. Various other polish, but thems the big changes. Lmk if you wanna chat
> tomorrow about this. I gave you a wall o' text to absorb. :)

Another issue that comes up today when I talk with Justin is having some sort of
a "natural span" that is not defined on site. An example use case is collecting
a variety of metrics on initial homepage load once Chrome receives the first
byte. It seems unlikely that we can mark console.time(..) anywhere.

[nednguyen, 2014-02-26 01:00:10]

https://codereview.chromium.org/165673008/diff/110001/tools/perf/measurements...
File tools/perf/measurements/timeline_based_measurement.py (right):

https://codereview.chromium.org/165673008/diff/110001/tools/perf/measurements...
tools/perf/measurements/timeline_based_measurement.py:42: raise
Exception('Unrecognized metric type: %s' % request.metric_type)
I think eventually we will want this to be configurable instead of hard coded. I
can imagine the scenario that GWS, bigtop, and chromium are interested in
different config of metric_type->metrics.
Can you add a TODO so this can read the mapping from metric_type to metric lists
from an external file?

[tonyg, 2014-02-26 17:20:03]

Love it :)

And thanks for explaining yesterday the design you have in mind.

Just a few more really minor nits/questions and then I think this is ready to
go.

https://codereview.chromium.org/165673008/diff/1/tools/telemetry/unittest_dat...
File tools/telemetry/unittest_data/measurement_request_enabled_page.html
(right):

https://codereview.chromium.org/165673008/diff/1/tools/telemetry/unittest_dat...
tools/telemetry/unittest_data/measurement_request_enabled_page.html:35:
console.time('MetricRequest.smoothness/CenterAnimation');
On 2014/02/22 14:55:19, tonyg wrote:
> Is it possible to use the standard User Timing marks and measures instead of
> console?

ping on this one?

https://codereview.chromium.org/165673008/diff/110001/tools/perf/measurements...
File tools/perf/measurements/timeline_based_measurement.py (right):

https://codereview.chromium.org/165673008/diff/110001/tools/perf/measurements...
tools/perf/measurements/timeline_based_measurement.py:5: from telemetry.page
import page_measurement
uber nit: alphabetize

https://codereview.chromium.org/165673008/diff/110001/tools/perf/measurements...
tools/perf/measurements/timeline_based_measurement.py:21:
ALL_INSTRUMENTATION_LEVELS = [
These levels are an awesome idea :)

I'm wondering if this would be even better. WDYT?

NO_OVERHEAD_LEVEL => Things like measuring page load time, memory consumption,
power consumption, framegrabber video capture, etc where you just checkpoint
before and after the benchmark without running anything extra during the
benchmark.

MINIMAL_OVERHEAD_LEVEL => Things like tracing, devtools timeline and android
video capture which have minimal overhead, but we will shy away from listing an
exact percent.

DEBUG_OVERHEAD_LEVEL => Things like recording skpictures, running perf profiler,
etc which definitely have non-negligible overhead.

https://codereview.chromium.org/165673008/diff/110001/tools/perf/metrics/time...
File tools/perf/metrics/timeline_based_metric_request.py (right):

https://codereview.chromium.org/165673008/diff/110001/tools/perf/metrics/time...
tools/perf/metrics/timeline_based_metric_request.py:15: return (m.group(1),
m.group(2))
style nit: 2 lines between top level definitions

[nduca, 2014-03-04 04:18:57]

> > Is it possible to use the standard User Timing marks and measures instead of
> > console?
Yeah we'll have to do that as a followup but I think thats the right direction.
I think the motivation to switch will be that console.time/timeEnd is actually
heinously intrusive, so I'm pretty sure we'll get it moved over rather than
saying we do and then... not. :)

Rest of updates forthcoming.

[nduca, 2014-03-04 05:33:51]

All set for landing i think. ptal, folk.

[nednguyen, 2014-03-04 15:38:46]

lgtm

LGTM with some nit

https://codereview.chromium.org/165673008/diff/130001/tools/perf/measurements...
File tools/perf/measurements/timeline_based_measurement.py (right):

https://codereview.chromium.org/165673008/diff/130001/tools/perf/measurements...
tools/perf/measurements/timeline_based_measurement.py:64: based on those flags.
Add examples of the interaction marker that the page need to make here or make a
reference to timeline_interaction_record_unittest.py?

https://codereview.chromium.org/165673008/diff/130001/tools/perf/metrics/time...
File tools/perf/metrics/timeline_based_metric.py (right):

https://codereview.chromium.org/165673008/diff/130001/tools/perf/metrics/time...
tools/perf/metrics/timeline_based_metric.py:15: this method should only compute
metric on the data between time_bound.
Compute metric on the data during interaction_record's start & end time?

https://codereview.chromium.org/165673008/diff/130001/tools/perf/metrics/time...
tools/perf/metrics/timeline_based_metric.py:19: results is an instance of
page.PageTestResults.
Our plan is using a results wrapper here to make sure that one always call
TimelineInteractionRecord.GetResultNameFor(..), isn't it? 

We can do this in another patch.

[tonyg, 2014-03-05 03:37:57]

lgtm

[nduca, 2014-03-06 02:35:03]

The CQ bit was checked by nduca@chromium.org

[commit-bot: I haz the power, 2014-03-06 03:22:03]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/nduca@chromium.org/165673008/150001

[commit-bot: I haz the power, 2014-03-06 04:14:29]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2014-03-06 04:14:30]

Retried try job too often on win_rel for step(s) telemetry_perf_unittests
http://build.chromium.org/p/tryserver.chromium/buildstatus?builder=win_rel&nu...

[nednguyen, 2014-03-06 16:29:27]

The CQ bit was checked by nednguyen@google.com

[commit-bot: I haz the power, 2014-03-06 16:29:39]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/nduca@chromium.org/165673008/150001

[commit-bot: I haz the power, 2014-03-06 16:32:11]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2014-03-06 16:32:13]

Try jobs failed on following builders: mac_chromium_compile_dbg

[nduca, 2014-03-06 18:41:17]

The CQ bit was checked by nduca@chromium.org

[nduca, 2014-03-06 18:43:41]

Notry=True because I believe failures to be flake after study.

[commit-bot: I haz the power, 2014-03-06 18:44:13]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/nduca@chromium.org/165673008/150001

[nduca, 2014-03-06 19:08:18]

The CQ bit was unchecked by nduca@chromium.org

[nduca, 2014-03-06 19:08:25]

The CQ bit was checked by nduca@chromium.org

[commit-bot: I haz the power, 2014-03-06 20:56:59]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/nduca@chromium.org/165673008/150001

[commit-bot: I haz the power, 2014-03-06 21:28:38]

Change committed as 255434