Telemetry Test for WebRTC Rendering.

tools/telemetry/telemetry/web_perf/metrics/webrtc_rendering_timeline.py

Index: tools/telemetry/telemetry/web_perf/metrics/webrtc_rendering_timeline.py
diff --git a/tools/telemetry/telemetry/web_perf/metrics/webrtc_rendering_timeline.py b/tools/telemetry/telemetry/web_perf/metrics/webrtc_rendering_timeline.py
new file mode 100644
index 0000000000000000000000000000000000000000..c9623572953ab76a24d46ccd0b5aef67582c8cf4
--- /dev/null
+++ b/tools/telemetry/telemetry/web_perf/metrics/webrtc_rendering_timeline.py
@@ -0,0 +1,162 @@
+# Copyright 2015 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+import logging
+
+from telemetry.value import list_of_scalar_values
+from telemetry.value import list_of_string_values
+from telemetry.value import scalar
+from telemetry.web_perf.metrics import timeline_based_metric
+from telemetry.web_perf.metrics import webrtc_rendering_stats as stats_helper
+
+WEB_MEDIA_PLAYER_MS_EVENT = 'WebMediaPlayerMS::UpdateCurrentFrame'
+

[eakuefner, 2015/09/28 19:49:48]

nit: 2 newlines instead of 3

[cpaulin (no longer in chrome), 2015/09/29 15:50:25]

Done.

+
+
+class WebRtcRenderingTimelineMetric(timeline_based_metric.TimelineBasedMetric):
+  """WebrtcRenderingTimelineMetric calculates metric for WebMediaPlayerMS.
+
+  The following metrics are added to the results:
+    WebRtcRendering_drift_time usec

[eakuefner, 2015/09/28 19:49:48]

The units of the values you add should match entries in unit-info.json to be
compatible with the perf dashboard:
https://code.google.com/p/chromium/codesearch#chromium/src/tools/telemetry/te...
The file is kind of a mess but we want to avoid adding new units to it if at all
possible.

For 'usec' you probably want 'us' which is already defined, and for 'FPS' you
probably want 'fps'. '%' will work as long as the improvement direction for
those values is down (smaller is better).

Also, is it intentional that the capitalization is different for the name of
this metric in particular compared to the rest (WebRtc vs WebRTC)?

[cpaulin (no longer in chrome), 2015/09/29 15:50:25]

Ethan, 
I fixed the WebRtcRendering_ occurences, also changed 'usec' to 'us'.
For the '%' case, sometimes low percentage is good like for
WebRTCRendering_percent_out_of_sync, but for WebRTCRendering_smoothness_score
higher is better, how do you deal with this? Thanks

[eakuefner, 2015/09/29 18:09:27]

Great question. For a little background, we're currently midway through the
process of decoupling improvement_direction from unit-info.json, and moving
towards specifying all improvement directions in value callsites.

The improvement_direction field already exists in values, so you can use this
functionality by writing:

from telemetry.value import improvement_direction

to bring in the improvement direction module, and then setting
improvement_direction=improvement_direction.UP or improvement_direction.DOWN at
the sites where you're constructing values. For now, if you don't explicitly set
an improvement direction, we'll still use the values from unit-info.json but it
would be great if you could specify an improvement_direction for all your
values.

I need to do some minor plumbing on the perf dashboard to make this work, and
I've opened https://github.com/catapult-project/catapult/issues/1541 for that. I
should be able to get this done today, and I'll CC you on the CL.

[cpaulin (no longer in chrome), 2015/09/29 18:59:05]

Done.

+    WebRTCRendering_std_dev_drift_time usec
+    WebRTCRendering_percent_badly_out_of_sync %
+    WebRTCRendering_percent_out_of_sync %
+    WebRTCRendering_fps FPS
+    WebRTCRendering_smoothness_score %
+    WebRTCRendering_freezing_score %
+    WebRTCRendering_rendering_length_error %
+  """
+
+  def __init__(self):
+    super(WebRtcRenderingTimelineMetric, self).__init__()
+
+  @staticmethod
+  def IsMediaPlayerMSEvent(event):
+    """Verify that the event is a webmediaplayerMS event."""
+    return event.name == WEB_MEDIA_PLAYER_MS_EVENT
+
+  @staticmethod
+  def IsEventInInteraction(event, interaction):

[eakuefner, 2015/09/28 19:49:48]

Ned has recently added IsEventInInteractions here:
https://codereview.chromium.org/1357423008

Please prefer that over this.

[cpaulin (no longer in chrome), 2015/09/29 15:50:25]

Done.

+    """Verify that the event belong to the given interaction."""
+    return interaction[0].start <= event.start <= interaction[0].end
+
+  def AddResults(self, model, renderer_thread, interactions, results):
+    """Adding metrics to the results."""
+    assert interactions
+    found_events = []
+    for event in renderer_thread.parent.IterAllEvents(
+        event_predicate=self.IsMediaPlayerMSEvent):
+      if self.IsEventInInteraction(event, interactions):
+        found_events.append(event)
+    stats_parser = stats_helper.WebMediaPlayerMsRenderingStats(found_events)
+    rendering_stats = stats_parser.GetTimeStats()
+    none_reason = None
+    if not rendering_stats:
+      rendering_stats = dict.fromkeys([
+          'drift_time'
+          'mean_drift_time',
+          'std_dev_drift_time',
+          'percent_badly_out_of_sync',
+          'percent_out_of_sync',
+          'smoothness_score',
+          'freezing_score',
+          'rendering_length_error',
+          'fps',
+          'frame_distribution'])
+      none_reason = "No WebMediaPlayerMS::UpdateCurrentFrame event found"

[eakuefner, 2015/09/28 19:49:48]

nit: single quotes

[cpaulin (no longer in chrome), 2015/09/29 15:50:25]

Done.

+    logging.info('rendering stats : %s', rendering_stats)

[nednguyen, 2015/09/28 20:56:43]

Remove this logging

[cpaulin (no longer in chrome), 2015/09/29 15:50:25]

Done.

+    results.AddValue(list_of_scalar_values.ListOfScalarValues(
+        results.current_page,
+        'WebRtcRendering_drift_time',
+        'usec',
+        rendering_stats['drift_time'],
+        important=True,
+        description='Drift time for a rendered frame',
+        none_value_reason=none_reason))
+
+    results.AddValue(scalar.ScalarValue(
+        results.current_page,
+        'WebRTCRendering_mean_drift_time',
+        'usec',
+        rendering_stats['mean_drift_time'],
+        important=True,
+        description='Mean drift time for frames',
+        none_value_reason=none_reason))
+
+    results.AddValue(scalar.ScalarValue(
+        results.current_page,
+        'WebRTCRendering_std_dev_drift_time',
+        'usec',
+        rendering_stats['std_dev_drift_time'],
+        important=True,
+        description='Standard deviation of drift time for frames',
+        none_value_reason=none_reason))
+
+    results.AddValue(scalar.ScalarValue(
+        results.current_page,
+        'WebRTCRendering_percent_badly_out_of_sync',
+        '%',
+        rendering_stats['percent_badly_out_of_sync'],
+        important=True,
+        description='Percentage of frame which drifted more than 2 VSYNC',
+        none_value_reason=none_reason))
+
+    results.AddValue(scalar.ScalarValue(
+        results.current_page,
+        'WebRTCRendering_percent_out_of_sync',
+        '%',
+        rendering_stats['percent_out_of_sync'],
+        important=True,
+        description='Percentage of frame which drifted more than 1 VSYNC',
+        none_value_reason=none_reason))
+
+    # Make the output distribution a list since
+    # no facilities for dict values exist (yet).
+    frame_distribution_list = []
+    for key, value in rendering_stats['frame_distribution'].iteritems():
+      temp = '%s:%s' % (key, value)
+      frame_distribution_list.append(temp)
+    results.AddValue(list_of_string_values.ListOfStringValues(
+        results.current_page,
+        'WebRtcRendering_frame_distribution',
+        'frames:occurences',
+        frame_distribution_list,
+        important=True,
+        description='Output distribution of frames',
+        none_value_reason=none_reason))
+
+    results.AddValue(scalar.ScalarValue(
+        results.current_page,
+        'WebRTCRendering_fps',
+        'FPS',
+        rendering_stats['fps'],
+        important=True,
+        description='Calculated Frame Rate of video rendering',
+        none_value_reason=none_reason))
+
+    results.AddValue(scalar.ScalarValue(
+        results.current_page,
+        'WebRTCRendering_smoothness_score',
+        '%',
+        rendering_stats['smoothness_score'],
+        important=True,
+        description='Smoothness score of rendering',
+        none_value_reason=none_reason))
+
+    results.AddValue(scalar.ScalarValue(
+        results.current_page,
+        'WebRTCRendering_freezing_score',
+        '%',
+        rendering_stats['freezing_score'],
+        important=True,
+        description='Freezing score of rendering',
+        none_value_reason=none_reason))
+
+    results.AddValue(scalar.ScalarValue(
+        results.current_page,
+        'WebRTCRendering_rendering_length_error',
+        '%',
+        rendering_stats['rendering_length_error'],
+        important=True,
+        description='Rendering length error rate',
+        none_value_reason=none_reason))