Entry point for bisect sample comparison.

tracing/tracing/metrics/compare_samples.py

Index: tracing/tracing/metrics/compare_samples.py
diff --git a/tracing/tracing/metrics/compare_samples.py b/tracing/tracing/metrics/compare_samples.py
new file mode 100644
index 0000000000000000000000000000000000000000..57951022424eff0b3e6e14586a06456619a3a0e5
--- /dev/null
+++ b/tracing/tracing/metrics/compare_samples.py
@@ -0,0 +1,47 @@
+# Copyright 2016 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+import os
+
+import tracing_project
+import vinn
+
+FORMAT_TO_METHOD = {
+    'chartjson': 'compareCharts',
+    'valueset': 'compareValuesets',
+    'buildbot': 'compareBuildbotOutputs'
+}
+
+_COMPARE_SAMPLES_CMD_LINE = os.path.join(
+    os.path.dirname(__file__), 'compare_samples_cmdline.html')
+
+
+def CompareSamples(sample_a, sample_b, metric, data_format='chartjson'):
+  """Compare the values of a metric from two samples from benchmark output.
+
+  Args:
+    sample_a, sample_b (str): comma-separated lists of paths to the benchmark
+        output.
+    metric (str): Metric name in slash-separated format [2 or 3 part].
+    data_format (str): The format the samples are in. Supported values are:
+        'chartjson', 'valueset', 'buildbot'.
+  Returns:
+    JSON encoded dict with the values parsed form the samples and the result of
+    the hypothesis testing comparison of the samples under the 'result' key.
+    Possible values for the result key are:
+      'NEED_MORE_DATA', 'REJECT_THE_NULL' and 'FAIL_TO_REJECT_THE_NULL'.

[eakuefner, 2016/09/29 17:01:11]

seems like we don't really need "THE_NULL"; maybe just "REJECT" and
"FAIL_TO_REJECT", or at the very least trim "THE" if you don't like that.

[RobertoCN, 2016/10/01 01:04:22]

Done.

+    Where the null hypothesis is that the samples belong to the same population.
+    i.e. a 'REJECT_THE_NULL' result would make it reasonable to conclude that
+    there is a significant difference between the samples. (e.g. a perf
+    regression).
+  """
+
+  method = FORMAT_TO_METHOD[data_format]
+  project = tracing_project.TracingProject()
+  all_source_paths = list(project.source_paths)
+
+  return vinn.RunFile(
+      _COMPARE_SAMPLES_CMD_LINE,
+      source_paths=all_source_paths,
+      js_args=[method, sample_a, sample_b, metric])