[Telemetry] Add Benchmark.ShouldTearDownStateAfterEachStorySetRun

telemetry/telemetry/benchmark.py

Index: telemetry/telemetry/benchmark.py
diff --git a/telemetry/telemetry/benchmark.py b/telemetry/telemetry/benchmark.py
index c67bad5c4a1953b0bef6beb48e9f7e13dc34454c..14770994203623c1613e36100112dc6c76bcf03a 100644
--- a/telemetry/telemetry/benchmark.py
+++ b/telemetry/telemetry/benchmark.py
@@ -100,7 +100,7 @@ class Benchmark(command_line.Command):
 
   @classmethod
   def ShouldTearDownStateAfterEachStoryRun(cls):
-    """Override this method to tear down state after each story run.
+    """Override to specify whether to tear down state after each story run.
 
     Tearing down all states after each story run, e.g., clearing profiles,
     stopping the browser, stopping local server, etc. So the browser will not be
@@ -113,6 +113,22 @@ class Benchmark(command_line.Command):
     return False
 
   @classmethod
+  def ShouldTearDownStateAfterEachStorySetRun(cls):
+    """Override to specify whether to tear down state after each story set run.
+
+    Defaults to True in order to reset the state and make individual story set
+    repeats more independent of each other. The intended effect is to average
+    out noise in measurements between repeats.
+
+    Long running benchmarks willing to stess test the browser and have it run
+    for long periods of time may switch this value to False.
+
+    This should only be used by TimelineBasedMeasurement (TBM) benchmarks, but
+    not by PageTest based benchmarks.
+    """
+    return True

[nednguyen, 2016/06/24 14:44:50]

This will cause a behavior change to all existing benchmarks that's using
pageset-repeat. I think we should make this False for now, then send an email to
telemetry-announce@ telling people about this, then switch this to True.

[perezju, 2016/06/24 15:12:36]

ok, sgtm

+
+  @classmethod
   def AddCommandLineArgs(cls, parser):
     group = optparse.OptionGroup(parser, '%s test options' % cls.Name())
     cls.AddBenchmarkCommandLineArgs(group)