Allow seeding the random layout test order and write out seed.

third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py

Index: third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py
diff --git a/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py b/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py
index 67ca8bce188b6127c091266b79da0dc5c264aa6c..1aa5620fb52ae4bdaaa4e27820c95828bb272741 100644
--- a/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py
+++ b/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py
@@ -89,6 +89,7 @@ class Manager(object):
         self._wptserve_started = False
         self._websockets_server_started = False
 
+        self._random_seed = None
         self._results_directory = self._port.results_directory()
         self._finder = LayoutTestFinder(self._port, self._options)
         self._runner = LayoutTestRunner(self._options, self._port, self._printer, self._results_directory, self._test_is_slow)
@@ -192,6 +193,8 @@ class Manager(object):
 
         if not self._options.dry_run:
             self._write_json_files(summarized_full_results, summarized_failing_results, initial_results, running_all_tests)
+            if self._random_seed:
+                self._write_random_seed()
 
             if self._options.write_full_results_to:
                 self._filesystem.copyfile(self._filesystem.join(self._results_directory, "full_results.json"),
@@ -253,11 +256,15 @@ class Manager(object):
         if self._options.order == 'natural':
             tests_to_run.sort(key=self._port.test_key)
         elif self._options.order == 'random':
-            random.shuffle(tests_to_run)
-        elif self._options.order == 'random-seeded':
-            rnd = random.Random()
-            rnd.seed(4)  # http://xkcd.com/221/
-            rnd.shuffle(tests_to_run)
+            if self._options.seed is not None:
+                self._random_seed = self._options.seed
+            else:
+                self._random_seed = int(time.time())

[ojan, 2016/09/06 16:53:29]

I think we should do a fixed value. Using current time is almost always a
mistake because it's then hard to reproduce flakiness you encounter.

[qyearsley, 2016/09/06 17:22:48]

If the seed is output with the results, it should then be possible to reproduce
flakiness that is encountered by passing --order=random --seed=<seed from prior
run>.

One possible advantage of a random seed based on current time by default is that
more time-dependent flakiness would be revealed over the course of multiple test
runs. Although, the order should also change over time as the number of tests
changes, so an order-dependent flaky test shouldn't be able to "hide" for too
long.

Given that, does it still seem like a fixed seed would be a good idea?

[jeffcarp, 2016/09/06 17:22:51]

Is that still true if we log the seed in results.json?

[ojan, 2016/09/06 21:27:11]

Think of this from the perspective of a sheriff trying to figure out why a
failure happened. It will be hard to reason about current time. Admittedly, a
fixed seed is hard to reason about as well, but at least that only changes when
the list of test to run changes.

[jeffcarp, 2016/09/06 22:57:30]

If this would make it hard to tell whether a failing test is flaky or
order-dependent, should we investigate other methods to make sure tests are
order-independent? Or what if we started with a small set of tests that are
known to be order-independent, run those in random order, and slowly move more
tests into that bucket?

[ojan, 2016/09/06 23:07:43]

I think this is a good idea. You probably want it to be a blacklist so that when
you're done moving things you can get rid of the concept of order-dependent
tests. Concretely, we could have an OrderDependentTests file that lets you list
directories or individual tests and the order they should run in. Then you can
remove one directory at a time from that list.

Theoretically, we could even run the order-independent tests on swarming while
we're working on emptying the OrderDependentTest file. I don't know how
complicated that would be to do. If it's easy, I think it'd be a good idea. If
complicated, it might not be worth it.

+            _log.info("Test random order seed: %d", self._random_seed)
+            tests_to_run.sort()
+            rand = random.Random()
+            rand.seed(self._random_seed)
+            rand.shuffle(tests_to_run)
 
         tests_to_run, tests_in_other_chunks = self._finder.split_into_chunks(tests_to_run)
         self._expectations.add_extra_skipped_tests(tests_in_other_chunks)
@@ -265,6 +272,14 @@ class Manager(object):
 
         return tests_to_run, tests_to_skip
 
+    def _write_random_seed(self):
+        """Writes the random seed used for randomized test order."""
+        assert self._options.order == 'random' and self._random_seed
+        path = self._filesystem.join(self._results_directory, 'random-seed.txt')
+        contents = '%d\n' % self._random_seed
+        _log.info('%s %s', path, contents)
+        self._filesystem.write_text_file(path, contents)

[qyearsley, 2016/09/05 00:15:28]

We could also potentially:
 - Not write the seed to the results directory; to reproduce a test order would
then involve getting the seed that's logged when the test was run.
 - Also write the test order. This might sometimes be helpful, but may be
unnecessary.

[Dirk Pranke, 2016/09/06 01:19:57]

I would include the seed as a field in the results.json file, along with all of
the other records of the test run. I don't think we should write a separate file
just for this.

I would probably also log the seed that was used at the beginning of the run, as
part of the printer.print_config() function called at run_webkit_tests.py:544,
rather than logging it here.

[mithro, 2016/09/06 02:02:25]

I'm happy with Dirk's suggestions. 

The aim is to be able to find the random seed uses for a given run so it can be
reproduced. This includes both when run on the bots and when run locally.

[qyearsley, 2016/09/06 17:22:48]

Aye, SGTM

+
     def _test_input_for_file(self, test_file):
         return TestInput(test_file,
                          self._options.slow_time_out_ms if self._test_is_slow(test_file) else self._options.time_out_ms,