blink/run-webkit-tests: Allow seeding the random test run order and write out value.

third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py

Index: third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py
diff --git a/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py b/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py
index 7449169ac273b47626ebb3637c3d0e39c177e9a0..b5e6b6b38eb2dda18596d2d18e2128a2ca2bd525 100644
--- a/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py
+++ b/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py
@@ -40,6 +40,7 @@ The Manager object has a constructor and one main method called run.
 import json
 import logging
 import random
+import re
 import sys
 import time
 
@@ -89,6 +90,7 @@ class Manager(object):
         self._wptserve_started = False
         self._websockets_server_started = False
 
+        self._random_seed = None
         self._results_directory = self._port.results_directory()
         self._finder = LayoutTestFinder(self._port, self._options)
         self._runner = LayoutTestRunner(self._options, self._port, self._printer, self._results_directory, self._test_is_slow)
@@ -119,6 +121,16 @@ class Manager(object):
         if exit_code:
             return test_run_results.RunDetails(exit_code=exit_code)
 
+        # Write out the random seed and the order we are running the tests in.
+        if not self._options.dry_run:
+            if self._random_seed:
+                self._filesystem.write_text_file(
+                    self._filesystem.join(self._results_directory, "random-seed.txt"),
+                    "%s\n" % self._random_seed)
+            self._filesystem.write_text_file(
+                self._filesystem.join(self._results_directory, "tests-run-order.txt"),
+                "\n".join(tests_to_run))

[Dirk Pranke, 2016/06/21 23:47:13]

How does this relate to your "write a json file out" ?

See also the comment I had in that CL that I'd just log the seed instead ...

[mithro, 2016/06/22 08:40:08]

Logging the random seed into the json environment file would be an alternative
to logging to this file. I didn't want to block this patch on having the json
environment recording figured out and a simple txt file seemed the simplest
option.
I do log the random seed used on line 277. 

However, unless I'm missing something, the log output doesn't end up in the
results output? As we are generating a lot of layout test results with different
random seeds, it makes things a lot easier to have this information recorded in
the output result somewhere. Happy to put it somewhere else if you have a better
/ more appropriate idea.

The tests-run-order.txt was/is really useful when looking into these problems
but it is less required. I can remove this bit (or put it behind a flag?) it if
you prefer.

+
         # Don't retry failures if an explicit list of tests was passed in.
         if self._options.retry_failures is None:
             should_retry_failures = len(paths) < len(test_names)
@@ -252,11 +264,23 @@ class Manager(object):
         # if used, contains alphabetically consecutive tests.
         if self._options.order == 'natural':
             tests_to_run.sort(key=self._port.test_key)
-        elif self._options.order == 'random':
-            random.shuffle(tests_to_run)
-        elif self._options.order == 'random-seeded':
+        elif self._options.order.startswith('random'):
+            rnd = random.Random()

[qyearsley, 2016/06/22 17:30:16]

This line could be removed (it appears below).

+
+            groups = re.match('random-seeded(=([0-9]+))?', self._options.order)

[qyearsley, 2016/06/22 17:30:16]

Instead of specifying seeds with an option like `--order=random-seeded=123`,
what do you think about adding a separate option "--seed" which takes an
integer? Then we also wouldn't need to have both --order=random and
--order=random-seeded; whether it's using an input seed or generating a new seed
could depend on whether --seed is passed in. For example:

    ...
    elif self._options.order == 'random':
        if self._options.seed is not None:
            seed = self._options.seed
        else:
            seed = int(time.time())
        rnd = random.Random()
        test_to_run.sort() 
        rnd.shuffle(tests_to_run)
        

[Dirk Pranke, 2016/06/24 01:53:40]

+1 to splitting it into a separate argument.

+            if not groups:
+                seed = int(time.time())
+            elif not groups.group(2):
+                seed = 4  # http://xkcd.com/221/

[qyearsley, 2016/06/22 17:30:16]

It's probably not necessary to necessary to keep this option here anymore; I
don't think anything depends on using --order=random-seeded.

+            else:
+                seed = int(groups.group(2))
+            _log.info("Test random order seed: %i", seed)
+            self._random_seed = seed
+
+            # Make sure the tests are in sorted before we randomize them.
+            tests_to_run.sort()
             rnd = random.Random()
-            rnd.seed(4)  # http://xkcd.com/221/
+            rnd.seed(seed)
             rnd.shuffle(tests_to_run)
 
         tests_to_run, tests_in_other_chunks = self._finder.split_into_chunks(tests_to_run)