Index: docs/testing/identifying_tests_that_depend_on_order.md |
diff --git a/docs/testing/identifying_tests_that_depend_on_order.md b/docs/testing/identifying_tests_that_depend_on_order.md |
new file mode 100644 |
index 0000000000000000000000000000000000000000..e62ef1fca22b0cc678625e64eb8c556c9c80550d |
--- /dev/null |
+++ b/docs/testing/identifying_tests_that_depend_on_order.md |
@@ -0,0 +1,80 @@ |
+ |
+# Fixing layout test flakiness |
+ |
+We'd like to stamp out all the tests that have ordering dependencies. This helps |
+make the tests more reliable and, eventually, will make it so we can run tests |
+in a random order and avoid new ordering dependencies being introduced. To get |
+there, we need to weed out and fix all the existing ordering dependencies. |
+ |
+## Diagnosing test ordering flakiness |
+ |
+These are steps for diagnosing ordering flakiness once you have a test that you |
+believe depends on an earlier test running. |
+ |
+### Bisect test ordering |
+ |
+1. Run the tests such that the test in question fails. |
+2. Run `./Tools/Scripts/print-test-ordering` and save the output to a file. This |
+ outputs the tests run in the order they were run on each content_shell |
+ instance. |
+3. Create a file that contains only the tests run on that worker in the same |
+ order as in your saved output file. The last line in the file should be the |
+ failing test. |
+4. Run |
+ `./Tools/Scripts/bisect-test-ordering --test-list=path/to/file/from/step/3` |
+ |
+The bisect-test-ordering script should spit out a list of tests at the end that |
+causes the test to fail. |
+ |
+*** promo |
+At the moment bisect-test-ordering only allows you to find tests that fail due |
+to a previous test running. It's a small change to the script to make it work |
+for tests that pass due to a previous test running (i.e. to figure out which |
+test it depends on running before it). Contact ojan@chromium if you're |
+interested in adding that feature to the script. |
+*** |
+ |
+### Manual bisect |
+ |
+Instead of running `bisect-test-ordering`, you can manually do the work of step |
+4 above. |
+ |
+1. `run-webkit-tests --child-processes=1 --order=none --test-list=path/to/file/from/step/3` |
+2. If the test doesn't fail here, then the test itself is probably just flaky. |
+ If it does, remove some lines from the file and repeat step 1. Continue |
+ repeating until you've found the dependency. If the test fails when run by |
+ itself, but passes on the bots, that means that it depends on another test to |
+ pass. In this case, you need to generate the list of tests run by |
+ `run-webkit-tests --order=natural` and repeat this process to find which test |
+ causes the test in question to *pass* (e.g. |
+ [crbug.com/262793](https://crbug.com/262793)). |
+3. File a bug and give it the |
+ [LayoutTestOrdering](https://crbug.com/?q=label:LayoutTestOrdering) label, |
+ e.g. [crbug.com/262787](https://crbug.com/262787) or |
+ [crbug.com/262791](https://crbug.com/262791). |
+ |
+### Finding test ordering flakiness |
+ |
+#### Run tests in a random order and diagnose failures |
+ |
+1. Run `run-webkit-tests --order=random --no-retry` |
+2. Run `./Tools/Scripts/print-test-ordering` and save the output to a file. This |
+ outputs the tests run in the order they were run on each content_shell |
+ instance. |
+3. Run the diagnosing steps from above to figure out which tests |
+ |
+Run `run-webkit-tests --run-singly --no-retry`. This starts up a new |
+content_shell instance for each test. Tests that fail when run in isolation but |
+pass when run as part of the full test suite represent some state that we're not |
+properly resetting between test runs or some state that we're not properly |
+setting when starting up content_shell. You might want to run with |
+`--time-out-ms=60000` to weed out tests that timeout due to waiting on |
+content_shell startup time. |
+ |
+#### Diagnose especially flaky tests |
+ |
+1. Load |
+ https://test-results.appspot.com/dashboards/overview.html#group=%40ToT%20Blink&flipCount=12 |
+2. Tweak the flakiness threshold to the desired level of flakiness. |
+3. Click on *webkit_tests* to get that list of flaky tests. |
+4. Diagnose the source of flakiness for that test. |