Add flaky test classification feature to test suites.

Add flaky test classification feature to test suites.

Test expectations can now include outcome: FLAKY.

The test runner can now run a class of tests (flaky|non-flaky|all). All tests are in the non-flaky class that are not marked as FLAKY.

The slash correction for windows is now pulled into the test name method. Currently the progress output on windows contains a mixture of / and \.

R=jkummerow@chromium.org

Committed: https://code.google.com/p/v8/source/detail?r=16080

[Michael Achenbach, 2013-08-06 10:01:43]

PTAL

[Jakob Kummerow, 2013-08-06 11:11:01]

LGTM with comments.

https://codereview.chromium.org/22381003/diff/5001/tools/run-tests.py
File tools/run-tests.py (right):

https://codereview.chromium.org/22381003/diff/5001/tools/run-tests.py#newcode97
tools/run-tests.py:97: result.add_option("--classification",
"classification" is too generic. How about "--flaky-tests" or
"--run-flaky-tests" instead? The options' names could stay the same.

https://codereview.chromium.org/22381003/diff/5001/tools/testrunner/local/tes...
File tools/testrunner/local/testsuite.py (right):

https://codereview.chromium.org/22381003/diff/5001/tools/testrunner/local/tes...
tools/testrunner/local/testsuite.py:70: return testcase.path.replace("\\", "/")
Good catch!

https://codereview.chromium.org/22381003/diff/5001/tools/testrunner/local/tes...
tools/testrunner/local/testsuite.py:92: return cls == "flaky" and not flaky or
cls == "non-flaky" and flaky
I'm paranoid about operator precedence. Please add ():
    return (cls == "flaky" and not flaky) or (cls == "non-flaky" and flaky)

[Michael Achenbach, 2013-08-06 12:02:06]

PTAL. Made some renaming.

[Jakob Kummerow, 2013-08-06 14:30:25]

lgtm

[Michael Achenbach, 2013-08-06 14:39:45]

Committed patchset #2 manually as r16080 (presubmit successful).