Added tools for finding and purging flaky tests

Added tools for finding and purging flaky tests

These may be helpful


Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=97560

[clee, 2011-08-19 15:01:58]

Uploading both right now just to get them on the table. Only find_flakies has
been cleaned up though.

[M-A Ruel, 2011-08-19 15:07:01]

lgtm

http://codereview.chromium.org/7688004/diff/1/tools/purge_flakies.py
File tools/purge_flakies.py (right):

http://codereview.chromium.org/7688004/diff/1/tools/purge_flakies.py#newcode3
tools/purge_flakies.py:3: 
Just add the copyright header there

http://codereview.chromium.org/7688004/diff/1/tools/purge_flakies.py#newcode84
tools/purge_flakies.py:84: 
remove the extra line there

[clee, 2011-08-19 16:53:57]

purge_flakies ready 
added a few comments to find_flakies too

http://codereview.chromium.org/7688004/diff/1/tools/purge_flakies.py
File tools/purge_flakies.py (right):

http://codereview.chromium.org/7688004/diff/1/tools/purge_flakies.py#newcode3
tools/purge_flakies.py:3: 
On 2011/08/19 15:07:01, Marc-Antoine Ruel wrote:
> Just add the copyright header there

Done.

http://codereview.chromium.org/7688004/diff/1/tools/purge_flakies.py#newcode84
tools/purge_flakies.py:84: 
On 2011/08/19 15:07:01, Marc-Antoine Ruel wrote:
> remove the extra line there

Done.

[Paweł Hajdan Jr., 2011-08-19 18:35:50]

http://codereview.chromium.org/7688004/diff/4001/tools/find_flakies.py
File tools/find_flakies.py (right):

http://codereview.chromium.org/7688004/diff/4001/tools/find_flakies.py#newcode1
tools/find_flakies.py:1: #!/usr/bin/env python
Add a new directory for those like toos/flakiness. Avoid abbreviations like
flakies, use full name "flakiness" or "flaky".

http://codereview.chromium.org/7688004/diff/4001/tools/find_flakies.py#newcode29
tools/find_flakies.py:29: # read a previously written data file
nit: Comments should start with a capital letter and end with a dot. Applies to
entire CL.

http://codereview.chromium.org/7688004/diff/4001/tools/purge_flakies.py
File tools/purge_flakies.py (right):

http://codereview.chromium.org/7688004/diff/4001/tools/purge_flakies.py#newcode6
tools/purge_flakies.py:6: """Runs all the test cases in a given test in parallel
with itself, to get at
Would it make sense to combine this in a single file with the previous tool?
Requiring people to run two scripts is more complicated than saying "run this
one script, it will take a lot of time but will finally print the useful
results". It also makes running it overnight more effective.

[clee, 2011-08-19 21:06:35]

It's all in one file now, but the defaults may need to be tweaked a little for
optimal overnight use.

http://codereview.chromium.org/7688004/diff/4001/tools/find_flakies.py
File tools/find_flakies.py (right):

http://codereview.chromium.org/7688004/diff/4001/tools/find_flakies.py#newcode1
tools/find_flakies.py:1: #!/usr/bin/env python
On 2011/08/19 18:35:50, Paweł Hajdan Jr. wrote:
> Add a new directory for those like toos/flakiness. Avoid abbreviations like
> flakies, use full name "flakiness" or "flaky".

Done.

http://codereview.chromium.org/7688004/diff/4001/tools/find_flakies.py#newcode29
tools/find_flakies.py:29: # read a previously written data file
On 2011/08/19 18:35:50, Paweł Hajdan Jr. wrote:
> nit: Comments should start with a capital letter and end with a dot. Applies
to
> entire CL.

Done.

http://codereview.chromium.org/7688004/diff/4001/tools/purge_flakies.py
File tools/purge_flakies.py (right):

http://codereview.chromium.org/7688004/diff/4001/tools/purge_flakies.py#newcode6
tools/purge_flakies.py:6: """Runs all the test cases in a given test in parallel
with itself, to get at
On 2011/08/19 18:35:50, Paweł Hajdan Jr. wrote:
> Would it make sense to combine this in a single file with the previous tool?
> Requiring people to run two scripts is more complicated than saying "run this
> one script, it will take a lot of time but will finally print the useful
> results". It also makes running it overnight more effective.

Ideally yes, but the command line options to the two scripts are different, and
the formats of the output files are different. It's very easy to write something
that just runs the two procedures with default options, but quite confusing to
allow argument specification. I'll just do it with default arguments here, and
then you can call the functions directly to change them.

[commit-bot: I haz the power, 2011-08-20 05:32:25]

Change committed as 97560