Chromium Code Reviews| Index: media/tools/layout_tests/README |
| diff --git a/media/tools/layout_tests/README b/media/tools/layout_tests/README |
| new file mode 100644 |
| index 0000000000000000000000000000000000000000..61e95b047cd8d723bc04d4c24f532dc6ce0a7466 |
| --- /dev/null |
| +++ b/media/tools/layout_tests/README |
| @@ -0,0 +1,82 @@ |
| +Layout test analyzer script. |
| + |
| +* Prerequisite: |
| + |
| +This script requires PySVN (http://pysvn.tigris.org/) to be installed on the |
| +machine. |
| + |
| +* How to execute |
| + |
| +python layouttest_analyzer.py (Please run with '-h' for available command-line |
| + options) |
|
dennis_jeffrey
2011/08/31 17:49:40
(optional) You may want to also give a full exampl
imasaki1
2011/08/31 19:08:19
Done.
|
| + |
| +* Directory structure |
|
dennis_jeffrey
2011/08/31 17:49:40
The description of the directory structure seems t
imasaki1
2011/08/31 19:08:19
Done.
|
| + |
| +Design overview: |
| + |
| +* Execution overflow overview. |
|
dennis_jeffrey
2011/08/31 17:49:40
I'm not sure what you mean by 'overflow' here; cou
imasaki1
2011/08/31 19:08:19
Done.
|
| + |
| +First, using PySVN, the script collects layout test of interest |
| +(specified in command-line parameter and testnames/*.csv file) from |
| +Webkit SVN repository as well as test description. Then, it gets the |
| +the latest test expectation file and does the simple parsing. The |
|
dennis_jeffrey
2011/08/31 17:49:40
remove 'the' at the start of this line (it's dupli
imasaki1
2011/08/31 19:08:19
Done.
|
| +parsed test expectation is joined with the layout tests using test name |
| +as its join key. The tests are classified into 'whole', 'skip', |
| +'nonskip' test groups. 'Whole' contains all tests, 'skip' contains |
| +tests are skipped as specified in the test expectation file, 'nonskip' |
|
dennis_jeffrey
2011/08/31 17:49:40
'tests are' --> 'tests that are'
imasaki1
2011/08/31 19:08:19
Done.
|
| +tests are in the test expectation file and not skipped (these are tests |
| +that needs attention). |
|
dennis_jeffrey
2011/08/31 17:49:40
'needs' --> 'need'
imasaki1
2011/08/31 19:08:19
Done.
|
| + |
| +Next, the script reads the previous latest analyzer results, then, |
| +compares it with current results for each test group ('whole', 'skip', |
| +and 'nonskip'). It also looks into the SVN test expectation history |
| +diff for the time period for the test names of interest. |
| + |
| +Thirdly, all obtained information sent out to a email list specified |
| +in the parameters in pretty HTML format. |
| + |
| +Lastly, the trend graph is updated using the current results. |
|
dennis_jeffrey
2011/08/31 17:49:40
(optional) For lines 19-38, it might be a little e
imasaki1
2011/08/31 19:08:19
Done.
|
| + |
| +* Components |
| + |
| +** LayoutTests: |
| + |
| +A class to store test names in layout tests. The test names (including |
| +regular expression patterns) are read from a CSV file and used for |
| +getting layout test names from Webkit SVN as well as test description. |
| + |
| +** TestExpectations: |
| + |
| +A class to model the content of test expectation file for analysis. |
| +The raw test expectation file can be found in |
| +|DEFAULT_TEST_EXPECTATION_LOCATION|. It is necessary to parse this |
| +file and store meaningful information for the analysis (joining with |
| +existing layout tests using a test name). Instance variable |
| +|all_test_expectation_info| is used. A test name such as |
| +'media/video-source-type.html' is used for the key to store |
| +information. However, a test name can appear multiple times in the |
| +test expectation file. So, the map should keep all the occurrence |
| +information. For example, the current test expectation file has the |
| +following two entries: |
| + |
| +BUGWK58587 LINUX DEBUG GPU : media/video-zoom.html = IMAGE |
| +BUGCR86714 MAC GPU : media/video-zoom.html = CRASH IMAGE |
| + |
| +In this case, all_test_expectation_info['media/video-zoom.html'] will |
| +have a list with two elements, each of which is the map of the test |
| +expectation information. |
| + |
| +** TestExpectationsHistory: |
| + |
| +A class to represent history of the test expectation file. The history |
| +is obtained by calling PySVN.log()/diff() APIs using the specified time |
| +period. |
| + |
| +** TrendGraph: |
| + |
| +A class to manage trend graph which is using Google Visualization |
| + APIs. Google Visualization API |
| + (http://code.google.com/apis/chart/interactive/docs/gallery/annotatedtimeline.html) |
| + is used to present the historical analyzer result. Currently, data |
| + is directly written to JavaScript file using file in-place |
| + replacement for simplicity. |