Chromium Code Reviews| OLD | NEW |
|---|---|
| (Empty) | |
| 1 Layout test analyzer script. | |
| 2 | |
| 3 * Prerequisite: | |
| 4 | |
| 5 This script requires PySVN (http://pysvn.tigris.org/) to be installed on the | |
| 6 machine. | |
| 7 | |
| 8 * How to execute | |
| 9 | |
| 10 python layouttest_analyzer.py (Please run with '-h' for available command-line | |
| 11 options) | |
|
dennis_jeffrey
2011/08/31 17:49:40
(optional) You may want to also give a full exampl
imasaki1
2011/08/31 19:08:19
Done.
| |
| 12 | |
| 13 * Directory structure | |
|
dennis_jeffrey
2011/08/31 17:49:40
The description of the directory structure seems t
imasaki1
2011/08/31 19:08:19
Done.
| |
| 14 | |
| 15 Design overview: | |
| 16 | |
| 17 * Execution overflow overview. | |
|
dennis_jeffrey
2011/08/31 17:49:40
I'm not sure what you mean by 'overflow' here; cou
imasaki1
2011/08/31 19:08:19
Done.
| |
| 18 | |
| 19 First, using PySVN, the script collects layout test of interest | |
| 20 (specified in command-line parameter and testnames/*.csv file) from | |
| 21 Webkit SVN repository as well as test description. Then, it gets the | |
| 22 the latest test expectation file and does the simple parsing. The | |
|
dennis_jeffrey
2011/08/31 17:49:40
remove 'the' at the start of this line (it's dupli
imasaki1
2011/08/31 19:08:19
Done.
| |
| 23 parsed test expectation is joined with the layout tests using test name | |
| 24 as its join key. The tests are classified into 'whole', 'skip', | |
| 25 'nonskip' test groups. 'Whole' contains all tests, 'skip' contains | |
| 26 tests are skipped as specified in the test expectation file, 'nonskip' | |
|
dennis_jeffrey
2011/08/31 17:49:40
'tests are' --> 'tests that are'
imasaki1
2011/08/31 19:08:19
Done.
| |
| 27 tests are in the test expectation file and not skipped (these are tests | |
| 28 that needs attention). | |
|
dennis_jeffrey
2011/08/31 17:49:40
'needs' --> 'need'
imasaki1
2011/08/31 19:08:19
Done.
| |
| 29 | |
| 30 Next, the script reads the previous latest analyzer results, then, | |
| 31 compares it with current results for each test group ('whole', 'skip', | |
| 32 and 'nonskip'). It also looks into the SVN test expectation history | |
| 33 diff for the time period for the test names of interest. | |
| 34 | |
| 35 Thirdly, all obtained information sent out to a email list specified | |
| 36 in the parameters in pretty HTML format. | |
| 37 | |
| 38 Lastly, the trend graph is updated using the current results. | |
|
dennis_jeffrey
2011/08/31 17:49:40
(optional) For lines 19-38, it might be a little e
imasaki1
2011/08/31 19:08:19
Done.
| |
| 39 | |
| 40 * Components | |
| 41 | |
| 42 ** LayoutTests: | |
| 43 | |
| 44 A class to store test names in layout tests. The test names (including | |
| 45 regular expression patterns) are read from a CSV file and used for | |
| 46 getting layout test names from Webkit SVN as well as test description. | |
| 47 | |
| 48 ** TestExpectations: | |
| 49 | |
| 50 A class to model the content of test expectation file for analysis. | |
| 51 The raw test expectation file can be found in | |
| 52 |DEFAULT_TEST_EXPECTATION_LOCATION|. It is necessary to parse this | |
| 53 file and store meaningful information for the analysis (joining with | |
| 54 existing layout tests using a test name). Instance variable | |
| 55 |all_test_expectation_info| is used. A test name such as | |
| 56 'media/video-source-type.html' is used for the key to store | |
| 57 information. However, a test name can appear multiple times in the | |
| 58 test expectation file. So, the map should keep all the occurrence | |
| 59 information. For example, the current test expectation file has the | |
| 60 following two entries: | |
| 61 | |
| 62 BUGWK58587 LINUX DEBUG GPU : media/video-zoom.html = IMAGE | |
| 63 BUGCR86714 MAC GPU : media/video-zoom.html = CRASH IMAGE | |
| 64 | |
| 65 In this case, all_test_expectation_info['media/video-zoom.html'] will | |
| 66 have a list with two elements, each of which is the map of the test | |
| 67 expectation information. | |
| 68 | |
| 69 ** TestExpectationsHistory: | |
| 70 | |
| 71 A class to represent history of the test expectation file. The history | |
| 72 is obtained by calling PySVN.log()/diff() APIs using the specified time | |
| 73 period. | |
| 74 | |
| 75 ** TrendGraph: | |
| 76 | |
| 77 A class to manage trend graph which is using Google Visualization | |
| 78 APIs. Google Visualization API | |
| 79 (http://code.google.com/apis/chart/interactive/docs/gallery/annotatedtimeline. html) | |
| 80 is used to present the historical analyzer result. Currently, data | |
| 81 is directly written to JavaScript file using file in-place | |
| 82 replacement for simplicity. | |
| OLD | NEW |