OLD | NEW |
---|---|
(Empty) | |
1 Layout test analyzer script. | |
2 | |
3 * Prerequisite: | |
4 | |
5 This script requires PySVN (http://pysvn.tigris.org/) to be installed on the | |
6 machine. | |
7 | |
8 * How to execute | |
9 | |
10 python layouttest_analyzer.py (Please run with '-h' for available command-line | |
11 options) | |
12 | |
13 Example: python layouttest_analyzer.py -r imasaki@chromium.org | |
14 -t /var/www/analyzer/graph.html | |
15 | |
16 * File/Directory structure | |
17 | |
18 layout_tests/ | |
19 README: this file. | |
20 *.py: python scripts. | |
21 tmp/: temp files. | |
22 test_data/: data for unit tests. | |
23 graph/: the default location for graph files. | |
24 anno/: the default location for bug annotation files. | |
25 result/: the default location for the analyzer results. | |
26 | |
27 * Execution overview. | |
28 | |
29 1) using PySVN, the script collects layout test of interest | |
30 (specified in command-line parameter and testnames/*.csv file) from | |
31 Webkit SVN repository as well as test description. Then, it gets the | |
32 latest test expectation file and does the simple parsing. The | |
33 parsed test expectation is joined with the layout tests using test name | |
34 as its join key. The tests are classified into 'whole', 'skip', | |
35 'nonskip' test groups. 'Whole' contains all tests, 'skip' contains | |
36 tests that are skipped as specified in the test expectation file, 'nonskip' | |
37 tests are in the test expectation file and not skipped (these are tests | |
38 that need attention). | |
39 | |
40 2) the script reads the previous latest analyzer results, then, | |
41 compares it with current results for each test group ('whole', 'skip', | |
42 and 'nonskip'). It also looks into the SVN test expectation history | |
43 diff for the time period for the test names of interest. | |
44 | |
45 3) all obtained information sent out to a email list specified | |
dennis_jeffrey
2011/09/01 00:29:13
'sent out' --> 'is sent out'
imasaki1
2011/09/01 03:08:23
Done.
| |
46 in the parameters in pretty HTML format. | |
47 | |
48 4) the trend graph is updated using the current results. | |
49 | |
50 * Components | |
51 | |
52 ** LayoutTests: | |
53 | |
54 A class to store test names in layout tests. The test names (including | |
55 regular expression patterns) are read from a CSV file and used for | |
56 getting layout test names from Webkit SVN as well as test description. | |
57 | |
58 ** TestExpectations: | |
59 | |
60 A class to model the content of test expectation file for analysis. | |
61 The raw test expectation file can be found in | |
62 |DEFAULT_TEST_EXPECTATION_LOCATION|. It is necessary to parse this | |
63 file and store meaningful information for the analysis (joining with | |
64 existing layout tests using a test name). Instance variable | |
65 |all_test_expectation_info| is used. A test name such as | |
66 'media/video-source-type.html' is used for the key to store | |
67 information. However, a test name can appear multiple times in the | |
68 test expectation file. So, the map should keep all the occurrence | |
69 information. For example, the current test expectation file has the | |
70 following two entries: | |
71 | |
72 BUGWK58587 LINUX DEBUG GPU : media/video-zoom.html = IMAGE | |
73 BUGCR86714 MAC GPU : media/video-zoom.html = CRASH IMAGE | |
74 | |
75 In this case, all_test_expectation_info['media/video-zoom.html'] will | |
76 have a list with two elements, each of which is the map of the test | |
77 expectation information. | |
78 | |
79 ** TestExpectationsHistory: | |
80 | |
81 A class to represent history of the test expectation file. The history | |
82 is obtained by calling PySVN.log()/diff() APIs using the specified time | |
83 period. | |
84 | |
85 ** TrendGraph: | |
86 | |
87 A class to manage trend graph which is using Google Visualization | |
88 APIs. Google Visualization API | |
89 (http://code.google.com/apis/chart/interactive/docs/gallery/annotatedtimeline. html) | |
90 is used to present the historical analyzer result. Currently, data | |
91 is directly written to JavaScript file using file in-place | |
92 replacement for simplicity. | |
OLD | NEW |