OLD | NEW |
(Empty) | |
| 1 Layout test analyzer script. |
| 2 |
| 3 * Prerequisite: |
| 4 |
| 5 This script requires PySVN (http://pysvn.tigris.org/) to be installed on the |
| 6 machine. |
| 7 |
| 8 * How to execute |
| 9 |
| 10 python layouttest_analyzer.py (Please run with '-h' for available command-line |
| 11 options) |
| 12 |
| 13 Example: python layouttest_analyzer.py -r imasaki@chromium.org |
| 14 -t /var/www/analyzer/graph.html |
| 15 |
| 16 * File/Directory structure |
| 17 |
| 18 layout_tests/ |
| 19 README: this file. |
| 20 *.py: python scripts. |
| 21 tmp/: temp files. |
| 22 test_data/: data for unit tests. |
| 23 graph/: the default location for graph files. |
| 24 anno/: the default location for bug annotation files. |
| 25 result/: the default location for the analyzer results. |
| 26 testname/: the default location for the test group definition files in CSV. |
| 27 |
| 28 * Execution overview. |
| 29 |
| 30 1) using PySVN, the script collects layout test of interest |
| 31 (specified in command-line parameter and testnames/*.csv file) from |
| 32 Webkit SVN repository as well as test description. Then, it gets the |
| 33 latest test expectation file and does the simple parsing. The |
| 34 parsed test expectation is joined with the layout tests using test name |
| 35 as its join key. The tests are classified into 'whole', 'skip', |
| 36 'nonskip' test groups. 'Whole' contains all tests, 'skip' contains |
| 37 tests that are skipped as specified in the test expectation file, 'nonskip' |
| 38 tests are in the test expectation file and not skipped (these are tests |
| 39 that need attention). |
| 40 |
| 41 2) the script reads the previous latest analyzer results, then, |
| 42 compares it with current results for each test group ('whole', 'skip', |
| 43 and 'nonskip'). It also looks into the SVN test expectation history |
| 44 diff for the time period for the test names of interest. |
| 45 |
| 46 3) all obtained information is sent out to a email list specified |
| 47 in the parameters in pretty HTML format. |
| 48 |
| 49 4) the trend graph is updated using the current results. |
| 50 |
| 51 * Components |
| 52 |
| 53 ** LayoutTests: |
| 54 |
| 55 A class to store test names in layout tests. The test names (including |
| 56 regular expression patterns) are read from a CSV file and used for |
| 57 getting layout test names from Webkit SVN as well as test description. |
| 58 |
| 59 ** TestExpectations: |
| 60 |
| 61 A class to model the content of test expectation file for analysis. |
| 62 The raw test expectation file can be found in |
| 63 |DEFAULT_TEST_EXPECTATION_LOCATION|. It is necessary to parse this |
| 64 file and store meaningful information for the analysis (joining with |
| 65 existing layout tests using a test name). Instance variable |
| 66 |all_test_expectation_info| is used. A test name such as |
| 67 'media/video-source-type.html' is used for the key to store |
| 68 information. However, a test name can appear multiple times in the |
| 69 test expectation file. So, the map should keep all the occurrence |
| 70 information. For example, the current test expectation file has the |
| 71 following two entries: |
| 72 |
| 73 BUGWK58587 LINUX DEBUG GPU : media/video-zoom.html = IMAGE |
| 74 BUGCR86714 MAC GPU : media/video-zoom.html = CRASH IMAGE |
| 75 |
| 76 In this case, all_test_expectation_info['media/video-zoom.html'] will |
| 77 have a list with two elements, each of which is the map of the test |
| 78 expectation information. |
| 79 |
| 80 ** TestExpectationsHistory: |
| 81 |
| 82 A class to represent history of the test expectation file. The history |
| 83 is obtained by calling PySVN.log()/diff() APIs using the specified time |
| 84 period. |
| 85 |
| 86 ** TrendGraph: |
| 87 |
| 88 A class to manage trend graph which is using Google Visualization |
| 89 APIs. Google Visualization API |
| 90 (http://code.google.com/apis/chart/interactive/docs/gallery/annotatedtimeline.
html) |
| 91 is used to present the historical analyzer result. Currently, data |
| 92 is directly written to JavaScript file using file in-place |
| 93 replacement for simplicity. |
OLD | NEW |