Index: media/tools/layout_tests/README |
diff --git a/media/tools/layout_tests/README b/media/tools/layout_tests/README |
new file mode 100644 |
index 0000000000000000000000000000000000000000..61e95b047cd8d723bc04d4c24f532dc6ce0a7466 |
--- /dev/null |
+++ b/media/tools/layout_tests/README |
@@ -0,0 +1,82 @@ |
+Layout test analyzer script. |
+ |
+* Prerequisite: |
+ |
+This script requires PySVN (http://pysvn.tigris.org/) to be installed on the |
+machine. |
+ |
+* How to execute |
+ |
+python layouttest_analyzer.py (Please run with '-h' for available command-line |
+ options) |
dennis_jeffrey
2011/08/31 17:49:40
(optional) You may want to also give a full exampl
imasaki1
2011/08/31 19:08:19
Done.
|
+ |
+* Directory structure |
dennis_jeffrey
2011/08/31 17:49:40
The description of the directory structure seems t
imasaki1
2011/08/31 19:08:19
Done.
|
+ |
+Design overview: |
+ |
+* Execution overflow overview. |
dennis_jeffrey
2011/08/31 17:49:40
I'm not sure what you mean by 'overflow' here; cou
imasaki1
2011/08/31 19:08:19
Done.
|
+ |
+First, using PySVN, the script collects layout test of interest |
+(specified in command-line parameter and testnames/*.csv file) from |
+Webkit SVN repository as well as test description. Then, it gets the |
+the latest test expectation file and does the simple parsing. The |
dennis_jeffrey
2011/08/31 17:49:40
remove 'the' at the start of this line (it's dupli
imasaki1
2011/08/31 19:08:19
Done.
|
+parsed test expectation is joined with the layout tests using test name |
+as its join key. The tests are classified into 'whole', 'skip', |
+'nonskip' test groups. 'Whole' contains all tests, 'skip' contains |
+tests are skipped as specified in the test expectation file, 'nonskip' |
dennis_jeffrey
2011/08/31 17:49:40
'tests are' --> 'tests that are'
imasaki1
2011/08/31 19:08:19
Done.
|
+tests are in the test expectation file and not skipped (these are tests |
+that needs attention). |
dennis_jeffrey
2011/08/31 17:49:40
'needs' --> 'need'
imasaki1
2011/08/31 19:08:19
Done.
|
+ |
+Next, the script reads the previous latest analyzer results, then, |
+compares it with current results for each test group ('whole', 'skip', |
+and 'nonskip'). It also looks into the SVN test expectation history |
+diff for the time period for the test names of interest. |
+ |
+Thirdly, all obtained information sent out to a email list specified |
+in the parameters in pretty HTML format. |
+ |
+Lastly, the trend graph is updated using the current results. |
dennis_jeffrey
2011/08/31 17:49:40
(optional) For lines 19-38, it might be a little e
imasaki1
2011/08/31 19:08:19
Done.
|
+ |
+* Components |
+ |
+** LayoutTests: |
+ |
+A class to store test names in layout tests. The test names (including |
+regular expression patterns) are read from a CSV file and used for |
+getting layout test names from Webkit SVN as well as test description. |
+ |
+** TestExpectations: |
+ |
+A class to model the content of test expectation file for analysis. |
+The raw test expectation file can be found in |
+|DEFAULT_TEST_EXPECTATION_LOCATION|. It is necessary to parse this |
+file and store meaningful information for the analysis (joining with |
+existing layout tests using a test name). Instance variable |
+|all_test_expectation_info| is used. A test name such as |
+'media/video-source-type.html' is used for the key to store |
+information. However, a test name can appear multiple times in the |
+test expectation file. So, the map should keep all the occurrence |
+information. For example, the current test expectation file has the |
+following two entries: |
+ |
+BUGWK58587 LINUX DEBUG GPU : media/video-zoom.html = IMAGE |
+BUGCR86714 MAC GPU : media/video-zoom.html = CRASH IMAGE |
+ |
+In this case, all_test_expectation_info['media/video-zoom.html'] will |
+have a list with two elements, each of which is the map of the test |
+expectation information. |
+ |
+** TestExpectationsHistory: |
+ |
+A class to represent history of the test expectation file. The history |
+is obtained by calling PySVN.log()/diff() APIs using the specified time |
+period. |
+ |
+** TrendGraph: |
+ |
+A class to manage trend graph which is using Google Visualization |
+ APIs. Google Visualization API |
+ (http://code.google.com/apis/chart/interactive/docs/gallery/annotatedtimeline.html) |
+ is used to present the historical analyzer result. Currently, data |
+ is directly written to JavaScript file using file in-place |
+ replacement for simplicity. |