OLD | NEW |
1 # Extending the Layout Framework | 1 # How to Extend the Layout Test Framework |
2 | 2 |
3 # Introduction | 3 The Layout Test Framework that Blink uses is a regression testing tool that is |
4 The Layout Test Framework that Blink uses is a regression testing tool that is m
ulti-platform and it has a large amount of tools that help test varying types of
regression, such as pixel diffs, text diffs, etc. The framework is mainly used
by Blink, however it was made to be extensible so that other projects can use it
test different parts of chrome (such as Print Preview). This is a guide to help
people who want to actually the framework to test whatever they want. | 4 multi-platform and it has a large amount of tools that help test varying types |
5 | 5 of regression, such as pixel diffs, text diffs, etc. The framework is mainly |
6 # Background | 6 used by Blink, however it was made to be extensible so that other projects can |
7 Before you can start actually extending the framework, you should be familiar wi
th how to use it. This wiki is basically all you need to learn how to use it | 7 use it test different parts of chrome (such as Print Preview). This is a guide |
| 8 to help people who want to actually the framework to test whatever they want. |
| 9 |
| 10 [TOC] |
| 11 |
| 12 ## Background |
| 13 |
| 14 Before you can start actually extending the framework, you should be familiar |
| 15 with how to use it. This wiki is basically all you need to learn how to use it |
8 http://www.chromium.org/developers/testing/webkit-layout-tests | 16 http://www.chromium.org/developers/testing/webkit-layout-tests |
9 | 17 |
10 # How to Extend the Framework | 18 ## How to Extend the Framework |
| 19 |
11 There are two parts to actually extending framework to test a piece of software. | 20 There are two parts to actually extending framework to test a piece of software. |
12 The first part is extending certain files in: | 21 The first part is extending certain files in: |
13 src/third\_party/Webkit/Tools/Scripts/webkitpy/layout\_tests/ | 22 [/third_party/Webkit/Tools/Scripts/webkitpy/layout_tests/](/third_party/Webkit/T
ools/Scripts/webkitpy/layout_tests/) |
14 The code in webkitpy/layout\_tests is the layout test framework itself | 23 The code in `webkitpy/layout_tests` is the layout test framework itself |
15 | 24 |
16 The second part is creating a driver (program) to actually communicate the layou
t test framework. This part is significantly more tricky and dependant on what e
xactly exactly is being tested. | 25 The second part is creating a driver (program) to actually communicate the |
17 | 26 layout test framework. This part is significantly more tricky and dependant on |
18 ## Part 1: | 27 what exactly exactly is being tested. |
19 This part isn’t too difficult. There are basically two classes that need to be e
xtended (ideally, just inherited from). These classes are: | 28 |
20 Driver | 29 ### Part 1 |
21 Located in layout\_tests/port/driver.py | 30 |
22 Each instance of this is the class that will actually an instance of the program
that produces the test data (program in Part 2). | 31 This part isn’t too difficult. There are basically two classes that need to be |
23 Port | 32 extended (ideally, just inherited from). These classes are: |
24 Located in layout\_tests/port/base.py | 33 |
25 This class is responsible creating drivers with the correct settings, giving acc
ess to certain OS functionality to access expected files, etc. | 34 Driver |
26 | 35 |
27 | 36 Located in `layout_tests/port/driver.py`. Each instance of this is the class |
28 | 37 that will actually an instance of the program that produces the test data |
29 | 38 (program in Part 2). |
30 | 39 |
31 ### Extending Driver: | 40 Port |
32 As said, Driver launches the program from Part 2. Said program will communicate
with the driver class to receive instructions and send back data. All of the wor
k for driver gets done in Driver.run\_test. Everything else is a helper or initi
alization function. | 41 |
33 run\_test() steps: | 42 Located in `layout_tests/port/base.py`. This class is responsible creating |
34 1. On the very first call of this function, it will actually run the test prog
ram. On every subsequent call to this function, at the beginning it will verify
that the process doesn’t need to be restarted, and if it does, it will create a
new instance of the test program. | 43 drivers with the correct settings, giving access to certain OS functionality to |
35 1. It will then create a command to send the program | 44 access expected files, etc. |
36 * This command generally consists of an html file path for the test program
to navigate to. | 45 |
37 * After creating it, the command is sent | 46 #### Extending Driver |
38 1. After the command has been sent, it will then wait for data from the progra
m. | 47 |
39 * It will actually wait for 2 blocks of data. | 48 As said, Driver launches the program from Part 2. Said program will communicate |
40 * The first part being text or audio data. This part is required (the prog
ram will always send something, even an empty string) | 49 with the driver class to receive instructions and send back data. All of the |
41 * The second block is optional and is image data and an image hash (md5) t
his block of data is used for pixel tests | 50 work for driver gets done in `Driver.run_test`. Everything else is a helper or |
42 1. After it has received all the data, it will proceed to check if the program
has timed out or crashed, and if so fail this instance of the test (it can be r
etried later if need be). | 51 initialization function. |
43 | 52 |
44 Luckily, run\_test() most likely doesn’t need to be overridden unless extra bloc
ks of data need to be sent to/read from the test program. However, you do need t
o know how it works because it will influence what functions you need to overrid
e. Here are the ones you’re probably going to need to override | 53 `run_test()` steps: |
45 cmd\_line | 54 |
46 This function creates a set of command line arguments to run the test program, s
o the function will almost certainly need to be overridden. | 55 1. On the very first call of this function, it will actually run the test |
47 It creates the command line to run the program. Driver uses subprocess.popen to
create the process, which takes the name of the test program and any options it
might need. | 56 program. On every subsequent call to this function, at the beginning it will |
48 The first item in the list of arguments should be the path to test program using
this function: | 57 verify that the process doesn’t need to be restarted, and if it does, it |
49 self._port._path\_to\_driver() | 58 will create a new instance of the test program. |
50 This is an absolute path to the test program. | 59 1. It will then create a command to send the program |
51 This is the bare minimum you need to get the driver to launch the test program,
however if you have options you need to append, just append them to the list. | 60 * This command generally consists of an html file path for the test |
52 start | 61 program to navigate to. |
53 If your program has any special startup needs, then this will be the place to pu
t it. | 62 * After creating it, the command is sent |
54 | 63 1. After the command has been sent, it will then wait for data from the |
55 That’s mostly it. The Driver class has almost all the functionality you could wa
nt, so there isn’t much to override here. If extra data needs to be read or sent
, extra data members should be added to ContentBlock. | 64 program. |
56 | 65 * It will actually wait for 2 blocks of data. |
57 ### Extending Port: | 66 * The first part being text or audio data. This part is required (the |
58 This class is responsible for providing functionality such as where to look for
tests, where to store test results, what driver to run, what timeout to use, wha
t kind of files can be run, etc. It provides a lot of functionality, however it
isn’t really sufficient because it doesn’t account of platform specific problems
, therefore port itself shouldn’t be extend. Instead LinuxPort, WinPort, and Mac
Port (and maybe the android port class) should be extended as they provide platf
orm specific overrides/extensions that implement most of the important functiona
lity. While there are many functions in Port, overriding one function will affec
t most of the other ones to get the desired behavior. For example, if layout\_te
sts\_dir() is overriden, not only will the code look for tests in that directory
, but it will find the correct TestExpectations file, the platform specific expe
cted files, etc. | 67 program will always send something, even an empty string) |
| 68 * The second block is optional and is image data and an image hash |
| 69 (md5) this block of data is used for pixel tests |
| 70 1. After it has received all the data, it will proceed to check if the program |
| 71 has timed out or crashed, and if so fail this instance of the test (it can |
| 72 be retried later if need be). |
| 73 |
| 74 Luckily, `run_test()` most likely doesn’t need to be overridden unless extra |
| 75 blocks of data need to be sent to/read from the test program. However, you do |
| 76 need to know how it works because it will influence what functions you need to |
| 77 override. Here are the ones you’re probably going to need to override |
| 78 |
| 79 cmd_line |
| 80 |
| 81 This function creates a set of command line arguments to run the test program, |
| 82 so the function will almost certainly need to be overridden. |
| 83 |
| 84 It creates the command line to run the program. `Driver` uses `subprocess.popen` |
| 85 to create the process, which takes the name of the test program and any options |
| 86 it might need. |
| 87 |
| 88 The first item in the list of arguments should be the path to test program using |
| 89 this function: |
| 90 |
| 91 self._port._path_to_driver() |
| 92 |
| 93 This is an absolute path to the test program. This is the bare minimum you need |
| 94 to get the driver to launch the test program, however if you have options you |
| 95 need to append, just append them to the list. |
| 96 |
| 97 start |
| 98 |
| 99 If your program has any special startup needs, then this will be the place to |
| 100 put it. |
| 101 |
| 102 That’s mostly it. The Driver class has almost all the functionality you could |
| 103 want, so there isn’t much to override here. If extra data needs to be read or |
| 104 sent, extra data members should be added to `ContentBlock`. |
| 105 |
| 106 #### Extending Port |
| 107 |
| 108 This class is responsible for providing functionality such as where to look for |
| 109 tests, where to store test results, what driver to run, what timeout to use, |
| 110 what kind of files can be run, etc. It provides a lot of functionality, however |
| 111 it isn’t really sufficient because it doesn’t account of platform specific |
| 112 problems, therefore port itself shouldn’t be extend. Instead LinuxPort, WinPort, |
| 113 and MacPort (and maybe the android port class) should be extended as they |
| 114 provide platform specific overrides/extensions that implement most of the |
| 115 important functionality. While there are many functions in Port, overriding one |
| 116 function will affect most of the other ones to get the desired behavior. For |
| 117 example, if `layout_tests_dir()` is overriden, not only will the code look for |
| 118 tests in that directory, but it will find the correct TestExpectations file, the |
| 119 platform specific expected files, etc. |
59 | 120 |
60 Here are some of the functions that most likely need to be overridden. | 121 Here are some of the functions that most likely need to be overridden. |
61 * driver\_class | 122 * `driver_class` |
62 * This should be overridden to allow the testing program to actually run. By
default the code will run content\_shell, which might or might not be what you
want. | 123 * This should be overridden to allow the testing program to actually run. |
63 * It should be overridden to return the driver extension class created earli
er.This function doesn’t return an instance on the driver, just the class itself
. | 124 By default the code will run content_shell, which might or might not be |
64 * driver\_name | 125 what you want. |
65 * This should return the name of the program test p. By default it returns ‘
content\_shell’, but you want to have it return the program you want to run, suc
h as chrome or browser\_tests. | 126 * It should be overridden to return the driver extension class created |
66 * layout\_tests\_dir | 127 earlier. This function doesn’t return an instance on the driver, just |
67 * This tells the port where to look for all the and everything associated wi
th them such as resources files. | 128 the class itself. |
68 * By default it returns absolute path to the webkit tests. | 129 * `driver_name` |
69 * If you are planning on running something in the chromium src/ directory, t
here are helper functions to allow you to return a path relative to the base of
the chromium src directory. | 130 * This should return the name of the program test p. By default it returns |
70 | 131 ‘content_shell’, but you want to have it return the program you want to |
71 The rest of the functions can definitely be overridden for your projects specifi
c needs, however these are the bare minimum needed to get it running. There are
also functions you can override to make certain actions that aren’t on by defaul
t always take place. For example, the layout test framework always checks for sy
stem dependencies unless you pass in a switch. If you want them disabled for you
r project, just override check\_sys\_deps to always return OK. This way you don’
t need to pass in so many switches. | 132 run, such as `chrome` or `browser_tests`. |
72 | 133 * `layout_tests_dir` |
73 As said earlier, you should override LinuxPort, MacPort, and/or WinPort. You sho
uld create a class that implements the platform independent overrides (such as d
river\_class) and then create a separate class for each platform specific port o
f your program that inherits from the class with the independent overrides and t
he platform port you want. For example, you might want to have a different timeo
ut for your project, but on Windows the timeout needs to be vastly different tha
n the others. In this case you can just create a default override that every cla
ss uses except your Windows port. In that port you can just override the functio
n again to provide the specific timeout you need. This way you don’t need to mai
ntain the same function on each platform if they all do the same thing. | 134 * This tells the port where to look for all the and everything associated |
74 | 135 with them such as resources files. |
75 For Driver and Port that’s basically it unless you need to make many odd modific
ations. Lots of functionality is already there so you shouldn’t really need to d
o much. | 136 * By default it returns absolute path to the webkit tests. |
76 | 137 * If you are planning on running something in the chromium src/ directory, |
77 ## Part 2: | 138 there are helper functions to allow you to return a path relative to the |
78 This is the part where you create the program that your driver class launches. T
his part is very application dependent, so it will not be a guide on how impleme
nt certain features, just what should be implemented and the order in which even
ts should occur and some guidelines about what to do/not do. For a good example
of how to implement your test program, look at MockDRT in mock\_drt.pyin the sam
e directory as base.py and driver.py. It goes through all the steps described be
low and is very clear and concise. It is written in python, but your driver can
be anything that can be run by subprocess.popen and has stdout, stdin, stderr. | 139 base of the chromium src directory. |
79 | 140 |
80 ### Goals | 141 The rest of the functions can definitely be overridden for your projects |
81 Your goal for this part of the project is to create a program (or extend a progr
am) to interface with the layout test framework. The layout test framework will
communicate with this program to tell it what to do and it will accept data from
this program to perform the regression testing or create new base line files. | 142 specific needs, however these are the bare minimum needed to get it running. |
82 | 143 There are also functions you can override to make certain actions that aren’t on |
83 ### Structure | 144 by default always take place. For example, the layout test framework always |
| 145 checks for system dependencies unless you pass in a switch. If you want them |
| 146 disabled for your project, just override `check_sys_deps` to always return OK. |
| 147 This way you don’t need to pass in so many switches. |
| 148 |
| 149 As said earlier, you should override LinuxPort, MacPort, and/or WinPort. You |
| 150 should create a class that implements the platform independent overrides (such |
| 151 as `driver_class`) and then create a separate class for each platform specific |
| 152 port of your program that inherits from the class with the independent overrides |
| 153 and the platform port you want. For example, you might want to have a different |
| 154 timeout for your project, but on Windows the timeout needs to be vastly |
| 155 different than the others. In this case you can just create a default override |
| 156 that every class uses except your Windows port. In that port you can just |
| 157 override the function again to provide the specific timeout you need. This way |
| 158 you don’t need to maintain the same function on each platform if they all do the |
| 159 same thing. |
| 160 |
| 161 For `Driver` and `Port` that’s basically it unless you need to make many odd |
| 162 modifications. Lots of functionality is already there so you shouldn’t really |
| 163 need to do much. |
| 164 |
| 165 ### Part 2 |
| 166 |
| 167 This is the part where you create the program that your driver class launches. |
| 168 This part is very application dependent, so it will not be a guide on how |
| 169 implement certain features, just what should be implemented and the order in |
| 170 which events should occur and some guidelines about what to do/not do. For a |
| 171 good example of how to implement your test program, look at MockDRT in |
| 172 `mock_drt.pyin` the same directory as `base.py` and `driver.py`. It goes through |
| 173 all the steps described below and is very clear and concise. It is written in |
| 174 python, but your driver can be anything that can be run by `subprocess.popen` |
| 175 and has stdout, stdin, stderr. |
| 176 |
| 177 #### Goals |
| 178 |
| 179 Your goal for this part of the project is to create a program (or extend a |
| 180 program) to interface with the layout test framework. The layout test framework |
| 181 will communicate with this program to tell it what to do and it will accept data |
| 182 from this program to perform the regression testing or create new base line |
| 183 files. |
| 184 |
| 185 #### Structure |
| 186 |
84 This is how your code should be laid out. | 187 This is how your code should be laid out. |
85 1. Initialization | 188 |
86 * The creation of any directories or the launching of any programs should be
done here and should be done once. | 189 1. Initialization |
87 * After the program is initialized, “#READY\n” should be sent to progress th
e run\_test() in the driver. | 190 * The creation of any directories or the launching of any programs should |
88 1. Infinite Loop (!) | 191 be done here and should be done once. |
89 * After initialization, your program needs to actually wait for input, then
process that input to carry out the test. In the context of layout testing, the
content\_shell needs to wait for an html file to navigate to, render it, then co
nvert that rendering to a PNG. It does this constantly, until a signal/message i
s sent to indicate that no more tests should be processed | 192 * After the program is initialized, “#READY\n” should be sent to progress |
90 * Details: | 193 the `run_test()` in the driver. |
91 * The first thing you need is your test file path and any other additional
information about the test that is required (this is sent during the write() st
ep in run\_tests() is driver.py. This information will be passed through stdin a
nd is just one large string, with each part of the command being split with apos
trophes (ex: “/path’foo” is path to the test file, then foo is some setting that
your program might need). | 194 1. Infinite Loop (!) |
92 * After that, your program should act on this input, how it does this is d
ependent on your program, however in content\_shell, this would be the part wher
e it navigates to the test file, then renders it. After the program acts on the
input, it needs to send some text to the driver code to indicate that it has act
ed on the input. This text will indicate something that you want to test. For ex
ample, if you want to make sure you program always prints “foo” you should send
it to the driver. If the program every prints “bar” (or anything else), that wou
ld indicate a failure and the test will fail. | 195 * After initialization, your program needs to actually wait for input, |
93 * Then you need to send any image data in the same manner as you did for s
tep ii. | 196 then process that input to carry out the test. In the context of layout |
94 * Cleanup everything related to processing the input from step i, then go
back to step i. | 197 testing, the `content_shell` needs to wait for an html file to navigate |
95 * This is where the ‘infinite’ loop part comes in, your program should c
onstantly accept input from the driver until the driver indicates that there are
no more tests to run. The driver does this by closing stdin, which will cause s
td::cin to go into a bad state. However, you can also modify the driver to send
a special string such as ‘QUIT’ to exit the while loop. | 198 to, render it, then convert that rendering to a PNG. It does this |
| 199 constantly, until a signal/message is sent to indicate that no more |
| 200 tests should be processed |
| 201 * Details: |
| 202 * The first thing you need is your test file path and any other |
| 203 additional information about the test that is required (this is sent |
| 204 during the write() step in `run_tests()` is `driver.py`. This |
| 205 information will be passed through stdin and is just one large |
| 206 string, with each part of the command being split with apostrophes |
| 207 (ex: “/path’foo” is path to the test file, then foo is some setting |
| 208 that your program might need). |
| 209 * After that, your program should act on this input, how it does this |
| 210 is dependent on your program, however in `content_shell`, this would |
| 211 be the part where it navigates to the test file, then renders it. |
| 212 After the program acts on the input, it needs to send some text to |
| 213 the driver code to indicate that it has acted on the input. This |
| 214 text will indicate something that you want to test. For example, if |
| 215 you want to make sure you program always prints “foo” you should |
| 216 send it to the driver. If the program every prints “bar” (or |
| 217 anything else), that would indicate a failure and the test will |
| 218 fail. |
| 219 * Then you need to send any image data in the same manner as you did |
| 220 for step 2. |
| 221 * Cleanup everything related to processing the input from step i, then |
| 222 go back to step 1. |
| 223 * This is where the ‘infinite’ loop part comes in, your program |
| 224 should constantly accept input from the driver until the driver |
| 225 indicates that there are no more tests to run. The driver does this |
| 226 by closing stdin, which will cause std::cin to go into a bad state. |
| 227 However, you can also modify the driver to send a special string |
| 228 such as ‘QUIT’ to exit the while loop. |
96 | 229 |
97 That’s basically what the skeleton of your program should be. | 230 That’s basically what the skeleton of your program should be. |
98 | 231 |
99 ### Details: | 232 ### Details |
100 This is information about how to do some specific things, such as sending data t
o the layout test framework. | 233 |
101 * Content Blocks | 234 This is information about how to do some specific things, such as sending data |
102 * The layout test framework accepts output from your program in blocks of da
ta through stdout. Therefore, printing to stdout is really sending data to the l
ayout test framework. | 235 to the layout test framework. |
103 * Structure of block | 236 |
104 * “Header: Data\n” | 237 * Content Blocks |
105 * Header indicates what type of data will be sent through. A list of val
id headers is listed in Driver.py. | 238 * The layout test framework accepts output from your program in blocks of |
106 * Data is the data that you actually want to send. For pixel tests, you
want to send the actual PNG data here. | 239 data through stdout. Therefore, printing to stdout is really sending |
107 * The newline is needed to indicate the end of a header. | 240 data to the layout test framework. |
108 * End of a content block | 241 * Structure of block |
109 * To indicate the end of a a content block and cause the driver to progr
ess, you need to write “#EOF\n” to stdout (mandatory) and to stderr for certain
types of content, such as image data. | 242 * “Header: Data\n” |
110 * Multiple headers per block | 243 * Header indicates what type of data will be sent through. A list |
111 * Some blocks require different sets of data. For PNGs, not only is the
PNG needed, but so is a hash of the bitmap used to create the PNG. | 244 of valid headers is listed in `Driver.py`. |
112 * In this case this is how your output should look. | 245 * Data is the data that you actually want to send. For pixel |
113 * “Content-type: image/png\n” | 246 tests, you want to send the actual PNG data here. |
114 * “ActualHash: hashData\n” | 247 * The newline is needed to indicate the end of a header. |
115 * “Content-Length: lengthOfPng\n” | 248 * End of a content block |
116 * “pngdata” | 249 * To indicate the end of a a content block and cause the driver to |
117 * This part doesn’t need a header specifying that you are sending pn
g data, just send it | 250 progress, you need to write “#EOF\n” to stdout (mandatory) and |
118 * “#EOF\n” on both stdout and stderr | 251 to stderr for certain types of content, such as image data. |
119 * To see the structure of the data required, look at the read\_block fun
ctions in Driver.py | 252 * Multiple headers per block |
120 | 253 * Some blocks require different sets of data. For PNGs, not only |
121 | 254 is the PNG needed, but so is a hash of the bitmap used to create |
122 | 255 the PNG. |
123 | 256 * In this case this is how your output should look. |
124 | 257 * “Content-type: image/png\n” |
125 | 258 * “ActualHash: hashData\n” |
| 259 * “Content-Length: lengthOfPng\n” |
| 260 * “pngdata” |
| 261 * This part doesn’t need a header specifying that you are |
| 262 sending png data, just send it |
| 263 * “#EOF\n” on both stdout and stderr |
| 264 * To see the structure of the data required, look at the |
| 265 `read_block` functions in Driver.py |
OLD | NEW |