Chromium Code Reviews
chromiumcodereview-hr@appspot.gserviceaccount.com (chromiumcodereview-hr) | Please choose your nickname with Settings | Help | Chromium Project | Gerrit Changes | Sign out
(26)

Side by Side Diff: docs/how_to_extend_layout_test_framework.md

Issue 1324603002: [Docs] Another round of stylistic fixes. (Closed) Base URL: https://chromium.googlesource.com/chromium/src.git@master
Patch Set: Created 5 years, 3 months ago
Use n/p to move between diff chunks; N/P to move between comments. Draft comments are only viewable by you.
Jump to:
View unified diff | Download patch
« no previous file with comments | « docs/gtk_vs_views_gtk.md ('k') | docs/include_what_you_use.md » ('j') | no next file with comments »
Toggle Intra-line Diffs ('i') | Expand Comments ('e') | Collapse Comments ('c') | Show Comments Hide Comments ('s')
OLDNEW
1 # Extending the Layout Framework 1 # How to Extend the Layout Test Framework
2 2
3 # Introduction 3 The Layout Test Framework that Blink uses is a regression testing tool that is
4 The Layout Test Framework that Blink uses is a regression testing tool that is m ulti-platform and it has a large amount of tools that help test varying types of regression, such as pixel diffs, text diffs, etc. The framework is mainly used by Blink, however it was made to be extensible so that other projects can use it test different parts of chrome (such as Print Preview). This is a guide to help people who want to actually the framework to test whatever they want. 4 multi-platform and it has a large amount of tools that help test varying types
5 5 of regression, such as pixel diffs, text diffs, etc. The framework is mainly
6 # Background 6 used by Blink, however it was made to be extensible so that other projects can
7 Before you can start actually extending the framework, you should be familiar wi th how to use it. This wiki is basically all you need to learn how to use it 7 use it test different parts of chrome (such as Print Preview). This is a guide
8 to help people who want to actually the framework to test whatever they want.
9
10 [TOC]
11
12 ## Background
13
14 Before you can start actually extending the framework, you should be familiar
15 with how to use it. This wiki is basically all you need to learn how to use it
8 http://www.chromium.org/developers/testing/webkit-layout-tests 16 http://www.chromium.org/developers/testing/webkit-layout-tests
9 17
10 # How to Extend the Framework 18 ## How to Extend the Framework
19
11 There are two parts to actually extending framework to test a piece of software. 20 There are two parts to actually extending framework to test a piece of software.
12 The first part is extending certain files in: 21 The first part is extending certain files in:
13 src/third\_party/Webkit/Tools/Scripts/webkitpy/layout\_tests/ 22 [/third_party/Webkit/Tools/Scripts/webkitpy/layout_tests/](/third_party/Webkit/T ools/Scripts/webkitpy/layout_tests/)
14 The code in webkitpy/layout\_tests is the layout test framework itself 23 The code in `webkitpy/layout_tests` is the layout test framework itself
15 24
16 The second part is creating a driver (program) to actually communicate the layou t test framework. This part is significantly more tricky and dependant on what e xactly exactly is being tested. 25 The second part is creating a driver (program) to actually communicate the
17 26 layout test framework. This part is significantly more tricky and dependant on
18 ## Part 1: 27 what exactly exactly is being tested.
19 This part isn’t too difficult. There are basically two classes that need to be e xtended (ideally, just inherited from). These classes are: 28
20 Driver 29 ### Part 1
21 Located in layout\_tests/port/driver.py 30
22 Each instance of this is the class that will actually an instance of the program that produces the test data (program in Part 2). 31 This part isn’t too difficult. There are basically two classes that need to be
23 Port 32 extended (ideally, just inherited from). These classes are:
24 Located in layout\_tests/port/base.py 33
25 This class is responsible creating drivers with the correct settings, giving acc ess to certain OS functionality to access expected files, etc. 34 Driver
26 35
27 36 Located in `layout_tests/port/driver.py`. Each instance of this is the class
28 37 that will actually an instance of the program that produces the test data
29 38 (program in Part 2).
30 39
31 ### Extending Driver: 40 Port
32 As said, Driver launches the program from Part 2. Said program will communicate with the driver class to receive instructions and send back data. All of the wor k for driver gets done in Driver.run\_test. Everything else is a helper or initi alization function. 41
33 run\_test() steps: 42 Located in `layout_tests/port/base.py`. This class is responsible creating
34 1. On the very first call of this function, it will actually run the test prog ram. On every subsequent call to this function, at the beginning it will verify that the process doesn’t need to be restarted, and if it does, it will create a new instance of the test program. 43 drivers with the correct settings, giving access to certain OS functionality to
35 1. It will then create a command to send the program 44 access expected files, etc.
36 * This command generally consists of an html file path for the test program to navigate to. 45
37 * After creating it, the command is sent 46 #### Extending Driver
38 1. After the command has been sent, it will then wait for data from the progra m. 47
39 * It will actually wait for 2 blocks of data. 48 As said, Driver launches the program from Part 2. Said program will communicate
40 * The first part being text or audio data. This part is required (the prog ram will always send something, even an empty string) 49 with the driver class to receive instructions and send back data. All of the
41 * The second block is optional and is image data and an image hash (md5) t his block of data is used for pixel tests 50 work for driver gets done in `Driver.run_test`. Everything else is a helper or
42 1. After it has received all the data, it will proceed to check if the program has timed out or crashed, and if so fail this instance of the test (it can be r etried later if need be). 51 initialization function.
43 52
44 Luckily, run\_test() most likely doesn’t need to be overridden unless extra bloc ks of data need to be sent to/read from the test program. However, you do need t o know how it works because it will influence what functions you need to overrid e. Here are the ones you’re probably going to need to override 53 `run_test()` steps:
45 cmd\_line 54
46 This function creates a set of command line arguments to run the test program, s o the function will almost certainly need to be overridden. 55 1. On the very first call of this function, it will actually run the test
47 It creates the command line to run the program. Driver uses subprocess.popen to create the process, which takes the name of the test program and any options it might need. 56 program. On every subsequent call to this function, at the beginning it will
48 The first item in the list of arguments should be the path to test program using this function: 57 verify that the process doesn’t need to be restarted, and if it does, it
49 self._port._path\_to\_driver() 58 will create a new instance of the test program.
50 This is an absolute path to the test program. 59 1. It will then create a command to send the program
51 This is the bare minimum you need to get the driver to launch the test program, however if you have options you need to append, just append them to the list. 60 * This command generally consists of an html file path for the test
52 start 61 program to navigate to.
53 If your program has any special startup needs, then this will be the place to pu t it. 62 * After creating it, the command is sent
54 63 1. After the command has been sent, it will then wait for data from the
55 That’s mostly it. The Driver class has almost all the functionality you could wa nt, so there isn’t much to override here. If extra data needs to be read or sent , extra data members should be added to ContentBlock. 64 program.
56 65 * It will actually wait for 2 blocks of data.
57 ### Extending Port: 66 * The first part being text or audio data. This part is required (the
58 This class is responsible for providing functionality such as where to look for tests, where to store test results, what driver to run, what timeout to use, wha t kind of files can be run, etc. It provides a lot of functionality, however it isn’t really sufficient because it doesn’t account of platform specific problems , therefore port itself shouldn’t be extend. Instead LinuxPort, WinPort, and Mac Port (and maybe the android port class) should be extended as they provide platf orm specific overrides/extensions that implement most of the important functiona lity. While there are many functions in Port, overriding one function will affec t most of the other ones to get the desired behavior. For example, if layout\_te sts\_dir() is overriden, not only will the code look for tests in that directory , but it will find the correct TestExpectations file, the platform specific expe cted files, etc. 67 program will always send something, even an empty string)
68 * The second block is optional and is image data and an image hash
69 (md5) this block of data is used for pixel tests
70 1. After it has received all the data, it will proceed to check if the program
71 has timed out or crashed, and if so fail this instance of the test (it can
72 be retried later if need be).
73
74 Luckily, `run_test()` most likely doesn’t need to be overridden unless extra
75 blocks of data need to be sent to/read from the test program. However, you do
76 need to know how it works because it will influence what functions you need to
77 override. Here are the ones you’re probably going to need to override
78
79 cmd_line
80
81 This function creates a set of command line arguments to run the test program,
82 so the function will almost certainly need to be overridden.
83
84 It creates the command line to run the program. `Driver` uses `subprocess.popen`
85 to create the process, which takes the name of the test program and any options
86 it might need.
87
88 The first item in the list of arguments should be the path to test program using
89 this function:
90
91 self._port._path_to_driver()
92
93 This is an absolute path to the test program. This is the bare minimum you need
94 to get the driver to launch the test program, however if you have options you
95 need to append, just append them to the list.
96
97 start
98
99 If your program has any special startup needs, then this will be the place to
100 put it.
101
102 That’s mostly it. The Driver class has almost all the functionality you could
103 want, so there isn’t much to override here. If extra data needs to be read or
104 sent, extra data members should be added to `ContentBlock`.
105
106 #### Extending Port
107
108 This class is responsible for providing functionality such as where to look for
109 tests, where to store test results, what driver to run, what timeout to use,
110 what kind of files can be run, etc. It provides a lot of functionality, however
111 it isn’t really sufficient because it doesn’t account of platform specific
112 problems, therefore port itself shouldn’t be extend. Instead LinuxPort, WinPort,
113 and MacPort (and maybe the android port class) should be extended as they
114 provide platform specific overrides/extensions that implement most of the
115 important functionality. While there are many functions in Port, overriding one
116 function will affect most of the other ones to get the desired behavior. For
117 example, if `layout_tests_dir()` is overriden, not only will the code look for
118 tests in that directory, but it will find the correct TestExpectations file, the
119 platform specific expected files, etc.
59 120
60 Here are some of the functions that most likely need to be overridden. 121 Here are some of the functions that most likely need to be overridden.
61 * driver\_class 122 * `driver_class`
62 * This should be overridden to allow the testing program to actually run. By default the code will run content\_shell, which might or might not be what you want. 123 * This should be overridden to allow the testing program to actually run.
63 * It should be overridden to return the driver extension class created earli er.This function doesn’t return an instance on the driver, just the class itself . 124 By default the code will run content_shell, which might or might not be
64 * driver\_name 125 what you want.
65 * This should return the name of the program test p. By default it returns ‘ content\_shell’, but you want to have it return the program you want to run, suc h as chrome or browser\_tests. 126 * It should be overridden to return the driver extension class created
66 * layout\_tests\_dir 127 earlier. This function doesn’t return an instance on the driver, just
67 * This tells the port where to look for all the and everything associated wi th them such as resources files. 128 the class itself.
68 * By default it returns absolute path to the webkit tests. 129 * `driver_name`
69 * If you are planning on running something in the chromium src/ directory, t here are helper functions to allow you to return a path relative to the base of the chromium src directory. 130 * This should return the name of the program test p. By default it returns
70 131 ‘content_shell’, but you want to have it return the program you want to
71 The rest of the functions can definitely be overridden for your projects specifi c needs, however these are the bare minimum needed to get it running. There are also functions you can override to make certain actions that aren’t on by defaul t always take place. For example, the layout test framework always checks for sy stem dependencies unless you pass in a switch. If you want them disabled for you r project, just override check\_sys\_deps to always return OK. This way you don’ t need to pass in so many switches. 132 run, such as `chrome` or `browser_tests`.
72 133 * `layout_tests_dir`
73 As said earlier, you should override LinuxPort, MacPort, and/or WinPort. You sho uld create a class that implements the platform independent overrides (such as d river\_class) and then create a separate class for each platform specific port o f your program that inherits from the class with the independent overrides and t he platform port you want. For example, you might want to have a different timeo ut for your project, but on Windows the timeout needs to be vastly different tha n the others. In this case you can just create a default override that every cla ss uses except your Windows port. In that port you can just override the functio n again to provide the specific timeout you need. This way you don’t need to mai ntain the same function on each platform if they all do the same thing. 134 * This tells the port where to look for all the and everything associated
74 135 with them such as resources files.
75 For Driver and Port that’s basically it unless you need to make many odd modific ations. Lots of functionality is already there so you shouldn’t really need to d o much. 136 * By default it returns absolute path to the webkit tests.
76 137 * If you are planning on running something in the chromium src/ directory,
77 ## Part 2: 138 there are helper functions to allow you to return a path relative to the
78 This is the part where you create the program that your driver class launches. T his part is very application dependent, so it will not be a guide on how impleme nt certain features, just what should be implemented and the order in which even ts should occur and some guidelines about what to do/not do. For a good example of how to implement your test program, look at MockDRT in mock\_drt.pyin the sam e directory as base.py and driver.py. It goes through all the steps described be low and is very clear and concise. It is written in python, but your driver can be anything that can be run by subprocess.popen and has stdout, stdin, stderr. 139 base of the chromium src directory.
79 140
80 ### Goals 141 The rest of the functions can definitely be overridden for your projects
81 Your goal for this part of the project is to create a program (or extend a progr am) to interface with the layout test framework. The layout test framework will communicate with this program to tell it what to do and it will accept data from this program to perform the regression testing or create new base line files. 142 specific needs, however these are the bare minimum needed to get it running.
82 143 There are also functions you can override to make certain actions that aren’t on
83 ### Structure 144 by default always take place. For example, the layout test framework always
145 checks for system dependencies unless you pass in a switch. If you want them
146 disabled for your project, just override `check_sys_deps` to always return OK.
147 This way you don’t need to pass in so many switches.
148
149 As said earlier, you should override LinuxPort, MacPort, and/or WinPort. You
150 should create a class that implements the platform independent overrides (such
151 as `driver_class`) and then create a separate class for each platform specific
152 port of your program that inherits from the class with the independent overrides
153 and the platform port you want. For example, you might want to have a different
154 timeout for your project, but on Windows the timeout needs to be vastly
155 different than the others. In this case you can just create a default override
156 that every class uses except your Windows port. In that port you can just
157 override the function again to provide the specific timeout you need. This way
158 you don’t need to maintain the same function on each platform if they all do the
159 same thing.
160
161 For `Driver` and `Port` that’s basically it unless you need to make many odd
162 modifications. Lots of functionality is already there so you shouldn’t really
163 need to do much.
164
165 ### Part 2
166
167 This is the part where you create the program that your driver class launches.
168 This part is very application dependent, so it will not be a guide on how
169 implement certain features, just what should be implemented and the order in
170 which events should occur and some guidelines about what to do/not do. For a
171 good example of how to implement your test program, look at MockDRT in
172 `mock_drt.pyin` the same directory as `base.py` and `driver.py`. It goes through
173 all the steps described below and is very clear and concise. It is written in
174 python, but your driver can be anything that can be run by `subprocess.popen`
175 and has stdout, stdin, stderr.
176
177 #### Goals
178
179 Your goal for this part of the project is to create a program (or extend a
180 program) to interface with the layout test framework. The layout test framework
181 will communicate with this program to tell it what to do and it will accept data
182 from this program to perform the regression testing or create new base line
183 files.
184
185 #### Structure
186
84 This is how your code should be laid out. 187 This is how your code should be laid out.
85 1. Initialization 188
86 * The creation of any directories or the launching of any programs should be done here and should be done once. 189 1. Initialization
87 * After the program is initialized, “#READY\n” should be sent to progress th e run\_test() in the driver. 190 * The creation of any directories or the launching of any programs should
88 1. Infinite Loop (!) 191 be done here and should be done once.
89 * After initialization, your program needs to actually wait for input, then process that input to carry out the test. In the context of layout testing, the content\_shell needs to wait for an html file to navigate to, render it, then co nvert that rendering to a PNG. It does this constantly, until a signal/message i s sent to indicate that no more tests should be processed 192 * After the program is initialized, “#READY\n” should be sent to progress
90 * Details: 193 the `run_test()` in the driver.
91 * The first thing you need is your test file path and any other additional information about the test that is required (this is sent during the write() st ep in run\_tests() is driver.py. This information will be passed through stdin a nd is just one large string, with each part of the command being split with apos trophes (ex: “/path’foo” is path to the test file, then foo is some setting that your program might need). 194 1. Infinite Loop (!)
92 * After that, your program should act on this input, how it does this is d ependent on your program, however in content\_shell, this would be the part wher e it navigates to the test file, then renders it. After the program acts on the input, it needs to send some text to the driver code to indicate that it has act ed on the input. This text will indicate something that you want to test. For ex ample, if you want to make sure you program always prints “foo” you should send it to the driver. If the program every prints “bar” (or anything else), that wou ld indicate a failure and the test will fail. 195 * After initialization, your program needs to actually wait for input,
93 * Then you need to send any image data in the same manner as you did for s tep ii. 196 then process that input to carry out the test. In the context of layout
94 * Cleanup everything related to processing the input from step i, then go back to step i. 197 testing, the `content_shell` needs to wait for an html file to navigate
95 * This is where the ‘infinite’ loop part comes in, your program should c onstantly accept input from the driver until the driver indicates that there are no more tests to run. The driver does this by closing stdin, which will cause s td::cin to go into a bad state. However, you can also modify the driver to send a special string such as ‘QUIT’ to exit the while loop. 198 to, render it, then convert that rendering to a PNG. It does this
199 constantly, until a signal/message is sent to indicate that no more
200 tests should be processed
201 * Details:
202 * The first thing you need is your test file path and any other
203 additional information about the test that is required (this is sent
204 during the write() step in `run_tests()` is `driver.py`. This
205 information will be passed through stdin and is just one large
206 string, with each part of the command being split with apostrophes
207 (ex: “/path’foo” is path to the test file, then foo is some setting
208 that your program might need).
209 * After that, your program should act on this input, how it does this
210 is dependent on your program, however in `content_shell`, this would
211 be the part where it navigates to the test file, then renders it.
212 After the program acts on the input, it needs to send some text to
213 the driver code to indicate that it has acted on the input. This
214 text will indicate something that you want to test. For example, if
215 you want to make sure you program always prints “foo” you should
216 send it to the driver. If the program every prints “bar” (or
217 anything else), that would indicate a failure and the test will
218 fail.
219 * Then you need to send any image data in the same manner as you did
220 for step 2.
221 * Cleanup everything related to processing the input from step i, then
222 go back to step 1.
223 * This is where the ‘infinite’ loop part comes in, your program
224 should constantly accept input from the driver until the driver
225 indicates that there are no more tests to run. The driver does this
226 by closing stdin, which will cause std::cin to go into a bad state.
227 However, you can also modify the driver to send a special string
228 such as ‘QUIT’ to exit the while loop.
96 229
97 That’s basically what the skeleton of your program should be. 230 That’s basically what the skeleton of your program should be.
98 231
99 ### Details: 232 ### Details
100 This is information about how to do some specific things, such as sending data t o the layout test framework. 233
101 * Content Blocks 234 This is information about how to do some specific things, such as sending data
102 * The layout test framework accepts output from your program in blocks of da ta through stdout. Therefore, printing to stdout is really sending data to the l ayout test framework. 235 to the layout test framework.
103 * Structure of block 236
104 * “Header: Data\n” 237 * Content Blocks
105 * Header indicates what type of data will be sent through. A list of val id headers is listed in Driver.py. 238 * The layout test framework accepts output from your program in blocks of
106 * Data is the data that you actually want to send. For pixel tests, you want to send the actual PNG data here. 239 data through stdout. Therefore, printing to stdout is really sending
107 * The newline is needed to indicate the end of a header. 240 data to the layout test framework.
108 * End of a content block 241 * Structure of block
109 * To indicate the end of a a content block and cause the driver to progr ess, you need to write “#EOF\n” to stdout (mandatory) and to stderr for certain types of content, such as image data. 242 * “Header: Data\n”
110 * Multiple headers per block 243 * Header indicates what type of data will be sent through. A list
111 * Some blocks require different sets of data. For PNGs, not only is the PNG needed, but so is a hash of the bitmap used to create the PNG. 244 of valid headers is listed in `Driver.py`.
112 * In this case this is how your output should look. 245 * Data is the data that you actually want to send. For pixel
113 * “Content-type: image/png\n” 246 tests, you want to send the actual PNG data here.
114 * “ActualHash: hashData\n” 247 * The newline is needed to indicate the end of a header.
115 * “Content-Length: lengthOfPng\n” 248 * End of a content block
116 * “pngdata” 249 * To indicate the end of a a content block and cause the driver to
117 * This part doesn’t need a header specifying that you are sending pn g data, just send it 250 progress, you need to write “#EOF\n” to stdout (mandatory) and
118 * “#EOF\n” on both stdout and stderr 251 to stderr for certain types of content, such as image data.
119 * To see the structure of the data required, look at the read\_block fun ctions in Driver.py 252 * Multiple headers per block
120 253 * Some blocks require different sets of data. For PNGs, not only
121 254 is the PNG needed, but so is a hash of the bitmap used to create
122 255 the PNG.
123 256 * In this case this is how your output should look.
124 257 * “Content-type: image/png\n”
125 258 * “ActualHash: hashData\n”
259 * “Content-Length: lengthOfPng\n”
260 * “pngdata”
261 * This part doesn’t need a header specifying that you are
262 sending png data, just send it
263 * “#EOF\n” on both stdout and stderr
264 * To see the structure of the data required, look at the
265 `read_block` functions in Driver.py
OLDNEW
« no previous file with comments | « docs/gtk_vs_views_gtk.md ('k') | docs/include_what_you_use.md » ('j') | no next file with comments »

Powered by Google App Engine
This is Rietveld 408576698