OLD | NEW |
1 Correctness Testing | 1 Correctness Testing |
2 =================== | 2 =================== |
3 | 3 |
4 Skia correctness testing is primarily served by a tool named DM. | 4 Skia correctness testing is primarily served by a tool named DM. |
5 This is a quickstart to building and running DM. | 5 This is a quickstart to building and running DM. |
6 | 6 |
7 ~~~ | 7 ~~~ |
8 $ ./gyp_skia | 8 $ ./gyp_skia |
9 $ ninja -C out/Debug dm | 9 $ ninja -C out/Debug dm |
10 $ out/Debug/dm -v -w dm_output | 10 $ out/Debug/dm -v -w dm_output |
(...skipping 117 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... |
128 dm.json is used by our automated testing system, so you can ignore it if you | 128 dm.json is used by our automated testing system, so you can ignore it if you |
129 like. It contains a listing of each test run and a checksum of the image | 129 like. It contains a listing of each test run and a checksum of the image |
130 generated for that run. (Boring technical detail: it is not a checksum of the | 130 generated for that run. (Boring technical detail: it is not a checksum of the |
131 .png file, but rather a checksum of the raw pixels used to create that .png.) | 131 .png file, but rather a checksum of the raw pixels used to create that .png.) |
132 | 132 |
133 Unit tests don't generally output anything but a status update when they pass. | 133 Unit tests don't generally output anything but a status update when they pass. |
134 If a test fails, DM will print out its assertion failures, both at the time | 134 If a test fails, DM will print out its assertion failures, both at the time |
135 they happen and then again all together after everything is done running. | 135 they happen and then again all together after everything is done running. |
136 These failures are also included in the dm.json file. | 136 These failures are also included in the dm.json file. |
137 | 137 |
138 DM does not currently have a built-in image comparison facility, though we | 138 DM has a simple facility to compare against the results of a previous run: |
139 might add one. For now, you can use the skdiff tool to compare corresponding | |
140 images from two runs of DM. | |
141 ~~~ | 139 ~~~ |
142 $ ./gyp_skia | 140 $ ./gyp_skia |
143 $ ninja -C out/Debug dm | 141 $ ninja -C out/Debug dm |
| 142 $ out/Debug/dm -w good |
| 143 |
| 144 (do some work) |
| 145 |
| 146 $ ./gyp_skia |
| 147 $ ninja -C out/Debug dm |
| 148 $ out/Debug/dm -r good -w bad |
| 149 ~~~ |
| 150 When using `-r`, DM will display a failure for any test that didn't produce the |
| 151 same image as the `good` run. |
| 152 |
| 153 For anything fancier, I suggest using skdiff: |
| 154 ~~~ |
| 155 $ ./gyp_skia |
| 156 $ ninja -C out/Debug dm |
144 $ out/Debug/dm -w good | 157 $ out/Debug/dm -w good |
145 | 158 |
146 (do some work) | 159 (do some work) |
147 | 160 |
148 $ ./gyp_skia | 161 $ ./gyp_skia |
149 $ ninja -C out/Debug dm | 162 $ ninja -C out/Debug dm |
150 $ out/Debug/dm -w bad | 163 $ out/Debug/dm -w bad |
151 | 164 |
152 $ ninja -C out/Debug skdiff | 165 $ ninja -C out/Debug skdiff |
153 $ mkdir diff | 166 $ mkdir diff |
154 $ out/Debug/skdiff good bad diff | 167 $ out/Debug/skdiff good bad diff |
155 | 168 |
156 (open diff/index.html in your web browser) | 169 (open diff/index.html in your web browser) |
157 ~~~ | 170 ~~~ |
158 | 171 |
159 That's the basics of DM. DM supports many other modes and flags. Here are a | 172 That's the basics of DM. DM supports many other modes and flags. Here are a |
160 few examples you might find handy. | 173 few examples you might find handy. |
161 ~~~ | 174 ~~~ |
162 $ out/Debug/dm --help # Print all flags, their defaults, and a brief expl
anation of each. | 175 $ out/Debug/dm --help # Print all flags, their defaults, and a brief expl
anation of each. |
163 $ out/Debug/dm --src tests # Run only unit tests. | 176 $ out/Debug/dm --src tests # Run only unit tests. |
164 $ out/Debug/dm --nocpu # Test only GPU-backed work. | 177 $ out/Debug/dm --nocpu # Test only GPU-backed work. |
165 $ out/Debug/dm --nogpu # Test only CPU-backed work. | 178 $ out/Debug/dm --nogpu # Test only CPU-backed work. |
166 $ out/Debug/dm --match blur # Run only work with "blur" in its name. | 179 $ out/Debug/dm --match blur # Run only work with "blur" in its name. |
167 $ out/Debug/dm --dryRun # Don't really do anything, just print out what we'
d do. | 180 $ out/Debug/dm --dryRun # Don't really do anything, just print out what we'
d do. |
168 ~~~ | 181 ~~~ |
OLD | NEW |