OLD | NEW |
(Empty) | |
| 1 # GYP (Generate Your Projects) Tests |
| 2 |
| 3 -- |
| 4 |
| 5 Status: Draft (as of 2009-08-18) |
| 6 |
| 7 Steven Knight <sgk@chromium.org> |
| 8 _et al._ |
| 9 |
| 10 Modified: 2009-08-18 |
| 11 |
| 12 [TOC] |
| 13 |
| 14 ## Introduction |
| 15 |
| 16 This document describes the GYP testing infrastructure, |
| 17 as provided by the `TestGyp.py` module. |
| 18 |
| 19 These tests emphasize testing the _behavior_ of the |
| 20 various GYP-generated build configurations: |
| 21 Visual Studio, Xcode, SCons, Make, etc. |
| 22 The goal is _not_ to test the output of the GYP generators by, |
| 23 for example, comparing a GYP-generated Makefile |
| 24 against a set of known "golden" Makefiles |
| 25 (although the testing infrastructure could |
| 26 be used to write those kinds of tests). |
| 27 The idea is that the generated build configuration files |
| 28 could be completely written to add a feature or fix a bug |
| 29 so long as they continue to support the functional behaviors |
| 30 defined by the tests: building programs, shared libraries, etc. |
| 31 |
| 32 ## "Hello, world!" GYP test configuration |
| 33 |
| 34 Here is an actual test configuration, |
| 35 a simple build of a C program to print `"Hello, world!"`. |
| 36 |
| 37 ``` |
| 38 $ ls -l test/hello |
| 39 total 20 |
| 40 -rw-r--r-- 1 knight knight 312 Jul 30 20:22 gyptest-all.py |
| 41 -rw-r--r-- 1 knight knight 307 Jul 30 20:22 gyptest-default.py |
| 42 -rwxr-xr-x 1 knight knight 326 Jul 30 20:22 gyptest-target.py |
| 43 -rw-r--r-- 1 knight knight 98 Jul 30 20:22 hello.c |
| 44 -rw-r--r-- 1 knight knight 142 Jul 30 20:22 hello.gyp |
| 45 $ |
| 46 ``` |
| 47 |
| 48 The `gyptest-*.py` files are three separate tests (test scripts) |
| 49 that use this configuration. The first one, `gyptest-all.py`, |
| 50 looks like this: |
| 51 |
| 52 ``` |
| 53 #!/usr/bin/env python |
| 54 |
| 55 """ |
| 56 Verifies simplest-possible build of a "Hello, world!" program |
| 57 using an explicit build target of 'all'. |
| 58 """ |
| 59 |
| 60 import TestGyp |
| 61 |
| 62 test = TestGyp.TestGyp() |
| 63 |
| 64 test.run_gyp('hello.gyp') |
| 65 |
| 66 test.build_all('hello.gyp') |
| 67 |
| 68 test.run_built_executable('hello', stdout="Hello, world!\n") |
| 69 |
| 70 test.pass_test() |
| 71 ``` |
| 72 |
| 73 The test script above runs GYP against the specified input file |
| 74 (`hello.gyp`) to generate a build configuration. |
| 75 It then tries to build the `'all'` target |
| 76 (or its equivalent) using the generated build configuration. |
| 77 Last, it verifies that the build worked as expected |
| 78 by running the executable program (`hello`) |
| 79 that was just presumably built by the generated configuration, |
| 80 and verifies that the output from the program |
| 81 matches the expected `stdout` string (`"Hello, world!\n"`). |
| 82 |
| 83 Which configuration is generated |
| 84 (i.e., which build tool to test) |
| 85 is specified when the test is run; |
| 86 see the next section. |
| 87 |
| 88 Surrounding the functional parts of the test |
| 89 described above are the header, |
| 90 which should be basically the same for each test |
| 91 (modulo a different description in the docstring): |
| 92 |
| 93 ``` |
| 94 #!/usr/bin/env python |
| 95 |
| 96 """ |
| 97 Verifies simplest-possible build of a "Hello, world!" program |
| 98 using an explicit build target of 'all'. |
| 99 """ |
| 100 |
| 101 import TestGyp |
| 102 |
| 103 test = TestGyp.TestGyp() |
| 104 ``` |
| 105 |
| 106 Similarly, the footer should be the same in every test: |
| 107 |
| 108 ``` |
| 109 test.pass_test() |
| 110 ``` |
| 111 |
| 112 ## Running tests |
| 113 |
| 114 Test scripts are run by the `gyptest.py` script. |
| 115 You can specify (an) explicit test script(s) to run: |
| 116 |
| 117 ``` |
| 118 $ python gyptest.py test/hello/gyptest-all.py |
| 119 PYTHONPATH=/home/knight/src/gyp/trunk/test/lib |
| 120 TESTGYP_FORMAT=scons |
| 121 /usr/bin/python test/hello/gyptest-all.py |
| 122 PASSED |
| 123 $ |
| 124 ``` |
| 125 |
| 126 If you specify a directory, all test scripts |
| 127 (scripts prefixed with `gyptest-`) underneath |
| 128 the directory will be run: |
| 129 |
| 130 ``` |
| 131 $ python gyptest.py test/hello |
| 132 PYTHONPATH=/home/knight/src/gyp/trunk/test/lib |
| 133 TESTGYP_FORMAT=scons |
| 134 /usr/bin/python test/hello/gyptest-all.py |
| 135 PASSED |
| 136 /usr/bin/python test/hello/gyptest-default.py |
| 137 PASSED |
| 138 /usr/bin/python test/hello/gyptest-target.py |
| 139 PASSED |
| 140 $ |
| 141 ``` |
| 142 |
| 143 Or you can specify the `-a` option to run all scripts |
| 144 in the tree: |
| 145 |
| 146 ``` |
| 147 $ python gyptest.py -a |
| 148 PYTHONPATH=/home/knight/src/gyp/trunk/test/lib |
| 149 TESTGYP_FORMAT=scons |
| 150 /usr/bin/python test/configurations/gyptest-configurations.py |
| 151 PASSED |
| 152 /usr/bin/python test/defines/gyptest-defines.py |
| 153 PASSED |
| 154 . |
| 155 . |
| 156 . |
| 157 . |
| 158 /usr/bin/python test/variables/gyptest-commands.py |
| 159 PASSED |
| 160 $ |
| 161 ``` |
| 162 |
| 163 If any tests fail during the run, |
| 164 the `gyptest.py` script will report them in a |
| 165 summary at the end. |
| 166 |
| 167 ## Debugging tests |
| 168 |
| 169 Tests that create intermediate output do so under the gyp/out/testworkarea |
| 170 directory. On test completion, intermediate output is cleaned up. To preserve |
| 171 this output, set the environment variable PRESERVE=1. This can be handy to |
| 172 inspect intermediate data when debugging a test. |
| 173 |
| 174 You can also set PRESERVE\_PASS=1, PRESERVE\_FAIL=1 or PRESERVE\_NO\_RESULT=1 |
| 175 to preserve output for tests that fall into one of those categories. |
| 176 |
| 177 # Specifying the format (build tool) to use |
| 178 |
| 179 By default, the `gyptest.py` script will generate configurations for |
| 180 the "primary" supported build tool for the platform you're on: |
| 181 Visual Studio on Windows, |
| 182 Xcode on Mac, |
| 183 and (currently) SCons on Linux. |
| 184 An alternate format (build tool) may be specified |
| 185 using the `-f` option: |
| 186 |
| 187 ``` |
| 188 $ python gyptest.py -f make test/hello/gyptest-all.py |
| 189 PYTHONPATH=/home/knight/src/gyp/trunk/test/lib |
| 190 TESTGYP_FORMAT=make |
| 191 /usr/bin/python test/hello/gyptest-all.py |
| 192 PASSED |
| 193 $ |
| 194 ``` |
| 195 |
| 196 Multiple tools may be specified in a single pass as |
| 197 a comma-separated list: |
| 198 |
| 199 ``` |
| 200 $ python gyptest.py -f make,scons test/hello/gyptest-all.py |
| 201 PYTHONPATH=/home/knight/src/gyp/trunk/test/lib |
| 202 TESTGYP_FORMAT=make |
| 203 /usr/bin/python test/hello/gyptest-all.py |
| 204 PASSED |
| 205 TESTGYP_FORMAT=scons |
| 206 /usr/bin/python test/hello/gyptest-all.py |
| 207 PASSED |
| 208 $ |
| 209 ``` |
| 210 |
| 211 ## Test script functions and methods |
| 212 |
| 213 The `TestGyp` class contains a lot of functionality |
| 214 intended to make it easy to write tests. |
| 215 This section describes the most useful pieces for GYP testing. |
| 216 |
| 217 (The `TestGyp` class is actually a subclass of more generic |
| 218 `TestCommon` and `TestCmd` base classes |
| 219 that contain even more functionality than is |
| 220 described here.) |
| 221 |
| 222 ### Initialization |
| 223 |
| 224 The standard initialization formula is: |
| 225 |
| 226 ``` |
| 227 import TestGyp |
| 228 test = TestGyp.TestGyp() |
| 229 ``` |
| 230 |
| 231 This copies the contents of the directory tree in which |
| 232 the test script lives to a temporary directory for execution, |
| 233 and arranges for the temporary directory's removal on exit. |
| 234 |
| 235 By default, any comparisons of output or file contents |
| 236 must be exact matches for the test to pass. |
| 237 If you need to use regular expressions for matches, |
| 238 a useful alternative initialization is: |
| 239 |
| 240 ``` |
| 241 import TestGyp |
| 242 test = TestGyp.TestGyp(match = TestGyp.match_re, |
| 243 diff = TestGyp.diff_re)` |
| 244 ``` |
| 245 |
| 246 ### Running GYP |
| 247 |
| 248 The canonical invocation is to simply specify the `.gyp` file to be executed: |
| 249 |
| 250 ``` |
| 251 test.run_gyp('file.gyp') |
| 252 ``` |
| 253 |
| 254 Additional GYP arguments may be specified: |
| 255 |
| 256 ``` |
| 257 test.run_gyp('file.gyp', arguments=['arg1', 'arg2', ...]) |
| 258 ``` |
| 259 |
| 260 To execute GYP from a subdirectory (where, presumably, the specified file |
| 261 lives): |
| 262 |
| 263 ``` |
| 264 test.run_gyp('file.gyp', chdir='subdir') |
| 265 ``` |
| 266 |
| 267 ### Running the build tool |
| 268 |
| 269 Running the build tool requires passing in a `.gyp` file, which may be used to |
| 270 calculate the name of a specific build configuration file (such as a MSVS |
| 271 solution file corresponding to the `.gyp` file). |
| 272 |
| 273 There are several different `.build_*()` methods for invoking different types |
| 274 of builds. |
| 275 |
| 276 To invoke a build tool with an explicit `all` target (or equivalent): |
| 277 |
| 278 ``` |
| 279 test.build_all('file.gyp') |
| 280 ``` |
| 281 |
| 282 To invoke a build tool with its default behavior (for example, executing `make` |
| 283 with no targets specified): |
| 284 |
| 285 ``` |
| 286 test.build_default('file.gyp') |
| 287 ``` |
| 288 |
| 289 To invoke a build tool with an explicit specified target: |
| 290 |
| 291 ``` |
| 292 test.build_target('file.gyp', 'target') |
| 293 ``` |
| 294 |
| 295 ### Running executables |
| 296 |
| 297 The most useful method executes a program built by the GYP-generated |
| 298 configuration: |
| 299 |
| 300 ``` |
| 301 test.run_built_executable('program') |
| 302 ``` |
| 303 |
| 304 The `.run_built_executable()` method will account for the actual built target |
| 305 output location for the build tool being tested, as well as tack on any |
| 306 necessary executable file suffix for the platform (for example `.exe` on |
| 307 Windows). |
| 308 |
| 309 `stdout=` and `stderr=` keyword arguments specify expected standard output and |
| 310 error output, respectively. Failure to match these (if specified) will cause |
| 311 the test to fail. An explicit `None` value will suppress that verification: |
| 312 |
| 313 ``` |
| 314 test.run_built_executable('program', |
| 315 stdout="expect this output\n", |
| 316 stderr=None) |
| 317 ``` |
| 318 |
| 319 Note that the default values are `stdout=None` and `stderr=''` (that is, no |
| 320 check for standard output, and error output must be empty). |
| 321 |
| 322 Arbitrary executables (not necessarily those built by GYP) can be executed with |
| 323 the lower-level `.run()` method: |
| 324 |
| 325 ``` |
| 326 test.run('program') |
| 327 ``` |
| 328 |
| 329 The program must be in the local directory (that is, the temporary directory |
| 330 for test execution) or be an absolute path name. |
| 331 |
| 332 ### Fetching command output |
| 333 |
| 334 ``` |
| 335 test.stdout() |
| 336 ``` |
| 337 |
| 338 Returns the standard output from the most recent executed command (including |
| 339 `.run_gyp()`, `.build_*()`, or `.run*()` methods). |
| 340 |
| 341 ``` |
| 342 test.stderr() |
| 343 ``` |
| 344 |
| 345 Returns the error output from the most recent executed command (including |
| 346 `.run_gyp()`, `.build_*()`, or `.run*()` methods). |
| 347 |
| 348 ### Verifying existence or non-existence of files or directories |
| 349 |
| 350 ``` |
| 351 test.must_exist('file_or_dir') |
| 352 ``` |
| 353 |
| 354 Verifies that the specified file or directory exists, and fails the test if it |
| 355 doesn't. |
| 356 |
| 357 ``` |
| 358 test.must_not_exist('file_or_dir') |
| 359 ``` |
| 360 |
| 361 Verifies that the specified file or directory does not exist, and fails the |
| 362 test if it does. |
| 363 |
| 364 ### Verifying file contents |
| 365 |
| 366 ``` |
| 367 test.must_match('file', 'expected content\n') |
| 368 ``` |
| 369 |
| 370 Verifies that the content of the specified file match the expected string, and |
| 371 fails the test if it does not. By default, the match must be exact, but |
| 372 line-by-line regular expressions may be used if the `TestGyp` object was |
| 373 initialized with `TestGyp.match_re`. |
| 374 |
| 375 ``` |
| 376 test.must_not_match('file', 'expected content\n') |
| 377 ``` |
| 378 |
| 379 Verifies that the content of the specified file does _not_ match the expected |
| 380 string, and fails the test if it does. By default, the match must be exact, |
| 381 but line-by-line regular expressions may be used if the `TestGyp` object was |
| 382 initialized with `TestGyp.match_re`. |
| 383 |
| 384 ``` |
| 385 test.must_contain('file', 'substring') |
| 386 ``` |
| 387 |
| 388 Verifies that the specified file contains the specified substring, and fails |
| 389 the test if it does not. |
| 390 |
| 391 ``` |
| 392 test.must_not_contain('file', 'substring') |
| 393 ``` |
| 394 |
| 395 Verifies that the specified file does not contain the specified substring, and |
| 396 fails the test if it does. |
| 397 |
| 398 ``` |
| 399 test.must_contain_all_lines(output, lines) |
| 400 ``` |
| 401 |
| 402 Verifies that the output string contains all of the "lines" in the specified |
| 403 list of lines. In practice, the lines can be any substring and need not be |
| 404 `\n`-terminaed lines per se. If any line is missing, the test fails. |
| 405 |
| 406 ``` |
| 407 test.must_not_contain_any_lines(output, lines) |
| 408 ``` |
| 409 |
| 410 Verifies that the output string does _not_ contain any of the "lines" in the |
| 411 specified list of lines. In practice, the lines can be any substring and need |
| 412 not be `\n`-terminaed lines per se. If any line exists in the output string, |
| 413 the test fails. |
| 414 |
| 415 ``` |
| 416 test.must_contain_any_line(output, lines) |
| 417 ``` |
| 418 |
| 419 Verifies that the output string contains at least one of the "lines" in the |
| 420 specified list of lines. In practice, the lines can be any substring and need |
| 421 not be `\n`-terminaed lines per se. If none of the specified lines is present, |
| 422 the test fails. |
| 423 |
| 424 ### Reading file contents |
| 425 |
| 426 ``` |
| 427 test.read('file') |
| 428 ``` |
| 429 |
| 430 Returns the contents of the specified file. Directory elements contained in a |
| 431 list will be joined: |
| 432 |
| 433 ``` |
| 434 test.read(['subdir', 'file']) |
| 435 ``` |
| 436 |
| 437 ### Test success or failure |
| 438 |
| 439 ``` |
| 440 test.fail_test() |
| 441 ``` |
| 442 |
| 443 Fails the test, reporting `FAILED` on standard output and exiting with an exit |
| 444 status of `1`. |
| 445 |
| 446 ``` |
| 447 test.pass_test() |
| 448 ``` |
| 449 |
| 450 Passes the test, reporting `PASSED` on standard output and exiting with an exit |
| 451 status of `0`. |
| 452 |
| 453 ``` |
| 454 test.no_result() |
| 455 ``` |
| 456 |
| 457 Indicates the test had no valid result (i.e., the conditions could not be |
| 458 tested because of an external factor like a full file system). Reports `NO |
| 459 RESULT` on standard output and exits with a status of `2`. |
OLD | NEW |