| OLD | NEW |
|---|
| (Empty) |
| 1 # GYP (Generate Your Projects) Tests | |
| 2 | |
| 3 -- | |
| 4 | |
| 5 Status: Draft (as of 2009-08-18) | |
| 6 | |
| 7 Steven Knight <sgk@chromium.org> | |
| 8 _et al._ | |
| 9 | |
| 10 Modified: 2009-08-18 | |
| 11 | |
| 12 [TOC] | |
| 13 | |
| 14 ## Introduction | |
| 15 | |
| 16 This document describes the GYP testing infrastructure, | |
| 17 as provided by the `TestGyp.py` module. | |
| 18 | |
| 19 These tests emphasize testing the _behavior_ of the | |
| 20 various GYP-generated build configurations: | |
| 21 Visual Studio, Xcode, SCons, Make, etc. | |
| 22 The goal is _not_ to test the output of the GYP generators by, | |
| 23 for example, comparing a GYP-generated Makefile | |
| 24 against a set of known "golden" Makefiles | |
| 25 (although the testing infrastructure could | |
| 26 be used to write those kinds of tests). | |
| 27 The idea is that the generated build configuration files | |
| 28 could be completely written to add a feature or fix a bug | |
| 29 so long as they continue to support the functional behaviors | |
| 30 defined by the tests: building programs, shared libraries, etc. | |
| 31 | |
| 32 ## "Hello, world!" GYP test configuration | |
| 33 | |
| 34 Here is an actual test configuration, | |
| 35 a simple build of a C program to print `"Hello, world!"`. | |
| 36 | |
| 37 ``` | |
| 38 $ ls -l test/hello | |
| 39 total 20 | |
| 40 -rw-r--r-- 1 knight knight 312 Jul 30 20:22 gyptest-all.py | |
| 41 -rw-r--r-- 1 knight knight 307 Jul 30 20:22 gyptest-default.py | |
| 42 -rwxr-xr-x 1 knight knight 326 Jul 30 20:22 gyptest-target.py | |
| 43 -rw-r--r-- 1 knight knight 98 Jul 30 20:22 hello.c | |
| 44 -rw-r--r-- 1 knight knight 142 Jul 30 20:22 hello.gyp | |
| 45 $ | |
| 46 ``` | |
| 47 | |
| 48 The `gyptest-*.py` files are three separate tests (test scripts) | |
| 49 that use this configuration. The first one, `gyptest-all.py`, | |
| 50 looks like this: | |
| 51 | |
| 52 ``` | |
| 53 #!/usr/bin/env python | |
| 54 | |
| 55 """ | |
| 56 Verifies simplest-possible build of a "Hello, world!" program | |
| 57 using an explicit build target of 'all'. | |
| 58 """ | |
| 59 | |
| 60 import TestGyp | |
| 61 | |
| 62 test = TestGyp.TestGyp() | |
| 63 | |
| 64 test.run_gyp('hello.gyp') | |
| 65 | |
| 66 test.build_all('hello.gyp') | |
| 67 | |
| 68 test.run_built_executable('hello', stdout="Hello, world!\n") | |
| 69 | |
| 70 test.pass_test() | |
| 71 ``` | |
| 72 | |
| 73 The test script above runs GYP against the specified input file | |
| 74 (`hello.gyp`) to generate a build configuration. | |
| 75 It then tries to build the `'all'` target | |
| 76 (or its equivalent) using the generated build configuration. | |
| 77 Last, it verifies that the build worked as expected | |
| 78 by running the executable program (`hello`) | |
| 79 that was just presumably built by the generated configuration, | |
| 80 and verifies that the output from the program | |
| 81 matches the expected `stdout` string (`"Hello, world!\n"`). | |
| 82 | |
| 83 Which configuration is generated | |
| 84 (i.e., which build tool to test) | |
| 85 is specified when the test is run; | |
| 86 see the next section. | |
| 87 | |
| 88 Surrounding the functional parts of the test | |
| 89 described above are the header, | |
| 90 which should be basically the same for each test | |
| 91 (modulo a different description in the docstring): | |
| 92 | |
| 93 ``` | |
| 94 #!/usr/bin/env python | |
| 95 | |
| 96 """ | |
| 97 Verifies simplest-possible build of a "Hello, world!" program | |
| 98 using an explicit build target of 'all'. | |
| 99 """ | |
| 100 | |
| 101 import TestGyp | |
| 102 | |
| 103 test = TestGyp.TestGyp() | |
| 104 ``` | |
| 105 | |
| 106 Similarly, the footer should be the same in every test: | |
| 107 | |
| 108 ``` | |
| 109 test.pass_test() | |
| 110 ``` | |
| 111 | |
| 112 ## Running tests | |
| 113 | |
| 114 Test scripts are run by the `gyptest.py` script. | |
| 115 You can specify (an) explicit test script(s) to run: | |
| 116 | |
| 117 ``` | |
| 118 $ python gyptest.py test/hello/gyptest-all.py | |
| 119 PYTHONPATH=/home/knight/src/gyp/trunk/test/lib | |
| 120 TESTGYP_FORMAT=scons | |
| 121 /usr/bin/python test/hello/gyptest-all.py | |
| 122 PASSED | |
| 123 $ | |
| 124 ``` | |
| 125 | |
| 126 If you specify a directory, all test scripts | |
| 127 (scripts prefixed with `gyptest-`) underneath | |
| 128 the directory will be run: | |
| 129 | |
| 130 ``` | |
| 131 $ python gyptest.py test/hello | |
| 132 PYTHONPATH=/home/knight/src/gyp/trunk/test/lib | |
| 133 TESTGYP_FORMAT=scons | |
| 134 /usr/bin/python test/hello/gyptest-all.py | |
| 135 PASSED | |
| 136 /usr/bin/python test/hello/gyptest-default.py | |
| 137 PASSED | |
| 138 /usr/bin/python test/hello/gyptest-target.py | |
| 139 PASSED | |
| 140 $ | |
| 141 ``` | |
| 142 | |
| 143 Or you can specify the `-a` option to run all scripts | |
| 144 in the tree: | |
| 145 | |
| 146 ``` | |
| 147 $ python gyptest.py -a | |
| 148 PYTHONPATH=/home/knight/src/gyp/trunk/test/lib | |
| 149 TESTGYP_FORMAT=scons | |
| 150 /usr/bin/python test/configurations/gyptest-configurations.py | |
| 151 PASSED | |
| 152 /usr/bin/python test/defines/gyptest-defines.py | |
| 153 PASSED | |
| 154 . | |
| 155 . | |
| 156 . | |
| 157 . | |
| 158 /usr/bin/python test/variables/gyptest-commands.py | |
| 159 PASSED | |
| 160 $ | |
| 161 ``` | |
| 162 | |
| 163 If any tests fail during the run, | |
| 164 the `gyptest.py` script will report them in a | |
| 165 summary at the end. | |
| 166 | |
| 167 ## Debugging tests | |
| 168 | |
| 169 Tests that create intermediate output do so under the gyp/out/testworkarea | |
| 170 directory. On test completion, intermediate output is cleaned up. To preserve | |
| 171 this output, set the environment variable PRESERVE=1. This can be handy to | |
| 172 inspect intermediate data when debugging a test. | |
| 173 | |
| 174 You can also set PRESERVE\_PASS=1, PRESERVE\_FAIL=1 or PRESERVE\_NO\_RESULT=1 | |
| 175 to preserve output for tests that fall into one of those categories. | |
| 176 | |
| 177 # Specifying the format (build tool) to use | |
| 178 | |
| 179 By default, the `gyptest.py` script will generate configurations for | |
| 180 the "primary" supported build tool for the platform you're on: | |
| 181 Visual Studio on Windows, | |
| 182 Xcode on Mac, | |
| 183 and (currently) SCons on Linux. | |
| 184 An alternate format (build tool) may be specified | |
| 185 using the `-f` option: | |
| 186 | |
| 187 ``` | |
| 188 $ python gyptest.py -f make test/hello/gyptest-all.py | |
| 189 PYTHONPATH=/home/knight/src/gyp/trunk/test/lib | |
| 190 TESTGYP_FORMAT=make | |
| 191 /usr/bin/python test/hello/gyptest-all.py | |
| 192 PASSED | |
| 193 $ | |
| 194 ``` | |
| 195 | |
| 196 Multiple tools may be specified in a single pass as | |
| 197 a comma-separated list: | |
| 198 | |
| 199 ``` | |
| 200 $ python gyptest.py -f make,scons test/hello/gyptest-all.py | |
| 201 PYTHONPATH=/home/knight/src/gyp/trunk/test/lib | |
| 202 TESTGYP_FORMAT=make | |
| 203 /usr/bin/python test/hello/gyptest-all.py | |
| 204 PASSED | |
| 205 TESTGYP_FORMAT=scons | |
| 206 /usr/bin/python test/hello/gyptest-all.py | |
| 207 PASSED | |
| 208 $ | |
| 209 ``` | |
| 210 | |
| 211 ## Test script functions and methods | |
| 212 | |
| 213 The `TestGyp` class contains a lot of functionality | |
| 214 intended to make it easy to write tests. | |
| 215 This section describes the most useful pieces for GYP testing. | |
| 216 | |
| 217 (The `TestGyp` class is actually a subclass of more generic | |
| 218 `TestCommon` and `TestCmd` base classes | |
| 219 that contain even more functionality than is | |
| 220 described here.) | |
| 221 | |
| 222 ### Initialization | |
| 223 | |
| 224 The standard initialization formula is: | |
| 225 | |
| 226 ``` | |
| 227 import TestGyp | |
| 228 test = TestGyp.TestGyp() | |
| 229 ``` | |
| 230 | |
| 231 This copies the contents of the directory tree in which | |
| 232 the test script lives to a temporary directory for execution, | |
| 233 and arranges for the temporary directory's removal on exit. | |
| 234 | |
| 235 By default, any comparisons of output or file contents | |
| 236 must be exact matches for the test to pass. | |
| 237 If you need to use regular expressions for matches, | |
| 238 a useful alternative initialization is: | |
| 239 | |
| 240 ``` | |
| 241 import TestGyp | |
| 242 test = TestGyp.TestGyp(match = TestGyp.match_re, | |
| 243 diff = TestGyp.diff_re)` | |
| 244 ``` | |
| 245 | |
| 246 ### Running GYP | |
| 247 | |
| 248 The canonical invocation is to simply specify the `.gyp` file to be executed: | |
| 249 | |
| 250 ``` | |
| 251 test.run_gyp('file.gyp') | |
| 252 ``` | |
| 253 | |
| 254 Additional GYP arguments may be specified: | |
| 255 | |
| 256 ``` | |
| 257 test.run_gyp('file.gyp', arguments=['arg1', 'arg2', ...]) | |
| 258 ``` | |
| 259 | |
| 260 To execute GYP from a subdirectory (where, presumably, the specified file | |
| 261 lives): | |
| 262 | |
| 263 ``` | |
| 264 test.run_gyp('file.gyp', chdir='subdir') | |
| 265 ``` | |
| 266 | |
| 267 ### Running the build tool | |
| 268 | |
| 269 Running the build tool requires passing in a `.gyp` file, which may be used to | |
| 270 calculate the name of a specific build configuration file (such as a MSVS | |
| 271 solution file corresponding to the `.gyp` file). | |
| 272 | |
| 273 There are several different `.build_*()` methods for invoking different types | |
| 274 of builds. | |
| 275 | |
| 276 To invoke a build tool with an explicit `all` target (or equivalent): | |
| 277 | |
| 278 ``` | |
| 279 test.build_all('file.gyp') | |
| 280 ``` | |
| 281 | |
| 282 To invoke a build tool with its default behavior (for example, executing `make` | |
| 283 with no targets specified): | |
| 284 | |
| 285 ``` | |
| 286 test.build_default('file.gyp') | |
| 287 ``` | |
| 288 | |
| 289 To invoke a build tool with an explicit specified target: | |
| 290 | |
| 291 ``` | |
| 292 test.build_target('file.gyp', 'target') | |
| 293 ``` | |
| 294 | |
| 295 ### Running executables | |
| 296 | |
| 297 The most useful method executes a program built by the GYP-generated | |
| 298 configuration: | |
| 299 | |
| 300 ``` | |
| 301 test.run_built_executable('program') | |
| 302 ``` | |
| 303 | |
| 304 The `.run_built_executable()` method will account for the actual built target | |
| 305 output location for the build tool being tested, as well as tack on any | |
| 306 necessary executable file suffix for the platform (for example `.exe` on | |
| 307 Windows). | |
| 308 | |
| 309 `stdout=` and `stderr=` keyword arguments specify expected standard output and | |
| 310 error output, respectively. Failure to match these (if specified) will cause | |
| 311 the test to fail. An explicit `None` value will suppress that verification: | |
| 312 | |
| 313 ``` | |
| 314 test.run_built_executable('program', | |
| 315 stdout="expect this output\n", | |
| 316 stderr=None) | |
| 317 ``` | |
| 318 | |
| 319 Note that the default values are `stdout=None` and `stderr=''` (that is, no | |
| 320 check for standard output, and error output must be empty). | |
| 321 | |
| 322 Arbitrary executables (not necessarily those built by GYP) can be executed with | |
| 323 the lower-level `.run()` method: | |
| 324 | |
| 325 ``` | |
| 326 test.run('program') | |
| 327 ``` | |
| 328 | |
| 329 The program must be in the local directory (that is, the temporary directory | |
| 330 for test execution) or be an absolute path name. | |
| 331 | |
| 332 ### Fetching command output | |
| 333 | |
| 334 ``` | |
| 335 test.stdout() | |
| 336 ``` | |
| 337 | |
| 338 Returns the standard output from the most recent executed command (including | |
| 339 `.run_gyp()`, `.build_*()`, or `.run*()` methods). | |
| 340 | |
| 341 ``` | |
| 342 test.stderr() | |
| 343 ``` | |
| 344 | |
| 345 Returns the error output from the most recent executed command (including | |
| 346 `.run_gyp()`, `.build_*()`, or `.run*()` methods). | |
| 347 | |
| 348 ### Verifying existence or non-existence of files or directories | |
| 349 | |
| 350 ``` | |
| 351 test.must_exist('file_or_dir') | |
| 352 ``` | |
| 353 | |
| 354 Verifies that the specified file or directory exists, and fails the test if it | |
| 355 doesn't. | |
| 356 | |
| 357 ``` | |
| 358 test.must_not_exist('file_or_dir') | |
| 359 ``` | |
| 360 | |
| 361 Verifies that the specified file or directory does not exist, and fails the | |
| 362 test if it does. | |
| 363 | |
| 364 ### Verifying file contents | |
| 365 | |
| 366 ``` | |
| 367 test.must_match('file', 'expected content\n') | |
| 368 ``` | |
| 369 | |
| 370 Verifies that the content of the specified file match the expected string, and | |
| 371 fails the test if it does not. By default, the match must be exact, but | |
| 372 line-by-line regular expressions may be used if the `TestGyp` object was | |
| 373 initialized with `TestGyp.match_re`. | |
| 374 | |
| 375 ``` | |
| 376 test.must_not_match('file', 'expected content\n') | |
| 377 ``` | |
| 378 | |
| 379 Verifies that the content of the specified file does _not_ match the expected | |
| 380 string, and fails the test if it does. By default, the match must be exact, | |
| 381 but line-by-line regular expressions may be used if the `TestGyp` object was | |
| 382 initialized with `TestGyp.match_re`. | |
| 383 | |
| 384 ``` | |
| 385 test.must_contain('file', 'substring') | |
| 386 ``` | |
| 387 | |
| 388 Verifies that the specified file contains the specified substring, and fails | |
| 389 the test if it does not. | |
| 390 | |
| 391 ``` | |
| 392 test.must_not_contain('file', 'substring') | |
| 393 ``` | |
| 394 | |
| 395 Verifies that the specified file does not contain the specified substring, and | |
| 396 fails the test if it does. | |
| 397 | |
| 398 ``` | |
| 399 test.must_contain_all_lines(output, lines) | |
| 400 ``` | |
| 401 | |
| 402 Verifies that the output string contains all of the "lines" in the specified | |
| 403 list of lines. In practice, the lines can be any substring and need not be | |
| 404 `\n`-terminaed lines per se. If any line is missing, the test fails. | |
| 405 | |
| 406 ``` | |
| 407 test.must_not_contain_any_lines(output, lines) | |
| 408 ``` | |
| 409 | |
| 410 Verifies that the output string does _not_ contain any of the "lines" in the | |
| 411 specified list of lines. In practice, the lines can be any substring and need | |
| 412 not be `\n`-terminaed lines per se. If any line exists in the output string, | |
| 413 the test fails. | |
| 414 | |
| 415 ``` | |
| 416 test.must_contain_any_line(output, lines) | |
| 417 ``` | |
| 418 | |
| 419 Verifies that the output string contains at least one of the "lines" in the | |
| 420 specified list of lines. In practice, the lines can be any substring and need | |
| 421 not be `\n`-terminaed lines per se. If none of the specified lines is present, | |
| 422 the test fails. | |
| 423 | |
| 424 ### Reading file contents | |
| 425 | |
| 426 ``` | |
| 427 test.read('file') | |
| 428 ``` | |
| 429 | |
| 430 Returns the contents of the specified file. Directory elements contained in a | |
| 431 list will be joined: | |
| 432 | |
| 433 ``` | |
| 434 test.read(['subdir', 'file']) | |
| 435 ``` | |
| 436 | |
| 437 ### Test success or failure | |
| 438 | |
| 439 ``` | |
| 440 test.fail_test() | |
| 441 ``` | |
| 442 | |
| 443 Fails the test, reporting `FAILED` on standard output and exiting with an exit | |
| 444 status of `1`. | |
| 445 | |
| 446 ``` | |
| 447 test.pass_test() | |
| 448 ``` | |
| 449 | |
| 450 Passes the test, reporting `PASSED` on standard output and exiting with an exit | |
| 451 status of `0`. | |
| 452 | |
| 453 ``` | |
| 454 test.no_result() | |
| 455 ``` | |
| 456 | |
| 457 Indicates the test had no valid result (i.e., the conditions could not be | |
| 458 tested because of an external factor like a full file system). Reports `NO | |
| 459 RESULT` on standard output and exits with a status of `2`. | |
| OLD | NEW |
|---|
Chromium Code Reviews
Issue 1400043002:
Promote documents in md-pages branch. (Closed)
Base URL: https://chromium.googlesource.com/external/gyp@master