diff options
Diffstat (limited to 'node_modules/node-gyp/gyp/docs/Testing.md')
| -rw-r--r-- | node_modules/node-gyp/gyp/docs/Testing.md | 450 |
1 files changed, 0 insertions, 450 deletions
diff --git a/node_modules/node-gyp/gyp/docs/Testing.md b/node_modules/node-gyp/gyp/docs/Testing.md deleted file mode 100644 index a52031e..0000000 --- a/node_modules/node-gyp/gyp/docs/Testing.md +++ /dev/null @@ -1,450 +0,0 @@ -# Testing - -NOTE: this document is outdated and needs to be updated. Read with your own discretion. - -## Introduction - -This document describes the GYP testing infrastructure, -as provided by the `TestGyp.py` module. - -These tests emphasize testing the _behavior_ of the -various GYP-generated build configurations: -Visual Studio, Xcode, SCons, Make, etc. -The goal is _not_ to test the output of the GYP generators by, -for example, comparing a GYP-generated Makefile -against a set of known "golden" Makefiles -(although the testing infrastructure could -be used to write those kinds of tests). -The idea is that the generated build configuration files -could be completely written to add a feature or fix a bug -so long as they continue to support the functional behaviors -defined by the tests: building programs, shared libraries, etc. - -## "Hello, world!" GYP test configuration - -Here is an actual test configuration, -a simple build of a C program to print `"Hello, world!"`. - -``` - $ ls -l test/hello - total 20 - -rw-r--r-- 1 knight knight 312 Jul 30 20:22 gyptest-all.py - -rw-r--r-- 1 knight knight 307 Jul 30 20:22 gyptest-default.py - -rwxr-xr-x 1 knight knight 326 Jul 30 20:22 gyptest-target.py - -rw-r--r-- 1 knight knight 98 Jul 30 20:22 hello.c - -rw-r--r-- 1 knight knight 142 Jul 30 20:22 hello.gyp - $ -``` - -The `gyptest-*.py` files are three separate tests (test scripts) -that use this configuration. The first one, `gyptest-all.py`, -looks like this: - -``` - #!/usr/bin/env python - - """ - Verifies simplest-possible build of a "Hello, world!" program - using an explicit build target of 'all'. - """ - - import TestGyp - - test = TestGyp.TestGyp() - - test.run_gyp('hello.gyp') - - test.build_all('hello.gyp') - - test.run_built_executable('hello', stdout="Hello, world!\n") - - test.pass_test() -``` - -The test script above runs GYP against the specified input file -(`hello.gyp`) to generate a build configuration. -It then tries to build the `'all'` target -(or its equivalent) using the generated build configuration. -Last, it verifies that the build worked as expected -by running the executable program (`hello`) -that was just presumably built by the generated configuration, -and verifies that the output from the program -matches the expected `stdout` string (`"Hello, world!\n"`). - -Which configuration is generated -(i.e., which build tool to test) -is specified when the test is run; -see the next section. - -Surrounding the functional parts of the test -described above are the header, -which should be basically the same for each test -(modulo a different description in the docstring): - -``` - #!/usr/bin/env python - - """ - Verifies simplest-possible build of a "Hello, world!" program - using an explicit build target of 'all'. - """ - - import TestGyp - - test = TestGyp.TestGyp() -``` - -Similarly, the footer should be the same in every test: - -``` - test.pass_test() -``` - -## Running tests - -Test scripts are run by the `gyptest.py` script. -You can specify (an) explicit test script(s) to run: - -``` - $ python gyptest.py test/hello/gyptest-all.py - PYTHONPATH=/home/knight/src/gyp/trunk/test/lib - TESTGYP_FORMAT=scons - /usr/bin/python test/hello/gyptest-all.py - PASSED - $ -``` - -If you specify a directory, all test scripts -(scripts prefixed with `gyptest-`) underneath -the directory will be run: - -``` - $ python gyptest.py test/hello - PYTHONPATH=/home/knight/src/gyp/trunk/test/lib - TESTGYP_FORMAT=scons - /usr/bin/python test/hello/gyptest-all.py - PASSED - /usr/bin/python test/hello/gyptest-default.py - PASSED - /usr/bin/python test/hello/gyptest-target.py - PASSED - $ -``` - -Or you can specify the `-a` option to run all scripts -in the tree: - -``` - $ python gyptest.py -a - PYTHONPATH=/home/knight/src/gyp/trunk/test/lib - TESTGYP_FORMAT=scons - /usr/bin/python test/configurations/gyptest-configurations.py - PASSED - /usr/bin/python test/defines/gyptest-defines.py - PASSED - . - . - . - . - /usr/bin/python test/variables/gyptest-commands.py - PASSED - $ -``` - -If any tests fail during the run, -the `gyptest.py` script will report them in a -summary at the end. - -## Debugging tests - -Tests that create intermediate output do so under the gyp/out/testworkarea -directory. On test completion, intermediate output is cleaned up. To preserve -this output, set the environment variable PRESERVE=1. This can be handy to -inspect intermediate data when debugging a test. - -You can also set PRESERVE\_PASS=1, PRESERVE\_FAIL=1 or PRESERVE\_NO\_RESULT=1 -to preserve output for tests that fall into one of those categories. - -# Specifying the format (build tool) to use - -By default, the `gyptest.py` script will generate configurations for -the "primary" supported build tool for the platform you're on: -Visual Studio on Windows, -Xcode on Mac, -and (currently) SCons on Linux. -An alternate format (build tool) may be specified -using the `-f` option: - -``` - $ python gyptest.py -f make test/hello/gyptest-all.py - PYTHONPATH=/home/knight/src/gyp/trunk/test/lib - TESTGYP_FORMAT=make - /usr/bin/python test/hello/gyptest-all.py - PASSED - $ -``` - -Multiple tools may be specified in a single pass as -a comma-separated list: - -``` - $ python gyptest.py -f make,scons test/hello/gyptest-all.py - PYTHONPATH=/home/knight/src/gyp/trunk/test/lib - TESTGYP_FORMAT=make - /usr/bin/python test/hello/gyptest-all.py - PASSED - TESTGYP_FORMAT=scons - /usr/bin/python test/hello/gyptest-all.py - PASSED - $ -``` - -## Test script functions and methods - -The `TestGyp` class contains a lot of functionality -intended to make it easy to write tests. -This section describes the most useful pieces for GYP testing. - -(The `TestGyp` class is actually a subclass of more generic -`TestCommon` and `TestCmd` base classes -that contain even more functionality than is -described here.) - -### Initialization - -The standard initialization formula is: - -``` - import TestGyp - test = TestGyp.TestGyp() -``` - -This copies the contents of the directory tree in which -the test script lives to a temporary directory for execution, -and arranges for the temporary directory's removal on exit. - -By default, any comparisons of output or file contents -must be exact matches for the test to pass. -If you need to use regular expressions for matches, -a useful alternative initialization is: - -``` - import TestGyp - test = TestGyp.TestGyp(match = TestGyp.match_re, - diff = TestGyp.diff_re)` -``` - -### Running GYP - -The canonical invocation is to simply specify the `.gyp` file to be executed: - -``` - test.run_gyp('file.gyp') -``` - -Additional GYP arguments may be specified: - -``` - test.run_gyp('file.gyp', arguments=['arg1', 'arg2', ...]) -``` - -To execute GYP from a subdirectory (where, presumably, the specified file -lives): - -``` - test.run_gyp('file.gyp', chdir='subdir') -``` - -### Running the build tool - -Running the build tool requires passing in a `.gyp` file, which may be used to -calculate the name of a specific build configuration file (such as a MSVS -solution file corresponding to the `.gyp` file). - -There are several different `.build_*()` methods for invoking different types -of builds. - -To invoke a build tool with an explicit `all` target (or equivalent): - -``` - test.build_all('file.gyp') -``` - -To invoke a build tool with its default behavior (for example, executing `make` -with no targets specified): - -``` - test.build_default('file.gyp') -``` - -To invoke a build tool with an explicit specified target: - -``` - test.build_target('file.gyp', 'target') -``` - -### Running executables - -The most useful method executes a program built by the GYP-generated -configuration: - -``` - test.run_built_executable('program') -``` - -The `.run_built_executable()` method will account for the actual built target -output location for the build tool being tested, as well as tack on any -necessary executable file suffix for the platform (for example `.exe` on -Windows). - -`stdout=` and `stderr=` keyword arguments specify expected standard output and -error output, respectively. Failure to match these (if specified) will cause -the test to fail. An explicit `None` value will suppress that verification: - -``` - test.run_built_executable('program', - stdout="expect this output\n", - stderr=None) -``` - -Note that the default values are `stdout=None` and `stderr=''` (that is, no -check for standard output, and error output must be empty). - -Arbitrary executables (not necessarily those built by GYP) can be executed with -the lower-level `.run()` method: - -``` - test.run('program') -``` - -The program must be in the local directory (that is, the temporary directory -for test execution) or be an absolute path name. - -### Fetching command output - -``` - test.stdout() -``` - -Returns the standard output from the most recent executed command (including -`.run_gyp()`, `.build_*()`, or `.run*()` methods). - -``` - test.stderr() -``` - -Returns the error output from the most recent executed command (including -`.run_gyp()`, `.build_*()`, or `.run*()` methods). - -### Verifying existence or non-existence of files or directories - -``` - test.must_exist('file_or_dir') -``` - -Verifies that the specified file or directory exists, and fails the test if it -doesn't. - -``` - test.must_not_exist('file_or_dir') -``` - -Verifies that the specified file or directory does not exist, and fails the -test if it does. - -### Verifying file contents - -``` - test.must_match('file', 'expected content\n') -``` - -Verifies that the content of the specified file match the expected string, and -fails the test if it does not. By default, the match must be exact, but -line-by-line regular expressions may be used if the `TestGyp` object was -initialized with `TestGyp.match_re`. - -``` - test.must_not_match('file', 'expected content\n') -``` - -Verifies that the content of the specified file does _not_ match the expected -string, and fails the test if it does. By default, the match must be exact, -but line-by-line regular expressions may be used if the `TestGyp` object was -initialized with `TestGyp.match_re`. - -``` - test.must_contain('file', 'substring') -``` - -Verifies that the specified file contains the specified substring, and fails -the test if it does not. - -``` - test.must_not_contain('file', 'substring') -``` - -Verifies that the specified file does not contain the specified substring, and -fails the test if it does. - -``` - test.must_contain_all_lines(output, lines) -``` - -Verifies that the output string contains all of the "lines" in the specified -list of lines. In practice, the lines can be any substring and need not be -`\n`-terminated lines per se. If any line is missing, the test fails. - -``` - test.must_not_contain_any_lines(output, lines) -``` - -Verifies that the output string does _not_ contain any of the "lines" in the -specified list of lines. In practice, the lines can be any substring and need -not be `\n`-terminated lines per se. If any line exists in the output string, -the test fails. - -``` - test.must_contain_any_line(output, lines) -``` - -Verifies that the output string contains at least one of the "lines" in the -specified list of lines. In practice, the lines can be any substring and need -not be `\n`-terminated lines per se. If none of the specified lines is present, -the test fails. - -### Reading file contents - -``` - test.read('file') -``` - -Returns the contents of the specified file. Directory elements contained in a -list will be joined: - -``` - test.read(['subdir', 'file']) -``` - -### Test success or failure - -``` - test.fail_test() -``` - -Fails the test, reporting `FAILED` on standard output and exiting with an exit -status of `1`. - -``` - test.pass_test() -``` - -Passes the test, reporting `PASSED` on standard output and exiting with an exit -status of `0`. - -``` - test.no_result() -``` - -Indicates the test had no valid result (i.e., the conditions could not be -tested because of an external factor like a full file system). Reports `NO -RESULT` on standard output and exits with a status of `2`. |