diff --git a/tests/README.md b/tests/README.md index 65af2a0bd9..6f027bddeb 100644 --- a/tests/README.md +++ b/tests/README.md @@ -8,6 +8,41 @@ SPDX-License-Identifier: curl # Running + See the "Requires to run" section for prerequisites. + + In the root of the curl repository: + + ./configure && make && make test + + To run a specific set of tests (e.g. 303 and 410): + + make test TFLAGS="303 410" + + To run the tests faster, pass the -j (parallelism) flag: + + make test TFLAGS="-j10" + + "make test" builds the test suite support code and invokes the 'runtests.pl' + perl script to run all the tests. The value of `TFLAGS` is passed + directly to 'runtests.pl'. + + When you run tests via make, the flags `-a` and `-s` are passed, meaning + to continue running tests even after one fails, and to emit short output. + + If you'd like to not use those flags, you can run 'runtests.pl' directly. + You must `chdir` into the tests directory, then you can run it like so: + + ./runtests.pl 303 410 + + You must have run `make test` at least once first to build the support code. + + To see what flags are available for runtests.pl, and what output it emits, run: + + man ./tests/runtests.1 + + After a test fails, examine the tests/log directory for stdout, stderr, and + output from the servers used in the test. + ## Requires to run - perl (and a unix-style shell) @@ -75,44 +110,6 @@ SPDX-License-Identifier: curl used, set the environment variable `NGHTTPX`. The default can also be changed by specifying `--with-test-nghttpx=` as argument to `configure`. -### Run - - `./configure && make && make test`. This builds the test suite support code - and invokes the 'runtests.pl' perl script to run all the tests. Edit the top - variables of that script in case you have some specific needs, or run the - script manually (after the support code has been built). - - The script breaks on the first test that doesn't do OK. Use `-a` to prevent - the script from aborting on the first error. Run the script with `-v` for - more verbose output. Use `-d` to run the test servers with debug output - enabled as well. Specifying `-k` keeps all the log files generated by the - test intact. - - Use `-s` for shorter output, or pass test numbers to run specific tests only - (like `./runtests.pl 3 4` to test 3 and 4 only). It also supports test case - ranges with 'to', as in `./runtests.pl 3 to 9` which runs the seven tests - from 3 to 9. Any test numbers starting with ! are disabled, as are any test - numbers found in the files `data/DISABLED` or `data/DISABLED.local` (one per - line). The latter is meant for local temporary disables and will be ignored - by git. - - Test cases mentioned in `DISABLED` can still be run if `-f` is provided. - - When `-s` is not present, each successful test will display on one line the - test number and description and on the next line a set of flags, the test - result, current test sequence, total number of tests to be run and an - estimated amount of time to complete the test run. The flags consist of - these letters describing what is checked in this test: - - s stdout - d data - u upload - p protocol - o output - e exit code - m memory - v valgrind - ### Shell startup scripts Tests which use the ssh test server, SCP/SFTP/SOCKS tests, might be badly diff --git a/tests/runtests.1 b/tests/runtests.1 index 571e5ce062..6d895d2b9b 100644 --- a/tests/runtests.1 +++ b/tests/runtests.1 @@ -53,6 +53,35 @@ this keyword. Remember that the exclamation marks and spaces will need to be quoted somehow when entered at many command shells. Prefix a keyword with a tilde (~) to still run it, but ignore the results. + +.SH "OUTPUT" + +When running without -s (short output), for instance when running runtests.pl +directly rather than via make, each test will emits a pair of lines like this: + +Test 0045...[simple HTTP Location: without protocol in initial URL] +--pd---e-v- OK (45 out of 1427, remaining: 16:08, took 6.188s, duration: 00:31) + +the first line contains the test number and a description. On the second line, +the characters at the beginning are flags indicating which aspects of curl's +behavior were checked by the test: + + s stdout + r stderr + p protocol + d data + u upload + P proxy + o output + e exit code + m memory + v valgrind + E the test was run event-based + +The remainder of the second line contains the test result, current test sequence, +total number of tests to be run and an estimated amount of time to complete the +test run. + .SH OPTIONS .IP "-a" Continue running the rest of the test cases even if one test fails. By @@ -91,9 +120,6 @@ start. Run the given test(s) with gdb as a windowed application. .IP "-h, --help" Displays a help text about this program's command line options. -.IP "-k" -Keep output and log files in log/ after a test run, even if no error was -detected. Useful for debugging. .IP "-j[num]" Spawn num processes to run tests. This defaults to 0 to run tests serially within a single process. Using a number greater than one allows multiple tests @@ -101,6 +127,9 @@ to run in parallel, speeding up a test run. The optimum number is dependent on the system and set of tests to run, but 7*number of CPU cores is a good figure to start with, or 1.3*number of CPU cores if Valgrind is in use. Enabling parallel tests is not recommended in conjunction with the \-g option. +.IP "-k" +Keep output and log files in log/ after a test run, even if no error was +detected. Useful for debugging. .IP "-L " Load and execute the specified file which should contain perl code. This option allows one to change \fIruntests.pl\fP behaviour by overwriting diff --git a/tests/runtests.pl b/tests/runtests.pl index 32e0680378..e7b9f01f6a 100755 --- a/tests/runtests.pl +++ b/tests/runtests.pl @@ -23,6 +23,8 @@ # ########################################################################### +# For documentation, run `man ./runtests.1` and see README.md. + # Experimental hooks are available to run tests remotely on machines that # are able to run curl but are unable to run the test harness. # The following sections need to be modified: @@ -1503,7 +1505,7 @@ sub singletest_check { } else { - $ok .= "-"; # protocol not checked + $ok .= "-"; # proxy not checked } my $outputok;