curl/tests/unit
Viktor Szakats 71cf0d1fca
tests: speed up builds with single-binary test bundles
Add support for single-block binaries that contain all libtests and
unit tests respectively.

Enable with:
- autotools: `--enable-test-bundles`
- cmake: `-DCURL_TEST_BUNDLES=ON`

(They are compatible with `--enable-unity` and `-DCMAKE_UNITY_BUILD=ON`
options, for further speed-up.)

Makes libtests and unit tests build _fast_, needing little disk space
even in static mode. Similar to CMake unity mode, but with a custom
script, also supporting autotools builds.

The price is having to deal with symbols/macros colliding between
`lib*.c` and `unit*.c` sources. Maybe with naming conventions or other
solutions this can be improved gradually and reduce the need for manual
intervention by `mk-bundle.mk`. I've included a script that does the bulk
of detecting name collisions.

Also:
- CI: enable test bundles.
- CI: build tests in more jobs.
- lib2305: fix FILE handle leak.
- unit1661: fix memleak found by torture test by releasing the `bufref`
  structure in `unit_stop()` that was allocated in `unit_setup()`.
  ```
  test 1661...[bufref unit tests]
  Leak detected: memory still allocated: 13 bytes
   allocated by /home/runner/work/curl/curl/tests/unit/unit1661.c:70
   1661: torture FAILED: function number 1 in test.
  ```
  Ref: https://github.com/curl/curl/actions/runs/10967279334/job/30456745290?pr=14772#step:8:41

Similar test suite builds with autotools default and cmake+bundle+unity:
- GHA/Linux: 33s vs 7s
  https://github.com/curl/curl/actions/runs/10705668823/job/29681617374
- GHA/macOS 34s vs 2s
  https://github.com/curl/curl/actions/runs/10705668813/job/29681632885
- GHA/FreeBSD: 15m25 vs 6m21 (full workflow time, ~qemu)
  https://github.com/curl/curl/actions/runs/10705668811/job/29681607915
- GHA/Cygwin: 9m52 vs 32s
  https://github.com/curl/curl/actions/runs/10705668809/job/29681609965
- GHA/MSYS2: 3m52 vs 14s
  https://github.com/curl/curl/actions/runs/10705668808/job/29681624295
- GHA/mingw-w64: 5m45 vs 30s
  https://github.com/curl/curl/actions/runs/10705668808/job/29681628787

Autotools test suite builds compared between master -> `--enable-test-bundles`:
- GHA/Linux: 33s -> 9s (run tests: 22m23 -> 20m44)
  https://github.com/curl/curl/actions/runs/10710030193/job/29695932185
  https://github.com/curl/curl/actions/runs/10967831456/job/30458220344
- GHA/macOS: 25s -> 4s (run tests: 2m58 -> 2m24)
  https://github.com/curl/curl/actions/runs/10710030195/job/29695938444
  https://github.com/curl/curl/actions/runs/10967831452/job/30458225762
- GHA/non-native (FreeBSD): 4m8 -> 3m12 (full workflow time, ~qemu)
  https://github.com/curl/curl/actions/runs/10710030198/job/29695928401
  https://github.com/curl/curl/actions/runs/10967831458/job/30458212692
- GHA/Cygwin: 9m25 -> 1m9 (run tests: 9m19 -> 3m28)
  https://github.com/curl/curl/actions/runs/10710030212/job/29695928213
  https://github.com/curl/curl/actions/runs/10967831453/job/30458213268
- GHA/MSYS2: 3m54 -> 32s (run tests: 6m3 -> 3m59)
  https://github.com/curl/curl/actions/runs/10710030190/job/29704850591
  https://github.com/curl/curl/actions/runs/10967831449/job/30459280005
- GHA/mingw-w64: 5m42 -> 1m5 (run tests: 7m41 -> 5m36)
  https://github.com/curl/curl/actions/runs/10710030190/job/29704852058
  https://github.com/curl/curl/actions/runs/10967831449/job/30459280862
- Azure MSYS2 mingw64 openssl: 38m55 -> 11m58
  https://dev.azure.com/daniel0244/curl/_build/results?buildId=25546&view=logs&j=b58b8c59-0f61-52e9-0f9e-fad562a1e77f&t=0f9230a7-3b10-53ca-9938-700ece377c5e
  https://dev.azure.com/daniel0244/curl/_build/results?buildId=25547&view=logs&jobId=39473db1-3945-55d5-deb5-c218fad88dce&j=b58b8c59-0f61-52e9-0f9e-fad562a1e77f&t=0f9230a7-3b10-53ca-9938-700ece377c5e
- Azure Ubuntu default: 2m15 -> 55s (all build)
  https://dev.azure.com/daniel0244/curl/_build/results?buildId=25546&view=logs&j=9d58b9ac-e1e6-53b6-f83a-1f9f1d912522&t=a6b38d83-e7cf-5a9b-c762-a178412717b7
  https://dev.azure.com/daniel0244/curl/_build/results?buildId=25547&view=logs&jobId=39473db1-3945-55d5-deb5-c218fad88dce&j=9d58b9ac-e1e6-53b6-f83a-1f9f1d912522&t=a6b38d83-e7cf-5a9b-c762-a178412717b7

Cmake test suite builds compared between master -> `-DCURL_TEST_BUNDLES=ON` + unity:
- GHA/Linux: 29s -> 7s (run tests: 4m50 -> 4m57, 20m43 -> 20m45)
  https://github.com/curl/curl/actions/runs/10710030193/job/29695941814
  https://github.com/curl/curl/actions/runs/10705668823/job/29681622201
- GHA/Linux old: 44s -> 13s (bundle+no unity) (run tests: 5m5 -> 5m6)
  https://github.com/curl/curl/actions/runs/10718264094/job/29719794727
  https://github.com/curl/curl/actions/runs/10718653175/job/29721009613
- GHA/macOS: 32s -> 2s (run tests: 2m43 -> 2m40)
  https://github.com/curl/curl/actions/runs/10710030195/job/29695931956
  https://github.com/curl/curl/actions/runs/10705668813/job/29681638937
- GHA/non-native (*BSD): inconclusive (full workflow time, ~qemu)
  https://github.com/curl/curl/actions/runs/10710030198
  https://github.com/curl/curl/actions/runs/10705668811
- GHA/Cygwin: 3m9 -> 32s
  https://github.com/curl/curl/actions/runs/10710030212/job/29695929075
  https://github.com/curl/curl/actions/runs/10705668809/job/29681609965
- GHA/MSYS2: 2m24 -> 14s
  https://github.com/curl/curl/actions/runs/10710030190/job/29704850996
  https://github.com/curl/curl/actions/runs/10705668808/job/29681624295
- GHA/mingw-w64: 3m56 -> 30s (run tests: 4m2 -> 3m52)
  https://github.com/curl/curl/actions/runs/10710030190/job/29704852219
  https://github.com/curl/curl/actions/runs/10705668808/job/29681631393
- GHA/mingw-w64-old: 7m19 -> 1m44 (run tests: 3m30 -> 2m53)
  https://github.com/curl/curl/actions/runs/10710030190/job/29704849763
  https://github.com/curl/curl/actions/runs/10705668808/job/29681622329
- GHA/MSVC: 3m22 -> 13s (run tests: 9m43 -> 4m22)
  https://github.com/curl/curl/actions/runs/10710030190/job/29704850411
  https://github.com/curl/curl/actions/runs/10705668808/job/29681623313
- AppVeyor CI MSVC 2008: 4m3 -> 45s (full build)
- AppVeyor CI MSVC 2010: 2m56 -> 1m8 (full build)
- AppVeyor CI MSVC 2022: 10m19 -> 2m23 (full build)
  https://ci.appveyor.com/project/curlorg/curl/builds/50538455
  https://ci.appveyor.com/project/curlorg/curl/builds/50536558
- AppVeyor CI total build time: 10m30 (master) -> 6m48 (unity) -> 4m5 (bundle) -> 3m24 (bundle+unity) -> 5m7 (bundle+unity+all jobs building tests)

Closes #14772
2024-09-22 09:51:15 +02:00
..
.gitignore tests: speed up builds with single-binary test bundles 2024-09-22 09:51:15 +02:00
CMakeLists.txt tests: speed up builds with single-binary test bundles 2024-09-22 09:51:15 +02:00
curlcheck.h tests: make the unit test result type CURLcode 2024-05-12 18:53:07 +02:00
Makefile.am tests: speed up builds with single-binary test bundles 2024-09-22 09:51:15 +02:00
Makefile.inc tests: speed up builds with single-binary test bundles 2024-09-22 09:51:15 +02:00
README.md tests: fixup tests/data/Makefile.am references 2024-08-04 15:28:29 +02:00
unit1300.c unit1300: fix checksrc longline warnings 2024-08-21 03:37:29 -04:00
unit1302.c tests: speed up builds with single-binary test bundles 2024-09-22 09:51:15 +02:00
unit1303.c tests: speed up builds with single-binary test bundles 2024-09-22 09:51:15 +02:00
unit1304.c tests: speed up builds with single-binary test bundles 2024-09-22 09:51:15 +02:00
unit1305.c tests: speed up builds with single-binary test bundles 2024-09-22 09:51:15 +02:00
unit1307.c tests: fix MSVC unreachable code warnings in unit tests 2023-03-11 18:54:21 -08:00
unit1308.c build: tidy up deprecation suppression, enable warnings for clang 2024-09-21 00:59:52 +02:00
unit1309.c lib/src: white space edits to comply better with code style 2024-09-19 14:59:12 +02:00
unit1323.c tests: fix bad printf format flags in test code 2023-08-07 10:46:14 +02:00
unit1330.c copyright: update all copyright lines and remove year ranges 2023-01-03 09:19:21 +01:00
unit1394.c windows: use built-in _WIN32 macro to detect Windows 2023-11-22 15:42:25 +00:00
unit1395.c autotools: stop setting -std=gnu89 with --enable-warnings 2023-11-20 22:28:17 +00:00
unit1396.c copyright: update all copyright lines and remove year ranges 2023-01-03 09:19:21 +01:00
unit1397.c gskit: remove 2023-08-07 20:57:48 +02:00
unit1398.c tests: speed up builds with single-binary test bundles 2024-09-22 09:51:15 +02:00
unit1399.c lib/src: white space edits to comply better with code style 2024-09-19 14:59:12 +02:00
unit1600.c copyright: update all copyright lines and remove year ranges 2023-01-03 09:19:21 +01:00
unit1601.c lib: add ability to disable auths individually 2023-09-07 17:45:06 +02:00
unit1602.c src: fix potential macro confusion in cmake unity builds 2024-08-22 10:45:04 +02:00
unit1603.c src: fix potential macro confusion in cmake unity builds 2024-08-22 10:45:04 +02:00
unit1604.c tidy-up: OS names 2024-08-04 19:17:45 +02:00
unit1605.c copyright: update all copyright lines and remove year ranges 2023-01-03 09:19:21 +01:00
unit1606.c tests: speed up builds with single-binary test bundles 2024-09-22 09:51:15 +02:00
unit1607.c build: prefer USE_IPV6 macro internally (was: ENABLE_IPV6) 2024-04-13 08:33:26 +00:00
unit1608.c test1608: make it build and get skipped without shuffle DNS support 2023-08-17 17:17:59 +02:00
unit1609.c hash: provide asserts to verify API use 2024-08-15 08:54:19 +02:00
unit1610.c lib: prefer CURL_SHA256_DIGEST_LENGTH over the unprefixed name 2024-08-13 10:04:06 +02:00
unit1611.c md4: only build when used 2023-05-13 09:57:35 +02:00
unit1612.c lib: add ability to disable auths individually 2023-09-07 17:45:06 +02:00
unit1614.c CI: move Azure jobs to GHA, fix fallouts, sshserver, runtests tweaks 2024-09-16 10:00:17 +02:00
unit1615.c curl_sha512_256: fix symbol collisions with nettle library 2024-08-13 10:04:06 +02:00
unit1616.c src: fix potential macro confusion in cmake unity builds 2024-08-22 10:45:04 +02:00
unit1620.c tests: extend user/password parsing test1620 2024-05-25 23:48:57 +02:00
unit1621.c unit tests: use the unit test infrastructure better 2023-04-01 08:59:38 -07:00
unit1650.c doh: cleanups 2024-09-06 08:58:52 +02:00
unit1651.c build: fix some -Wsign-conversion/-Warith-conversion warnings 2023-12-19 12:45:28 +00:00
unit1652.c tests: speed up builds with single-binary test bundles 2024-09-22 09:51:15 +02:00
unit1653.c checksrc: disallow spaces before labels 2023-05-18 20:45:04 +02:00
unit1654.c llist: remove direct struct accesses, use only functions 2024-08-12 13:18:10 +02:00
unit1655.c doh: cleanups 2024-09-06 08:58:52 +02:00
unit1656.c x509asn1: unittests and fixes for gtime2str 2024-07-30 23:08:59 +02:00
unit1660.c unit1660: fix unreachable code warning in no-SSL builds 2024-09-19 23:06:09 +02:00
unit1661.c tests: speed up builds with single-binary test bundles 2024-09-22 09:51:15 +02:00
unit1663.c tests: speed up builds with single-binary test bundles 2024-09-22 09:51:15 +02:00
unit2600.c lib/src: white space edits to comply better with code style 2024-09-19 14:59:12 +02:00
unit2601.c lib/src: white space edits to comply better with code style 2024-09-19 14:59:12 +02:00
unit2602.c trace: make tracing available in non-debug builds 2023-08-03 17:32:25 +02:00
unit2603.c CI: move Azure jobs to GHA, fix fallouts, sshserver, runtests tweaks 2024-09-16 10:00:17 +02:00
unit2604.c CI: move Azure jobs to GHA, fix fallouts, sshserver, runtests tweaks 2024-09-16 10:00:17 +02:00
unit3200.c tests: make the unit test result type CURLcode 2024-05-12 18:53:07 +02:00
unit3205.c rustls: add support for setting TLS version and ciphers 2024-08-16 09:55:02 +02:00

Unit tests

The goal is to add tests for all functions in libcurl. If functions are too big and complicated, we should split them into smaller and testable ones.

Build Unit Tests

./configure --enable-debug is required for the unit tests to build. To enable unit tests, there will be a separate static libcurl built that will be used exclusively for linking unit test programs. Just build everything as normal, and then you can run the unit test cases as well.

Run Unit Tests

Unit tests are run as part of the regular test suite. If you have built everything to run unit tests, to can do 'make test' at the root level. Or you can cd tests and make and then invoke individual unit tests with ./runtests.pl NNNN where NNNN is the specific test number.

Debug Unit Tests

If a specific test fails you will get told. The test case then has output left in the %LOGDIR subdirectory, but most importantly you can re-run the test again using gdb by doing ./runtests.pl -g NNNN. That is, add a -g to make it start up gdb and run the same case using that.

Write Unit Tests

We put tests that focus on an area or a specific function into a single C source file. The source file should be named unitNNNN.c where NNNN is a previously unused number.

Add your test to tests/unit/Makefile.inc (if it is a unit test). Add your test data file name to tests/data/Makefile.am

You also need a separate file called tests/data/testNNNN (using the same number) that describes your test case. See the test1300 file for inspiration and the tests/FILEFORMAT.md documentation.

For the actual C file, here's a simple example:


    #include "curlcheck.h"

    #include "a libcurl header.h" /* from the lib dir */

    static CURLcode unit_setup( void )
    {
      /* whatever you want done first */
      return CURLE_OK;
    }

    static void unit_stop( void )
    {
      /* done before shutting down and exiting */
    }

    UNITTEST_START

      /* here you start doing things and checking that the results are good */

      fail_unless( size == 0 , "initial size should be zero" );
      fail_if( head == NULL , "head should not be initiated to NULL" );

      /* you end the test code like this: */

    UNITTEST_STOP