Commit Graph

153 Commits

Author SHA1 Message Date
Viktor Szakats
4633e41fc5
formatting: move MAKEFLAGS first 2025-02-26 22:26:25 +01:00
Daniel Stenberg
71f190cbbc
GHA: enable the RTSP tests on macOS gcc-12 2025-02-26 07:58:54 +01:00
Viktor Szakats
67559aa8ce
CI: dump non-pre-fill configure log on pre-fill check fail
To help debugging builds where the actual feature check is broken.

Follow-up to e7adf3e837 #15841
Closes #16369
2025-02-17 16:59:56 +01:00
Viktor Szakats
e7adf3e837
cmake: add pre-fill for Unix, enable in GHA/macos, verify pre-fills
TL;DR: Save 10 minutes of CI time for GHA/macos jobs using pre-fills and
add pre-fill verification for Apple and Windows. Also restores Xcode job
and saves 1.5-10 minutes configuring iOS jobs.

Pre-filling feature detection results can bring down the CMake configure
step to ~5 seconds on most GHA runners, ~10 seconds in slow envs like
Cygwin/MSYS2.

The potential savings per job are:
- 5-40 (average 19) seconds on GHA/macos (33 jobs)
- ~10 seconds on GHA for iOS GNU Makefile (1 job)
- 1.5-10 minutes on GHA for iOS Xcode generator (1 job)
- 10 seconds on GHA/linux with native Ubuntu (12 jobs)
- 40 seconds for Cygwin/MSYS2 (2 jobs)
- 5-10 seconds for virtualized BSDs, native CPU (3 jobs)
- ~60 seconds for virtualized BSDs, emulated CPU (1 job)

On native Windows pre-filling has been in place for a long time and
saving 8 minutes (VS2019-VS2015) to 1.5-2 minutes (VS2022), 3 minutes
(VS2022 UWP), and 30-60 seconds (MinGW), per CI job.

The downside is that detection results need to be manually collected and
filtered to those that universally apply to all platforms that they are
enabled on. Another downside is that by using a cache, we're not running
the actual detections, and thus won't catch regressions in them. It
means we must make sure that the cache is solid and matches with actual
detections results. An upside is that it gives a rough overview of which
features are available on which platforms. Another upside is pre-filled
values do work for feature detections skipped for cross-builds, e.g.
`HAVE_WRITABLE_ARGV`.

This PR adds a pre-fill cache that supports all Unixes (except OmniOS)
used in CI, and makes it usable with an internal option. It also enables
it for GHA/macos CI jobs, where the maximum savings are. And also for
the two iOS [1] and two Cygwin/MSYS2 jobs. The latters don't have
pre-fill checks and we can drop them if they turn into a hassle.

Saving:
- 10 minutes of CI time per GHA/macos workflow run. [2]
- ~80 seconds per GHA/windows workflow run with Cygwin/MSYS2.
  (offsetting the cost of pre-fill verifications)
- 1.5-10 minutes per GHA/non-native runs with iOS jobs. [3]

You can enable pre-fill locally with `-D_CURL_PREFILL=ON`. It's
experimental, and if you experience a problem, file a PR or an Issue.

This PR also adds a pre-fill checker for macOS and MinGW/MSVC Windows
GHA jobs to catch if the cache diverges from real detections. It also
adds this logic to AppVeyor, but doesn't enable it due to the perf
penalty of 2 minutes mininum.

The pre-fill checker works by configuring out-of-tree with and without
pre-fill, then diffing their `lib/curl_config.h` outputs.

Exceptions are 3 detection results exposed indirectly [4], and missing
to expose 2, of which one is the C89 header `stddef.h`. While we assume
the C99 `stdint.h` available outside iOS. We can expose them in the
future, if necessary.

The pre-fill checks cost in total:
- ~20 seconds for macOS
- ~40 seconds for MinGW on GHA
- ~80 seconds for MSVC on GHA (UWP would be 2x this)

An extra time saving potential is caching type sizes. They are
well-known, and seldom change, esp. in CI. GHA/Windows jobs spend 8-17
seconds per job on these ~12 feature checks. ~5s on Cygwin/MSYS2. Couple
of seconds on other platforms. (This PR doesn't make this optimization.)

Another opportunity is doing the same for autotools, which typically
spends more time in the configuration step than cmake.

[1] Xcode job restored as a
follow-up to be5f20202c #16302

[2] GHA/macos cmake configure times in seconds:
Job                                              |  Bef. | After |  Gain
:----------------------------------------------- | ----: | ----: | ----:
CM clang GnuTLS !ldap krb5                       |  21.2 |   4.5 |  16.7
CM clang LibreSSL !ldap heimdal c-ares +examples |  13.3 |   3.9 |   9.4
CM clang OpenSSL +static libssh +examples        |  20.0 |   4.6 |  15.4
CM clang OpenSSL IDN clang-tidy~ (w/chkprefill)  |  15.7 |  18.6 |  -2.9
CM clang OpenSSL gsasl rtmp AppleIDN             |  25.0 |   4.7 |  20.3
CM clang OpenSSL torture !FTP                    |  15.3 |   4.5 |  10.8
CM clang OpenSSL torture FTP                     |  25.0 |   5.9 |  19.1
CM clang SecureTransport debug                   |  18.0 |   3.8 |  14.2
CM clang macos-13 SecureTransport                |  45.8 |  12.4 |  33.4
CM clang macos-14 SecureTransport                |  15.8 |   4.6 |  11.2
CM clang macos-15 SecureTransport                |  26.8 |   6.1 |  20.7
CM clang mbedTLS openldap brotli zstd            |  15.1 |   6.5 |   8.6
CM clang wolfSSL !ldap brotli zstd               |  27.0 |   4.4 |  22.6
CM gcc-12 GnuTLS !ldap krb5                      |  39.1 |   8.7 |  30.4
CM gcc-12 LibreSSL !ldap heimdal c-ares +examples|  23.8 |   7.2 |  16.6
CM gcc-12 OpenSSL +static libssh +examples       |  20.7 |   8.5 |  12.2
CM gcc-12 OpenSSL gsasl rtmp AppleIDN            |  23.1 |  10.1 |  13.0
CM gcc-12 SecureTransport debug                  |  21.1 |   4.8 |  16.3
CM gcc-12 mbedTLS openldap brotli zstd           |  21.4 |   5.8 |  15.6
CM gcc-12 wolfSSL !ldap brotli zstd              |  21.1 |   6.9 |  14.2
CM gcc-14 macos-13 SecureTransport               |  61.9 |  18.7 |  43.2
CM gcc-14 macos-14 SecureTransport               |  30.5 |   6.4 |  24.1
CM gcc-14 macos-15 SecureTransport               |  32.7 |   8.4 |  24.3
CM llvm@15 GnuTLS !ldap krb5                     |  21.1 |   7.5 |  13.6
CM llvm@15 LibreSSL !ldap heimdal c-ares +exampl~|  24.6 |   6.8 |  17.8
CM llvm@15 OpenSSL +static libssh +examples      |  19.0 |   6.4 |  12.6
CM llvm@15 OpenSSL gsasl rtmp AppleIDN           |  19.0 |   8.2 |  10.8
CM llvm@15 SecureTransport debug                 |  18.0 |   5.4 |  12.6
CM llvm@15 macos-13 SecureTransport              |  66.2 |  25.7 |  40.5
CM llvm@15 macos-14 SecureTransport              |  31.9 |   6.1 |  25.8
CM llvm@15 mbedTLS openldap brotli zstd          |  19.5 |   8.9 |  10.6
CM llvm@15 wolfSSL !ldap brotli zstd             |  24.3 |   5.9 |  18.4
CM llvm@18 macos-15 SecureTransport              |  33.8 |   6.4 |  27.4
Total                                            | 856.8 | 257.3 | 599.5
Before: https://github.com/curl/curl/actions/runs/13311042735/job/37173478424
After: https://github.com/curl/curl/actions/runs/13313927119/job/37183206426?pr=15841

[3] iOS:
Before: https://github.com/curl/curl/actions/runs/13326401704?pr=15841
After: https://github.com/curl/curl/actions/runs/13332177764?pr=15841

[4] detection results exposed indirectly in `curl_config.h`:
- `HAVE_FILE_OFFSET_BITS` via `_FILE_OFFSET_BITS`
- `HAVE_GETHOSTBYNAME_R_*_REENTRANT` via `NEED_REENTRANT`
- `HAVE_SOCKADDR_IN6_SIN6_ADDR` via `USE_IPV6`

Closes #15841
2025-02-16 01:59:59 +01:00
Viktor Szakats
ffd6132da6
cmake: drop LDAP_DEPRECATED=1 macro, to sync with autotools
We set this macro to silence a warning inside `openldap.h`. With this
warning now silenced by using `-isystem`, we can drop it. Also it never
had to be set to `1`.

Also enable OpenLDAP in a CMake GHA/macos job.

Follow-up to 445fb81237 #14763
Follow-up to 751e168d93 #12024

Closes #16146
2025-02-07 00:07:38 +01:00
Viktor Szakats
5ce5c62662
GHA: fixup CM/AM order within Apple steps, move name: first, quote them
Closes #16092
2025-01-26 15:18:59 +01:00
Viktor Szakats
f1dbe68172
GHA: tidy up quotes, checkout order, silence Android/CMake warnings
- move checkout step right before use.
- quotes in yaml and bash.
- unfold single-line run commands.
- set `CMAKE_WARN_DEPRECATED=OFF` for CMake with Android NDK to avoid
  a wall of useless deprecation warnings hiding useful output.
  These warnings happen in 3rd-party scripts, and unfixable in curl.

Closes #16042
2025-01-19 15:03:16 +01:00
Daniel Stenberg
97d278fd76
GHA: enable the SSL session cache in a few builds
Closes #15989
2025-01-13 23:40:23 +01:00
Viktor Szakats
fabfa8e402
clang-tidy: add to CI, add cmake support, fix fallouts
build:
- autotools: fix to build generated sources for the `tidy` target.
- autotools: allow passing custom clang-tidy options via
  `CURL_CLANG_TIDYFLAGS` env.
- cmake: add `CURL_CLANG_TIDY` option to configure for `clang-tidy`.
  Also add:
  - `CLANG_TIDY` variable to customize the `clang-tidy` tool.
  - `CURL_CLANG_TIDYFLAGS` to pass custom options to `clang-tidy`.
- apply `--enable-werror` and `-DCURL_WERROR=ON` to `clang-tidy`.

CI/GHA:
- add clang-tidy job for Linux, using autotools and clang-tidy v18.
  This one needs to disable `clang-analyzer-valist.Uninitialized`
  to avoid false positives:
  https://github.com/llvm/llvm-project/issues/40656
  Duration: 5.5 minutes
- add clang-tidy job for macOS, using cmake and clang-tidy v19.
  This one also covers tests and examples, and doesn't hit the false
  positives seen with llvm v18 and earlier.
  Duration: 4.5 minutes
- Linux/macOS: skip installing test dependencies when not building or
  running tests.

fix fallouts reported by `clang-tidy`:
- lib:
  - cf-h2-proxy: unused assignment in non-debug builds.
  - cf-socket: silence warning.
    FIXME: https://github.com/curl/curl/pull/15825#issuecomment-2561867769
  - ftp: NULL passed to `strncmp()`.
  - http2: NULL-ptr deref.
  - mprintf: silence warning.
- src/tool_writeout: NULL passed to `fputs()`.
- examples:
  - invalid file pointers.
  - missing `fclose()`.
- tests:
  - http/clients/hx-download: memory leaks on error.
  - http/clients/hx-download: memory leak on repeat `-r` option.
  - server: double `fclose()`.
    https://www.man7.org/linux/man-pages/man3/fclose.3.html
  - server: invalid file pointer/handle.
  - server/getpart: unused assignments.
  - server/mqttd: leak on failed `realloc()`.
  - server/tftpd: NULL passed to `strcmp()`.

Closes #15825
2024-12-27 13:42:32 +01:00
Viktor Szakats
421e592db2
cmake: add librtmp Find module
The new detection method also allows to enable librtmp without using
OpenSSL as a curl TLS backend at the same time.

Also:
- implement manual version detection for librtmp.
  Version info is in hex. With CMake 3.13 and newer, extract it as a hex
  number. With earlier CMake version, just strip the leading zeroes.
  Doing more here seems overkill because librtmp has been standing
  at 2.3/2.4 for a decade now. Bumping into hex digits seems unlikely
  before deprecating CMake 3.13 support.
  librtmp advertises v2.4 via its `pkg-config` module, and v2.3 via
  its public header. The latter shows up in `curl -V` and either can
  be shown at configure-time depending on detection method.
  This isn't a curl bug.
- GHA/macos: enable rtmp in a job.
- apply the "half-detection" fix to the Find module.
  `librtmp` is also affected (in CI too), because it depends on libssl and
  libcrypto.

Closes #15832
2024-12-27 12:52:23 +01:00
Viktor Szakats
435ea241ba
GHA/macos: drop custom macos-version-min options
Drop them, except for Secure Transport jobs where they may trigger
different code paths.

Also drop unused `matrix.build.cflags` variable.

Follow-up to ef90ee39e1 #15763
Closes #15766
2024-12-17 19:15:50 +01:00
Stefan Eissing
ef90ee39e1
lib: supress deprecation warnings in apple builds
On apple builds, the gssapi/ldap/securetransport headers deprecate
almost everything which leads to a wall of compiler warnings on use in
code.

Suppress those warning that may hide other warnings/errors.

Closes #15763
2024-12-17 14:54:34 +01:00
Viktor Szakats
f2adb3b6d7
cmake: extend zlib's AUTO option to brotli, zstd and enable if found
- make `curl_dependency_option()` more generic.

- extend `CURL_BROTLI` and `CURL_ZSTD` options to accept
  `AUTO` in addition to existing `ON` and `OFF`.

- change `CURL_BROTLI` and `CURL_ZSTD` option default
  to `AUTO`. Was: `OFF`.
  It brings cmake behavior closer to `./configure`.
  Still different:
  - `./configure` defaults to `off` which means to check default
    locations. cmake checks more locations by default.
    (Also tried `NO_CMAKE_PATH`, but then it checked less locations.)
  - cmake returns both `brotlicommon` and `brotlidec` libs,
    while `./configure` only returns the latter.

- ci: drop explicit cmake options, that are now unnecessary.

- GHA/configure-vs-cmake: make adjustments to make tests pass.

Closes #15431
2024-12-17 04:06:45 +01:00
Viktor Szakats
ba9fe58d43
GHA: set persist-credentials: false
Suggested by zizmor GHA analysis tool.

Also:
- Move GH variables within single-quotes.
- Prefer single-quotes in shell code. (tidy-up)

Ref: https://github.com/actions/checkout/issues/485
Ref: https://github.com/actions/checkout/pull/1687
Ref: https://woodruffw.github.io/zizmor/

Closes #15746
2024-12-16 18:00:22 +01:00
Viktor Szakats
2f03242316
build: fix ECH to always enable HTTPS RR
The ECH feature cannot be built without HTTPS RR.

ECH automatically implied HTTPS RR in `./configure` but not in CMake,
winbuild, documentation.

Also update documentation and CI configs.

Follow-up to a362962b72 #11922
Closes #15648
2024-11-28 11:55:05 +01:00
Viktor Szakats
7fbd066641
GHA/macos: enable ECH in wolfSSL jobs
Homebrew wolfSSL builds recently enabled all features.
It allows to enable ECH in curl for these jobs.

97d1ed6e6d
https://github.com/Homebrew/homebrew-core/pull/191561

Closes #15607
2024-11-17 20:13:13 +01:00
Viktor Szakats
3c6bf10863
GHA/macos: follow Homebrew and switch to pkgconf
Homebrew switched to `pkgconf`, and now pkg-config installs an extra
package. Update package list to avoid that.

The side-effect of `pkgconf` is that this former log message:
```
--   Package 'libcrypto', required by 'libssh2', not found
```
https://github.com/curl/curl/actions/runs/11779568834/job/32808325442#step:7:84

is replaced by this, and repeated 10 times:
```
Package libcrypto was not found in the pkg-config search path.
Perhaps you should add the directory containing `libcrypto.pc'
to the PKG_CONFIG_PATH environment variable
Package 'libcrypto', required by 'libssh2', not found
```
https://github.com/curl/curl/actions/runs/11792711391/job/32846858320#step:7:85

Closes #15575
2024-11-13 18:19:38 +01:00
Daniel Stenberg
cbafcec50b
curl: --test-duphandle in debug builds runs "duphandled"
Using this option (only available in debug builds) makes curl always
call curl_easy_duphandle() on the handle before using it.

To help us catch curl_easy_duphandle() mistakes better.

Add a CI job using this.

Bonus: the previous runtests option -e is now also supported as
--test-event

Closes #15504
2024-11-08 13:22:47 +01:00
Viktor Szakats
4d8ab1f1b8
GHA/macos: let gcc dictate the configured Apple SDK
As discovered earlier, Homebrew gcc is built against a specific Apple
SDK version and doesn't work when matched up with a different version,
e.g. the one advertised as default by the macos runner image.

Before this patch this was resolved with brute force by zapping the
hack-layer gcc component to avoid the bad interference. This worked
for us, but it's fragile, accidental and doesn't translate to
real-world build environments. Thus, impractical.

Avoid this by explicitly selecting the SDK version gcc was built for and
meant to be used with, as shown by `gcc --print-sysroot`.

It assumes that the gcc binaries preinstalled on the runner images
always ship with the SDK version they reference. It also assumes
this works with and without `brew update`.

Also:
- add 4 quick build-only jobs to test all gcc/macos combos.
- list SDKs offered via CommandLineTools.

Suggested-by: Bo Anderson
Ref: https://github.com/Homebrew/homebrew-core/issues/194778#issuecomment-2462764619

Follow-up to c349bd668c #14097
Closes #15518
2024-11-08 10:48:12 +01:00
Viktor Szakats
087f77d855
GHA/macos: drop WebSockets from job names
It's enabled in all jobs by default now.

Follow-up to cd63629733 #15076
2024-11-06 01:43:55 +01:00
renovate[bot]
38c57bdf0d
GHA: update five dependencies
- rojopolis/spellcheck-github-actions digest to ab8ac45
- nghttp2/nghttp2 to v1.64.0
- actions/cache digest to 6849a64
- github/codeql-action digest to 6624720
- Update actions/checkout digest to 11bd719

Closes #15341
Closes #15346
Closes #15365
Closes #15366
Closes #15387
2024-10-24 10:03:31 +02:00
Viktor Szakats
52851d3256
GHA/macos: use test-torture target for torture tests
They used `test-ci` before this patch.

Closes #15369
2024-10-22 20:20:42 +02:00
Viktor Szakats
1e01e2b549
GHA/macos: merge autotools and cmake jobs
To match other workflows and to avoid repetition in rules.

Also:
- fix build example step for cmake. update a job to use it.
- use `cmake` to invoke the builds (instead of ninja directly).
- extend test 2100 exclusion to more jobs.
  It fails with all `!debug gcc-12` jobs with autotools.
  With cmake this only happened for gcc-12 Secure Transport jobs
  for some reason.

Closes #15312
2024-10-16 23:25:18 +02:00
Viktor Szakats
e8a007de0d
GHA: optimize test prereq steps
- Linux: move test and pytest prereqs right before test run.
  - returns build phase results faster.
  - allows skipping steps for jobs that don't need them.
  - makes dependencies more transparent.
- sync prereq install step names.
- use `tests/requirements.txt` more.

Closes #15275
2024-10-13 11:34:59 +02:00
renovate[bot]
ba68eb02fa
CI: update rojopolis/spellcheck, actions/checkout, actions/upload-artifact
- update rojopolis/spellcheck-github-actions digest to 7ff888c
- update actions/checkout digest to eef6144
- update actions/upload-artifact digest to b4b15b8

Closes #15207
Closes #15184
Closes #15187
2024-10-11 23:53:42 +02:00
Viktor Szakats
7bff686476
ci: dump curl_config.h to log in all jobs
Also:
- GHA/windows: merge full and brief dump into a single job step.
- fix shellcheck warning 'useless cat'.

Closes #15266
2024-10-11 15:01:29 +02:00
Viktor Szakats
5b2d6448bb
GHA/mac: simplify detecting SDK version bound to GCC
Previous method extracted this information from the "hack layer"
headers' copyright comments.

It turns out that the same information can be obtained by querying
the default SDK root via a GCC option.

According to this test run, they do match:
https://github.com/curl/curl/actions/runs/11264283159/job/31323963221?pr=15193

Closes #15241
2024-10-10 11:18:07 +02:00
Viktor Szakats
e377c91766
GHA: add Linux and macOS mbedTLS jobs, fix issue
- update mbedTLS repo URL.
- switch local mbedTLS build to use CMake, and Ninja.
  CMake build is required to create and install mbedTLS `pkg-config`
  files. (as of v3.6.1)
  `-DCMAKE_POSITION_INDEPENDENT_CODE=ON` required to avoid this error
  when linking mbedtls to `libcurl.so`:
  ```
  /usr/bin/ld: /home/runner/mbedtls/lib/libmbedcrypto.a(cipher.c.o): warning: relocation against `mbedtls_cipher_base_lookup_table' in read-only section `.text'
  /usr/bin/ld: /home/runner/mbedtls/lib/libmbedtls.a(ssl_tls.c.o): relocation R_X86_64_PC32 against symbol `mbedtls_x509_crt_profile_suiteb' can not be used when making a shared object; recompile with -fPIC
  /usr/bin/ld: final link failed: bad value
  ```
  Ref: https://github.com/curl/curl/actions/runs/11245069259/job/31264386723#step:40:43
- make local mbedTLS build 10x smaller by omitting programs and tests.
- GHA/linux: fix cmake warning by adding `-B .` option.
- GHA/linux: add build-only cmake job for packaged mbedTLS (2.x).
- fix compiler warning when building with mbedTLS 2.x:
  ```
  /home/runner/work/curl/curl/lib/vtls/mbedtls.c:344:1: error: ‘mbed_cipher_suite_get_str’ defined but not used [-Werror=unused-function]
    344 | mbed_cipher_suite_get_str(uint16_t id, char *buf, size_t buf_size,
        | ^~~~~~~~~~~~~~~~~~~~~~~~~
  ```
  Ref: https://github.com/curl/curl/actions/runs/11244999065/job/31264168295#step:40:50

Also in preparation for #15193
Closes #15208
2024-10-09 02:46:01 +02:00
Viktor Szakats
a71bc67f2c
GHA/macos: comment spelling and clarity
Closes #15166
2024-10-07 12:31:21 +02:00
Viktor Szakats
bc6072d245
GHA/macos: update comment with new Xcode default for macos-13 [ci skip] 2024-10-02 02:16:59 +02:00
Viktor Szakats
96fc2b88f3
GHA/macos: drop unsupported Xcode version references
Closes #15115
2024-10-02 01:52:36 +02:00
Viktor Szakats
bf44536e23
GHA/macos: delete macos-12 jobs, update matrix for macos-14
- Days of `macos-12` are numbered:
  https://github.com/actions/runner-images/issues/10721
  Drop 5 affected jobs, replaced by `macos-15` ones added earlier today.

- Drop Xcode 14 and 16 from `macos-14` jobs.
  Following up upstream announcement:
  https://github.com/actions/runner-images/issues/10703
  (No active jobs are affected by this.)

Follow-up to 4b4ff444dd #15108
Closes #15113
2024-10-01 23:02:34 +02:00
Viktor Szakats
4b4ff444dd
GHA/macos: Sequoia chores, fixes for llvm 18
- add `macos-15` Sequoia to combinations jobs.

- add `llvm@18` to combinations jobs for `macos-15`.

- fix cmake, autotools configurations for llvm 18.
  It requires `CMAKE_OSX_SYSROOT` for cmake, and `--sysroot` within `CC`
  for both cmake and autotools.
  (Also add `--target` to `CC` for all jobs.)

- stop `brew update` on `macos-12`.
  `macos-12` is Monterey. Homebrew stopped providing binaries for it
  after the latest macOS was out. It means newer versions of packages
  have to be built from source, which is slow and undesired. Try to
  avoid it using this trick. If it doesn't work, `macos-12` jobs will
  have to be dropped.

- update Xcode support matrix.

Ref: https://github.com/actions/runner-images/blob/main/images/macos/macos-15-arm64-Readme.md
Closes #15108
2024-10-01 19:27:56 +02:00
dependabot[bot]
303c0cf74b
CI: bump actions/checkout from 4.1.7 to 4.2.0
Bumps [actions/checkout](https://github.com/actions/checkout) from 4.1.7 to 4.2.0.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](692973e3d9...d632683dd7)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Closes #15095
2024-10-01 15:21:36 +02:00
Daniel Stenberg
cd63629733
CI/winbuild: remove enabling of websocket - done by default now
Closes #15076
2024-09-30 07:58:54 +02:00
Stefan Eissing
7f3d598276
CI: improvements in test reliability and performance
- CI default: raise parallelism to 20
- CI valgind: set parallelism to 6
- CI non-native: adapt parallelism for OS builds
- CI Windows: no longer ignore FTP, TFTP, MQTT and SMTP
- CI Windows: restrict test timeout to 10 minutes
- CI Windows: do not run tests for msh3 build
- tests, various: restrict curl invocation to ipv4 to avoid talking to
  any ipv6 test server running in parallel
- tests: requiring http/2 server no longer needs to mention http server
- test 190: use a fixed timeout of 10 seconds instead of %FTPTIME2, as
  that value rises under parallel load beyond what the server is waiting
  and then produces different results
- test 1540: add debug logging
- testrunner: add verify checks for http/2 and http/3 server

Closes #15040
2024-09-27 16:47:37 +02:00
Viktor Szakats
71cf0d1fca
tests: speed up builds with single-binary test bundles
Add support for single-block binaries that contain all libtests and
unit tests respectively.

Enable with:
- autotools: `--enable-test-bundles`
- cmake: `-DCURL_TEST_BUNDLES=ON`

(They are compatible with `--enable-unity` and `-DCMAKE_UNITY_BUILD=ON`
options, for further speed-up.)

Makes libtests and unit tests build _fast_, needing little disk space
even in static mode. Similar to CMake unity mode, but with a custom
script, also supporting autotools builds.

The price is having to deal with symbols/macros colliding between
`lib*.c` and `unit*.c` sources. Maybe with naming conventions or other
solutions this can be improved gradually and reduce the need for manual
intervention by `mk-bundle.mk`. I've included a script that does the bulk
of detecting name collisions.

Also:
- CI: enable test bundles.
- CI: build tests in more jobs.
- lib2305: fix FILE handle leak.
- unit1661: fix memleak found by torture test by releasing the `bufref`
  structure in `unit_stop()` that was allocated in `unit_setup()`.
  ```
  test 1661...[bufref unit tests]
  Leak detected: memory still allocated: 13 bytes
   allocated by /home/runner/work/curl/curl/tests/unit/unit1661.c:70
   1661: torture FAILED: function number 1 in test.
  ```
  Ref: https://github.com/curl/curl/actions/runs/10967279334/job/30456745290?pr=14772#step:8:41

Similar test suite builds with autotools default and cmake+bundle+unity:
- GHA/Linux: 33s vs 7s
  https://github.com/curl/curl/actions/runs/10705668823/job/29681617374
- GHA/macOS 34s vs 2s
  https://github.com/curl/curl/actions/runs/10705668813/job/29681632885
- GHA/FreeBSD: 15m25 vs 6m21 (full workflow time, ~qemu)
  https://github.com/curl/curl/actions/runs/10705668811/job/29681607915
- GHA/Cygwin: 9m52 vs 32s
  https://github.com/curl/curl/actions/runs/10705668809/job/29681609965
- GHA/MSYS2: 3m52 vs 14s
  https://github.com/curl/curl/actions/runs/10705668808/job/29681624295
- GHA/mingw-w64: 5m45 vs 30s
  https://github.com/curl/curl/actions/runs/10705668808/job/29681628787

Autotools test suite builds compared between master -> `--enable-test-bundles`:
- GHA/Linux: 33s -> 9s (run tests: 22m23 -> 20m44)
  https://github.com/curl/curl/actions/runs/10710030193/job/29695932185
  https://github.com/curl/curl/actions/runs/10967831456/job/30458220344
- GHA/macOS: 25s -> 4s (run tests: 2m58 -> 2m24)
  https://github.com/curl/curl/actions/runs/10710030195/job/29695938444
  https://github.com/curl/curl/actions/runs/10967831452/job/30458225762
- GHA/non-native (FreeBSD): 4m8 -> 3m12 (full workflow time, ~qemu)
  https://github.com/curl/curl/actions/runs/10710030198/job/29695928401
  https://github.com/curl/curl/actions/runs/10967831458/job/30458212692
- GHA/Cygwin: 9m25 -> 1m9 (run tests: 9m19 -> 3m28)
  https://github.com/curl/curl/actions/runs/10710030212/job/29695928213
  https://github.com/curl/curl/actions/runs/10967831453/job/30458213268
- GHA/MSYS2: 3m54 -> 32s (run tests: 6m3 -> 3m59)
  https://github.com/curl/curl/actions/runs/10710030190/job/29704850591
  https://github.com/curl/curl/actions/runs/10967831449/job/30459280005
- GHA/mingw-w64: 5m42 -> 1m5 (run tests: 7m41 -> 5m36)
  https://github.com/curl/curl/actions/runs/10710030190/job/29704852058
  https://github.com/curl/curl/actions/runs/10967831449/job/30459280862
- Azure MSYS2 mingw64 openssl: 38m55 -> 11m58
  https://dev.azure.com/daniel0244/curl/_build/results?buildId=25546&view=logs&j=b58b8c59-0f61-52e9-0f9e-fad562a1e77f&t=0f9230a7-3b10-53ca-9938-700ece377c5e
  https://dev.azure.com/daniel0244/curl/_build/results?buildId=25547&view=logs&jobId=39473db1-3945-55d5-deb5-c218fad88dce&j=b58b8c59-0f61-52e9-0f9e-fad562a1e77f&t=0f9230a7-3b10-53ca-9938-700ece377c5e
- Azure Ubuntu default: 2m15 -> 55s (all build)
  https://dev.azure.com/daniel0244/curl/_build/results?buildId=25546&view=logs&j=9d58b9ac-e1e6-53b6-f83a-1f9f1d912522&t=a6b38d83-e7cf-5a9b-c762-a178412717b7
  https://dev.azure.com/daniel0244/curl/_build/results?buildId=25547&view=logs&jobId=39473db1-3945-55d5-deb5-c218fad88dce&j=9d58b9ac-e1e6-53b6-f83a-1f9f1d912522&t=a6b38d83-e7cf-5a9b-c762-a178412717b7

Cmake test suite builds compared between master -> `-DCURL_TEST_BUNDLES=ON` + unity:
- GHA/Linux: 29s -> 7s (run tests: 4m50 -> 4m57, 20m43 -> 20m45)
  https://github.com/curl/curl/actions/runs/10710030193/job/29695941814
  https://github.com/curl/curl/actions/runs/10705668823/job/29681622201
- GHA/Linux old: 44s -> 13s (bundle+no unity) (run tests: 5m5 -> 5m6)
  https://github.com/curl/curl/actions/runs/10718264094/job/29719794727
  https://github.com/curl/curl/actions/runs/10718653175/job/29721009613
- GHA/macOS: 32s -> 2s (run tests: 2m43 -> 2m40)
  https://github.com/curl/curl/actions/runs/10710030195/job/29695931956
  https://github.com/curl/curl/actions/runs/10705668813/job/29681638937
- GHA/non-native (*BSD): inconclusive (full workflow time, ~qemu)
  https://github.com/curl/curl/actions/runs/10710030198
  https://github.com/curl/curl/actions/runs/10705668811
- GHA/Cygwin: 3m9 -> 32s
  https://github.com/curl/curl/actions/runs/10710030212/job/29695929075
  https://github.com/curl/curl/actions/runs/10705668809/job/29681609965
- GHA/MSYS2: 2m24 -> 14s
  https://github.com/curl/curl/actions/runs/10710030190/job/29704850996
  https://github.com/curl/curl/actions/runs/10705668808/job/29681624295
- GHA/mingw-w64: 3m56 -> 30s (run tests: 4m2 -> 3m52)
  https://github.com/curl/curl/actions/runs/10710030190/job/29704852219
  https://github.com/curl/curl/actions/runs/10705668808/job/29681631393
- GHA/mingw-w64-old: 7m19 -> 1m44 (run tests: 3m30 -> 2m53)
  https://github.com/curl/curl/actions/runs/10710030190/job/29704849763
  https://github.com/curl/curl/actions/runs/10705668808/job/29681622329
- GHA/MSVC: 3m22 -> 13s (run tests: 9m43 -> 4m22)
  https://github.com/curl/curl/actions/runs/10710030190/job/29704850411
  https://github.com/curl/curl/actions/runs/10705668808/job/29681623313
- AppVeyor CI MSVC 2008: 4m3 -> 45s (full build)
- AppVeyor CI MSVC 2010: 2m56 -> 1m8 (full build)
- AppVeyor CI MSVC 2022: 10m19 -> 2m23 (full build)
  https://ci.appveyor.com/project/curlorg/curl/builds/50538455
  https://ci.appveyor.com/project/curlorg/curl/builds/50536558
- AppVeyor CI total build time: 10m30 (master) -> 6m48 (unity) -> 4m5 (bundle) -> 3m24 (bundle+unity) -> 5m7 (bundle+unity+all jobs building tests)

Closes #14772
2024-09-22 09:51:15 +02:00
Viktor Szakats
d83b528a80
tidy-up: spelling
C89, Schannel, Secure Transport, contractions.

Cherry-picked from #14692
Closes #14996
2024-09-22 09:51:15 +02:00
Viktor Szakats
c5e3d8ba94
GHA: speed up builds in torture jobs, tidy up
- use cmake with ninja.
- drop unnecessary packages.

Also:
- Linux: switch to Ubuntu 20.24 runner. Initially for newer packages,
  then left it there for variation and as canary.
- Linux: make it easier to enable valgrind by dropping `-n` tflags.
- Linux: show `curl -V` after build.
- Linux: sync job names with macOS.
- Linux, macOS: review and adjust job timeouts.
- Linux, macOS: sync job configs more:
  Linux: enable libssh2, macOS: enable brotli, zstd, libssh2.
- macOS: prefer `source` (over manual `PATH` editing) for `venv` setup.
- macOS: drop redundant `-DENABLE_CURLDEBUG=ON`.
- macOS: drop redundant `-n` tflags. There is no valgrind on macOS.
- macOS: allow overriding test parallelism per job.
- macOS: unlock all disabled tests for torture jobs.

Speed-ups (configure/build stage):
- Linux !FTP: 83s -> 36s
  before: https://github.com/curl/curl/actions/runs/10948030198/job/30398134647
  after: https://github.com/curl/curl/actions/runs/10961987675/job/30440179978?pr=14972
- Linux FTP: 79s -> 32s
  before: https://github.com/curl/curl/actions/runs/10948030198/job/30398137070
  after: https://github.com/curl/curl/actions/runs/10961987675/job/30440182406?pr=14972
- macOS !FTP: 98s -> 29s
  before: https://github.com/curl/curl/actions/runs/10960141275/job/30434081412?pr=14972
  after: https://github.com/curl/curl/actions/runs/10960141275/job/30434082357?pr=14972
- macOS FTP: 129s -> 29s
  before: https://github.com/curl/curl/actions/runs/10960141275/job/30434082041?pr=14972
  after: https://github.com/curl/curl/actions/runs/10960141275/job/30434083050?pr=14972

Closes #14972
2024-09-21 00:32:00 +02:00
Viktor Szakats
60c3d04465
autotools: add support for 'unity' builds, enable in CI
Implement the "unity" builds as known from CMake, but for autotools.
It's limited to `lib` and `src` (CMake also supports it in `tests`).

Enable with: `--enable-unity` (disabled by default)

Unity builds speed up builds significantly. Cygwin and Windows builds in
particular, but the effect is noticeable on most systems. It also allows
discovering unity issues with autotools, benefitting also CMake when
building the same combination. In CI it makes turnaround times quicker.

This closes build performance with CMake. autotools still lags behind
because it builds shared and static libcurl in two, separate passes.
CMake does it in one. Manpage compilation isn't batched, it is in CMake.
After unity and test bundle support the slowest parts of the build are
the configuration phase (which is effectively a tedious, non-parallel,
compilation and/or linking of 300+ tiny programs. The next bottleneck
is compiling individual examples and finally test servers (only slow
with autotools).

The autotools implementation is slightly less efficient than CMake,
because 3 sources are permanently excluded while in CMake this isn't
necessary and solved more efficiently while building libtests. There is
also no 'unity' support for tests, making them a less efficient also.

Enable it in CI for most `configure` jobs. Except in GHA/dist (though
it works fine there too), to use the default config there. Also skip for
the Linux AWC-LC job where it made builds time a few seconds longer
(reason undiscovered.)

Autotools test suite builds compared between master -> `--enable-unity`:
- GHA/Linux: 32s -> 12s
  https://github.com/curl/curl/actions/runs/10705668823/job/29681617374
  https://github.com/curl/curl/actions/runs/10742978889/job/29796766297
- GHA/macOS: 37s -> 10s
  https://github.com/curl/curl/actions/runs/10705668813/job/29681632885
  https://github.com/curl/curl/actions/runs/10742978699/job/29796768875
- GHA/FreeBSD: 15m25 -> 10m58 (full workflow time, ~qemu)
  https://github.com/curl/curl/actions/runs/10705668811/job/29681607915
  https://github.com/curl/curl/actions/runs/10742978937/job/29796766115
- GHA/Cygwin: 3m32 -> 1m21
  https://github.com/curl/curl/actions/runs/10705668809/job/29681609965
  https://github.com/curl/curl/actions/runs/10742978645/job/29796756933
- GHA/MSYS2: 2m42 -> 50s
  https://github.com/curl/curl/actions/runs/10705668808/job/29681621166
  https://github.com/curl/curl/actions/runs/10742978662/job/29799739289
- GHA/mingw-w64: 5m32 -> 1m23
  https://github.com/curl/curl/actions/runs/10705668808/job/29681628787
  https://github.com/curl/curl/actions/runs/10742978662/job/29799741568

Closes #14815
2024-09-20 23:53:33 +02:00
Viktor Szakats
45202cbba4
cmake: separate target for examples, optimize CI, fix fallouts
- Move `docs/examples` builds under a separate target.

- Make `BUILD_EXAMPLES` default to `ON`. It means to generate the rules
  for `docs/examples` by default, but not build them. To build them,
  an explicit `make curl-examples` (or ninja, etc) command is necessary.
  This syncs behaviour with autotools, and also how both cmake and
  autotools are building tests.

- GHA: update cmake jobs to use the new way of building examples.

- GHA: move examples build step at the end of the job, after building
  and running tests. This allows to have build and test run results
  faster, and leave the seldom-changing examples build to the end.
  Building examples is the slowest build step with no practical way to
  make them fast.

- appveyor: enable building examples in two old-MSVC jobs.

- examples: fix examples to build cleanly with old MSVC versions.

- GHA/non-native: move example build log under a GHA foldable section.

- GHA/windows: move building examples into separate step for Linux cross
  jobs.

Follow-up to dfdd978f7c #13491
Closes #14906
2024-09-20 23:53:33 +02:00
Viktor Szakats
e53523fef0
CI: move Azure jobs to GHA, fix fallouts, sshserver, runtests tweaks
CI:

- GHA/windows: enable OpenSSH server, SysInternals `handle`, `impacket`.
  Skip `impacket` on MSYS2 due to install failure.
  Skip OpenSSH server for old/standalone mingw-w64 (building curl
  without SSH there.)
- GHA/windows: make test tool installs a separate step.
- GHA/cygwin: enable OpenSSH server.
  Skip `impacket`: it's compiling for 7 minutes then breaks.
  Skip `stunnel` due to sluggish test run performance.
  (This update is unrelated to Azure jobs.)
- GHA/linux: migrate Linux jobs from Azure CI.
- GHA/linux: migrate scanbuild job from Azure CI.
- GHA/linux: enable libssh2 in a job. Also enable valgrind.
- CI/windows: enable SSPI in two jobs.
- CI/windows: disable zlib in one more job.
- CI/windows: improve `if` condition checking GnuTLS.
- CI/windows: ignore SFTP/SCP tests as necessary.
  - universally ignore SCP tests, they fail everywhere.
  - ignore test 612.
  - ignore test 613 616 618 with MSYS2 mingw-w64.
  - ignore test 614 with libssh.
  - ignore all SFTP with MSYS2 native.
  - ignore all SFTP with vcpkg with `libssh2[core,zlib]`.
  - ignore a couple of SFTP tests with MSYS2 mingw-w64.
  (This matches settings on Azure CI.)
- GHA/windows: ignore failing 1451 'Basic SMB request' test for
  old mingw-w64 7.3.0 (but not for 9.5.0!):
  ```
  2024-09-11 21:45:59,738 ERROR smbComNegotiate: b'NT LM 0.12\x00' is not in list
  [...]
  curl: (7) Could not connect to server
  [...]
  FAIL 1451: 'Basic SMB request' SMB
  ```
  Ref: https://github.com/curl/curl/actions/runs/10816280746/job/30007130770#step:13:3546
- CI/Azure: delete, now moved to GHA.

sshserver:

- sshserver: fix permissions for SSH host key on Windows, allowing sshd
  to launch.
- sshserver: fix initializing config paths with Cygwin/MSYS2 sshd.
- sshserver: fix initializing config paths in parallel builds.
- sshserver: delete redundant `DenyUsers`. This also opens the way
  to allow multiple usernames.
- sshserver: fix `AllowUsers` for GHA/windows, by allowing the
  domainless username again.
  Follow-up to 3ee7c676ec #5721
- sshserver: fix `Deprecated option` warnings in `sshd_config`:
  ```
  D:/a/curl/curl/bld/tests/log/2/server/curl_sshd_config line 7: Deprecated option AuthorizedKeysFile2
  D:/a/curl/curl/bld/tests/log/2/server/curl_sshd_config line 25: Deprecated option KeyRegenerationInterval
  D:/a/curl/curl/bld/tests/log/2/server/curl_sshd_config line 35: Deprecated option RhostsRSAAuthentication
  D:/a/curl/curl/bld/tests/log/2/server/curl_sshd_config line 36: Deprecated option RSAAuthentication
  D:/a/curl/curl/bld/tests/log/2/server/curl_sshd_config line 37: Deprecated option ServerKeyBits
  D:/a/curl/curl/bld/tests/log/2/server/curl_sshd_config line 41: Deprecated option UseLogin
  ```
- sshserver: fix `Deprecated option` warnings in `ssh_config` with
  Cygwin/MSYS2.
- sshserver: fix dumping config files due to the filenames missing their
  full paths.
- sshserver: add workaround to make `logmsg` messages visible.
  Before this patch they only went to a file and never shown.

runtests:

- runtests: log details when these Windows commands are called:
  `handle`, `taskkill`, `tasklist`.
- runtests: add documentation links to Windows tools:
  `handle`, `taskkill`, `tasklist`, `icacls`.
- runtests: add `-t` (kill whole tree) option to `taskkill` in
  `servers.pm`, syncing it with the other `taskkill` call.
  Follow-up to bc72a78a11 #14488
- runtests: show warning if Sysinternals `handle` tool is missing.
- runtests: drop Windows XP Home compatibility `tskill` call.
  The call was made on all Windows versions. It's possibly overkill to
  do this, because XP Home is probably rarely used for running curl
  tests these days. In case it's needed, it'd be better to put it under
  an explicit option.
- runtests: show Perl version and path.

Fix/silence fallouts:

- unit2603: fix building with disabled HTTP support.
- unit2604: silence `-Woverlength-strings` warnings in C89 mode.
- test437, test1614: fix to pass with no-IPv6 builds.

Closes #14859
2024-09-16 10:00:17 +02:00
Viktor Szakats
c3a7145a4f
GHA/macos: tidy-up
- drop redundant `brew autoconf`.
  It's automatically installed via `automake`.
- whitespace
2024-09-15 13:57:20 +02:00
Viktor Szakats
8709404fe9
GHA/macos: make impacket found by tests
Also move impacket installation right before the test run to avoid
spending 10s installing if the build fails.

Closes #14913
2024-09-15 03:12:48 +02:00
Viktor Szakats
c89cc09ab0
GHA/macos: replace make with ninja for cmake builds
Ninja performs better than GNU Make. Also the generation step is a bit
faster. The cost is installing ninja where missing from the image, and
there is no pre-existing package install step. Otherwise the
installation is fast because the tool is tiny.

(It would probably make sense to include ninja on the runner image; it's
a near zero-cost optimization opportunity for all CI jobs using cmake.)

Speed-ups:

Build step: 3s -> 1s.
Build tests step: 37s -> 22s.

before: https://github.com/curl/curl/actions/runs/10865247916/job/30151507582?pr=14911
after: https://github.com/curl/curl/actions/runs/10865301046/job/30151610055?pr=14910

Closes #14910
2024-09-15 03:09:04 +02:00
Viktor Szakats
51fc89965b
GHA/macos: tidy-ups, install impacket for cmake jobs
- install impacket for cmake jobs.

- stop installing autotools for cmake jobs.
  (also saving the extra seconds necessary for impacket install.)

- delete no longer necessary openssl pkg-config patch.

- `brew unlink openssl` unconditionally. This is safe, we no longer need
  it at the default path, and the condition were never met anyway for
  a while now.

- stop installing openssl, it's there on the base image.
  (it was also missed for cmake, but that's okay)

before: https://github.com/curl/curl/actions/runs/10864438947/job/30149718802
after: https://github.com/curl/curl/actions/runs/10865247916/job/30151507582

Closes #14911
2024-09-15 02:34:17 +02:00
Daniel Stenberg
b29caf0ba2
GHA/macOS: add an -e test
Adds -e to runtests for using the event-based API in the tool.

Closes #14876
2024-09-12 11:26:33 +02:00
Viktor Szakats
6004f96734
cmake: default CURL_DISABLE_LDAPS to the value of CURL_DISABLE_LDAP
After this patch LDAPS is disabled by default when LDAP is manually
disabled.

This makes it unnecessary to disable them in sync manually just to avoid
a `CMakeLists.txt` warning.

Syncs behavior with `./configure`.

Closes #14758
2024-09-02 21:03:01 +02:00
Viktor Szakats
d620ec6791
CI: add test timeouts, more cmake build tests, fix VS2010 C warning
- GHA/macos: set timeout for test runs.
  Double the value for autotools to fit torture tests.
- GHA/cygwin: reduce test run timeout for autotools.
- GHA/cygwin: enable building tests with cmake.
- GHA/windows: enable building tests with MSYS cmake.
- GHA/windows: enable building tests with MSVC UWP.
- appveyor: enable building tests with VS2008 x86 and VS2010 x64.
- tests: add workaround compiler warnings when building with VS2010:
  ```
  tests\server\util.c(482): warning C4306: 'type cast' : conversion from 'int' to 'void (__cdecl *)(int)' of greater size
  tests\server\util.c(486): warning C4306: 'type cast' : conversion from 'int' to 'void (__cdecl *)(int)' of greater size
  tests\server\util.c(490): warning C4306: 'type cast' : conversion from 'int' to 'void (__cdecl *)(int)' of greater size
  tests\server\util.c(720): warning C4306: 'type cast' : conversion from 'int' to 'void (__cdecl *)(int)' of greater size
  tests\server\util.c(726): warning C4306: 'type cast' : conversion from 'int' to 'void (__cdecl *)(int)' of greater size
  tests\server\util.c(732): warning C4306: 'type cast' : conversion from 'int' to 'void (__cdecl *)(int)' of greater size
  tests\server\util.c(781): warning C4306: 'type cast' : conversion from 'int' to 'void (__cdecl *)(int)' of greater size
  tests\server\util.c(785): warning C4306: 'type cast' : conversion from 'int' to 'void (__cdecl *)(int)' of greater size
  tests\server\util.c(789): warning C4306: 'type cast' : conversion from 'int' to 'void (__cdecl *)(int)' of greater size
  ```
  Ref: https://ci.appveyor.com/project/curlorg/curl/builds/50485633/job/4ujlwxod3cexmn2q#L1535

With this, tests are built in all GHA cygwin/windows jobs.

Timeouts avoid long runs with runaway issues, example:
https://github.com/curl/curl/actions/runs/10575522173

Closes #14700
2024-08-27 23:40:48 +02:00
Viktor Szakats
2c9331be46
GHA/macos: drop options no longer necessary
Follow-up to 7673c12929 #14674

Closes #14693
2024-08-26 16:31:46 +02:00