- When searching for existing connections, interpret the
default CURL_HTTP_VERSION_2TLS as "anything goes". This
will allow us to reuse HTTP/3 connections better
- add 'http/1.1' as allowed protocol identifier in Alt-Svc
files
- add test_02_0[345] for testing protocol selection on
provided alt-svc files
Fixes#14890
Reported-by: MacKenzie
Closes#14966
Or no-HTTP, no-HSTS builds, also MSH3 builds.
CMake, VS2022, Debug, x64, no SSL, Static, Build-only:
```
C:\projects\curl\tests\unit\unit1660.c(46,1): error C2220: the following warning is treated as an error
C:\projects\curl\tests\unit\unit1660.c(46,1): warning C4702: unreachable code
```
Ref: https://ci.appveyor.com/project/curlorg/curl/builds/50531210/job/cjewvo9agavthr3o#L216
Required-by: #14922
Cherry-picked from #14772Closes#14971
There shouldn't be anything displayed during a normal run, but only if
server debugging is enabled. Also, set log_file to a magic value to
disable it, otherwise impacket installs its own logger that messes with
what we want.
Closes#14950
Collect all ready runners from select() and process in a loop. This
assures fairness in processing among all runners.
Formerly, only the first ready runner in the list of all was processed,
leading to later runners being delayed in processing and reporting
overly long test durations.
Also, reduce the backend idle timeout for the h2/h3 test servers so that
process shutdowns take less time.
Closes#14967
- cmake: drop `configure.os`.
This also includes OS version, but thus far it's not important enough
to include it.
- autotools: drop redundant, autotools-only `{target|host}.vendor`.
(it's part of the triplet in `{target|host}`.)
- swap order to `*.cpu` -> `*.os` to match triplet-order.
- cmake: drop redundant `target`.
It's manually filled and only in a (so far) few CI jobs. Let's revisit
when this becomes useful.
- move `buildinfo.txt` to build root.
- dist: add `buildinfo.txt` to `DISTCLEANFILES`.
- autotools: detect human readable compiler version.
- autotools: replace `XXYY` `compiler.version` with "X.Y"-style.
(also to match cmake.)
- autotools: use distinct `compiler_id` for Apple clang: `APPLECLANG`.
To match cmake and also because the the "X.Y"-style version number
is the Apple version, while `XXYY` was a value roughly translated to
mainline llvm/clang version.
- show buildinfo at the end of the configure stage, when run in CI, or
when `CURL_BUILDINFO` or `CURL_CI` env is set.
Follow-up to 1fdea16846#14802
Assisted-by: Dan Fandrich
Ref: https://github.com/curl/curl/pull/14802#issuecomment-2334942991Closes#14822
Some of the `.in` files were listed in `EXTRA_DIST`. Delete them.
`.in` files (passed to `AC_CONFIG_FILES`) are added automatically
to the distro by autotools.
Closes#14821
- drop `tasklist` call before `taskkill`.
`taskkill` offers two ways to kill a `pid`:
1. `-pid <pid>`
If `<pid>` is missing it returns 128 and outputs:
```
ERROR: The process "<pid>" not found.
```
2. `-fi "PID eq <pid>"`
If `<pid>` is missing, it returns 0 and outputs:
```
INFO: No tasks running with the specified criteria.
```
The curl runner script doesn't check the result of the call and both
stdout and stderr are redirected to NUL.
Meaning the `tasklist` calls pre-verifying if the PID exists are not
necessary and we can drop them to put less strain on the runner
environment.
- log a `taskkill` call missed earlier.
Follow-up to e53523fef0#14859
- streamline `taskkill` calls by using the `-pid` option
(was `-fi <filter-expression>`).
- make `taskkill` in `pidterm()` use `-t` to kill the process tree.
Ref: #11009Closes#14959
- perform torture tests with '-j2' for shorter runtime
- when waiting on test results overly long, log the tests
waited for and eventually log the test log directories
for easier analysis what is wrong in CI jobs.
- sockfilt.c: treat the windows errno 109 (ERROR_BROKEN_PIPE)
as a socket closed by the client and do not exit.
- when verifying https server, do not in addition check
the http server behind it also
- when tearing down the stunnel of a non-responsive https
server, tear down the http server with it
Closes#14960
Use the same fallback for content-disposition cases as for regular -O
Add test692: verify -JO with URL without a file name
Reported-by: Brian Inglis
Fixes#14939Closes#14940
CI:
- GHA/windows: enable OpenSSH server, SysInternals `handle`, `impacket`.
Skip `impacket` on MSYS2 due to install failure.
Skip OpenSSH server for old/standalone mingw-w64 (building curl
without SSH there.)
- GHA/windows: make test tool installs a separate step.
- GHA/cygwin: enable OpenSSH server.
Skip `impacket`: it's compiling for 7 minutes then breaks.
Skip `stunnel` due to sluggish test run performance.
(This update is unrelated to Azure jobs.)
- GHA/linux: migrate Linux jobs from Azure CI.
- GHA/linux: migrate scanbuild job from Azure CI.
- GHA/linux: enable libssh2 in a job. Also enable valgrind.
- CI/windows: enable SSPI in two jobs.
- CI/windows: disable zlib in one more job.
- CI/windows: improve `if` condition checking GnuTLS.
- CI/windows: ignore SFTP/SCP tests as necessary.
- universally ignore SCP tests, they fail everywhere.
- ignore test 612.
- ignore test 613 616 618 with MSYS2 mingw-w64.
- ignore test 614 with libssh.
- ignore all SFTP with MSYS2 native.
- ignore all SFTP with vcpkg with `libssh2[core,zlib]`.
- ignore a couple of SFTP tests with MSYS2 mingw-w64.
(This matches settings on Azure CI.)
- GHA/windows: ignore failing 1451 'Basic SMB request' test for
old mingw-w64 7.3.0 (but not for 9.5.0!):
```
2024-09-11 21:45:59,738 ERROR smbComNegotiate: b'NT LM 0.12\x00' is not in list
[...]
curl: (7) Could not connect to server
[...]
FAIL 1451: 'Basic SMB request' SMB
```
Ref: https://github.com/curl/curl/actions/runs/10816280746/job/30007130770#step:13:3546
- CI/Azure: delete, now moved to GHA.
sshserver:
- sshserver: fix permissions for SSH host key on Windows, allowing sshd
to launch.
- sshserver: fix initializing config paths with Cygwin/MSYS2 sshd.
- sshserver: fix initializing config paths in parallel builds.
- sshserver: delete redundant `DenyUsers`. This also opens the way
to allow multiple usernames.
- sshserver: fix `AllowUsers` for GHA/windows, by allowing the
domainless username again.
Follow-up to 3ee7c676ec#5721
- sshserver: fix `Deprecated option` warnings in `sshd_config`:
```
D:/a/curl/curl/bld/tests/log/2/server/curl_sshd_config line 7: Deprecated option AuthorizedKeysFile2
D:/a/curl/curl/bld/tests/log/2/server/curl_sshd_config line 25: Deprecated option KeyRegenerationInterval
D:/a/curl/curl/bld/tests/log/2/server/curl_sshd_config line 35: Deprecated option RhostsRSAAuthentication
D:/a/curl/curl/bld/tests/log/2/server/curl_sshd_config line 36: Deprecated option RSAAuthentication
D:/a/curl/curl/bld/tests/log/2/server/curl_sshd_config line 37: Deprecated option ServerKeyBits
D:/a/curl/curl/bld/tests/log/2/server/curl_sshd_config line 41: Deprecated option UseLogin
```
- sshserver: fix `Deprecated option` warnings in `ssh_config` with
Cygwin/MSYS2.
- sshserver: fix dumping config files due to the filenames missing their
full paths.
- sshserver: add workaround to make `logmsg` messages visible.
Before this patch they only went to a file and never shown.
runtests:
- runtests: log details when these Windows commands are called:
`handle`, `taskkill`, `tasklist`.
- runtests: add documentation links to Windows tools:
`handle`, `taskkill`, `tasklist`, `icacls`.
- runtests: add `-t` (kill whole tree) option to `taskkill` in
`servers.pm`, syncing it with the other `taskkill` call.
Follow-up to bc72a78a11#14488
- runtests: show warning if Sysinternals `handle` tool is missing.
- runtests: drop Windows XP Home compatibility `tskill` call.
The call was made on all Windows versions. It's possibly overkill to
do this, because XP Home is probably rarely used for running curl
tests these days. In case it's needed, it'd be better to put it under
an explicit option.
- runtests: show Perl version and path.
Fix/silence fallouts:
- unit2603: fix building with disabled HTTP support.
- unit2604: silence `-Woverlength-strings` warnings in C89 mode.
- test437, test1614: fix to pass with no-IPv6 builds.
Closes#14859
Since ASCII transfers on FTP means sending CRLF line endings, we should
still keep converting them to LF-only on platforms where text files
typically do not use CRLF.
This also DOES NOT convert existing CRLF line endings on ASCII uploads
but only does stand-alone LF => CRLF.
Regression from eeb7c12807 shipped in 8.10.0
Reported-by: finkjsc on github
Fixes#14873Closes#14875
- when server are killed by a test case, do not wait for the server lock
file to go away. These tests are mostly about client timeouts and the
server will hang until killed.
- when killing a server successfully, check for a remaining lock file,
log its existence and remove it.
- lower the delay timings on SLOWDOWN by half
- add SLOWDOWNDATA server command to only slow down the FTP data bytes,
not the control ones.
- lower some timeout values
Closes#14835
When uploading data from stdin ('-T -'), and the EOS was only detected
on a 0-length read, the EOS was not forwarded to the filters. This led
HTTP/2 to hang on not forwarding this to the server.
Added test_07_14 to reproduce and verify.
Fixes#14870
Reported-by: nekopsykose on github
Closes#14877
Modern debian reports a soft limit of 134217724 which makes this test
run for minutes to allocate all file descriptors. Impose a cap of 256k,
so we do not run this tests on systems with larger rlimits.
Closes#14857
Do not give up connect on servers that are in draining state. This might
indicate the QUIC server restarting and the UDP packet routing still
hitting the instance shutting down.
Instead keep on connecting until the overall TIMEOUT fires.
Closes#14863
Also:
- read `buildinfo.txt` from `runtests.pl` and dump it to the log.
- cmake: show `CROSS` target flag for cross-builds.
- cmake: add logic to detect arguments passed via the command-line.
It is meant to help filling out missing datapoints in the testclutch
matrix.
Closes#14802
Follow-up to a07ba37b5e which did not
solve the issue of corrent polling for FTP active data connections.
Added test cases for active up-/download.
Closes#14786
Always try ipv6 addresses first, ipv4 second after a delay.
If neither ipv4/6 are amongst the supplied addresses, start a happy
eyeballer for the first address family present. This is for AF_UNIX
connects.
Fixes#14761
Reported-by: janedenone on hackerone
Closes#14768
The SSL_Session object is mutated during connection inside openssl,
and it might not be thread-safe. Besides, according to documentation
of openssl:
```
SSL_SESSION objects keep internal link information about the session
cache list, when being inserted into one SSL_CTX object's session
cache. One SSL_SESSION object, regardless of its reference count,
must therefore only be used with one SSL_CTX object (and the SSL
objects created from this SSL_CTX object).
```
If I understand correctly, it is not safe to share it even in a
single thread.
Instead, serialize the SSL_SESSION before adding it to the cache,
and deserialize it after retrieving it from the cache, so that no
concurrent write to the same object is infeasible.
Also
- add a ci test for thread sanitizer
- add a test for sharing ssl sessions concurrently
- avoid redefining memory functions when not building libcurl, but
including the soruce in libtest
- increase the concurrent connections limit in sws
Notice that there are fix for a global data race for openssl which
is not yet release. The fix is cherry pick for the ci test with
thread sanitizer.
d8def79838Closes#14751
Change mingw-w64 printf format checks in public curl headers to use
`__MINGW_PRINTF_FORMAT` instead of `gnu_printf`. This syncs the format
checker with format string macros published via `curl/system.h`. (Also
disable format checks for mingw-w64 older than 3.0.0 (2013-09-20) and
classic-mingw, which do not support this macro.)
This fixes bogus format checker `-Wformat` warnings in 3rd party code
using curl format strings with the curl printf functions, when using
mingw-w64 7.0.0 (2019-11-10) and older (with GCC, MSVCRT).
It also allows to delete two workaounds for this within curl itself:
- setting `-D__USE_MINGW_ANSI_STDIO=1` for mingw-w64 via cmake and
configure for `docs/examples` and `tests/http/clients`.
Ref: c730c8549b#14640
The format check macro is incompatible (depending on mingw-w64 version
and configuration) with the C99 `%z` (`size_t`) format string used
internally by curl.
To work around this problem, override the format check style in curl
public headers to use `gnu_printf`. This is compatible with `%z` in all
mingw-w64 versions and allows keeping the C99 format strings internally.
Also:
- lib/ws.c: add missing space to an error message.
- docs/examples/ftpgetinfo.c: fix to use standard printf.
Ref: #14643 (take 1)
Follow-up to 3829759bd0#12489Closes#14703
Existing C macro lacked build-level counterparts.
Add them in this patch.
- cmake: `-DCURL_DISABLE_SHA512_256=ON`
- autotools: `--disable-sha512-256`
Also drop the checker exception from `test1165.pl`.
Follow-up to cbe41d151d#12897Closes#14753
Also fix issues:
- cmake: fix `CURL_DISABLE_HTTP_AUTH` option
- cmake: fix `CURL_DISABLE_SHUFFLE_DNS` option
Fixes:
```
Present in CMakeLists.txt, not propagated via curl_config.h.cmake: CURL_DISABLE_HTTP_AUTH
Present in CMakeLists.txt, not propagated via curl_config.h.cmake: CURL_DISABLE_SHUFFLE_DNS
```
Ref: https://github.com/curl/curl/actions/runs/10655027540/job/29532054141?pr=14754#step:11:2090Closes#14754
Some POP3 commands are multi-line, e.g. have responses terminated by a
last line with '.', but some are not. Define the known command
properties and fix response handling.
Add test case for STAT.
Fixes#14677
Reported-by: ralfjunker on github
Closes#14707
When a markdown quoted section using 4-space indentation was converted
to nroff, managen previously caused a newline to appear after the
leading .nf. This fix makes sure that newline is inserted *before* .nf
as intended.
This is perhaps most notable in the HTML version of rendered manpages if
the quoted sections use different colors or similar.
Closes#14732
All sections defined with the mode="text" attribute now get line endings
normalized so that comparisons become line ending agnostic. Removes the
previous problem of figuring out how exactly different Windows
environments should be treated in this regard.
Closes#14717
Extended the test format and runtest.pl so that the verify/upload part
can be marked using crlf newlines even when the client/file does not
have it.
Closes#14717
Update the script for test 1013 and 1014 to require:
- case-sensitive match for the curl feature list.
(Continue to allow case-difference for protocols. They've always been
in uppercase within curl config.)
- matching order for the protocol list.
(Continue to allow any order for features. autotools builds on
platforms without `sort -f` need it. E.g. Old Linux CI)
Also:
- fix casing of the `gsasl` feature in `configure`, to match `curl -V`
and cmake.
- delete obsolete comment.
Closes#14706
Adds test cases to check that plain http: with HTTP/2 works
via 'Upgrade: h2c' or --http2-prior-knowledge'.
Also added tests to check connection reused in these situations.
Closes#14694
Update IP related information at the connection and the transfer in two
places only: once the filter chain connects and when a transfer is added
to a connection. The latter only updates on reuse when the filters
already are connected.
The only user of that information before a full connect is the HAProxy
filter. Add cfilter CF_QUERY_IP_INFO query to let it find the
information from the filters "below".
This solves two issues with the previous version:
- updates where often done twice with the same info
- happy eyeballing filter "forks" could overwrite each others
updates before the full winner was determined.
Closes#14699
This is a better match for what they do and the general "cpool"
var/function prefix works well.
The pool now handles very long hostnames correctly.
The following changes have been made:
* 'struct connectdata', e.g. connections, keep new members
named `destination` and ' destination_len' that fully specifies
interface+port+hostname of where the connection is going to.
This is used in the pool for "bundling" of connections with
the same destination. There is no limit on the length any more.
* Locking: all locks are done inside conncache.c when calling
into the pool and released on return. This eliminates hazards
of the callers keeping track.
* 'struct connectbundle' is now internal to the pool. It is no
longer referenced by a connection.
* 'bundle->multiuse' no longer exists. HTTP/2 and 3 and TLS filters
no longer need to set it. Instead, the multi checks on leaving
MSTATE_CONNECT or MSTATE_CONNECTING if the connection is now
multiplexed and new, e.g. not conn->bits.reuse. In that case
the processing of pending handles is triggered.
* The pool's init is provided with a callback to invoke on all
connections being discarded. This allows the cleanups in
`Curl_disconnect` to run, wherever it is decided to retire
a connection.
* Several pool operations can now be fully done with one call.
Pruning dead connections, upkeep and checks on pool limits
can now directly discard connections and need no longer return
those to the caller for doing that (as we have now the callback
described above).
* Finding a connection for reuse is now done via `Curl_cpool_find()`
and the caller provides callbacks to evaluate the connection
candidates.
* The 'Curl_cpool_check_limits()' now directly uses the max values
that may be set in the transfer's multi. No need to pass them
around. Curl_multi_max_host_connections() and
Curl_multi_max_total_connections() are gone.
* Add method 'Curl_node_llist()' to get the llist a node is in.
Used in cpool to verify connection are indeed in the list (or
not in any list) as they need to.
I left the conncache.[ch] as is for now and also did not touch the
documentation. If we update that outside the feature window, we can
do this in a separate PR.
Multi-thread safety is not achieved by this PR, but since more details
on how pools operate are now "internal" it is a better starting
point to go for this in the future.
Closes#14662
- GHA/macos: set timeout for test runs.
Double the value for autotools to fit torture tests.
- GHA/cygwin: reduce test run timeout for autotools.
- GHA/cygwin: enable building tests with cmake.
- GHA/windows: enable building tests with MSYS cmake.
- GHA/windows: enable building tests with MSVC UWP.
- appveyor: enable building tests with VS2008 x86 and VS2010 x64.
- tests: add workaround compiler warnings when building with VS2010:
```
tests\server\util.c(482): warning C4306: 'type cast' : conversion from 'int' to 'void (__cdecl *)(int)' of greater size
tests\server\util.c(486): warning C4306: 'type cast' : conversion from 'int' to 'void (__cdecl *)(int)' of greater size
tests\server\util.c(490): warning C4306: 'type cast' : conversion from 'int' to 'void (__cdecl *)(int)' of greater size
tests\server\util.c(720): warning C4306: 'type cast' : conversion from 'int' to 'void (__cdecl *)(int)' of greater size
tests\server\util.c(726): warning C4306: 'type cast' : conversion from 'int' to 'void (__cdecl *)(int)' of greater size
tests\server\util.c(732): warning C4306: 'type cast' : conversion from 'int' to 'void (__cdecl *)(int)' of greater size
tests\server\util.c(781): warning C4306: 'type cast' : conversion from 'int' to 'void (__cdecl *)(int)' of greater size
tests\server\util.c(785): warning C4306: 'type cast' : conversion from 'int' to 'void (__cdecl *)(int)' of greater size
tests\server\util.c(789): warning C4306: 'type cast' : conversion from 'int' to 'void (__cdecl *)(int)' of greater size
```
Ref: https://ci.appveyor.com/project/curlorg/curl/builds/50485633/job/4ujlwxod3cexmn2q#L1535
With this, tests are built in all GHA cygwin/windows jobs.
Timeouts avoid long runs with runaway issues, example:
https://github.com/curl/curl/actions/runs/10575522173Closes#14700
Before this patch, `libhostname.so` and `chkhostname` were a test
facility for overriding `gethostname()` in non-debug builds on
Linux and other Unix platforms supporting `LD_PRELOAD`.
`gethostname()` has a single use with SMTP.
The alternative way to override `gethostname()` is building in debug
mode, which allows to do this via the `CURL_GETHOSTNAME` env, on all
platforms.
Drop the `LD_PRELOAD` solution in favour of the above.
Also:
- delete inactive NTLM code with a `gethostname()` call made from it.
- streamline NTLM code by dropping a `printf()` and a macro.
- tests: stop setting `CURL_GETHOSTNAME` where unnecessary.
Closes#14695
- show `OpenSSL v3+` when detected (as in `./configure`).
(this string also makes its way to `curl-config`.)
- prefer `unset(VAR)` over `set(VAR)`.
Same effect, but `unset()` tells the intent unambiguously.
https://cmake.org/cmake/help/latest/command/set.html
- drop "implementation" from an `option()` description.
- FindGSS: replace legacy keyword alias with modern alternative.
https://cmake.org/cmake/help/latest/command/get_filename_component.html
- move `CURL_STATIC_CRT` logic next to its `option()`.
- improve order of `libcurl.pc`/`curl-config` variable init lines.
- tests: drop/shorten custom target names.
They inflated generated make files by 550KB.
Keep target name logic for sync between code snippets.
Follow-up to a2ef5d36b3#14660
- clear a variable after use.
- restore `STATUS` for `Features:`/`Protocols:` `message()`s:
Without it the output goes to stderr, and appears in red in CMake GUI.
It doesn't seem possible to show a line on stdout without leading
underscores to match `curl -V` and `./configure` output.
Partial revert of acbc6b703f#14197
- WindowsCache: move `HAVE_LINUX_TCP_H` into the header group.
- move strings to the same line as their `STRING` keyword.
- formatting in generated code.
- delete bogus comment.
- unfold lines for readability.
- fix a too long line. (for cmakelint)
- missing quotes, whitespace, comments.
Closes#14610
The missing message only uses a single line now and has a consistent
prefix making it easier for log parsers to skip. Remove a precheck test
looking for impacket in test1451 because it's incomplete, unnecessary
and in the wrong layer; the test harness will automatically discover
that the smbserver won't start and the test was only checking a single
dependency. Also update the tests README to no longer mention Python 2.
The servers were switched to Python 3 several years ago and no attempt
is being made to maintain Python 2 compatibility
Closes#14669
- reuse local variable names.
- sync `PROJECT_LABEL`, add where missing.
- namespace all target names.
- bind header directories to each target.
- tests/server: limit `CURL_STATICLIB` to Windows (as in autotools.)
- drop functions with a single caller.
Closes#14660
It was previously wrongly verifying the input in its URL encoded format
when setting the hostname component with curl_url_set(), so it wrongly
rejected '%'.
Now it URL decodes the name appropriately before the check.
Added tests to lib1560 to verify that a fine %-code is okay and that a
bad %-code (that decodes to '%') is rejected.
Regression from 0a0c9b6dfa, shipped in 8.0.0
Fixes#14656
Reported-by: Venkat Krishna R
Closes#14657
- Install stunnel.
- Regenerate certificates (as SecureTransport requires a validity period
less than 398 days).
- Restart server if it is unresponsive.
- Do not hardcode the SHA-256 base64 public pinned key.
- Ignore test 313 as SecureTransport does not support crl file.
- Ignore tests 1631 and 1632 as SecureTransport is not yet able to shut
down FTP over HTTPS gracefully.
- Add a CMake target for generating certificates.
Closes#14486
Previously this test allowed several error values when setting options.
This made this test miss #14629.
Now, errors are generally not accepted for setopts:
- numerical setopts accept CURLE_BAD_FUNCTION_ARGUMENT for funny input
- the first setopt to an option accepts CURLE_NOT_BUILT_IN or
CURLE_UNKNOWN_OPTION for when they are disabled/not built-in
- there is an allowlist concept for some return code for some variables,
managed at the top of the mk-lib1521.pl script
In curl.h: remove the OBSOLETE named values from the setopt list.
Closes#14634
Running the http pytest had to be done from tests directory or above,
because the repeat argument fixture was defined in tests/conftest.py.
However, the repeat argument is not needed because its functionality
can be provided by pytest-repeat as documented in the test's
README.md. So, removed the pytest_addoption function for the repeat
argument and the pytest_report_header function is moved to
tests/http/conftest.py.
TODO: Remove repeat argument from all tests. As a stopgap, a
one-element list is defined for it for now.
Closes#14611
Add tweak for mingw-w64 when building tests/http/client programs to
avoid a bogus `-Wformat` warning when using mingw-w64 v7.0.0 or older.
The warning is bogus because these programs use curl's `printf()`
implementation that is guaranteed to support that format spec.
Add this for both CMake and autotools. (But only CMake is CI tested with
an old toolchain.)
Apply the workaround to `docs/examples`, and fix an example to use
curl's `printf()` with `CURL_FORMAT_CURL_OFF_T`.
Reintroduce curl `printf()` calls into `tests/http/client`, via #14625.
Also restore large number masks to a printf, changed earlier in #14382.
Follow-up to 232302f88a#14382
Ref: https://github.com/curl/curl/pull/14625#issuecomment-2302361737Closes#14640
Sources used `lib/curlx.h` with both `ENABLE_CURLX_PRINTF` set and unset
before including it.
In a cmake "unity" batch where the first included source had it unset,
the next sources did not get the macros requested with
`ENABLE_CURLX_PRINTF` because `lib/curl.x` had already been included
without them.
Fix it by by making the macros enabled permanently and globally for
internal sources, and dropping `ENABLE_CURLX_PRINTF`.
This came up while testing unity builds with smaller batches. The full,
default unity build where all `src` is bundled up in a single unit, was
not affected.
Fixes:
```
$ cmake -B build -DCMAKE_UNITY_BUILD=ON -DCMAKE_UNITY_BUILD_BATCH_SIZE=15
$ make -C build
...
curl/src/tool_getparam.c: In function ‘getparameter’:
curl/src/tool_getparam.c:2409:11: error: implicit declaration of function ‘msnprintf’; did you mean ‘vsnprintf’? [-Wimplicit-function-declaration]
2409 | msnprintf(buffer, sizeof(buffer), "%" CURL_FORMAT_CURL_OFF_T "-",
| ^~~~~~~~~
| vsnprintf
curl/src/tool_getparam.c:2409:11: warning: nested extern declaration of ‘msnprintf’ [-Wnested-externs]
[...]
```
Reported-by: Daniel Stenberg
Bug: https://github.com/curl/curl/pull/14626#issuecomment-2301663491Closes#14632
Change TLS proto version on the test httpd server to test setting
combinations of --tls13-ciphers and --ciphers.
To not let the changed config of the httpd server bleed into the next
test, clean and reload on each test. Because a reload is slow, only
do this if the config is different than the loaded config. For this
the httpd.reload_if_config_changed() method is added.
Overloading of autouse fixtures does not seem to work. For the test
httpd server to be reloaded with a clean config in test_18_methods,
to not be affected by the config changes in test_17_ssl_use, the two
class scope fixtures of test_18_methods are now combined.
Closes#14589
- Renames Curl_readwrite() to Curl_sendrecv() to reflect that it
is mainly about talking to the server, not reads or writes to the
client. Add a `nowp` parameter since the single caller already
has this.
- Curl_sendrecv() now runs all possible operations whenever it is
called and either it had been polling sockets or the 'select_bits'
are set.
POLL_IN/POLL_OUT are not always directly related to send/recv
operations. Filters like HTTP/2, QUIC or TLS may monitor reverse
directions. If a transfer does not want to send (KEEP_SEND), it
will not do so, as before. Same for receives.
- Curl_update_timer() now checks the absolute timestamp of an expiry
and the last/new timeout to determine if the application needs
to stop/start/restart its timer. This fixes edge cases where
updates did not happen as they should have.
- improved --test-event curl_easy_perform() simulation to handle
situations where no sockets are registered but a timeout is
in place.
- fixed bug in events_socket() that complained about removing
a socket that was unknown, when indeed it had removed the socket
just before, only it was the last in the list
- fixed conncache's internal handle to carry the multi instance
(where the cache has one) so that operations on the closure handle
trigger event callbacks correctly.
- fixed conncache to not POLL_REMOVE a socket twice when a conneciton
was closed.
Closes#14561
- move variable dump to a GHA foldable group.
- minimize scope for an include().
- rename `HIDES_CURL_PRIVATE_SYMBOLS` to `CURL_HIDES_PRIVATE_SYMBOLS`,
to keep it in the curl namespace.
- drop quotes from a version number.
- add missing `Makefile.inc` var refs to comment.
- FindNGTCP2: rename internal var to underscore/lowercase.
- FindBearSSL, FindGSS: whitespace.
Closes#14571
- add set/get functions for the custom data in a tree node
- use Curl_timediff for time comparisons instead of the custom macro, as they
do the exact same things.
- add asserts to catch mistakes better
- updated test 1309 accordingly
Closes#14562
urle.scheme, urle.user, urle.password and urle.options mistakenly
operated on the original URL instead of the *effective* (last) URL.
Add test 474 to verify.
Reported-by: Gruber Glass
Fixes#14550Closes#14560
- set `CARES_NO_DEPRECATED` to avoid c-ares deprecation warnings.
Like autotools does.
- drop unused c-ares header directory when building libtests and test
servers.
- disable TrackMemory (aka `CURLDEBUG`) feature for MSVC builds.
It fails on test 558 and 1330, saying that TrackMemory isn't working.
Left a FIXME about it.
- GHA/macos: enable c-ares in a cmake job.
- GHA/windows: enable c-ares in MSVC job.
Fixes#14202
- GHA/windows: add c-ares mingw autotools job with tests.
(move `--with-windows-unicode` option from 'default' job to
this one to keep the former "default".)
Put these tests on ignore for now:
```
FAIL 472: 'aws-sigv4 with query' HTTP, aws-sigv4
FAIL 1299: 'Send "OPTIONS *" with --request-target' HTTP, --request-target
FAIL 1613: 'Send "OPTIONS *" with --request-target to a proxy' HTTP, HTTP proxy, --request-target
```
Ref: https://github.com/curl/curl/actions/runs/10388126947/job/28765761441#step:15:5902
472: fails with mingw despite Unicode enabled.
1299, 1613: seem like a case of expanding '*' to a filename.
- GHA/windows: enable c-ares in mingw cmake job.
Closes#14541
- converted the Curl_hash_count() macro to a function
- Discourage accessing struct fields directly
- Document the internal API in HASH.md
Closes#14503
- bearssl, c-ares, gss, libpsl, libssh2, mbedtls:
Before this patch these Find modules returned results via
`<NAME>_INCLUDE_DIR` and `<NAME>_LIBRARY`.
This patch makes them return `<NAME>_INCLUDE_DIRS` (note the `S`)
and `<NAME>_LIBRARIES` like other modules already did.
- bearssl, mbedtls:
Before this patch these Find modules allowed custom configuration
via `<NAME>_INCLUDE_DIRS` (note the `S`).
This patch makes them accept `<NAME>_INCLUDE_DIR`, like the rest of
the modules did.
Deprecate the old variables, but keep accepting them for
compatibility.
- bearssl: add missing `mark_as_advanced()` call.
Closes#14542
Remove check of server output as upload may abort before request
could fully be sent, so server output may be completely missing.
Test already used a 1 second delay to mitigate timing. This change
makes timing no longer an issue.
Closes#14525
Configure the AES 256 instead of the AES 128 cipher in the test httpd to
make scorecard testing between httpd and caddy more comparable.
Adapt test_17 expectations, now that AES 128 can no longer, but 256 can
now be negotiated.
Closes#14502
List tests explicitly ignored, but passing. It can be useful to catch
ignore rules that are no longer necessary because the tests are not
failing anymore.
To not clutter the default view, hide this information under a GitHub
Actions foldable group.
Closes#14457
`SHA512_256_BLOCK_SIZE`, `SHA512_256_DIGEST_SIZE` macros were both
defined within curl and also in the nettle library required by GnuTLS.
Fix it by namespacing the curl macros.
Cherry-picked from #14495Closes#14514
Already used in `vtls.h`. Prefer this curl-namespaced name over the
unprefixed `SHA256_DIGEST_LENGTH`. The latter is also defined by TLS
backends with a potential to cause issues.
Also stop relying on externel headers setting this constant. It's
already defined in `vtls.h` on curl's behalf, do this also for `lib`.
Cherry-picked from #14495Closes#14513
Silence bogus MSVC warning C4232. Use the method already used
for similar cases earlier.
Also fixup existing suppressions to use pragma push/pop.
```
lib\vquic\curl_ngtcp2.c(709,40): error C2220: the following warning is treated as an error
lib\vquic\curl_ngtcp2.c(709,40): warning C4232: nonstandard extension used: 'client_initial': address of dllimport 'ngtcp2_crypto_client_initial_cb' is not static, identity not guaranteed
lib\vquic\curl_ngtcp2.c(709,40): warning C4232: nonstandard extension used: 'recv_crypto_data': address of dllimport 'ngtcp2_crypto_recv_crypto_data_cb' is not static, identity not guaran
lib\vquic\curl_ngtcp2.c(709,40): warning C4232: nonstandard extension used: 'encrypt': address of dllimport 'ngtcp2_crypto_encrypt_cb' is not static, identity not guaranteed
lib\vquic\curl_ngtcp2.c(709,40): warning C4232: nonstandard extension used: 'decrypt': address of dllimport 'ngtcp2_crypto_decrypt_cb' is not static, identity not guaranteed
lib\vquic\curl_ngtcp2.c(709,40): warning C4232: nonstandard extension used: 'hp_mask': address of dllimport 'ngtcp2_crypto_hp_mask_cb' is not static, identity not guaranteed
lib\vquic\curl_ngtcp2.c(709,40): warning C4232: nonstandard extension used: 'recv_retry': address of dllimport 'ngtcp2_crypto_recv_retry_cb' is not static, identity not guaranteed
lib\vquic\curl_ngtcp2.c(709,40): warning C4232: nonstandard extension used: 'update_key': address of dllimport 'ngtcp2_crypto_update_key_cb' is not static, identity not guaranteed
lib\vquic\curl_ngtcp2.c(709,40): warning C4232: nonstandard extension used: 'delete_crypto_aead_ctx': address of dllimport 'ngtcp2_crypto_delete_crypto_aead_ctx_cb' is not static, identit
lib\vquic\curl_ngtcp2.c(709,40): warning C4232: nonstandard extension used: 'delete_crypto_cipher_ctx': address of dllimport 'ngtcp2_crypto_delete_crypto_cipher_ctx_cb' is not static, ide
lib\vquic\curl_ngtcp2.c(709,40): warning C4232: nonstandard extension used: 'get_path_challenge_data': address of dllimport 'ngtcp2_crypto_get_path_challenge_data_cb' is not static, ident
```
Ref: https://github.com/curl/curl/actions/runs/10343459009/job/28627621355#step:10:30
Cherry-picked from #14495
Co-authored-by: Tal Regev
Ref: #14383Closes#14510
- make sure to exclude failing tests when libidn2 is detected by
default.
- ignore test 1560 results. Seen to fail with libidn2.
I'm not sure why this test was not executed earlier:
https://github.com/curl/curl/actions/runs/10354610889/job/28660309355#step:13:3647
- runtests: recognize `libidn2` as a feature.
- move IDN test exclusions from GHA/windows to `tests/data/DISABLED`.
- GHA/windows: drop default `-DUSE_LIBIDN2=ON` cmake config.
Cherry-picked from #14495Closes#14519
Before, setting CURLOPT_SSLVERSION with wolfSSL restricted the the tls
proto to just the specified version. Now it properly supports a range.
So it can set the min and max tls proto (max requires wolfSSL 4.2.0).
Bump the absolute minimum required version of wolfSSL to 3.4.6 (released
2015) because it is needed for the wolfSSL_CTX_SetMinVersion() function.
Closes#14480
Rename internal macros to match their `libcurl.pc` metadata counterpart.
Also apply these to the `curl-config.in` template.
- `CPPFLAG_CURL_STATICLIB` -> `LIBCURL_PC_CFLAGS`
- `LIBCURL_LIBS` -> `LIBCURL_PC_LIBS_PRIVATE`
- `LIBCURL_NO_SHARED` -> `LIBCURL_PC_LIBS`
Closes#14476
- Turned them all into functions to also do asserts etc.
- The llist related structs got all their fields renamed in order to make
sure no existing code remains using direct access.
- Each list node struct now points back to the list it "lives in", so
Curl_node_remove() no longer needs the list pointer.
- Rename the node struct and some of the access functions.
- Added lots of ASSERTs to verify API being used correctly
- Fix some cases of API misuse
Add docs/LLIST.md documenting the internal linked list API.
Closes#14485
- prefix local variables with underscore and convert to lowercase.
- list variables accepted by `libcurl.pc` and `curl-config` templates.
- quote more string literals.
Follow-up to 919394ee64#14450Closes#14462
Log progress only at start and end of transfer to give normalized
output when upload data is only partially sent or temporarily blocked.
Fixes test with CURL_DBG_SOCK_WBLOCK=90 set.
Closes#14454
- tidy up two `MATCHES` expression by avoiding macros expansion and
adding quotes. Then convert then to `STREQUAL` to match other places
in the code doing the same checks.
- fix setting `_ALL_SOURCE` for AIX to match what autotools does.
- delete stray `_ALL_SOURCE` reference from `lib/config_riscos.h`
- simplify/fix two `STREQUAL ""` checks.
The one in the `openssl_check_symbol_exists()` macro succeeded
regardless of the value. The other could return TRUE when
`CMAKE_OSX_SYSROOT` was undefined.
- delete code for CMake versions (<3.7) we no longer support.
- prefer `LIST(APPEND ...)` to extend `CURL_LIBS`.
- use `CURL_LIBS` to add the `network` lib for Haiku.
Before this patch it was done via raw C flags. I could not test this.
- move `_WIN32_WINNT`-related code next to each other.
It also moves detection to the top, allowing more code to use
the result.
- merge two `WIN32` blocks.
- rename internal variables to underscore + lowercase.
- unwrap a line, indent, whitespace.
Closes#14450
- quote string literals.
In the hope it improves syntax-highlighting and readability.
- use lowercase, underscore-prefixed local var names.
As a hint for scope, to help readability.
- prefer `pkg_search_module` (over `pkg_check_modules`).
They are the same, but `pkg_search_module` stops searching
at the first hit.
- more `IN LISTS` in `foreach()`.
- OtherTests.cmake: clear `CMAKE_EXTRA_INCLUDE_FILES` after use.
- add `PROJECT_LABEL` for http/client and unit test targets.
- sync `Find*` module comments and formatting.
- drop a few local variables.
- drop bogus `CARES_LIBRARIES` from comment.
- unquote numeric literal.
Follow-up to acbc6b703f#14197Closes#14388
- rely on the new flush to handle blocked sends. No longer
do simulated EAGAIN on (partially) blocked sends with their
need to handle repeats.
- fix some debug handling CURL_SMALLREQSEND env var
- add some assertings in request.c for affirming we do it right
- enhance assertion output in test_16 for easier analysis
Closes#14435
This disambiguates the source code being tested. The output format is
the same as when testing out of a git repo, but with no description and
a long hash.
Ref: #14363Closes#14429
(in debug-builds)
Fix implementation in curl using libuv to process parallel transfers.
Add pytest capabilities to run test cases with --test-event.
- fix uv_timer handling to carry correct 'data' pointing to uv context.
- fix uv_loop handling to reap and add transfers when possible
- fix return code when a transfer errored
Closes#14413
- QUIT is not an important FTP command
- curl only sends it "best effort", meaning it might not be sent
- it is a known "flaky" thing in test output because of this
Closes#14404
- sync build-dir/source-dir header path order with autotools, by
including build-dir first, then source-dir.
This prevents out-of-tree builds breaking due to leftover generated
headers in the source tree.
- tests/unit: move `src` ahead of `libtest` in header path, syncing with
autotools.
- stop adding non-existing generated `include` dir to header path.
There are no generated `include` headers and this directory is either
missing in out-of-tree builds or the same as the one already added
globally via the root `CMakeLists.txt`.
- lib: stop adding a duplicate source include directory to the header
path.
It's already added globally via the root `CMakeLists.txt`.
- lib: stop adding the project root to the header path.
- docs/examples: drop internal header paths.
Examples do not and should not use internal headers.
- replace `curl_setup_once.h` in comments with `curl_setup.h`,
the header actually used, and also referred to in autotools comments.
- add comment why we need `src` in include path for `tests/server`.
- add quotes around header directories.
Closes#14416
The documented and mandated step has been to not use buildconf but to
invoke 'autoreconf -fi' for four years already.
This change only drops buildconf from the release tarball, it remains
present in git for now.
Follow-up to 85868537d6Closes#14412
- extend existing Linux workflow with CMake support.
Including running pytest the first time with CMake.
- cmake: generate `tests/config` and `tests/http/config.ini`.
Required for pytest tests.
Uses basic detection logic. Feel free to take it from here.
Also dump config files in a CI step for debugging purposes.
- cmake: build `tests/http/clients` programs.
- fix portability issues with `tests/http/clients` programs.
Some of them use `getopt()`, which is not supported by MSVC.
Fix the rest to compile in CI (old-mingw-w64, MSVC, Windows).
- GHA/linux: add CMake job matching an existing autotools one.
- GHA/linux: test `-DCURL_LIBCURL_VERSIONED_SYMBOLS=ON`
in the new CMake job.
- reorder testdeps to build server, client tests first and then
libtests and units, to catch errors in the more complex/unique
sources earlier.
- sort list in `tests/http/clients/Makefile.inc`.
Closes#14382
If a request containing two headers that have equivalent prefixes (ex.
"x-amz-meta-test:test" and "x-amz-meta-test-two:test2") AWS expects the
header with the shorter name to come first. The previous implementation
used `strcmp` on the full header. Using the example, this would result
in a comparison between the ':' and '-' chars and sort
"x-amz-meta-test-two" before "x-amz-meta-test", which produces a
different "StringToSign" than the one calculated by AWS.
Test 1976 verifies
Closes#14370
Bring setting ciphers with WolfSSL in line with other SSL backends,
to make the curl interface more consistent across the backends.
Now the tls1.3 ciphers are set with the --tls13-ciphers option, when
not set the default tls1.3 ciphers are used. The tls1.2 (1.1, 1.0)
ciphers are set with the --ciphers option, when not set the default
tls1.2 ciphers are used. The ciphers available for the connection
are now a union of the tls1.3 and tls1.2 ciphers.
This changes the behaviour for WolfSSL when --ciphers is set, but
--tls13-ciphers is not set. Now the ciphers set with --ciphers
are combined with the default tls1.3 ciphers, whereas before solely
the ciphers of --ciphers were used.
Thus before when no tls1.3 ciphers were specified in --ciphers,
tls1.3 was completely disabled. This might not be what the user
expected, especially as this does not happen with OpenSSL.
Closes#14385
Bring setting ciphers with mbedTLS in line with other SSL backends,
to make the curl interface more consistent across the backends.
Now the tls1.3 ciphers are set with the --tls13-ciphers option, when
not set the default tls1.3 ciphers are used. The tls1.2 (1.1, 1.0)
ciphers are set with the --ciphers option, when not set the default
tls1.2 ciphers are used. The ciphers available for the connection
are now a union of the tls1.3 and tls1.2 ciphers.
This changes the behaviour for mbedTLS when --ciphers is set, but
--tls13-ciphers is not set. Now the ciphers set with --ciphers
are combined with the default tls1.3 ciphers, whereas before solely
the ciphers of --ciphers were used.
Thus before when no tls1.3 ciphers were specified in --ciphers,
tls1.3 was completely disabled. This might not be what the user
expected, especially as this does not happen with OpenSSL.
Closes#14384
add --with-libuv to configure to (optionally) use it in debug-builds to
drive the event-based API
Use curl_multi_socket_action() and friends to drive parallel transfers.
tests/README has brief documentation for this
Closes#14298
- replace the counting of upload lengths with the new eos send flag
- improve frequency of stream draining to happen less on events where it
is not needed
- this PR is based on #14220
http2, cf-h2-proxy: fix EAGAINed out buffer
- in adjust pollset and shutdown handling, a non-empty `ctx->outbufq`
must trigger send polling, irregardless of http/2 flow control
- in http2, fix retry handling of blocked GOAWAY frame
test case improvement:
- let client 'upload-pausing' handle http versions
Closes#14253
With this option, the entire download is skipped if the selected target
filename already exists when the opertion is about to begin.
Test 994, 995 and 996 verify.
Ref: #11012Closes#13993
revert f6cb3c63#14338
Setting SSLHonorCipherOrder to on means it honors the server cipher
order. From the documentation: "When choosing a cipher during an SSLv3
or TLSv1 handshake, normally the client's preference is used. If this
directive is enabled, the server's preference will be used instead."
Also the commit inhibits test_17_07_ssl_ciphers. The test tries to
tests if all the ciphers specified, and only those, are properly set
in curl. For that to work we need have cases where some or all ciphers
do no intersect with the cipher-set of the server. We need to be able
to assert a failed connection based on a cipher set mismatch.
That is why a restricted set of ciphers is used on the server. This
set is so chosen that it contains the well known most secure ciphers.
Except with the slower aes256 variant intentionally left out, to be
able to test above described.
As test_17_07_ssl_ciphers is currently the only test that tests the
functioning of the --ciphers and --tls13-ciphers options, it is
important that its coverage is as good as possible.
Closes#14381
Use these words and casing more consistently across text, comments and
one curl tool output:
AIX, ALPN, ANSI, BSD, Cygwin, Darwin, FreeBSD, GitHub, HP-UX, Linux,
macOS, MS-DOS, MSYS, MinGW, NTLM, POSIX, Solaris, UNIX, Unix, Unicode,
WINE, WebDAV, Win32, winbind, WinIDN, Windows, Windows CE, Winsock.
Mostly OS names and a few more.
Also a couple of other minor text fixups.
Closes#14360
Since the documentation text blob might be gzipped, it needs to search
for what to output in a streaming manner. It then first searches for
"\nALL OPTIONS".
Then, it looks for the start to display at "\n -[option]" and stops
again at "\n -". Except for the last option in the man page, which
ends at "\nFILES" - the subtitle for the section following all options
in the manpage.
Test 1707 to 1710 verify
Closes#13997
... or pick the last directory part from the path if available.
Instead of returning error.
Add test 690 and 691 to verify. Test 76 and 2036 no longer apply.
Closes#13988
- tidy-up comments.
- use lowercase, underscore prefixed names for internal variables.
- use `IN LISTS` and `IN ITEMS` in `foreach()` loops.
- rename variable name `OUTPUT` to a more distinctive one.
- tidy-up `STREQUAL` syntax.
- delete commented code.
- indent/whitespace.
Closes#14197
Replace Curl_resolv_unlock() with Curl_resolv_unlink():
-replace inuse member with refcount in Curl_dns_entry
- pass Curl_dns_entry ** to unlink, so it gets always cleared
- solve potential (but unlikley) UAF in FTP's handling of looked up
Curl_dns_entry. Esp. do not use addr information after unlinking an entry.
In reality, the unlink will not free memory, as the dns entry is still
referenced by the hostcache. But this is not safe and relying on no other
code pruning the cache in the meantime.
- pass permanent flag when adding a dns entry instead of fixing timestamp
afterwards.
url.c: fold several static *resolve_* functions into one.
Closes#14195
The test-ci target now uses 2 processes by default, but the amount of
parallelism is tuned for each CI service and build environment based on
results of a number of test runs. Some CI services use super-
oversubscribed build machines that can barely run the curl tests
already with no parallelism without frequently failing with
timing-induced failures. These continue to be run without parallelism.
Other services provide two fast, unloaded cores and these run with 14
processes, which is a good default for this kind of environment.
Here's a summary of the number of test processes by CI service:
Appveyor - 2 (Windows MSVC), 1 (others)
Azure - 2
Circle CI - 14
Cirrus - 28 (macOS), 14 (Linux), 7 (FreeBSD), 5 (macOS torture), 2 (Windows)
GitHub Actions - 3 (macOS), 2 (Linux)
Some of these are a bit conservative to keep timing-induced flakiness down.
The net result is that the first test results should arrive only
3 minutes after a commit submission.
Changes merged via separate commits:
- 2a7c8b27fd#14171
- 72341068a2
- efce544418#14244
- c6cf411bac
Ref: #10818Closes#11510
It's a no-op since
d162fca69a#9333 (2022-08-18).
Also revert 476499c75c that is
no longer necessary: move `Makefile.inc` back into `Makefile.am`.
Closes#14357
Raise the limit for certification information from 10 thousand to 100
thousand bytes. Certificates can be larger than 10k.
Change the infof() debug output to add '...' at the end when the max
limit it can handle is exceeded.
Reported-by: Sergio Durigan Junior
Fixes#14352Closes#14354
