As the parsing and address "regeneration" are done anyway, we might as
well use the updated version in the result and thereby A) get a
normalized (and lower cased) version of the address and B) avoid a
strcpy().
Updated test 1560 to verify.
Closes#15143
Due to timings on paused response receive, the error code may vary due
to the location where it is detected that the server closed the transfer
prematurely.
Be more lenient in always allowing PARTIAL_FILE as ok.
Closes#15138
`Curl_bufq_unwrite()` used the head instead of the tail chunk to shrink
the bufq's content. Fix this and add test case that checks correct
behaviour.
Amended test 2601 accordingly.
Reported-by: Chris Stubbs
Closes#15136
Generate the certificate serial numbers automatically instead of doing
from shell (or Perl earlier).
Fixes intermittent CI failures due to the shell-based random generator
generating the same serial number twice:
```
$ openssl ca -config EdelCurlRoot-ca.cnf -revoke Server-localhost0h-sv.crt
Using configuration from EdelCurlRoot-ca.cnf
ERROR:Already revoked, serial number 66FDB23A
make: *** [../../../tests/certs/Server-localhost0h-sv.pem] Error 1
```
https://github.com/curl/curl/actions/runs/11151401083/job/30994755798?pr=15128#step:10:498
Follow-up to fa461b4eff#14486
Follow-up to fa69b41c77#13307Closes#15129
Instead of calling the shell and external tools, rely on Perl functions
like `Cwd::getcwd()`, `Cwd::abs_path()`, `Cygwin::posix_to_win_path()`,
`Cygwin::win_to_posix_path()` to retrieve the current directory and
convert between POSIX and Windows formats.
This adds native Windows Perl support, avoids most failure modes and
makes format guessing and other internal functions unnecessary.
Also:
- delete unused `sys_native_path()`.
- delete redundant `normalize_path()` because Perl `abs_path()` already
does it.
Cherry-picked from #14949Closes#15111
For all tests using -O that were previously relying on a debug build and
the CURL_TESTDIR environment variable, use the plain --output-dir option
instead so that they can run proper in non-debug builds.
Closes#15114
By keeping the headers in memory until we know the target file name,
then output them all.
Previously this option combination would cause an error.
Add test 1310 and 1492 to verify. Adjusted test 1460 to work in the new
conditions.
Closes#15110
poll() on macOS 10.12 was deemed broken in 2016 when we discovered that
it misbehaves when provided with no sockets to wait for. The
HAVE_POLL_FINE is used to mark a poll() implementation that behaves
correctly: it *should* still wait the timeout time.
curl has therefore opted to use select() on Apple operating systems ever
since. To avoid the risk that this or other breakage cause problems.
However, using select() internally is also bad because it suffers from
problems when using file descriptors beyond 1024.
This change makes poll() used if it is present, but if there is no
sockets to wait for it avoids using poll() and instead falls back to
select() - but without any sockets to wait for there is no 1024 problem.
This removes all previous special-handling involving HAVE_POLL_FINE.
ref: https://daniel.haxx.se/blog/2016/10/11/poll-on-mac-10-12-is-broken/Closes#15096
When uploading FTP with unknown length, we write a last 0-length chunk
with the EOS flag set. OpenSSL's SSL_write() errors on such a write.
Skip writing 0-length data to TLS backends instead.
Add test in FTPS for such uploads to verify.
Fixes#15101
Reported-by: Denis Goleshchikhin
Closes#15102
- add variable for the null device filename and use that in Perl code.
- initialize this variable with `NUL` on native Windows.
- add `%DEV_NULL` variable and use it in tests.
Fixes `The system cannot find the path specified.` messages seen when
running `runtests.pl` with native Windows Perl.
Also adjust code to not break mcedit syntax highlighting.
Cherry-picked from #14949Closes#15098
To ensure Perl invocations within tests call the same Perl binary used
for running the tests, as specified or auto-detected via `PERL` env
(autotools) or `PERL_EXECUTABLE` setting (cmake). Instead of the first
`perl` executable found in `PATH`.)
Cherry-picked from #14949Closes#15097
Before this patch not all tests used `%FILE_PWD` with the `file://`
protocol.
Keep `%PWD` for test1145, to keep it fail on Windows like the test
expects.
Cherry-picked from #14949Closes#15090
- CI default: raise parallelism to 20
- CI valgind: set parallelism to 6
- CI non-native: adapt parallelism for OS builds
- CI Windows: no longer ignore FTP, TFTP, MQTT and SMTP
- CI Windows: restrict test timeout to 10 minutes
- CI Windows: do not run tests for msh3 build
- tests, various: restrict curl invocation to ipv4 to avoid talking to
any ipv6 test server running in parallel
- tests: requiring http/2 server no longer needs to mention http server
- test 190: use a fixed timeout of 10 seconds instead of %FTPTIME2, as
that value rises under parallel load beyond what the server is waiting
and then produces different results
- test 1540: add debug logging
- testrunner: add verify checks for http/2 and http/3 server
Closes#15040
Our test servers run either on ipv4 *or* on ipv6, as requested.
A test case using 'localhost' or '*.local' in the url needs to
run with the specific version of the server started.
Otherwise, curl's "happy eyeball"ing will connect to another
server that may be running due to parallel testing or for some
other reasons.
Note that port reuse here depends on the OS strategy and it
seems netbsd is especially likely to hit this.
Closes#15060
Check responsiveness of http/3 server when running.
Also, a test case with http/2 or http/3 server requirement
now implicitly drags in a 'http' server and we need no longer
mention that in testdata.
Closes#15058
The variable FTPTIME2 may, on a loaded test server, become so
large that the timeout does not happen before the fixed 60 seconds
the server waits.
Closes#15056
Windows sometimes has issues when opening the same file twice, so these
test two situations where that could potentially occur.
Reported-by: ralfjunker on github
Ref: #15043Closes#15045
Test expected a connect to a port no one is listening to immediately
fail. But Windows has its internal retry logic that may fail this.
As fix, multi_perform()/multi_wait() until transfer is done.
Closes#15054
SANITIZE_ALLOW_TRUNCATE and SANITIZE_ALLOW_COLONS were never used by
code, thus only making the code complicated for no good use.
Since nothing should truncate, using strncpy() is wrong.
Two cases of malloc + copy replaced with proper strdup() calls.
Fixup unit test 1604 accordingly.
Closes#15047
This is the %CLIENT6IP variable - but without outmost brackets since
some commmand lines need the address without the brackets. With this
variable we can run three more tests without prechecks.
Closes#15039
Also introduce 'notexists' for verifying that directory entries do not
exist after a test. Now an explicit supported feature instead of using
"funny" perl in postcheck.
Closes#15046
It enables running pytests in cmake jobs, regardless of underlying build
tool choice (= makes it work with ninja.)
Also:
- drop pytest logic launching `make` and exiting in case of failure.
Maybe there is a better way and keep this functionality somehow, bind
it to a command-line option? make it fail softly?
- GHA/linux: invoke pytest via the build, not directly.
- autotools: add missing dummy runtests targets when cross-compiling.
Closes#15034
- lib557: suppress `-Wformat-overflow` warning in source.
Fixes:
```
lib557.c: In function ‘test_float_formatting’:
lib557.c:1408:37: error: ‘%*f’ directive output of 2147483648 bytes exceeds ‘INT_MAX’ [-Werror=format-overflow=]
1408 | curl_msnprintf(buf, sizeof(buf), "%*f", INT_MIN, 9.1);
| ^~~
lib557.c:1408:3: note: ‘curl_msnprintf’ output 2147483649 bytes
1408 | curl_msnprintf(buf, sizeof(buf), "%*f", INT_MIN, 9.1);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
```
Ref: https://app.circleci.com/pipelines/github/curl/curl/10226/workflows/87642ee9-cda6-4916-8206-c82aac5f595e/jobs/107669?invite=true#step-106-40996_46
The root cause of why this option gets enabled remains undiscovered.
Reported-by: Daniel Stenberg
Fixes#15008
Follow-up to 71cf0d1fca#14772
- build: drop `-Wno-format-overflow` from picky warning list.
These options only get used with picky warnings enabled.
Follow-up to 145f87b9e8#14598
- unit1652: suppress in source (and not rely on picky warnings anymore.)
Closes#15012
Make `test-ci` not depend on the `testdeps` target.
`test-ci` is designed to run curl tests in CI. In CI we build all
necessary dependencies explicitly beforehand, and they are always ready
when calling the `test-ci` step. Thus, it isn't necessary to enforce
them via a dependency rule. Dropping it saves redundant work and delay
in CI jobs.
The `testdeps` dependency should not normally be a problem. It's
supposed to be a no-op if those targets are already built. In practice
however, it causes a delay and/or an actual rebuild of those
dependencies depending on generator (= build tool) used and other
factors.
As observed in the GHA/windows workflow, the `testdeps` dependency:
- with Ninja, causes no delay, and no extra work:
https://github.com/curl/curl/actions/runs/10980099984/job/30485440389#step:25:18
- with GNU Make, caused a re-evaluation of `testdeps` targets,
but did not actually rebuild them. This re-evaluation took a
noticeable time (esp. with non-bundled tests):
https://github.com/curl/curl/actions/runs/10980099984/job/30485440155#step:14:11 (with bundles)
https://github.com/curl/curl/actions/runs/10973851013/job/30471690331#step:14:11 (w/o bundles)
verbose: https://github.com/curl/curl/actions/runs/10980506956/job/30486434629#step:14:13
- with MSBuild, caused a re-evaluation of `testdeps` targets, and
triggered a _rebuild_:
https://github.com/curl/curl/actions/runs/10980099984/job/30485435968#step:14:19 (with bundles)
https://github.com/curl/curl/actions/runs/10973851013/job/30471689714#step:14:19 (w/o bundles)
verbose: https://github.com/curl/curl/actions/runs/10980506956/job/30486436368#step:14:48
It's suspected that our use of
`-DCMAKE_VS_GLOBALS=TrackFileAccess=false` in CI is contributing to
this. This option is supposed to affect incremental builds only. For
some reason it affects CI builds too, even though they are not
incremental, and no sources are changed between build steps.
Reported-by: Aki Sakurai
Ref: #14999
Notice that `test-*` targets depending on `testdeps` is NOT sufficient
to build everything to run tests, e.g. it misses to build the curl tool,
which is essential. This is also true for autotools' `test-ci` target
which misses to build libcurl and the curl tool.
Perhaps it'd be best to drop `testdeps` as a dependency for _all_
`test-*` targets and make it official to require building dependencies
manually. Alternatively these targets could be fixed to rebuild
everything necessary to run tests.
Closes#15001
Add `CURL_CA_SEARCH_SAFE` build-time option to enable CA bundle search
in the `curl` tool directory. The lookup method was already used to find
`.curlrc` and `_curlrc` (on Windows). On Windows it overrides the unsafe
default `SearchPath()` method.
Enable with:
- cmake: `-DCURL_CA_SEARCH_SAFE=ON`
- autotools: `--enable-ca-search-safe`
- raw: `CPPFLAGS=-DCURL_CA_SEARCH_SAFE`
On Windows, before this patch the whole `PATH` was searched for
a CA bundle. `PATH` may contain unwanted or world-writable locations,
including the current directory. Searching them all is convenient to
pick up any CA bundle, but not secure.
The Muldersoft curl distro implements such CA search via a custom
patch for Windows:
cd652d4792/patch/curl_tool_doswin.diff (L50)
MSYS2/mingw-w64 distro has also been rolling a patch solving this:
https://github.com/msys2/MINGW-packages/blob/master/mingw-w64-curl/0001-Make-cURL-relocatable.patchhttps://github.com/msys2/MINGW-packages/blob/master/mingw-w64-curl/pathtools.c
Also add option to fully disable Windows CA search:
- cmake: `-DCURL_DISABLE_CA_SEARCH=ON`
- autotools: `--disable-ca-search`
- raw: `CPPFLAGS=-DCURL_DISABLE_CA_SEARCH`.
Both options are considered EXPERIMENTAL, with possible incompatible
changes or even (partial) removal in the future, depending on feedback.
An alternative, secure option is to embed the CA bundle into the binary.
Safe search can be extended to other platforms if necessary or useful,
by using `_NSGetExecutablePath()` (macOS),
`/proc/self/exe` (Linux/Cygwin), or `argv[0]`.
Closes#14582
Suppress deprecation warnings the closest to the deprecated code, using
`CURL_IGNORE_DEPRECATION()`. Then drop build-specific suppressions, and
file-wide ones. The latter is not compatible with Unity mode. Also
replace manual suppressions with a macro to apply to all compilers with
deprecation warning support. Also enable deprecation warnings for clang.
- curl/curl.h: enable deprecation warnings for clang.
- docs/examples: stop setting `CURL_DISABLE_DEPRECATION` with autotools.
Suppression moved to C-level earlier. Syncs with cmake.
Follow-up to 5fc61a37c1#14123
- tests/http/clients: stop setting `CURL_DISABLE_DEPRECATION` in
autotools. If it becomes necessary in the future, it can be done in
C via the macro. Syncs with cmake.
- lib1545: stop setting `CURL_DISABLE_DEPRECATION` in autotools.
Drop guard from test source.
Follow-up to 0f10360073#12444
- libtest, unit: replace `CURL_DISABLE_DEPRECATION` with
`CURL_IGNORE_DEPRECATION()`.
- docs/examples: replace pragmas with `CURL_IGNORE_DEPRECATION()`.
Closes#14789
- When searching for existing connections, interpret the
default CURL_HTTP_VERSION_2TLS as "anything goes". This
will allow us to reuse HTTP/3 connections better
- add 'http/1.1' as allowed protocol identifier in Alt-Svc
files
- add test_02_0[345] for testing protocol selection on
provided alt-svc files
Fixes#14890
Reported-by: MacKenzie
Closes#14966
Or no-HTTP, no-HSTS builds, also MSH3 builds.
CMake, VS2022, Debug, x64, no SSL, Static, Build-only:
```
C:\projects\curl\tests\unit\unit1660.c(46,1): error C2220: the following warning is treated as an error
C:\projects\curl\tests\unit\unit1660.c(46,1): warning C4702: unreachable code
```
Ref: https://ci.appveyor.com/project/curlorg/curl/builds/50531210/job/cjewvo9agavthr3o#L216
Required-by: #14922
Cherry-picked from #14772Closes#14971
There shouldn't be anything displayed during a normal run, but only if
server debugging is enabled. Also, set log_file to a magic value to
disable it, otherwise impacket installs its own logger that messes with
what we want.
Closes#14950
Collect all ready runners from select() and process in a loop. This
assures fairness in processing among all runners.
Formerly, only the first ready runner in the list of all was processed,
leading to later runners being delayed in processing and reporting
overly long test durations.
Also, reduce the backend idle timeout for the h2/h3 test servers so that
process shutdowns take less time.
Closes#14967
- cmake: drop `configure.os`.
This also includes OS version, but thus far it's not important enough
to include it.
- autotools: drop redundant, autotools-only `{target|host}.vendor`.
(it's part of the triplet in `{target|host}`.)
- swap order to `*.cpu` -> `*.os` to match triplet-order.
- cmake: drop redundant `target`.
It's manually filled and only in a (so far) few CI jobs. Let's revisit
when this becomes useful.
- move `buildinfo.txt` to build root.
- dist: add `buildinfo.txt` to `DISTCLEANFILES`.
- autotools: detect human readable compiler version.
- autotools: replace `XXYY` `compiler.version` with "X.Y"-style.
(also to match cmake.)
- autotools: use distinct `compiler_id` for Apple clang: `APPLECLANG`.
To match cmake and also because the the "X.Y"-style version number
is the Apple version, while `XXYY` was a value roughly translated to
mainline llvm/clang version.
- show buildinfo at the end of the configure stage, when run in CI, or
when `CURL_BUILDINFO` or `CURL_CI` env is set.
Follow-up to 1fdea16846#14802
Assisted-by: Dan Fandrich
Ref: https://github.com/curl/curl/pull/14802#issuecomment-2334942991Closes#14822
Some of the `.in` files were listed in `EXTRA_DIST`. Delete them.
`.in` files (passed to `AC_CONFIG_FILES`) are added automatically
to the distro by autotools.
Closes#14821
- drop `tasklist` call before `taskkill`.
`taskkill` offers two ways to kill a `pid`:
1. `-pid <pid>`
If `<pid>` is missing it returns 128 and outputs:
```
ERROR: The process "<pid>" not found.
```
2. `-fi "PID eq <pid>"`
If `<pid>` is missing, it returns 0 and outputs:
```
INFO: No tasks running with the specified criteria.
```
The curl runner script doesn't check the result of the call and both
stdout and stderr are redirected to NUL.
Meaning the `tasklist` calls pre-verifying if the PID exists are not
necessary and we can drop them to put less strain on the runner
environment.
- log a `taskkill` call missed earlier.
Follow-up to e53523fef0#14859
- streamline `taskkill` calls by using the `-pid` option
(was `-fi <filter-expression>`).
- make `taskkill` in `pidterm()` use `-t` to kill the process tree.
Ref: #11009Closes#14959
- perform torture tests with '-j2' for shorter runtime
- when waiting on test results overly long, log the tests
waited for and eventually log the test log directories
for easier analysis what is wrong in CI jobs.
- sockfilt.c: treat the windows errno 109 (ERROR_BROKEN_PIPE)
as a socket closed by the client and do not exit.
- when verifying https server, do not in addition check
the http server behind it also
- when tearing down the stunnel of a non-responsive https
server, tear down the http server with it
Closes#14960
Use the same fallback for content-disposition cases as for regular -O
Add test692: verify -JO with URL without a file name
Reported-by: Brian Inglis
Fixes#14939Closes#14940
CI:
- GHA/windows: enable OpenSSH server, SysInternals `handle`, `impacket`.
Skip `impacket` on MSYS2 due to install failure.
Skip OpenSSH server for old/standalone mingw-w64 (building curl
without SSH there.)
- GHA/windows: make test tool installs a separate step.
- GHA/cygwin: enable OpenSSH server.
Skip `impacket`: it's compiling for 7 minutes then breaks.
Skip `stunnel` due to sluggish test run performance.
(This update is unrelated to Azure jobs.)
- GHA/linux: migrate Linux jobs from Azure CI.
- GHA/linux: migrate scanbuild job from Azure CI.
- GHA/linux: enable libssh2 in a job. Also enable valgrind.
- CI/windows: enable SSPI in two jobs.
- CI/windows: disable zlib in one more job.
- CI/windows: improve `if` condition checking GnuTLS.
- CI/windows: ignore SFTP/SCP tests as necessary.
- universally ignore SCP tests, they fail everywhere.
- ignore test 612.
- ignore test 613 616 618 with MSYS2 mingw-w64.
- ignore test 614 with libssh.
- ignore all SFTP with MSYS2 native.
- ignore all SFTP with vcpkg with `libssh2[core,zlib]`.
- ignore a couple of SFTP tests with MSYS2 mingw-w64.
(This matches settings on Azure CI.)
- GHA/windows: ignore failing 1451 'Basic SMB request' test for
old mingw-w64 7.3.0 (but not for 9.5.0!):
```
2024-09-11 21:45:59,738 ERROR smbComNegotiate: b'NT LM 0.12\x00' is not in list
[...]
curl: (7) Could not connect to server
[...]
FAIL 1451: 'Basic SMB request' SMB
```
Ref: https://github.com/curl/curl/actions/runs/10816280746/job/30007130770#step:13:3546
- CI/Azure: delete, now moved to GHA.
sshserver:
- sshserver: fix permissions for SSH host key on Windows, allowing sshd
to launch.
- sshserver: fix initializing config paths with Cygwin/MSYS2 sshd.
- sshserver: fix initializing config paths in parallel builds.
- sshserver: delete redundant `DenyUsers`. This also opens the way
to allow multiple usernames.
- sshserver: fix `AllowUsers` for GHA/windows, by allowing the
domainless username again.
Follow-up to 3ee7c676ec#5721
- sshserver: fix `Deprecated option` warnings in `sshd_config`:
```
D:/a/curl/curl/bld/tests/log/2/server/curl_sshd_config line 7: Deprecated option AuthorizedKeysFile2
D:/a/curl/curl/bld/tests/log/2/server/curl_sshd_config line 25: Deprecated option KeyRegenerationInterval
D:/a/curl/curl/bld/tests/log/2/server/curl_sshd_config line 35: Deprecated option RhostsRSAAuthentication
D:/a/curl/curl/bld/tests/log/2/server/curl_sshd_config line 36: Deprecated option RSAAuthentication
D:/a/curl/curl/bld/tests/log/2/server/curl_sshd_config line 37: Deprecated option ServerKeyBits
D:/a/curl/curl/bld/tests/log/2/server/curl_sshd_config line 41: Deprecated option UseLogin
```
- sshserver: fix `Deprecated option` warnings in `ssh_config` with
Cygwin/MSYS2.
- sshserver: fix dumping config files due to the filenames missing their
full paths.
- sshserver: add workaround to make `logmsg` messages visible.
Before this patch they only went to a file and never shown.
runtests:
- runtests: log details when these Windows commands are called:
`handle`, `taskkill`, `tasklist`.
- runtests: add documentation links to Windows tools:
`handle`, `taskkill`, `tasklist`, `icacls`.
- runtests: add `-t` (kill whole tree) option to `taskkill` in
`servers.pm`, syncing it with the other `taskkill` call.
Follow-up to bc72a78a11#14488
- runtests: show warning if Sysinternals `handle` tool is missing.
- runtests: drop Windows XP Home compatibility `tskill` call.
The call was made on all Windows versions. It's possibly overkill to
do this, because XP Home is probably rarely used for running curl
tests these days. In case it's needed, it'd be better to put it under
an explicit option.
- runtests: show Perl version and path.
Fix/silence fallouts:
- unit2603: fix building with disabled HTTP support.
- unit2604: silence `-Woverlength-strings` warnings in C89 mode.
- test437, test1614: fix to pass with no-IPv6 builds.
Closes#14859
Since ASCII transfers on FTP means sending CRLF line endings, we should
still keep converting them to LF-only on platforms where text files
typically do not use CRLF.
This also DOES NOT convert existing CRLF line endings on ASCII uploads
but only does stand-alone LF => CRLF.
Regression from eeb7c12807 shipped in 8.10.0
Reported-by: finkjsc on github
Fixes#14873Closes#14875
- when server are killed by a test case, do not wait for the server lock
file to go away. These tests are mostly about client timeouts and the
server will hang until killed.
- when killing a server successfully, check for a remaining lock file,
log its existence and remove it.
- lower the delay timings on SLOWDOWN by half
- add SLOWDOWNDATA server command to only slow down the FTP data bytes,
not the control ones.
- lower some timeout values
Closes#14835
When uploading data from stdin ('-T -'), and the EOS was only detected
on a 0-length read, the EOS was not forwarded to the filters. This led
HTTP/2 to hang on not forwarding this to the server.
Added test_07_14 to reproduce and verify.
Fixes#14870
Reported-by: nekopsykose on github
Closes#14877
Modern debian reports a soft limit of 134217724 which makes this test
run for minutes to allocate all file descriptors. Impose a cap of 256k,
so we do not run this tests on systems with larger rlimits.
Closes#14857
Do not give up connect on servers that are in draining state. This might
indicate the QUIC server restarting and the UDP packet routing still
hitting the instance shutting down.
Instead keep on connecting until the overall TIMEOUT fires.
Closes#14863
Also:
- read `buildinfo.txt` from `runtests.pl` and dump it to the log.
- cmake: show `CROSS` target flag for cross-builds.
- cmake: add logic to detect arguments passed via the command-line.
It is meant to help filling out missing datapoints in the testclutch
matrix.
Closes#14802
Follow-up to a07ba37b5e which did not
solve the issue of corrent polling for FTP active data connections.
Added test cases for active up-/download.
Closes#14786
Always try ipv6 addresses first, ipv4 second after a delay.
If neither ipv4/6 are amongst the supplied addresses, start a happy
eyeballer for the first address family present. This is for AF_UNIX
connects.
Fixes#14761
Reported-by: janedenone on hackerone
Closes#14768
The SSL_Session object is mutated during connection inside openssl,
and it might not be thread-safe. Besides, according to documentation
of openssl:
```
SSL_SESSION objects keep internal link information about the session
cache list, when being inserted into one SSL_CTX object's session
cache. One SSL_SESSION object, regardless of its reference count,
must therefore only be used with one SSL_CTX object (and the SSL
objects created from this SSL_CTX object).
```
If I understand correctly, it is not safe to share it even in a
single thread.
Instead, serialize the SSL_SESSION before adding it to the cache,
and deserialize it after retrieving it from the cache, so that no
concurrent write to the same object is infeasible.
Also
- add a ci test for thread sanitizer
- add a test for sharing ssl sessions concurrently
- avoid redefining memory functions when not building libcurl, but
including the soruce in libtest
- increase the concurrent connections limit in sws
Notice that there are fix for a global data race for openssl which
is not yet release. The fix is cherry pick for the ci test with
thread sanitizer.
d8def79838Closes#14751
Change mingw-w64 printf format checks in public curl headers to use
`__MINGW_PRINTF_FORMAT` instead of `gnu_printf`. This syncs the format
checker with format string macros published via `curl/system.h`. (Also
disable format checks for mingw-w64 older than 3.0.0 (2013-09-20) and
classic-mingw, which do not support this macro.)
This fixes bogus format checker `-Wformat` warnings in 3rd party code
using curl format strings with the curl printf functions, when using
mingw-w64 7.0.0 (2019-11-10) and older (with GCC, MSVCRT).
It also allows to delete two workaounds for this within curl itself:
- setting `-D__USE_MINGW_ANSI_STDIO=1` for mingw-w64 via cmake and
configure for `docs/examples` and `tests/http/clients`.
Ref: c730c8549b#14640
The format check macro is incompatible (depending on mingw-w64 version
and configuration) with the C99 `%z` (`size_t`) format string used
internally by curl.
To work around this problem, override the format check style in curl
public headers to use `gnu_printf`. This is compatible with `%z` in all
mingw-w64 versions and allows keeping the C99 format strings internally.
Also:
- lib/ws.c: add missing space to an error message.
- docs/examples/ftpgetinfo.c: fix to use standard printf.
Ref: #14643 (take 1)
Follow-up to 3829759bd0#12489Closes#14703
Existing C macro lacked build-level counterparts.
Add them in this patch.
- cmake: `-DCURL_DISABLE_SHA512_256=ON`
- autotools: `--disable-sha512-256`
Also drop the checker exception from `test1165.pl`.
Follow-up to cbe41d151d#12897Closes#14753
Also fix issues:
- cmake: fix `CURL_DISABLE_HTTP_AUTH` option
- cmake: fix `CURL_DISABLE_SHUFFLE_DNS` option
Fixes:
```
Present in CMakeLists.txt, not propagated via curl_config.h.cmake: CURL_DISABLE_HTTP_AUTH
Present in CMakeLists.txt, not propagated via curl_config.h.cmake: CURL_DISABLE_SHUFFLE_DNS
```
Ref: https://github.com/curl/curl/actions/runs/10655027540/job/29532054141?pr=14754#step:11:2090Closes#14754
Some POP3 commands are multi-line, e.g. have responses terminated by a
last line with '.', but some are not. Define the known command
properties and fix response handling.
Add test case for STAT.
Fixes#14677
Reported-by: ralfjunker on github
Closes#14707
When a markdown quoted section using 4-space indentation was converted
to nroff, managen previously caused a newline to appear after the
leading .nf. This fix makes sure that newline is inserted *before* .nf
as intended.
This is perhaps most notable in the HTML version of rendered manpages if
the quoted sections use different colors or similar.
Closes#14732
All sections defined with the mode="text" attribute now get line endings
normalized so that comparisons become line ending agnostic. Removes the
previous problem of figuring out how exactly different Windows
environments should be treated in this regard.
Closes#14717
Extended the test format and runtest.pl so that the verify/upload part
can be marked using crlf newlines even when the client/file does not
have it.
Closes#14717
Update the script for test 1013 and 1014 to require:
- case-sensitive match for the curl feature list.
(Continue to allow case-difference for protocols. They've always been
in uppercase within curl config.)
- matching order for the protocol list.
(Continue to allow any order for features. autotools builds on
platforms without `sort -f` need it. E.g. Old Linux CI)
Also:
- fix casing of the `gsasl` feature in `configure`, to match `curl -V`
and cmake.
- delete obsolete comment.
Closes#14706
Adds test cases to check that plain http: with HTTP/2 works
via 'Upgrade: h2c' or --http2-prior-knowledge'.
Also added tests to check connection reused in these situations.
Closes#14694
Update IP related information at the connection and the transfer in two
places only: once the filter chain connects and when a transfer is added
to a connection. The latter only updates on reuse when the filters
already are connected.
The only user of that information before a full connect is the HAProxy
filter. Add cfilter CF_QUERY_IP_INFO query to let it find the
information from the filters "below".
This solves two issues with the previous version:
- updates where often done twice with the same info
- happy eyeballing filter "forks" could overwrite each others
updates before the full winner was determined.
Closes#14699
This is a better match for what they do and the general "cpool"
var/function prefix works well.
The pool now handles very long hostnames correctly.
The following changes have been made:
* 'struct connectdata', e.g. connections, keep new members
named `destination` and ' destination_len' that fully specifies
interface+port+hostname of where the connection is going to.
This is used in the pool for "bundling" of connections with
the same destination. There is no limit on the length any more.
* Locking: all locks are done inside conncache.c when calling
into the pool and released on return. This eliminates hazards
of the callers keeping track.
* 'struct connectbundle' is now internal to the pool. It is no
longer referenced by a connection.
* 'bundle->multiuse' no longer exists. HTTP/2 and 3 and TLS filters
no longer need to set it. Instead, the multi checks on leaving
MSTATE_CONNECT or MSTATE_CONNECTING if the connection is now
multiplexed and new, e.g. not conn->bits.reuse. In that case
the processing of pending handles is triggered.
* The pool's init is provided with a callback to invoke on all
connections being discarded. This allows the cleanups in
`Curl_disconnect` to run, wherever it is decided to retire
a connection.
* Several pool operations can now be fully done with one call.
Pruning dead connections, upkeep and checks on pool limits
can now directly discard connections and need no longer return
those to the caller for doing that (as we have now the callback
described above).
* Finding a connection for reuse is now done via `Curl_cpool_find()`
and the caller provides callbacks to evaluate the connection
candidates.
* The 'Curl_cpool_check_limits()' now directly uses the max values
that may be set in the transfer's multi. No need to pass them
around. Curl_multi_max_host_connections() and
Curl_multi_max_total_connections() are gone.
* Add method 'Curl_node_llist()' to get the llist a node is in.
Used in cpool to verify connection are indeed in the list (or
not in any list) as they need to.
I left the conncache.[ch] as is for now and also did not touch the
documentation. If we update that outside the feature window, we can
do this in a separate PR.
Multi-thread safety is not achieved by this PR, but since more details
on how pools operate are now "internal" it is a better starting
point to go for this in the future.
Closes#14662
- GHA/macos: set timeout for test runs.
Double the value for autotools to fit torture tests.
- GHA/cygwin: reduce test run timeout for autotools.
- GHA/cygwin: enable building tests with cmake.
- GHA/windows: enable building tests with MSYS cmake.
- GHA/windows: enable building tests with MSVC UWP.
- appveyor: enable building tests with VS2008 x86 and VS2010 x64.
- tests: add workaround compiler warnings when building with VS2010:
```
tests\server\util.c(482): warning C4306: 'type cast' : conversion from 'int' to 'void (__cdecl *)(int)' of greater size
tests\server\util.c(486): warning C4306: 'type cast' : conversion from 'int' to 'void (__cdecl *)(int)' of greater size
tests\server\util.c(490): warning C4306: 'type cast' : conversion from 'int' to 'void (__cdecl *)(int)' of greater size
tests\server\util.c(720): warning C4306: 'type cast' : conversion from 'int' to 'void (__cdecl *)(int)' of greater size
tests\server\util.c(726): warning C4306: 'type cast' : conversion from 'int' to 'void (__cdecl *)(int)' of greater size
tests\server\util.c(732): warning C4306: 'type cast' : conversion from 'int' to 'void (__cdecl *)(int)' of greater size
tests\server\util.c(781): warning C4306: 'type cast' : conversion from 'int' to 'void (__cdecl *)(int)' of greater size
tests\server\util.c(785): warning C4306: 'type cast' : conversion from 'int' to 'void (__cdecl *)(int)' of greater size
tests\server\util.c(789): warning C4306: 'type cast' : conversion from 'int' to 'void (__cdecl *)(int)' of greater size
```
Ref: https://ci.appveyor.com/project/curlorg/curl/builds/50485633/job/4ujlwxod3cexmn2q#L1535
With this, tests are built in all GHA cygwin/windows jobs.
Timeouts avoid long runs with runaway issues, example:
https://github.com/curl/curl/actions/runs/10575522173Closes#14700
Before this patch, `libhostname.so` and `chkhostname` were a test
facility for overriding `gethostname()` in non-debug builds on
Linux and other Unix platforms supporting `LD_PRELOAD`.
`gethostname()` has a single use with SMTP.
The alternative way to override `gethostname()` is building in debug
mode, which allows to do this via the `CURL_GETHOSTNAME` env, on all
platforms.
Drop the `LD_PRELOAD` solution in favour of the above.
Also:
- delete inactive NTLM code with a `gethostname()` call made from it.
- streamline NTLM code by dropping a `printf()` and a macro.
- tests: stop setting `CURL_GETHOSTNAME` where unnecessary.
Closes#14695
- show `OpenSSL v3+` when detected (as in `./configure`).
(this string also makes its way to `curl-config`.)
- prefer `unset(VAR)` over `set(VAR)`.
Same effect, but `unset()` tells the intent unambiguously.
https://cmake.org/cmake/help/latest/command/set.html
- drop "implementation" from an `option()` description.
- FindGSS: replace legacy keyword alias with modern alternative.
https://cmake.org/cmake/help/latest/command/get_filename_component.html
- move `CURL_STATIC_CRT` logic next to its `option()`.
- improve order of `libcurl.pc`/`curl-config` variable init lines.
- tests: drop/shorten custom target names.
They inflated generated make files by 550KB.
Keep target name logic for sync between code snippets.
Follow-up to a2ef5d36b3#14660
- clear a variable after use.
- restore `STATUS` for `Features:`/`Protocols:` `message()`s:
Without it the output goes to stderr, and appears in red in CMake GUI.
It doesn't seem possible to show a line on stdout without leading
underscores to match `curl -V` and `./configure` output.
Partial revert of acbc6b703f#14197
- WindowsCache: move `HAVE_LINUX_TCP_H` into the header group.
- move strings to the same line as their `STRING` keyword.
- formatting in generated code.
- delete bogus comment.
- unfold lines for readability.
- fix a too long line. (for cmakelint)
- missing quotes, whitespace, comments.
Closes#14610
The missing message only uses a single line now and has a consistent
prefix making it easier for log parsers to skip. Remove a precheck test
looking for impacket in test1451 because it's incomplete, unnecessary
and in the wrong layer; the test harness will automatically discover
that the smbserver won't start and the test was only checking a single
dependency. Also update the tests README to no longer mention Python 2.
The servers were switched to Python 3 several years ago and no attempt
is being made to maintain Python 2 compatibility
Closes#14669
- reuse local variable names.
- sync `PROJECT_LABEL`, add where missing.
- namespace all target names.
- bind header directories to each target.
- tests/server: limit `CURL_STATICLIB` to Windows (as in autotools.)
- drop functions with a single caller.
Closes#14660
It was previously wrongly verifying the input in its URL encoded format
when setting the hostname component with curl_url_set(), so it wrongly
rejected '%'.
Now it URL decodes the name appropriately before the check.
Added tests to lib1560 to verify that a fine %-code is okay and that a
bad %-code (that decodes to '%') is rejected.
Regression from 0a0c9b6dfa, shipped in 8.0.0
Fixes#14656
Reported-by: Venkat Krishna R
Closes#14657
- Install stunnel.
- Regenerate certificates (as SecureTransport requires a validity period
less than 398 days).
- Restart server if it is unresponsive.
- Do not hardcode the SHA-256 base64 public pinned key.
- Ignore test 313 as SecureTransport does not support crl file.
- Ignore tests 1631 and 1632 as SecureTransport is not yet able to shut
down FTP over HTTPS gracefully.
- Add a CMake target for generating certificates.
Closes#14486
Previously this test allowed several error values when setting options.
This made this test miss #14629.
Now, errors are generally not accepted for setopts:
- numerical setopts accept CURLE_BAD_FUNCTION_ARGUMENT for funny input
- the first setopt to an option accepts CURLE_NOT_BUILT_IN or
CURLE_UNKNOWN_OPTION for when they are disabled/not built-in
- there is an allowlist concept for some return code for some variables,
managed at the top of the mk-lib1521.pl script
In curl.h: remove the OBSOLETE named values from the setopt list.
Closes#14634
Running the http pytest had to be done from tests directory or above,
because the repeat argument fixture was defined in tests/conftest.py.
However, the repeat argument is not needed because its functionality
can be provided by pytest-repeat as documented in the test's
README.md. So, removed the pytest_addoption function for the repeat
argument and the pytest_report_header function is moved to
tests/http/conftest.py.
TODO: Remove repeat argument from all tests. As a stopgap, a
one-element list is defined for it for now.
Closes#14611
Add tweak for mingw-w64 when building tests/http/client programs to
avoid a bogus `-Wformat` warning when using mingw-w64 v7.0.0 or older.
The warning is bogus because these programs use curl's `printf()`
implementation that is guaranteed to support that format spec.
Add this for both CMake and autotools. (But only CMake is CI tested with
an old toolchain.)
Apply the workaround to `docs/examples`, and fix an example to use
curl's `printf()` with `CURL_FORMAT_CURL_OFF_T`.
Reintroduce curl `printf()` calls into `tests/http/client`, via #14625.
Also restore large number masks to a printf, changed earlier in #14382.
Follow-up to 232302f88a#14382
Ref: https://github.com/curl/curl/pull/14625#issuecomment-2302361737Closes#14640
Sources used `lib/curlx.h` with both `ENABLE_CURLX_PRINTF` set and unset
before including it.
In a cmake "unity" batch where the first included source had it unset,
the next sources did not get the macros requested with
`ENABLE_CURLX_PRINTF` because `lib/curl.x` had already been included
without them.
Fix it by by making the macros enabled permanently and globally for
internal sources, and dropping `ENABLE_CURLX_PRINTF`.
This came up while testing unity builds with smaller batches. The full,
default unity build where all `src` is bundled up in a single unit, was
not affected.
Fixes:
```
$ cmake -B build -DCMAKE_UNITY_BUILD=ON -DCMAKE_UNITY_BUILD_BATCH_SIZE=15
$ make -C build
...
curl/src/tool_getparam.c: In function ‘getparameter’:
curl/src/tool_getparam.c:2409:11: error: implicit declaration of function ‘msnprintf’; did you mean ‘vsnprintf’? [-Wimplicit-function-declaration]
2409 | msnprintf(buffer, sizeof(buffer), "%" CURL_FORMAT_CURL_OFF_T "-",
| ^~~~~~~~~
| vsnprintf
curl/src/tool_getparam.c:2409:11: warning: nested extern declaration of ‘msnprintf’ [-Wnested-externs]
[...]
```
Reported-by: Daniel Stenberg
Bug: https://github.com/curl/curl/pull/14626#issuecomment-2301663491Closes#14632
Change TLS proto version on the test httpd server to test setting
combinations of --tls13-ciphers and --ciphers.
To not let the changed config of the httpd server bleed into the next
test, clean and reload on each test. Because a reload is slow, only
do this if the config is different than the loaded config. For this
the httpd.reload_if_config_changed() method is added.
Overloading of autouse fixtures does not seem to work. For the test
httpd server to be reloaded with a clean config in test_18_methods,
to not be affected by the config changes in test_17_ssl_use, the two
class scope fixtures of test_18_methods are now combined.
Closes#14589
- Renames Curl_readwrite() to Curl_sendrecv() to reflect that it
is mainly about talking to the server, not reads or writes to the
client. Add a `nowp` parameter since the single caller already
has this.
- Curl_sendrecv() now runs all possible operations whenever it is
called and either it had been polling sockets or the 'select_bits'
are set.
POLL_IN/POLL_OUT are not always directly related to send/recv
operations. Filters like HTTP/2, QUIC or TLS may monitor reverse
directions. If a transfer does not want to send (KEEP_SEND), it
will not do so, as before. Same for receives.
- Curl_update_timer() now checks the absolute timestamp of an expiry
and the last/new timeout to determine if the application needs
to stop/start/restart its timer. This fixes edge cases where
updates did not happen as they should have.
- improved --test-event curl_easy_perform() simulation to handle
situations where no sockets are registered but a timeout is
in place.
- fixed bug in events_socket() that complained about removing
a socket that was unknown, when indeed it had removed the socket
just before, only it was the last in the list
- fixed conncache's internal handle to carry the multi instance
(where the cache has one) so that operations on the closure handle
trigger event callbacks correctly.
- fixed conncache to not POLL_REMOVE a socket twice when a conneciton
was closed.
Closes#14561
- move variable dump to a GHA foldable group.
- minimize scope for an include().
- rename `HIDES_CURL_PRIVATE_SYMBOLS` to `CURL_HIDES_PRIVATE_SYMBOLS`,
to keep it in the curl namespace.
- drop quotes from a version number.
- add missing `Makefile.inc` var refs to comment.
- FindNGTCP2: rename internal var to underscore/lowercase.
- FindBearSSL, FindGSS: whitespace.
Closes#14571
- add set/get functions for the custom data in a tree node
- use Curl_timediff for time comparisons instead of the custom macro, as they
do the exact same things.
- add asserts to catch mistakes better
- updated test 1309 accordingly
Closes#14562
urle.scheme, urle.user, urle.password and urle.options mistakenly
operated on the original URL instead of the *effective* (last) URL.
Add test 474 to verify.
Reported-by: Gruber Glass
Fixes#14550Closes#14560
- set `CARES_NO_DEPRECATED` to avoid c-ares deprecation warnings.
Like autotools does.
- drop unused c-ares header directory when building libtests and test
servers.
- disable TrackMemory (aka `CURLDEBUG`) feature for MSVC builds.
It fails on test 558 and 1330, saying that TrackMemory isn't working.
Left a FIXME about it.
- GHA/macos: enable c-ares in a cmake job.
- GHA/windows: enable c-ares in MSVC job.
Fixes#14202
- GHA/windows: add c-ares mingw autotools job with tests.
(move `--with-windows-unicode` option from 'default' job to
this one to keep the former "default".)
Put these tests on ignore for now:
```
FAIL 472: 'aws-sigv4 with query' HTTP, aws-sigv4
FAIL 1299: 'Send "OPTIONS *" with --request-target' HTTP, --request-target
FAIL 1613: 'Send "OPTIONS *" with --request-target to a proxy' HTTP, HTTP proxy, --request-target
```
Ref: https://github.com/curl/curl/actions/runs/10388126947/job/28765761441#step:15:5902
472: fails with mingw despite Unicode enabled.
1299, 1613: seem like a case of expanding '*' to a filename.
- GHA/windows: enable c-ares in mingw cmake job.
Closes#14541
- converted the Curl_hash_count() macro to a function
- Discourage accessing struct fields directly
- Document the internal API in HASH.md
Closes#14503
- bearssl, c-ares, gss, libpsl, libssh2, mbedtls:
Before this patch these Find modules returned results via
`<NAME>_INCLUDE_DIR` and `<NAME>_LIBRARY`.
This patch makes them return `<NAME>_INCLUDE_DIRS` (note the `S`)
and `<NAME>_LIBRARIES` like other modules already did.
- bearssl, mbedtls:
Before this patch these Find modules allowed custom configuration
via `<NAME>_INCLUDE_DIRS` (note the `S`).
This patch makes them accept `<NAME>_INCLUDE_DIR`, like the rest of
the modules did.
Deprecate the old variables, but keep accepting them for
compatibility.
- bearssl: add missing `mark_as_advanced()` call.
Closes#14542
Remove check of server output as upload may abort before request
could fully be sent, so server output may be completely missing.
Test already used a 1 second delay to mitigate timing. This change
makes timing no longer an issue.
Closes#14525
