Changes to make a curl built with OpenSSL + GnuTLS to run successfully
in our pytests. Run
CURL_SSL_BACKEND=openssl pytest
to test a TLS backend other than the default.
Closes#15443
handle/handle64 requires a literal match with the filenames it's
listing.
Also:
- make handle64 log messages more unique to help text searches.
- update a comment with Windows Perl info.
Cherry-picked from #14949Closes#15436
Add two test cases that connection using a hostname the server has no
certificate for. First, verify that the peer verification fail, as
expected. Second, provide '--insecure' to test that the connection
succeeded and returned some data.
Closes#15429
New args for scorecard.py:
* --remote ip:port to run tests against a remote server
* --start-only: to only start the servers, listing their ports
Start the server using "scorecard.py --start-only" on one machine and
then run the tests with "scorecard.py --remote ip:port" against that
machine.
Closes#15415
Extend `INSTALL-CMAKE` document with the list of available options,
a short description and default values.
The list may not be 100% complete.
There are no component boundaries in CMake, so the line is blurry
between curl options, CMake options, CMake Find modules options.
I included certain CMake options that seemed useful, and/or have
dedicated use withing curl's CMake source. But, all CMake built-in
options are usable, as documented upstream in CMake.
The naming of the options has a heritage and the inconsistencies with
it, including a lack of clear namespace. This may be subject to future
updates, also after figuring out which name has special meaning within
CMake and/or CMake projects out of unwritten convention or something
more tangible.
CMake allows to initialize any internal variable via `-D`. This may be
useful to pre-initialize/override feature check results. The list
doesn't contain these, and they remain officially undocumented.
Also:
- make adjustments to keep the spellchecker happy.
- retrofit description changes to the cmake sources.
- stop documenting deprecated `Find*` variables.
Reported-by: Daniel Stenberg
Fixes https://github.com/curl/curl/discussions/14885Closes#15388
Use session cache for QUIC when built with quictls or wolfSSL.
Add test_017_10 for verifying QUIC TLS session reuse when built with
quictls, gnutls or wolfssl.
Closes#15358
Add length to session saves, making it clear that we are storing a byte
blob and allowing memcmp() on sameness check.
Remove some pytest skips for bearssl to see if they now work properly in
CI.
Closes#15395
Use mbedtls_ssl_session_load() and mbedtls_ssl_session_save() to convert
TLS sessions to byte blobs for the session cache.
Fix a skip message to better indicate why the test is skipped for
mbedtls.
Closes#15398
Sort TESTINFO lines by description within the number of skipped test.
It makes the list of skipped test groups easier to diff/compare between
jobs and runs.
Closes#15374
They complete in 4 and 7 minutes, and do not hold back the main Linux
workflow.
Also:
- bump default parallelism for `test-torture` target to `-j20`
(was: `-j2`).
- drop redundant package install from `rustls` jobs.
Closes#15360
It reduces the number of synonym variables in the code.
Makes it easier to grok and grep.
- replace `CURL_SOURCE_DIR`
with `PROJECT_SOURCE_DIR`.
- replace `CURL_BINARY_DIR`
with `PROJECT_BINARY_DIR` or `CMAKE_CURRENT_BINARY_DIR`.
- replace a single use of `CMAKE_BINARY_DIR`
with `PROJECT_BINARY_DIR`.
- replace `CMAKE_CURRENT_*_DIR`
with `PROJECT_*_DIR` where it makes the code more uniform.
- quote an argument (formatting).
Closes#15331
We used to include a special mod_h2 in our CI that supports the
directive H2MaxDataFrameLen for test_02_20. Since then, ubuntu-lastest
includes a more recent apache httpd. Let's see if we can live without
the special.
Closes#15353
When a server signals EOS from its side and the curl upload is
unfinished and the server has not given a positive HTTP status response,
auto RST the stream to signal that the upload is incomplete and that the
whole transfer can be stopped.
Fixes the case where the server responds with 413 on an upload but does
not RST the stream from its side, as httpd and others do.
Reported-by: jkamp-aws on github
Fixes#15316Closes#15325
By renaming from a temporary file name to the .c once completed. This
avoids the risk that the checksrc job tries to verify the file before it
is complete, in parallel build setups.
Reported-by: Dan Frandrich
Fixes#15258Closes#15327
Once SSL_shutdown() has been called, OpenSSL does not really seem to
like it when it is called again and the other side has some finally data
to deliver.
Instead SSL_read() needs to be used solely, once the close notify has
been sent from curl's side.
Closes#15321
Override the system default config in test_17_09, since we want to check
all TLS versions. Provide own, empty config file to gnutls, so that any
system wide file has no effect.
The latest ubunu image in GH CI disables TLS 1.0 and 1.1
system wide for GnuTLS. Good intentions.
Closes#15310
vsftpd 3.0 at least writes its version number to stdin (!) instead of
stderr. This works due for backwards compatibility reasons in UNIX, so
we must check stdin for anything written there to reliably parse the
version string.
Closes#15278
Fix new issues found by `proselint`.
Also:
- silence this technical warning:
```
:0: DeprecationWarning: /home/runner/.proselintrc was found instead of a JSON file. Rename to /home/runner/.proselintrc.json.
```
- fix an input filename.
`proselints` fails now if an input file is missing.
Reported-by: Jay Satiro
Bug: https://github.com/curl/curl/pull/15291#issuecomment-2410505100Closes#15293
For TLSv1.3, if supported, observer special return code to retrieve
newly arrived session from mbedTLS.
Adjust test expectations now that TLSv1.3 session resumption works in
mbedTLS >= 3.6.0.
Based on #14135 by @ad-chaos
Closes#15245
Register a callback to get notified of new SSL sessions by wolfSSL.
Remove the explicit session retrieval after handshake, since this does
not work for TLSv1.3.
Adjust test expectations now that TLSv1.3 session resumption works
in wolfSSL.
Closes#15243
Add session reuse for QUIC transfers using GnuTLS. This does not include
support for TLS early data, yet.
Fix check of early data support in common GnuTLS init code to not access
the filter context, as the struct varies between TCP and QUIC
connections.
Closes#15265
This indicates that the file was created by curl which can help a user
determine the origin of a file. Like the other attributes, this is only
enabled with the --xattr option.
Closes#15237
Based on #14135, implement TLSv1.3 earlydata support for the curl
command line, libcurl and its implementation in GnuTLS.
If a known TLS session announces early data support, and the feature is
enabled *and* it is not a "connect-only" transfer, delay the TLS
handshake until the first request is being sent.
- Add --tls-earldata as new boolean command line option for curl.
- Add CURLSSLOPT_EARLYDATA to libcurl to enable use of the feature.
- Add CURLINFO_EARLYDATA_SENT_T to libcurl, reporting the amount of
bytes sent and accepted/rejected by the server.
Implementation details:
- store the ALPN protocol selected at the SSL session.
- When reusing the session and enabling earlydata, use exactly
that ALPN protocol for negoptiation with the server. When the
sessions ALPN does not match the connections ALPN, earlydata
will not be enabled.
- Check that the server selected the correct ALPN protocol for
an earlydata connect. If the server does not confirm or reports
something different, the connect fails.
- HTTP/2: delay sending the initial SETTINGS frames during connect,
if not connect-only.
Verification:
- add test_02_32 to verify earlydata GET with nghttpx.
- add test_07_70 to verify earlydata PUT with nghttpx.
- add support in 'hx-download', 'hx-upload' clients for the feature
Assisted-by: ad-chaos on github
Closes#15211
- on load, only replace existing HSTS entries if there is a full host
match
- on matching, prefer a full host match and secondary the longest tail
subdomain match
Closes#15210
Since we see Windows fails of 1915, add tracing and a connect timeout.
The test uses a port no one is supposed to listen on, but Windows has
this weird wait logic. So, set a short timeout.
Closes#15107
As the parsing and address "regeneration" are done anyway, we might as
well use the updated version in the result and thereby A) get a
normalized (and lower cased) version of the address and B) avoid a
strcpy().
Updated test 1560 to verify.
Closes#15143
Due to timings on paused response receive, the error code may vary due
to the location where it is detected that the server closed the transfer
prematurely.
Be more lenient in always allowing PARTIAL_FILE as ok.
Closes#15138
`Curl_bufq_unwrite()` used the head instead of the tail chunk to shrink
the bufq's content. Fix this and add test case that checks correct
behaviour.
Amended test 2601 accordingly.
Reported-by: Chris Stubbs
Closes#15136
Generate the certificate serial numbers automatically instead of doing
from shell (or Perl earlier).
Fixes intermittent CI failures due to the shell-based random generator
generating the same serial number twice:
```
$ openssl ca -config EdelCurlRoot-ca.cnf -revoke Server-localhost0h-sv.crt
Using configuration from EdelCurlRoot-ca.cnf
ERROR:Already revoked, serial number 66FDB23A
make: *** [../../../tests/certs/Server-localhost0h-sv.pem] Error 1
```
https://github.com/curl/curl/actions/runs/11151401083/job/30994755798?pr=15128#step:10:498
Follow-up to fa461b4eff#14486
Follow-up to fa69b41c77#13307Closes#15129
Instead of calling the shell and external tools, rely on Perl functions
like `Cwd::getcwd()`, `Cwd::abs_path()`, `Cygwin::posix_to_win_path()`,
`Cygwin::win_to_posix_path()` to retrieve the current directory and
convert between POSIX and Windows formats.
This adds native Windows Perl support, avoids most failure modes and
makes format guessing and other internal functions unnecessary.
Also:
- delete unused `sys_native_path()`.
- delete redundant `normalize_path()` because Perl `abs_path()` already
does it.
Cherry-picked from #14949Closes#15111
For all tests using -O that were previously relying on a debug build and
the CURL_TESTDIR environment variable, use the plain --output-dir option
instead so that they can run proper in non-debug builds.
Closes#15114
By keeping the headers in memory until we know the target file name,
then output them all.
Previously this option combination would cause an error.
Add test 1310 and 1492 to verify. Adjusted test 1460 to work in the new
conditions.
Closes#15110
poll() on macOS 10.12 was deemed broken in 2016 when we discovered that
it misbehaves when provided with no sockets to wait for. The
HAVE_POLL_FINE is used to mark a poll() implementation that behaves
correctly: it *should* still wait the timeout time.
curl has therefore opted to use select() on Apple operating systems ever
since. To avoid the risk that this or other breakage cause problems.
However, using select() internally is also bad because it suffers from
problems when using file descriptors beyond 1024.
This change makes poll() used if it is present, but if there is no
sockets to wait for it avoids using poll() and instead falls back to
select() - but without any sockets to wait for there is no 1024 problem.
This removes all previous special-handling involving HAVE_POLL_FINE.
ref: https://daniel.haxx.se/blog/2016/10/11/poll-on-mac-10-12-is-broken/Closes#15096
When uploading FTP with unknown length, we write a last 0-length chunk
with the EOS flag set. OpenSSL's SSL_write() errors on such a write.
Skip writing 0-length data to TLS backends instead.
Add test in FTPS for such uploads to verify.
Fixes#15101
Reported-by: Denis Goleshchikhin
Closes#15102
- add variable for the null device filename and use that in Perl code.
- initialize this variable with `NUL` on native Windows.
- add `%DEV_NULL` variable and use it in tests.
Fixes `The system cannot find the path specified.` messages seen when
running `runtests.pl` with native Windows Perl.
Also adjust code to not break mcedit syntax highlighting.
Cherry-picked from #14949Closes#15098
To ensure Perl invocations within tests call the same Perl binary used
for running the tests, as specified or auto-detected via `PERL` env
(autotools) or `PERL_EXECUTABLE` setting (cmake). Instead of the first
`perl` executable found in `PATH`.)
Cherry-picked from #14949Closes#15097
Before this patch not all tests used `%FILE_PWD` with the `file://`
protocol.
Keep `%PWD` for test1145, to keep it fail on Windows like the test
expects.
Cherry-picked from #14949Closes#15090
- CI default: raise parallelism to 20
- CI valgind: set parallelism to 6
- CI non-native: adapt parallelism for OS builds
- CI Windows: no longer ignore FTP, TFTP, MQTT and SMTP
- CI Windows: restrict test timeout to 10 minutes
- CI Windows: do not run tests for msh3 build
- tests, various: restrict curl invocation to ipv4 to avoid talking to
any ipv6 test server running in parallel
- tests: requiring http/2 server no longer needs to mention http server
- test 190: use a fixed timeout of 10 seconds instead of %FTPTIME2, as
that value rises under parallel load beyond what the server is waiting
and then produces different results
- test 1540: add debug logging
- testrunner: add verify checks for http/2 and http/3 server
Closes#15040
Our test servers run either on ipv4 *or* on ipv6, as requested.
A test case using 'localhost' or '*.local' in the url needs to
run with the specific version of the server started.
Otherwise, curl's "happy eyeball"ing will connect to another
server that may be running due to parallel testing or for some
other reasons.
Note that port reuse here depends on the OS strategy and it
seems netbsd is especially likely to hit this.
Closes#15060
Check responsiveness of http/3 server when running.
Also, a test case with http/2 or http/3 server requirement
now implicitly drags in a 'http' server and we need no longer
mention that in testdata.
Closes#15058
The variable FTPTIME2 may, on a loaded test server, become so
large that the timeout does not happen before the fixed 60 seconds
the server waits.
Closes#15056
Windows sometimes has issues when opening the same file twice, so these
test two situations where that could potentially occur.
Reported-by: ralfjunker on github
Ref: #15043Closes#15045
Test expected a connect to a port no one is listening to immediately
fail. But Windows has its internal retry logic that may fail this.
As fix, multi_perform()/multi_wait() until transfer is done.
Closes#15054
SANITIZE_ALLOW_TRUNCATE and SANITIZE_ALLOW_COLONS were never used by
code, thus only making the code complicated for no good use.
Since nothing should truncate, using strncpy() is wrong.
Two cases of malloc + copy replaced with proper strdup() calls.
Fixup unit test 1604 accordingly.
Closes#15047
This is the %CLIENT6IP variable - but without outmost brackets since
some commmand lines need the address without the brackets. With this
variable we can run three more tests without prechecks.
Closes#15039
Also introduce 'notexists' for verifying that directory entries do not
exist after a test. Now an explicit supported feature instead of using
"funny" perl in postcheck.
Closes#15046
It enables running pytests in cmake jobs, regardless of underlying build
tool choice (= makes it work with ninja.)
Also:
- drop pytest logic launching `make` and exiting in case of failure.
Maybe there is a better way and keep this functionality somehow, bind
it to a command-line option? make it fail softly?
- GHA/linux: invoke pytest via the build, not directly.
- autotools: add missing dummy runtests targets when cross-compiling.
Closes#15034
- lib557: suppress `-Wformat-overflow` warning in source.
Fixes:
```
lib557.c: In function ‘test_float_formatting’:
lib557.c:1408:37: error: ‘%*f’ directive output of 2147483648 bytes exceeds ‘INT_MAX’ [-Werror=format-overflow=]
1408 | curl_msnprintf(buf, sizeof(buf), "%*f", INT_MIN, 9.1);
| ^~~
lib557.c:1408:3: note: ‘curl_msnprintf’ output 2147483649 bytes
1408 | curl_msnprintf(buf, sizeof(buf), "%*f", INT_MIN, 9.1);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
```
Ref: https://app.circleci.com/pipelines/github/curl/curl/10226/workflows/87642ee9-cda6-4916-8206-c82aac5f595e/jobs/107669?invite=true#step-106-40996_46
The root cause of why this option gets enabled remains undiscovered.
Reported-by: Daniel Stenberg
Fixes#15008
Follow-up to 71cf0d1fca#14772
- build: drop `-Wno-format-overflow` from picky warning list.
These options only get used with picky warnings enabled.
Follow-up to 145f87b9e8#14598
- unit1652: suppress in source (and not rely on picky warnings anymore.)
Closes#15012
Make `test-ci` not depend on the `testdeps` target.
`test-ci` is designed to run curl tests in CI. In CI we build all
necessary dependencies explicitly beforehand, and they are always ready
when calling the `test-ci` step. Thus, it isn't necessary to enforce
them via a dependency rule. Dropping it saves redundant work and delay
in CI jobs.
The `testdeps` dependency should not normally be a problem. It's
supposed to be a no-op if those targets are already built. In practice
however, it causes a delay and/or an actual rebuild of those
dependencies depending on generator (= build tool) used and other
factors.
As observed in the GHA/windows workflow, the `testdeps` dependency:
- with Ninja, causes no delay, and no extra work:
https://github.com/curl/curl/actions/runs/10980099984/job/30485440389#step:25:18
- with GNU Make, caused a re-evaluation of `testdeps` targets,
but did not actually rebuild them. This re-evaluation took a
noticeable time (esp. with non-bundled tests):
https://github.com/curl/curl/actions/runs/10980099984/job/30485440155#step:14:11 (with bundles)
https://github.com/curl/curl/actions/runs/10973851013/job/30471690331#step:14:11 (w/o bundles)
verbose: https://github.com/curl/curl/actions/runs/10980506956/job/30486434629#step:14:13
- with MSBuild, caused a re-evaluation of `testdeps` targets, and
triggered a _rebuild_:
https://github.com/curl/curl/actions/runs/10980099984/job/30485435968#step:14:19 (with bundles)
https://github.com/curl/curl/actions/runs/10973851013/job/30471689714#step:14:19 (w/o bundles)
verbose: https://github.com/curl/curl/actions/runs/10980506956/job/30486436368#step:14:48
It's suspected that our use of
`-DCMAKE_VS_GLOBALS=TrackFileAccess=false` in CI is contributing to
this. This option is supposed to affect incremental builds only. For
some reason it affects CI builds too, even though they are not
incremental, and no sources are changed between build steps.
Reported-by: Aki Sakurai
Ref: #14999
Notice that `test-*` targets depending on `testdeps` is NOT sufficient
to build everything to run tests, e.g. it misses to build the curl tool,
which is essential. This is also true for autotools' `test-ci` target
which misses to build libcurl and the curl tool.
Perhaps it'd be best to drop `testdeps` as a dependency for _all_
`test-*` targets and make it official to require building dependencies
manually. Alternatively these targets could be fixed to rebuild
everything necessary to run tests.
Closes#15001
Add `CURL_CA_SEARCH_SAFE` build-time option to enable CA bundle search
in the `curl` tool directory. The lookup method was already used to find
`.curlrc` and `_curlrc` (on Windows). On Windows it overrides the unsafe
default `SearchPath()` method.
Enable with:
- cmake: `-DCURL_CA_SEARCH_SAFE=ON`
- autotools: `--enable-ca-search-safe`
- raw: `CPPFLAGS=-DCURL_CA_SEARCH_SAFE`
On Windows, before this patch the whole `PATH` was searched for
a CA bundle. `PATH` may contain unwanted or world-writable locations,
including the current directory. Searching them all is convenient to
pick up any CA bundle, but not secure.
The Muldersoft curl distro implements such CA search via a custom
patch for Windows:
cd652d4792/patch/curl_tool_doswin.diff (L50)
MSYS2/mingw-w64 distro has also been rolling a patch solving this:
https://github.com/msys2/MINGW-packages/blob/master/mingw-w64-curl/0001-Make-cURL-relocatable.patchhttps://github.com/msys2/MINGW-packages/blob/master/mingw-w64-curl/pathtools.c
Also add option to fully disable Windows CA search:
- cmake: `-DCURL_DISABLE_CA_SEARCH=ON`
- autotools: `--disable-ca-search`
- raw: `CPPFLAGS=-DCURL_DISABLE_CA_SEARCH`.
Both options are considered EXPERIMENTAL, with possible incompatible
changes or even (partial) removal in the future, depending on feedback.
An alternative, secure option is to embed the CA bundle into the binary.
Safe search can be extended to other platforms if necessary or useful,
by using `_NSGetExecutablePath()` (macOS),
`/proc/self/exe` (Linux/Cygwin), or `argv[0]`.
Closes#14582
