In curl 8.12 I tried to improve the logic on how we handle connections
that "upgrade" to TLS later, e.g. with a STARTTLS. I found the existing
code hard to read in this regard. But of course, the "improvements" blew
up in my face.
We fixed issues with imap, opo3, smtp in 8.12.1, but ftp was no longer
reusing existing, upgraded control connections as before. This PR adds
checks in our pytest FTP tests that verify reuse is happening as
intended.
I rewrote the logic in url.c again, so that the new test checks now pass.
Reported-by: Zenju on github
Fixes#16384Closes#16392
Adds a "cw-pause" client writer in the PROTOCOL phase that buffers
output when the client paused the transfer. This prevents content
decoding from blowing the buffer in the "cw-out" writer.
Added test_02_35 that downloads 2 100MB gzip bombs in parallel and
pauses after 1MB of decoded 0's.
This is a solution to issue #16280, with some limitations:
- cw-out still needs buffering of its own, since it can be paused
"in the middle" of a write that started with some KB of gzipped
zeros and exploded into several MB of calls to cw-out.
- cw-pause will then start buffering on its own *after* the write
that caused the pause. cw-pause has no buffer limits, but the
data it buffers is still content-encoded.
Protocols like http/1.1 stop receiving, h2/h3 have window sizes,
so the cw-pause buffer should not grow out of control, at least
for these protocols.
- the current limit on cw-out's buffer is ~75MB (for whatever
historical reason). A potential content-encoding that blows 16KB
(the common h2 chunk size) into > 75MB would still blow the buffer,
making the transfer fail. A gzip of 0's makes 16KB into ~16MB, so
that still works.
A better solution would be to allow CURLE_AGAIN handling in the client
writer chain and make all content encoders handle that. This would stop
explosion of encoding on a pause right away. But this is a large change
of the deocoder operations.
Reported-by: lf- on github
Fixes#16280Closes#16296
Add a 'wanted' major HTTP version bitmask next to the 'allowed' bitmask
in HTTP version negotiation. This will try connections as specified in
'wanted', but enabled Alt-Svc and HTTPS-RR to redirect to other major
HTTP versions, if those are 'allowed'.
Changes libcurl internal default to `CURL_HTTP_VERSION_NONE` and removes
the code in curl that sets `CURL_HTTP_VERSION_2TLS` if the command line
does not say anything else.
Closes#16117
Remove `blocking` argument from cfilter's connect method.
Implement blocking behaviour in Curl_conn_connect() instead for all
filter chains.
Update filters implementations. Several of which did never use the
paramter (QUIC for example). Simplifies connect handling in TLS filters
that no longer need to loop
Fixed a blocking connect call in FTP when waiting on a socket accept()
which only worked because the filter did not implement it.
Closes#16397
Around 2.3x speed-up parsing many large hexadecimal numbers. The decimal and
octal parser get marginally faster.
Still very readable, compact and easy to follow code.
Tweaks
- combine the max and the overflow check, gains 3ns/num (use a separate
check outside of the loop instead for max < base)
- one less indirection in the pointer, gains 3ns/num
- using the table lookup for hex nums, gains 5ns/num
- unfold the num_digit() macro, gains 3s/num
- use the hexasciitable unconditionally, gains 2ns/num
- use post-increment pointer in the table lookup, gains 1ns/num
- improved valid_digit() using the table for the hex case,
gains 26 ns/num
- use "max char" in valid_digit(), gains 3ns/num
Behavior changes:
- no longer returns STRE_TOO_BIG - only STRE_OVERFLOW
- does not move the char ** on error, which is probably better
Updated and extended test 1664 (significantly).
Closes#16374
Translate the `data->set.httpwant` which is one of the consts from the
public API (CURL_HTTP_VERSION_*) into a major version mask plus
additional flags for internal handling.
`Curl_http_neg_init()` does the translation and flags setting in http.c,
using new internal consts CURL_HTTP_V1x, CURL_HTTP_V2x and CURL_HTTP_V3x
for the major versions. The flags are
- only_10: when the application explicity asked fro HTTP/1.0
- h2_upgrade: when the application asks for upgrading 1.1 to 2.
- h2_prior_knowledge: when directly talking h2 without ALPN
- accept_09: when a HTTP/0.9 response is acceptable.
The Alt-Svc and HTTPS RR redirections from one ALPN to another obey the
allowed major versions. If a transfer has only h3 enabled, Alt-Svc
redirection to h2 is ignored.
This is the current implementation. It can be debated if Alt-Svc should
be able to override the allowed major versions. Added test_12_06 to
verify the current restriction.
Closes#16100
Give peers and `exportable` flag, set TRUE when sessions for this peer
should not be exported. This evalualtes if the peer uses confidential
information (like srp username/password), a client certificate OR if the
"ssl_peer_key" contains relative paths.
When SSL is configured with paths for relevant components, like CA trust
anchors, an attempt is made to make this path absolute. When that does
not work or the infrstructure is not available, the peer key is marked
as *local*.
Exporting sessions based on relative paths may lead to confusion when
later imported in another execution context.
Closes#16322
- replace `add_compile_options()`, `add_definitions()` with directory
properties. To harmonize this across all scripts. The new commands are
verbose, but describe better how they work. The syntax is also closer
to setting target properties, helps grepping.
- prefer `CMAKE_INSTALL_PREFIX` over `--prefix` (in tests, CI).
- tidy up cmake invocations.
- formatting.
Closes#16238
- add hex and octal parsers to the Curl_str_* family
- make curlx_strtoofft use these parsers
- remove all use of strtol() and strtoul() in library code
- generally use Curl_str_* more than strtoofft, for stricter parsing
- supports 64-bit universally, instead of 'long' which differs in size
between platforms
Extended the unit test 1664 to verify hex and octal parsing.
Closes#16336
Instead of strtoul() and strtol() calls.
Easier API with better integer overflow detection and built-in max check
that now comes automatic everywhere this is used.
Closes#16319
To reduce the number `-Wunused-macro` compiler warnings:
- delete unused macros.
- comment out unused macro that are part of a set.
- move macros into the scope they are used.
This may be useful to enable by default, but there are tricky cases that
I didn't manage to fix and paused the effort. E.g. internal features
checks in `openssl.c`. There is more, once those are fixed.
Closes#16279
By dropping the unused enum wrappers for `AF_*` macros.
Also fix `./configure` to apply `--enable-werror` options to
`tests/servers`, to catch this next time.
Seen with Intel C compiler:
```
socksd.c(184): warning #188: enumerated type mixed with another type
socksd.c(881): warning #188: enumerated type mixed with another type
[...]
sws.c(76): warning #188: enumerated type mixed with another type
sws.c(229): warning #188: enumerated type mixed with another type
[...]
```
Ref: https://github.com/curl/curl/actions/runs/13296520425/job/37129676921#step:40:338Closes#16314
Before this patch curl code was redefining `getaddrinfo` and
`freeaddrinfo` system symbols to plug in its debug wrappers. This was
causing pains to avoid applying the redefinitions to system headers
defining these functions, and to the local debug wrappers. Especially
in unity builds. It also required workarounds for systems where these
symbols are already macros.
Introduce curl-namespaced macros for these functions and use them.
This allows to drop all workarounds and makes it work in all envs,
local targets and unity/bundle combinations.
Also drop GHA/windows workaround and use the same unity batch across
all jobs. Follow-up to 29e4eda631#16272
Ref: #16272
Ref: 71cf0d1fca#14772
Ref: 3efba94f77#14765
Ref: f7d5f47059#14399Closes#16274
Prior to this change %header{} and %output{} were explained in remarks
but not listed in the --write-out variable list.
Closes https://github.com/curl/curl/pull/16299
Reapply an accidental, and unrelated, change committed in
3c128966ed which unwantedly reverted
the preceding commit fcd5c2baff.
Caused by doing a local rebase when `git am` did not apply cleanly,
and most likely using a local base behind master.
Extend output filter to pick up backslashes. This makes them pass in CI
when run in the vcpkg MSVC job, for example.
Also:
- cmake: allow TrackMemory, aka `ENABLE_CURLDEBUG` again. Drop FIXME.
- cmake: drop stale TODO.
- runtests: include the word 'TrackMemory' in the message disabling it.
Follow-up to 9f23c8f201#14541
Follow-up to 94c596bbc5#16283Closes#16289
Keeping the relevant 'ssl_scache' in 'data->state' leads to problems
when the owner of the cache is cleaned up and this reference is left
dangling.
Remove the ref entirely and always find the ssl_scache at the current
share or multi.
Folded in #16260 (test 3208) to verify this fixes the bug with a
dangling reference when an easy handle is used with easy_perform first
and in a multi_perform after.
Ref: #16236Closes#16261
OpenSSL QUIC method errors on setting TLSv1.2 ciphers, where other
methods do not.
Refrain setting --ciphers when min TLS version is 1.3 or higher.
Refrain setting --tls13-ciphers when max TLS version is less than 1.3.
Add 2 test cases.
Fixes#16232
Reported-by: zzq1015 on github
Closes#16235
Add CMake test project consuming curl via these methods:
`FetchContent`, `add_subdirectory()`, `find_package()`.
Also:
- GHA/distcheck: run these tests in CI.
- cmakelint: exclude a warning for calling "wonky-cased" built-in
CMake functions, such as `FetchContent_Declare()`.
Closes#16126
It should be done if "threaded-resolver" is enabled, but the detection
was slightly broken for builds that add c-ares for the asyn-rr feature.
Closes#16226
In certain Windows configurations, Perl resides under `C:/Program Files`
causing tests to fail when executing Perl. Fix by quoting the command.
Seen in `dl-mingw` jobs when switching to the default `bash` shell
offered by the `windows-latest` runner on GHA.
Also:
- apply the same fix for `valgrind` for consistency.
- make more use of `shell_quote()` when passing the `srcdir` directory
over the command-line. This doesn't come up in CI, but seems like
good practice. There are lots more unquoted arguments and possibly
also commands.
```
-------e--- OK (940 out of 1537, remaining: 00:32, took 0.217s, duration: 00:50)
test 1167...[Verify curl prefix of public symbols in header files]
/C/Program Files/Git/usr/bin/perl -I. -ID:/a/curl/curl/tests returned 127, when expecting 0
1167: exit FAILED
[...]
=== Start of file stderr1167
sh: line 1: /C/Program: No such file or directory
```
Ref: https://github.com/curl/curl/actions/runs/13181757313/job/36794072190?pr=16217#step:13:2107Closes#16220
Sync the test path with test1515. If fixes the test when the perl tool
resides on a path with spaces in it. E.g. when using the perl from Git
for Windows. This is a workaround, there may be a better fix this
type of issue.
Similar fix for test1515: 38b055446a
Failure:
```
test 1516...[caching of manual libcurl DNS entries after dead connection]
lib1515.exe returned 3, when expecting 0
1516: exit FAILED
=== Start of file stderr1516
Test: lib1515
[...]
17:59:32.390000 == Info: Expire cleared
request http://testserver.example.com:63621/C:/Program Files/Git/path/15160001 failed with 3
Test ended with result 3
=== End of file stderr1516
```
Ref: https://github.com/curl/curl/actions/runs/13184790755/job/36804217128?pr=16217#step:13:2805Closes#16223
Made the netrc parser return a more consistent set of error codes and
error messages, and also return error properly if the .netrc file is
missing.
Add test 697 to verify error on missing netrc file.
Fixes#16163
Reported-by: arlt on github
Closes#16165
For ASN.1 tags with indefinite length, curl's own parser for TLS
backends that do not support certificate inspection calls itself
recursively. A malicious server certificate can then lead to high
recursion level exhausting the stack space.
This PR limits the recursion level to 16 which should be safe on all
architectures.
Added unit test 1657 to verify behaviour.
Fixes#16135
Reported-by: z2_
Closes#16137
The condition `!$cmdtype eq "perl"` (introduced in a4765b0551) is always
false. It checks whether a logical negation (giving true/false) is equal
to the string `"perl"`. This is impossible, so the logging never worked.
The intent was probably to negate the result of the string
comparison:`!($cmdtype eq "perl")` or simply `$cmdtype ne "perl"`.
Fixes#16128
Reported-by: Igor Todorovski
Closes#16129
While this is useful on CI to highlight issues in the scripts, it's a
recipe for pain when enabled on users' builds.
Suggested-by: Leon Timmermans
Ref: #16128
RFC 6455 Section 5.2 notes that for bits RSV1, RSV2, and RSV3 of the
framing header, a non-zero value that is not defined by a negotiated
extension MUST Fail the WebSocket connection.
Test 2310 verifies
Closes#16069
- Detach and disconnect an attached connection before performing.
Prior to this change it was not possible to safely reuse an easy handle
with an attached connection in a second call to curl_easy_perform. The
only known case of this is a connect-only type handle where the
connection was detached when curl_easy_perform returned, only to be
reattached by either curl_easy_send/recv.
This commit effectively reverts 2f8ecd5d and be82a360, the latter of
which treated the reuse as an error. Prior to that change undefined
behavior may occur in such a case.
Bug: https://curl.se/mail/lib-2025-01/0044.html
Reported-by: Aleksander Mazur
Closes https://github.com/curl/curl/pull/16008
It had shorthand aliases to launch `./configure` and
`./configure --with-openssl`. The former hasn't worked for a long while
because of missing TLS.
Its `ca-bundle` and `ca-firefox` targets have been broken for 2.5 years
till recently. These targets also exist in `./configure` and have been
working all along.
Also:
- cmake: add support `curl-ca-bundle` and `curl-ca-firefox` targets.
- tests/testcurl.pl: drop obsolete build logic.
Closes#16094
Works better if we later introduce another way to do the asynch RR
resolves (together with the threaded resolver) that does not use c-ares.
Closes#16090
Rework the way `tool_hugehelp.c` is included in builds.
After this patch, with `./configure` and CMake `tool_hugehelp.c` is only
compiled when building with manuals enabled. With manuals disabled this
source file is not used anymore. The method is similar to how
8a3740bc8e implemented `tool_ca_embed.c`.
`./configure` always generates it as before, otherwise the build fails.
- winbuild: rework to not need `buildconf.bat`, but automatically use
`tool_hugehelp.c` if present (e.g. when building from an official
source tarball) and enable `USE_MANUAL` accordingly.
- `buildconf.bat`: after dropping `tool_hugehelp.c` generation, the only
logic left was `cp Makefile.dist Makefile`. This allowed to launch
winbuild builds via GNU Make in a Git repo. Drop this option together
with the batch file.
- build `libcurltool` without `USE_MANUAL` macro to exclude the manual
and the dependence on the generator commands. Drop relying on
`UNITTESTS` for this purpose.
Follow-up to 96843f4ef7#16068
- `src/mkhelp.pl`: include `tool_hugehelp.h` before using `USE_MANUAL`
to have it set in `config-*.h` builds with source tarballs created
with manual but without zlib.
Closes#16081
- Change the swsbounce keyword to override the part number on a
subsequent request to the previous part number + 1.
Note the previous part number in this case is the part number that
was returned as a response to the previous request and contained
the swsbounce keyword.
Prior to this change swsbounce incremented the part number of the
subsequent request instead of overriding it, and did so in a more
limited fashion that prevented chaining swsbounce in multiple responses.
For example, if the test makes a request that causes the sws server to
return `<data>` as a response and that response contains `swsbounce`
then for the next response the sws server returns `<data1>`. If
`<data1>` also contains `swsbounce` then for the next response the sws
server now returns `<data2>` instead of the requested part.
Fixes https://github.com/curl/curl/discussions/16074
Closes https://github.com/curl/curl/pull/16085
Allow building with c-ares and yet use threaded resolver for the main
host A/AAAA resolving:
`--with-ares` provides the c-ares install path and defaults to use
c-ares for name resolving
`--with-threaded-resolver` still uses c-ares in the build (for HTTPS)
but uses the threaded resolver for "normal" resolves.
It works similarly for cmake: ENABLE_ARES enables ares, and if
ENABLE_THREADED_RESOLVER also is set, c-ares is used for HTTPS RR and
the threaded resolver for "normal" resolves.
HTTPSRR and c-ares-rr are new features return by curl_version_info() and
thus shown by curl -V.
The c-ares-rr feature bit is there to make it possible to distinguish
between builds using c-ares for all name resolves and builds that use
the threaded resolves for the regular name resolves and c-ares for
HTTPSRR only. "c-ares-rr" means it does not use c-ares for "plain" name
resolves.
HTTPSRR support is EXPERIMENTAL only.
Closes#16054
The variable `conn->httpversion` was used for several purposes and it
was unclear at which time the value represents what.
- rename `conn->httpversion` to `conn->httpversion_seen`
This makes clear that the variable only records the last
HTTP version seen on the connection - if any. And that it
no longer is an indication of what version to use.
- Change Alt-Svc handling to no longer modify `conn->httpversion`
but set `data->state.httpwant` for influencing the HTTP version
to use on a transfer.
- Add `data->req.httpversion_sent` to have a record of what
HTTP version was sent in a request
- Add connection filter type CF_TYPE_HTTP
- Add filter query `CF_QUERY_HTTP_VERSION` to ask what HTTP
filter version is in place
- Lookup filters HTTP version instead of using `conn->httpversion`
Test test_12_05 now switches to HTTP/1.1 correctly and the
expectations have been fixed.
Removed the connection fitler "is_httpN()" checks and using
the version query instead.
Closes#16073
