Remove `blocking` argument from cfilter's connect method.
Implement blocking behaviour in Curl_conn_connect() instead for all
filter chains.
Update filters implementations. Several of which did never use the
paramter (QUIC for example). Simplifies connect handling in TLS filters
that no longer need to loop
Fixed a blocking connect call in FTP when waiting on a socket accept()
which only worked because the filter did not implement it.
Closes#16397
- add VS2019 job, with Schannel + OpenSSL 1.0.2.
First MultiSSL job here and add the last missing modern VS version.
- fix builds with mixed ALPN capabilities in MultiSSL unity builds.
Caused by reusing `HAS_ALPN` between TLS modules without
resetting it. Fix it by using unique names for each backend.
- merge a VS2010 job into a VS2012. With MultiSSL and x86 OpenSSL.
- make a job static.
- fix `Shared`/`Static` in a job name.
- add `Shared` to job names.
Closes#16231
mbedtls is picky when a mbedtls_ssl_write) was previously blocked. It
requires to be called with the same amount of bytes again, or it will
lose bytes, e.g. reporting all was sent but they were not. Remember the
blocked length and use that when set.
Reported-by: Tamás Bálint Misius
Fixes#15801Closes#15846
When a QUIC TLS session announced early data support and
'CURLSSLOPT_EARLYDATA' is set for the transfer, send initial request and
body (up to the 128k we buffer) as 0RTT when curl is built with
ngtcp2+gnutls.
QUIC 0RTT needs not only the TLS session but the QUIC transport
paramters as well. Store those and the earlydata max value together with
the session in the cache.
Add test case for h3 use of this. Enable quic early data in nghttpx for
testing.
Closes#15667
Described in detail in internal doc TLS-SESSIONS.md
Main points:
- use a new `ssl_peer_key` for cache lookups by connection filters
- recognize differences between TLSv1.3 and other tickets
* TLSv1.3 tickets are single-use, cache can hold several of them for a peer
* TLSv1.2 are reused, keep only a single one per peer
- differentiate between ticket BLOB to store (that could be persisted) and object instances
- use put/take/return pattern for cache access
- remember TLS version, ALPN protocol, time received and lifetime of ticket
- auto-expire tickets after their lifetime
Closes#15774
For TLS backends that don't need these functions, they now use plain
NULL pointers instead of setting a function that does nothing.
Helps making it clearer that a specific TLS handler does not provide
anything specific for that action.
Closes#15772
Run mbedtls' psa_crypt_init() in the general global init, optionally
protected by mbedtls locks when available.
CI: when building mbedtls, enabled thread safety
Reported-by: wxiaoguang on github
Fixes#15500Closes#15505
Since data can be NULL in here, calling failf() can be bad. This should
also be a terribly rare error so the lack of error message for this
should be manageable.
Reported-by: wxiaoguang on github
Fixes#15485Closes#15486
Use mbedtls_ssl_session_load() and mbedtls_ssl_session_save() to convert
TLS sessions to byte blobs for the session cache.
Fix a skip message to better indicate why the test is skipped for
mbedtls.
Closes#15398
For TLSv1.3, if supported, observer special return code to retrieve
newly arrived session from mbedTLS.
Adjust test expectations now that TLSv1.3 session resumption works in
mbedTLS >= 3.6.0.
Based on #14135 by @ad-chaos
Closes#15245
Based on #14135, implement TLSv1.3 earlydata support for the curl
command line, libcurl and its implementation in GnuTLS.
If a known TLS session announces early data support, and the feature is
enabled *and* it is not a "connect-only" transfer, delay the TLS
handshake until the first request is being sent.
- Add --tls-earldata as new boolean command line option for curl.
- Add CURLSSLOPT_EARLYDATA to libcurl to enable use of the feature.
- Add CURLINFO_EARLYDATA_SENT_T to libcurl, reporting the amount of
bytes sent and accepted/rejected by the server.
Implementation details:
- store the ALPN protocol selected at the SSL session.
- When reusing the session and enabling earlydata, use exactly
that ALPN protocol for negoptiation with the server. When the
sessions ALPN does not match the connections ALPN, earlydata
will not be enabled.
- Check that the server selected the correct ALPN protocol for
an earlydata connect. If the server does not confirm or reports
something different, the connect fails.
- HTTP/2: delay sending the initial SETTINGS frames during connect,
if not connect-only.
Verification:
- add test_02_32 to verify earlydata GET with nghttpx.
- add test_07_70 to verify earlydata PUT with nghttpx.
- add support in 'hx-download', 'hx-upload' clients for the feature
Assisted-by: ad-chaos on github
Closes#15211
- update mbedTLS repo URL.
- switch local mbedTLS build to use CMake, and Ninja.
CMake build is required to create and install mbedTLS `pkg-config`
files. (as of v3.6.1)
`-DCMAKE_POSITION_INDEPENDENT_CODE=ON` required to avoid this error
when linking mbedtls to `libcurl.so`:
```
/usr/bin/ld: /home/runner/mbedtls/lib/libmbedcrypto.a(cipher.c.o): warning: relocation against `mbedtls_cipher_base_lookup_table' in read-only section `.text'
/usr/bin/ld: /home/runner/mbedtls/lib/libmbedtls.a(ssl_tls.c.o): relocation R_X86_64_PC32 against symbol `mbedtls_x509_crt_profile_suiteb' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: bad value
```
Ref: https://github.com/curl/curl/actions/runs/11245069259/job/31264386723#step:40:43
- make local mbedTLS build 10x smaller by omitting programs and tests.
- GHA/linux: fix cmake warning by adding `-B .` option.
- GHA/linux: add build-only cmake job for packaged mbedTLS (2.x).
- fix compiler warning when building with mbedTLS 2.x:
```
/home/runner/work/curl/curl/lib/vtls/mbedtls.c:344:1: error: ‘mbed_cipher_suite_get_str’ defined but not used [-Werror=unused-function]
344 | mbed_cipher_suite_get_str(uint16_t id, char *buf, size_t buf_size,
| ^~~~~~~~~~~~~~~~~~~~~~~~~
```
Ref: https://github.com/curl/curl/actions/runs/11244999065/job/31264168295#step:40:50
Also in preparation for #15193Closes#15208
booleans should use the type 'bool' and set the value to TRUE/FALSE
non-booleans should not be 'bool' and should not set the value to
TRUE/FALSE
Closes#15123
Patch started out for working around compiler warnings in BearSSL latest
tarball release v0.6 (2018-08-14) and Apple clang 14 with CMake.
Then turned into patching CMake and `Makefile.mk` builds to use
`-isystem` instead `-I` when adding header directories for
dependencies. This avoids compiler warnings in dependency headers,
syncing behaviour with autotools.
Also:
- `Makefile.mk`: add support for BearSSL.
- delete warning suppression for mbedTLS headers. No longer necessary
after this patch.
Follow-up to 434db995a7#12720
Silenced BearSSL warnings:
```
In file included from curl/lib/vtls/bearssl.c:28:
In file included from bearssl/inc/bearssl.h:127:
bearssl/inc/bearssl_hash.h:727:5: warning: 'BR_DOXYGEN_IGNORE' is not defined, evaluates to 0 [-Wundef]
^
bearssl/inc/bearssl_hash.h:745:5: warning: 'BR_DOXYGEN_IGNORE' is not defined, evaluates to 0 [-Wundef]
^
In file included from curl/lib/vtls/bearssl.c:28:
In file included from bearssl/inc/bearssl.h:136:
bearssl/inc/bearssl_ssl.h:1253:20: warning: implicit conversion loses integer precision: 'unsigned int' to 'uint16_t' (aka 'unsigned short') [-Wimplicit-int-conversion]
cc->version_min = version_min;
~ ^~~~~~~~~~~
bearssl/inc/bearssl_ssl.h:1254:20: warning: implicit conversion loses integer precision: 'unsigned int' to 'uint16_t' (aka 'unsigned short') [-Wimplicit-int-conversion]
cc->version_max = version_max;
~ ^~~~~~~~~~~
bearssl/inc/bearssl_ssl.h:1327:28: warning: implicit conversion loses integer precision: 'size_t' (aka 'unsigned long') to 'uint16_t' (aka 'unsigned short') [-Wimplicit-int-conversion]
ctx->protocol_names_num = num;
~ ^~~
5 warnings generated.
```
(These warnings were fixed in BearSSL Git master in 2019 via
2893441f2efd4603ddd6d7f49011bdda096a4a87 and
ecdf89770ee82dfea6186fb4369cff3d06cd852e.)
Also these two cases, which are caused by an unidentified component
(outside curl) cranking up MSVC warnings in external headers to `/W4`
when ZLIB is deselected:
https://github.com/curl/curl/pull/14859#issuecomment-2351809153
mbedTLS 3.6.1:
```
C:\vcpkg\installed\x64-windows\include\psa\crypto_struct.h(254,13): error C2220: the following warning is treated as an error [D:\a\curl\curl\bld\lib\libcurl_object.vcxproj]
(compiling source file 'CMakeFiles/libcurl_object.dir/Unity/unity_0_c.c')
C:\vcpkg\installed\x64-windows\include\psa\crypto_struct.h(254,13): warning C4200: nonstandard extension used: zero-sized array in struct/union [D:\a\curl\curl\bld\lib\libcurl_object.vcxproj]
(compiling source file 'CMakeFiles/libcurl_object.dir/Unity/unity_0_c.c')
```
Ref: https://github.com/curl/curl/actions/runs/10842694205/job/30107466989?pr=14859#step:10:29
nghttp3 1.5.0:
```
C:\vcpkg\installed\x64-windows\include\nghttp3\nghttp3.h(2678,1): error C2220: the following warning is treated as an error [D:\a\curl\curl\bld\lib\libcurl_object.vcxproj]
(compiling source file 'CMakeFiles/libcurl_object.dir/Unity/unity_0_c.c')
C:\vcpkg\installed\x64-windows\include\nghttp3\nghttp3.h(2678,1): warning C4324: 'nghttp3_pri': structure was padded due to alignment specifier [D:\a\curl\curl\bld\lib\libcurl_object.vcxproj]
(compiling source file 'CMakeFiles/libcurl_object.dir/Unity/unity_0_c.c')
```
Ref: https://github.com/curl/curl/actions/runs/10871875297/job/30166233862?pr=14859#step:10:28Closes#14763
With mbedTLS if the minimum version of TLS is set to 1.3,
MBEDTLS_SSL_VERIFY_OPTIONAL is not available in client mode. See:
https://github.com/Mbed-TLS/mbedtls/blob/2ca6c285/library/ssl_tls.c#L1357
Also, there might be plans to remove it completely in future mbedTLS
versions.
Switch to always use MBEDTLS_SSL_VERIFY_REQUIRED. If verifypeer or
verifyhost are disabled the corresponding error flags are cleared in the
verify callback function. That is also where verification errors are
logged.
Closes#14591
After TLS handshare, indicate which TLS version was negotiated in
addition to the cipher in the handshake completed log message.
Also use the verify callback for certificate logging and collection.
This allows things to work even when MBEDTLS_SSL_KEEP_PEER_CERTIFICATE
is disabled in the mbedtls library.
And lastly, catch certificate validation errors later so we can give the
user more informative error messages that indicate what the failure was
from certificate validation.
Tested on both current LTS versions (2.28 and 3.6).
Closes#14444
When using CMake Unity build.
- use unique name for `set_ssl_version_min_max()`
Fixes collision between GnuTLS, mbedTLS and SecureTransport.
```
lib\vtls\mbedtls.c(317,1): error C2084: function 'CURLcode set_ssl_version_min_max(Curl_easy *,ssl_peer *,ssl_primary_config *,const char **,const char *)' already has a body
lib\vtls\mbedtls.c(837,49): warning C4133: 'function': incompatible types - from 'Curl_cfilter *' to 'Curl_easy *'
lib\vtls\mbedtls.c(837,53): warning C4133: 'function': incompatible types - from 'Curl_easy *' to 'ssl_peer *'
lib\vtls\mbedtls.c(837,25): error C2198: 'set_ssl_version_min_max': too few arguments for call
```
- use unique name for `do_file_type()`
Fixes collision between GnuTLS, OpenSSL and wolfSSL.
```
lib\vtls\openssl.c(1053,12): error C2084: function 'gnutls_x509_crt_fmt_t do_file_type(const char *)' already has a body
```
Ref: https://github.com/curl/curl/actions/runs/10341162641/job/28622681573?pr=14484#step:10:31
Cherry-picked from #14495Closes#14516
Channel Bindings are used to tie the session context to a specific TLS
channel. This is to provide additional proof of valid identity,
mitigating authentication relay attacks.
Major web servers have the ability to require (None/Accept/Require)
GSSAPI channel binding, rendering Curl unable to connect to such
websites unless support for channel bindings is implemented.
IIS calls this feature Extended Protection (EPA), which is used in
Enterprise environments using Kerberos for authentication.
This change require krb5 >= 1.19, otherwise channel bindings won't be
forwarded through SPNEGO.
Co-Authored-By: Steffen Kieß <947515+steffen-kiess@users.noreply.github.com>
Closes#13098
Bring setting ciphers with mbedTLS in line with other SSL backends,
to make the curl interface more consistent across the backends.
Now the tls1.3 ciphers are set with the --tls13-ciphers option, when
not set the default tls1.3 ciphers are used. The tls1.2 (1.1, 1.0)
ciphers are set with the --ciphers option, when not set the default
tls1.2 ciphers are used. The ciphers available for the connection
are now a union of the tls1.3 and tls1.2 ciphers.
This changes the behaviour for mbedTLS when --ciphers is set, but
--tls13-ciphers is not set. Now the ciphers set with --ciphers
are combined with the default tls1.3 ciphers, whereas before solely
the ciphers of --ciphers were used.
Thus before when no tls1.3 ciphers were specified in --ciphers,
tls1.3 was completely disabled. This might not be what the user
expected, especially as this does not happen with OpenSSL.
Closes#14384
Adds a `bool eos` flag to send methods to indicate that the data
is the last chunk the invovled transfer wants to send to the server.
This will help protocol filters like HTTP/2 and 3 to forward the
stream's EOF flag and also allow to EAGAIN such calls when buffers
are not yet fully flushed.
Closes#14220
Adds a `bool eos` flag to send methods to indicate that the data is the
last chunk the invovled transfer wants to send to the server.
This will help protocol filters like HTTP/2 and 3 to forward the
stream's EOF flag and also allow to EAGAIN such calls when buffers are
not yet fully flushed.
Closes#14220
- deduplicate the code in many tls backends that check
for an existing id and delete it before adding the new one
- rename ssl_primary_config's `sessionid` bool to `cache_session`
Closes#14121
Based on the standards and guidelines we use for our documentation.
- expand contractions (they're => they are etc)
- host name = > hostname
- file name => filename
- user name = username
- man page => manpage
- run-time => runtime
- set-up => setup
- back-end => backend
- a HTTP => an HTTP
- Two spaces after a period => one space after period
Closes#14073
When libcurl discards a connection there are two phases this may go
through: "shutdown" and "closing". If a connection is aborted, the
shutdown phase is skipped and it is closed right away.
The connection filters attached to the connection implement the phases
in their `do_shutdown()` and `do_close()` callbacks. Filters carry now a
`shutdown` flags next to `connected` to keep track of the shutdown
operation.
Filters are shut down from top to bottom. If a filter is not connected,
its shutdown is skipped. Notable filters that *do* something during
shutdown are HTTP/2 and TLS. HTTP/2 sends the GOAWAY frame. TLS sends
its close notify and expects to receive a close notify from the server.
As sends and receives may EAGAIN on the network, a shutdown is often not
successful right away and needs to poll the connection's socket(s). To
facilitate this, such connections are placed on a new shutdown list
inside the connection cache.
Since managing this list requires the cooperation of a multi handle,
only the connection cache belonging to a multi handle is used. If a
connection was in another cache when being discarded, it is removed
there and added to the multi's cache. If no multi handle is available at
that time, the connection is shutdown and closed in a one-time,
best-effort attempt.
When a multi handle is destroyed, all connection still on the shutdown
list are discarded with a final shutdown attempt and close. In curl
debug builds, the environment variable `CURL_GRACEFUL_SHUTDOWN` can be
set to make this graceful with a timeout in milliseconds given by the
variable.
The shutdown list is limited to the max number of connections configured
for a multi cache. Set via CURLMOPT_MAX_TOTAL_CONNECTIONS. When the
limit is reached, the oldest connection on the shutdown list is
discarded.
- In multi_wait() and multi_waitfds(), collect all connection caches
involved (each transfer might carry its own) into a temporary list.
Let each connection cache on the list contribute sockets and
POLLIN/OUT events it's connections are waiting for.
- in multi_perform() collect the connection caches the same way and let
them peform their maintenance. This will make another non-blocking
attempt to shutdown all connections on its shutdown list.
- for event based multis (multi->socket_cb set), add the sockets and
their poll events via the callback. When `multi_socket()` is invoked
for a socket not known by an active transfer, forward this to the
multi's cache for processing. On closing a connection, remove its
socket(s) via the callback.
TLS connection filters MUST NOT send close nofity messages in their
`do_close()` implementation. The reason is that a TLS close notify
signals a success. When a connection is aborted and skips its shutdown
phase, the server needs to see a missing close notify to detect
something has gone wrong.
A graceful shutdown of FTP's data connection is performed implicitly
before regarding the upload/download as complete and continuing on the
control connection. For FTP without TLS, there is just the socket close
happening. But with TLS, the sent/received close notify signals that the
transfer is complete and healthy. Servers like `vsftpd` verify that and
reject uploads without a TLS close notify.
- added test_19_* for shutdown related tests
- test_19_01 and test_19_02 test for TCP RST packets
which happen without a graceful shutdown and should
no longer appear otherwise.
- add test_19_03 for handling shutdowns by the server
- add test_19_04 for handling shutdowns by curl
- add test_19_05 for event based shutdowny by server
- add test_30_06/07 and test_31_06/07 for shutdown checks
on FTP up- and downloads.
Closes#13976
This adds connection shutdown infrastructure and first use for FTP. FTP
data connections, when not encountering an error, are now shut down in a
blocking way with a 2sec timeout.
- add cfilter `Curl_cft_shutdown` callback
- keep a shutdown start timestamp and timeout at connectdata
- provide shutdown timeout default and member in
`data->set.shutdowntimeout`.
- provide methods for starting, interrogating and clearing
shutdown timers
- provide `Curl_conn_shutdown_blocking()` to shutdown the
`sockindex` filter chain in a blocking way. Use that in FTP.
- add `Curl_conn_cf_poll()` to wait for socket events during
shutdown of a connection filter chain.
This gets the monitoring sockets and events via the filters
"adjust_pollset()" methods. This gives correct behaviour when
shutting down a TLS connection through a HTTP/2 proxy.
- Implement shutdown for all socket filters
- for HTTP/2 and h2 proxying to send GOAWAY
- for TLS backends to the best of their capabilities
- for tcp socket filter to make a final, nonblocking
receive to avoid unwanted RST states
- add shutdown forwarding to happy eyeballers and
https connect ballers when applicable.
Closes#13904
- decouple need to recv/send from negotiation state, we need
this later in shutdown handling as well
- move ssl enums from urldata.h to vtls_int.h
- implement use of `connssl->io_need` in vtls.c. and all backends
Closes#13879
- add special sauce to disable unwanted peer verification by mbedtls
when negotiating TLS v1.3
- add special sauce for MBEDTLS_ERR_SSL_RECEIVED_NEW_SESSION_TICKET
return code on *writing* TLS data. We assume the data had not been
written and EAGAIN.
- return correct Curl error code when peer verification failed.
- disable test_08_05 with 50 HTTP/1.1 connections, as mbedtls reports a
memory allocation failed during handshake.
- bump CI mbedtls version to 3.6.0
Fixes#13653Closes#13838
- add `Curl_hash_add2()` that passes a destructor function for
the element added. Call element destructor instead of hash
destructor if present.
- multi: add `proto_hash` for protocol related information,
remove `struct multi_ssl_backend_data`.
- openssl: use multi->proto_hash to keep x509 shared store
- schannel: use multi->proto_hash to keep x509 shared store
- vtls: remove Curl_free_multi_ssl_backend_data() and its
equivalents in the TLS backends
Closes#13345
- send the TLS close notify message when cloding down
the mbedtls connection filter
- this is a "least" effort version and, as other TLS filters,
is lacking a graceful send/receive/timeout for a really
clean shutdown.
Closes#13745
- add session with destructor callback
- remove vtls `session_free` method
- let `Curl_ssl_addsessionid()` take ownership
of session object, freeing it also on failures
- change tls backend use
- test_17, add tests for SSL session resumption
Closes#13386
Use a lookup list to set the cipher suites, allowing the
ciphers to be set by either openssl or IANA names.
To keep the binary size of the lookup list down we compress
each entry in the cipher list down to 2 + 6 bytes using the
C preprocessor.
Closes#13442
