Give more detailed reasons for certificate failures where available in
gnutls and wolfssi to allow user to understand the cause of the failure.
Closes#14501
When using CMake Unity build.
- use unique name for `set_ssl_version_min_max()`
Fixes collision between GnuTLS, mbedTLS and SecureTransport.
```
lib\vtls\mbedtls.c(317,1): error C2084: function 'CURLcode set_ssl_version_min_max(Curl_easy *,ssl_peer *,ssl_primary_config *,const char **,const char *)' already has a body
lib\vtls\mbedtls.c(837,49): warning C4133: 'function': incompatible types - from 'Curl_cfilter *' to 'Curl_easy *'
lib\vtls\mbedtls.c(837,53): warning C4133: 'function': incompatible types - from 'Curl_easy *' to 'ssl_peer *'
lib\vtls\mbedtls.c(837,25): error C2198: 'set_ssl_version_min_max': too few arguments for call
```
- use unique name for `do_file_type()`
Fixes collision between GnuTLS, OpenSSL and wolfSSL.
```
lib\vtls\openssl.c(1053,12): error C2084: function 'gnutls_x509_crt_fmt_t do_file_type(const char *)' already has a body
```
Ref: https://github.com/curl/curl/actions/runs/10341162641/job/28622681573?pr=14484#step:10:31
Cherry-picked from #14495Closes#14516
Before, setting CURLOPT_SSLVERSION with wolfSSL restricted the the tls
proto to just the specified version. Now it properly supports a range.
So it can set the min and max tls proto (max requires wolfSSL 4.2.0).
Bump the absolute minimum required version of wolfSSL to 3.4.6 (released
2015) because it is needed for the wolfSSL_CTX_SetMinVersion() function.
Closes#14480
Channel Bindings are used to tie the session context to a specific TLS
channel. This is to provide additional proof of valid identity,
mitigating authentication relay attacks.
Major web servers have the ability to require (None/Accept/Require)
GSSAPI channel binding, rendering Curl unable to connect to such
websites unless support for channel bindings is implemented.
IIS calls this feature Extended Protection (EPA), which is used in
Enterprise environments using Kerberos for authentication.
This change require krb5 >= 1.19, otherwise channel bindings won't be
forwarded through SPNEGO.
Co-Authored-By: Steffen Kieß <947515+steffen-kiess@users.noreply.github.com>
Closes#13098
Bring setting ciphers with WolfSSL in line with other SSL backends,
to make the curl interface more consistent across the backends.
Now the tls1.3 ciphers are set with the --tls13-ciphers option, when
not set the default tls1.3 ciphers are used. The tls1.2 (1.1, 1.0)
ciphers are set with the --ciphers option, when not set the default
tls1.2 ciphers are used. The ciphers available for the connection
are now a union of the tls1.3 and tls1.2 ciphers.
This changes the behaviour for WolfSSL when --ciphers is set, but
--tls13-ciphers is not set. Now the ciphers set with --ciphers
are combined with the default tls1.3 ciphers, whereas before solely
the ciphers of --ciphers were used.
Thus before when no tls1.3 ciphers were specified in --ciphers,
tls1.3 was completely disabled. This might not be what the user
expected, especially as this does not happen with OpenSSL.
Closes#14385
Improve handling of shutdown when sending gets blocked.
Add workaround for <https://github.com/wolfSSL/wolfssl/issues/7784>
where wolfSSL keeps on adding close notify messages to its outgoing
buffer on ever attempt.
Closes#14376
wolfSSL supports setting certificates/private keys from memory blobs
which allow us to implement both CURLOPT_SSLCERT_BLOB and
CURLOPT_SSLKEY_BLOB options.
Closes#14018
Adds a `bool eos` flag to send methods to indicate that the data
is the last chunk the invovled transfer wants to send to the server.
This will help protocol filters like HTTP/2 and 3 to forward the
stream's EOF flag and also allow to EAGAIN such calls when buffers
are not yet fully flushed.
Closes#14220
Replace Curl_resolv_unlock() with Curl_resolv_unlink():
-replace inuse member with refcount in Curl_dns_entry
- pass Curl_dns_entry ** to unlink, so it gets always cleared
- solve potential (but unlikley) UAF in FTP's handling of looked up
Curl_dns_entry. Esp. do not use addr information after unlinking an entry.
In reality, the unlink will not free memory, as the dns entry is still
referenced by the hostcache. But this is not safe and relying on no other
code pruning the cache in the meantime.
- pass permanent flag when adding a dns entry instead of fixing timestamp
afterwards.
url.c: fold several static *resolve_* functions into one.
Closes#14195
When sharing the x509 store in wolfSSL, always use an explicitly
constructed one, as the SSLCTX might have "only" an internal one which
is not obeying reference count lifetimes.
Fixes#14278
Reported-by: Alex Snast
Closes#14279
Adds a `bool eos` flag to send methods to indicate that the data is the
last chunk the invovled transfer wants to send to the server.
This will help protocol filters like HTTP/2 and 3 to forward the
stream's EOF flag and also allow to EAGAIN such calls when buffers are
not yet fully flushed.
Closes#14220
Fix the file of wolfssl.c because of this warning/error:
```
curl\lib\vtls\wolfssl.c(1017,42): error C2220: the following warning is treated as an error [curl\bld\lib\libcurl_object.vcxproj]
curl\lib\vtls\wolfssl.c(1017,42): warning C4267: 'function': conversion from 'size_t' to 'unsigned long', possible loss of data [curl\bld\lib\libcurl_object.vcxproj]
```
`size_t` in MSVC is different. Change it to `unsigned long` because
`wolfSSL_ERR_error_string_n` last argument is defined as
`unsigned long`.
Closes#14193
Currently we're using WOLFSSL_MAX_ERROR_SZ to define the error buffer
size, this value is user defined which means it can be overwritten with
-DWOLFSSL_MAX_ERROR_SZ=512 when building wolfssl and this overwrite is
not exported to the users of wolfssl.
Instead of relying on WOLFSSL_MAX_ERROR_SZ we'll just use a 256 bytes
error buffer and use wolfSSL_ERR_error_string_n to fill it thus dropping
the dependency on WOLFSSL_MAX_ERROR_SZ altogether.
Closes#14114
- deduplicate the code in many tls backends that check
for an existing id and delete it before adding the new one
- rename ssl_primary_config's `sessionid` bool to `cache_session`
Closes#14121
Based on the standards and guidelines we use for our documentation.
- expand contractions (they're => they are etc)
- host name = > hostname
- file name => filename
- user name = username
- man page => manpage
- run-time => runtime
- set-up => setup
- back-end => backend
- a HTTP => an HTTP
- Two spaces after a period => one space after period
Closes#14073
When libcurl discards a connection there are two phases this may go
through: "shutdown" and "closing". If a connection is aborted, the
shutdown phase is skipped and it is closed right away.
The connection filters attached to the connection implement the phases
in their `do_shutdown()` and `do_close()` callbacks. Filters carry now a
`shutdown` flags next to `connected` to keep track of the shutdown
operation.
Filters are shut down from top to bottom. If a filter is not connected,
its shutdown is skipped. Notable filters that *do* something during
shutdown are HTTP/2 and TLS. HTTP/2 sends the GOAWAY frame. TLS sends
its close notify and expects to receive a close notify from the server.
As sends and receives may EAGAIN on the network, a shutdown is often not
successful right away and needs to poll the connection's socket(s). To
facilitate this, such connections are placed on a new shutdown list
inside the connection cache.
Since managing this list requires the cooperation of a multi handle,
only the connection cache belonging to a multi handle is used. If a
connection was in another cache when being discarded, it is removed
there and added to the multi's cache. If no multi handle is available at
that time, the connection is shutdown and closed in a one-time,
best-effort attempt.
When a multi handle is destroyed, all connection still on the shutdown
list are discarded with a final shutdown attempt and close. In curl
debug builds, the environment variable `CURL_GRACEFUL_SHUTDOWN` can be
set to make this graceful with a timeout in milliseconds given by the
variable.
The shutdown list is limited to the max number of connections configured
for a multi cache. Set via CURLMOPT_MAX_TOTAL_CONNECTIONS. When the
limit is reached, the oldest connection on the shutdown list is
discarded.
- In multi_wait() and multi_waitfds(), collect all connection caches
involved (each transfer might carry its own) into a temporary list.
Let each connection cache on the list contribute sockets and
POLLIN/OUT events it's connections are waiting for.
- in multi_perform() collect the connection caches the same way and let
them peform their maintenance. This will make another non-blocking
attempt to shutdown all connections on its shutdown list.
- for event based multis (multi->socket_cb set), add the sockets and
their poll events via the callback. When `multi_socket()` is invoked
for a socket not known by an active transfer, forward this to the
multi's cache for processing. On closing a connection, remove its
socket(s) via the callback.
TLS connection filters MUST NOT send close nofity messages in their
`do_close()` implementation. The reason is that a TLS close notify
signals a success. When a connection is aborted and skips its shutdown
phase, the server needs to see a missing close notify to detect
something has gone wrong.
A graceful shutdown of FTP's data connection is performed implicitly
before regarding the upload/download as complete and continuing on the
control connection. For FTP without TLS, there is just the socket close
happening. But with TLS, the sent/received close notify signals that the
transfer is complete and healthy. Servers like `vsftpd` verify that and
reject uploads without a TLS close notify.
- added test_19_* for shutdown related tests
- test_19_01 and test_19_02 test for TCP RST packets
which happen without a graceful shutdown and should
no longer appear otherwise.
- add test_19_03 for handling shutdowns by the server
- add test_19_04 for handling shutdowns by curl
- add test_19_05 for event based shutdowny by server
- add test_30_06/07 and test_31_06/07 for shutdown checks
on FTP up- and downloads.
Closes#13976
When user sets CURLOPT_SSLCERT but leaves CURLOPT_SSLKEY unset assume
the path passed in CURLOPT_SSLCERT holds the ssl key which is what we do
in openssl implementation.
Fixes#14007Closes#14008
This adds connection shutdown infrastructure and first use for FTP. FTP
data connections, when not encountering an error, are now shut down in a
blocking way with a 2sec timeout.
- add cfilter `Curl_cft_shutdown` callback
- keep a shutdown start timestamp and timeout at connectdata
- provide shutdown timeout default and member in
`data->set.shutdowntimeout`.
- provide methods for starting, interrogating and clearing
shutdown timers
- provide `Curl_conn_shutdown_blocking()` to shutdown the
`sockindex` filter chain in a blocking way. Use that in FTP.
- add `Curl_conn_cf_poll()` to wait for socket events during
shutdown of a connection filter chain.
This gets the monitoring sockets and events via the filters
"adjust_pollset()" methods. This gives correct behaviour when
shutting down a TLS connection through a HTTP/2 proxy.
- Implement shutdown for all socket filters
- for HTTP/2 and h2 proxying to send GOAWAY
- for TLS backends to the best of their capabilities
- for tcp socket filter to make a final, nonblocking
receive to avoid unwanted RST states
- add shutdown forwarding to happy eyeballers and
https connect ballers when applicable.
Closes#13904
- decouple need to recv/send from negotiation state, we need
this later in shutdown handling as well
- move ssl enums from urldata.h to vtls_int.h
- implement use of `connssl->io_need` in vtls.c. and all backends
Closes#13879
- add `Curl_hash_add2()` that passes a destructor function for
the element added. Call element destructor instead of hash
destructor if present.
- multi: add `proto_hash` for protocol related information,
remove `struct multi_ssl_backend_data`.
- openssl: use multi->proto_hash to keep x509 shared store
- schannel: use multi->proto_hash to keep x509 shared store
- vtls: remove Curl_free_multi_ssl_backend_data() and its
equivalents in the TLS backends
Closes#13345
- add session with destructor callback
- remove vtls `session_free` method
- let `Curl_ssl_addsessionid()` take ownership
of session object, freeing it also on failures
- change tls backend use
- test_17, add tests for SSL session resumption
Closes#13386
Reduced size of dynamically_allocated_data structure.
Reduced number of stored values in enum dupstring and enum dupblob. This
affects the reduced array placed in the UserDefined structure.
Closes#13188
Fixes:
test 2034...[simple HTTPS GET with DER public key pinning]
==61829== 22,610 (3,744 direct, 18,866 indirect) bytes in 1 blocks are definitely lost in loss record 51 of 54
==61829== at 0x484BB74: malloc (vg_replace_malloc.c:446)
==61829== by 0x4B53A80: wolfSSL_Malloc (memory.c:344)
==61829== by 0x4C1C8E1: wolfSSL_X509_new (x509.c:5326)
==61829== by 0x4C3977D: d2i_X509orX509REQ (x509.c:3628)
==61829== by 0x4C1D1F4: wolfSSL_X509_d2i (x509.c:3664)
==61829== by 0x4C1C37B: wolfSSL_X509_dup (x509.c:13425)
==61829== by 0x4C197DB: wolfSSL_get_peer_certificate (ssl.c:18765)
==61829== by 0x33297C: wolfssl_connect_step2 (wolfssl.c:875)
==61829== by 0x331669: wolfssl_connect_common (wolfssl.c:1287)
==61829== by 0x3303E9: wolfssl_connect_nonblocking (wolfssl.c:1319)
==61829== by 0x32FE89: ssl_connect_nonblocking (vtls.c:510)
==61829== by 0x32DBE5: ssl_cf_connect (vtls.c:1679)
==61829== by 0x27ABD7: Curl_conn_cf_connect (cfilters.c:307)
==61829== by 0x27D9CF: cf_setup_connect (connect.c:1199)
==61829== by 0x27ABD7: Curl_conn_cf_connect (cfilters.c:307)
==61829== by 0x283CEA: cf_hc_baller_connect (cf-https-connect.c:135)
Closes#13272
Calling the function isn't necessary and causes the build
to fail when wolfSSL has been compiled with NO_WOLFSSL_STUB:
Making all in opts
CCLD curl
ld: error: undefined symbol: wolfSSL_BIO_set_init
>>> referenced by wolfssl.c:235 (vtls/wolfssl.c:235)
>>> libcurl_la-wolfssl.o:(wolfssl_bio_cf_create) in archive ../lib/.libs/libcurl.a
cc: error: linker command failed with exit code 1 (use -v to see invocation)
*** Error code 1
Closes#13164
- When curl sees a TCP close from the peer, do not start a TLS shutdown.
TLS shutdown is a handshake and if the peer already closed the
connection, it is not interested in participating.
Reported-by: dfdity on github
Assisted-by: Jiří Bok
Assisted-by: Pēteris Caune
Fixes#10290Closes#13087
https://best.openssf.org/Compiler-Hardening-Guides/Compiler-Options-Hardening-Guide-for-C-and-C++.html
as of 2023-11-29 [1].
Enable new recommended warnings (except `-Wsign-conversion`):
- enable `-Wformat=2` for clang (in both cmake and autotools).
- add `CURL_PRINTF()` internal attribute and mark functions accepting
printf arguments with it. This is a copy of existing
`CURL_TEMP_PRINTF()` but using `__printf__` to make it compatible
with redefinting the `printf` symbol:
https://gcc.gnu.org/onlinedocs/gcc-3.0.4/gcc_5.html#SEC94
- fix `CURL_PRINTF()` and existing `CURL_TEMP_PRINTF()` for
mingw-w64 and enable it on this platform.
- enable `-Wimplicit-fallthrough`.
- enable `-Wtrampolines`.
- add `-Wsign-conversion` commented with a FIXME.
- cmake: enable `-pedantic-errors` the way we do it with autotools.
Follow-up to d5c0351055#2747
- lib/curl_trc.h: use `CURL_FORMAT()`, this also fixes it to enable format
checks. Previously it was always disabled due to the internal `printf`
macro.
Fix them:
- fix bug where an `set_ipv6_v6only()` call was missed in builds with
`--disable-verbose` / `CURL_DISABLE_VERBOSE_STRINGS=ON`.
- add internal `FALLTHROUGH()` macro.
- replace obsolete fall-through comments with `FALLTHROUGH()`.
- fix fallthrough markups: Delete redundant ones (showing up as
warnings in most cases). Add missing ones. Fix indentation.
- silence `-Wformat-nonliteral` warnings with llvm/clang.
- fix one `-Wformat-nonliteral` warning.
- fix new `-Wformat` and `-Wformat-security` warnings.
- fix `CURL_FORMAT_SOCKET_T` value for mingw-w64. Also move its
definition to `lib/curl_setup.h` allowing use in `tests/server`.
- lib: fix two wrongly passed string arguments in log outputs.
Co-authored-by: Jay Satiro
- fix new `-Wformat` warnings on mingw-w64.
[1] 56c0fde389/docs/Compiler-Hardening-Guides/Compiler-Options-Hardening-Guide-for-C-and-C%2B%2B.mdCloses#12489
- add `struct ssl_peer` to keep hostname, dispname and sni
for a filter
- allocate `sni` for use in VTLS backend
- eliminate `Curl_ssl_snihost()` and its use of the download buffer
- use ssl_peer in SSL and QUIC filters
Closes#12349
Connection filter had a `get_select_socks()` method, inspired by the
various `getsocks` functions involved during the lifetime of a
transfer. These, depending on transfer state (CONNECT/DO/DONE/ etc.),
return sockets to monitor and flag if this shall be done for POLLIN
and/or POLLOUT.
Due to this design, sockets and flags could only be added, not
removed. This led to problems in filters like HTTP/2 where flow control
prohibits the sending of data until the peer increases the flow
window. The general transfer loop wants to write, adds POLLOUT, the
socket is writeable but no data can be written.
This leads to cpu busy loops. To prevent that, HTTP/2 did set the
`SEND_HOLD` flag of such a blocked transfer, so the transfer loop cedes
further attempts. This works if only one such filter is involved. If a
HTTP/2 transfer goes through a HTTP/2 proxy, two filters are
setting/clearing this flag and may step on each other's toes.
Connection filters `get_select_socks()` is replaced by
`adjust_pollset()`. They get passed a `struct easy_pollset` that keeps
up to `MAX_SOCKSPEREASYHANDLE` sockets and their `POLLIN|POLLOUT`
flags. This struct is initialized in `multi_getsock()` by calling the
various `getsocks()` implementations based on transfer state, as before.
After protocol handlers/transfer loop have set the sockets and flags
they want, the `easy_pollset` is *always* passed to the filters. Filters
"higher" in the chain are called first, starting at the first
not-yet-connection one. Each filter may add sockets and/or change
flags. When all flags are removed, the socket itself is removed from the
pollset.
Example:
* transfer wants to send, adds POLLOUT
* http/2 filter has a flow control block, removes POLLOUT and adds
POLLIN (it is waiting on a WINDOW_UPDATE from the server)
* TLS filter is connected and changes nothing
* h2-proxy filter also has a flow control block on its tunnel stream,
removes POLLOUT and adds POLLIN also.
* socket filter is connected and changes nothing
* The resulting pollset is then mixed together with all other transfers
and their pollsets, just as before.
Use of `SEND_HOLD` is no longer necessary in the filters.
All filters are adapted for the changed method. The handling in
`multi.c` has been adjusted, but its state handling the the protocol
handlers' `getsocks` method are untouched.
The most affected filters are http/2, ngtcp2, quiche and h2-proxy. TLS
filters needed to be adjusted for the connecting handshake read/write
handling.
No noticeable difference in performance was detected in local scorecard
runs.
Closes#11833
This define is set in wolfssl's options.h file when this function and
feature is present. Handles both builds with the feature explicitly
disabled and wolfSSL versions before 5.5.2 - which introduced this API
call.
Closes#12108
By using unique static function/variable names in source files
implementing these interfaces.
- OpenLDAP combined with any SSH backend.
- MultiSSL with mbedTLS, OpenSSL, wolfSSL, SecureTransport.
Closes#12027
The default wolfSSL_CTX_load_verify_locations() function is quite picky
with the certificates it loads and will for example return error if just
one of the certs has expired.
With the *_ex() function and its WOLFSSL_LOAD_FLAG_IGNORE_ERR flag, it
behaves more similar to what OpenSSL does by default.
Even the set of default certs on my Debian unstable has several expired
ones.
Assisted-by: Juliusz Sosinowicz
Assisted-by: Michael Osipov
Closes#11987
- added test cases for various code paths
- fixed handling of blocked write when stream had
been closed inbetween attempts
- re-enabled DEBUGASSERT on send with smaller data size
- in debug builds, environment variables can be set to simulate a slow
network when sending data. cf-socket.c and vquic.c support
* CURL_DBG_SOCK_WBLOCK: percentage of send() calls that should be
answered with a EAGAIN. TCP/UNIX sockets.
This is chosen randomly.
* CURL_DBG_SOCK_WPARTIAL: percentage of data that shall be written
to the network. TCP/UNIX sockets.
Example: 80 means a send with 1000 bytes would only send 800
This is applied to every send.
* CURL_DBG_QUIC_WBLOCK: percentage of send() calls that should be
answered with EAGAIN. QUIC only.
This is chosen randomly.
Closes#11756
