- deduplicate the code in many tls backends that check
for an existing id and delete it before adding the new one
- rename ssl_primary_config's `sessionid` bool to `cache_session`
Closes#14121
Based on the standards and guidelines we use for our documentation.
- expand contractions (they're => they are etc)
- host name = > hostname
- file name => filename
- user name = username
- man page => manpage
- run-time => runtime
- set-up => setup
- back-end => backend
- a HTTP => an HTTP
- Two spaces after a period => one space after period
Closes#14073
When libcurl discards a connection there are two phases this may go
through: "shutdown" and "closing". If a connection is aborted, the
shutdown phase is skipped and it is closed right away.
The connection filters attached to the connection implement the phases
in their `do_shutdown()` and `do_close()` callbacks. Filters carry now a
`shutdown` flags next to `connected` to keep track of the shutdown
operation.
Filters are shut down from top to bottom. If a filter is not connected,
its shutdown is skipped. Notable filters that *do* something during
shutdown are HTTP/2 and TLS. HTTP/2 sends the GOAWAY frame. TLS sends
its close notify and expects to receive a close notify from the server.
As sends and receives may EAGAIN on the network, a shutdown is often not
successful right away and needs to poll the connection's socket(s). To
facilitate this, such connections are placed on a new shutdown list
inside the connection cache.
Since managing this list requires the cooperation of a multi handle,
only the connection cache belonging to a multi handle is used. If a
connection was in another cache when being discarded, it is removed
there and added to the multi's cache. If no multi handle is available at
that time, the connection is shutdown and closed in a one-time,
best-effort attempt.
When a multi handle is destroyed, all connection still on the shutdown
list are discarded with a final shutdown attempt and close. In curl
debug builds, the environment variable `CURL_GRACEFUL_SHUTDOWN` can be
set to make this graceful with a timeout in milliseconds given by the
variable.
The shutdown list is limited to the max number of connections configured
for a multi cache. Set via CURLMOPT_MAX_TOTAL_CONNECTIONS. When the
limit is reached, the oldest connection on the shutdown list is
discarded.
- In multi_wait() and multi_waitfds(), collect all connection caches
involved (each transfer might carry its own) into a temporary list.
Let each connection cache on the list contribute sockets and
POLLIN/OUT events it's connections are waiting for.
- in multi_perform() collect the connection caches the same way and let
them peform their maintenance. This will make another non-blocking
attempt to shutdown all connections on its shutdown list.
- for event based multis (multi->socket_cb set), add the sockets and
their poll events via the callback. When `multi_socket()` is invoked
for a socket not known by an active transfer, forward this to the
multi's cache for processing. On closing a connection, remove its
socket(s) via the callback.
TLS connection filters MUST NOT send close nofity messages in their
`do_close()` implementation. The reason is that a TLS close notify
signals a success. When a connection is aborted and skips its shutdown
phase, the server needs to see a missing close notify to detect
something has gone wrong.
A graceful shutdown of FTP's data connection is performed implicitly
before regarding the upload/download as complete and continuing on the
control connection. For FTP without TLS, there is just the socket close
happening. But with TLS, the sent/received close notify signals that the
transfer is complete and healthy. Servers like `vsftpd` verify that and
reject uploads without a TLS close notify.
- added test_19_* for shutdown related tests
- test_19_01 and test_19_02 test for TCP RST packets
which happen without a graceful shutdown and should
no longer appear otherwise.
- add test_19_03 for handling shutdowns by the server
- add test_19_04 for handling shutdowns by curl
- add test_19_05 for event based shutdowny by server
- add test_30_06/07 and test_31_06/07 for shutdown checks
on FTP up- and downloads.
Closes#13976
This adds connection shutdown infrastructure and first use for FTP. FTP
data connections, when not encountering an error, are now shut down in a
blocking way with a 2sec timeout.
- add cfilter `Curl_cft_shutdown` callback
- keep a shutdown start timestamp and timeout at connectdata
- provide shutdown timeout default and member in
`data->set.shutdowntimeout`.
- provide methods for starting, interrogating and clearing
shutdown timers
- provide `Curl_conn_shutdown_blocking()` to shutdown the
`sockindex` filter chain in a blocking way. Use that in FTP.
- add `Curl_conn_cf_poll()` to wait for socket events during
shutdown of a connection filter chain.
This gets the monitoring sockets and events via the filters
"adjust_pollset()" methods. This gives correct behaviour when
shutting down a TLS connection through a HTTP/2 proxy.
- Implement shutdown for all socket filters
- for HTTP/2 and h2 proxying to send GOAWAY
- for TLS backends to the best of their capabilities
- for tcp socket filter to make a final, nonblocking
receive to avoid unwanted RST states
- add shutdown forwarding to happy eyeballers and
https connect ballers when applicable.
Closes#13904
- decouple need to recv/send from negotiation state, we need
this later in shutdown handling as well
- move ssl enums from urldata.h to vtls_int.h
- implement use of `connssl->io_need` in vtls.c. and all backends
Closes#13879
The versions check wrongly complained and return error if the *minimum*
version was set to something less than 1.3. QUIC is always TLS 1.3, but
that means minimum 1.2 is still fine to ask for.
This also renames the local variable to make the mistake harder to make
in the future.
Regression shipped in 8.8.0
Follow-up to 3210101088
Reported-by: fds242 on github
Fixes#13799Closes#13802
- add `Curl_hash_add2()` that passes a destructor function for
the element added. Call element destructor instead of hash
destructor if present.
- multi: add `proto_hash` for protocol related information,
remove `struct multi_ssl_backend_data`.
- openssl: use multi->proto_hash to keep x509 shared store
- schannel: use multi->proto_hash to keep x509 shared store
- vtls: remove Curl_free_multi_ssl_backend_data() and its
equivalents in the TLS backends
Closes#13345
- Don't call the keylog function if it has already logged the key.
For old OpenSSL versions and its forks that do not have support for
OpenSSL's keylog callback, libcurl has its own legacy key logging
function that logs the TLS 1.2 (and earlier) key (client random + master
key) on a single line.
Prior to this change, since e7de80e8 (precedes 8.8.0), the legacy key
logging function could write the same key line more than once (usually
twice) due to some incorrect logic.
Closes https://github.com/curl/curl/pull/13683
- Revert to the legacy TLS 1.2 key logging code for LibreSSL.
- Document SSLKEYLOGFILE for LibreSSL is TLS 1.2 max.
Prior to this change if the user specified a filename in the
SSLKEYLOGFILE environment variable and was using LibreSSL 3.5.0+ then
an empty file would be created and no keys would be logged.
This is effectively a revert of e43474b4 which changed openssl.c to use
SSL_CTX_set_keylog_callback for LibreSSL 3.5.0+. Unfortunately LibreSSL
added that function only as a stub that doesn't actually do anything.
Reported-by: Gonçalo Carvalho
Fixes https://github.com/curl/curl/issues/13672
Closes https://github.com/curl/curl/pull/13682
- add session with destructor callback
- remove vtls `session_free` method
- let `Curl_ssl_addsessionid()` take ownership
of session object, freeing it also on failures
- change tls backend use
- test_17, add tests for SSL session resumption
Closes#13386
Before this patch, two macros were used to guard IPv6 features in curl
sources: `ENABLE_IPV6` and `USE_IPV6`. This patch makes the source use
the latter for consistency with other similar switches.
`-DENABLE_IPV6` remains accepted for compatibility as a synonym for
`-DUSE_IPV6`, when passed to the compiler.
`ENABLE_IPV6` also remains the name of the CMake and `Makefile.vc`
options to control this feature.
Closes#13349
- Use data->multi and not data->multi_easy to refer to the active multi.
The easy handle's active multi is always data->multi.
This is a follow up to 757dfdf which changed curl so that an easy handle
used with the easy interface and then multi interface cannot have two
different multi handles associated with it at the same time
(data->multi_easy from the easy interface and data->multi from the multi
interface).
Closes https://github.com/curl/curl/pull/12665
Reduced size of dynamically_allocated_data structure.
Reduced number of stored values in enum dupstring and enum dupblob. This
affects the reduced array placed in the UserDefined structure.
Closes#13188
- When curl sees a TCP close from the peer, do not start a TLS shutdown.
TLS shutdown is a handshake and if the peer already closed the
connection, it is not interested in participating.
Reported-by: dfdity on github
Assisted-by: Jiří Bok
Assisted-by: Pēteris Caune
Fixes#10290Closes#13087
- improve info logging when peer verification fails to indicate
if DNS name or ip address has been tried to match
- add test case for contacting https proxy with ip address
- add pytest env check on loaded credentials and re-issue
when they are no longer valid
- disable proxy ip address test for bearssl, since not supported there
Ref: #12831Closes#12838
To prevent that it gets used in a subsequent transfer that skips the
verifystatus check since that check can't be done when the session id is
reused.
Reported-by: Hiroki Kurosawa
Closes#12760
https://best.openssf.org/Compiler-Hardening-Guides/Compiler-Options-Hardening-Guide-for-C-and-C++.html
as of 2023-11-29 [1].
Enable new recommended warnings (except `-Wsign-conversion`):
- enable `-Wformat=2` for clang (in both cmake and autotools).
- add `CURL_PRINTF()` internal attribute and mark functions accepting
printf arguments with it. This is a copy of existing
`CURL_TEMP_PRINTF()` but using `__printf__` to make it compatible
with redefinting the `printf` symbol:
https://gcc.gnu.org/onlinedocs/gcc-3.0.4/gcc_5.html#SEC94
- fix `CURL_PRINTF()` and existing `CURL_TEMP_PRINTF()` for
mingw-w64 and enable it on this platform.
- enable `-Wimplicit-fallthrough`.
- enable `-Wtrampolines`.
- add `-Wsign-conversion` commented with a FIXME.
- cmake: enable `-pedantic-errors` the way we do it with autotools.
Follow-up to d5c0351055#2747
- lib/curl_trc.h: use `CURL_FORMAT()`, this also fixes it to enable format
checks. Previously it was always disabled due to the internal `printf`
macro.
Fix them:
- fix bug where an `set_ipv6_v6only()` call was missed in builds with
`--disable-verbose` / `CURL_DISABLE_VERBOSE_STRINGS=ON`.
- add internal `FALLTHROUGH()` macro.
- replace obsolete fall-through comments with `FALLTHROUGH()`.
- fix fallthrough markups: Delete redundant ones (showing up as
warnings in most cases). Add missing ones. Fix indentation.
- silence `-Wformat-nonliteral` warnings with llvm/clang.
- fix one `-Wformat-nonliteral` warning.
- fix new `-Wformat` and `-Wformat-security` warnings.
- fix `CURL_FORMAT_SOCKET_T` value for mingw-w64. Also move its
definition to `lib/curl_setup.h` allowing use in `tests/server`.
- lib: fix two wrongly passed string arguments in log outputs.
Co-authored-by: Jay Satiro
- fix new `-Wformat` warnings on mingw-w64.
[1] 56c0fde389/docs/Compiler-Hardening-Guides/Compiler-Options-Hardening-Guide-for-C-and-C%2B%2B.mdCloses#12489
Lower the barrier to enable `infof_certstack()` from OpenSSL 3 to
OpenSSL 1.1.x, and LibreSSL 3.6 or upper.
With the caveat, that "group name" and "type name" are missing from
the log output with these TLS backends.
Follow-up to b6e6d4ff8f#12030
Reviewed-by: Daniel Stenberg
Closes#12385
- build quictls with `no-deprecated` in CI to have test coverage for
this OpenSSL 3 configuration.
- don't call `OpenSSL_add_all_algorithms()`, `OpenSSL_add_all_digests()`.
The caller code is meant for OpenSSL 3, while these two functions were
only necessary before OpenSSL 1.1.0. They are missing from OpenSSL 3
if built with option `no-deprecated`, causing build errors:
```
vtls/openssl.c:4097:3: error: call to undeclared function 'OpenSSL_add_all_algorithms'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
vtls/openssl.c:4098:3: error: call to undeclared function 'OpenSSL_add_all_digests'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
```
Ref: https://ci.appveyor.com/project/curlorg/curl-for-win/builds/48587418?fullLog=true#L7667
Regression from b6e6d4ff8f#12030
Bug: https://github.com/curl/curl/issues/12380#issuecomment-1822944669
Reviewed-by: Alex Bozarth
- vquic/curl_ngtcp2: fix using `SSL_get_peer_certificate` with
`no-deprecated` quictls 3 builds.
Do it by moving an existing solution for this from `vtls/openssl.c`
to `vtls/openssl.h` and adjusting caller code.
```
vquic/curl_ngtcp2.c:1950:19: error: implicit declaration of function 'SSL_get_peer_certificate'; did you mean 'SSL_get1_peer_certificate'? [-Wimplicit-function-declaration]
```
Ref: https://github.com/curl/curl/actions/runs/6960723097/job/18940818625#step:24:1178
- curl_ntlm_core: fix `-Wunused-parameter`, `-Wunused-variable` and
`-Wunused-function` when trying to build curl with NTLM enabled but
without the necessary TLS backend (with DES) support.
Closes#12384
- add `struct ssl_peer` to keep hostname, dispname and sni
for a filter
- allocate `sni` for use in VTLS backend
- eliminate `Curl_ssl_snihost()` and its use of the download buffer
- use ssl_peer in SSL and QUIC filters
Closes#12349
Uniformly use `OPENSSL_VERSION_NUMBER` to check for OpenSSL version.
Before this patch some places used `OPENSSL_VERSION_MAJOR`.
Also fix `lib/md4.c`, which included `opensslconf.h`, but that doesn't
define any version number in these implementations: BoringSSL, AWS-LC,
LibreSSL, wolfSSL. (Only in mainline OpenSSL/quictls). Switch that to
`opensslv.h`. This wasn't causing a deeper problem because the code is
looking for v3, which is only provided by OpenSSL/quictls as of now.
According to https://github.com/openssl/openssl/issues/17517, the macro
`OPENSSL_VERSION_NUMBER` is safe to use and not deprecated.
Reviewed-by: Marcel Raad
Closes#12298
Since vanilla OpenSSL does not support the QUIC API I think it helps
users to identify the correct OpenSSL fork in version output. The best
(crude) way to do that right now seems to be to check if ngtcp2 support
is enabled.
Closes#12270
- If CURLSSLOPT_NATIVE_CA on Windows then import from intermediate CA
"CA" store after importing from root CA "ROOT" store.
This change allows curl to work in situations where a server does not
send all intermediate certs and they are present in the "CA" store (the
store with intermediate CAs). This is already allowed by the Schannel
backend.
Also this change makes partial chain verification possible for those
certs since we allow partial chain verification by default for OpenSSL
(unless CURLSSLOPT_NO_PARTIALCHAIN). This is not allowed by the Schannel
backend.
Prior to this change CURLSSLOPT_NATIVE_CA only imported "ROOT" certs.
Fixes https://github.com/curl/curl/issues/12155
Closes https://github.com/curl/curl/pull/12185
Connection filter had a `get_select_socks()` method, inspired by the
various `getsocks` functions involved during the lifetime of a
transfer. These, depending on transfer state (CONNECT/DO/DONE/ etc.),
return sockets to monitor and flag if this shall be done for POLLIN
and/or POLLOUT.
Due to this design, sockets and flags could only be added, not
removed. This led to problems in filters like HTTP/2 where flow control
prohibits the sending of data until the peer increases the flow
window. The general transfer loop wants to write, adds POLLOUT, the
socket is writeable but no data can be written.
This leads to cpu busy loops. To prevent that, HTTP/2 did set the
`SEND_HOLD` flag of such a blocked transfer, so the transfer loop cedes
further attempts. This works if only one such filter is involved. If a
HTTP/2 transfer goes through a HTTP/2 proxy, two filters are
setting/clearing this flag and may step on each other's toes.
Connection filters `get_select_socks()` is replaced by
`adjust_pollset()`. They get passed a `struct easy_pollset` that keeps
up to `MAX_SOCKSPEREASYHANDLE` sockets and their `POLLIN|POLLOUT`
flags. This struct is initialized in `multi_getsock()` by calling the
various `getsocks()` implementations based on transfer state, as before.
After protocol handlers/transfer loop have set the sockets and flags
they want, the `easy_pollset` is *always* passed to the filters. Filters
"higher" in the chain are called first, starting at the first
not-yet-connection one. Each filter may add sockets and/or change
flags. When all flags are removed, the socket itself is removed from the
pollset.
Example:
* transfer wants to send, adds POLLOUT
* http/2 filter has a flow control block, removes POLLOUT and adds
POLLIN (it is waiting on a WINDOW_UPDATE from the server)
* TLS filter is connected and changes nothing
* h2-proxy filter also has a flow control block on its tunnel stream,
removes POLLOUT and adds POLLIN also.
* socket filter is connected and changes nothing
* The resulting pollset is then mixed together with all other transfers
and their pollsets, just as before.
Use of `SEND_HOLD` is no longer necessary in the filters.
All filters are adapted for the changed method. The handling in
`multi.c` has been adjusted, but its state handling the the protocol
handlers' `getsocks` method are untouched.
The most affected filters are http/2, ngtcp2, quiche and h2-proxy. TLS
filters needed to be adjusted for the connecting handshake read/write
handling.
No noticeable difference in performance was detected in local scorecard
runs.
Closes#11833
Currently the verbose output does not include which algorithms are used
for the signature and key exchange when using OpenSSL. Including the
algorithms used will enable better debugging when working on using new
algorithm implementations. Know what algorithms are used has become more
important with the fast growing research into new quantum-safe
algorithms.
This implementation includes a build time check for the OpenSSL version
to use a new function that will be included in OpenSSL 3.2 that was
introduced in openssl/openssl@6866824
Based-on-patch-by: Martin Schmatz <mrt@zurich.ibm.com>
Closes#12030
While the struct is still public in OpenSSL, there is a (somewhat
inconvenient) accessor. Use it to remain compatible if it becomes opaque
in the future.
Closes#12038
By using unique static function/variable names in source files
implementing these interfaces.
- OpenLDAP combined with any SSH backend.
- MultiSSL with mbedTLS, OpenSSL, wolfSSL, SecureTransport.
Closes#12027
- If SSL shutdown is not finished then make an additional call to
SSL_read to gather additional tracing.
- Fix http2 and h2-proxy filters to forward do_close() calls to the next
filter.
For example h2 and SSL shutdown before and after this change:
Before:
Curl_conn_close -> cf_hc_close -> Curl_conn_cf_discard_chain ->
ssl_cf_destroy
After:
Curl_conn_close -> cf_hc_close -> cf_h2_close -> cf_setup_close ->
ssl_cf_close
Note that currently the tracing does not show output on the connection
closure handle. Refer to discussion in #11878.
Ref: https://github.com/curl/curl/discussions/11878
Closes https://github.com/curl/curl/pull/11858
