Enable TLS Early Data for wolfSSL:
- merge WOLFSSL_CTX and WOLFSSL setup from ngtcp2 with the general
implemenation in wolfssl.c
- enable for QUIC via ngtcp2
- give Curl_vquic_tls_init() a `struct alpn_spec` like used for the TCP
case. Adapt gnutls and other users.
- enable pytest test cases for early data with wolfSSL
and while this messes up wolfssl.c anyway, do
- rename all struct/functions with prefix 'wolfssl_' to 'wssl_' to not
pollute that name prefix
- rename `ctx/handle` to `ssl_ctx/ssl`, as used in openssl case
Closes#16167
Remove `blocking` argument from cfilter's connect method.
Implement blocking behaviour in Curl_conn_connect() instead for all
filter chains.
Update filters implementations. Several of which did never use the
paramter (QUIC for example). Simplifies connect handling in TLS filters
that no longer need to loop
Fixed a blocking connect call in FTP when waiting on a socket accept()
which only worked because the filter did not implement it.
Closes#16397
The variable `conn->httpversion` was used for several purposes and it
was unclear at which time the value represents what.
- rename `conn->httpversion` to `conn->httpversion_seen`
This makes clear that the variable only records the last
HTTP version seen on the connection - if any. And that it
no longer is an indication of what version to use.
- Change Alt-Svc handling to no longer modify `conn->httpversion`
but set `data->state.httpwant` for influencing the HTTP version
to use on a transfer.
- Add `data->req.httpversion_sent` to have a record of what
HTTP version was sent in a request
- Add connection filter type CF_TYPE_HTTP
- Add filter query `CF_QUERY_HTTP_VERSION` to ask what HTTP
filter version is in place
- Lookup filters HTTP version instead of using `conn->httpversion`
Test test_12_05 now switches to HTTP/1.1 correctly and the
expectations have been fixed.
Removed the connection fitler "is_httpN()" checks and using
the version query instead.
Closes#16073
Make transfer attach/detach to/from connections chepaer.
- the "attach" event was no longer implemented by any filter
- the "detach" did the same as the "done" event for the filters
who still implemented it. It should be superfluous as the "done"
must always happen.
Closes#16067
When a QUIC TLS session announced early data support and
'CURLSSLOPT_EARLYDATA' is set for the transfer, send initial request and
body (up to the 128k we buffer) as 0RTT when curl is built with
ngtcp2+gnutls.
QUIC 0RTT needs not only the TLS session but the QUIC transport
paramters as well. Store those and the earlydata max value together with
the session in the cache.
Add test case for h3 use of this. Enable quic early data in nghttpx for
testing.
Closes#15667
Described in detail in internal doc TLS-SESSIONS.md
Main points:
- use a new `ssl_peer_key` for cache lookups by connection filters
- recognize differences between TLSv1.3 and other tickets
* TLSv1.3 tickets are single-use, cache can hold several of them for a peer
* TLSv1.2 are reused, keep only a single one per peer
- differentiate between ticket BLOB to store (that could be persisted) and object instances
- use put/take/return pattern for cache access
- remember TLS version, ALPN protocol, time received and lifetime of ticket
- auto-expire tickets after their lifetime
Closes#15774
booleans should use the type 'bool' and set the value to TRUE/FALSE
non-booleans should not be 'bool' and should not set the value to
TRUE/FALSE
Closes#15123
- fix MSH3 static symbol clash.
- fix Quiche static symbol clash.
- fix local macro clash with BearSSL header.
- fix local macro clash with OmniOS system header.
```
In file included from ../../lib/urldata.h:197,
from ../../lib/altsvc.c:32,
from libcurlall.c:2:
../../lib/cf-socket.h:55:25: error: expected ':', ',', ';', '}' or '__attribute__' before '.' token
55 | #define sa_addr _sa_ex_u.addr
| ^
In file included from ../../lib/urldata.h:197,
from ../../lib/altsvc.c:32,
from libcurlall.c:2:
../../lib/cf-socket.h:55:25: error: expected ':', ',', ';', '}' or '__attribute__' before '.' token
55 | #define sa_addr _sa_ex_u.addr
| ^
```
Ref: https://github.com/curl/curl/actions/runs/10738314933/job/29781644299?pr=14772#step:3:6115
Discovered while adding support for "unity" builds for autotools.
Required-by: #14922
Cherry-picked from #14815Closes#14932
Do not give up connect on servers that are in draining state. This might
indicate the QUIC server restarting and the UDP packet routing still
hitting the instance shutting down.
Instead keep on connecting until the overall TIMEOUT fires.
Closes#14863
This is a better match for what they do and the general "cpool"
var/function prefix works well.
The pool now handles very long hostnames correctly.
The following changes have been made:
* 'struct connectdata', e.g. connections, keep new members
named `destination` and ' destination_len' that fully specifies
interface+port+hostname of where the connection is going to.
This is used in the pool for "bundling" of connections with
the same destination. There is no limit on the length any more.
* Locking: all locks are done inside conncache.c when calling
into the pool and released on return. This eliminates hazards
of the callers keeping track.
* 'struct connectbundle' is now internal to the pool. It is no
longer referenced by a connection.
* 'bundle->multiuse' no longer exists. HTTP/2 and 3 and TLS filters
no longer need to set it. Instead, the multi checks on leaving
MSTATE_CONNECT or MSTATE_CONNECTING if the connection is now
multiplexed and new, e.g. not conn->bits.reuse. In that case
the processing of pending handles is triggered.
* The pool's init is provided with a callback to invoke on all
connections being discarded. This allows the cleanups in
`Curl_disconnect` to run, wherever it is decided to retire
a connection.
* Several pool operations can now be fully done with one call.
Pruning dead connections, upkeep and checks on pool limits
can now directly discard connections and need no longer return
those to the caller for doing that (as we have now the callback
described above).
* Finding a connection for reuse is now done via `Curl_cpool_find()`
and the caller provides callbacks to evaluate the connection
candidates.
* The 'Curl_cpool_check_limits()' now directly uses the max values
that may be set in the transfer's multi. No need to pass them
around. Curl_multi_max_host_connections() and
Curl_multi_max_total_connections() are gone.
* Add method 'Curl_node_llist()' to get the llist a node is in.
Used in cpool to verify connection are indeed in the list (or
not in any list) as they need to.
I left the conncache.[ch] as is for now and also did not touch the
documentation. If we update that outside the feature window, we can
do this in a separate PR.
Multi-thread safety is not achieved by this PR, but since more details
on how pools operate are now "internal" it is a better starting
point to go for this in the future.
Closes#14662
`data->id` is unique in *most* situations, but not in all. If a libcurl
application uses more than one connection cache, they will overlap. This
is a rare situations, but libcurl apps do crazy things. However, for
informative things, like tracing, `data->id` is superior, since it
assigns new ids in curl's serial curl_easy_perform() use.
Introduce `data->mid` which is a unique identifer inside one multi
instance, assigned on multi_add_handle() and cleared on
multi_remove_handle().
Use the `mid` in DoH operations and also in h2/h3 stream hashes.
Reported-by: 罗朝辉
Fixes#14414Closes#14499
Members of the filter context, like stream hash and buffers, need to be
initialized early and protected by a flag to also avoid double cleanup.
This allow the context to be used safely before a connect() is started
and the other parts of the context are set up.
Closes#14505
- Turned them all into functions to also do asserts etc.
- The llist related structs got all their fields renamed in order to make
sure no existing code remains using direct access.
- Each list node struct now points back to the list it "lives in", so
Curl_node_remove() no longer needs the list pointer.
- Rename the node struct and some of the access functions.
- Added lots of ASSERTs to verify API being used correctly
- Fix some cases of API misuse
Add docs/LLIST.md documenting the internal linked list API.
Closes#14485
Instead of having an especially "unique" linked list handler for the
main list of easy handles within the multi handle, this now uses a
regular Curl_llist for this as well.
With this change, it is also clearer that every easy handle added to a
multi handle belongs to one and only one out of three different lists:
process - the general one for normal transfer processing
pending - queued up waiting to get a connection (MSTATE_PENDING)
msgsent - transfer completed (MSTATE_MSGSENT)
An easy handle must therefore be removed from the current list before it
gets added to another.
Closes#14474
Adds a `bool eos` flag to send methods to indicate that the data
is the last chunk the invovled transfer wants to send to the server.
This will help protocol filters like HTTP/2 and 3 to forward the
stream's EOF flag and also allow to EAGAIN such calls when buffers
are not yet fully flushed.
Closes#14220
Adds a `bool eos` flag to send methods to indicate that the data is the
last chunk the invovled transfer wants to send to the server.
This will help protocol filters like HTTP/2 and 3 to forward the
stream's EOF flag and also allow to EAGAIN such calls when buffers are
not yet fully flushed.
Closes#14220
Based on the standards and guidelines we use for our documentation.
- expand contractions (they're => they are etc)
- host name = > hostname
- file name => filename
- user name = username
- man page => manpage
- run-time => runtime
- set-up => setup
- back-end => backend
- a HTTP => an HTTP
- Two spaces after a period => one space after period
Closes#14073
This adds connection shutdown infrastructure and first use for FTP. FTP
data connections, when not encountering an error, are now shut down in a
blocking way with a 2sec timeout.
- add cfilter `Curl_cft_shutdown` callback
- keep a shutdown start timestamp and timeout at connectdata
- provide shutdown timeout default and member in
`data->set.shutdowntimeout`.
- provide methods for starting, interrogating and clearing
shutdown timers
- provide `Curl_conn_shutdown_blocking()` to shutdown the
`sockindex` filter chain in a blocking way. Use that in FTP.
- add `Curl_conn_cf_poll()` to wait for socket events during
shutdown of a connection filter chain.
This gets the monitoring sockets and events via the filters
"adjust_pollset()" methods. This gives correct behaviour when
shutting down a TLS connection through a HTTP/2 proxy.
- Implement shutdown for all socket filters
- for HTTP/2 and h2 proxying to send GOAWAY
- for TLS backends to the best of their capabilities
- for tcp socket filter to make a final, nonblocking
receive to avoid unwanted RST states
- add shutdown forwarding to happy eyeballers and
https connect ballers when applicable.
Closes#13904
- set the idle timeout transport parameter
in milliseconds as documented by quiche
- do not calculate the idle timeout, rely on
quiche handling it
Closes#13581
- quiche: error transfers that try to receive on a closed
or draining connection
- ngtcp2: use callback for extending max bidi streams. This
allows more precise calculation of MAX_CONCURRENT as we
only can start a new stream when the server acknowledges
the close - not when we locally have closed it.
- remove a fprintf() from h2-download client to avoid excess
log files on tests timing out.
Closes#13475
- when a connection close is detected, all ongoing transfers
need to expire bc no more POLL events are likely to happen
for them.
Fixes#13439
Reported-by: Jay Satiro
Closes#13447
- add `Curl_hash_offt` as hashmap between a `curl_off_t` and
an object. Use this in h2+h3 connection filters to associate
`data->id` with the internal stream state.
- changed implementations of all affected connection filters
- removed `h2_ctx*` and `h3_ctx*` from `struct HTTP` and thus
the easy handle
- solves the problem of attaching "foreign protocol" easy handles
during connection shutdown
Test 1616 verifies the new hash functions.
Closes#13204
- add curl_int64_t signed 64-bit type for lib use
- define CURL_PRId64, CURL_PRIu64 format ids
- use curl_int64_t in vquic
curl_int64_t signed complements the existing curl_uint64_t unsigned.
Note that `curl_int64_t` and `int64_t` are assignable from each other
but not identical. Some platforms with 64 long type defint int64_t as
"long long" (staring at macOS) which messes up things like pointers and
format identifiers.
Closes https://github.com/curl/curl/pull/13293
new struct ip_quadruple for holding local/remote addr+port
- used in data->info and conn and cf-socket.c
- copy back and forth complete struct
- add 'secondary' to conn
- use secondary in reporting success for ftp 2nd connection
Reported-by: DasKutti on github
Fixes#13084Closes#13090
- let `multi_getsock()` initialize the pollset in what the
transfer state requires in regards to SEND/RECV
- change connection filters `adjust_pollset()` implementation
to react on the presence of POLLIN/-OUT in the pollset and
no longer check CURL_WANT_SEND/CURL_WANT_RECV
- cf-socket will no longer add POLLIN on its own
- http2 and http/3 filters will only do adjustments if the
passed pollset wants to POLLIN/OUT for the transfer on
the socket. This is similar to the HTTP/2 proxy filter
and works in stacked filters.
Closes#12640
do not add a socket for POLLIN when the transfer does not want to send
(for example is paused).
Follow-up to 47f5b1a
Reported-by: bubbleguuum on github
Fixes#12632Closes#12633
Prior to this change if a send failed on a stream in an invalid state
(according to quiche) and not marked as closed (according to libcurl)
then the send function would return CURLE_SEND_ERROR.
We already have similar code for ngtcp2 to return CURLE_HTTP3 in this
case.
Caught by test test_07_upload.py: test_07_22_upload_parallel_fail.
Fixes https://github.com/curl/curl/issues/12590
Closes https://github.com/curl/curl/pull/12597
- use `data->state.dselect_bits` everywhere instead
- remove `bool *comeback` parameter as non-zero
`data->state.dselect_bits` will indicate that IO is
incomplete.
Closes#12512
- when a connect immediately goes into DRAINING state, do
not attempt retries in the QUIC connection filter. Instead,
return CURLE_WEIRD_SERVER_REPLY
- When eyeballing, interpret CURLE_WEIRD_SERVER_REPLY as an
inconclusive answer. When all addresses have been attempted,
rewind the address list once on an inconclusive answer.
- refs #11832 where connects were retried indefinitely until
the overall timeout fired
Closes#12400
Windows compilers define `_WIN32` automatically. Windows SDK headers
or build env defines `WIN32`, or we have to take care of it. The
agreement seems to be that `_WIN32` is the preferred practice here.
Make the source code rely on that to detect we're building for Windows.
Public `curl.h` was using `WIN32`, `__WIN32__` and `CURL_WIN32` for
Windows detection, next to the official `_WIN32`. After this patch it
only uses `_WIN32` for this. Also, make it stop defining `CURL_WIN32`.
There is a slight chance these break compatibility with Windows
compilers that fail to define `_WIN32`. I'm not aware of any obsolete
or modern compiler affected, but in case there is one, one possible
solution is to define this macro manually.
grepping for `WIN32` remains useful to discover Windows-specific code.
Also:
- extend `checksrc` to ensure we're not using `WIN32` anymore.
- apply minor formatting here and there.
- delete unnecessary checks for `!MSDOS` when `_WIN32` is present.
Co-authored-by: Jay Satiro
Reviewed-by: Daniel Stenberg
Closes#12376
- refs #12356 where a UAF is reported when closing a connection
with a stream whose easy handle was cleaned up already
- handle DETACH events same as DONE events in h2/h3 filters
Fixes#12356
Reported-by: Paweł Wegner
Closes#12364
- add `struct ssl_peer` to keep hostname, dispname and sni
for a filter
- allocate `sni` for use in VTLS backend
- eliminate `Curl_ssl_snihost()` and its use of the download buffer
- use ssl_peer in SSL and QUIC filters
Closes#12349
In recent quiche, transport parameter API is separated
with quiche_conn_peer_transport_params().
(https://github.com/cloudflare/quiche/pull/1575)
It breaks with bulding with latest(post 0.18.0) quiche.
Closes#12180
GCC 14 introduces a new -Walloc-size included in -Wextra which gives:
```
src/tool_operate.c: In function ‘add_per_transfer’:
src/tool_operate.c:213:5: warning: allocation of insufficient size ‘1’ for type ‘struct per_transfer’ with size ‘480’ [-Walloc-size]
213 | p = calloc(sizeof(struct per_transfer), 1);
| ^
src/var.c: In function ‘addvariable’:
src/var.c:361:5: warning: allocation of insufficient size ‘1’ for type ‘struct var’ with size ‘32’ [-Walloc-size]
361 | p = calloc(sizeof(struct var), 1);
| ^
```
The calloc prototype is:
```
void *calloc(size_t nmemb, size_t size);
```
So, just swap the number of members and size arguments to match the
prototype, as we're initialising 1 struct of size `sizeof(struct
...)`. GCC then sees we're not doing anything wrong.
Closes#12292
Connection filter had a `get_select_socks()` method, inspired by the
various `getsocks` functions involved during the lifetime of a
transfer. These, depending on transfer state (CONNECT/DO/DONE/ etc.),
return sockets to monitor and flag if this shall be done for POLLIN
and/or POLLOUT.
Due to this design, sockets and flags could only be added, not
removed. This led to problems in filters like HTTP/2 where flow control
prohibits the sending of data until the peer increases the flow
window. The general transfer loop wants to write, adds POLLOUT, the
socket is writeable but no data can be written.
This leads to cpu busy loops. To prevent that, HTTP/2 did set the
`SEND_HOLD` flag of such a blocked transfer, so the transfer loop cedes
further attempts. This works if only one such filter is involved. If a
HTTP/2 transfer goes through a HTTP/2 proxy, two filters are
setting/clearing this flag and may step on each other's toes.
Connection filters `get_select_socks()` is replaced by
`adjust_pollset()`. They get passed a `struct easy_pollset` that keeps
up to `MAX_SOCKSPEREASYHANDLE` sockets and their `POLLIN|POLLOUT`
flags. This struct is initialized in `multi_getsock()` by calling the
various `getsocks()` implementations based on transfer state, as before.
After protocol handlers/transfer loop have set the sockets and flags
they want, the `easy_pollset` is *always* passed to the filters. Filters
"higher" in the chain are called first, starting at the first
not-yet-connection one. Each filter may add sockets and/or change
flags. When all flags are removed, the socket itself is removed from the
pollset.
Example:
* transfer wants to send, adds POLLOUT
* http/2 filter has a flow control block, removes POLLOUT and adds
POLLIN (it is waiting on a WINDOW_UPDATE from the server)
* TLS filter is connected and changes nothing
* h2-proxy filter also has a flow control block on its tunnel stream,
removes POLLOUT and adds POLLIN also.
* socket filter is connected and changes nothing
* The resulting pollset is then mixed together with all other transfers
and their pollsets, just as before.
Use of `SEND_HOLD` is no longer necessary in the filters.
All filters are adapted for the changed method. The handling in
`multi.c` has been adjusted, but its state handling the the protocol
handlers' `getsocks` method are untouched.
The most affected filters are http/2, ngtcp2, quiche and h2-proxy. TLS
filters needed to be adjusted for the connecting handshake read/write
handling.
No noticeable difference in performance was detected in local scorecard
runs.
Closes#11833
- configure a 120s idle timeout on our side of the connection
- track the timestamp when actual socket IO happens
- check IO timestamp to our *and* the peer's idle timeouts
in "is this connection alive" checks
Reported-by: calvin2021y on github
Fixes#12064Closes#12077
