Windows compilers define `_WIN32` automatically. Windows SDK headers
or build env defines `WIN32`, or we have to take care of it. The
agreement seems to be that `_WIN32` is the preferred practice here.
Make the source code rely on that to detect we're building for Windows.
Public `curl.h` was using `WIN32`, `__WIN32__` and `CURL_WIN32` for
Windows detection, next to the official `_WIN32`. After this patch it
only uses `_WIN32` for this. Also, make it stop defining `CURL_WIN32`.
There is a slight chance these break compatibility with Windows
compilers that fail to define `_WIN32`. I'm not aware of any obsolete
or modern compiler affected, but in case there is one, one possible
solution is to define this macro manually.
grepping for `WIN32` remains useful to discover Windows-specific code.
Also:
- extend `checksrc` to ensure we're not using `WIN32` anymore.
- apply minor formatting here and there.
- delete unnecessary checks for `!MSDOS` when `_WIN32` is present.
Co-authored-by: Jay Satiro
Reviewed-by: Daniel Stenberg
Closes#12376
- refs #12356 where a UAF is reported when closing a connection
with a stream whose easy handle was cleaned up already
- handle DETACH events same as DONE events in h2/h3 filters
Fixes#12356
Reported-by: Paweł Wegner
Closes#12364
- add `struct ssl_peer` to keep hostname, dispname and sni
for a filter
- allocate `sni` for use in VTLS backend
- eliminate `Curl_ssl_snihost()` and its use of the download buffer
- use ssl_peer in SSL and QUIC filters
Closes#12349
In recent quiche, transport parameter API is separated
with quiche_conn_peer_transport_params().
(https://github.com/cloudflare/quiche/pull/1575)
It breaks with bulding with latest(post 0.18.0) quiche.
Closes#12180
GCC 14 introduces a new -Walloc-size included in -Wextra which gives:
```
src/tool_operate.c: In function ‘add_per_transfer’:
src/tool_operate.c:213:5: warning: allocation of insufficient size ‘1’ for type ‘struct per_transfer’ with size ‘480’ [-Walloc-size]
213 | p = calloc(sizeof(struct per_transfer), 1);
| ^
src/var.c: In function ‘addvariable’:
src/var.c:361:5: warning: allocation of insufficient size ‘1’ for type ‘struct var’ with size ‘32’ [-Walloc-size]
361 | p = calloc(sizeof(struct var), 1);
| ^
```
The calloc prototype is:
```
void *calloc(size_t nmemb, size_t size);
```
So, just swap the number of members and size arguments to match the
prototype, as we're initialising 1 struct of size `sizeof(struct
...)`. GCC then sees we're not doing anything wrong.
Closes#12292
Connection filter had a `get_select_socks()` method, inspired by the
various `getsocks` functions involved during the lifetime of a
transfer. These, depending on transfer state (CONNECT/DO/DONE/ etc.),
return sockets to monitor and flag if this shall be done for POLLIN
and/or POLLOUT.
Due to this design, sockets and flags could only be added, not
removed. This led to problems in filters like HTTP/2 where flow control
prohibits the sending of data until the peer increases the flow
window. The general transfer loop wants to write, adds POLLOUT, the
socket is writeable but no data can be written.
This leads to cpu busy loops. To prevent that, HTTP/2 did set the
`SEND_HOLD` flag of such a blocked transfer, so the transfer loop cedes
further attempts. This works if only one such filter is involved. If a
HTTP/2 transfer goes through a HTTP/2 proxy, two filters are
setting/clearing this flag and may step on each other's toes.
Connection filters `get_select_socks()` is replaced by
`adjust_pollset()`. They get passed a `struct easy_pollset` that keeps
up to `MAX_SOCKSPEREASYHANDLE` sockets and their `POLLIN|POLLOUT`
flags. This struct is initialized in `multi_getsock()` by calling the
various `getsocks()` implementations based on transfer state, as before.
After protocol handlers/transfer loop have set the sockets and flags
they want, the `easy_pollset` is *always* passed to the filters. Filters
"higher" in the chain are called first, starting at the first
not-yet-connection one. Each filter may add sockets and/or change
flags. When all flags are removed, the socket itself is removed from the
pollset.
Example:
* transfer wants to send, adds POLLOUT
* http/2 filter has a flow control block, removes POLLOUT and adds
POLLIN (it is waiting on a WINDOW_UPDATE from the server)
* TLS filter is connected and changes nothing
* h2-proxy filter also has a flow control block on its tunnel stream,
removes POLLOUT and adds POLLIN also.
* socket filter is connected and changes nothing
* The resulting pollset is then mixed together with all other transfers
and their pollsets, just as before.
Use of `SEND_HOLD` is no longer necessary in the filters.
All filters are adapted for the changed method. The handling in
`multi.c` has been adjusted, but its state handling the the protocol
handlers' `getsocks` method are untouched.
The most affected filters are http/2, ngtcp2, quiche and h2-proxy. TLS
filters needed to be adjusted for the connecting handshake read/write
handling.
No noticeable difference in performance was detected in local scorecard
runs.
Closes#11833
- configure a 120s idle timeout on our side of the connection
- track the timestamp when actual socket IO happens
- check IO timestamp to our *and* the peer's idle timeouts
in "is this connection alive" checks
Reported-by: calvin2021y on github
Fixes#12064Closes#12077
- refs #11342 where errors with git https interactions
were observed
- problem was caused by 1st sends of size larger than 64KB
which resulted in later retries of 64KB only
- limit sending of 1st block to 64KB
- adjust h2/h3 filters to cope with parsing the HTTP/1.1
formatted request in chunks
- introducing Curl_nwrite() as companion to Curl_write()
for the many cases where the sockindex is already known
Fixes#11342 (again)
Closes#11803
- added test cases for various code paths
- fixed handling of blocked write when stream had
been closed inbetween attempts
- re-enabled DEBUGASSERT on send with smaller data size
- in debug builds, environment variables can be set to simulate a slow
network when sending data. cf-socket.c and vquic.c support
* CURL_DBG_SOCK_WBLOCK: percentage of send() calls that should be
answered with a EAGAIN. TCP/UNIX sockets.
This is chosen randomly.
* CURL_DBG_SOCK_WPARTIAL: percentage of data that shall be written
to the network. TCP/UNIX sockets.
Example: 80 means a send with 1000 bytes would only send 800
This is applied to every send.
* CURL_DBG_QUIC_WBLOCK: percentage of send() calls that should be
answered with EAGAIN. QUIC only.
This is chosen randomly.
Closes#11756
In parallel with ngtcp2, quiche also offers the `quiche_conn_on_timeout`
interface for the application to invoke upon timer
expiration. Therefore, invoking the `on_timeout` function of the
Connection is crucial to ensure seamless functionality of quiche with
timeout events.
Closes#11654
- refs #11449 where a segfault is reported when IP Eyeballing did
not immediately connect but made several attempts
- The transfer initiating the eyeballing was initialized too early,
leadding to references to the filter instance that was then
replaced in the subsequent eyeball attempts. That led to a use
after free in the buffer handling for the transfer
- transfers are initiated now more lazy (like in the ngtcp2 filter),
when the stream is actually opened
- suppress reporting on quiche event errors for "other" transfers
than the current one to not fail a transfer due to faults in
another one.
- revert recent return value handling for quiche_h3_recv_body()
to not indicate an error but an EAGAIN situation. We wish quiche
would document what functions return.
Fixes#11449Closes#11469
Reported-by: ウさん
- refs #11449 where weirdness in quiche multi connection tranfers was
observed
- fixes lookup of transfer for a quiche event to take the connection
into account
- formerly, a transfer with the same stream_id, but on another connection
could be found
Closes#11462
- fix HTTP/2 check to not declare a connection dead when
the read attempt results in EAGAIN
- add H2-PROXY alive check as for HTTP/2 that was missing
and is needed
- add attach/detach around Curl_conn_is_alive() and remove
these in filter methods
- add checks for number of connections used in some test_10
proxy tunneling tests
Closes#11368
- Use CURL_FORMAT_CURL_OFF_T where %zd was erroneously used for some
curl_off_t variables.
- Use %zu where %zd was erroneously used for some size_t variables.
Prior to this change some of the Windows CI tests were failing because
in Windows 32-bit targets have a 32-bit size_t and a 64-bit curl_off_t.
When %zd was used for some curl_off_t variables then only the lower
32-bits was read and the upper 32-bits would be read for part or all of
the next specifier.
Fixes https://github.com/curl/curl/issues/11327
Closes https://github.com/curl/curl/pull/11321
- add an `id` long to Curl_easy, -1 on init
- once added to a multi (or its own multi), it gets
a non-negative number assigned by the connection cache
- `id` is unique among all transfers using the same
cache until reaching LONG_MAX where it will wrap
around. So, not unique eternally.
- CURLINFO_CONN_ID returns the connection id attached to
data or, if none present, data->state.lastconnect_id
- variables and type declared in tool for write out
Closes#11185
- ngtcp2 and quiche implementations relied on the DONE_SEND event
to forward the EOF for uploads to the libraries. This often
result in a last 0 length EOF data. Tracking the amount of
data left to upload allows EOF indication earlier.
- refs #11205 where CloudFlare DoH servers did not like to
receive the initial upload DATA without EOF and returned
a 400 Bad Request
Reported-by: Sergey Fionov
Fixes#11205Closes#11207
This works around #11138, by doubling the limit, and should be a
relatively safe fix.
Ideally the buffer would grow as needed and there would be no need for a
limit? But that might be follow-up material.
Fixes#11138Closes#11139
- `drain` was used by http/2 and http/3 implementations to indicate
that the transfer requires send/recv independant from its socket
poll state. Intended as a counter, it was used as bool flag only.
- a similar mechanism exists on `connectdata->cselect_bits` where
specific protocols can indicate something similar, only for the
whole connection.
- `cselect_bits` are cleard in transfer.c on use and, importantly,
also set when the transfer loop expended its `maxloops` tries.
`drain` was not cleared by transfer and the http2/3 implementations
had to take care of that.
- `dselect_bits` is cleared *and* set by the transfer loop. http2/3
does no longer clear it, only set when new events happen.
This change unifies the handling of socket poll overrides, extending
`cselect_bits` by a easy handle specific value and a common treatment in
transfers.
Closes#11005
- with `--proxy-http2` allow h2 ALPN negotiation to
forward proxies
- applies to http: requests against a https: proxy only,
as https: requests will auto-tunnel
- adding a HTTP/1 request parser in http1.c
- removed h2h3.c
- using new request parser in nghttp2 and all h3 backends
- adding test 2603 for request parser
- adding h2 proxy test cases to test_10_*
scorecard.py: request scoring accidentally always run curl
with '-v'. Removed that, expect double numbers.
labeller: added http1.* and h2-proxy sources to detection
Closes#10967
- callbacks and filter methods might be invoked at unexpected
times, e.g. when the transfer's stream_ctx has not been initialized
yet or, more likely, has already been taken down.
- check for existance of stream_ctx in such places and return
an error or silently succeed the call.
Closes#10951
- remove NGHTTP2 members of `struct HTTP`
- add `void *h2_ctx` to `struct HTTP`
- add `void *h3_ctx` to `struct HTTP`
- separate h2/h3 pointers are needed for eyeballing
- manage local stream_ctx in http implementations
Closes#10877
- currently only on debug build and when env variable
CURL_PROXY_TUNNEL_H2 is present.
- will ALPN negotiate with the proxy server and switch
tunnel filter based on the protocol negotiated.
- http/1.1 tunnel code moved into cf-h1-proxy.[ch]
- http/2 tunnel code implemented in cf-h2-proxy.[ch]
- tunnel start and ALPN set remains in http_proxy.c
- moving all haproxy related code into cf-haproxy.[ch]
VTLS changes
- SSL filters rely solely on the "alpn" specification they
are created with and no longer check conn->bits.tls_enable_alpn.
- checks on which ALPN specification to use (or none at all) are
done in vtls.c when creating the filter.
Testing
- added a nghttpx forward proxy to the pytest setup that
speaks HTTP/2 and forwards all requests to the Apache httpd
forward proxy server.
- extending test coverage in test_10 cases
- adding proxy tests for direct/tunnel h1/h2 use of basic auth.
- adding test for http/1.1 and h2 proxy tunneling to pytest
Closes#10780
- ngtcp2: using bufq for recv stream data
- internal stream_ctx instead of `struct HTTP` members
for quiche, ngtcp2 and msh3
- no more QUIC related members in `struct HTTP`
- experimental use of recvmmsg(), disabled by default
- testing on my old debian box shows no throughput improvements.
- leaving it in, but disabled, for future revisit
- vquic: common UDP receive code for ngtcp2 and quiche
- vquic: common UDP send code for ngtcp2 and quiche
- added pytest skips for known msh3 failures
- fix unit2601 to survive torture testing
- quiche: using latest `master` from quiche and enabling large download
tests, now that key change is supported
- fixing test_07_21 where retry handling of starting a stream
was faulty
- msh3: use bufq for recv buffering headers and data
- msh3: replace fprintf debug logging with LOG_CF where possible
- msh3: force QUIC expire timers on recv/send to have more than
1 request per second served
Closes#10772
- use the defined, but so far not used, KEEP_SEND_HOLD bit for flow
control based suspend of sending in transfers.
Prior to this change KEEP_SEND_PAUSE bit was used instead, but that can
interfere with pausing streams from the user side via curl_easy_pause.
Fixes https://github.com/curl/curl/issues/10751
Closes https://github.com/curl/curl/pull/10753
- add parameter to `conn_is_alive()` cfilter method that returns
if there is input data waiting on the connection
- refrain from re-using connnection from the cache that have
input pending
- adapt http/2 and http/3 alive checks to digest pending input
to check the connection state
- remove check_cxn method from openssl as that was just doing
what the socket filter now does.
- add tests for connection reuse with special server configs
Closes#10690
- time_connect was not updated when the overall connection failed,
e.g. when SSL verification was unsuccessful, refs #10670
- rework gather those values to interrogate involved filters,
also from all eyeballing attempts, to report the maximum of
those values.
- added 3 test cases in test_06 to check reported values on
successful, partially failed and totally failed connections.
Reported-by: Master Inspire
Fixes#10670Closes#10671
vquic stabilization
- udp send code shared between ngtcp2 and quiche
- quiche handling of data and events improved
ngtcp2 and pytest improvements
- fixes handling of "drain" situations, discovered in scorecard
tests with the Caddy server.
- improvements in handling transfers that have already data or
are already closed to make an early return on recv
pytest
- adding caddy tests when available
scorecard improvemnts.
- using correct caddy port
- allowing tests for only httpd or caddy
Closes#10451
New cfilter HTTP-CONNECT for h3/h2/http1.1 eyeballing.
- filter is installed when `--http3` in the tool is used (or
the equivalent CURLOPT_ done in the library)
- starts a QUIC/HTTP/3 connect right away. Should that not
succeed after 100ms (subject to change), a parallel attempt
is started for HTTP/2 and HTTP/1.1 via TCP
- both attempts are subject to IPv6/IPv4 eyeballing, same
as happens for other connections
- tie timeout to the ip-version HAPPY_EYEBALLS_TIMEOUT
- use a `soft` timeout at half the value. When the soft timeout
expires, the HTTPS-CONNECT filter checks if the QUIC filter
has received any data from the server. If not, it will start
the HTTP/2 attempt.
HTTP/3(ngtcp2) improvements.
- setting call_data in all cfilter calls similar to http/2 and vtls filters
for use in callback where no stream data is available.
- returning CURLE_PARTIAL_FILE for prematurely terminated transfers
- enabling pytest test_05 for h3
- shifting functionality to "connect" UDP sockets from ngtcp2
implementation into the udp socket cfilter. Because unconnected
UDP sockets are weird. For example they error when adding to a
pollset.
HTTP/3(quiche) improvements.
- fixed upload bug in quiche implementation, now passes 251 and pytest
- error codes on stream RESET
- improved debug logs
- handling of DRAIN during connect
- limiting pending event queue
HTTP/2 cfilter improvements.
- use LOG_CF macros for dynamic logging in debug build
- fix CURLcode on RST streams to be CURLE_PARTIAL_FILE
- enable pytest test_05 for h2
- fix upload pytests and improve parallel transfer performance.
GOAWAY handling for ngtcp2/quiche
- during connect, when the remote server refuses to accept new connections
and closes immediately (so the local conn goes into DRAIN phase), the
connection is torn down and a another attempt is made after a short grace
period.
This is the behaviour observed with nghttpx when we tell it to shut
down gracefully. Tested in pytest test_03_02.
TLS improvements
- ALPN selection for SSL/SSL-PROXY filters in one vtls set of functions, replaces
copy of logic in all tls backends.
- standardized the infof logging of offered ALPNs
- ALPN negotiated: have common function for all backends that sets alpn proprty
and connection related things based on the negotiated protocol (or lack thereof).
- new tests/tests-httpd/scorecard.py for testing h3/h2 protocol implementation.
Invoke:
python3 tests/tests-httpd/scorecard.py --help
for usage.
Improvements on gathering connect statistics and socket access.
- new CF_CTRL_CONN_REPORT_STATS cfilter control for having cfilters
report connection statistics. This is triggered when the connection
has completely connected.
- new void Curl_pgrsTimeWas(..) method to report a timer update with
a timestamp of when it happend. This allows for updating timers
"later", e.g. a connect statistic after full connectivity has been
reached.
- in case of HTTP eyeballing, the previous changes will update
statistics only from the filter chain that "won" the eyeballing.
- new cfilter query CF_QUERY_SOCKET for retrieving the socket used
by a filter chain.
Added methods Curl_conn_cf_get_socket() and Curl_conn_get_socket()
for convenient use of this query.
- Change VTLS backend to query their sub-filters for the socket when
checks during the handshake are made.
HTTP/3 documentation on how https eyeballing works.
TLS improvements
- ALPN selection for SSL/SSL-PROXY filters in one vtls set of functions, replaces
copy of logic in all tls backends.
- standardized the infof logging of offered ALPNs
- ALPN negotiated: have common function for all backends that sets alpn proprty
and connection related things based on the negotiated protocol (or lack thereof).
Scorecard with Caddy.
- configure can be run with `--with-test-caddy=path` to specify which caddy to use for testing
- tests/tests-httpd/scorecard.py now measures download speeds with caddy
pytest improvements
- adding Makfile to clean gen dir
- adding nghttpx rundir creation on start
- checking httpd version 2.4.55 for test_05 cases where it is needed. Skipping with message if too old.
- catch exception when checking for caddy existance on system.
Closes#10349
- new functions and macros for cfilter debugging
- set CURL_DEBUG with names of cfilters where debug logging should be
enabled
- use GNUC __attribute__ to enable printf format checks during compile
Closes#10271
- ECONNECTREFUSED has not its own fail message in quic filters
- Debug logging in connect eyballing improved
- Fix bug in ngtcp2/quiche that could lead to false success reporting.
Reported-by: Divy Le Ray
Fixes#10245Closes#10248
- quiche in debug mode did not build, fixed.
- moved all vquic implementation files to prefix curl_* to avoid
the potential mixups between provided .h files and our own.
- quich passes test 2500 and 2502. 2501, the POST, fail with
the body being rejected. Quich bug?
Closes#10242
