- connect timeout was used at half the configured value, if the
destination had 1 ip version 4 and other version 6 addresses
(or the other way around)
- extended test2600 to reproduce these cases
Reported-by: Michael Kaufmann
Fixes#10514Closes#10517
New cfilter HTTP-CONNECT for h3/h2/http1.1 eyeballing.
- filter is installed when `--http3` in the tool is used (or
the equivalent CURLOPT_ done in the library)
- starts a QUIC/HTTP/3 connect right away. Should that not
succeed after 100ms (subject to change), a parallel attempt
is started for HTTP/2 and HTTP/1.1 via TCP
- both attempts are subject to IPv6/IPv4 eyeballing, same
as happens for other connections
- tie timeout to the ip-version HAPPY_EYEBALLS_TIMEOUT
- use a `soft` timeout at half the value. When the soft timeout
expires, the HTTPS-CONNECT filter checks if the QUIC filter
has received any data from the server. If not, it will start
the HTTP/2 attempt.
HTTP/3(ngtcp2) improvements.
- setting call_data in all cfilter calls similar to http/2 and vtls filters
for use in callback where no stream data is available.
- returning CURLE_PARTIAL_FILE for prematurely terminated transfers
- enabling pytest test_05 for h3
- shifting functionality to "connect" UDP sockets from ngtcp2
implementation into the udp socket cfilter. Because unconnected
UDP sockets are weird. For example they error when adding to a
pollset.
HTTP/3(quiche) improvements.
- fixed upload bug in quiche implementation, now passes 251 and pytest
- error codes on stream RESET
- improved debug logs
- handling of DRAIN during connect
- limiting pending event queue
HTTP/2 cfilter improvements.
- use LOG_CF macros for dynamic logging in debug build
- fix CURLcode on RST streams to be CURLE_PARTIAL_FILE
- enable pytest test_05 for h2
- fix upload pytests and improve parallel transfer performance.
GOAWAY handling for ngtcp2/quiche
- during connect, when the remote server refuses to accept new connections
and closes immediately (so the local conn goes into DRAIN phase), the
connection is torn down and a another attempt is made after a short grace
period.
This is the behaviour observed with nghttpx when we tell it to shut
down gracefully. Tested in pytest test_03_02.
TLS improvements
- ALPN selection for SSL/SSL-PROXY filters in one vtls set of functions, replaces
copy of logic in all tls backends.
- standardized the infof logging of offered ALPNs
- ALPN negotiated: have common function for all backends that sets alpn proprty
and connection related things based on the negotiated protocol (or lack thereof).
- new tests/tests-httpd/scorecard.py for testing h3/h2 protocol implementation.
Invoke:
python3 tests/tests-httpd/scorecard.py --help
for usage.
Improvements on gathering connect statistics and socket access.
- new CF_CTRL_CONN_REPORT_STATS cfilter control for having cfilters
report connection statistics. This is triggered when the connection
has completely connected.
- new void Curl_pgrsTimeWas(..) method to report a timer update with
a timestamp of when it happend. This allows for updating timers
"later", e.g. a connect statistic after full connectivity has been
reached.
- in case of HTTP eyeballing, the previous changes will update
statistics only from the filter chain that "won" the eyeballing.
- new cfilter query CF_QUERY_SOCKET for retrieving the socket used
by a filter chain.
Added methods Curl_conn_cf_get_socket() and Curl_conn_get_socket()
for convenient use of this query.
- Change VTLS backend to query their sub-filters for the socket when
checks during the handshake are made.
HTTP/3 documentation on how https eyeballing works.
TLS improvements
- ALPN selection for SSL/SSL-PROXY filters in one vtls set of functions, replaces
copy of logic in all tls backends.
- standardized the infof logging of offered ALPNs
- ALPN negotiated: have common function for all backends that sets alpn proprty
and connection related things based on the negotiated protocol (or lack thereof).
Scorecard with Caddy.
- configure can be run with `--with-test-caddy=path` to specify which caddy to use for testing
- tests/tests-httpd/scorecard.py now measures download speeds with caddy
pytest improvements
- adding Makfile to clean gen dir
- adding nghttpx rundir creation on start
- checking httpd version 2.4.55 for test_05 cases where it is needed. Skipping with message if too old.
- catch exception when checking for caddy existance on system.
Closes#10349
- add test2600 as a unit test that triggers various connect conditions
and monitors behaviour, available in a debug build only.
- this exposed edge cases in connect.c that have been fixed
Closes#10312
- new functions and macros for cfilter debugging
- set CURL_DEBUG with names of cfilters where debug logging should be
enabled
- use GNUC __attribute__ to enable printf format checks during compile
Closes#10271
- ECONNECTREFUSED has not its own fail message in quic filters
- Debug logging in connect eyballing improved
- Fix bug in ngtcp2/quiche that could lead to false success reporting.
Reported-by: Divy Le Ray
Fixes#10245Closes#10248
- they are mostly pointless in all major jurisdictions
- many big corporations and projects already don't use them
- saves us from pointless churn
- git keeps history for us
- the year range is kept in COPYING
checksrc is updated to allow non-year using copyright statements
Closes#10205
Refactoring of connection setup and happy eyeballing. Move
nghttp2. ngtcp2, quiche and msh3 into connection filters.
- eyeballing cfilter that uses sub-filters for performing parallel connects
- socket cfilter for all transport types, including QUIC
- QUIC implementations in cfilter, can now participate in eyeballing
- connection setup is more dynamic in order to adapt to what filter did
really connect. Relevant to see if a SSL filter needs to be added or
if SSL has already been provided
- HTTP/3 test cases similar to HTTP/2
- multiuse of parallel transfers for HTTP/3, tested for ngtcp2 and quiche
- Fix for data attach/detach in VTLS filters that could lead to crashes
during parallel transfers.
- Eliminating setup() methods in cfilters, no longer needed.
- Improving Curl_conn_is_alive() to replace Curl_connalive() and
integrated ssl alive checks into cfilter.
- Adding CF_CNTRL_CONN_INFO_UPDATE to tell filters to update
connection into and persist it at the easy handle.
- Several more cfilter related cleanups and moves:
- stream_weigth and dependency info is now wrapped in struct
Curl_data_priority
- Curl_data_priority members depend is available in HTTP2|HTTP3
- Curl_data_priority members depend on NGHTTP2 support
- handling init/reset/cleanup of priority part of url.c
- data->state.priority same struct, but shallow copy for compares only
- PROTOPT_STREAM has been removed
- Curl_conn_is_mulitplex() now available to check on capability
- Adding query method to connection filters.
- ngtcp2+quiche: implementing query for max concurrent transfers.
- Adding is_alive and keep_alive cfilter methods. Adding DATA_SETUP event.
- setting keepalive timestamp on connect
- DATA_SETUP is called after the connection has been completely
setup (but may not connected yet) to allow filters to initialize
data members they use.
- there is no socket to be had with msh3, it is unclear how select
shall work
- manual test via "curl --http3 https://curl.se" fail with "empty
reply from server".
- Various socket/conn related cleanups:
- Curl_socket is now Curl_socket_open and in cf-socket.c
- Curl_closesocket is now Curl_socket_close and in cf-socket.c
- Curl_ssl_use has been replaced with Cur_conn_is_ssl
- Curl_conn_tcp_accepted_set has been split into
Curl_conn_tcp_listen_set and Curl_conn_tcp_accepted_set
with a clearer purpose
Closes#10141
- fixes `Curl_ssl_cf_get_ssl()` to detect also the first filter instance
as ssl (refs #10053)
- replaces `Curl_ssl_use()` with the correct `Curl_conn_is_ssl()`
Closes#10054Fixes#10053
Reported-by: Patrick Monnerat
- `Curl_ssl_get_config()` now returns the first config if no SSL proxy
filter is active
- socket filter starts connection only on first invocation of its
connect method
Fixes#9982Closes#9983
Commit 3b16575ae9 removed support for
building on Novell Netware, but a few leftover traces remained. This
removes the last bits.
Closes: #9966
Reviewed-by: Daniel Stenberg <daniel@haxx.se>
- almost all backend calls pass the Curl_cfilter intance instead of
connectdata+sockindex
- ssl_connect_data is remove from struct connectdata and made internal
to vtls
- ssl_connect_data is allocated in the added filter, kept at cf->ctx
- added function to let a ssl filter access its ssl_primary_config and
ssl_config_data this selects the propert subfields in conn and data,
for filters added as plain or proxy
- adjusted all backends to use the changed api
- adjusted all backends to access config data via the exposed
functions, no longer using conn or data directly
cfilter renames for clear purpose:
- methods `Curl_conn_*(data, conn, sockindex)` work on the complete
filter chain at `sockindex` and connection `conn`.
- methods `Curl_cf_*(cf, ...)` work on a specific Curl_cfilter
instance.
- methods `Curl_conn_cf()` work on/with filter instances at a
connection.
- rebased and resolved some naming conflicts
- hostname validation (und session lookup) on SECONDARY use the same
name as on FIRST (again).
new debug macros and removing connectdata from function signatures where not
needed.
adapting schannel for new Curl_read_plain paramter.
Closes#9919
- Adding Curl_conn_is_ip_connected() to check if network connectivity
has been reached
- having ftp wait for network connectivity before proceeding with
transfers.
Fixes test failures 1631 and 1632 with hyper.
Closes#9952
- general construct/destroy in connectdata
- default implementations of callback functions
- connect: cfilters for connect and accept
- socks: cfilter for socks proxying
- http_proxy: cfilter for http proxy tunneling
- vtls: cfilters for primary and proxy ssl
- change in general handling of data/conn
- Curl_cfilter_setup() sets up filter chain based on data settings,
if none are installed by the protocol handler setup
- Curl_cfilter_connect() boot straps filters into `connected` status,
used by handlers and multi to reach further stages
- Curl_cfilter_is_connected() to check if a conn is connected,
e.g. all filters have done their work
- Curl_cfilter_get_select_socks() gets the sockets and READ/WRITE
indicators for multi select to work
- Curl_cfilter_data_pending() asks filters if the have incoming
data pending for recv
- Curl_cfilter_recv()/Curl_cfilter_send are the general callbacks
installed in conn->recv/conn->send for io handling
- Curl_cfilter_attach_data()/Curl_cfilter_detach_data() inform filters
and addition/removal of a `data` from their connection
- adding vtl functions to prevent use of Curl_ssl globals directly
in other parts of the code.
Reviewed-by: Daniel Stenberg
Closes#9855
- Use brackets for the IPv6 address shown in verbose message when the
format is address:port so that it is less confusing.
Before: Trying 2606:4700:4700::1111:443...
After: Trying [2606:4700:4700::1111]:443...
Bug: https://curl.se/mail/archive-2022-02/0041.html
Reported-by: David Hu
Closes#9635
The "Failed to connect to" message after a connection failure would
include the strerror message based on the presumed previous socket
error, but in times it seems that error number is not set when reaching
this code and therefore it would include the wrong error message.
The strerror message is now removed from here and the curl_easy_strerror
error is used instead.
Reported-by: Edoardo Lolletti
Fixes#9549Closes#9554
This internal-use-only storage type can be bumped to a curl_off_t once
we need to use bit 32 as the previous 'unsigned int' can no longer hold
them all then.
The websocket protocols take bit 30 and 31 so they are the last ones
that fit within 32 bits - but cannot properly be exported through APIs
since those use *signed* 32 bit types (long) in places.
Closes#9481
So that an address used from the DNS cache that was previously used for
QUIC can be reused for TCP and vice versa.
To make this possible, set conn->transport to "unix" for unix domain
connections ... and store the transport struct field in an unsigned char
to use less space.
Reported-by: ウさん
Fixes#9274Closes#9276
The options were added in #6341 and d13179d, but cause problems: Lots of
POLLIN event occurs but recvfrom read nothing.
Reported-by: Tatsuhiro Tsujikawa
Fixes#9209Closes#9215
Add licensing and copyright information for all files in this repository. This
either happens in the file itself as a comment header or in the file
`.reuse/dep5`.
This commit also adds a Github workflow to check pull requests and adapts
copyright.pl to the changes.
Closes#8869
Allow curl to send larger UDP datagram if Path MTU Discovery finds the
availability of larger path MTU. To make it work and not to send
fragmented packet, we need to set DF bit. That makes send(2) fail with
EMSGSIZE if UDP datagram is too large. In that case, just let it be
lost. This patch enables DF bit for Linux only.
Closes#8883
Earlier if CURLOPT_LOCALPORT + CURLOPT_LOCALPORTRANGE would go past port
65535 the code would fall back to random port rather than giving up.
Closes#8862
Make ngtcp2+quictls correctly acknowledge `CURLOPT_SSL_VERIFYPEER` and
`CURLOPT_SSL_VERIFYHOST`.
The name check now uses a function from lib/vtls/openssl.c which will
need attention for when TLS is not done by OpenSSL or is disabled while
QUIC is enabled.
Possibly the servercert() function in openssl.c should be adjusted to be
able to use for both regular TLS and QUIC.
Ref: #8173Closes#8178
... and make connect_init() refusing trying to tunnel protocols marked
as not working. Avoids a double-free.
Reported-by: Even Rouault
Fixes#8018Closes#8020
Regression. In d6a37c23a3 (7.75.0) we removed the duplicated storage
(connection + easy handle), so this info needs be extracted again even
for re-used connections.
Add test 435 to verify
Reported-by: Max Dymond
Fixes#7660Closes#7662
Commit dbd16c3e2 cleaned up the logic for traversing the addrinfos,
but the move left a conditional on ai which no longer is needed as
the while loop reevaluation will cover it.
Closes#7511
Reviewed-by: Carlo Marcelo Arenas Belón
Reviewed-by: Daniel Stenberg <daniel@haxx.se>
0842175 (not in any release) used the wrong format specifier (long int)
for timediff_t. On an OS such as Windows libcurl's timediff_t (usually
64-bit) is bigger than long int (32-bit). In 32-bit Windows builds the
upper 32-bits of the timediff_t were erroneously then used by the next
format specifier. Usually since the timeout isn't larger than 32-bits
this would result in null as a pointer to the string with the reason for
the connection failing. On other OSes or maybe other compilers it could
probably result in garbage values (ie crash on deref).
Before:
Failed to connect to localhost port 12345 after 1201 ms: (nil)
After:
Failed to connect to localhost port 12345 after 1203 ms: Connection refused
Closes https://github.com/curl/curl/pull/7449
- the data needs to be "line-based" anyway since it's also passed to the
debug callback/application
- it makes infof() work like failf() and consistency is good
- there's an assert that triggers on newlines in the format string
- Also removes a few instances of "..."
- Removes the code that would append "..." to the end of the data *iff*
it was truncated in infof()
Closes#7357
- Check whether a connection has succeded before checking whether it's
timed out.
This means if we've connected quickly, but subsequently been
descheduled, we allow the connection to succeed. Note, if we timeout,
but between checking the timeout, and connecting to the server the
connection succeeds, we will allow it to go ahead. This is viewed as
an acceptable trade off.
- Add additional failf logging around failed connection attempts to
propogate the cause up to the caller.
Co-Authored-by: Martin Howarth
Closes#7178
In some situations, it was possible that a transfer was setup to
use an specific IP version, but due do DNS caching or connection
reuse, it ended up using a different IP version from requested.
This commit changes the effect of CURLOPT_IPRESOLVE from simply
restricting address resolution to preventing the wrong connection
type being used, when choosing a connection from the pool, and
to restricting what addresses could be used when establishing
a new connection.
It is important that all addresses versions are resolved, even if
not used in that transfer in particular, because the result is
cached, and could be useful for a different transfer with a
different CURLOPT_IPRESOLVE setting.
Closes#6853
The duration of a connect and the total transfer are calculated from two
different time-stamps. It can end up with the total timeout triggering
before the connect timeout expires and we should make sure to
acknowledge whichever timeout that is reached first.
This is especially notable when a transfer first sits in PENDING, as
that time is counted in the total time but the connect timeout is based
on the time since the handle changed to the CONNECT state.
The CONNECTTIMEOUT is per connect attempt. The TIMEOUT is for the entire
operation.
Fixes#6744Closes#6745
Reported-by: Andrei Bica
Assisted-by: Jay Satiro
The Curl_easy pointer struct entry in connectdata is now gone. Just
before commit 215db086e0 landed on January 8, 2021 there were 919
references to conn->data.
Closes#6608
As the info is already stored in the transfer handle anyway, there's no
need to carry around a duplicate buffer for the life-time of the handle.
Closes#6534
... and use 'int' for ports. We don't use 'unsigned short' since -1 is
still often used internally to signify "unknown value" and 0 - 65535 are
all valid port numbers.
Closes#6534
... in most cases instead of 'struct connectdata *' but in some cases in
addition to.
- We mostly operate on transfers and not connections.
- We need the transfer handle to log, store data and more. Everything in
libcurl is driven by a transfer (the CURL * in the public API).
- This work clarifies and separates the transfers from the connections
better.
- We should avoid "conn->data". Since individual connections can be used
by many transfers when multiplexing, making sure that conn->data
points to the current and correct transfer at all times is difficult
and has been notoriously error-prone over the years. The goal is to
ultimately remove the conn->data pointer for this reason.
Closes#6425
The linux kernel does not report all ICMP errors back to userspace due
to historical reasons.
IP*_RECVERR sockopt must be turned on to have the correct behaviour
which is to pass all ICMP errors to userspace.
See https://bugzilla.kernel.org/show_bug.cgi?id=202355Closes#6341
If supported, defer port selection until connect() time
if --interface is given and source port is 0.
Reproducer:
* start fast webserver on port 80
* starve system of ephemeral ports
$ sysctl net.ipv4.ip_local_port_range="60990 60999"
* start a curl/libcurl "crawler"
$curl --keepalive --parallel --parallel-immediate --head --interface
127.0.0.2 "http://127.0.0.[1-254]/file[001-002].txt"
current result:
(possible some successful data)
curl: (45) bind failed with errno 98: Address already in use
result after patch:
(complete success or few connections failing, higlhy depending on load)
Fail only when all the possible 4-tuple combinations are exhausted,
which is impossible to do when port is selected at bind() time becuse
the kernel does not know if socket will be listen()'ed on or connect'ed
yet.
Closes#6295
Valgrind will complain that ssrem buffer usage if not explicit
initialized, hence initialize it to zero.
This completes the change intially started in commit 2c0d721215 ('ftp:
retry getpeername for FTP with TCP_FASTOPEN') where the ssloc buffer has
a similar memset to zero.
Signed-off-by: Hans-Christian Noren Egtvedt <hegtvedt@cisco.com>
Closes#6289
In the case of TFO, the remote host name is not resolved at the
connetion time.
For FTP that has lead to missing hostname for the secondary connection.
Therefore the name resolution is done at the time, when FTP requires it.
Fixes#6252Closes#6265Closes#6282
Setting a timeout to INT_MAX could cause an immediate error to get
returned as timeout because of an overflow when different values of
'now' were used.
This is primarily fixed by having Curl_pgrsTime() return the "now" when
TIMER_STARTSINGLE is set so that the parent function will continue using
that time.
Reported-by: Ionuț-Francisc Oancea
Fixes#5583Closes#5847
Failures clearly returned from a (SOCKS) proxy now causes this return
code. Previously the situation was not very clear as what would be
returned and when.
In addition: when this error code is returned, an application can use
CURLINFO_PROXY_ERROR to query libcurl for the detailed error, which then
returns a value from the new 'CURLproxycode' enum.
Closes#5770
For QUIC but also for regular TCP when the second family runs out of IPs
with a failure while the first family is still trying to connect.
Separated the timeout handling for IPv4 and IPv6 connections when they
both have a number of addresses to iterate over.
- Stick to a single unified way to use structs
- Make checksrc complain on 'typedef struct {'
- Allow them in tests, public headers and examples
- Let MD4_CTX, MD5_CTX, and SHA256_CTX typedefs remain as they actually
typedef different types/structs depending on build conditions.
Closes#5338
- Document in Curl_timeleft's comment block that returning 0 signals no
timeout (ie there's infinite time left).
- Fix SOCKS' Curl_blockread_all for the case when no timeout was set.
Prior to this change if the timeout had a value of 0 and that was passed
to SOCKET_READABLE it would return right away instead of blocking. That
was likely because it was not well understood that when Curl_timeleft
returns 0 it is not a timeout of 0 ms but actually means no timeout.
Ref: https://github.com/curl/curl/pull/5214#issuecomment-612512360
Closes https://github.com/curl/curl/pull/5220
Restores the --head functionality to the curl utility which extracts
'protocol' that is stored that way.
Reported-by: James Fuller
Fixes#5196Closes#5198
Make sure each separate index in connn->tempaddr[] is used for a fixed
family (and only that family) during the connection process.
If family one takes a long time and family two fails immediately, the
previous logic could misbehave and retry the same family two address
repeatedly.
Reported-by: Paul Vixie
Reported-by: Jay Satiro
Fixes#5083Fixes#4954Closes#5089
The ngtcp2 QUIC backend was using the MSG_DONTWAIT flag for send/recv
in order to perform nonblocking operations. On Windows this flag does
not exist. Instead, the socket must be set to nonblocking mode via
ioctlsocket.
This change sets the nonblocking flag on UDP sockets used for QUIC on
all platforms so the use of MSG_DONTWAIT is not needed.
Fixes#4531Closes#4532
Previosly all connect() failures would return CURLE_COULDNT_CONNECT, no
matter what errno said.
This makes for example --retry work on these transfer failures.
Reported-by: Nathaniel J. Smith
Fixes#4461
Clsoes #4462
