This is a better match for what they do and the general "cpool"
var/function prefix works well.
The pool now handles very long hostnames correctly.
The following changes have been made:
* 'struct connectdata', e.g. connections, keep new members
named `destination` and ' destination_len' that fully specifies
interface+port+hostname of where the connection is going to.
This is used in the pool for "bundling" of connections with
the same destination. There is no limit on the length any more.
* Locking: all locks are done inside conncache.c when calling
into the pool and released on return. This eliminates hazards
of the callers keeping track.
* 'struct connectbundle' is now internal to the pool. It is no
longer referenced by a connection.
* 'bundle->multiuse' no longer exists. HTTP/2 and 3 and TLS filters
no longer need to set it. Instead, the multi checks on leaving
MSTATE_CONNECT or MSTATE_CONNECTING if the connection is now
multiplexed and new, e.g. not conn->bits.reuse. In that case
the processing of pending handles is triggered.
* The pool's init is provided with a callback to invoke on all
connections being discarded. This allows the cleanups in
`Curl_disconnect` to run, wherever it is decided to retire
a connection.
* Several pool operations can now be fully done with one call.
Pruning dead connections, upkeep and checks on pool limits
can now directly discard connections and need no longer return
those to the caller for doing that (as we have now the callback
described above).
* Finding a connection for reuse is now done via `Curl_cpool_find()`
and the caller provides callbacks to evaluate the connection
candidates.
* The 'Curl_cpool_check_limits()' now directly uses the max values
that may be set in the transfer's multi. No need to pass them
around. Curl_multi_max_host_connections() and
Curl_multi_max_total_connections() are gone.
* Add method 'Curl_node_llist()' to get the llist a node is in.
Used in cpool to verify connection are indeed in the list (or
not in any list) as they need to.
I left the conncache.[ch] as is for now and also did not touch the
documentation. If we update that outside the feature window, we can
do this in a separate PR.
Multi-thread safety is not achieved by this PR, but since more details
on how pools operate are now "internal" it is a better starting
point to go for this in the future.
Closes#14662
Before this patch, `libhostname.so` and `chkhostname` were a test
facility for overriding `gethostname()` in non-debug builds on
Linux and other Unix platforms supporting `LD_PRELOAD`.
`gethostname()` has a single use with SMTP.
The alternative way to override `gethostname()` is building in debug
mode, which allows to do this via the `CURL_GETHOSTNAME` env, on all
platforms.
Drop the `LD_PRELOAD` solution in favour of the above.
Also:
- delete inactive NTLM code with a `gethostname()` call made from it.
- streamline NTLM code by dropping a `printf()` and a macro.
- tests: stop setting `CURL_GETHOSTNAME` where unnecessary.
Closes#14695
The missing message only uses a single line now and has a consistent
prefix making it easier for log parsers to skip. Remove a precheck test
looking for impacket in test1451 because it's incomplete, unnecessary
and in the wrong layer; the test harness will automatically discover
that the smbserver won't start and the test was only checking a single
dependency. Also update the tests README to no longer mention Python 2.
The servers were switched to Python 3 several years ago and no attempt
is being made to maintain Python 2 compatibility
Closes#14669
- Install stunnel.
- Regenerate certificates (as SecureTransport requires a validity period
less than 398 days).
- Restart server if it is unresponsive.
- Do not hardcode the SHA-256 base64 public pinned key.
- Ignore test 313 as SecureTransport does not support crl file.
- Ignore tests 1631 and 1632 as SecureTransport is not yet able to shut
down FTP over HTTPS gracefully.
- Add a CMake target for generating certificates.
Closes#14486
Previously this test allowed several error values when setting options.
This made this test miss #14629.
Now, errors are generally not accepted for setopts:
- numerical setopts accept CURLE_BAD_FUNCTION_ARGUMENT for funny input
- the first setopt to an option accepts CURLE_NOT_BUILT_IN or
CURLE_UNKNOWN_OPTION for when they are disabled/not built-in
- there is an allowlist concept for some return code for some variables,
managed at the top of the mk-lib1521.pl script
In curl.h: remove the OBSOLETE named values from the setopt list.
Closes#14634
urle.scheme, urle.user, urle.password and urle.options mistakenly
operated on the original URL instead of the *effective* (last) URL.
Add test 474 to verify.
Reported-by: Gruber Glass
Fixes#14550Closes#14560
Remove check of server output as upload may abort before request
could fully be sent, so server output may be completely missing.
Test already used a 1 second delay to mitigate timing. This change
makes timing no longer an issue.
Closes#14525
- make sure to exclude failing tests when libidn2 is detected by
default.
- ignore test 1560 results. Seen to fail with libidn2.
I'm not sure why this test was not executed earlier:
https://github.com/curl/curl/actions/runs/10354610889/job/28660309355#step:13:3647
- runtests: recognize `libidn2` as a feature.
- move IDN test exclusions from GHA/windows to `tests/data/DISABLED`.
- GHA/windows: drop default `-DUSE_LIBIDN2=ON` cmake config.
Cherry-picked from #14495Closes#14519
Log progress only at start and end of transfer to give normalized
output when upload data is only partially sent or temporarily blocked.
Fixes test with CURL_DBG_SOCK_WBLOCK=90 set.
Closes#14454
- QUIT is not an important FTP command
- curl only sends it "best effort", meaning it might not be sent
- it is a known "flaky" thing in test output because of this
Closes#14404
If a request containing two headers that have equivalent prefixes (ex.
"x-amz-meta-test:test" and "x-amz-meta-test-two:test2") AWS expects the
header with the shorter name to come first. The previous implementation
used `strcmp` on the full header. Using the example, this would result
in a comparison between the ':' and '-' chars and sort
"x-amz-meta-test-two" before "x-amz-meta-test", which produces a
different "StringToSign" than the one calculated by AWS.
Test 1976 verifies
Closes#14370
With this option, the entire download is skipped if the selected target
filename already exists when the opertion is about to begin.
Test 994, 995 and 996 verify.
Ref: #11012Closes#13993
Use these words and casing more consistently across text, comments and
one curl tool output:
AIX, ALPN, ANSI, BSD, Cygwin, Darwin, FreeBSD, GitHub, HP-UX, Linux,
macOS, MS-DOS, MSYS, MinGW, NTLM, POSIX, Solaris, UNIX, Unix, Unicode,
WINE, WebDAV, Win32, winbind, WinIDN, Windows, Windows CE, Winsock.
Mostly OS names and a few more.
Also a couple of other minor text fixups.
Closes#14360
Since the documentation text blob might be gzipped, it needs to search
for what to output in a streaming manner. It then first searches for
"\nALL OPTIONS".
Then, it looks for the start to display at "\n -[option]" and stops
again at "\n -". Except for the last option in the man page, which
ends at "\nFILES" - the subtitle for the section following all options
in the manpage.
Test 1707 to 1710 verify
Closes#13997
... or pick the last directory part from the path if available.
Instead of returning error.
Add test 690 and 691 to verify. Test 76 and 2036 no longer apply.
Closes#13988
It's a no-op since
d162fca69a#9333 (2022-08-18).
Also revert 476499c75c that is
no longer necessary: move `Makefile.inc` back into `Makefile.am`.
Closes#14357
Set the initial stream window size to 64KB and increase that to the 10MB
we used to start with on the first server reply, unless a rate limit is
in effect.
Continously monitory changes to the transfers rate limit and adjust the
stream window size accordingly. `max_recv_speed` is a transfer propert
that can be changed during processing by a callback.
Closes#14326
Instead of providing a fixed single synthetic response in the test
server itself. To allow us to better use *different* directory listings
in different test cases. In this change, most listings remain the same
as before.
The wildcard match tests still use synthetic responses but we should fix
that as well.
Updated numerous test cases to use this.
Closes#14295
- generate AVAILABILITY manpage sections automatically - for consistent
wording
- allows us to double-check against other documumentation (symbols-in-versions
etc)
- enables proper automation/scripting based on this data
- lots of them were wrong or missing in the manpages
- several of them repeated (sometimes mismatching) backend support info
Add test 1488 to verify "added-in" version numbers against
symbols-in-versions.
Closes#14217
Some feature names used in tests had minor differences compared to
the well-known ones from `curl -V`. This patch syncs them to make test
results easier to grok.
Closes#14183
When CRLF line end conversion was enabled (--crlf), input after the last
newline in the upload buffer was not sent, if the buffer contained a
newline.
Reported-by: vuonganh1993 on github
Fixes#14165Closes#14169
- disbable this test on WIN32 platforms. It uses the file describtor '1'
as valid socket without events. Not portable.
- reduce trace output somewhat on other runs
Fixes#14177
Reported-by: Viktor Szakats
Closes#14191
The commit 6483813b was missing changes necessitated by 2abfc75 that
causes a crash. Also, use ARRAYSIZE() for cleaner code.
Follow-up to 6483813b
Ref #14055
This eliminates the need to run an extra help subcommand to get the
possible categories, reducing the friction in getting relevant help. The
help wording was also slightly tweaked for grammatical accuracy.
Closes#14055
Option cleanups:
--get is not upload
--form* are post
- added several options into ldap, smtp, imap and pop3
- shortened the category descriptions in the list
category curl fixes:
--create-dirs removed from 'curl'
--ftp-create-dirs removed from 'curl'
--netrc moved to 'auth' from 'curl'
--netrc-file moved to 'auth' from 'curl'
--netrc-optional moved to 'auth' from 'curl'
--no-buffer moved to 'output' from 'curl'
--no-clobber removed from 'curl'
--output removed from 'curl'
--output-dir removed from 'curl'
--remove-on-error removed from 'curl'
Add a "global" category:
- Made all "global" options set this category
Add a "deprecated" category:
- Moved the deprecated options to it (maybe they should not be in any
category long term)
Add a 'timeout' category
- Put a number of appropriate options in it
Add an 'ldap' category
- Put the LDAP related option in there
Remove categories "ECH" and "ipfs"
- They should not be categories. Had only one single option each.
Remove category "misc"
- It should not be a category as it is impossible to know when to browse
it.
--use-ascii moved to ftp and output
--xattr moved to output
--service-name moved to auth
Managen fixes:
- errors if an option is given a category name that is not already setup
for in code
- verifies that options set `scope: global` also is put in category
`global´
Closes#14101
Based on the standards and guidelines we use for our documentation.
- expand contractions (they're => they are etc)
- host name = > hostname
- file name => filename
- user name = username
- man page => manpage
- run-time => runtime
- set-up => setup
- back-end => backend
- a HTTP => an HTTP
- Two spaces after a period => one space after period
Closes#14073
When an individual file ended with a quote (typically an example), the
render function would return without ending the quote correctly with a
".fi" (fill in) in the manpage output.
This made the additional text provided below to render wrongly.
Closes#14048
Fix issues detected.
Also:
- One of the `.vc` files used LF EOLs, while the other didn't.
Make that one also use LF EOLs, as this is apparently supported by
`nmake`.
- Drop `.dsw` and `.btn` types from `.gitattributes`.
The repository doesn't use them.
- Sync section order with the rest of files in
`tests/certs/EdelCurlRoot-ca.prm`.
- Indent/align `.prm` and `.pem` files.
- Delete dummy `[something]` section from `.prm` and `.pem` files.
Mental note:
MSVC `.sln` files seem to accept spaces for indentation and also support
LF line-endings. I cannot test this and I don't know what's more
convenient when updating them, so left them as-is, with specific
exclusions.
Closes#14031
When libcurl discards a connection there are two phases this may go
through: "shutdown" and "closing". If a connection is aborted, the
shutdown phase is skipped and it is closed right away.
The connection filters attached to the connection implement the phases
in their `do_shutdown()` and `do_close()` callbacks. Filters carry now a
`shutdown` flags next to `connected` to keep track of the shutdown
operation.
Filters are shut down from top to bottom. If a filter is not connected,
its shutdown is skipped. Notable filters that *do* something during
shutdown are HTTP/2 and TLS. HTTP/2 sends the GOAWAY frame. TLS sends
its close notify and expects to receive a close notify from the server.
As sends and receives may EAGAIN on the network, a shutdown is often not
successful right away and needs to poll the connection's socket(s). To
facilitate this, such connections are placed on a new shutdown list
inside the connection cache.
Since managing this list requires the cooperation of a multi handle,
only the connection cache belonging to a multi handle is used. If a
connection was in another cache when being discarded, it is removed
there and added to the multi's cache. If no multi handle is available at
that time, the connection is shutdown and closed in a one-time,
best-effort attempt.
When a multi handle is destroyed, all connection still on the shutdown
list are discarded with a final shutdown attempt and close. In curl
debug builds, the environment variable `CURL_GRACEFUL_SHUTDOWN` can be
set to make this graceful with a timeout in milliseconds given by the
variable.
The shutdown list is limited to the max number of connections configured
for a multi cache. Set via CURLMOPT_MAX_TOTAL_CONNECTIONS. When the
limit is reached, the oldest connection on the shutdown list is
discarded.
- In multi_wait() and multi_waitfds(), collect all connection caches
involved (each transfer might carry its own) into a temporary list.
Let each connection cache on the list contribute sockets and
POLLIN/OUT events it's connections are waiting for.
- in multi_perform() collect the connection caches the same way and let
them peform their maintenance. This will make another non-blocking
attempt to shutdown all connections on its shutdown list.
- for event based multis (multi->socket_cb set), add the sockets and
their poll events via the callback. When `multi_socket()` is invoked
for a socket not known by an active transfer, forward this to the
multi's cache for processing. On closing a connection, remove its
socket(s) via the callback.
TLS connection filters MUST NOT send close nofity messages in their
`do_close()` implementation. The reason is that a TLS close notify
signals a success. When a connection is aborted and skips its shutdown
phase, the server needs to see a missing close notify to detect
something has gone wrong.
A graceful shutdown of FTP's data connection is performed implicitly
before regarding the upload/download as complete and continuing on the
control connection. For FTP without TLS, there is just the socket close
happening. But with TLS, the sent/received close notify signals that the
transfer is complete and healthy. Servers like `vsftpd` verify that and
reject uploads without a TLS close notify.
- added test_19_* for shutdown related tests
- test_19_01 and test_19_02 test for TCP RST packets
which happen without a graceful shutdown and should
no longer appear otherwise.
- add test_19_03 for handling shutdowns by the server
- add test_19_04 for handling shutdowns by curl
- add test_19_05 for event based shutdowny by server
- add test_30_06/07 and test_31_06/07 for shutdown checks
on FTP up- and downloads.
Closes#13976
- Parse etag and content-disposition headers for 3xx replies.
For example, a server may send a content-disposition filename header
with a redirect reply (3xx) but not with the final response (2xx).
Without this change curl would ignore the server's specified filename
and continue to use the filename extracted from the user-specified URL.
Prior to this change, 75d79a4 had limited etag and content-disposition
to 2xx replies only.
Tests-by: Daniel Stenberg
Reported-by: Morgan Willcock
Fixes https://github.com/curl/curl/issues/13302Closes#13484
