This struct field MUST remain what the application set it to, so that
handle reuse and handle duplication work.
Instead, the request state bit 'no_body' is introduced for code flows
that need to change this in run-time.
Closes#9888
- general construct/destroy in connectdata
- default implementations of callback functions
- connect: cfilters for connect and accept
- socks: cfilter for socks proxying
- http_proxy: cfilter for http proxy tunneling
- vtls: cfilters for primary and proxy ssl
- change in general handling of data/conn
- Curl_cfilter_setup() sets up filter chain based on data settings,
if none are installed by the protocol handler setup
- Curl_cfilter_connect() boot straps filters into `connected` status,
used by handlers and multi to reach further stages
- Curl_cfilter_is_connected() to check if a conn is connected,
e.g. all filters have done their work
- Curl_cfilter_get_select_socks() gets the sockets and READ/WRITE
indicators for multi select to work
- Curl_cfilter_data_pending() asks filters if the have incoming
data pending for recv
- Curl_cfilter_recv()/Curl_cfilter_send are the general callbacks
installed in conn->recv/conn->send for io handling
- Curl_cfilter_attach_data()/Curl_cfilter_detach_data() inform filters
and addition/removal of a `data` from their connection
- adding vtl functions to prevent use of Curl_ssl globals directly
in other parts of the code.
Reviewed-by: Daniel Stenberg
Closes#9855
Adds a new option to control the maximum time that a cached
certificate store may be retained for.
Currently only the OpenSSL backend implements support for
caching certificate stores.
Closes#9620
No point in having two entry points for the same functions.
Also merged the *safe* function treatment into these so that they can
also be used when one or both pointers are NULL.
Closes#9837
- Replace `Github` with `GitHub`.
- Replace `windows` with `Windows`
- Replace `advice` with `advise` where a verb is used.
- A few fixes on removing repeated words.
- Replace `a HTTP` with `an HTTP`
Closes#9802
Even though `PICKY_COMPILER=ON` is the default, warnings were not
enabled when using llvm/clang, because `CMAKE_COMPILER_IS_CLANG` was
always false (in my tests at least).
This is the single use of this variable in curl, and in a different
place we already use `CMAKE_C_COMPILER_ID MATCHES "Clang"`, which works
as expected, so change the condition to use that instead.
Also fix the warnings uncovered by the above:
- lib: add casts to silence clang warnings
- schannel: add casts to silence clang warnings in ALPN code
Assuming the code is correct, solve the warnings with a cast.
This particular build case isn't CI tested.
There is a chance the warning is relevant for some platforms, perhaps
Windows 32-bit ARM7.
Closes#9783
For both IPv4 and IPv6 addresses. Now also checks IPv6 addresses "correctly"
and not with string comparisons.
Split out the noproxy checks and functionality into noproxy.c
Added unit test 1614 to verify checking functions.
Reported-by: Mathieu Carbonneaux
Fixes#9773Fixes#5745Closes#9775
When the user name is provided in the URL it is URL encoded there, but
when used for authentication the encoded version should be used.
Regression introduced after 7.83.0
Reported-by: Jonas Haag
Fixes#9709Closes#9715
This is a strcmp() alternative function for comparing "secrets",
designed to take the same time no matter the content to not leak
match/non-match info to observers based on how fast it is.
The time this function takes is only a function of the shortest input
string.
Reported-by: Trail of Bits
Closes#9658
According to `docs/INTERNALS.md`, internal function names spanning source
files start with uppercase `Curl_`. Bring these two functions in
alignment with this.
This also stops exporting them from `libcurl.dll` in autotools builds.
Reviewed-by: Daniel Stenberg
Closes#9598
Move the curl_prot_t to its own conditional block. Introduce symbol
PROTO_TYPE_SMALL to control it.
Fix a cast in a curl_prot_t assignment.
Remove an outdated comment.
Follow-up to cd5ca80.
Closes#9534
This also returns error CURLE_UNSUPPORTED_PROTOCOL rather than
CURLE_BAD_FUNCTION_ARGUMENT when a listed protocol name is not found.
A new schemelen parameter is added to Curl_builtin_scheme() to support
this extended use.
Note that disabled protocols are not recognized anymore.
Tests adapted accordingly.
Closes#9472
When the parser is not allowed to guess scheme, it should consider the
word ending at the first colon to be the scheme, independently of number
of slashes.
The parser now checks that the scheme is known before it counts slashes,
to improve the error messge for URLs with unknown schemes and maybe no
slashes.
When following redirects, no scheme guessing is allowed and therefore
this change effectively prevents redirects to unknown schemes such as
"data".
Fixes#9503
This internal-use-only storage type can be bumped to a curl_off_t once
we need to use bit 32 as the previous 'unsigned int' can no longer hold
them all then.
The websocket protocols take bit 30 and 31 so they are the last ones
that fit within 32 bits - but cannot properly be exported through APIs
since those use *signed* 32 bit types (long) in places.
Closes#9481
Slightly faster with more robust code. Uses fewer and smaller mallocs.
- remove two fields from the URL handle struct
- reduce copies and allocs
- use dynbuf buffers more instead of custom malloc + copies
- uses dynbuf to build the host name in reduces serial alloc+free within
the same function.
- move dedotdotify into urlapi.c and make it static, not strdup the input
and optimize it by checking for . and / before using strncmp
- remove a few strlen() calls
- add Curl_dyn_setlen() that can "trim" an existing dynbuf
Closes#9408
Next Protocol Negotiation is a TLS extension that was created and used
for agreeing to use the SPDY protocol (the precursor to HTTP/2) for
HTTPS. In the early days of HTTP/2, before the spec was finalized and
shipped, the protocol could be enabled using this extension with some
servers.
curl supports the NPN extension with some TLS backends since then, with
a command line option `--npn` and in libcurl with
`CURLOPT_SSL_ENABLE_NPN`.
HTTP/2 proper is made to use the ALPN (Application-Layer Protocol
Negotiation) extension and the NPN extension has no purposes
anymore. The HTTP/2 spec was published in May 2015.
Today, use of NPN in the wild should be extremely rare and most likely
totally extinct. Chrome removed NPN support in Chrome 51, shipped in
June 2016. Removed in Firefox 53, April 2017.
Closes#9307
If the user is specified as part of the URL, and the same user exists
in .netrc, Authorization header was not sent at all.
The user and password fields were assigned in conn->user and password
but the user was not assigned to data->state.aptr, which is the field
that is used in output_auth_headers and friends.
Fix by assigning the user also to aptr.
Amends commit d1237ac906.
Fixes#9243
- If, after parsing netrc, there is a password with no username then
set a blank username.
This used to be the case prior to 7d600ad (precedes 7.82). Note
parseurlandfillconn already does the same thing for URLs.
Reported-by: Raivis <standsed@users.noreply.github.com>
Testing-by: Domen Kožar
Fixes https://github.com/curl/curl/issues/8653Closes#9334Closes#9066
This commit splits the branch-heavy resolve_server() function into
various sub-functions, in order to reduce the amount of nested
if/else-statements.
Beside this, it also removes many else-sequences, by returning in the
previous if-statement.
Closes#9283
So that an address used from the DNS cache that was previously used for
QUIC can be reused for TCP and vice versa.
To make this possible, set conn->transport to "unix" for unix domain
connections ... and store the transport struct field in an unsigned char
to use less space.
Reported-by: ウさん
Fixes#9274Closes#9276
and make 'dnstype' in 'struct dnsprobe' use the DNStype to fix the icc compiler warning:
doh.c(924): error #188: enumerated type mixed with another type
Reported-by: Matthew Thompson
Ref #9156Closes#9174
Add licensing and copyright information for all files in this repository. This
either happens in the file itself as a comment header or in the file
`.reuse/dep5`.
This commit also adds a Github workflow to check pull requests and adapts
copyright.pl to the changes.
Closes#8869
The .netrc parser now accepts strings within double-quotes in order to
deal with for example passwords containing white space - which
previously was not possible.
A password that starts with a double-quote also ends with one, and
double-quotes themselves are escaped with backslashes, like \". It also
supports \n, \r and \t for newline, carriage return and tabs
respectively.
If the password does not start with a double quote, it will end at first
white space and no escaping is performed.
WARNING: this change is not entirely backwards compatible. If anyone
previously used a double-quote as the first letter of their password,
the parser will now get it differently compared to before. This is
highly unfortunate but hard to avoid.
Reported-by: ImpatientHippo on GitHub
Fixes#8908Closes#8937
Usage:
curl -x "socks5h://localhost/run/tor/socks" "https://example.com"
Updated runtests.pl to run a socksd server listening on unix socket
Added tests test1467 test1468
Added documentation for proxy command line option and socks proxy
options
Closes#8668
These two options were only ever used for the OpenSSL backend for
versions before 1.1.0. They were never used for other backends and they
are not used with recent OpenSSL versions. They were never used much by
applications.
The defines RANDOM_FILE and EGD_SOCKET can still be set at build-time
for ancient EOL OpenSSL versions.
Closes#8670
To simplify, and also since the returned name is not the full actual
name used for the check. The port number and zone id is also involved,
so just showing the name is misleading.
Closes#8750
Also move static function safecmp() as non-static Curl_safecmp() since
its purpose is needed at several places.
Bug: https://curl.se/docs/CVE-2022-22576.html
CVE-2022-22576
Closes#8746
Reverts 5de8d84098 (May 2014, shipped in 7.37.0) and the
follow-up changes done afterward.
Keep the dot in names for everything except the SNI to make curl behave
more similar to current browsers. This means 'name' and 'name.' send the
same SNI for different 'Host:' headers.
Updated test 1322 accordingly
Fixes#8290
Reported-by: Charles Cazabon
Closes#8320
The tools.ietf.org domain has been deprecated a while now, with the
links being redirected to datatracker.ietf.org.
Rather than make people eat that redirect time, this change switches the
URL to a more canonical source.
Closes#8317
1. The function would only ever return CURLE_OK anyway
2. Only one caller actually used the return code
3. Most callers did (void)Curl_disconnect()
Closes#8303
Instad of having all callers pass in the maximum length, always use
it. The passed in length is instead used only as the length of the
target buffer for to storing the scheme name in, if used.
Added the scheme max length restriction to the curl_url_set.3 man page.
Follow-up to 45bcb2eaa7Closes#8047
Add curl_url_strerror() to convert CURLUcode into readable string and
facilitate easier troubleshooting in programs using URL API.
Extend CURLUcode with CURLU_LAST for iteration in unit tests.
Update man pages with a mention of new function.
Update example code and tests with new functionality where it fits.
Closes#7605
We have and provide Curl_strerror() internally for a reason: strerror()
is not necessarily thread-safe so we should always try to avoid it.
Extended checksrc to warn for this, but feature the check disabled by
default and only enable it in lib/
Closes#7685
warning C4189: 'netrc_user_changed': local variable is initialized but
not referenced
warning C4189: 'netrc_passwd_changed': local variable is initialized but
not referenced
Closes#7423
- the data needs to be "line-based" anyway since it's also passed to the
debug callback/application
- it makes infof() work like failf() and consistency is good
- there's an assert that triggers on newlines in the format string
- Also removes a few instances of "..."
- Removes the code that would append "..." to the end of the data *iff*
it was truncated in infof()
Closes#7357
Coverity (CID 1486645) pointed out a use of curl_url_get() in the
parse_proxy function where the return code wasn't checked. A
(void)-prefix makes the intention obvious.
Closes#7320
Fix typos in code comments which repeat various words. In trivial
cases, just delete the repeated word. Reword the affected sentence in
"lib/url.c" for it to make sense.
Closes#7303
Reviewed-by: Daniel Gustafsson <daniel@yesql.se>
- Check whether a connection has succeded before checking whether it's
timed out.
This means if we've connected quickly, but subsequently been
descheduled, we allow the connection to succeed. Note, if we timeout,
but between checking the timeout, and connecting to the server the
connection succeeds, we will allow it to go ahead. This is viewed as
an acceptable trade off.
- Add additional failf logging around failed connection attempts to
propogate the cause up to the caller.
Co-Authored-by: Martin Howarth
Closes#7178
Unicode Windows builds use UTF-8 strings internally in libcurl,
so make sure to call the UTF-8 flavour of the libidn2 API. Also
document that Windows builds with libidn2 and UNICODE do expect
CURLOPT_URL as an UTF-8 string.
Reported-by: dEajL3kA on github
Assisted-by: Jay Satiro
Reviewed-by: Marcel Raad
Closes#7246Fixes#7228
Also, use a single function library-wide for detecting if a given hostname is
a numerical IP address.
Reported-by: Harry Sintonen
Fixes#7146Closes#7149
In some situations, it was possible that a transfer was setup to
use an specific IP version, but due do DNS caching or connection
reuse, it ended up using a different IP version from requested.
This commit changes the effect of CURLOPT_IPRESOLVE from simply
restricting address resolution to preventing the wrong connection
type being used, when choosing a connection from the pool, and
to restricting what addresses could be used when establishing
a new connection.
It is important that all addresses versions are resolved, even if
not used in that transfer in particular, because the result is
cached, and could be useful for a different transfer with a
different CURLOPT_IPRESOLVE setting.
Closes#6853
The libssh2 backend has SSH session associated with the connection but
the callback context is the easy handle, so when a connection gets
attached to a transfer, the protocol handler now allows for a custom
function to get used to set things up correctly.
Reported-by: Michael O'Farrell
Fixes#6898Closes#7078
Both were used for the same purposes and there was no logical separation
between them. Combined, this also saves 16 bytes in less holes in my
test build.
Closes#6798
The Curl_easy pointer struct entry in connectdata is now gone. Just
before commit 215db086e0 landed on January 8, 2021 there were 919
references to conn->data.
Closes#6608
- New libcurl options CURLOPT_DOH_SSL_VERIFYHOST,
CURLOPT_DOH_SSL_VERIFYPEER and CURLOPT_DOH_SSL_VERIFYSTATUS do the
same as their respective counterparts.
- New curl tool options --doh-insecure and --doh-cert-status do the same
as their respective counterparts.
Prior to this change DOH SSL certificate verification settings for
verifyhost and verifypeer were supposed to be inherited respectively
from CURLOPT_SSL_VERIFYHOST and CURLOPT_SSL_VERIFYPEER, but due to a bug
were not. As a result DOH verification remained at the default, ie
enabled, and it was not possible to disable. This commit changes
behavior so that the DOH verification settings are independent and not
inherited.
Ref: https://github.com/curl/curl/pull/4579#issuecomment-554723676
Fixes https://github.com/curl/curl/issues/4578
Closes https://github.com/curl/curl/pull/6597
HTTP auth "accidentally" worked before this cleanup since the code would
always overwrite the connection credentials with the credentials from
the most recent transfer and since HTTP auth is typically done first
thing, this has not been an issue. It was still wrong and subject to
possible race conditions or future breakage if the sequence of functions
would change.
The data.set.str[] strings MUST remain unmodified exactly as set by the
user, and the credentials to use internally are instead set/updated in
state.aptr.*
Added test 675 to verify different credentials used in two requests done
over a reused HTTP connection, which previously behaved wrongly.
Fixes#6542Closes#6545
Rename it to 'httpwant' and make a cloned field in the state struct as
well for run-time updates.
Also: refuse non-supported HTTP versions. Verified with test 129.
Closes#6585
As the info is already stored in the transfer handle anyway, there's no
need to carry around a duplicate buffer for the life-time of the handle.
Closes#6534
... and use 'int' for ports. We don't use 'unsigned short' since -1 is
still often used internally to signify "unknown value" and 0 - 65535 are
all valid port numbers.
Closes#6534
... instead of having it static within the Curl_easy struct. This takes
away 1176 bytes (18%) from the Curl_easy struct that aren't used very
often and instead makes the code allocate it when needed.
Closes#6492
... in most cases instead of 'struct connectdata *' but in some cases in
addition to.
- We mostly operate on transfers and not connections.
- We need the transfer handle to log, store data and more. Everything in
libcurl is driven by a transfer (the CURL * in the public API).
- This work clarifies and separates the transfers from the connections
better.
- We should avoid "conn->data". Since individual connections can be used
by many transfers when multiplexing, making sure that conn->data
points to the current and correct transfer at all times is difficult
and has been notoriously error-prone over the years. The goal is to
ultimately remove the conn->data pointer for this reason.
Closes#6425
... and not in the connection setup, as for multiplexed transfers the
connection setup might be skipped and then the transfer would end up
without the set user-agent!
Reported-by: Flameborn on github
Assisted-by: Andrey Gursky
Assisted-by: Jay Satiro
Assisted-by: Mike Gelfand
Fixes#6312Closes#6417
When doing HTTP authentication and a port number set with CURLOPT_PORT,
the code would previously have the URL's port number override as if it
had been a redirect to an absolute URL.
Added test 1568 to verify.
Reported-by: UrsusArctos on github
Fixes#6397Closes#6400
We currently use both spellings the british "behaviour" and the american
"behavior". However "behavior" is more used in the project so I think
it's worth dropping the british name.
Closes#6395
This commit introduces a "gophers" handler inside the gopher protocol if
USE_SSL is defined. This protocol is no different than the usual gopher
prococol, with the added TLS encapsulation upon connecting. The protocol
has been adopted in the gopher community, and many people have enabled
TLS in their gopher daemons like geomyidae(8), and clients, like clic(1)
and hurl(1).
I have not implemented test units for this protocol because my knowledge
of Perl is sub-par. However, for someone more knowledgeable it might be
fairly trivial, because the same test that tests the plain gopher
protocol can be used for "gophers" just by adding a TLS listener.
Signed-off-by: parazyd <parazyd@dyne.org>
Closes#6208
The command line tool also independently sets --ftp-skip-pasv-ip by
default.
Ten test cases updated to adapt the modified --libcurl output.
Bug: https://curl.se/docs/CVE-2020-8284.html
CVE-2020-8284
Reported-by: Varnavas Papaioannou
- enable in the build (configure)
- header parsing
- host name lookup
- unit tests for the above
- CI build
- CURL_VERSION_HSTS bit
- curl_version_info support
- curl -V output
- curl-config --features
- CURLOPT_HSTS_CTRL
- man page for CURLOPT_HSTS_CTRL
- curl --hsts (sets CURLOPT_HSTS_CTRL and works with --libcurl)
- man page for --hsts
- save cache to disk
- load cache from disk
- CURLOPT_HSTS
- man page for CURLOPT_HSTS
- added docs/HSTS.md
- fixed --version docs
- adjusted curl_easy_duphandle
Closes#5896
The cache content is not duplicated, like other caches, but the setting
and specified file name are.
Test 1908 is extended to verify this somewhat. Since the duplicated
handle gets the same file name, the test unfortunately overwrites the
same file twice (with different contents) which makes it hard to check
automatically.
Closes#5923
`USE_WINDOWS_SSPI` without `USE_WIN32_CRYPTO` but with any other DES
backend is fine, but was excluded before.
This also fixes test 1013 as the condition for SMB support in
configure.ac didn't match the condition in the source code. Now it
does.
Fixes https://github.com/curl/curl/issues/1262
Closes https://github.com/curl/curl/pull/5771
Since commit f3d501dc67, if proxy support is disabled, MSVC warns:
url.c : warning C4701: potentially uninitialized local variable
'hostaddr' used
url.c : error C4703: potentially uninitialized local pointer variable
'hostaddr' used
That could actually only happen if both `conn->bits.proxy` and
`CURL_DISABLE_PROXY` were enabled.
Initialize it to NULL to silence the warning.
Closes https://github.com/curl/curl/pull/5638
Follow-up to c4e6968127
When a new transfer is created, as a resuly of an acknowledged push,
that transfer needs a download buffer allocated.
Closes#5590
Since the connection can be used by many independent requests (using
HTTP/2 or HTTP/3), things like user-agent and other transfer-specific
data MUST NOT be kept connection oriented as it could lead to requests
getting the wrong string for their requests. This struct data was
lingering like this due to old HTTP1 legacy thinking where it didn't
mattered..
Fixes#5566Closes#5567
