Commit Graph

1835 Commits

Author SHA1 Message Date
Nathan Moinvaziri
6ea9388157
url: remove duplicate call to Curl_conncache_remove_conn when pruning
- remove unnecessary prunedead struct from prune_dead_connections
- rename extract_if_dead to prune_if_dead for clarity

Closes #13710
2024-05-21 16:25:21 +02:00
Daniel Stenberg
0e37b42dc9
url: make parse_login_details use memdup0
Also make the user and password arguments mandatory, since all code
paths in libcurl used them anyway.

Adapted unit test case 1620 to the new rules.

Closes #13584
2024-05-10 22:53:12 +02:00
Daniel Stenberg
c294f9cb56
lib: make protocol handlers store scheme name lowercase
- saves a lowercase operation when the "[scheme]_proxy" name is
  generated
- appears less "shouting"
- update test 970, 972, 1438 and 1536

Closes #13553
2024-05-08 09:39:30 +02:00
Daniel Stenberg
926fb00405
request: make Curl_req_init return void
Since it could not return error and therefore this change removes dead
code for the caller.

Spotted by CodeSonar.

Closes #13423
2024-04-19 23:42:33 +02:00
Daniel Stenberg
25236c6a80
multi: remove the unused Curl_preconnect function
The implementation has been removed, no point in keeping it around.

Follow-up to 476adfeac0

Closes #13422
2024-04-19 15:06:48 +02:00
Jiwoo Park
5fb018494d
url: fix use of an uninitialized variable
Closes #13399
2024-04-17 22:34:47 +02:00
Viktor Szakats
49f83c30e4
lib: merge ENABLE_QUIC C macro into USE_HTTP3
Before this patch `lib/curl_setup.h` defined these two macros right
next to each other, then the source code used them interchangeably.

After this patch, `USE_HTTP3` guards all HTTP/3 / QUIC features.
(Like `USE_HTTP2` does for HTTP/2.) `ENABLE_QUIC` is no longer used.

This patch doesn't change the way HTTP/3 is enabled via autotools
or CMake. Builders who enabled HTTP/3 manually by defining both of
these macros via `CPPFLAGS` can now delete `-DENABLE_QUIC`.

Closes #13352
2024-04-13 08:33:27 +00:00
Viktor Szakats
e411c98f70
build: prefer USE_IPV6 macro internally (was: ENABLE_IPV6)
Before this patch, two macros were used to guard IPv6 features in curl
sources: `ENABLE_IPV6` and `USE_IPV6`. This patch makes the source use
the latter for consistency with other similar switches.

`-DENABLE_IPV6` remains accepted for compatibility as a synonym for
`-DUSE_IPV6`, when passed to the compiler.

`ENABLE_IPV6` also remains the name of the CMake and `Makefile.vc`
options to control this feature.

Closes #13349
2024-04-13 08:33:26 +00:00
Daniel Stenberg
50def7c881
NTLM_WB: drop support
The feature has not worked for months and has been marked as DEPRECATED
for six+ months.

Closes #13249
2024-04-08 13:58:58 +02:00
MAntoniak
f46385d36d
urldata: remove fields not used depending on used features
Reduced size of dynamically_allocated_data structure.

Reduced number of stored values in enum dupstring and enum dupblob. This
affects the reduced array placed in the UserDefined structure.

Closes #13188
2024-04-05 16:06:22 +02:00
Daniel Stenberg
d5e83eb745
url: do not URL decode proxy crendentials
The two options CURLOPT_PROXYUSERNAME and CURLOPT_PROXYPASSWORD set the
actual names as-is, not URL encoded.

Modified test 503 to use percent-encoded strings in the credential
strings that should be passed on as-is.

Reported-by: Sergey Ogryzkov
Fixes #13265
Closes #13270
2024-04-04 11:35:19 +02:00
Stefan Eissing
80a3b830cc
http: expect 100 rework
Move all handling of HTTP's `Expect: 100-continue` feature into a client
reader. Add sending flag `KEEP_SEND_TIMED` that triggers transfer
sending on general events like a timer.

HTTP installs a `CURL_CR_PROTOCOL` reader when announcing `Expect:
100-continue`. That reader works as follows:

- on first invocation, records time, starts the `EXPIRE_100_TIMEOUT`
  timer, disables `KEEP_SEND`, enables `KEEP_SEND_TIMER` and returns 0,
  eos=FALSE like a paused upload.

- on subsequent invocation it checks if the timer has expired. If so, it
  enables `KEEP_SEND` and switches to passing through reads to the
  underlying readers.

Transfer handling's `readwrite()` will be invoked when a timer expires
(like `EXPIRE_100_TIMEOUT`) or when data from the server arrives. Seeing
`KEEP_SEND_TIMER`, it will try to upload more data, which triggers
reading from the client readers again. Which then may lead to a new
pausing or cause the upload to start.

Flags and timestamps connected to this have been moved from
`SingleRequest` into the reader's context.

Closes #13110
2024-03-18 12:41:56 +01:00
Stefan Eissing
4e4e8af1f6
lib: move 'done' parameter to SingleRequests
A transfer may do several `SingleRequest`s for its success. This happens
regularly for authentication, follows and retries on failed connections.
The "readwrite()" calls and functions connected to those carried a `bool
*done` parameter to indicate that the current `SingleRequest` is over.
This may happen before `upload_done` or `download_done` bits of
`SingleRequest` are set.

The problem with that is now `write_resp()` protocol handlers are
invoked in places where the `bool *done` cannot be passed up to the
caller. Instead of being a bool in the call chain, it needs to become a
member of `SingleRequest`, reflecting its state.

This removes the `bool *done` parameter and adds the `done` bit to
`SingleRequest` instead. It adds `Curl_req_soft_reset()` for using a
`SingleRequest` in a follow up, clearing `done` and other
flags/counters.

Closes #13096
2024-03-11 23:27:02 +01:00
Stefan Eissing
fcef00db1a
lib: keep conn IP information together
new struct ip_quadruple for holding local/remote addr+port

- used in data->info and conn and cf-socket.c
- copy back and forth complete struct
- add 'secondary' to conn
- use secondary in reporting success for ftp 2nd connection

Reported-by: DasKutti on github
Fixes #13084
Closes #13090
2024-03-09 23:24:58 +01:00
Stefan Eissing
a586b8ca40
lib: client reader polish
- seek_func/seek_client, use transfer values only
    - remove copies held in `struct connectdata`, use only
      ever `data->set.seek_func`
    - resolves possible issues in multiuse connections
    - new mime post reader eliminates need to ever overwriting this

- websockets, remove empty Curl_ws_done() function

Closes #13079
2024-03-08 13:11:17 +01:00
Stefan Eissing
e3905de819
lib: further send/upload handling polish
- Move all the "upload_done" handling to request.c

  - add possibility to abort sending of a request
  - add `Curl_req_done_sending()` for checks
  - transfer.c: readwrite_upload() now clean

- removing data->state.ulbuf and data->req.upload_fromhere

  - as well as data->req.upload_present
  - set data->req.upload_done on having read all from
    the client and completely flushed the send buffer

- tftp, remove setting of data->req.upload_fromhere

  - serves no purpose as `upload_present` is not set
    and the data itself is directly `sendto()` anyway

- smtp, make upload EOB conversion a client reader
- xfer_ulbuf addition

  - add xfer_ulbuf for borrowing, similar to xfer_buf
  - use in file upload
  - use in c-hyper body sending

- h1-proxy, remove init of data->state.uilbuf that is never used
- smb, add own send_buf instead of using data->state.ulbuf

Closes #13010
2024-03-04 08:42:56 +01:00
Stefan Eissing
3755153571
lib: Curl_read/Curl_write clarifications
- replace `Curl_read()`, `Curl_write()` and `Curl_nwrite()` to
  clarify when and at what level they operate
- send/recv of transfer related data is now done via
  `Curl_xfer_send()/Curl_xfer_recv()` which no longer has
  socket/socketindex as parameter. It decides on the transfer
  setup of `conn->sockfd` and `conn->writesockfd` on which
  connection filter chain to operate.
- send/recv on a specific connection filter chain is done via
  `Curl_conn_send()/Curl_conn_recv()` which get the socket index
  as parameter.
- rename `Curl_setup_transfer()` to `Curl_xfer_setup()` for
  naming consistency
- clarify that the special CURLE_AGAIN hangling to return
  `CURLE_OK` with length 0 only applies to `Curl_xfer_send()`
  and CURLE_AGAIN is returned by all other send() variants.
- fix a bug in websocket `curl_ws_recv()` that mixed up data
  when it arrived in more than a single chunk

The method for sending not just raw bytes, but bytes that are either
"headers" or "body". The send abstraction stack, to to bottom, now is:

* `Curl_req_send()`: has parameter to indicate amount of header bytes,
  buffers all data.
* `Curl_xfer_send()`: knows on which socket index to send, returns
  amount of bytes sent.
* `Curl_conn_send()`: called with socket index, returns amount of bytes
  sent.

In addition there is `Curl_req_flush()` for writing out all buffered
bytes.

`Curl_req_send()` is active for requests without body,
`Curl_buffer_send()` still being used for others. This is because the
special quirks need to be addressed in future parts:

* `expect-100` handling
* `Curl_fillreadbuffer()` needs to add directly to the new
  `data->req.sendbuf`
* special body handlings, like `chunked` encodings and line end
  conversions will be moved into something like a Client Reader.

In functions of the pattern `CURLcode xxx_send(..., ssize_t *written)`,
replace the `ssize_t` with a `size_t`. It makes no sense to allow for negative
values as the returned `CURLcode` already specifies error conditions. This
allows easier handling of lengths without casting.

Closes #12964
2024-02-27 14:13:56 +01:00
Stefan Eissing
5929822114
lib: send rework
Curl_read/Curl_write clarifications

- replace `Curl_read()`, `Curl_write()` and `Curl_nwrite()` to 1clarify
  when and at what level they operate

- send/recv of transfer related data is now done via
  `Curl_xfer_send()/Curl_xfer_recv()` which no longer has
  socket/socketindex as parameter. It decides on the transfer setup of
  `conn->sockfd` and `conn->writesockfd` on which connection filter
  chain to operate.

- send/recv on a specific connection filter chain is done via
  `Curl_conn_send()/Curl_conn_recv()` which get the socket index as
  parameter.

- rename `Curl_setup_transfer()` to `Curl_xfer_setup()` for naming
  consistency

- clarify that the special CURLE_AGAIN handling to return `CURLE_OK`
  with length 0 only applies to `Curl_xfer_send()` and CURLE_AGAIN is
  returned by all other send() variants.

SingleRequest reshuffling

- move functions into request.[ch]
- differentiate between reset and free
- add Curl_req_done() to perform last actions
- add a send `bufq` to SingleRequest for future use in keeping upload data

Closes #12963
2024-02-27 08:58:10 +01:00
Stefan Eissing
2abfa3833b
http: move headers collecting to writer
- add a client writer that does "push" response
  headers written to the client if the headers api
  is enabled
- remove special handling in sendf.c
- needs to be installed very early on connection
  setup to catch CONNECT response headers

Closes #12880
2024-02-26 09:44:15 +01:00
MAntoniak
e26c362544
lib: remove curl_mimepart object when CURL_DISABLE_MIME
Remove curl_mimepart object from UserDefined structure when
CURL_DISABLE_MIME flag is active. Reduce size of UserDefined structure.

Also remove unreachable code: when CURL_DISABLE_MIME is set, httpreq can
never have HTTPREQ_POST_MIME value and the same goes for the
CURL_DISABLE_FORM_API flag and the HTTPREQ_POST_FORM value

Closes #12948
2024-02-26 09:22:34 +01:00
Stefan Eissing
476adfeac0
multi: add xfer_buf to multi handle
- can be borrowed by transfer during recv-write operation
- needs to be released before borrowing again
- adjustis size to `data->set.buffer_size`
- used in transfer.c readwrite_data()

Closes #12805
2024-02-09 09:43:50 +01:00
Jay Wu
c5801a28c5
url: don't set default CA paths for Secure Transport backend
As the default for this backend is the native CA store.

Closes #12704
2024-01-16 10:48:36 +01:00
Stefan Eissing
d7b6ce64ce
lib: replace readwrite with write_resp
This clarifies the handling of server responses by folding the code for
the complicated protocols into their protocol handlers. This concerns
mainly HTTP and its bastard sibling RTSP.

The terms "read" and "write" are often used without clear context if
they refer to the connect or the client/application side of a
transfer. This PR uses "read/write" for operations on the client side
and "send/receive" for the connection, e.g. server side. If this is
considered useful, we can revisit renaming of further methods in another
PR.

Curl's protocol handler `readwrite()` method been changed:

```diff
-  CURLcode (*readwrite)(struct Curl_easy *data, struct connectdata *conn,
-                        const char *buf, size_t blen,
-                        size_t *pconsumed, bool *readmore);
+  CURLcode (*write_resp)(struct Curl_easy *data, const char *buf, size_t blen,
+                         bool is_eos, bool *done);
```

The name was changed to clarify that this writes reponse data to the
client side. The parameter changes are:

* `conn` removed as it always operates on `data->conn`
* `pconsumed` removed as the method needs to handle all data on success
* `readmore` removed as no longer necessary
* `is_eos` as indicator that this is the last call for the transfer
  response (end-of-stream).
* `done` TRUE on return iff the transfer response is to be treated as
  finished

This change affects many files only because of updated comments in
handlers that provide no implementation. The real change is that the
HTTP protocol handlers now provide an implementation.

The HTTP protocol handlers `write_resp()` implementation will get passed
**all** raw data of a server response for the transfer. The HTTP/1.x
formatted status and headers, as well as the undecoded response
body. `Curl_http_write_resp_hds()` is used internally to parse the
response headers and pass them on. This method is public as the RTSP
protocol handler also uses it.

HTTP/1.1 "chunked" transport encoding is now part of the general
*content encoding* writer stack, just like other encodings. A new flag
`CLIENTWRITE_EOS` was added for the last client write. This allows
writers to verify that they are in a valid end state. The chunked
decoder will check if it indeed has seen the last chunk.

The general response handling in `transfer.c:466` happens in function
`readwrite_data()`. This mainly operates now like:

```
static CURLcode readwrite_data(data, ...)
{
  do {
    Curl_xfer_recv_resp(data, buf)
    ...
    Curl_xfer_write_resp(data, buf)
    ...
  } while(interested);
  ...
}
```

All the response data handling is implemented in
`Curl_xfer_write_resp()`. It calls the protocol handler's `write_resp()`
implementation if available, or does the default behaviour.

All raw response data needs to pass through this function. Which also
means that anyone in possession of such data may call
`Curl_xfer_write_resp()`.

Closes #12480
2024-01-13 17:23:42 +01:00
Stefan Eissing
d0cb2c7a1f
url: init conn->sockfd and writesockfd to CURL_SOCKET_BAD
Also add more tracing to test 19

Follow-up to a0f9480

Fixes #12657
Closes #12659
2024-01-08 16:51:13 +01:00
Daniel Stenberg
907eea0804
Revert "urldata: move async resolver state from easy handle to connectdata"
This reverts commit 56a4db2e4e (#12198)

We want the c-ares channel to be held in the easy handle, not per
connection - for performance.

Closes #12524
2023-12-15 12:57:35 +01:00
Daniel Stenberg
45cf4755e7
url: for disabled protocols, mention if found in redirect
To help users better understand where the URL (and denied scheme) comes
from. Also removed "in libcurl" from the message, since the disabling
can be done by the application.

The error message now says "not supported" or "disabled" depending on
why it was denied:

 Protocol "hej" not supported
 Protocol "http" disabled

And in redirects:

 Protocol "hej" not supported (in redirect)
 Protocol "http" disabled (in redirect)

Reported-by: Mauricio Scheffer
Fixes #12465
Closes #12469
2023-12-06 23:05:21 +01:00
Stefan Eissing
5b65e7d1ae
transfer: cleanup done+excess handling
- add `SingleRequest->download_done` as indicator that
  all download bytes have been received
- remove `stop_reading` bool from readwrite functions
- move excess body handling into client download writer

Closes #12371
2023-11-24 13:22:07 +01:00
Viktor Szakats
e9a7d4a1c8
windows: use built-in _WIN32 macro to detect Windows
Windows compilers define `_WIN32` automatically. Windows SDK headers
or build env defines `WIN32`, or we have to take care of it. The
agreement seems to be that `_WIN32` is the preferred practice here.
Make the source code rely on that to detect we're building for Windows.

Public `curl.h` was using `WIN32`, `__WIN32__` and `CURL_WIN32` for
Windows detection, next to the official `_WIN32`. After this patch it
only uses `_WIN32` for this. Also, make it stop defining `CURL_WIN32`.

There is a slight chance these break compatibility with Windows
compilers that fail to define `_WIN32`. I'm not aware of any obsolete
or modern compiler affected, but in case there is one, one possible
solution is to define this macro manually.

grepping for `WIN32` remains useful to discover Windows-specific code.

Also:

- extend `checksrc` to ensure we're not using `WIN32` anymore.

- apply minor formatting here and there.

- delete unnecessary checks for `!MSDOS` when `_WIN32` is present.

Co-authored-by: Jay Satiro
Reviewed-by: Daniel Stenberg

Closes #12376
2023-11-22 15:42:25 +00:00
Stefan Eissing
1c37d472a0
url: ConnectionExists revisited
- have common pattern of `if not match, continue`
- revert pages long if()s to return early
- move dead connection check to later since it may
  be relatively expensive
- check multiuse also when NOT building with NGHTTP2
- for MULTIUSE bundles, verify that the inspected
  connection indeed supports multiplexing when in use
  (bundles may contain a mix of connection, afaict)

Closes #12373
2023-11-22 15:10:58 +01:00
Daniel Stenberg
b2d8f3f7f9
url: find scheme with a "perfect hash"
Instead of a loop to scan over the potentially 30+ scheme names, this
uses a "perfect hash" table. This works fine because the set of schemes
is known and cannot change in a build. The hash algorithm and table size
is made to only make a single scheme index per table entry.

The perfect hash is generated by a separate tool (scripts/schemetable.c)

Closes #12347
2023-11-19 13:59:24 +01:00
Viktor Szakats
a5daac6009
url: fix -Wzero-length-array with no protocols
Fixes:
```
./lib/url.c:178:56: warning: use of an empty initializer is a C2x extension [-Wc2x-extensions]
  178 | static const struct Curl_handler * const protocols[] = {
      |                                                        ^
./lib/url.c:178:56: warning: zero size arrays are an extension [-Wzero-length-array]
```

Closes #12344
2023-11-17 11:26:40 +00:00
Viktor Szakats
47c00fce4e
url: fix builds with CURL_DISABLE_HTTP
Fixes:
```
./lib/url.c:456:35: error: no member named 'formp' in 'struct UrlState'
  456 |   Curl_mime_cleanpart(data->state.formp);
      |                       ~~~~~~~~~~~ ^
```

Regression from 74b87a8af1 #11682

Closes #12343
2023-11-17 11:26:40 +00:00
Daniel Stenberg
70e3b534d5
urldata: move cookielist from UserDefined to UrlState
1. Because the value is not strictly set with a setopt option.

2. Because otherwise when duping a handle when all the set.* fields are
   first copied and an error happens (think out of memory mid-function),
   the function would easily free the list *before* it was deep-copied,
   which could lead to a double-free.

Closes #12323
2023-11-15 09:42:30 +01:00
Daniel Stenberg
289b486ffa
urldata: move hstslist from 'set' to 'state'
To make it work properly with curl_easy_duphandle(). This, because
duphandle duplicates the entire 'UserDefined' struct by plain copy while
'hstslist' is a linked curl_list of file names. This would lead to a
double-free when the second of the two involved easy handles were
closed.

Closes #12315
2023-11-13 15:36:24 +01:00
Stefan Eissing
3e6254f819
url: proxy ssl connection reuse fix
- tunnel https proxy used for http: transfers does
  no check if proxy-ssl configuration matches
- test cases added, test_10_12 fails on 8.4.0

Closes #12255
2023-11-03 16:43:45 +01:00
Stefan Eissing
bbdbd4b025
vtls: late clone of connection ssl config
- perform connection cache matching against `data->set.ssl.primary`
  and proxy counterpart
- fully clone connection ssl config only when connection is used

Closes #12237
2023-11-02 14:30:42 +01:00
Stefan Eissing
bf0e278a3c
vtls: cleanup SSL config management
- remove `Curl_ssl_get_config()`, no longer needed

Closes #12204
2023-10-31 14:29:36 +01:00
Stefan Eissing
39547ae64d
url: protocol handler lookup tidy-up
- rename lookup to what it does
- use ARRAYSIZE instead of NULL check for end
- offer alternate lookup for 0-terminated strings

Closes #12216
2023-10-27 16:55:54 +02:00
Stefan Eissing
56a4db2e4e
urldata: move async resolver state from easy handle to connectdata
- resolving is done for a connection, not for every transfer
- save create/dup/free of a cares channel for each transfer
- check values of setopt calls against a local channel if no
  connection has been attached yet, when needed.

Closes #12198
2023-10-26 17:29:17 +02:00
Daniel Stenberg
fc077bc786
urldata: move the 'internal' boolean to the state struct
... where all the other state bits for the easy handles live.

Closes #12165
2023-10-20 16:59:52 +02:00
Daniel Stenberg
b0bee93dfe
url: don't touch the multi handle when closing internal handles
Reported-by: Maksymilian Arciemowicz
Closes #12165
2023-10-20 16:59:43 +02:00
Daniel Stenberg
43215842f2
url: fall back to http/https proxy env-variable if ws/wss not set
Reported-by: Craig Andrews
Fixes #12031
Closes #12058
2023-10-08 11:29:10 +02:00
Jay Satiro
cf577bca84 url: fix typo 2023-10-07 18:43:14 -04:00
Stefan Eissing
0bd9e137e3
lib: move handling of data->req.writer_stack into Curl_client_write()
- move definitions from content_encoding.h to sendf.h
- move create/cleanup/add code into sendf.c
- installed content_encoding writers will always be called
  on Curl_client_write(CLIENTWRITE_BODY)
- Curl_client_cleanup() frees writers and tempbuffers from
  paused transfers, irregardless of protocol

Closes #11908
2023-09-28 10:00:13 +02:00
Jay Satiro
ab18c04218 url: fix netrc info message
- Fix netrc info message to use the generic ".netrc" filename if the
  user did not specify a netrc location.

- Update --netrc doc to add that recent versions of curl on Windows
  prefer .netrc over _netrc.

Before:
* Couldn't find host google.com in the (nil) file; using defaults

After:
* Couldn't find host google.com in the .netrc file; using defaults

Closes https://github.com/curl/curl/pull/11904
2023-09-24 03:37:13 -04:00
vvb2060
746dbc1488
lib: failf/infof compiler warnings
Closes #11874
2023-09-18 09:56:25 +02:00
Wyatt O'Day
e92edfbef6
lib: add ability to disable auths individually
Both with configure and cmake

Closes #11490
2023-09-07 17:45:06 +02:00
John Bampton
7378f9cc32
misc: remove duplicate words
Closes #11740
2023-08-29 10:39:31 +02:00
Daniel Stenberg
5e2beb3395
spelling: use 'reuse' not 're-use' in code and elsewhere
Unify the spelling as both versions were previously used intermittently

Closes #11717
2023-08-23 23:22:36 +02:00
Daniel Stenberg
74b87a8af1
lib: move mimepost data from ->req.p.http to ->state
When the legacy CURLOPT_HTTPPOST option is used, it gets converted into
the modem mimpost struct at first use. This data is (now) kept for the
entire transfer and not only per single HTTP request. This re-enables
rewind in the beginning of the second request instead of in end of the
first, as brought by 1b39731.

The request struct is per-request data only.

Extend test 650 to verify.

Fixes #11680
Reported-by: yushicheng7788 on github
Closes #11682
2023-08-17 23:49:57 +02:00
Daniel Stenberg
e67718eef7
lib: --disable-bindlocal builds curl without local binding support 2023-08-17 17:17:59 +02:00
Stefan Eissing
e12b39e133
trace: make tracing available in non-debug builds
Add --trace-config to curl

Add curl_global_trace() to libcurl

Closes #11421
2023-08-03 17:32:25 +02:00
Daniel Stenberg
d135d040df
url: change default value for CURLOPT_MAXREDIRS to 30
It was previously unlimited by default, but that's not a sensible
default. While changing this has a remote risk of breaking an existing
use case, I figure it is more likely to actually save users from loops.

Closes #11581
2023-08-02 23:59:59 +02:00
Daniel Stenberg
d4618a3fc4
http2: avoid too early connection re-use/multiplexing
HTTP/1 connections that are upgraded to HTTP/2 should not be picked up
for reuse and multiplexing by other handles until the 101 switching
process is completed.

Lots-of-debgging-by: Stefan Eissing
Reported-by: Richard W.M. Jones
Bug: https://curl.se/mail/lib-2023-07/0045.html
Closes #11557
2023-08-01 11:30:07 +02:00
Daniel Stenberg
5413d24aae
url: remove infof() output for "still name resolving"
The message does not help and might get spewed a lot during times.

Reported-by: yushicheng7788 on github
Fixes #11394
Closes #11529
2023-07-27 23:55:21 +02:00
Daniel Stenberg
dd4d1a2695
transfer: clear credentials when redirecting to absolute URL
Make sure the user and password for the second request is taken from the
redirected-to URL.

Add test case 899 to verify.

Reported-by: James Lucas
Fixes #11410
Closes #11412
2023-07-09 11:12:13 +02:00
Stefan Eissing
873b958d0b
http2: h2 and h2-PROXY connection alive check fixes
- fix HTTP/2 check to not declare a connection dead when
  the read attempt results in EAGAIN
- add H2-PROXY alive check as for HTTP/2 that was missing
  and is needed
- add attach/detach around Curl_conn_is_alive() and remove
  these in filter methods
- add checks for number of connections used in some test_10
  proxy tunneling tests

Closes #11368
2023-06-22 17:32:15 +02:00
Jay Satiro
fd306e55a0 lib: fix some format specifiers
- Use CURL_FORMAT_CURL_OFF_T where %zd was erroneously used for some
  curl_off_t variables.

- Use %zu where %zd was erroneously used for some size_t variables.

Prior to this change some of the Windows CI tests were failing because
in Windows 32-bit targets have a 32-bit size_t and a 64-bit curl_off_t.
When %zd was used for some curl_off_t variables then only the lower
32-bits was read and the upper 32-bits would be read for part or all of
the next specifier.

Fixes https://github.com/curl/curl/issues/11327
Closes https://github.com/curl/curl/pull/11321
2023-06-17 01:38:04 -04:00
Stefan Eissing
e024d5665d
lib: add CURLINFO_CONN_ID and CURLINFO_XFER_ID
- add an `id` long to Curl_easy, -1 on init
- once added to a multi (or its own multi), it gets
  a non-negative number assigned by the connection cache
- `id` is unique among all transfers using the same
  cache until reaching LONG_MAX where it will wrap
  around. So, not unique eternally.
- CURLINFO_CONN_ID returns the connection id attached to
  data or, if none present, data->state.lastconnect_id
- variables and type declared in tool for write out

Closes #11185
2023-06-12 23:53:00 +02:00
Daniel Stenberg
8e85764b7b
lib: remove unused functions, make single-use static
Closes #11174
2023-05-23 11:08:51 +02:00
Emanuele Torre
f198d33e8d
checksrc: disallow spaces before labels
Out of 415 labels throughout the code base, 86 of those labels were
not at the start of the line. Which means labels always at the start of
the line is the favoured style overall with 329 instances.

Out of the 86 labels not at the start of the line:
* 75 were indented with the same indentation level of the following line
* 8 were indented with exactly one space
* 2 were indented with one fewer indentation level then the following
  line
* 1 was indented with the indentation level of the following line minus
  three space (probably unintentional)

Co-Authored-By: Viktor Szakats

Closes #11134
2023-05-18 20:45:04 +02:00
Daniel Stenberg
f1e5d8a731
url: provide better error message when URLs fail to parse
By providing the URL API error message into the error message.

Ref: #11129
Closes #11137
2023-05-18 20:42:17 +02:00
Shohei Maeda
ac5ad52142
url: fix null dispname for --connect-to option
Closes #11106
2023-05-12 17:52:07 +02:00
Daniel Stenberg
21e7e44fb9
url: fix PVS nits
- expression 'hostptr' is always true
- a part of conditional expression is always true: proxypasswd
- expression 'proxyuser' is always true
- avoid multiple Curl_now() calls in allocate_conn

Ref: #10929
Closes #10959
2023-04-14 09:29:33 +02:00
Daniel Stenberg
4cfa5bcc9a
urlapi: cleanups
- move host checks together
- simplify the scheme parser loop and the end of host name parser
- avoid itermediate buffer storing in multiple places
- reduce scope for several variables
- skip the Curl_dyn_tail() call for speed
- detect IPv6 earlier and skip extra checks for such hosts
- normalize directly in dynbuf instead of itermediate buffer
- split out the IPv6 parser into its own funciton
- call the IPv6 parser directly for ipv6 addresses
- remove (unused) special treatment of % in host names
- junkscan() once in the beginning instead of scattered
- make junkscan return error code
- remove unused query management from dedotdotify()
- make Curl_parse_login_details use memchr
- more use of memchr() instead of strchr() and less strlen() calls
- make junkscan check and return the URL length

An optimized build runs one of my benchmark URL parsing programs ~41%
faster using this branch. (compared against the shipped 7.88.1 library
in Debian)

Closes #10935
2023-04-13 08:41:40 +02:00
Daniel Stenberg
712e5f1e7f
CURLPROXY_HTTPS2: for HTTPS proxy that may speak HTTP/2
Setting this proxy type allows curl to negotiate and use HTTP/2 with
HTTPS proxies.

Closes #10900
2023-04-08 00:27:53 +02:00
Daniel Stenberg
5fec927374
url: remove call to Curl_llist_destroy in Curl_close
A list that is created with a NULL "destructor" does not need to be
destroyed. Not calling it is faster than calling it.

Closes #10846
2023-03-27 23:36:16 +02:00
Daniel Stenberg
843b3baa3e
multi: remove PENDING + MSGSENT handles from the main linked list
As they are not driving transfers or any socket activity, the main loop
does not need to iterate over these handles. A performance improvement.

They are instead only held in their own separate lists.

'data->multi' is kept a pointer to the multi handle as long as the easy
handle is actually part of it even when the handle is moved to the
pending/msgsent lists. It needs to know which multi handle it belongs
to, if for example curl_easy_cleanup() is called before the handle is
removed from the multi handle.

Alll 'data->multi' pointers of handles still part of the multi handle
gets cleared by curl_multi_cleanup() which "orphans" all previously
attached easy handles.

This is take 2. The first version was reverted for the 8.0.1 release.

Assisted-by: Stefan Eissing
Closes #10801
2023-03-26 17:43:28 +02:00
Daniel Stenberg
e5e8815532
lib: skip Curl_llist_destroy calls
Linked lists themselves do not carry any allocations, so for the lists
that do not have have a set destructor we can just skip the
Curl_llist_destroy() call and save CPU time.

Closes #10764
2023-03-15 00:10:21 +01:00
Daniel Stenberg
8f4608468b
ftp: add more conditions for connection reuse
Reported-by: Harry Sintonen
Closes #10730
2023-03-13 09:07:23 +01:00
Daniel Stenberg
cb49e67303
url: only reuse connections with same GSS delegation
Reported-by: Harry Sintonen
Closes #10731
2023-03-10 23:30:34 +01:00
Daniel Stenberg
f384d40718
url: remove dummy protocol handler
Just two added checks were needed saves a whole handler struct.

Closes #10727
2023-03-10 23:19:20 +01:00
Daniel Stenberg
af369db4d3
url: fix the SSH connection reuse check
Reported-by: Harry Sintonen
Closes #10735
2023-03-10 17:54:32 +01:00
Jay Satiro
b559ef6f36 url: fix cookielist memleak when curl_easy_reset
- Free set.cookelist in Curl_freeset instead of Curl_close.

Prior to this change the cookielist linked list wasn't freed by
curl_easy_reset which calls Curl_freeset to free all set.

Bug: https://github.com/curl/curl/issues/10694#issuecomment-1458619157
Reported-by: Sergey Ryabinin

Closes https://github.com/curl/curl/pull/10709
2023-03-10 03:49:16 -05:00
Stefan Eissing
7c5637b8b4
url: fix logic in connection reuse to deny reuse on "unclean" connections
- add parameter to `conn_is_alive()` cfilter method that returns
  if there is input data waiting on the connection
- refrain from re-using connnection from the cache that have
  input pending
- adapt http/2 and http/3 alive checks to digest pending input
  to check the connection state
- remove check_cxn method from openssl as that was just doing
  what the socket filter now does.
- add tests for connection reuse with special server configs

Closes #10690
2023-03-07 12:50:31 +01:00
Stefan Eissing
821f6e2a89
CURLOPT_PIPEWAIT: allow waited reuse also for subsequent connections
As tested in test_02_07, when firing off 200 urls with --parallel, 199
wait for the first connection to be established. if that is multiuse,
urls are added up to its capacity.

The first url over capacity opens another connection. But subsequent
urls found the same situation and open a connection too. They should
have waited for the second connection to actually connect and make its
capacity known.

This change fixes that by

- setting `connkeep()` early in the HTTP setup handler. as otherwise
  a new connection is marked as closeit by default and not considered
  for multiuse at all
- checking the "connected" status for a candidate always and continuing
  to PIPEWAIT if no alternative is found.

pytest:
- removed "skip" from test_02_07
- added test_02_07b to check that http/1.1 continues to work as before

Closes #10456
2023-02-21 11:12:48 +01:00
Stefan Eissing
e187d69e48
urldata: remove now from struct SingleRequest - not needed
Closes #10549
2023-02-17 23:23:33 +01:00
Pronyushkin Petr
690c43b3ee
url: fix part of conditional expression is always true
Closes #10407
2023-02-06 08:51:31 +01:00
Stefan Eissing
5651a36d1a
cf-socket: improvements in socket I/O handling
- Curl_write_plain/Curl_read_plain have been eliminated. Last code use
  now uses Curl_conn_send/recv so that requests use conn->send/revc
  callbacks which defaults to cfilters use.
- Curl_recv_plain/Curl_send_plain have been internalized in cf-socket.c.
- USE_RECV_BEFORE_SEND_WORKAROUND (active on Windows) has been moved
  into cf-socket.c. The pre_recv buffer is held at the socket filter
  context.  `postponed_data` structures have been removed from
  `connectdata`.
- the hanger in HTTP/2 request handling was a result of read buffering
  on all sends and the multi handling is not prepared for this. The
  following happens:

   - multi preforms on a HTTP/2 easy handle
   - h2 reads and processes data
   - this leads to a send of h2 data
   - which receives and buffers before the send
   - h2 returns
   - multi selects on the socket, but no data arrives (its in the buffer already)
   the workaround now receives data in a loop as long as there is something in
   the buffer. The real fix would be for multi to change, so that `data_pending`
   is evaluated before deciding to wait on the socket.

io_buffer, optional, in cf-socket.c, http/2 sets state.drain if lower
filter have pending data.

This io_buffer is only available/used when the
-DUSE_RECV_BEFORE_SEND_WORKAROUND is active, e.g. on Windows
configurations. It also maintains the original checks on protocol
handler being HTTP and conn->send/recv not being replaced.

The HTTP/2 (nghttp2) cfilter now sets data->state.drain when it finds
out that the "lower" filter chain has still pending data at the end of
its IO operation. This prevents the processing from becoming stalled.

Closes #10280
2023-01-31 11:23:59 +01:00
Stefan Eissing
f8da4f2f2d vtls: fix hostname handling in filters
- Copy the hostname and dispname to ssl_connect_data.

Use a copy instead of referencing the `connectdata` instance since this
may get free'ed on connection reuse.

Reported-by: Stefan Talpalaru
Reported-by: sergio-nsk@users.noreply.github.com

Fixes https://github.com/curl/curl/issues/10273
Fixes https://github.com/curl/curl/issues/10309

Closes https://github.com/curl/curl/pull/10310
2023-01-20 00:40:18 -05:00
Daniel Stenberg
7ad8a7ba9e
noproxy: support for space-separated names is deprecated
To be removed in July 2024.

Assisted-by: Michael Osipov
Fixes #10209
Closes #10215
2023-01-04 09:15:37 +01:00
Andrei Rybak
bb393e521f
lib: fix typos in comments which repeat a word
Remove erroneously duplicated words in code comments of files
`lib.connect.c` and `lib/url.c`.

Closes #10220
2023-01-04 09:12:58 +01:00
Stefan Eissing
6a8d7ef981
cf-socket: keep sockaddr local in the socket filters
- copy `struct Curl_addrinfo` on filter setup into context
- remove `struct Curl_addrinfoi *` with `struct Curl_sockaddr_ex *` in
  connectdata that is set and NULLed by the socket filter
- this means we have no reference to the resolver info in connectdata or
  its filters
- trigger the CF_CTRL_CONN_INFO_UPDATE event when the complete filter
  chain reaches connected status
- update easy handle connection information on CF_CTRL_DATA_SETUP event.

Closes #10213
2023-01-03 17:33:29 +01:00
Daniel Stenberg
2bc1d775f5
copyright: update all copyright lines and remove year ranges
- they are mostly pointless in all major jurisdictions
- many big corporations and projects already don't use them
- saves us from pointless churn
- git keeps history for us
- the year range is kept in COPYING

checksrc is updated to allow non-year using copyright statements

Closes #10205
2023-01-03 09:19:21 +01:00
Daniel Stenberg
becfe2ec78
urldata: cease storing TLS auth type
The only TLS auth type libcurl ever supported is SRP and that is the
default type. Since nobody ever sets any other type, there is no point
in wasting space to store the set type and code to check the type.

If TLS auth is used, SRP is now implied.

Closes #10181
2022-12-31 16:56:21 +01:00
Stefan Eissing
71b7e01610
lib: connect/h2/h3 refactor
Refactoring of connection setup and happy eyeballing. Move
nghttp2. ngtcp2, quiche and msh3 into connection filters.

 - eyeballing cfilter that uses sub-filters for performing parallel connects
 - socket cfilter for all transport types, including QUIC
 - QUIC implementations in cfilter, can now participate in eyeballing
 - connection setup is more dynamic in order to adapt to what filter did
   really connect.  Relevant to see if a SSL filter needs to be added or
   if SSL has already been provided
 - HTTP/3 test cases similar to HTTP/2
 - multiuse of parallel transfers for HTTP/3, tested for ngtcp2 and quiche

 - Fix for data attach/detach in VTLS filters that could lead to crashes
   during parallel transfers.
 - Eliminating setup() methods in cfilters, no longer needed.
 - Improving Curl_conn_is_alive() to replace Curl_connalive() and
   integrated ssl alive checks into cfilter.
 - Adding CF_CNTRL_CONN_INFO_UPDATE to tell filters to update
   connection into and persist it at the easy handle.

 - Several more cfilter related cleanups and moves:
   - stream_weigth and dependency info is now wrapped in struct
     Curl_data_priority
   - Curl_data_priority members depend is available in HTTP2|HTTP3
   - Curl_data_priority members depend on NGHTTP2 support
   - handling init/reset/cleanup of priority part of url.c
   - data->state.priority same struct, but shallow copy for compares only

 - PROTOPT_STREAM has been removed
   - Curl_conn_is_mulitplex() now available to check on capability

 - Adding query method to connection filters.
   - ngtcp2+quiche: implementing query for max concurrent transfers.

 - Adding is_alive and keep_alive cfilter methods. Adding DATA_SETUP event.
   - setting keepalive timestamp on connect
   - DATA_SETUP is called after the connection has been completely
     setup (but may not connected yet) to allow filters to initialize
     data members they use.

 - there is no socket to be had with msh3, it is unclear how select
   shall work

 - manual test via "curl --http3 https://curl.se" fail with "empty
   reply from server".

 - Various socket/conn related cleanups:
   - Curl_socket is now Curl_socket_open and in cf-socket.c
   - Curl_closesocket is now Curl_socket_close and in cf-socket.c
   - Curl_ssl_use has been replaced with Cur_conn_is_ssl
   - Curl_conn_tcp_accepted_set has been split into
     Curl_conn_tcp_listen_set and Curl_conn_tcp_accepted_set
     with a clearer purpose

Closes #10141
2022-12-30 16:43:19 +01:00
Daniel Stenberg
076a2f6291
share: add sharing of HSTS cache among handles
Closes #10138
2022-12-27 15:22:01 +01:00
Hannah Schierling
a5cf4a87cf
url: fix build with --disable-cookies
Struct `UserDefined` has no member `cookielist` if
`CURL_DISABLE_COOKIES` is defined.

Follow-up to af5999a

Closes #10158
2022-12-26 09:27:39 +01:00
Daniel Stenberg
db07301fbb
urldata: remove unused struct fields, made more conditional
- source_quote, source_prequote and source_postquote have not been used since
  5e0d9aea3; September 2006

- make several fields conditional on proxy support

- make three quote struct fields conditional on FTP || SSH

- make 'mime_options' depend on MIME

- make trailer_* fields depend on HTTP

- change 'gssapi_delegation' from long to unsigned char

- make 'localportrange' unsigned short instead of int

- conn->trailer now depends on HTTP

Closes #10147
2022-12-23 08:45:05 +01:00
Daniel Stenberg
f4883ba966
urldata: make set.http200aliases conditional on HTTP being present
And make a few SSH-only fields depend on SSH

Closes #10140
2022-12-22 23:07:46 +01:00
Daniel Stenberg
af5999a674
urldata: move the cookefilelist to the 'set' struct
The cookiefile entries are set into the handle and should remain set for
the lifetime of the handle so that duplicating it also duplicates the
list. Therefore, the struct field is moved from 'state' to 'set'.

Fixes #10133
Closes #10134
2022-12-22 16:06:37 +01:00
Daniel Stenberg
cc0aaf6e82
idn: rename the files to idn.[ch] and hold all IDN functions
Closes #10094
2022-12-15 22:56:58 +01:00
Daniel Stenberg
301812bec2
idn: remove Curl_win32_ascii_to_idn
It was not used. Introduce a new IDN header for the prototype(s).

Closes #10094
2022-12-15 22:56:52 +01:00
Patrick Monnerat
eb559c8056
mime: relax easy/mime structures binding
Deprecation and removal of codeset conversion support from the library
have released the strict need for an early binding of mime structures to
an easy handle (https://github.com/curl/curl/commit/2610142).

This constraint currently forces to create the handle before the mime
structure and the latter cannot be attached to another handle once
created (see https://curl.se/mail/lib-2022-08/0027.html).

This commit removes the handle pointers from the mime structures
allowing more flexibility on their use.

When an easy handle is duplicated, bound mime structures must however
still be duplicated too as their components hold send-time dynamic
information.

Closes #9927
2022-11-28 09:09:54 +01:00
Stefan Eissing
f18956d0ca
cfilter: re-add conn as parameter to cfilter setup methods
- `Curl_ssl_get_config()` now returns the first config if no SSL proxy
  filter is active

- socket filter starts connection only on first invocation of its
  connect method

Fixes #9982
Closes #9983
2022-11-26 16:56:46 +01:00
Stefan Eissing
af22c2a546
vtls: localization of state data in filters
- almost all backend calls pass the Curl_cfilter intance instead of
   connectdata+sockindex
 - ssl_connect_data is remove from struct connectdata and made internal
   to vtls
 - ssl_connect_data is allocated in the added filter, kept at cf->ctx

 - added function to let a ssl filter access its ssl_primary_config and
   ssl_config_data this selects the propert subfields in conn and data,
   for filters added as plain or proxy
 - adjusted all backends to use the changed api
 - adjusted all backends to access config data via the exposed
   functions, no longer using conn or data directly

cfilter renames for clear purpose:

 - methods `Curl_conn_*(data, conn, sockindex)` work on the complete
   filter chain at `sockindex` and connection `conn`.
 - methods `Curl_cf_*(cf, ...)` work on a specific Curl_cfilter
   instance.
 - methods `Curl_conn_cf()` work on/with filter instances at a
   connection.
 - rebased and resolved some naming conflicts
 - hostname validation (und session lookup) on SECONDARY use the same
   name as on FIRST (again).

new debug macros and removing connectdata from function signatures where not
needed.

adapting schannel for new Curl_read_plain paramter.

Closes #9919
2022-11-22 14:25:50 +01:00
Daniel Stenberg
37ca6f0f9a
url: move back the IDN conversion of proxy names
Regression: in commit 53bcf55 we moved the IDN conversion calls to
happen before the HSTS checks. But the HSTS checks are only done on the
server host name, not the proxy names. By moving the proxy name IDN
conversions, we accidentally broke the verbose output showing the proxy
name.

This change moves back the IDN conversions for the proxy names to the
place in the code path they were before 53bcf55.

Reported-by: Andy Stamp
Fixes #9937
Closes #9939
2022-11-18 08:22:05 +01:00
Alexandre Ferrieux
49798cac83
CURLOPT_QUICK_EXIT: don't wait for DNS thread on exit
Fixes #2975
Closes #9147
2022-11-18 00:17:27 +01:00
Daniel Stenberg
b7e4c41aa3
url: only acknowledge fresh_reuse for non-followed transfers
... to make sure NTLM auth sticks to the connection it needs, as
verified by 2032.

Follow-up to fa0b922761

Assisted-by: Stefan Eissing
Closes #9905
2022-11-14 14:53:15 +01:00
Daniel Stenberg
bf12c2bed6
lib: remove bad set.opt_no_body assignments
This struct field MUST remain what the application set it to, so that
handle reuse and handle duplication work.

Instead, the request state bit 'no_body' is introduced for code flows
that need to change this in run-time.

Closes #9888
2022-11-11 15:56:05 +01:00