Commit Graph

13126 Commits

Author SHA1 Message Date
Stefan Eissing
88332049ea
http/2: unstick uploads
- refs #11157 and #11175 where uploads get stuck or lead to RST streams
- fixes our h2 send behaviour to continue sending in the nghttp2 session
  as long as it wants to. This will empty our send buffer as long as
  the remote stream/connection window allows.
- in case the window is exhausted, the data remaining in the send buffer
  will wait for a WINDOW_UPDATE from the server. Which is a socket event
  that engages our transfer loop again
- the problem in the issue was that we did not exhaust the window, but
  left data in the sendbuffer and no further socket events did happen.
  The server was just waiting for us to send more.
- relatedly, there was an issue fixed that closing a stream with KEEP_HOLD
  set kept the transfer from shutting down - as it should have - leading
  to a timeout.

Closes #11176
2023-05-22 16:19:13 +02:00
Daniel Stenberg
954c7dfb91
sectransp.c: make the code c89 compatible
Follow-up to dd2bb48552

Reported-by: FeignClaims on github
Fixes #11155
Closes #11159
2023-05-21 14:02:11 +02:00
Emanuele Torre
eef076baa6
Revert "urlapi: respect CURLU_ALLOW_SPACE and CURLU_NO_AUTHORITY for redirects"
This reverts commit df6c2f7b54.
(It only keep the test case that checks redirection to an absolute URL
without hostname and CURLU_NO_AUTHORITY).

I originally wanted to make CURLU_ALLOW_SPACE accept spaces in the
hostname only because I thought
curl_url_set(CURLUPART_URL, CURLU_ALLOW_SPACE) was already accepting
them, and they were only not being accepted in the hostname when
curl_url_set(CURLUPART_URL) was used for a redirection.

That is not actually the case, urlapi never accepted hostnames with
spaces, and a hostname with a space in it never makes sense.
I probably misread the output of my original test when I they were
normally accepted when using CURLU_ALLOW_SPACE, and not redirecting.

Some other URL parsers seems to allow space in the host part of the URL,
e.g. both python3's urllib.parse module, and Chromium's javascript URL
object allow spaces (chromium percent escapes the spaces with %20),
(they also both ignore TABs, and other whitespace characters), but those
URLs with spaces in the hostname are useless, neither python3's requests
module nor Chromium's window.location can actually use them.

There is no reason to add support for URLs with spaces in the host,
since it was not a inconsistency bug; let's revert that patch before it
makes it into release. Sorry about that.

I also reverted the extra check for CURLU_NO_AUTHORITY since that does
not seem to be necessary, CURLU_NO_AUTHORITY already worked for
redirects.

Closes #11169
2023-05-21 13:59:04 +02:00
Stefan Eissing
0cab1359a1
http2: upload improvements
Make send buffer smaller to have progress and "upload done" reporting
closer to reality. Fix handling of send "drain" condition to no longer
trigger once the transfer loop reports it is done sending. Also do not
trigger the send "drain" on RST streams.

Background:
- a upload stall was reported in #11157 that timed out
- test_07_33a reproduces a problem with such a stall if the
  server 404s the request and RSTs the stream.
- test_07_33b verifies a successful PUT, using the parameters
  from #11157 and checks success

Ref: #11157
Closes #11165
2023-05-20 23:07:45 +02:00
Stefan Eissing
1886eef7fa
http2: increase stream window size to 10 MB
Reported-by: pandada8 on github

Fixes #11162
Closes #11167
2023-05-20 23:05:07 +02:00
Daniel Stenberg
54ce13d3ff
lib: rename struct 'http_req' to 'httpreq'
Because FreeBSD 14 kidnapped the name.
Ref: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=271526

Fixes #11163
Closes #11164
2023-05-20 23:03:01 +02:00
Stefan Eissing
7ab94d7d57
ngtcp2: proper handling of uint64_t when adjusting send buffer
Fixes #11149
Closes #11153
2023-05-19 16:23:19 +02:00
Stefan Eissing
e0ddfc8e05
ngtcp2: fix compiler warning about possible null-deref
- compiler analyzer did not include the call context for this
  static function where the condition had already been checked.
- eleminating the problem by making stream a call parameter

Fixes #11147
Closes #11151
2023-05-19 16:19:56 +02:00
Antoine Pitrou
d65321f939
select: avoid returning an error on EINTR from select() or poll()
This was already done for the poll() and select() calls
made directly from Curl_poll(), but was missed in
Curl_wait_ms(), which is called when there are no fds
to wait on.

Fixes #11135
Closes #11143
2023-05-19 16:16:26 +02:00
Daniel Stenberg
5b4bcc6ede
vquic.c: make recvfrom_packets static, avoid compiler warning
warning: no previous prototype for 'recvfrom_packets'

Reported-by: Keitagit-kun on github
Fixes #11146
Closes #11148
2023-05-19 16:15:22 +02:00
Daniel Stenberg
92772e6d39
urlapi: allow numerical parts in the host name
It can only be an IPv4 address if all parts are all digits and no more than
four parts, otherwise it is a host name. Even slightly wrong IPv4 will now be
passed through as a host name.

Regression from 17a15d8846 shipped in 8.1.0

Extended test 1560 accordingly.

Reported-by: Pavel Kalyugin
Fixes #11129
Closes #11131
2023-05-19 16:01:26 +02:00
Emilio Cobos Álvarez
77c9a9845b
http2: double http request parser max line length
This works around #11138, by doubling the limit, and should be a
relatively safe fix.

Ideally the buffer would grow as needed and there would be no need for a
limit? But that might be follow-up material.

Fixes #11138
Closes #11139
2023-05-19 01:09:36 +02:00
Deal(一线灵)
446061e6cb
cmake: repair cross compiling
It cannot *run* code for testing purposes when cross-compiling.

Closes #11130
2023-05-18 21:13:12 +02:00
Stefan Eissing
408eb87bb3
cf-socket: completely remove the disabled USE_RECV_BEFORE_SEND_WORKAROUND
Closes #11118
2023-05-18 20:55:16 +02:00
Emanuele Torre
df6c2f7b54
urlapi: respect CURLU_ALLOW_SPACE and CURLU_NO_AUTHORITY for redirects
curl_url_set(uh, CURLUPART_URL, redirurl, flags)  was not respecing
CURLU_ALLOW_SPACE and CURLU_NO_AUTHORITY in the host part of redirurl
when redirecting to an absolute URL.

Closes #11136
2023-05-18 20:52:59 +02:00
Colin Cross
e743425bc6
hostip: move easy_lock.h include above curl_memory.h
Similar to #9561, move easy_lock.h above curl_memory.h to fix building
against musl libc.

Closes #11140
2023-05-18 20:50:22 +02:00
Emanuele Torre
f198d33e8d
checksrc: disallow spaces before labels
Out of 415 labels throughout the code base, 86 of those labels were
not at the start of the line. Which means labels always at the start of
the line is the favoured style overall with 329 instances.

Out of the 86 labels not at the start of the line:
* 75 were indented with the same indentation level of the following line
* 8 were indented with exactly one space
* 2 were indented with one fewer indentation level then the following
  line
* 1 was indented with the indentation level of the following line minus
  three space (probably unintentional)

Co-Authored-By: Viktor Szakats

Closes #11134
2023-05-18 20:45:04 +02:00
Daniel Stenberg
d334b02924
cookie: update the comment on cookie length and size limits
To refer to the proper cookie RFC and the upcoming RFC refresh.

Closes #11127
2023-05-18 20:43:25 +02:00
Daniel Stenberg
f1e5d8a731
url: provide better error message when URLs fail to parse
By providing the URL API error message into the error message.

Ref: #11129
Closes #11137
2023-05-18 20:42:17 +02:00
Daniel Stenberg
f446258f02
hostip: include easy_lock.h before using GLOBAL_INIT_IS_THREADSAFE
Since that header file is the only place that define can be defined.

Reported-by: Marc Deslauriers

Follow-up to 13718030ad

Closes #11121
2023-05-17 08:03:10 +02:00
Daniel Silverstone
e915b6904c
ssh: Add support for libssh2 read timeout
Hook the new (1.11.0 or newer) libssh2 support for setting a read timeout
into the SERVER_RESPONSE_TIMEOUT option.  With this done, clients can use
the standard curl response timeout setting to also control the time that
libssh2 will wait for packets from a slow server.  This is necessary to
enable use of very slow SFTP servers.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

Closes #10965
2023-05-15 13:20:47 +02:00
Marcel Raad
2a154560c0
md(4|5): don't use deprecated iOS functions
They are marked as deprecated in iOS 13.0, which might result in
warnings-as-errors.

Also, use `*_MIN_REQUIRED` instead of `*_MIN_ALLOWED`, which seems to
be what's currently used.

Bug: https://github.com/curl/curl/issues/11098
Closes https://github.com/curl/curl/pull/11102
2023-05-13 09:57:54 +02:00
Marcel Raad
e587598534
md4: only build when used
Its only usage in curl_ntlm_core.c is guarded by `USE_CURL_NTLM_CORE`,
so let's use this here too.

Ref: https://github.com/curl/curl/issues/11098
Closes https://github.com/curl/curl/pull/11102
2023-05-13 09:57:35 +02:00
Shohei Maeda
ac5ad52142
url: fix null dispname for --connect-to option
Closes #11106
2023-05-12 17:52:07 +02:00
Daniel Stenberg
cd1c611b24
headers: clear (possibly) lingering pointer in init
The "prevhead" pointer is used for the headers storage but was not
cleared correctly in init, which made it possible to act up when a
handle is reused.

Reported-by: Steve Herrell
Fixes #11101
Closes #11103
2023-05-12 17:48:20 +02:00
Daniel Stenberg
10998e3b89
ngtcp2: use 0.15.0
- nghttp3 0.11.0
- nghttp2 1.53.0

Adapt to new API calls

Closes #11031
2023-05-12 17:25:38 +02:00
Jay Satiro
2cc1c93f45 openssl: fix indent 2023-05-10 14:03:30 -04:00
Daniel Stenberg
8e6abece40
hostip: use time_t for storing oldest DNS entry
Theoretically, the oldest time could overflow an int. In practice that
won't happen, but let's do this to please analyzers.

Follow-up to 9ed7d56e04

Pointed out by Coverity.
Closes #11094
2023-05-09 17:56:19 +02:00
Daniel Stenberg
f7170a8f2e
http: free the url before storing a new copy
To avoid a memory-leak.

Reported-by: Hiroki Kurosawa

Closes #11093
2023-05-09 09:47:30 +02:00
Daniel Stenberg
8e21b1a05f
libssh2: free fingerprint better
Reported-by: Wei Chong Tan
Closes #11088
2023-05-08 16:47:03 +02:00
Daniel Stenberg
9ed7d56e04
hostip: enforce a maximum DNS cache size independent of timeout value
To reduce the damage an application can cause if using -1 or other
ridiculous timeout values and letting the cache live long times.

The maximum number of entries in the DNS cache is now totally
arbitrarily and hard-coded set to 29999.

Closes #11084
2023-05-08 14:55:26 +02:00
Daniel Stenberg
f62557276a
hostip: store dns timeout as 'int'
... because it set and held as an 'int' elsewhere and can never be
larger.
2023-05-08 14:55:19 +02:00
Daniel Stenberg
fb7886b9c9
transfer: refuse POSTFIELDS + RESUME_FROM combo
The code assumes that such a resume is wanting to continue an upload
using the read callback, and since POSTFIELDS is done without callback
libcurl will just misbehave.

This combo will make the transfer fail with CURLE_BAD_FUNCTION_ARGUMENT
with an explanation in the error message.

Reported-by: Smackd0wn on github
Fixes #11081
Closes #11083
2023-05-08 14:10:34 +02:00
François Michel
89f6fafedb
quiche: disable pacing while pacing is not actually performed
Closes #11068
2023-05-03 13:59:17 +02:00
Daniel Stenberg
d8df0d6db7
easy_cleanup: require a "good" handle to act
By insisting that the passed in handle is "good" (the magic number is
intact), this can limit the potential damage if a bad pointer is passed
in. Like when this function is called twice on the same handle pointer.

Ref: #10964
Closes #11061
2023-05-02 13:24:30 +02:00
Andreas Falkenhahn
b8d7746e3c amiga: Fix CA certificate paths for AmiSSL and MorphOS
AmiSSL stores certificates in `AmiSSL:Certs` and MorphOS stores them in
`MOSSYS:Data/SSL/curl-ca-bundle.crt`.

Closes https://github.com/curl/curl/pull/11059
2023-05-01 14:08:55 -04:00
Daniel Stenberg
b637166092
http2: (void)-mark when we explicitly ignore the return code
When h2_progress_egress() is called. Pointed out by Coverity.

Closes #11057
2023-04-30 11:26:07 +02:00
Daniel Stenberg
9ce7eee070
checksrc: find bad indentation in conditions without open brace
If the previous line starts with if/while/for AND ends with a closed
parenthesis and there's an equal number of open and closed parentheses
on that line, verify that this line is indented $indent more steps, if
not a cpp line.

Also adjust the fall-out from this fix.

Closes #11054
2023-04-28 23:11:00 +02:00
Daniel Stenberg
da9d9c3d1c
multi: add multi-ignore logic to multi_socket_action
The multi-ignore logic that was previously applied to
curl_multi_perform() (#10750) is here applied to the loop within
curl_multi_socket_action() to make it use the same optimization: most
handles have the same signal-ignore option state so this drastically
reduces the number of ignore/unignore calls per libcurl function invoke.

Follow-up to bc90308328

Closes #11045
2023-04-28 16:27:45 +02:00
Stefan Eissing
a9b7f72bc9
http2: do flow window accounting for cancelled streams
- nghttp2 does not free connection level window flow for
  aborted streams
- when closing transfers, make sure that any buffered
  response data is "given back" to the flow control window
- add tests test_02_22 and test_02_23 to reproduce

Closes #11052
2023-04-28 13:55:39 +02:00
Stefan Eissing
b0edf0b7da
pingpong: fix compiler warning "assigning an enum to unsigned char"
Closes #11050
2023-04-28 13:38:21 +02:00
Emanuele Torre
7f712399d5
checksrc: check for spaces before the colon of switch labels
Closes #11047
2023-04-27 23:26:50 +02:00
Daniel Stenberg
ff67da58c4
libssh: tell it to use SFTP non-blocking
Reported-by: Andreas Huebner
Fixes #11020
Closes #11039
2023-04-27 17:59:56 +02:00
Stefan Eissing
de4b0164f2
http2: enlarge the connection window
- fixes stalled connections

- Make the connection window large enough, so that there is
  some room left should 99/100 streams be PAUSED by the application

Reported-by: Paweł Wegner
Fixes #10988
Closes #11043
2023-04-27 17:44:55 +02:00
Daniel Stenberg
d567cca1de
checksrc: fix SPACEBEFOREPAREN for conditions starting with "*"
The open paren check wants to warn for spaces before open parenthesis
for if/while/for but also for any function call. In order to avoid
catching function pointer declarations, the logic allows a space if the
first character after the open parenthesis is an asterisk.

I also spotted what we did not include "switch" in the check but we should.

This check is a little lame, but we reduce this problem by not allowing
that space for if/while/for/switch.

Reported-by: Emanuele Torre
Closes #11044
2023-04-27 17:24:47 +02:00
Daniel Stenberg
bb0b245cc1
ws: fix CONT opcode check
Detected by Coverity. Follow-up to 930c00c259

Closes #11037
2023-04-27 11:08:48 +02:00
Daniel Stenberg
b7b1846275
urlapi: make internal function start with Curl_
Curl_url_set_authority() it is.

Follow-up to acd82c8bfd

Closes #11035
2023-04-27 08:36:51 +02:00
YX Hao
ca3f6decb9
cf-socket: turn off IPV6_V6ONLY on Windows if it is supported
IPV6_V6ONLY refs:
https://en.wikipedia.org/wiki/IPv6#IPv4-mapped_IPv6_addresses
https://github.com/golang/go/blob/master/src/net/ipsock_posix.go
https://en.wikipedia.org/wiki/Unix-like
https://learn.microsoft.com/en-us/windows/win32/winsock/ipproto-ipv6-socket-options

default value refs:
https://datatracker.ietf.org/doc/html/rfc3493#section-5.3
https://www.kernel.org/doc/html/latest/networking/ip-sysctl.html#proc-sys-net-ipv6-variables

Closes #10975
2023-04-26 23:39:57 +02:00
Daniel Stenberg
65f03e507f
urldata: shrink *select_bits int => unsigned char
- dselect_bits
- cselect_bits

... are using less than 8 bits. Changed types and moved them towards
the end of the structs to fit better.

Closes #11025
2023-04-26 23:36:45 +02:00
Stefan Eissing
acd82c8bfd
tests/http: more tests with specific clients
- Makefile support for building test specific clients in tests/http/clients
- auto-make of clients when invoking pytest
- added test_09_02 for server PUSH_PROMISEs using clients/h2-serverpush
- added test_02_21 for lib based downloads and pausing/unpausing transfers

curl url parser:
- added internal method `curl_url_set_authority()` for setting the
  authority part of a url (used for PUSH_PROMISE)

http2:
- made logging of PUSH_PROMISE handling nicer

Placing python test requirements in requirements.txt files
- separate files to base test suite and http tests since use
  and module lists differ
- using the files in the gh workflows

websocket test cases, fixes for we and bufq
- bufq: account for spare chunks in space calculation
- bufq: reset chunks that are skipped empty
- ws: correctly encode frames with 126 bytes payload
- ws: update frame meta information on first call of collect
  callback that fills user buffer
- test client ws-data: some test/reporting improvements

Closes #11006
2023-04-26 23:24:46 +02:00