- leave transfer loop when --limit-rate is in effect and has
been received
- adjust stream window size to --limit-rate plus some slack
to make the server observe the pacing we want
- add test case to confirm behaviour
Closes#11115
The code assumes that such a resume is wanting to continue an upload
using the read callback, and since POSTFIELDS is done without callback
libcurl will just misbehave.
This combo will make the transfer fail with CURLE_BAD_FUNCTION_ARGUMENT
with an explanation in the error message.
Reported-by: Smackd0wn on github
Fixes#11081Closes#11083
- `drain` was used by http/2 and http/3 implementations to indicate
that the transfer requires send/recv independant from its socket
poll state. Intended as a counter, it was used as bool flag only.
- a similar mechanism exists on `connectdata->cselect_bits` where
specific protocols can indicate something similar, only for the
whole connection.
- `cselect_bits` are cleard in transfer.c on use and, importantly,
also set when the transfer loop expended its `maxloops` tries.
`drain` was not cleared by transfer and the http2/3 implementations
had to take care of that.
- `dselect_bits` is cleared *and* set by the transfer loop. http2/3
does no longer clear it, only set when new events happen.
This change unifies the handling of socket poll overrides, extending
`cselect_bits` by a easy handle specific value and a common treatment in
transfers.
Closes#11005
By making sure we set state.upload based on the set.method value and not
independently as set.upload, we reduce confusion and mixup risks, both
internally and externally.
Closes#11017
As they are not driving transfers or any socket activity, the main loop
does not need to iterate over these handles. A performance improvement.
They are instead only held in their own separate lists.
'data->multi' is kept a pointer to the multi handle as long as the easy
handle is actually part of it even when the handle is moved to the
pending/msgsent lists. It needs to know which multi handle it belongs
to, if for example curl_easy_cleanup() is called before the handle is
removed from the multi handle.
Alll 'data->multi' pointers of handles still part of the multi handle
gets cleared by curl_multi_cleanup() which "orphans" all previously
attached easy handles.
This is take 2. The first version was reverted for the 8.0.1 release.
Assisted-by: Stefan Eissing
Closes#10801
- use the defined, but so far not used, KEEP_SEND_HOLD bit for flow
control based suspend of sending in transfers.
Prior to this change KEEP_SEND_PAUSE bit was used instead, but that can
interfere with pausing streams from the user side via curl_easy_pause.
Fixes https://github.com/curl/curl/issues/10751
Closes https://github.com/curl/curl/pull/10753
The feature is rarely used so this frees up data for the vast majority
of easy handles that don't use it.
Rename "protdata" to "ftpwc" since it is always an FTP wildcard struct
pointer. Made the state struct field an unsigned char to save space.
Closes#10639
- Change readwrite_upload() to call win_update_buffer_size() no more
than once a second to update SO_SNDBUF (send buffer limit).
Prior to this change during an upload readwrite_upload() could call
win_update_buffer_size() anywhere from hundreds of times per second to
an extreme test case of 100k per second (which is likely due to a bug,
see #10618). In the latter case WPA profiler showed
win_update_buffer_size was the highest capture count in
readwrite_upload. In any case the calls were excessive and unnecessary.
Ref: https://github.com/curl/curl/pull/2762
Closes https://github.com/curl/curl/pull/10611
- as reported in #10433, HTTP/2 uploads may stall when a response is
received before the upload is done. This happens when the
data->state.drain is set for such a transfer, as the special handling
in transfer.c from then on only cared about downloads.
- add continuation of uploads, if applicable, in this case.
- add pytest case test_07_12_upload_seq_large to reproduce this scenario
(although, current nghttp2 implementation is using drain less often)
Reported-by: Lucas Pardue
Fixes#10433Closes#10443
- they are mostly pointless in all major jurisdictions
- many big corporations and projects already don't use them
- saves us from pointless churn
- git keeps history for us
- the year range is kept in COPYING
checksrc is updated to allow non-year using copyright statements
Closes#10205
Refactoring of connection setup and happy eyeballing. Move
nghttp2. ngtcp2, quiche and msh3 into connection filters.
- eyeballing cfilter that uses sub-filters for performing parallel connects
- socket cfilter for all transport types, including QUIC
- QUIC implementations in cfilter, can now participate in eyeballing
- connection setup is more dynamic in order to adapt to what filter did
really connect. Relevant to see if a SSL filter needs to be added or
if SSL has already been provided
- HTTP/3 test cases similar to HTTP/2
- multiuse of parallel transfers for HTTP/3, tested for ngtcp2 and quiche
- Fix for data attach/detach in VTLS filters that could lead to crashes
during parallel transfers.
- Eliminating setup() methods in cfilters, no longer needed.
- Improving Curl_conn_is_alive() to replace Curl_connalive() and
integrated ssl alive checks into cfilter.
- Adding CF_CNTRL_CONN_INFO_UPDATE to tell filters to update
connection into and persist it at the easy handle.
- Several more cfilter related cleanups and moves:
- stream_weigth and dependency info is now wrapped in struct
Curl_data_priority
- Curl_data_priority members depend is available in HTTP2|HTTP3
- Curl_data_priority members depend on NGHTTP2 support
- handling init/reset/cleanup of priority part of url.c
- data->state.priority same struct, but shallow copy for compares only
- PROTOPT_STREAM has been removed
- Curl_conn_is_mulitplex() now available to check on capability
- Adding query method to connection filters.
- ngtcp2+quiche: implementing query for max concurrent transfers.
- Adding is_alive and keep_alive cfilter methods. Adding DATA_SETUP event.
- setting keepalive timestamp on connect
- DATA_SETUP is called after the connection has been completely
setup (but may not connected yet) to allow filters to initialize
data members they use.
- there is no socket to be had with msh3, it is unclear how select
shall work
- manual test via "curl --http3 https://curl.se" fail with "empty
reply from server".
- Various socket/conn related cleanups:
- Curl_socket is now Curl_socket_open and in cf-socket.c
- Curl_closesocket is now Curl_socket_close and in cf-socket.c
- Curl_ssl_use has been replaced with Cur_conn_is_ssl
- Curl_conn_tcp_accepted_set has been split into
Curl_conn_tcp_listen_set and Curl_conn_tcp_accepted_set
with a clearer purpose
Closes#10141
When the RECV bit is cleared because the response reading for this
transfer is complete, the read loop should be stopped. data_pending()
can otherwise still return TRUE and another read would be attempted.
Reported-by: Hide Ishikawa
Fixes#10172Closes#10174
The cookiefile entries are set into the handle and should remain set for
the lifetime of the handle so that duplicating it also duplicates the
list. Therefore, the struct field is moved from 'state' to 'set'.
Fixes#10133Closes#10134
This makes a big difference for cases when the rewind is not actually
necessary to perofm (for example HTTP response code 301 converts to GET)
and therefore the rewind can be avoided. In particular for situations
when that rewind fails, for example when reading from a pipe or similar.
Reported-by: Ali Utku Selen
Fixes#9735Closes#9958
- almost all backend calls pass the Curl_cfilter intance instead of
connectdata+sockindex
- ssl_connect_data is remove from struct connectdata and made internal
to vtls
- ssl_connect_data is allocated in the added filter, kept at cf->ctx
- added function to let a ssl filter access its ssl_primary_config and
ssl_config_data this selects the propert subfields in conn and data,
for filters added as plain or proxy
- adjusted all backends to use the changed api
- adjusted all backends to access config data via the exposed
functions, no longer using conn or data directly
cfilter renames for clear purpose:
- methods `Curl_conn_*(data, conn, sockindex)` work on the complete
filter chain at `sockindex` and connection `conn`.
- methods `Curl_cf_*(cf, ...)` work on a specific Curl_cfilter
instance.
- methods `Curl_conn_cf()` work on/with filter instances at a
connection.
- rebased and resolved some naming conflicts
- hostname validation (und session lookup) on SECONDARY use the same
name as on FIRST (again).
new debug macros and removing connectdata from function signatures where not
needed.
adapting schannel for new Curl_read_plain paramter.
Closes#9919
This struct field MUST remain what the application set it to, so that
handle reuse and handle duplication work.
Instead, the request state bit 'no_body' is introduced for code flows
that need to change this in run-time.
Closes#9888
- general construct/destroy in connectdata
- default implementations of callback functions
- connect: cfilters for connect and accept
- socks: cfilter for socks proxying
- http_proxy: cfilter for http proxy tunneling
- vtls: cfilters for primary and proxy ssl
- change in general handling of data/conn
- Curl_cfilter_setup() sets up filter chain based on data settings,
if none are installed by the protocol handler setup
- Curl_cfilter_connect() boot straps filters into `connected` status,
used by handlers and multi to reach further stages
- Curl_cfilter_is_connected() to check if a conn is connected,
e.g. all filters have done their work
- Curl_cfilter_get_select_socks() gets the sockets and READ/WRITE
indicators for multi select to work
- Curl_cfilter_data_pending() asks filters if the have incoming
data pending for recv
- Curl_cfilter_recv()/Curl_cfilter_send are the general callbacks
installed in conn->recv/conn->send for io handling
- Curl_cfilter_attach_data()/Curl_cfilter_detach_data() inform filters
and addition/removal of a `data` from their connection
- adding vtl functions to prevent use of Curl_ssl globals directly
in other parts of the code.
Reviewed-by: Daniel Stenberg
Closes#9855
- Replace `Github` with `GitHub`.
- Replace `windows` with `Windows`
- Replace `advice` with `advise` where a verb is used.
- A few fixes on removing repeated words.
- Replace `a HTTP` with `an HTTP`
Closes#9802
This also returns error CURLE_UNSUPPORTED_PROTOCOL rather than
CURLE_BAD_FUNCTION_ARGUMENT when a listed protocol name is not found.
A new schemelen parameter is added to Curl_builtin_scheme() to support
this extended use.
Note that disabled protocols are not recognized anymore.
Tests adapted accordingly.
Closes#9472
When the parser is not allowed to guess scheme, it should consider the
word ending at the first colon to be the scheme, independently of number
of slashes.
The parser now checks that the scheme is known before it counts slashes,
to improve the error messge for URLs with unknown schemes and maybe no
slashes.
When following redirects, no scheme guessing is allowed and therefore
this change effectively prevents redirects to unknown schemes such as
"data".
Fixes#9503
If not, reusing an easy handle to do a subsequent transfer would
continue the counter from the previous invoke, which then would make use
of the header API difficult/impossible as the request counter
mismatched.
Add libtest 1947 to verify.
Reported-by: Andrew Lambert
Fixes#9424Closes#9447
Append to the upload buffer when only small amount remains in buffer
rather than performing a separate tiny send to empty buffer.
Avoid degenerative upload behavior which might cause curl to send mostly
1-byte DATA frames after exhausing the h2 send window size
Related discussion: https://github.com/nghttp2/nghttp2/issues/1722
Signed-off-by: Glenn Strauss <gstrauss@gluelogic.com>
Closes#8965
Add licensing and copyright information for all files in this repository. This
either happens in the file itself as a comment header or in the file
`.reuse/dep5`.
This commit also adds a Github workflow to check pull requests and adapts
copyright.pl to the changes.
Closes#8869
Commit 0ef54abf52 accidentally used the conn variable before the
assertion for it being NULL. Fix by moving the assignment which use
conn to after the assertion.
Closes: #8857
Reviewed-by: Daniel Stenberg <daniel@haxx.se>
Add Curl_quic_idle which is called when no HTTP level read or write is
performed. It is a good place to handle timer expiry for QUIC transport
(.e.g, retransmission).
Closes#8698
Instad of having all callers pass in the maximum length, always use
it. The passed in length is instead used only as the length of the
target buffer for to storing the scheme name in, if used.
Added the scheme max length restriction to the curl_url_set.3 man page.
Follow-up to 45bcb2eaa7Closes#8047
- the data needs to be "line-based" anyway since it's also passed to the
debug callback/application
- it makes infof() work like failf() and consistency is good
- there's an assert that triggers on newlines in the format string
- Also removes a few instances of "..."
- Removes the code that would append "..." to the end of the data *iff*
it was truncated in infof()
Closes#7357
They were never officially allowed and slipped in only due to sloppy
parsing. Spaces (ascii 32) should be correctly encoded (to %20) before
being part of a URL.
The new flag bit CURLU_ALLOW_SPACE when a full URL is set, makes libcurl
allow spaces.
Updated test 1560 to verify.
Closes#7073
Check the FIRST for all other protocols.
This fixes a timeout in an ftps download. The server sends a TLS
close_notify message in the same packet as the file data. The
close_notify seems to not be handled in the schannel_recv function, so
libcurl is not aware that the server has closed the connection. Thus
libcurl ends up waiting for action on the socket until a timeout is
reached. With the secondary socket check added to the data_pending
function, the close_notify is properly handled, and the ftps transfer
terminates as expected.
Fixes#7068Closes#7069
Both were used for the same purposes and there was no logical separation
between them. Combined, this also saves 16 bytes in less holes in my
test build.
Closes#6798
HTTP auth "accidentally" worked before this cleanup since the code would
always overwrite the connection credentials with the credentials from
the most recent transfer and since HTTP auth is typically done first
thing, this has not been an issue. It was still wrong and subject to
possible race conditions or future breakage if the sequence of functions
would change.
The data.set.str[] strings MUST remain unmodified exactly as set by the
user, and the credentials to use internally are instead set/updated in
state.aptr.*
Added test 675 to verify different credentials used in two requests done
over a reused HTTP connection, which previously behaved wrongly.
Fixes#6542Closes#6545
Rename it to 'httpwant' and make a cloned field in the state struct as
well for run-time updates.
Also: refuse non-supported HTTP versions. Verified with test 129.
Closes#6585
and rename it from 'ftp_list_only' since it is also used for SSH and
POP3. The state is updated internally for 'type=D' FTP URLs.
Added test case 1570 to verify.
Closes#6578
... and make sure the code never updates 'set.prefer_ascii' as it breaks
handle reuse which should use the setting as the user specified it.
Added test 1569 to verify: it first makes an FTP transfer with ';type=A'
and then another without type on the same handle and the second should
then use binary. Previously, curl failed this.
Closes#6578
... in most cases instead of 'struct connectdata *' but in some cases in
addition to.
- We mostly operate on transfers and not connections.
- We need the transfer handle to log, store data and more. Everything in
libcurl is driven by a transfer (the CURL * in the public API).
- This work clarifies and separates the transfers from the connections
better.
- We should avoid "conn->data". Since individual connections can be used
by many transfers when multiplexing, making sure that conn->data
points to the current and correct transfer at all times is difficult
and has been notoriously error-prone over the years. The goal is to
ultimately remove the conn->data pointer for this reason.
Closes#6425
... and not in the connection setup, as for multiplexed transfers the
connection setup might be skipped and then the transfer would end up
without the set user-agent!
Reported-by: Flameborn on github
Assisted-by: Andrey Gursky
Assisted-by: Jay Satiro
Assisted-by: Mike Gelfand
Fixes#6312Closes#6417
When doing HTTP authentication and a port number set with CURLOPT_PORT,
the code would previously have the URL's port number override as if it
had been a redirect to an absolute URL.
Added test 1568 to verify.
Reported-by: UrsusArctos on github
Fixes#6397Closes#6400
We currently use both spellings the british "behaviour" and the american
"behavior". However "behavior" is more used in the project so I think
it's worth dropping the british name.
Closes#6395
When the initial request isn't possible to send in its entirety, the
remainder of request would be delivered to the debug callback as data
and would wrongly be counted internally as body-bytes sent.
Extended test 1295 to verify.
Closes#6328
This flag was applied to the connection struct that is released on
retry. These changes move the retry counter into Curl_easy struct that
lives across retries and retains the new connection.
Reported-by: Cherish98 on github
Fixes#5794Closes#5800
When the method is updated inside libcurl we must still not change the
method as set by the user as then repeated transfers with that same
handle might not execute the same operation anymore!
This fixes the libcurl part of #5462
Test 1633 added to verify.
Closes#5499
"Null-checking k->str suggests that it may be null, but it has already
been dereferenced on all paths leading to the check" - and it can't
legally be NULL at this point. Remove check.
Detected by Coverity CID 1463884
Closes#5495
... and free it as soon as the transfer is done. It removes the extra
alloc when a new size is set with setopt() and reduces memory for unused
easy handles.
In addition: the closure_handle now doesn't use an allocated buffer at
all but the smallest supported size as a stack based one.
Closes#5472
For HTTP 1.x, it's a protocol error when the server sends more bytes
than announced. If this happens, don't reuse the connection, because the
start position of the next response is undefined.
Closes#5440
A common set of functions instead of many separate implementations for
creating buffers that can grow when appending data to them. Existing
functionality has been ported over.
In my early basic testing, the total number of allocations seem at
roughly the same amount as before, possibly a few less.
See docs/DYNBUF.md for a description of the API.
Closes#5300
Prior to this change if there was a 303 reply to a PUT request then
the subsequent request to respond to that redirect would also be a PUT.
It was determined that was most likely incorrect based on the language
of the RFCs. Basically 303 means "see other" resource, which implies it
is most likely not the same resource, therefore we should not try to PUT
to that different resource.
Refer to the discussions in #5237 and #5248 for more information.
Fixes https://github.com/curl/curl/issues/5237
Closes https://github.com/curl/curl/pull/5248
Prior to this change in libcurl debug builds http2 stream closure was
erroneously referred to as connection closure.
Before:
* nread <= 0, server closed connection, bailing
After:
* nread == 0, stream closed, bailing
Closes https://github.com/curl/curl/pull/5118
When libcurl retries a connection due to it being "seemingly dead" or by
REFUSED_STREAM, it will now only do it up five times before giving up,
to avoid never-ending loops.
Reported-by: Dima Tisnek
Bug: https://curl.haxx.se/mail/lib-2020-03/0044.htmlCloses#5074
... since the socket might not actually be readable anymore when for
example the data is already buffered in the TLS layer.
Fixes#4966
Reported-by: Anders Berg
Closes#5000
- Disable warning C4127 "conditional expression is constant" globally
in curl_setup.h for when building with Microsoft's compiler.
This mainly affects building with the Visual Studio project files found
in the projects dir.
Prior to this change the cmake and winbuild build systems already
disabled 4127 globally for when building with Microsoft's compiler.
Also, 4127 was already disabled for all build systems in the limited
circumstance of the WHILE_FALSE macro which disabled the warning
specifically for while(0). This commit removes the WHILE_FALSE macro and
all other cruft in favor of disabling globally in curl_setup.
Background:
We have various macros that cause 0 or 1 to be evaluated, which would
cause warning C4127 in Visual Studio. For example this causes it:
#define Curl_resolver_asynch() 1
Full behavior is not clearly defined and inconsistent across versions.
However it is documented that since VS 2015 Update 3 Microsoft has
addressed this somewhat but not entirely, not warning on while(true) for
example.
Prior to this change some C4127 warnings occurred when I built with
Visual Studio using the generated projects in the projects dir.
Closes https://github.com/curl/curl/pull/4658
To make sure that the HTTP/2 state is initialized correctly for
duplicated handles. It would otherwise easily generate "spurious"
PRIORITY frames to get sent over HTTP/2 connections when duplicated easy
handles were used.
Reported-by: Daniel Silverstone
Fixes#4303Closes#4442
