Commit Graph

445 Commits

Author SHA1 Message Date
Stefan Eissing
f4b5c88ab7
http2: better support for --limit-rate
- leave transfer loop when --limit-rate is in effect and has
  been received
- adjust stream window size to --limit-rate plus some slack
  to make the server observe the pacing we want
- add test case to confirm behaviour

Closes #11115
2023-06-02 23:06:12 +02:00
Stefan Eissing
5c58cb0212
http2: fix EOF handling on uploads with auth negotiation
- doing a POST with `--digest` does an override on the initial request
  with `Content-Length: 0`, but the http2 filter was unaware of that
  and expected the originally request body. It did therefore not
  send a final DATA frame with EOF flag to the server.
- The fix overrides any initial notion of post size when the `done_send`
  event is triggered by the transfer loop, leading to the EOF that
  is necessary.
- refs #11194. The fault did not happen in testing, as Apache httpd
  never tries to read the request body of the initial request,
  sends the 401 reply and closes the stream. The server used in the
  reported issue however tried to read the EOF and timed out on the
  request.

Reported-by: Aleksander Mazur
Fixes #11194
Cloes #11200
2023-05-25 08:26:18 +02:00
Daniel Stenberg
8e85764b7b
lib: remove unused functions, make single-use static
Closes #11174
2023-05-23 11:08:51 +02:00
Daniel Stenberg
127eb0d83a
misc: fix spelling mistakes
Reported-by: musvaage on github
Fixes #11171
Closes #11172
2023-05-23 10:42:09 +02:00
Stefan Eissing
88332049ea
http/2: unstick uploads
- refs #11157 and #11175 where uploads get stuck or lead to RST streams
- fixes our h2 send behaviour to continue sending in the nghttp2 session
  as long as it wants to. This will empty our send buffer as long as
  the remote stream/connection window allows.
- in case the window is exhausted, the data remaining in the send buffer
  will wait for a WINDOW_UPDATE from the server. Which is a socket event
  that engages our transfer loop again
- the problem in the issue was that we did not exhaust the window, but
  left data in the sendbuffer and no further socket events did happen.
  The server was just waiting for us to send more.
- relatedly, there was an issue fixed that closing a stream with KEEP_HOLD
  set kept the transfer from shutting down - as it should have - leading
  to a timeout.

Closes #11176
2023-05-22 16:19:13 +02:00
Stefan Eissing
0cab1359a1
http2: upload improvements
Make send buffer smaller to have progress and "upload done" reporting
closer to reality. Fix handling of send "drain" condition to no longer
trigger once the transfer loop reports it is done sending. Also do not
trigger the send "drain" on RST streams.

Background:
- a upload stall was reported in #11157 that timed out
- test_07_33a reproduces a problem with such a stall if the
  server 404s the request and RSTs the stream.
- test_07_33b verifies a successful PUT, using the parameters
  from #11157 and checks success

Ref: #11157
Closes #11165
2023-05-20 23:07:45 +02:00
Stefan Eissing
1886eef7fa
http2: increase stream window size to 10 MB
Reported-by: pandada8 on github

Fixes #11162
Closes #11167
2023-05-20 23:05:07 +02:00
Emilio Cobos Álvarez
77c9a9845b
http2: double http request parser max line length
This works around #11138, by doubling the limit, and should be a
relatively safe fix.

Ideally the buffer would grow as needed and there would be no need for a
limit? But that might be follow-up material.

Fixes #11138
Closes #11139
2023-05-19 01:09:36 +02:00
Emanuele Torre
f198d33e8d
checksrc: disallow spaces before labels
Out of 415 labels throughout the code base, 86 of those labels were
not at the start of the line. Which means labels always at the start of
the line is the favoured style overall with 329 instances.

Out of the 86 labels not at the start of the line:
* 75 were indented with the same indentation level of the following line
* 8 were indented with exactly one space
* 2 were indented with one fewer indentation level then the following
  line
* 1 was indented with the indentation level of the following line minus
  three space (probably unintentional)

Co-Authored-By: Viktor Szakats

Closes #11134
2023-05-18 20:45:04 +02:00
Daniel Stenberg
b637166092
http2: (void)-mark when we explicitly ignore the return code
When h2_progress_egress() is called. Pointed out by Coverity.

Closes #11057
2023-04-30 11:26:07 +02:00
Stefan Eissing
a9b7f72bc9
http2: do flow window accounting for cancelled streams
- nghttp2 does not free connection level window flow for
  aborted streams
- when closing transfers, make sure that any buffered
  response data is "given back" to the flow control window
- add tests test_02_22 and test_02_23 to reproduce

Closes #11052
2023-04-28 13:55:39 +02:00
Stefan Eissing
de4b0164f2
http2: enlarge the connection window
- fixes stalled connections

- Make the connection window large enough, so that there is
  some room left should 99/100 streams be PAUSED by the application

Reported-by: Paweł Wegner
Fixes #10988
Closes #11043
2023-04-27 17:44:55 +02:00
Daniel Stenberg
b7b1846275
urlapi: make internal function start with Curl_
Curl_url_set_authority() it is.

Follow-up to acd82c8bfd

Closes #11035
2023-04-27 08:36:51 +02:00
Daniel Stenberg
65f03e507f
urldata: shrink *select_bits int => unsigned char
- dselect_bits
- cselect_bits

... are using less than 8 bits. Changed types and moved them towards
the end of the structs to fit better.

Closes #11025
2023-04-26 23:36:45 +02:00
Stefan Eissing
acd82c8bfd
tests/http: more tests with specific clients
- Makefile support for building test specific clients in tests/http/clients
- auto-make of clients when invoking pytest
- added test_09_02 for server PUSH_PROMISEs using clients/h2-serverpush
- added test_02_21 for lib based downloads and pausing/unpausing transfers

curl url parser:
- added internal method `curl_url_set_authority()` for setting the
  authority part of a url (used for PUSH_PROMISE)

http2:
- made logging of PUSH_PROMISE handling nicer

Placing python test requirements in requirements.txt files
- separate files to base test suite and http tests since use
  and module lists differ
- using the files in the gh workflows

websocket test cases, fixes for we and bufq
- bufq: account for spare chunks in space calculation
- bufq: reset chunks that are skipped empty
- ws: correctly encode frames with 126 bytes payload
- ws: update frame meta information on first call of collect
  callback that fills user buffer
- test client ws-data: some test/reporting improvements

Closes #11006
2023-04-26 23:24:46 +02:00
Stefan Eissing
5622e431ed
http2: always EXPIRE_RUN_NOW unpaused http/2 transfers
- just increasing the http/2 flow window does not necessarily
  make a server send new data. It may already have exhausted
  the window before

Closes #11005
2023-04-25 17:50:53 +02:00
Stefan Eissing
fbea71f293
http2: pass stream to http2_handle_stream_close to avoid NULL checks
Closes #11005
2023-04-25 17:50:45 +02:00
Stefan Eissing
cab2d56ea5
h2/h3: replace state.drain counter with state.dselect_bits
- `drain` was used by http/2 and http/3 implementations to indicate
  that the transfer requires send/recv independant from its socket
  poll state. Intended as a counter, it was used as bool flag only.
- a similar mechanism exists on `connectdata->cselect_bits` where
  specific protocols can indicate something similar, only for the
  whole connection.
- `cselect_bits` are cleard in transfer.c on use and, importantly,
  also set when the transfer loop expended its `maxloops` tries.
  `drain` was not cleared by transfer and the http2/3 implementations
  had to take care of that.
- `dselect_bits` is cleared *and* set by the transfer loop. http2/3
  does no longer clear it, only set when new events happen.

This change unifies the handling of socket poll overrides, extending
`cselect_bits` by a easy handle specific value and a common treatment in
transfers.

Closes #11005
2023-04-25 17:49:28 +02:00
Stefan Eissing
fc2f1e547a
http2: support HTTP/2 to forward proxies, non-tunneling
- with `--proxy-http2` allow h2 ALPN negotiation to
  forward proxies
- applies to http: requests against a https: proxy only,
  as https: requests will auto-tunnel
- adding a HTTP/1 request parser in http1.c
- removed h2h3.c
- using new request parser in nghttp2 and all h3 backends
- adding test 2603 for request parser
- adding h2 proxy test cases to test_10_*

scorecard.py: request scoring accidentally always run curl
with '-v'. Removed that, expect double numbers.

labeller: added http1.* and h2-proxy sources to detection

Closes #10967
2023-04-17 17:27:49 +02:00
Josh McCullough
233b4e4589
http2: fix typo in infof() call
Closes #10940
2023-04-13 08:39:56 +02:00
Stefan Eissing
c59b5b3c87
http2: fix copynpaste error reported by coverity
- move all code handling HTTP/2 frames for a particular
  stream into a separate function to keep from confusing
  the call `data` with the stream `data`.

Closes #10924
2023-04-11 23:46:53 +02:00
Daniel Stenberg
452b8e39ef
http2: avoid possible null pointer dereference
Reported-by: Dan Fandrich
Fixes #10920
Closes #10923
2023-04-11 12:29:03 +02:00
Daniel Stenberg
772e28ff76
http2: remove check for !data after it was already dereferenced
Pointed out by Coverity

Closes #10906
2023-04-07 23:49:51 +02:00
Stefan Eissing
632e0fbe70
http2: move HTTP/2 stream vars into local context
- remove NGHTTP2 members of `struct HTTP`
- add `void *h2_ctx` to `struct HTTP`
- add `void *h3_ctx` to `struct HTTP`
- separate h2/h3 pointers are needed for eyeballing
- manage local stream_ctx in http implementations

Closes #10877
2023-04-06 16:27:50 +02:00
Stefan Eissing
8f50e393ab
vtls and h2 improvements
- eliminate receive loop in vtls to fill buffer. This may
  lead to partial reads of data which is counter productive
- let http2 instead loop smarter to process pending network
  data without transfer switches

scorecard improvements
- do not start caddy when only httpd is requested
- allow curl -v to stderr file on --curl-verbose

Closes #10891
2023-04-06 09:41:54 +02:00
Stefan Eissing
744dcf22fa
http2: flow control and buffer improvements
- use bufq for send/receive of network data
- usd bufq for send/receive of stream data
- use HTTP/2 flow control with no-auto updates to control the
  amount of data we are buffering for a stream
  HTTP/2 stream window set to 128K after local tests, defined
  code constant for now
- elminiating PAUSEing nghttp2 processing when receiving data
  since a stream can now take in all DATA nghttp2 forwards

Improved scorecard and adjuste http2 stream window sizes
- scorecard improved output formatting and options default
- scorecard now also benchmarks small requests / second

Closes #10771
2023-03-30 23:11:26 +02:00
Stefan Eissing
06f65f771b http2: Use KEEP_SEND_HOLD for flow control in HTTP/2
- use the defined, but so far not used, KEEP_SEND_HOLD bit for flow
  control based suspend of sending in transfers.

Prior to this change KEEP_SEND_PAUSE bit was used instead, but that can
interfere with pausing streams from the user side via curl_easy_pause.

Fixes https://github.com/curl/curl/issues/10751
Closes https://github.com/curl/curl/pull/10753
2023-03-14 03:26:57 -04:00
Stefan Eissing
544a7f804d
http2: fix error handling during parallel operations
RST and connection close were not handled correctly during parallel
transfers, leading to aborted response bodies being reported complete.

Closes #10715
2023-03-10 23:52:53 +01:00
Stefan Eissing
7c5637b8b4
url: fix logic in connection reuse to deny reuse on "unclean" connections
- add parameter to `conn_is_alive()` cfilter method that returns
  if there is input data waiting on the connection
- refrain from re-using connnection from the cache that have
  input pending
- adapt http/2 and http/3 alive checks to digest pending input
  to check the connection state
- remove check_cxn method from openssl as that was just doing
  what the socket filter now does.
- add tests for connection reuse with special server configs

Closes #10690
2023-03-07 12:50:31 +01:00
Stefan Eissing
48cd032623
http2: fix handling of RST and GOAWAY to recognize partial transfers
- a reset transfer (HTTP/2 RST) did not always lead to the proper
  error message on receiving its response, leading to wrong reports
  of a successful transfer
- test_05_02 was able to trigger this condition with increased transfer
  count. The simulated response errors did not carry a 'Content-Length'
  so only proper RST handling could detect the abort
- When doing such transfers in parallel, a connection could enter the
  state where
  a) it had been closed (GOAWAY received)
  b) the RST had not been "seen" for the transfer yet
  or c) the GOAWAY announced an error and the last successful
  stream id was not checked against ongoing transfers

Closes #10693
2023-03-06 23:58:24 +01:00
Stefan Eissing
8151d7babc http2: fix code indent
Closes https://github.com/curl/curl/pull/10655
2023-03-02 00:09:53 -05:00
Stefan Eissing
d9ccc75b00 http2: fix upload busy loop
- Set KEEP_SEND_PAUSE when exhausting remote HTTP/2 window size of a
  stream.

- Clear KEEP_SEND_PAUSE when receiving HTTP/2 window updates on a paused
  stream.

- Also fix http2 send compiler warnings reported in #10449.

Prior to this change, starting in 71b7e016 which precedes 7.88.0,
libcurl may eat CPU during HTTP/2 upload.

Reported-by: Jay Satiro

Fixes https://github.com/curl/curl/issues/10449
Fixes https://github.com/curl/curl/issues/10618
Closes https://github.com/curl/curl/pull/10627
2023-03-01 04:13:49 -05:00
Stefan Eissing
cbe9d82c22
http2: fix for http2-prior-knowledge when reusing connections
- refs #10634 where errors in the HTTP/2 framing layer are observed.
- the bug was that on connection reuse, the code attempted to switch
  in yet another layer of HTTP/2 handling instead of detecting that
  this was already in place.
- added pytest testcase reproducing the issue.

Reported-by: rwmjones on github
Fixes #10634
Closes #10643
2023-02-28 23:04:00 +01:00
Harry Sintonen
87ed650d04
http2: set drain on stream end
Ensure that on_frame_recv() stream end will trigger a read if there is
pending data. Without this it could happen that the pending data is
never consumed.

This combined with https://github.com/curl/curl/pull/10529 should fix
https://github.com/curl/curl/issues/10525

Ref: https://github.com/curl/curl/issues/10525
Closes #10530
2023-02-16 09:33:52 +01:00
Stefan Eissing
3103de2053
http2: buffer/pausedata and output flush fix.
* do not process pending input data when copying pausedata to the
   caller
 * return CURLE_AGAIN if the output buffer could not be completely
   written out.

Ref: #10525
Closes #10529
2023-02-16 09:33:01 +01:00
Stefan Eissing
3de3ea6a64
HTTP/[23]: continue upload when state.drain is set
- as reported in #10433, HTTP/2 uploads may stall when a response is
  received before the upload is done. This happens when the
  data->state.drain is set for such a transfer, as the special handling
  in transfer.c from then on only cared about downloads.
- add continuation of uploads, if applicable, in this case.
- add pytest case test_07_12_upload_seq_large to reproduce this scenario
  (although, current nghttp2 implementation is using drain less often)

Reported-by: Lucas Pardue

Fixes #10433
Closes #10443
2023-02-09 09:13:30 +01:00
Stefan Eissing
8c762f5998
http2: minor buffer and error path fixes
- use memory buffer in full available size
- fail receive of reset/errored streams early

pytest:
- make test_05 error cases more reliable

Closes #10444
2023-02-09 08:38:01 +01:00
Stefan Eissing
ead2b2d4f6
http2: aggregate small SETTINGS/PRIO/WIN_UPDATE frames
add a small buffer to nghttp2 session sending in order to aggregate
small SETTINGS/PRIO/WIN_UPDATE frames that nghttp2 "writes" to the
callback individually.

Ref: #10389
Closes #10432
2023-02-08 10:41:15 +01:00
John Bampton
0065b146eb
misc: fix spelling
Closes #10437
2023-02-08 08:30:05 +01:00
Stefan Eissing
671158242d
connections: introduce http/3 happy eyeballs
New cfilter HTTP-CONNECT for h3/h2/http1.1 eyeballing.
- filter is installed when `--http3` in the tool is used (or
  the equivalent CURLOPT_ done in the library)
- starts a QUIC/HTTP/3 connect right away. Should that not
  succeed after 100ms (subject to change), a parallel attempt
  is started for HTTP/2 and HTTP/1.1 via TCP
- both attempts are subject to IPv6/IPv4 eyeballing, same
  as happens for other connections
- tie timeout to the ip-version HAPPY_EYEBALLS_TIMEOUT
- use a `soft` timeout at half the value. When the soft timeout
  expires, the HTTPS-CONNECT filter checks if the QUIC filter
  has received any data from the server. If not, it will start
  the HTTP/2 attempt.

HTTP/3(ngtcp2) improvements.
- setting call_data in all cfilter calls similar to http/2 and vtls filters
  for use in callback where no stream data is available.
- returning CURLE_PARTIAL_FILE for prematurely terminated transfers
- enabling pytest test_05 for h3
- shifting functionality to "connect" UDP sockets from ngtcp2
  implementation into the udp socket cfilter. Because unconnected
  UDP sockets are weird. For example they error when adding to a
  pollset.

HTTP/3(quiche) improvements.
- fixed upload bug in quiche implementation, now passes 251 and pytest
- error codes on stream RESET
- improved debug logs
- handling of DRAIN during connect
- limiting pending event queue

HTTP/2 cfilter improvements.
- use LOG_CF macros for dynamic logging in debug build
- fix CURLcode on RST streams to be CURLE_PARTIAL_FILE
- enable pytest test_05 for h2
- fix upload pytests and improve parallel transfer performance.

GOAWAY handling for ngtcp2/quiche
- during connect, when the remote server refuses to accept new connections
  and closes immediately (so the local conn goes into DRAIN phase), the
  connection is torn down and a another attempt is made after a short grace
  period.
  This is the behaviour observed with nghttpx when we tell it to  shut
  down gracefully. Tested in pytest test_03_02.

TLS improvements
- ALPN selection for SSL/SSL-PROXY filters in one vtls set of functions, replaces
  copy of logic in all tls backends.
- standardized the infof logging of offered ALPNs
- ALPN negotiated: have common function for all backends that sets alpn proprty
  and connection related things based on the negotiated protocol (or lack thereof).

- new tests/tests-httpd/scorecard.py for testing h3/h2 protocol implementation.
  Invoke:
    python3 tests/tests-httpd/scorecard.py --help
  for usage.

Improvements on gathering connect statistics and socket access.
- new CF_CTRL_CONN_REPORT_STATS cfilter control for having cfilters
  report connection statistics. This is triggered when the connection
  has completely connected.
- new void Curl_pgrsTimeWas(..) method to report a timer update with
  a timestamp of when it happend. This allows for updating timers
  "later", e.g. a connect statistic after full connectivity has been
  reached.
- in case of HTTP eyeballing, the previous changes will update
  statistics only from the filter chain that "won" the eyeballing.
- new cfilter query CF_QUERY_SOCKET for retrieving the socket used
  by a filter chain.
  Added methods Curl_conn_cf_get_socket() and Curl_conn_get_socket()
  for convenient use of this query.
- Change VTLS backend to query their sub-filters for the socket when
  checks during the handshake are made.

HTTP/3 documentation on how https eyeballing works.

TLS improvements
- ALPN selection for SSL/SSL-PROXY filters in one vtls set of functions, replaces
  copy of logic in all tls backends.
- standardized the infof logging of offered ALPNs
- ALPN negotiated: have common function for all backends that sets alpn proprty
  and connection related things based on the negotiated protocol (or lack thereof).

Scorecard with Caddy.
- configure can be run with `--with-test-caddy=path` to specify which caddy to use for testing
- tests/tests-httpd/scorecard.py now measures download speeds with caddy

pytest improvements
- adding Makfile to clean gen dir
- adding nghttpx rundir creation on start
- checking httpd version 2.4.55 for test_05 cases where it is needed. Skipping with message if too old.
- catch exception when checking for caddy existance on system.

Closes #10349
2023-02-02 09:57:34 +01:00
Stefan Eissing
5651a36d1a
cf-socket: improvements in socket I/O handling
- Curl_write_plain/Curl_read_plain have been eliminated. Last code use
  now uses Curl_conn_send/recv so that requests use conn->send/revc
  callbacks which defaults to cfilters use.
- Curl_recv_plain/Curl_send_plain have been internalized in cf-socket.c.
- USE_RECV_BEFORE_SEND_WORKAROUND (active on Windows) has been moved
  into cf-socket.c. The pre_recv buffer is held at the socket filter
  context.  `postponed_data` structures have been removed from
  `connectdata`.
- the hanger in HTTP/2 request handling was a result of read buffering
  on all sends and the multi handling is not prepared for this. The
  following happens:

   - multi preforms on a HTTP/2 easy handle
   - h2 reads and processes data
   - this leads to a send of h2 data
   - which receives and buffers before the send
   - h2 returns
   - multi selects on the socket, but no data arrives (its in the buffer already)
   the workaround now receives data in a loop as long as there is something in
   the buffer. The real fix would be for multi to change, so that `data_pending`
   is evaluated before deciding to wait on the socket.

io_buffer, optional, in cf-socket.c, http/2 sets state.drain if lower
filter have pending data.

This io_buffer is only available/used when the
-DUSE_RECV_BEFORE_SEND_WORKAROUND is active, e.g. on Windows
configurations. It also maintains the original checks on protocol
handler being HTTP and conn->send/recv not being replaced.

The HTTP/2 (nghttp2) cfilter now sets data->state.drain when it finds
out that the "lower" filter chain has still pending data at the end of
its IO operation. This prevents the processing from becoming stalled.

Closes #10280
2023-01-31 11:23:59 +01:00
Stefan Eissing
db91dbbf2c
curl_log: for failf/infof and debug logging implementations
- new functions and macros for cfilter debugging
 - set CURL_DEBUG with names of cfilters where debug logging should be
   enabled
 - use GNUC __attribute__ to enable printf format checks during compile

Closes #10271
2023-01-12 10:14:17 +01:00
Stefan Eissing
33ac97e1cb
tests-httpd: basic infra to run curl against an apache httpd plus nghttpx for h3
- adding '--with-test-httpd=<path>' to configure non-standard apache2
  install
- python env and base classes for running httpd
- basic tests for connectivity with h1/h2/h3
- adding test cases for truncated responses in http versions.
- adding goaway test for HTTP/3.
- adding "stuttering" tests with parallel downloads in chunks with
  varying delays between chunks.

- adding a curltest module to the httpd server, adding GOAWAY test.
    - mod_curltest now installs 2 handlers
      - 'echo': writing as response body what came as request body
      - 'tweak': with query parameters to tweak response behaviour
- marked known fails as skip for now

Closes #10175
2023-01-09 17:40:04 +01:00
Jay Satiro
fcd1b547d4 http2: fix compiler warning due to uninitialized variable
Prior to this change http2_cfilter_add could return an uninitialized
cfilter pointer in an OOM condition. In this case though, the pointer
is discarded and not dereferenced so there was no risk of a crash.
2023-01-03 17:15:26 -05:00
Daniel Stenberg
2bc1d775f5
copyright: update all copyright lines and remove year ranges
- they are mostly pointless in all major jurisdictions
- many big corporations and projects already don't use them
- saves us from pointless churn
- git keeps history for us
- the year range is kept in COPYING

checksrc is updated to allow non-year using copyright statements

Closes #10205
2023-01-03 09:19:21 +01:00
Daniel Stenberg
b637f35346
http2: when using printf %.*s, the length arg must be 'int'
Detected by Coverity CID 1518341

Closes #10203
2023-01-02 13:24:37 +01:00
Daniel Stenberg
ed18244df1
http2: in connisdead check, attach the connection before reading
Otherwise data->conn is NULL and things go wrong.

This problem caused occastional failures in test 359, 1700 and more
depending on timing and the alignment of various planets.

Assisted-by: Stefan Eissing

Closes #10199
2023-01-02 13:16:24 +01:00
Stefan Eissing
71b7e01610
lib: connect/h2/h3 refactor
Refactoring of connection setup and happy eyeballing. Move
nghttp2. ngtcp2, quiche and msh3 into connection filters.

 - eyeballing cfilter that uses sub-filters for performing parallel connects
 - socket cfilter for all transport types, including QUIC
 - QUIC implementations in cfilter, can now participate in eyeballing
 - connection setup is more dynamic in order to adapt to what filter did
   really connect.  Relevant to see if a SSL filter needs to be added or
   if SSL has already been provided
 - HTTP/3 test cases similar to HTTP/2
 - multiuse of parallel transfers for HTTP/3, tested for ngtcp2 and quiche

 - Fix for data attach/detach in VTLS filters that could lead to crashes
   during parallel transfers.
 - Eliminating setup() methods in cfilters, no longer needed.
 - Improving Curl_conn_is_alive() to replace Curl_connalive() and
   integrated ssl alive checks into cfilter.
 - Adding CF_CNTRL_CONN_INFO_UPDATE to tell filters to update
   connection into and persist it at the easy handle.

 - Several more cfilter related cleanups and moves:
   - stream_weigth and dependency info is now wrapped in struct
     Curl_data_priority
   - Curl_data_priority members depend is available in HTTP2|HTTP3
   - Curl_data_priority members depend on NGHTTP2 support
   - handling init/reset/cleanup of priority part of url.c
   - data->state.priority same struct, but shallow copy for compares only

 - PROTOPT_STREAM has been removed
   - Curl_conn_is_mulitplex() now available to check on capability

 - Adding query method to connection filters.
   - ngtcp2+quiche: implementing query for max concurrent transfers.

 - Adding is_alive and keep_alive cfilter methods. Adding DATA_SETUP event.
   - setting keepalive timestamp on connect
   - DATA_SETUP is called after the connection has been completely
     setup (but may not connected yet) to allow filters to initialize
     data members they use.

 - there is no socket to be had with msh3, it is unclear how select
   shall work

 - manual test via "curl --http3 https://curl.se" fail with "empty
   reply from server".

 - Various socket/conn related cleanups:
   - Curl_socket is now Curl_socket_open and in cf-socket.c
   - Curl_closesocket is now Curl_socket_close and in cf-socket.c
   - Curl_ssl_use has been replaced with Cur_conn_is_ssl
   - Curl_conn_tcp_accepted_set has been split into
     Curl_conn_tcp_listen_set and Curl_conn_tcp_accepted_set
     with a clearer purpose

Closes #10141
2022-12-30 16:43:19 +01:00
Nick Banks
ae98b85020
msh3: add support for request payload
Closes #10136
2022-12-28 13:52:58 +01:00
Stefan Eissing
af22c2a546
vtls: localization of state data in filters
- almost all backend calls pass the Curl_cfilter intance instead of
   connectdata+sockindex
 - ssl_connect_data is remove from struct connectdata and made internal
   to vtls
 - ssl_connect_data is allocated in the added filter, kept at cf->ctx

 - added function to let a ssl filter access its ssl_primary_config and
   ssl_config_data this selects the propert subfields in conn and data,
   for filters added as plain or proxy
 - adjusted all backends to use the changed api
 - adjusted all backends to access config data via the exposed
   functions, no longer using conn or data directly

cfilter renames for clear purpose:

 - methods `Curl_conn_*(data, conn, sockindex)` work on the complete
   filter chain at `sockindex` and connection `conn`.
 - methods `Curl_cf_*(cf, ...)` work on a specific Curl_cfilter
   instance.
 - methods `Curl_conn_cf()` work on/with filter instances at a
   connection.
 - rebased and resolved some naming conflicts
 - hostname validation (und session lookup) on SECONDARY use the same
   name as on FIRST (again).

new debug macros and removing connectdata from function signatures where not
needed.

adapting schannel for new Curl_read_plain paramter.

Closes #9919
2022-11-22 14:25:50 +01:00
Daniel Stenberg
3f039dfd6f
strcase: use curl_str(n)equal for case insensitive matches
No point in having two entry points for the same functions.

Also merged the *safe* function treatment into these so that they can
also be used when one or both pointers are NULL.

Closes #9837
2022-11-01 17:01:26 +01:00
Ayesh Karunaratne
4484270afc
misc: typo and grammar fixes
- Replace `Github` with `GitHub`.
- Replace `windows` with `Windows`
- Replace `advice` with `advise` where a verb is used.
- A few fixes on removing repeated words.
- Replace `a HTTP` with `an HTTP`

Closes #9802
2022-10-27 10:01:30 +02:00
Daniel Stenberg
664249d095
ws: initial websockets support
Closes #8995
2022-09-09 15:11:14 +02:00
Daniel Stenberg
eafc2b14ac
http2: make nghttp2 less picky about field whitespace
In nghttp2 1.49.0 it returns error on leading and trailing whitespace in
header fields according to language in the recently shipped RFC 9113.

nghttp2 1.50.0 introduces an option to switch off this strict check and
this change enables this option by default which should make curl behave
more similar to how it did with nghttp2 1.48.0 and earlier.

We might want to consider making this an option in the future.

Closes #9448
2022-09-08 14:52:45 +02:00
max.mehl
ad9bc5976d
copyright: make repository REUSE compliant
Add licensing and copyright information for all files in this repository. This
either happens in the file itself as a comment header or in the file
`.reuse/dep5`.

This commit also adds a Github workflow to check pull requests and adapts
copyright.pl to the changes.

Closes #8869
2022-06-13 09:13:00 +02:00
Daniel Stenberg
21ea13cfe1
http2: reject overly many push-promise headers
Getting more than a thousand of them is rather a sign of some kind of
attack.

Reported-by: Harry Sintonen
Bug: https://hackerone.com/reports/1589847
Closes #8962
2022-06-06 11:53:49 +02:00
steini2000
f9bc378ab7 http2: always debug print stream id in decimal with %u
Prior to this change the stream id shown could be hex or decimal which
was inconsistent and confusing.

Closes https://github.com/curl/curl/pull/8808
2022-05-12 01:31:32 -04:00
Daniel Stenberg
6968fb9d54
lib: remove exclamation marks
... from infof() and failf() calls. Make them less attention seeking.

Closes #8713
2022-04-16 11:55:05 +02:00
Daniel Stenberg
fda4b81635
http2: RST the stream if we stop it on our own will
For the "simulated 304" case the done-call isn't considered "premature"
but since the server didn't close the stream it needs to be reset to
stop delivering data.

Closes #8664
2022-04-01 13:23:04 +02:00
Daniel Stenberg
b5a9680577
http2: handle DONE called for the paused stream
As it could otherwise stall all streams on the connection

Reported-by: Evangelos Foutras
Fixes #8626
Closes #8664
2022-04-01 13:22:58 +02:00
Daniel Stenberg
d1e4a67734
header api: add curl_easy_header and curl_easy_nextheader
Add test 1940 to 1946 to verify.

Closes #8593
2022-03-22 08:24:22 +01:00
Daniel Stenberg
0694037032
http2: move two infof calls to debug-h2-only
and remove a superflous one

Ref: https://github.com/curl/curl/discussions/8498
Closes #8502
2022-02-24 11:00:12 +01:00
HenrikHolst
b807219292
misc: reduce strlen() calls with Curl_dyn_add()
Use STRCONST() to switch from Curl_dyn_add() to Curl_dyn_addn() for
string literals.

Closes #8398
2022-02-09 13:52:47 +01:00
Daniel Stenberg
2a1951519e
http2: fix the array copy to nghttp2_nv
Bug: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=44517
Follow-up to 9f985a11e7
Closes #8414
2022-02-09 12:18:50 +01:00
Daniel Stenberg
9f985a11e7
http2: use Curl_pseudo_headers 2022-02-07 13:56:06 +01:00
Daniel Stenberg
c85178a94c
http2: allow CURLOPT_HTTPHEADER change ":scheme"
The only h2 psuedo header that wasn't previously possible to change by a
user. This change also makes it impossible to send a HTTP/1 header that
starts with a colon, which I don't think anyone does anyway.

The other pseudo headers are possible to change indirectly by doing the
rightly crafted request.

Reported-by: siddharthchhabrap on github
Fixes #8381
Closes #8393
2022-02-07 13:55:57 +01:00
Daniel Stenberg
136f3e9d68
h2/h3: provide and refer to pseudo headers as defines
... and do sizeof() on the defines to use constants better.

Closes #8389
2022-02-05 11:08:43 +01:00
Daniel Stenberg
2610142139
lib: remove support for CURL_DOES_CONVERSIONS
TPF was the only user and support for that was dropped.

Closes #8378
2022-02-04 08:05:35 +01:00
Daniel Stenberg
b3f9c1aa09
http2:set_transfer_url() return early on OOM
If curl_url() returns NULL this should return early to avoid mistakes -
even if right now the subsequent function invokes are all OK.

Coverity (wrongly) pointed out this as a NULL deref.

Closes #8100
2021-12-06 14:32:29 +01:00
Daniel Stenberg
1fed8feaf1
http2: make getsock not wait for write if there's no remote window
While uploading, check for remote window availability in the getsock
function so that we don't wait for a writable socket if no data can be
sent.

Reported-by: Steini2000 on github
Fixes #7821
Closes #7839
2021-10-12 11:19:22 +02:00
Daniel Stenberg
901804ef95
Curl_http2_setup: don't change connection data on repeat invokes
Regression from 3cb8a74867 (releasde in 7.79.0). That change moved
transfer oriented inits to before the check but also erroneously moved a
few connection oriented ones, which causes problems.

Reported-by: Evangelos Foutras
Fixes #7730
Closes #7731
2021-09-16 11:23:55 +02:00
Daniel Stenberg
3cb8a74867
http2: Curl_http2_setup needs to init stream data in all invokes
Thus function was written to avoid doing multiple connection data
initializations, which is fine, but since it also initiates stream
related data it is crucial that it doesn't skip those even if called
again for the same connection. Solved by moving the stream
initializations before the "doing-it-again" check.

Reported-by: Inho Oh
Fixes #7630
Closes #7692
2021-09-11 23:01:23 +02:00
Daniel Stenberg
c364f5f6fa
Revert "http2: skip immediate parsing of payload following protocol switch"
This reverts commit 455a63c66f.

Reported-by: Tk Xiong
Fixes #7633
Closes #7648
2021-08-30 17:57:32 +02:00
Artur Sinila
ccffb6ec16
http2: revert call the handle-closed function correctly on closed stream
Reverts 252790c533

Assisted-by: Gergely Nagy
Fixes #7400
Closes #7525
2021-08-16 08:44:52 +02:00
Daniel Stenberg
e7416cfd2b
infof: remove newline from format strings, always append it
- the data needs to be "line-based" anyway since it's also passed to the
  debug callback/application

- it makes infof() work like failf() and consistency is good

- there's an assert that triggers on newlines in the format string

- Also removes a few instances of "..."

- Removes the code that would append "..." to the end of the data *iff*
  it was truncated in infof()

Closes #7357
2021-07-07 22:54:01 +02:00
Daniel Stenberg
63c7668182
version: turn version number functions into returning void
... as we never use the return codes from them.

Reviewed-by: Daniel Gustafsson
Closes #7319
2021-06-30 23:23:34 +02:00
Jay Satiro
c6cac5a96c http2: Clarify 'Using HTTP2' verbose message
- Change phrasing from multi-use to multiplexing since the former may
  not be as well understood.

Before: * Using HTTP2, server supports multi-use

After: * Using HTTP2, server supports multiplexing

Bug: https://github.com/curl/curl/discussions/7255
Reported-by: David Hu

Closes https://github.com/curl/curl/pull/7258
2021-06-15 03:18:45 -04:00
Shikha Sharma
a0709f9951
http2_connisdead: handle trailing GOAWAY better
When checking the connection the input processing returns error
immediately, we now consider that a dead connnection.

Bug: https://curl.se/mail/lib-2021-06/0001.html
Closes #7192
2021-06-05 23:45:31 +02:00
Daniel Stenberg
ef1a917a11
http2: init recvbuf struct for pushed streams
Debug builds would warn that these structs were not initialized properly
for pushed streams.

Ref: #7148
Closes #7153
2021-05-31 20:41:51 +02:00
Daniel Stenberg
0c55fbab45
conn: add 'attach' to protocol handler, make libssh2 use it
The libssh2 backend has SSH session associated with the connection but
the callback context is the easy handle, so when a connection gets
attached to a transfer, the protocol handler now allows for a custom
function to get used to set things up correctly.

Reported-by: Michael O'Farrell
Fixes #6898
Closes #7078
2021-05-17 17:57:50 +02:00
Daniel Stenberg
904b27d18d
http2: make sure pause is done on HTTP
Since the function is called for any protocol, we can't assume that the
HTTP struct is there without first making sure it is HTTP.

Reported-by: Denis Goleshchikhin
Fixes #7079
Closes #7080
2021-05-17 14:44:13 +02:00
Peng-Yu Chen
455a63c66f
http2: skip immediate parsing of payload following protocol switch
This is considered not harmful as a following http2_recv shall be
called very soon.

This is considered helpful in the specific situation where some
servers (e.g. nghttpx v1.43.0) may fulfill stream 1 immediately
following the return of HTTP status 101, other than waiting for
the client-side connection preface to arrive.

Fixes #7036
Closes #7040
2021-05-11 08:03:22 +02:00
Peng-Yu Chen
651a75e1e7
http2: use nghttp2_session_upgrade2 instead of nghttp2_session_upgrade
Following the upstream deprecation of nghttp2_session_upgrade.

Also provides further checks for requests with the HEAD method.

Closes #7041
2021-05-11 07:58:14 +02:00
Jay Satiro
ba0d3c28ca http2: fix potentially uninitialized variable
introduced several days ago in 3193170. caught by visual studio linker.
2021-05-05 03:16:06 -04:00
Kamil Dudka
3a6058cb97
http2: fix a resource leak in push_promise()
... detected by Coverity:

Error: RESOURCE_LEAK (CWE-772):
lib/http2.c:532: alloc_fn: Storage is returned from allocation function "duphandle".
lib/http2.c:532: var_assign: Assigning: "newhandle" = storage returned from "duphandle(data)".
lib/http2.c:552: noescape: Resource "newhandle" is not freed or pointed-to in "set_transfer_url".
lib/http2.c:555: leaked_storage: Variable "newhandle" going out of scope leaks the storage it points to.

Closes #6986
2021-05-01 23:40:14 +02:00
Kamil Dudka
3193170470
http2: fix resource leaks in set_transfer_url()
... detected by Coverity:

Error: RESOURCE_LEAK (CWE-772):
lib/http2.c:480: alloc_fn: Storage is returned from allocation function "curl_url". [Note: The source code implementation of the function has been overridden by a builtin model.]
lib/http2.c:480: var_assign: Assigning: "u" = storage returned from "curl_url()".
lib/http2.c:486: noescape: Resource "u" is not freed or pointed-to in "curl_url_set". [Note: The source code implementation of the function has been overridden by a builtin model.]
lib/http2.c:488: leaked_storage: Variable "u" going out of scope leaks the storage it points to.

Error: RESOURCE_LEAK (CWE-772):
lib/http2.c:480: alloc_fn: Storage is returned from allocation function "curl_url". [Note: The source code implementation of the function has been overridden by a builtin model.]
lib/http2.c:480: var_assign: Assigning: "u" = storage returned from "curl_url()".
lib/http2.c:493: noescape: Resource "u" is not freed or pointed-to in "curl_url_set". [Note: The source code implementation of the function has been overridden by a builtin model.]
lib/http2.c:495: leaked_storage: Variable "u" going out of scope leaks the storage it points to.

Error: RESOURCE_LEAK (CWE-772):
lib/http2.c:480: alloc_fn: Storage is returned from allocation function "curl_url". [Note: The source code implementation of the function has been overridden by a builtin model.]
lib/http2.c:480: var_assign: Assigning: "u" = storage returned from "curl_url()".
lib/http2.c:500: noescape: Resource "u" is not freed or pointed-to in "curl_url_set". [Note: The source code implementation of the function has been overridden by a builtin model.]
lib/http2.c:502: leaked_storage: Variable "u" going out of scope leaks the storage it points to.

Error: RESOURCE_LEAK (CWE-772):
lib/http2.c:480: alloc_fn: Storage is returned from allocation function "curl_url". [Note: The source code implementation of the function has been overridden by a builtin model.]
lib/http2.c:480: var_assign: Assigning: "u" = storage returned from "curl_url()".
lib/http2.c:505: noescape: Resource "u" is not freed or pointed-to in "curl_url_get". [Note: The source code implementation of the function has been overridden by a builtin model.]
lib/http2.c:507: leaked_storage: Variable "u" going out of scope leaks the storage it points to.

Closes #6986
2021-05-01 23:39:18 +02:00
Daniel Stenberg
063d3f3b96
tidy-up: make conditional checks more consistent
... remove '== NULL' and '!= 0'

Closes #6912
2021-04-22 09:10:17 +02:00
Daniel Stenberg
d8c4cdecb7
http2: remove DEBUG_HTTP2
Accidentally committed in 605e84235
2021-04-21 11:34:10 +02:00
Daniel Stenberg
605e842355
http2: move the stream error field to the per-transfer storage
Storing a stream error in the per-connection struct was an error that lead to
race conditions as subsequent stream handling could overwrite the error code
before it was used for the stream with the actual problem.

Closes #6910
2021-04-21 08:20:24 +02:00
Daniel Stenberg
252790c533
http2: call the handle-closed function correctly on closed stream
This was this one condition where the stream could be closed due to an
error and the function would still wrongly just return 0 for it.

Reported-by: Gergely Nagy
Fixes #6862
Closes #6910
2021-04-21 08:20:24 +02:00
Daniel Stenberg
95cbcec8f9
urldata: merge "struct DynamicStatic" into "struct UrlState"
Both were used for the same purposes and there was no logical separation
between them. Combined, this also saves 16 bytes in less holes in my
test build.

Closes #6798
2021-03-26 23:19:20 +01:00
Robert Ronto
068661bb02
http2: don't set KEEP_SEND when there's no more data to be sent
this should fix an issue where curl sometimes doesn't send out a request
with authorization info after a 401 is received over http2

Closes #6747
2021-03-15 23:29:35 +01:00
oxalica
d1f40078c1
http2: fail if connection terminated without END_STREAM
Closes #6736
2021-03-12 17:54:39 +01:00
Daniel Stenberg
723c0e75fa
http2: remove conn->data use
... but instead use a private alternative that points to the "driving
transfer" from the connection. We set the "user data" associated with
the connection to be the connectdata struct, but when we drive transfers
the code still needs to know the pointer to the transfer. We can change
the user data to become the Curl_easy handle, but with older nghttp2
version we cannot dynamically update that pointer properly when
different transfers are used over the same connection.

Closes #6520
2021-02-15 16:33:53 +01:00
Daniel Stenberg
1c3def53c7
urldata: remove duplicate 'upkeep_interval_ms' from connectdata
... and rely only on the value already set in Curl_easy.

Closes #6534
2021-01-27 09:19:08 +01:00
Daniel Stenberg
215db086e0
lib: pass in 'struct Curl_easy *' to most functions
... in most cases instead of 'struct connectdata *' but in some cases in
addition to.

- We mostly operate on transfers and not connections.

- We need the transfer handle to log, store data and more. Everything in
  libcurl is driven by a transfer (the CURL * in the public API).

- This work clarifies and separates the transfers from the connections
  better.

- We should avoid "conn->data". Since individual connections can be used
  by many transfers when multiplexing, making sure that conn->data
  points to the current and correct transfer at all times is difficult
  and has been notoriously error-prone over the years. The goal is to
  ultimately remove the conn->data pointer for this reason.

Closes #6425
2021-01-17 23:56:09 +01:00
Daniel Stenberg
aba01da639
failf: remove newline from formatting strings
... as failf adds one itself.

Also: add an assert() to failf() that triggers on a newline in the
format string!

Closes #6365
2020-12-25 23:42:16 +01:00
Daniel Stenberg
c7f95fa0cc
h2: do not wait for RECV on paused transfers
... as the socket might be readable all the time when paused and thus
causing a busy-loop.

Reported-by: Harry Sintonen
Reviewed-by: Jay Satiro
Fixes #6356
Closes #6357
2020-12-22 12:44:18 +01:00
Rikard Falkeborn
920f49a20b
infof/failf calls: fix format specifiers
Update a few format specifiers to match what is being printed.

Closes #6241
2020-11-24 13:18:41 +01:00
Daniel Stenberg
a95a6ce6b8
urldata: remove 'void *protop' and create the union 'p'
... to avoid the use of 'void *' for the protocol specific structs done
per transfer.

Closes #6238
2020-11-23 16:16:16 +01:00