Commit Graph

406 Commits

Author SHA1 Message Date
Stefan Eissing
671158242d
connections: introduce http/3 happy eyeballs
New cfilter HTTP-CONNECT for h3/h2/http1.1 eyeballing.
- filter is installed when `--http3` in the tool is used (or
  the equivalent CURLOPT_ done in the library)
- starts a QUIC/HTTP/3 connect right away. Should that not
  succeed after 100ms (subject to change), a parallel attempt
  is started for HTTP/2 and HTTP/1.1 via TCP
- both attempts are subject to IPv6/IPv4 eyeballing, same
  as happens for other connections
- tie timeout to the ip-version HAPPY_EYEBALLS_TIMEOUT
- use a `soft` timeout at half the value. When the soft timeout
  expires, the HTTPS-CONNECT filter checks if the QUIC filter
  has received any data from the server. If not, it will start
  the HTTP/2 attempt.

HTTP/3(ngtcp2) improvements.
- setting call_data in all cfilter calls similar to http/2 and vtls filters
  for use in callback where no stream data is available.
- returning CURLE_PARTIAL_FILE for prematurely terminated transfers
- enabling pytest test_05 for h3
- shifting functionality to "connect" UDP sockets from ngtcp2
  implementation into the udp socket cfilter. Because unconnected
  UDP sockets are weird. For example they error when adding to a
  pollset.

HTTP/3(quiche) improvements.
- fixed upload bug in quiche implementation, now passes 251 and pytest
- error codes on stream RESET
- improved debug logs
- handling of DRAIN during connect
- limiting pending event queue

HTTP/2 cfilter improvements.
- use LOG_CF macros for dynamic logging in debug build
- fix CURLcode on RST streams to be CURLE_PARTIAL_FILE
- enable pytest test_05 for h2
- fix upload pytests and improve parallel transfer performance.

GOAWAY handling for ngtcp2/quiche
- during connect, when the remote server refuses to accept new connections
  and closes immediately (so the local conn goes into DRAIN phase), the
  connection is torn down and a another attempt is made after a short grace
  period.
  This is the behaviour observed with nghttpx when we tell it to  shut
  down gracefully. Tested in pytest test_03_02.

TLS improvements
- ALPN selection for SSL/SSL-PROXY filters in one vtls set of functions, replaces
  copy of logic in all tls backends.
- standardized the infof logging of offered ALPNs
- ALPN negotiated: have common function for all backends that sets alpn proprty
  and connection related things based on the negotiated protocol (or lack thereof).

- new tests/tests-httpd/scorecard.py for testing h3/h2 protocol implementation.
  Invoke:
    python3 tests/tests-httpd/scorecard.py --help
  for usage.

Improvements on gathering connect statistics and socket access.
- new CF_CTRL_CONN_REPORT_STATS cfilter control for having cfilters
  report connection statistics. This is triggered when the connection
  has completely connected.
- new void Curl_pgrsTimeWas(..) method to report a timer update with
  a timestamp of when it happend. This allows for updating timers
  "later", e.g. a connect statistic after full connectivity has been
  reached.
- in case of HTTP eyeballing, the previous changes will update
  statistics only from the filter chain that "won" the eyeballing.
- new cfilter query CF_QUERY_SOCKET for retrieving the socket used
  by a filter chain.
  Added methods Curl_conn_cf_get_socket() and Curl_conn_get_socket()
  for convenient use of this query.
- Change VTLS backend to query their sub-filters for the socket when
  checks during the handshake are made.

HTTP/3 documentation on how https eyeballing works.

TLS improvements
- ALPN selection for SSL/SSL-PROXY filters in one vtls set of functions, replaces
  copy of logic in all tls backends.
- standardized the infof logging of offered ALPNs
- ALPN negotiated: have common function for all backends that sets alpn proprty
  and connection related things based on the negotiated protocol (or lack thereof).

Scorecard with Caddy.
- configure can be run with `--with-test-caddy=path` to specify which caddy to use for testing
- tests/tests-httpd/scorecard.py now measures download speeds with caddy

pytest improvements
- adding Makfile to clean gen dir
- adding nghttpx rundir creation on start
- checking httpd version 2.4.55 for test_05 cases where it is needed. Skipping with message if too old.
- catch exception when checking for caddy existance on system.

Closes #10349
2023-02-02 09:57:34 +01:00
Stefan Eissing
5651a36d1a
cf-socket: improvements in socket I/O handling
- Curl_write_plain/Curl_read_plain have been eliminated. Last code use
  now uses Curl_conn_send/recv so that requests use conn->send/revc
  callbacks which defaults to cfilters use.
- Curl_recv_plain/Curl_send_plain have been internalized in cf-socket.c.
- USE_RECV_BEFORE_SEND_WORKAROUND (active on Windows) has been moved
  into cf-socket.c. The pre_recv buffer is held at the socket filter
  context.  `postponed_data` structures have been removed from
  `connectdata`.
- the hanger in HTTP/2 request handling was a result of read buffering
  on all sends and the multi handling is not prepared for this. The
  following happens:

   - multi preforms on a HTTP/2 easy handle
   - h2 reads and processes data
   - this leads to a send of h2 data
   - which receives and buffers before the send
   - h2 returns
   - multi selects on the socket, but no data arrives (its in the buffer already)
   the workaround now receives data in a loop as long as there is something in
   the buffer. The real fix would be for multi to change, so that `data_pending`
   is evaluated before deciding to wait on the socket.

io_buffer, optional, in cf-socket.c, http/2 sets state.drain if lower
filter have pending data.

This io_buffer is only available/used when the
-DUSE_RECV_BEFORE_SEND_WORKAROUND is active, e.g. on Windows
configurations. It also maintains the original checks on protocol
handler being HTTP and conn->send/recv not being replaced.

The HTTP/2 (nghttp2) cfilter now sets data->state.drain when it finds
out that the "lower" filter chain has still pending data at the end of
its IO operation. This prevents the processing from becoming stalled.

Closes #10280
2023-01-31 11:23:59 +01:00
Stefan Eissing
db91dbbf2c
curl_log: for failf/infof and debug logging implementations
- new functions and macros for cfilter debugging
 - set CURL_DEBUG with names of cfilters where debug logging should be
   enabled
 - use GNUC __attribute__ to enable printf format checks during compile

Closes #10271
2023-01-12 10:14:17 +01:00
Stefan Eissing
33ac97e1cb
tests-httpd: basic infra to run curl against an apache httpd plus nghttpx for h3
- adding '--with-test-httpd=<path>' to configure non-standard apache2
  install
- python env and base classes for running httpd
- basic tests for connectivity with h1/h2/h3
- adding test cases for truncated responses in http versions.
- adding goaway test for HTTP/3.
- adding "stuttering" tests with parallel downloads in chunks with
  varying delays between chunks.

- adding a curltest module to the httpd server, adding GOAWAY test.
    - mod_curltest now installs 2 handlers
      - 'echo': writing as response body what came as request body
      - 'tweak': with query parameters to tweak response behaviour
- marked known fails as skip for now

Closes #10175
2023-01-09 17:40:04 +01:00
Jay Satiro
fcd1b547d4 http2: fix compiler warning due to uninitialized variable
Prior to this change http2_cfilter_add could return an uninitialized
cfilter pointer in an OOM condition. In this case though, the pointer
is discarded and not dereferenced so there was no risk of a crash.
2023-01-03 17:15:26 -05:00
Daniel Stenberg
2bc1d775f5
copyright: update all copyright lines and remove year ranges
- they are mostly pointless in all major jurisdictions
- many big corporations and projects already don't use them
- saves us from pointless churn
- git keeps history for us
- the year range is kept in COPYING

checksrc is updated to allow non-year using copyright statements

Closes #10205
2023-01-03 09:19:21 +01:00
Daniel Stenberg
b637f35346
http2: when using printf %.*s, the length arg must be 'int'
Detected by Coverity CID 1518341

Closes #10203
2023-01-02 13:24:37 +01:00
Daniel Stenberg
ed18244df1
http2: in connisdead check, attach the connection before reading
Otherwise data->conn is NULL and things go wrong.

This problem caused occastional failures in test 359, 1700 and more
depending on timing and the alignment of various planets.

Assisted-by: Stefan Eissing

Closes #10199
2023-01-02 13:16:24 +01:00
Stefan Eissing
71b7e01610
lib: connect/h2/h3 refactor
Refactoring of connection setup and happy eyeballing. Move
nghttp2. ngtcp2, quiche and msh3 into connection filters.

 - eyeballing cfilter that uses sub-filters for performing parallel connects
 - socket cfilter for all transport types, including QUIC
 - QUIC implementations in cfilter, can now participate in eyeballing
 - connection setup is more dynamic in order to adapt to what filter did
   really connect.  Relevant to see if a SSL filter needs to be added or
   if SSL has already been provided
 - HTTP/3 test cases similar to HTTP/2
 - multiuse of parallel transfers for HTTP/3, tested for ngtcp2 and quiche

 - Fix for data attach/detach in VTLS filters that could lead to crashes
   during parallel transfers.
 - Eliminating setup() methods in cfilters, no longer needed.
 - Improving Curl_conn_is_alive() to replace Curl_connalive() and
   integrated ssl alive checks into cfilter.
 - Adding CF_CNTRL_CONN_INFO_UPDATE to tell filters to update
   connection into and persist it at the easy handle.

 - Several more cfilter related cleanups and moves:
   - stream_weigth and dependency info is now wrapped in struct
     Curl_data_priority
   - Curl_data_priority members depend is available in HTTP2|HTTP3
   - Curl_data_priority members depend on NGHTTP2 support
   - handling init/reset/cleanup of priority part of url.c
   - data->state.priority same struct, but shallow copy for compares only

 - PROTOPT_STREAM has been removed
   - Curl_conn_is_mulitplex() now available to check on capability

 - Adding query method to connection filters.
   - ngtcp2+quiche: implementing query for max concurrent transfers.

 - Adding is_alive and keep_alive cfilter methods. Adding DATA_SETUP event.
   - setting keepalive timestamp on connect
   - DATA_SETUP is called after the connection has been completely
     setup (but may not connected yet) to allow filters to initialize
     data members they use.

 - there is no socket to be had with msh3, it is unclear how select
   shall work

 - manual test via "curl --http3 https://curl.se" fail with "empty
   reply from server".

 - Various socket/conn related cleanups:
   - Curl_socket is now Curl_socket_open and in cf-socket.c
   - Curl_closesocket is now Curl_socket_close and in cf-socket.c
   - Curl_ssl_use has been replaced with Cur_conn_is_ssl
   - Curl_conn_tcp_accepted_set has been split into
     Curl_conn_tcp_listen_set and Curl_conn_tcp_accepted_set
     with a clearer purpose

Closes #10141
2022-12-30 16:43:19 +01:00
Nick Banks
ae98b85020
msh3: add support for request payload
Closes #10136
2022-12-28 13:52:58 +01:00
Stefan Eissing
af22c2a546
vtls: localization of state data in filters
- almost all backend calls pass the Curl_cfilter intance instead of
   connectdata+sockindex
 - ssl_connect_data is remove from struct connectdata and made internal
   to vtls
 - ssl_connect_data is allocated in the added filter, kept at cf->ctx

 - added function to let a ssl filter access its ssl_primary_config and
   ssl_config_data this selects the propert subfields in conn and data,
   for filters added as plain or proxy
 - adjusted all backends to use the changed api
 - adjusted all backends to access config data via the exposed
   functions, no longer using conn or data directly

cfilter renames for clear purpose:

 - methods `Curl_conn_*(data, conn, sockindex)` work on the complete
   filter chain at `sockindex` and connection `conn`.
 - methods `Curl_cf_*(cf, ...)` work on a specific Curl_cfilter
   instance.
 - methods `Curl_conn_cf()` work on/with filter instances at a
   connection.
 - rebased and resolved some naming conflicts
 - hostname validation (und session lookup) on SECONDARY use the same
   name as on FIRST (again).

new debug macros and removing connectdata from function signatures where not
needed.

adapting schannel for new Curl_read_plain paramter.

Closes #9919
2022-11-22 14:25:50 +01:00
Daniel Stenberg
3f039dfd6f
strcase: use curl_str(n)equal for case insensitive matches
No point in having two entry points for the same functions.

Also merged the *safe* function treatment into these so that they can
also be used when one or both pointers are NULL.

Closes #9837
2022-11-01 17:01:26 +01:00
Ayesh Karunaratne
4484270afc
misc: typo and grammar fixes
- Replace `Github` with `GitHub`.
- Replace `windows` with `Windows`
- Replace `advice` with `advise` where a verb is used.
- A few fixes on removing repeated words.
- Replace `a HTTP` with `an HTTP`

Closes #9802
2022-10-27 10:01:30 +02:00
Daniel Stenberg
664249d095
ws: initial websockets support
Closes #8995
2022-09-09 15:11:14 +02:00
Daniel Stenberg
eafc2b14ac
http2: make nghttp2 less picky about field whitespace
In nghttp2 1.49.0 it returns error on leading and trailing whitespace in
header fields according to language in the recently shipped RFC 9113.

nghttp2 1.50.0 introduces an option to switch off this strict check and
this change enables this option by default which should make curl behave
more similar to how it did with nghttp2 1.48.0 and earlier.

We might want to consider making this an option in the future.

Closes #9448
2022-09-08 14:52:45 +02:00
max.mehl
ad9bc5976d
copyright: make repository REUSE compliant
Add licensing and copyright information for all files in this repository. This
either happens in the file itself as a comment header or in the file
`.reuse/dep5`.

This commit also adds a Github workflow to check pull requests and adapts
copyright.pl to the changes.

Closes #8869
2022-06-13 09:13:00 +02:00
Daniel Stenberg
21ea13cfe1
http2: reject overly many push-promise headers
Getting more than a thousand of them is rather a sign of some kind of
attack.

Reported-by: Harry Sintonen
Bug: https://hackerone.com/reports/1589847
Closes #8962
2022-06-06 11:53:49 +02:00
steini2000
f9bc378ab7 http2: always debug print stream id in decimal with %u
Prior to this change the stream id shown could be hex or decimal which
was inconsistent and confusing.

Closes https://github.com/curl/curl/pull/8808
2022-05-12 01:31:32 -04:00
Daniel Stenberg
6968fb9d54
lib: remove exclamation marks
... from infof() and failf() calls. Make them less attention seeking.

Closes #8713
2022-04-16 11:55:05 +02:00
Daniel Stenberg
fda4b81635
http2: RST the stream if we stop it on our own will
For the "simulated 304" case the done-call isn't considered "premature"
but since the server didn't close the stream it needs to be reset to
stop delivering data.

Closes #8664
2022-04-01 13:23:04 +02:00
Daniel Stenberg
b5a9680577
http2: handle DONE called for the paused stream
As it could otherwise stall all streams on the connection

Reported-by: Evangelos Foutras
Fixes #8626
Closes #8664
2022-04-01 13:22:58 +02:00
Daniel Stenberg
d1e4a67734
header api: add curl_easy_header and curl_easy_nextheader
Add test 1940 to 1946 to verify.

Closes #8593
2022-03-22 08:24:22 +01:00
Daniel Stenberg
0694037032
http2: move two infof calls to debug-h2-only
and remove a superflous one

Ref: https://github.com/curl/curl/discussions/8498
Closes #8502
2022-02-24 11:00:12 +01:00
HenrikHolst
b807219292
misc: reduce strlen() calls with Curl_dyn_add()
Use STRCONST() to switch from Curl_dyn_add() to Curl_dyn_addn() for
string literals.

Closes #8398
2022-02-09 13:52:47 +01:00
Daniel Stenberg
2a1951519e
http2: fix the array copy to nghttp2_nv
Bug: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=44517
Follow-up to 9f985a11e7
Closes #8414
2022-02-09 12:18:50 +01:00
Daniel Stenberg
9f985a11e7
http2: use Curl_pseudo_headers 2022-02-07 13:56:06 +01:00
Daniel Stenberg
c85178a94c
http2: allow CURLOPT_HTTPHEADER change ":scheme"
The only h2 psuedo header that wasn't previously possible to change by a
user. This change also makes it impossible to send a HTTP/1 header that
starts with a colon, which I don't think anyone does anyway.

The other pseudo headers are possible to change indirectly by doing the
rightly crafted request.

Reported-by: siddharthchhabrap on github
Fixes #8381
Closes #8393
2022-02-07 13:55:57 +01:00
Daniel Stenberg
136f3e9d68
h2/h3: provide and refer to pseudo headers as defines
... and do sizeof() on the defines to use constants better.

Closes #8389
2022-02-05 11:08:43 +01:00
Daniel Stenberg
2610142139
lib: remove support for CURL_DOES_CONVERSIONS
TPF was the only user and support for that was dropped.

Closes #8378
2022-02-04 08:05:35 +01:00
Daniel Stenberg
b3f9c1aa09
http2:set_transfer_url() return early on OOM
If curl_url() returns NULL this should return early to avoid mistakes -
even if right now the subsequent function invokes are all OK.

Coverity (wrongly) pointed out this as a NULL deref.

Closes #8100
2021-12-06 14:32:29 +01:00
Daniel Stenberg
1fed8feaf1
http2: make getsock not wait for write if there's no remote window
While uploading, check for remote window availability in the getsock
function so that we don't wait for a writable socket if no data can be
sent.

Reported-by: Steini2000 on github
Fixes #7821
Closes #7839
2021-10-12 11:19:22 +02:00
Daniel Stenberg
901804ef95
Curl_http2_setup: don't change connection data on repeat invokes
Regression from 3cb8a74867 (releasde in 7.79.0). That change moved
transfer oriented inits to before the check but also erroneously moved a
few connection oriented ones, which causes problems.

Reported-by: Evangelos Foutras
Fixes #7730
Closes #7731
2021-09-16 11:23:55 +02:00
Daniel Stenberg
3cb8a74867
http2: Curl_http2_setup needs to init stream data in all invokes
Thus function was written to avoid doing multiple connection data
initializations, which is fine, but since it also initiates stream
related data it is crucial that it doesn't skip those even if called
again for the same connection. Solved by moving the stream
initializations before the "doing-it-again" check.

Reported-by: Inho Oh
Fixes #7630
Closes #7692
2021-09-11 23:01:23 +02:00
Daniel Stenberg
c364f5f6fa
Revert "http2: skip immediate parsing of payload following protocol switch"
This reverts commit 455a63c66f.

Reported-by: Tk Xiong
Fixes #7633
Closes #7648
2021-08-30 17:57:32 +02:00
Artur Sinila
ccffb6ec16
http2: revert call the handle-closed function correctly on closed stream
Reverts 252790c533

Assisted-by: Gergely Nagy
Fixes #7400
Closes #7525
2021-08-16 08:44:52 +02:00
Daniel Stenberg
e7416cfd2b
infof: remove newline from format strings, always append it
- the data needs to be "line-based" anyway since it's also passed to the
  debug callback/application

- it makes infof() work like failf() and consistency is good

- there's an assert that triggers on newlines in the format string

- Also removes a few instances of "..."

- Removes the code that would append "..." to the end of the data *iff*
  it was truncated in infof()

Closes #7357
2021-07-07 22:54:01 +02:00
Daniel Stenberg
63c7668182
version: turn version number functions into returning void
... as we never use the return codes from them.

Reviewed-by: Daniel Gustafsson
Closes #7319
2021-06-30 23:23:34 +02:00
Jay Satiro
c6cac5a96c http2: Clarify 'Using HTTP2' verbose message
- Change phrasing from multi-use to multiplexing since the former may
  not be as well understood.

Before: * Using HTTP2, server supports multi-use

After: * Using HTTP2, server supports multiplexing

Bug: https://github.com/curl/curl/discussions/7255
Reported-by: David Hu

Closes https://github.com/curl/curl/pull/7258
2021-06-15 03:18:45 -04:00
Shikha Sharma
a0709f9951
http2_connisdead: handle trailing GOAWAY better
When checking the connection the input processing returns error
immediately, we now consider that a dead connnection.

Bug: https://curl.se/mail/lib-2021-06/0001.html
Closes #7192
2021-06-05 23:45:31 +02:00
Daniel Stenberg
ef1a917a11
http2: init recvbuf struct for pushed streams
Debug builds would warn that these structs were not initialized properly
for pushed streams.

Ref: #7148
Closes #7153
2021-05-31 20:41:51 +02:00
Daniel Stenberg
0c55fbab45
conn: add 'attach' to protocol handler, make libssh2 use it
The libssh2 backend has SSH session associated with the connection but
the callback context is the easy handle, so when a connection gets
attached to a transfer, the protocol handler now allows for a custom
function to get used to set things up correctly.

Reported-by: Michael O'Farrell
Fixes #6898
Closes #7078
2021-05-17 17:57:50 +02:00
Daniel Stenberg
904b27d18d
http2: make sure pause is done on HTTP
Since the function is called for any protocol, we can't assume that the
HTTP struct is there without first making sure it is HTTP.

Reported-by: Denis Goleshchikhin
Fixes #7079
Closes #7080
2021-05-17 14:44:13 +02:00
Peng-Yu Chen
455a63c66f
http2: skip immediate parsing of payload following protocol switch
This is considered not harmful as a following http2_recv shall be
called very soon.

This is considered helpful in the specific situation where some
servers (e.g. nghttpx v1.43.0) may fulfill stream 1 immediately
following the return of HTTP status 101, other than waiting for
the client-side connection preface to arrive.

Fixes #7036
Closes #7040
2021-05-11 08:03:22 +02:00
Peng-Yu Chen
651a75e1e7
http2: use nghttp2_session_upgrade2 instead of nghttp2_session_upgrade
Following the upstream deprecation of nghttp2_session_upgrade.

Also provides further checks for requests with the HEAD method.

Closes #7041
2021-05-11 07:58:14 +02:00
Jay Satiro
ba0d3c28ca http2: fix potentially uninitialized variable
introduced several days ago in 3193170. caught by visual studio linker.
2021-05-05 03:16:06 -04:00
Kamil Dudka
3a6058cb97
http2: fix a resource leak in push_promise()
... detected by Coverity:

Error: RESOURCE_LEAK (CWE-772):
lib/http2.c:532: alloc_fn: Storage is returned from allocation function "duphandle".
lib/http2.c:532: var_assign: Assigning: "newhandle" = storage returned from "duphandle(data)".
lib/http2.c:552: noescape: Resource "newhandle" is not freed or pointed-to in "set_transfer_url".
lib/http2.c:555: leaked_storage: Variable "newhandle" going out of scope leaks the storage it points to.

Closes #6986
2021-05-01 23:40:14 +02:00
Kamil Dudka
3193170470
http2: fix resource leaks in set_transfer_url()
... detected by Coverity:

Error: RESOURCE_LEAK (CWE-772):
lib/http2.c:480: alloc_fn: Storage is returned from allocation function "curl_url". [Note: The source code implementation of the function has been overridden by a builtin model.]
lib/http2.c:480: var_assign: Assigning: "u" = storage returned from "curl_url()".
lib/http2.c:486: noescape: Resource "u" is not freed or pointed-to in "curl_url_set". [Note: The source code implementation of the function has been overridden by a builtin model.]
lib/http2.c:488: leaked_storage: Variable "u" going out of scope leaks the storage it points to.

Error: RESOURCE_LEAK (CWE-772):
lib/http2.c:480: alloc_fn: Storage is returned from allocation function "curl_url". [Note: The source code implementation of the function has been overridden by a builtin model.]
lib/http2.c:480: var_assign: Assigning: "u" = storage returned from "curl_url()".
lib/http2.c:493: noescape: Resource "u" is not freed or pointed-to in "curl_url_set". [Note: The source code implementation of the function has been overridden by a builtin model.]
lib/http2.c:495: leaked_storage: Variable "u" going out of scope leaks the storage it points to.

Error: RESOURCE_LEAK (CWE-772):
lib/http2.c:480: alloc_fn: Storage is returned from allocation function "curl_url". [Note: The source code implementation of the function has been overridden by a builtin model.]
lib/http2.c:480: var_assign: Assigning: "u" = storage returned from "curl_url()".
lib/http2.c:500: noescape: Resource "u" is not freed or pointed-to in "curl_url_set". [Note: The source code implementation of the function has been overridden by a builtin model.]
lib/http2.c:502: leaked_storage: Variable "u" going out of scope leaks the storage it points to.

Error: RESOURCE_LEAK (CWE-772):
lib/http2.c:480: alloc_fn: Storage is returned from allocation function "curl_url". [Note: The source code implementation of the function has been overridden by a builtin model.]
lib/http2.c:480: var_assign: Assigning: "u" = storage returned from "curl_url()".
lib/http2.c:505: noescape: Resource "u" is not freed or pointed-to in "curl_url_get". [Note: The source code implementation of the function has been overridden by a builtin model.]
lib/http2.c:507: leaked_storage: Variable "u" going out of scope leaks the storage it points to.

Closes #6986
2021-05-01 23:39:18 +02:00
Daniel Stenberg
063d3f3b96
tidy-up: make conditional checks more consistent
... remove '== NULL' and '!= 0'

Closes #6912
2021-04-22 09:10:17 +02:00
Daniel Stenberg
d8c4cdecb7
http2: remove DEBUG_HTTP2
Accidentally committed in 605e84235
2021-04-21 11:34:10 +02:00
Daniel Stenberg
605e842355
http2: move the stream error field to the per-transfer storage
Storing a stream error in the per-connection struct was an error that lead to
race conditions as subsequent stream handling could overwrite the error code
before it was used for the stream with the actual problem.

Closes #6910
2021-04-21 08:20:24 +02:00
Daniel Stenberg
252790c533
http2: call the handle-closed function correctly on closed stream
This was this one condition where the stream could be closed due to an
error and the function would still wrongly just return 0 for it.

Reported-by: Gergely Nagy
Fixes #6862
Closes #6910
2021-04-21 08:20:24 +02:00
Daniel Stenberg
95cbcec8f9
urldata: merge "struct DynamicStatic" into "struct UrlState"
Both were used for the same purposes and there was no logical separation
between them. Combined, this also saves 16 bytes in less holes in my
test build.

Closes #6798
2021-03-26 23:19:20 +01:00
Robert Ronto
068661bb02
http2: don't set KEEP_SEND when there's no more data to be sent
this should fix an issue where curl sometimes doesn't send out a request
with authorization info after a 401 is received over http2

Closes #6747
2021-03-15 23:29:35 +01:00
oxalica
d1f40078c1
http2: fail if connection terminated without END_STREAM
Closes #6736
2021-03-12 17:54:39 +01:00
Daniel Stenberg
723c0e75fa
http2: remove conn->data use
... but instead use a private alternative that points to the "driving
transfer" from the connection. We set the "user data" associated with
the connection to be the connectdata struct, but when we drive transfers
the code still needs to know the pointer to the transfer. We can change
the user data to become the Curl_easy handle, but with older nghttp2
version we cannot dynamically update that pointer properly when
different transfers are used over the same connection.

Closes #6520
2021-02-15 16:33:53 +01:00
Daniel Stenberg
1c3def53c7
urldata: remove duplicate 'upkeep_interval_ms' from connectdata
... and rely only on the value already set in Curl_easy.

Closes #6534
2021-01-27 09:19:08 +01:00
Daniel Stenberg
215db086e0
lib: pass in 'struct Curl_easy *' to most functions
... in most cases instead of 'struct connectdata *' but in some cases in
addition to.

- We mostly operate on transfers and not connections.

- We need the transfer handle to log, store data and more. Everything in
  libcurl is driven by a transfer (the CURL * in the public API).

- This work clarifies and separates the transfers from the connections
  better.

- We should avoid "conn->data". Since individual connections can be used
  by many transfers when multiplexing, making sure that conn->data
  points to the current and correct transfer at all times is difficult
  and has been notoriously error-prone over the years. The goal is to
  ultimately remove the conn->data pointer for this reason.

Closes #6425
2021-01-17 23:56:09 +01:00
Daniel Stenberg
aba01da639
failf: remove newline from formatting strings
... as failf adds one itself.

Also: add an assert() to failf() that triggers on a newline in the
format string!

Closes #6365
2020-12-25 23:42:16 +01:00
Daniel Stenberg
c7f95fa0cc
h2: do not wait for RECV on paused transfers
... as the socket might be readable all the time when paused and thus
causing a busy-loop.

Reported-by: Harry Sintonen
Reviewed-by: Jay Satiro
Fixes #6356
Closes #6357
2020-12-22 12:44:18 +01:00
Rikard Falkeborn
920f49a20b
infof/failf calls: fix format specifiers
Update a few format specifiers to match what is being printed.

Closes #6241
2020-11-24 13:18:41 +01:00
Daniel Stenberg
a95a6ce6b8
urldata: remove 'void *protop' and create the union 'p'
... to avoid the use of 'void *' for the protocol specific structs done
per transfer.

Closes #6238
2020-11-23 16:16:16 +01:00
Daniel Stenberg
4d2f800677
curl.se: new home
Closes #6172
2020-11-04 23:59:47 +01:00
Daniel Stenberg
d70a5b5a0f
sendf: move the verbose-check into Curl_debug
Saves us from having the same check done everywhere.

Closes #6159
2020-11-02 22:49:36 +01:00
Daniel Stenberg
6434a73984
Curl_handler: add 'family' to each protocol
Makes get_protocol_family() faster and it moves the knowledge about the
"families" to each protocol handler, where it belongs.

Closes #5986
2020-09-21 23:30:16 +02:00
Laramie Leavitt
25a25f45ae
http: consolidate nghttp2_session_mem_recv() call paths
Previously there were several locations that called
nghttp2_session_mem_recv and handled responses slightly differently.
Those have been converted to call the existing
h2_process_pending_input() function.

Moved the end-of-session check to h2_process_pending_input() since the
only place the end-of-session state can change is after nghttp2
processes additional input frames.

This will likely fix the fuzzing error. While I don't have a root cause
the out-of-bounds read seems like a use after free, so moving the
nghttp2_session_check_request_allowed() call to a location with a
guaranteed nghttp2 session seems reasonable.

Also updated a few nghttp2 callsites to include error messages and added
a few additional error checks.

Closes #5648
2020-09-10 17:43:47 +02:00
Daniel Stenberg
7f187d897c
h2: repair trailer handling
The previous h2 trailer fix in 54a2b63 was wrong and caused a
regression: it cannot deal with trailers immediately when read since
they may be read off the connection by the wrong 'data' owner.

This change reverts the logic back to gathering all trailers into a
single buffer, like before 54a2b63.

Reported-by: Tadej Vengust
Fixes #5663
Closes #5769
2020-08-03 23:43:24 +02:00
Daniel Stenberg
954cd3eb48
CURL_PUSH_ERROROUT: allow the push callback to fail the parent stream
... by adding support for a new dedicated return code.

Suggested-by: Jonathan Cardoso
Assisted-by: Erik Johansson
URL: https://curl.haxx.se/mail/lib-2020-06/0099.html
Closes #5636
2020-07-16 00:24:29 +02:00
Daniel Stenberg
69bfde4515
http2: only do the *done() cleanups for HTTP
Follow-up to ef86daf4d3

Closes #5650
Fixes #5646
2020-07-04 01:13:50 +02:00
Laramie Leavitt
ef86daf4d3
http2: close the http2 connection when no more requests may be sent
Well-behaving HTTP2 servers send two GOAWAY messages. The first
message is a warning that indicates that the server is going to
stop accepting streams. The second one actually closes the stream.

nghttp2 reports this state (and the other state of no more stream
identifiers) via the call nghttp2_session_check_request_allowed().
In this state the client should not create more streams on the
session (tcp connection), and in curl this means that the server
has requested that the connection is closed.

It would be also be possible to put the connclose() call into the
on_http2_frame_recv() function that triggers on the GOAWAY message.

This fixes a bug seen when the client sees the following sequence of
frames:

// advisory GOAWAY
HTTP2 GOAWAY [stream-id = 0, promised-stream-id = -1]
... some additional frames

// final GOAWAY
HTTP2 GOAWAY [stream-id = 0, promised-stream-id = N ]

Before this change, curl will attempt to reuse the connection even
after the last stream, will encounter this error:

* Found bundle for host localhost: 0x5595f0a694e0 [can multiplex]
* Re-using existing connection! (#0) with host localhost
* Connected to localhost (::1) port 10443 (#0)
* Using Stream ID: 9 (easy handle 0x5595f0a72e30)
> GET /index.html?5 HTTP/2
> Host: localhost:10443
> user-agent: curl/7.68.0
> accept: */*
>
* stopped the pause stream!
* Connection #0 to host localhost left intact
curl: (16) Error in the HTTP2 framing layer

This error may posion the connection cache, causing future requests
which resolve to the same curl connection to go through the same error
path.

Closes #5643
2020-07-03 11:16:48 +02:00
Jeremy Maitin-Shepard
31f0e864c7
http2: fix nghttp2_strerror -> nghttp2_http2_strerror in debug messages
Confusingly, nghttp2 has two different error code enums:

- nghttp2_error, to be used with nghttp2_strerror
- nghttp2_error_code, to be used with nghttp2_http2_strerror

Closes #5641
2020-07-02 23:37:28 +02:00
Daniel Stenberg
646cc574f0
http2: set the correct URL in pushed transfers
...previously CURLINFO_EFFECTIVE_URL would report the URL of the
original "mother transfer", not the actually pushed resource.

Reported-by: Jonathan Cardoso Machado
Fixes #5589
Closes #5591
2020-06-25 09:31:21 +02:00
Daniel Stenberg
9c845be279
urldata: let the HTTP method be in the set.* struct
When the method is updated inside libcurl we must still not change the
method as set by the user as then repeated transfers with that same
handle might not execute the same operation anymore!

This fixes the libcurl part of #5462

Test 1633 added to verify.

Closes #5499
2020-06-02 16:30:36 +02:00
Daniel Stenberg
c4e6968127
url: alloc the download buffer at transfer start
... and free it as soon as the transfer is done. It removes the extra
alloc when a new size is set with setopt() and reduces memory for unused
easy handles.

In addition: the closure_handle now doesn't use an allocated buffer at
all but the smallest supported size as a stack based one.

Closes #5472
2020-05-30 23:14:33 +02:00
Vyron Tsingaras
34a5400de1
http2: keep trying to send pending frames after req.upload_done
Fixes #1410
Closes #5401
2020-05-15 15:49:18 +02:00
Daniel Stenberg
54a2b63c70
http2: simplify and clean up trailer handling
Triggered by a crash detected by OSS-Fuzz after the dynbuf introduction in
ed35d6590e. This should make the trailer handling more straight forward and
hopefully less error-prone.

Deliver the trailer header to the callback already at receive-time. No
longer caches the trailers to get delivered at end of stream.

Bug: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=22030
Closes #5348
2020-05-07 09:49:51 +02:00
Daniel Stenberg
ed35d6590e
dynbuf: introduce internal generic dynamic buffer functions
A common set of functions instead of many separate implementations for
creating buffers that can grow when appending data to them. Existing
functionality has been ported over.

In my early basic testing, the total number of allocations seem at
roughly the same amount as before, possibly a few less.

See docs/DYNBUF.md for a description of the API.

Closes #5300
2020-05-04 10:40:39 +02:00
Jay Satiro
347a374c56 http2: Fix erroneous debug message that h2 connection closed
Prior to this change in libcurl debug builds http2 stream closure was
erroneously referred to as connection closure.

Before:
* nread <= 0, server closed connection, bailing

After:
* nread == 0, stream closed, bailing

Closes https://github.com/curl/curl/pull/5118
2020-03-18 03:39:27 -04:00
Daniel Stenberg
e364546fb3
version: make curl_version* thread-safe without using global context
Closes #5010
2020-03-07 12:10:11 +01:00
Daniel Stenberg
15f51474c8
http2: make pausing/unpausing set/clear local stream window
This reduces the HTTP/2 window size to 32 MB since libcurl might have to
buffer up to this amount of data in memory and yet we don't want it set
lower to potentially impact tranfer performance on high speed networks.

Requires nghttp2 commit b3f85e2daa629
(https://github.com/nghttp2/nghttp2/pull/1444) to work properly, to end
up in the next release after 1.40.0.

Fixes #4939
Closes #4940
2020-02-27 10:35:51 +01:00
Daniel Stenberg
5808a0d0f5
http2: now require nghttp2 >= 1.12.0
To simplify our code and since earlier versions lack important function
calls libcurl needs to function correctly.

nghttp2 1.12.0 was relased on June 26, 2016.

Closes #4961
2020-02-21 16:54:47 +01:00
Daniel Stenberg
9607532873
ConnectionExists: respect the max_concurrent_streams limits
A regression made the code use 'multiplexed' as a boolean instead of the
counter it is intended to be. This made curl try to "over-populate"
connections with new streams.

This regression came with 41fcdf71a1, shipped in curl 7.65.0.

Also, respect the CURLMOPT_MAX_CONCURRENT_STREAMS value in the same
check.

Reported-by: Kunal Ekawde
Fixes #4779
Closes #4784
2020-01-13 15:44:58 +01:00
Daniel Stenberg
ee263de7a3
conncache: fix multi-thread use of shared connection cache
It could accidentally let the connection get used by more than one
thread, leading to double-free and more.

Reported-by: Christopher Reid
Fixes #4544
Closes #4557
2019-12-09 15:30:09 +01:00
Jay Satiro
9c1806ae46 build: Disable Visual Studio warning "conditional expression is constant"
- Disable warning C4127 "conditional expression is constant" globally
  in curl_setup.h for when building with Microsoft's compiler.

This mainly affects building with the Visual Studio project files found
in the projects dir.

Prior to this change the cmake and winbuild build systems already
disabled 4127 globally for when building with Microsoft's compiler.
Also, 4127 was already disabled for all build systems in the limited
circumstance of the WHILE_FALSE macro which disabled the warning
specifically for while(0). This commit removes the WHILE_FALSE macro and
all other cruft in favor of disabling globally in curl_setup.

Background:

We have various macros that cause 0 or 1 to be evaluated, which would
cause warning C4127 in Visual Studio. For example this causes it:

    #define Curl_resolver_asynch() 1

Full behavior is not clearly defined and inconsistent across versions.
However it is documented that since VS 2015 Update 3 Microsoft has
addressed this somewhat but not entirely, not warning on while(true) for
example.

Prior to this change some C4127 warnings occurred when I built with
Visual Studio using the generated projects in the projects dir.

Closes https://github.com/curl/curl/pull/4658
2019-12-01 19:01:02 -05:00
Daniel Stenberg
dcd7e37c3a
url: make Curl_close() NULLify the pointer too
This is the common pattern used in the code and by a unified approach we
avoid mistakes.

Closes #4534
2019-10-30 09:36:21 +01:00
Daniel Stenberg
95a4cfd888
http2_recv: a closed stream trumps pause state
... and thus should return 0, not EAGAIN.

Reported-by: Tom van der Woerdt
Fixes #4496
Closes #4505
2019-10-18 23:23:48 +02:00
Daniel Stenberg
59041f052a
http2: expire a timeout at end of stream
To make sure that transfer is being dealt with. Streams without
Content-Length need a final read to notice the end-of-stream state.

Reported-by: Tom van der Woerdt
Fixes #4496
2019-10-18 23:23:43 +02:00
Daniel Stenberg
8a00560de2
http2: move state-init from creation to pre-transfer
To make sure that the HTTP/2 state is initialized correctly for
duplicated handles. It would otherwise easily generate "spurious"
PRIORITY frames to get sent over HTTP/2 connections when duplicated easy
handles were used.

Reported-by: Daniel Silverstone
Fixes #4303
Closes #4442
2019-10-03 22:55:42 +02:00
Kunal Ekawde
c124e6b3c0
CURLMOPT_MAX_CONCURRENT_STREAMS: new setopt
Closes #4410
2019-10-02 07:47:48 +02:00
Daniel Stenberg
b259baabfe
http2: Expression 'stream->stream_id != - 1' is always true
PVS-Studio warning
Fixes #4402
2019-09-23 22:44:02 +02:00
Daniel Stenberg
f91b82e688
http2: A value is being subtracted from the unsigned variable
PVS-Studio warning
Fixes #4402
2019-09-23 22:44:02 +02:00
Barry Pollard
0023fce38d
http: lowercase headernames for HTTP/2 and HTTP/3
Closes #4401
Fixes #4400
2019-09-23 22:33:31 +02:00
Christoph M. Becker
7c596f5dea
http2: relax verification of :authority in push promise requests
If the :authority pseudo header field doesn't contain an explicit port,
we assume it is valid for the default port, instead of rejecting the
request for all ports.

Ref: https://curl.haxx.se/mail/lib-2019-09/0041.html

Closes #4365
2019-09-16 23:36:22 +02:00
Daniel Stenberg
c1b6a384f9
http2: when marked for closure and wanted to close == OK
It could otherwise return an error even when closed correctly if GOAWAY
had been received previously.

Reported-by: Tom van der Woerdt
Fixes #4267
Closes #4268
2019-08-26 22:44:59 +02:00
Daniel Stenberg
b1616dad8f
timediff: make it 64 bit (if possible) even with 32 bit time_t
... to make it hold microseconds too.

Fixes #4165
Closes #4168
2019-08-01 07:43:24 +02:00
Daniel Stenberg
a55faf33d4
cleanup: remove the 'numsocks' argument used in many places
It was used (intended) to pass in the size of the 'socks' array that is
also passed to these functions, but was rarely actually checked/used and
the array is defined to a fixed size of MAX_SOCKSPEREASYHANDLE entries
that should be used instead.

Closes #4169
2019-07-30 23:16:44 +02:00
Daniel Stenberg
fc6045f5d1
http2_recv: trigger another read when the last data is returned
... so that end-of-stream is detected properly.

Reported-by: Tom van der Woerdt
Fixes #4043
Closes #4160
2019-07-29 22:16:27 +02:00
Daniel Gustafsson
11d7fe118d headers: Remove no longer exported functions
There were a leftover few prototypes of Curl_ functions that we used to
export but no longer do, this removes those prototypes and cleans up any
comments still referring to them.

Curl_write32_le(), Curl_strcpy_url(), Curl_strlen_url(), Curl_up_free()
Curl_concat_url(), Curl_detach_connnection(), Curl_http_setup_conn()
were made static in 05b100aee2.
Curl_http_perhapsrewind() made static in 574aecee20.

For the remainder, I didn't trawl the Git logs hard enough to capture
their exact time of deletion, but they were all gone: Curl_splayprint(),
Curl_http2_send_request(), Curl_global_host_cache_dtor(),
Curl_scan_cache_used(), Curl_hostcache_destroy(), Curl_second_connect(),
Curl_http_auth_stage() and Curl_close_connections().

Closes #4096
Reviewed-by: Daniel Stenberg <daniel@haxx.se>
2019-07-10 19:26:40 +02:00
Daniel Stenberg
265f7f42f6
http2: call done_sending on end of upload
To make sure a HTTP/2 stream registers the end of stream.

Bug #4043 made me find this problem but this fix doesn't correct the
reported issue.

Closes #4068
2019-06-24 23:11:37 +02:00
Daniel Stenberg
c0c40ab075
http2: don't call stream-close on already closed streams
Closes #4055
2019-06-21 00:07:13 +02:00
Daniel Stenberg
ee6945c088
http2: remove CURL_DISABLE_TYPECHECK define
... in http2-less builds as it served no use.
2019-06-11 07:49:24 +02:00