Commit Graph

232 Commits

Author SHA1 Message Date
Daniel Stenberg
9cc246401e
source: avoid use of 'very' in comments 2024-10-17 13:36:33 +02:00
Stefan Eissing
fe8399f066
gnutls: use session cache for QUIC
Add session reuse for QUIC transfers using GnuTLS. This does not include
support for TLS early data, yet.

Fix check of early data support in common GnuTLS init code to not access
the filter context, as the struct varies between TCP and QUIC
connections.

Closes #15265
2024-10-11 23:37:47 +02:00
Stefan Eissing
962097b8dd
TLS: TLSv1.3 earlydata support for curl
Based on #14135, implement TLSv1.3 earlydata support for the curl
command line, libcurl and its implementation in GnuTLS.

If a known TLS session announces early data support, and the feature is
enabled *and* it is not a "connect-only" transfer, delay the TLS
handshake until the first request is being sent.

- Add --tls-earldata as new boolean command line option for curl.
- Add CURLSSLOPT_EARLYDATA to libcurl to enable use of the feature.
- Add CURLINFO_EARLYDATA_SENT_T to libcurl, reporting the amount of
  bytes sent and accepted/rejected by the server.

Implementation details:
- store the ALPN protocol selected at the SSL session.
- When reusing the session and enabling earlydata, use exactly
  that ALPN protocol for negoptiation with the server. When the
  sessions ALPN does not match the connections ALPN, earlydata
  will not be enabled.
- Check that the server selected the correct ALPN protocol for
  an earlydata connect. If the server does not confirm or reports
  something different, the connect fails.
- HTTP/2: delay sending the initial SETTINGS frames during connect,
  if not connect-only.

Verification:
- add test_02_32 to verify earlydata GET with nghttpx.
- add test_07_70 to verify earlydata PUT with nghttpx.
- add support in 'hx-download', 'hx-upload' clients for the feature

Assisted-by: ad-chaos on github
Closes #15211
2024-10-11 12:28:22 +02:00
Daniel Stenberg
bcec0840b0
lib: use bool/TRUE/FALSE properly
booleans should use the type 'bool' and set the value to TRUE/FALSE

non-booleans should not be 'bool' and should not set the value to
TRUE/FALSE

Closes #15123
2024-10-03 09:31:56 +02:00
Tatsuhiro Tsujikawa
7307c1a289
gtls: Add P12 format support
This change adds P12 format support for GnuTLS backend.

Closes #14991
2024-09-22 21:24:55 +02:00
Daniel Stenberg
fbf5d507ce
lib/src: white space edits to comply better with code style
... as checksrc now finds and complains about these.

Closes #14921
2024-09-19 14:59:12 +02:00
Stefan Eissing
6354b35dfe
gnutls: send all data
Turns out `gnutls_record_send()` does really what the name says: it
sends exactly one TLS record. If more than 16k are there to send, it
needs to be called again with new buffer offset and length.

Continue sending record until the input is all sent or a EAGAIN (or
fatal error) is returned by gnutls.

Closes #14722
2024-08-29 16:46:56 +02:00
Daniel Stenberg
aeb1a281ca
gtls: fix OCSP stapling management
Reported-by: Hiroki Kurosawa
Closes #14642
2024-08-22 11:11:14 +02:00
Stefan Eissing
623b877504
gnutls/wolfssl: improve error message when certificate fails
Give more detailed reasons for certificate failures where available in
gnutls and wolfssi to allow user to understand the cause of the failure.

Closes #14501
2024-08-14 11:47:01 +02:00
Viktor Szakats
71d3ab5813
vtls: fix static function name collisions between TLS backends
When using CMake Unity build.

- use unique name for `set_ssl_version_min_max()`
  Fixes collision between GnuTLS, mbedTLS and SecureTransport.
  ```
  lib\vtls\mbedtls.c(317,1): error C2084: function 'CURLcode set_ssl_version_min_max(Curl_easy *,ssl_peer *,ssl_primary_config *,const char **,const char *)' already has a body
  lib\vtls\mbedtls.c(837,49): warning C4133: 'function': incompatible types - from 'Curl_cfilter *' to 'Curl_easy *'
  lib\vtls\mbedtls.c(837,53): warning C4133: 'function': incompatible types - from 'Curl_easy *' to 'ssl_peer *'
  lib\vtls\mbedtls.c(837,25): error C2198: 'set_ssl_version_min_max': too few arguments for call
  ```

- use unique name for `do_file_type()`
  Fixes collision between GnuTLS, OpenSSL and wolfSSL.
  ```
  lib\vtls\openssl.c(1053,12): error C2084: function 'gnutls_x509_crt_fmt_t do_file_type(const char *)' already has a body
  ```

Ref: https://github.com/curl/curl/actions/runs/10341162641/job/28622681573?pr=14484#step:10:31
Cherry-picked from #14495
Closes #14516
2024-08-13 09:28:27 +02:00
Max Faxälv
0a5ea09a91
spnego_gssapi: implement TLS channel bindings for openssl
Channel Bindings are used to tie the session context to a specific TLS
channel. This is to provide additional proof of valid identity,
mitigating authentication relay attacks.

Major web servers have the ability to require (None/Accept/Require)
GSSAPI channel binding, rendering Curl unable to connect to such
websites unless support for channel bindings is implemented.

IIS calls this feature Extended Protection (EPA), which is used in
Enterprise environments using Kerberos for authentication.

This change require krb5 >= 1.19, otherwise channel bindings won't be
forwarded through SPNEGO.

Co-Authored-By: Steffen Kieß <947515+steffen-kiess@users.noreply.github.com>
Closes #13098
2024-08-12 19:16:54 +02:00
Stefan Eissing
35bf766280
http2: improved upload eos handling
- replace the counting of upload lengths with the new eos send flag
- improve frequency of stream draining to happen less on events where it
  is not needed
- this PR is based on #14220

http2, cf-h2-proxy: fix EAGAINed out buffer
- in adjust pollset and shutdown handling, a non-empty `ctx->outbufq`
  must trigger send polling, irregardless of http/2 flow control
- in http2, fix retry handling of blocked GOAWAY frame

test case improvement:
- let client 'upload-pausing' handle http versions

Closes #14253
2024-08-05 08:53:55 +02:00
Stefan Eissing
911c3166b6
lib: add eos flag to send methods
Adds a `bool eos` flag to send methods to indicate that the data
is the last chunk the invovled transfer wants to send to the server.

This will help protocol filters like HTTP/2 and 3 to forward the
stream's EOF flag and also allow to EAGAIN such calls when buffers
are not yet fully flushed.

Closes #14220
2024-08-03 19:53:54 +02:00
Daniel Stenberg
25321de30e
Revert "lib: send eos flag"
This reverts commit be93299f10.
2024-07-19 01:38:05 +02:00
Stefan Eissing
be93299f10
lib: send eos flag
Adds a `bool eos` flag to send methods to indicate that the data is the
last chunk the invovled transfer wants to send to the server.

This will help protocol filters like HTTP/2 and 3 to forward the
stream's EOF flag and also allow to EAGAIN such calls when buffers are
not yet fully flushed.

Closes #14220
2024-07-18 23:27:35 +02:00
Viktor Szakats
6343034dd1
tidy-up: adjust casing of project names
Mostly TLS/SSH project name.

Closes #14160
2024-07-12 13:56:16 +02:00
Stefan Eissing
46a26f122a
vtls: replace addsessionid with set_sessionid
- deduplicate the code in many tls backends that check
  for an existing id and delete it before adding the new one
- rename ssl_primary_config's `sessionid` bool to `cache_session`

Closes #14121
2024-07-09 23:14:58 +02:00
Daniel Stenberg
c074ba64a8
code: language cleanup in comments
Based on the standards and guidelines we use for our documentation.

 - expand contractions (they're => they are etc)
 - host name = > hostname
 - file name => filename
 - user name = username
 - man page => manpage
 - run-time => runtime
 - set-up => setup
 - back-end => backend
 - a HTTP => an HTTP
 - Two spaces after a period => one space after period

Closes #14073
2024-07-01 22:58:55 +02:00
Stefan Eissing
c9b95c0bb3
lib: graceful connection shutdown
When libcurl discards a connection there are two phases this may go
through: "shutdown" and "closing". If a connection is aborted, the
shutdown phase is skipped and it is closed right away.

The connection filters attached to the connection implement the phases
in their `do_shutdown()` and `do_close()` callbacks. Filters carry now a
`shutdown` flags next to `connected` to keep track of the shutdown
operation.

Filters are shut down from top to bottom. If a filter is not connected,
its shutdown is skipped. Notable filters that *do* something during
shutdown are HTTP/2 and TLS. HTTP/2 sends the GOAWAY frame. TLS sends
its close notify and expects to receive a close notify from the server.

As sends and receives may EAGAIN on the network, a shutdown is often not
successful right away and needs to poll the connection's socket(s). To
facilitate this, such connections are placed on a new shutdown list
inside the connection cache.

Since managing this list requires the cooperation of a multi handle,
only the connection cache belonging to a multi handle is used. If a
connection was in another cache when being discarded, it is removed
there and added to the multi's cache. If no multi handle is available at
that time, the connection is shutdown and closed in a one-time,
best-effort attempt.

When a multi handle is destroyed, all connection still on the shutdown
list are discarded with a final shutdown attempt and close. In curl
debug builds, the environment variable `CURL_GRACEFUL_SHUTDOWN` can be
set to make this graceful with a timeout in milliseconds given by the
variable.

The shutdown list is limited to the max number of connections configured
for a multi cache. Set via CURLMOPT_MAX_TOTAL_CONNECTIONS. When the
limit is reached, the oldest connection on the shutdown list is
discarded.

- In multi_wait() and multi_waitfds(), collect all connection caches
  involved (each transfer might carry its own) into a temporary list.
  Let each connection cache on the list contribute sockets and
  POLLIN/OUT events it's connections are waiting for.

- in multi_perform() collect the connection caches the same way and let
  them peform their maintenance. This will make another non-blocking
  attempt to shutdown all connections on its shutdown list.

- for event based multis (multi->socket_cb set), add the sockets and
  their poll events via the callback. When `multi_socket()` is invoked
  for a socket not known by an active transfer, forward this to the
  multi's cache for processing. On closing a connection, remove its
  socket(s) via the callback.

TLS connection filters MUST NOT send close nofity messages in their
`do_close()` implementation. The reason is that a TLS close notify
signals a success. When a connection is aborted and skips its shutdown
phase, the server needs to see a missing close notify to detect
something has gone wrong.

A graceful shutdown of FTP's data connection is performed implicitly
before regarding the upload/download as complete and continuing on the
control connection. For FTP without TLS, there is just the socket close
happening. But with TLS, the sent/received close notify signals that the
transfer is complete and healthy. Servers like `vsftpd` verify that and
reject uploads without a TLS close notify.

- added test_19_* for shutdown related tests
- test_19_01 and test_19_02 test for TCP RST packets
  which happen without a graceful shutdown and should
  no longer appear otherwise.
- add test_19_03 for handling shutdowns by the server
- add test_19_04 for handling shutdowns by curl
- add test_19_05 for event based shutdowny by server
- add test_30_06/07 and test_31_06/07 for shutdown checks
  on FTP up- and downloads.

Closes #13976
2024-06-26 08:33:17 +02:00
Daniel Stenberg
6b93190fae
gnutls: pass in SNI name, not hostname when checking cert
The function we use is called 'gnutls_x509_crt_check_hostname()' but if
we pass in the hostname with a trailing dot, the check fails. If we pass
in the SNI name, which cannot have a trailing dot, it succeeds for
https://pyropus.ca./

I consider this as a flaw in GnuTLS and have submitted this issue
upstream:

  https://gitlab.com/gnutls/gnutls/-/issues/1548

In order to work with old and existing GnuTLS versions, we still need
this change no matter how they view the issue or might change it in the
future.

Fixes #13428
Reported-by: Ryan Carsten Schmidt
Closes #13949
2024-06-14 13:19:20 +02:00
Stefan Eissing
385c62aabc
lib: xfer_setup and non-blocking shutdown
- clarify Curl_xfer_setup() with RECV/SEND flags and different calls for
  which socket they operate on. Add a shutdown flag for secondary
  sockets
- change Curl_xfer_setup() calls to new functions
- implement non-blocking connection shutdown at the end of receiving or
  sending a transfer

Closes #13913
2024-06-11 13:41:03 +02:00
Stefan Eissing
c31041b17e
connection: shutdown TLS (for FTP) better
This adds connection shutdown infrastructure and first use for FTP. FTP
data connections, when not encountering an error, are now shut down in a
blocking way with a 2sec timeout.

    - add cfilter `Curl_cft_shutdown` callback
    - keep a shutdown start timestamp and timeout at connectdata
    - provide shutdown timeout default and member in
      `data->set.shutdowntimeout`.
    - provide methods for starting, interrogating and clearing
      shutdown timers
    - provide `Curl_conn_shutdown_blocking()` to shutdown the
      `sockindex` filter chain in a blocking way. Use that in FTP.
    - add `Curl_conn_cf_poll()` to wait for socket events during
      shutdown of a connection filter chain.
      This gets the monitoring sockets and events via the filters
      "adjust_pollset()" methods. This gives correct behaviour when
      shutting down a TLS connection through a HTTP/2 proxy.
    - Implement shutdown for all socket filters
      - for HTTP/2 and h2 proxying to send GOAWAY
      - for TLS backends to the best of their capabilities
      - for tcp socket filter to make a final, nonblocking
        receive to avoid unwanted RST states
    - add shutdown forwarding to happy eyeballers and
      https connect ballers when applicable.

Closes #13904
2024-06-10 13:08:12 +02:00
Stefan Eissing
937ba94ed5
vtls: new io_need flags for poll handling
- decouple need to recv/send from negotiation state, we need
  this later in shutdown handling as well
- move ssl enums from urldata.h to vtls_int.h
- implement use of `connssl->io_need` in vtls.c. and all backends

Closes #13879
2024-06-05 09:03:38 +02:00
Stefan Eissing
5dd8f13bfc
gnutls: support CA caching
- similar to openssl, use a shared 'credentials' instance
  among TLS connections with a plain configuration.
- different to openssl, a connection with a client certificate
  is not eligible to sharing.
- document CURLOPT_CA_CACHE_TIMEOUT in man page

Closes #13795
2024-06-04 08:17:55 +02:00
Viktor Szakats
0887297100
lib/v*: tidy up types and casts
Also add a couple of negative checks.

Cherry-picked from #13489
Closes #13622
2024-06-02 19:27:17 +02:00
Daniel Stenberg
582743f2e7
openssl/gnutls: rectify the TLS version checks for QUIC
The versions check wrongly complained and return error if the *minimum*
version was set to something less than 1.3. QUIC is always TLS 1.3, but
that means minimum 1.2 is still fine to ask for.

This also renames the local variable to make the mistake harder to make
in the future.

Regression shipped in 8.8.0

Follow-up to 3210101088

Reported-by: fds242 on github
Fixes #13799
Closes #13802
2024-05-28 10:39:02 +02:00
Stefan Eissing
7bbad0c033
gnutls: improve TLS shutdown
local ftp upload tests sometimes failed with an invalid TLS record being
reported by gnutls. vsftp did log that the shutdown was not regarded as
clean, failing the control connection thereafter.

These changes make test_31_05 work reliable locally.

- on closing the SSL filter, shutdown READ *and* WRITE
- on closing, try a receive after shutdown is sent
- convert to DEBUGF to CURL_TRC_CF

Closes #13790
2024-05-28 10:37:52 +02:00
Stefan Eissing
e101a7a8b0
multi: add multi->proto_hash, a key-value store for protocol data
- add `Curl_hash_add2()` that passes a destructor function for
  the element added. Call element destructor instead of hash
  destructor if present.
- multi: add `proto_hash` for protocol related information,
  remove `struct multi_ssl_backend_data`.
- openssl: use multi->proto_hash to keep x509 shared store
- schannel: use multi->proto_hash to keep x509 shared store
- vtls: remove Curl_free_multi_ssl_backend_data() and its
  equivalents in the TLS backends

Closes #13345
2024-05-26 00:15:01 +02:00
Stefan Eissing
fb22459dc1
vtls: TLS session storage overhaul
- add session with destructor callback
- remove vtls `session_free` method
- let `Curl_ssl_addsessionid()` take ownership
  of session object, freeing it also on failures
- change tls backend use
- test_17, add tests for SSL session resumption

Closes #13386
2024-04-26 13:58:36 +02:00
Viktor Szakats
e411c98f70
build: prefer USE_IPV6 macro internally (was: ENABLE_IPV6)
Before this patch, two macros were used to guard IPv6 features in curl
sources: `ENABLE_IPV6` and `USE_IPV6`. This patch makes the source use
the latter for consistency with other similar switches.

`-DENABLE_IPV6` remains accepted for compatibility as a synonym for
`-DUSE_IPV6`, when passed to the compiler.

`ENABLE_IPV6` also remains the name of the CMake and `Makefile.vc`
options to control this feature.

Closes #13349
2024-04-13 08:33:26 +00:00
Stefan Eissing
8cee4c9238
gnutls: lazy init the trust settings
- delay loading of trust anchors and CRLs after the ClientHello
  has been sent off
- add tracing to IO operations
- on IO errors, return the CURLcode of the underlying filter

Closes #13339
2024-04-11 08:59:25 +02:00
Stefan Eissing
e7de80e8ce
tls: fix compile issues on old-linux CI
Follow-up to 3210101088
Closes #13325
2024-04-09 13:26:19 +02:00
Stefan Eissing
3210101088
tls: use shared init code for TCP+QUIC
Closes #13172
2024-04-09 09:08:05 +02:00
MAntoniak
f46385d36d
urldata: remove fields not used depending on used features
Reduced size of dynamically_allocated_data structure.

Reduced number of stored values in enum dupstring and enum dupblob. This
affects the reduced array placed in the UserDefined structure.

Closes #13188
2024-04-05 16:06:22 +02:00
Stefan Eissing
c765b04d11
TLS: start shutdown only when peer did not already close
- When curl sees a TCP close from the peer, do not start a TLS shutdown.
  TLS shutdown is a handshake and if the peer already closed the
  connection, it is not interested in participating.

Reported-by: dfdity on github
Assisted-by: Jiří Bok
Assisted-by: Pēteris Caune
Fixes #10290
Closes #13087
2024-03-15 09:19:58 +01:00
Baruch Siach
af520ac9fe
gnutls: fix build with --disable-verbose
infof() parameters must be defined event with --disable-verbose since
commit dac293cfb7 ("lib: apache style infof and trace
macros/functions").

Move also 'ptr' definition under !CURL_DISABLE_VERBOSE_STRINGS.

Fixes the following build failure:

In file included from ../lib/sendf.h:29,
                 from vtls/gtls.c:44:
vtls/gtls.c: In function 'Curl_gtls_verifyserver':
vtls/gtls.c:841:34: error: 'version' undeclared (first use in this function); did you mean 'session'?
  841 |         gnutls_protocol_get_name(version), ptr);
      |                                  ^~~~~~~

Closes #12505
2023-12-16 22:06:21 +01:00
Daniel Stenberg
7309b9cbbf
lib: strndup/memdup instead of malloc, memcpy and null-terminate
- bufref: use strndup
 - cookie: use strndup
 - formdata: use strndup
 - ftp: use strndup
 - gtls: use aprintf instead of malloc + strcpy * 2
 - http: use strndup
 - mbedtls: use strndup
 - md4: use memdup
 - ntlm: use memdup
 - ntlm_sspi: use strndup
 - pingpong: use memdup
 - rtsp: use strndup instead of malloc, memcpy and null-terminate
 - sectransp: use strndup
 - socks_gssapi.c: use memdup
 - vtls: use dynbuf instead of malloc, snprintf and memcpy
 - vtls: use strdup instead of malloc + memcpy
 - wolfssh: use strndup

Closes #12453
2023-12-07 08:47:44 +01:00
Stefan Eissing
fa714830e9
vtls/vquic, keep peer name information together
- add `struct ssl_peer` to keep hostname, dispname and sni
  for a filter
- allocate `sni` for use in VTLS backend
- eliminate `Curl_ssl_snihost()` and its use of the download buffer
- use ssl_peer in SSL and QUIC filters

Closes #12349
2023-11-19 13:55:22 +01:00
Stefan Eissing
47f5b1a37f
lib: introduce struct easy_poll_set for poll information
Connection filter had a `get_select_socks()` method, inspired by the
various `getsocks` functions involved during the lifetime of a
transfer. These, depending on transfer state (CONNECT/DO/DONE/ etc.),
return sockets to monitor and flag if this shall be done for POLLIN
and/or POLLOUT.

Due to this design, sockets and flags could only be added, not
removed. This led to problems in filters like HTTP/2 where flow control
prohibits the sending of data until the peer increases the flow
window. The general transfer loop wants to write, adds POLLOUT, the
socket is writeable but no data can be written.

This leads to cpu busy loops. To prevent that, HTTP/2 did set the
`SEND_HOLD` flag of such a blocked transfer, so the transfer loop cedes
further attempts. This works if only one such filter is involved. If a
HTTP/2 transfer goes through a HTTP/2 proxy, two filters are
setting/clearing this flag and may step on each other's toes.

Connection filters `get_select_socks()` is replaced by
`adjust_pollset()`. They get passed a `struct easy_pollset` that keeps
up to `MAX_SOCKSPEREASYHANDLE` sockets and their `POLLIN|POLLOUT`
flags. This struct is initialized in `multi_getsock()` by calling the
various `getsocks()` implementations based on transfer state, as before.

After protocol handlers/transfer loop have set the sockets and flags
they want, the `easy_pollset` is *always* passed to the filters. Filters
"higher" in the chain are called first, starting at the first
not-yet-connection one. Each filter may add sockets and/or change
flags. When all flags are removed, the socket itself is removed from the
pollset.

Example:

 * transfer wants to send, adds POLLOUT
 * http/2 filter has a flow control block, removes POLLOUT and adds
   POLLIN (it is waiting on a WINDOW_UPDATE from the server)
 * TLS filter is connected and changes nothing
 * h2-proxy filter also has a flow control block on its tunnel stream,
   removes POLLOUT and adds POLLIN also.
 * socket filter is connected and changes nothing
 * The resulting pollset is then mixed together with all other transfers
   and their pollsets, just as before.

Use of `SEND_HOLD` is no longer necessary in the filters.

All filters are adapted for the changed method. The handling in
`multi.c` has been adjusted, but its state handling the the protocol
handlers' `getsocks` method are untouched.

The most affected filters are http/2, ngtcp2, quiche and h2-proxy. TLS
filters needed to be adjusted for the connecting handshake read/write
handling.

No noticeable difference in performance was detected in local scorecard
runs.

Closes #11833
2023-10-25 09:34:32 +02:00
Daniel Stenberg
9cf4759354
gnutls: support CURLSSLOPT_NATIVE_CA
Remove the CURL_CA_FALLBACK logic. That build option was added to allow
primarily OpenSSL to use the default paths for loading the CA certs. For
GnuTLS it was instead made to load the "system certs", which is
different and not desirable.

The native CA store loading is now asked for with this option.

Follow-up to 7b55279d1d

Co-authored-by: Jay Satiro

Closes #12137
2023-10-21 14:41:06 +02:00
Viktor Szakats
781242ffa4
cmake: detect TLS-SRP in OpenSSL/wolfSSL/GnuTLS
With new option `CURL_DISABLE_SRP=ON` to force-disable it.
To match existing option and detection logic in autotools.

Also:
- fix detecting GnuTLS.
  We assume `nettle` as a GnuTLS dependency.
- add CMake GnuTLS CI job.
- bump AppVeyor CMake OpenSSL MSVC job to OpenSSL 1.1.1 (from 1.0.2)
  TLS-SRP fails to detect with 1.0.2 due to an OpenSSL header bug.
- fix compiler warning when building with GnuTLS and disabled TLS-SRP.
- fix comment typos, whitespace.

Ref: #11964

Closes #11967
2023-09-28 10:50:56 +00:00
Daniel Stenberg
5e2beb3395
spelling: use 'reuse' not 're-use' in code and elsewhere
Unify the spelling as both versions were previously used intermittently

Closes #11717
2023-08-23 23:22:36 +02:00
Viktor Szakats
3f8fc25720
cmake: add support for "unity" builds
Aka "jumbo" or "amalgamation" builds. It means to compile all sources
per target as a single C source. This is experimental.

You can enable it by passing `-DCMAKE_UNITY_BUILD=ON` to cmake.
It requires CMake 3.16 or newer.

It makes builds (much) faster, allows for better optimizations and tends
to promote less ambiguous code.

Also add a new AppVeyor CI job and convert an existing one to use
"unity" mode (one MSVC, one MinGW), and enable it for one macOS CI job.

Fix related issues:
- add missing include guard to `easy_lock.h`.
- rename static variables and functions (and a macro) with names reused
  across sources, or shadowed by local variables.
- add an `#undef` after use.
- add a missing `#undef` before use.
- move internal definitions from `ftp.h` to `ftp.c`.
- `curl_memory.h` fixes to make it work when included repeatedly.
- stop building/linking curlx bits twice for a static-mode curl tool.
  These caused doubly defined symbols in unity builds.
- silence missing extern declarations compiler warning for ` _CRT_glob`.
- fix extern declarations for `tool_freq` and `tool_isVistaOrGreater`.
- fix colliding static symbols in debug mode: `debugtime()` and
  `statename`.
- rename `ssl_backend_data` structure to unique names for each
  TLS-backend, along with the `ssl_connect_data` struct member
  referencing them. This required adding casts for each access.
- add workaround for missing `[P]UNICODE_STRING` types in certain Windows
  builds when compiling `lib/ldap.c`. To support "unity" builds, we had
  to enable `SCHANNEL_USE_BLACKLISTS` for Schannel (a Windows
  `schannel.h` option) _globally_. This caused an indirect inclusion of
  Windows `schannel.h` from `ldap.c` via `winldap.h` to have it enabled
  as well. This requires `[P]UNICODE_STRING` types, which is apperantly
  not defined automatically (as seen with both MSVS and mingw-w64).
  This patch includes `<subauth.h>` to fix it.
  Ref: https://github.com/curl/curl/runs/13987772013
  Ref: https://dev.azure.com/daniel0244/curl/_build/results?buildId=15827&view=logs&jobId=2c9f582d-e278-56b6-4354-f38a4d851906&j=2c9f582d-e278-56b6-4354-f38a4d851906&t=90509b00-34fa-5a81-35d7-5ed9569d331c
- tweak unity builds to compile `lib/memdebug.c` separately in memory
  trace builds to avoid PP confusion.
- force-disable unity for test programs.
- do not compile and link libcurl sources to libtests _twice_ when libcurl
  is built in static mode.

KNOWN ISSUES:
- running tests with unity builds may fail in cases.
- some build configurations/env may not compile in unity mode. E.g.:
  https://ci.appveyor.com/project/curlorg/curl/builds/47230972/job/51wfesgnfuauwl8q#L250

Ref: https://github.com/libssh2/libssh2/issues/1034
Ref: https://cmake.org/cmake/help/latest/prop_tgt/UNITY_BUILD.html
Ref: https://en.wikipedia.org/wiki/Unity_build

Closes #11095
2023-06-07 13:06:08 +00:00
Stefan Eissing
4ae2d9f24d
proxy: http2 proxy tunnel implementation
- currently only on debug build and when env variable
  CURL_PROXY_TUNNEL_H2 is present.
- will ALPN negotiate with the proxy server and switch
  tunnel filter based on the protocol negotiated.
- http/1.1 tunnel code moved into cf-h1-proxy.[ch]
- http/2 tunnel code implemented in cf-h2-proxy.[ch]
- tunnel start and ALPN set remains in http_proxy.c
- moving all haproxy related code into cf-haproxy.[ch]

VTLS changes
- SSL filters rely solely on the "alpn" specification they
  are created with and no longer check conn->bits.tls_enable_alpn.
- checks on which ALPN specification to use (or none at all) are
  done in vtls.c when creating the filter.

Testing
- added a nghttpx forward proxy to the pytest setup that
  speaks HTTP/2 and forwards all requests to the Apache httpd
  forward proxy server.
- extending test coverage in test_10 cases
- adding proxy tests for direct/tunnel h1/h2 use of basic auth.
- adding test for http/1.1 and h2 proxy tunneling to pytest

Closes #10780
2023-04-06 13:04:46 +02:00
Stefan Eissing
671158242d
connections: introduce http/3 happy eyeballs
New cfilter HTTP-CONNECT for h3/h2/http1.1 eyeballing.
- filter is installed when `--http3` in the tool is used (or
  the equivalent CURLOPT_ done in the library)
- starts a QUIC/HTTP/3 connect right away. Should that not
  succeed after 100ms (subject to change), a parallel attempt
  is started for HTTP/2 and HTTP/1.1 via TCP
- both attempts are subject to IPv6/IPv4 eyeballing, same
  as happens for other connections
- tie timeout to the ip-version HAPPY_EYEBALLS_TIMEOUT
- use a `soft` timeout at half the value. When the soft timeout
  expires, the HTTPS-CONNECT filter checks if the QUIC filter
  has received any data from the server. If not, it will start
  the HTTP/2 attempt.

HTTP/3(ngtcp2) improvements.
- setting call_data in all cfilter calls similar to http/2 and vtls filters
  for use in callback where no stream data is available.
- returning CURLE_PARTIAL_FILE for prematurely terminated transfers
- enabling pytest test_05 for h3
- shifting functionality to "connect" UDP sockets from ngtcp2
  implementation into the udp socket cfilter. Because unconnected
  UDP sockets are weird. For example they error when adding to a
  pollset.

HTTP/3(quiche) improvements.
- fixed upload bug in quiche implementation, now passes 251 and pytest
- error codes on stream RESET
- improved debug logs
- handling of DRAIN during connect
- limiting pending event queue

HTTP/2 cfilter improvements.
- use LOG_CF macros for dynamic logging in debug build
- fix CURLcode on RST streams to be CURLE_PARTIAL_FILE
- enable pytest test_05 for h2
- fix upload pytests and improve parallel transfer performance.

GOAWAY handling for ngtcp2/quiche
- during connect, when the remote server refuses to accept new connections
  and closes immediately (so the local conn goes into DRAIN phase), the
  connection is torn down and a another attempt is made after a short grace
  period.
  This is the behaviour observed with nghttpx when we tell it to  shut
  down gracefully. Tested in pytest test_03_02.

TLS improvements
- ALPN selection for SSL/SSL-PROXY filters in one vtls set of functions, replaces
  copy of logic in all tls backends.
- standardized the infof logging of offered ALPNs
- ALPN negotiated: have common function for all backends that sets alpn proprty
  and connection related things based on the negotiated protocol (or lack thereof).

- new tests/tests-httpd/scorecard.py for testing h3/h2 protocol implementation.
  Invoke:
    python3 tests/tests-httpd/scorecard.py --help
  for usage.

Improvements on gathering connect statistics and socket access.
- new CF_CTRL_CONN_REPORT_STATS cfilter control for having cfilters
  report connection statistics. This is triggered when the connection
  has completely connected.
- new void Curl_pgrsTimeWas(..) method to report a timer update with
  a timestamp of when it happend. This allows for updating timers
  "later", e.g. a connect statistic after full connectivity has been
  reached.
- in case of HTTP eyeballing, the previous changes will update
  statistics only from the filter chain that "won" the eyeballing.
- new cfilter query CF_QUERY_SOCKET for retrieving the socket used
  by a filter chain.
  Added methods Curl_conn_cf_get_socket() and Curl_conn_get_socket()
  for convenient use of this query.
- Change VTLS backend to query their sub-filters for the socket when
  checks during the handshake are made.

HTTP/3 documentation on how https eyeballing works.

TLS improvements
- ALPN selection for SSL/SSL-PROXY filters in one vtls set of functions, replaces
  copy of logic in all tls backends.
- standardized the infof logging of offered ALPNs
- ALPN negotiated: have common function for all backends that sets alpn proprty
  and connection related things based on the negotiated protocol (or lack thereof).

Scorecard with Caddy.
- configure can be run with `--with-test-caddy=path` to specify which caddy to use for testing
- tests/tests-httpd/scorecard.py now measures download speeds with caddy

pytest improvements
- adding Makfile to clean gen dir
- adding nghttpx rundir creation on start
- checking httpd version 2.4.55 for test_05 cases where it is needed. Skipping with message if too old.
- catch exception when checking for caddy existance on system.

Closes #10349
2023-02-02 09:57:34 +01:00
Stefan Eissing
9e93bd47c2 vtls: Manage current easy handle in nested cfilter calls
The previous implementation cleared `data` so the outer invocation lost
its data, which could lead to a crash.

Bug: https://github.com/curl/curl/issues/10336
Reported-by: Fujii Hironori

Closes https://github.com/curl/curl/pull/10340
2023-01-26 03:05:01 -05:00
Daniel Stenberg
2bc1d775f5
copyright: update all copyright lines and remove year ranges
- they are mostly pointless in all major jurisdictions
- many big corporations and projects already don't use them
- saves us from pointless churn
- git keeps history for us
- the year range is kept in COPYING

checksrc is updated to allow non-year using copyright statements

Closes #10205
2023-01-03 09:19:21 +01:00
Daniel Stenberg
becfe2ec78
urldata: cease storing TLS auth type
The only TLS auth type libcurl ever supported is SRP and that is the
default type. Since nobody ever sets any other type, there is no point
in wasting space to store the set type and code to check the type.

If TLS auth is used, SRP is now implied.

Closes #10181
2022-12-31 16:56:21 +01:00
Daniel Stenberg
df856cb5c9
vtls: use ALPN HTTP/1.0 when HTTP/1.0 is used
Previously libcurl would use the HTTP/1.1 ALPN id even when the
application specified HTTP/1.0.

Reported-by: William Tang
Ref: #10183
2022-12-31 16:53:21 +01:00
Stefan Eissing
27ec767ebd
gnutls: use common gnutls init and verify code for ngtcp2
Closes #10007
2022-12-01 10:57:36 +01:00