Commit Graph

61 Commits

Author SHA1 Message Date
Viktor Szakats
9eae606dfd
gitignore: restore explicit lists and fix them
Range matches are not supported with `+`: https://git-scm.com/docs/gitignore

Also sync `.gitignore` file lists with source files and alphasort them.

Follow-up to 5c31c2e670 #16093
Reported-by: Stefan Eissing
Fixes #16112
Fixes #16115
Closes #16116
2025-01-28 14:44:35 +01:00
Viktor Szakats
5c31c2e670
tidy-up: .gitignore lines mostly
- `.gitignore`: delete, dedupe and move rules upwards.
  Ref: 6389ba87b8 #13311
- `.gitignore`: fix generated test sources.
  Follow-up to 71cf0d1fca #14772
- `.gitignore`: replace exe listings with a wildcard.
- lib: move `setup-*.h` from `EXTRA_DIST` to `CURL_HFILES`.
- `makedebug.bat`: uppercase an argument to match docs.
- GHA/non-native: delete stray env.
  Follow-up to 12a6de2f66 #16043
- sort source lists.

Closes #16093
2025-01-27 20:59:46 +01:00
Viktor Szakats
7e814c8717
build: fix compiling with GCC 4.x versions
- silence false positive picky warnings.
- avoid "possible noreturn" warnings for standalone tests and examples.
- fix to compile without `#pragma GCC diagnostic push` support.
- fix "#pragma GCC diagnostic not allowed inside functions".

Prerequisite for #15975 that needs GCC 4.4 for the latest pre-built
CeGCC/mingw32ce toolchain for Windows CE.

Cherry-picked from #15975
Closes #16062
2025-01-22 11:26:15 +01:00
Stefan Eissing
ef3d7877d5
pytest: remove 'repeat' parameter
With the use of the pytest-repeat plugin, this parameter is no
longer necessary.

Closes #16033
2025-01-17 14:03:30 +01:00
Stefan Eissing
02edae54e8 websocket: fix message send corruption
- Fix a bug in EAGAIN handling when sending frames that led to a
  corrupted last byte of the frame sent.

- Restore sanity to curl_ws_send() behaviour:

  - Partial writes are reported as OK with the actual number of
    payload bytes sent.

  - CURLE_AGAIN is only returned when none of the payload bytes
    (or for 0-length frames, not all of the frame header bytes)
    could be sent.

  - curl_ws_send() now behaves like a common send() call.

- Change 'ws-data' test client to allow concurrent send/recv
  operations and vary frame sizes and repeat count.

- Add DEBUG env var CURL_WS_CHUNK_EAGAIN to simulate blocking
  after a chunk of an encoded websocket frame has been sent.

- Add tests.


Prior to this change data corruption may occur when sending websocket
messages due to two bugs:

1) 3e64569a (precedes 8.10.0) caused a data corruption bug in the last
   byte of frame of large messages.

2) curl_ws_send had non-traditional send behavior and could return
   CURLE_AGAIN with bytes sent and expect the caller to adjust buffer
   and buflen in a subsequent call. That behavior was not documented.


Reported-by: na-trium-144@users.noreply.github.com

Fixes https://github.com/curl/curl/issues/15865
Fixes https://github.com/curl/curl/issues/15865#issuecomment-2569870144
Closes https://github.com/curl/curl/pull/15901
2025-01-16 16:19:07 -05:00
Viktor Szakats
de3e662ce1
msvc: tidy up _CRT_*_NO_DEPRECATE definitions
Dedupe and migrate MSVC-specific warning suppressions to `curl_setup.h`.
Make cmake set `_CRT_SECURE_NO_DEPRECATE` for examples and standalone
tests, and stop setting `_CRT_NONSTDC_NO_DEPRECATE` for them.

Details:
- drop version guards. On ancient MSVC version these macro are a no-op.
- move to `curl_setup.h` from `config-win32*.h`.
- sync macro values with CMake.
- cmake: stop setting them globally in favour of `curl_setup.h`.
- cmake: re-add these macros to `docs/examples` and `tests/http/clients`,
  which do not use `curl_setup.h`.
- cmake: drop `_CRT_NONSTDC_NO_DEPRECATE` for examples and tests.
  They build fine without.
- update comments.

Closes #15960
2025-01-12 00:34:21 +01:00
Stefan Eissing
bd3c027ac9
conncache: count shutdowns against host and max limits
Count connections to a host against a possibly configured destination
limit. Trigger multi `connchange` when a connection has been shutdown,
so pending transfers can try to get a connection once again.

Reported-by: baranyaib90 on github
Fixes #15857
Closes #15879
2025-01-06 23:10:23 +01:00
Viktor Szakats
fabfa8e402
clang-tidy: add to CI, add cmake support, fix fallouts
build:
- autotools: fix to build generated sources for the `tidy` target.
- autotools: allow passing custom clang-tidy options via
  `CURL_CLANG_TIDYFLAGS` env.
- cmake: add `CURL_CLANG_TIDY` option to configure for `clang-tidy`.
  Also add:
  - `CLANG_TIDY` variable to customize the `clang-tidy` tool.
  - `CURL_CLANG_TIDYFLAGS` to pass custom options to `clang-tidy`.
- apply `--enable-werror` and `-DCURL_WERROR=ON` to `clang-tidy`.

CI/GHA:
- add clang-tidy job for Linux, using autotools and clang-tidy v18.
  This one needs to disable `clang-analyzer-valist.Uninitialized`
  to avoid false positives:
  https://github.com/llvm/llvm-project/issues/40656
  Duration: 5.5 minutes
- add clang-tidy job for macOS, using cmake and clang-tidy v19.
  This one also covers tests and examples, and doesn't hit the false
  positives seen with llvm v18 and earlier.
  Duration: 4.5 minutes
- Linux/macOS: skip installing test dependencies when not building or
  running tests.

fix fallouts reported by `clang-tidy`:
- lib:
  - cf-h2-proxy: unused assignment in non-debug builds.
  - cf-socket: silence warning.
    FIXME: https://github.com/curl/curl/pull/15825#issuecomment-2561867769
  - ftp: NULL passed to `strncmp()`.
  - http2: NULL-ptr deref.
  - mprintf: silence warning.
- src/tool_writeout: NULL passed to `fputs()`.
- examples:
  - invalid file pointers.
  - missing `fclose()`.
- tests:
  - http/clients/hx-download: memory leaks on error.
  - http/clients/hx-download: memory leak on repeat `-r` option.
  - server: double `fclose()`.
    https://www.man7.org/linux/man-pages/man3/fclose.3.html
  - server: invalid file pointer/handle.
  - server/getpart: unused assignments.
  - server/mqttd: leak on failed `realloc()`.
  - server/tftpd: NULL passed to `strcmp()`.

Closes #15825
2024-12-27 13:42:32 +01:00
Viktor Szakats
46093d9e0e
examples: make them compile with compatibility functions disabled (Windows)
For MinGW this is `-DNO_OLDNAMES`, with MSVC it is
`-D_CRT_DECLARE_NONSTDC_NAMES=0`.

There have been some support for this before this patch.
After this patch this is extended to all examples.

(And also the standalone http/client programs, if here.)

Cherry-picked from #15652
Closes #15789
2024-12-20 17:56:40 +01:00
Viktor Szakats
b9895b9d3f
cmake: namespace functions and macros
Prefix (or suffix) curl-defined macro and function names with `curl`.
To avoid collisions with non-curl CMake and to make them recognizable
as curl-specific calls.

Closes #15498
2024-12-16 21:55:00 +01:00
Randall S. Becker
aed732acb1
tests/http/clients: use proper sleep() call on NonStop
This fix impacts ws-data.c and ws-pingpong.c. sleep() replaced with
system PROCESS_DELAY_() having the same resolution as usleep().

Fixes #15711
Closes #15712

Signed-off-by: Randall S. Becker <randall.becker@nexbridge.ca>
2024-12-12 14:30:28 +01:00
Stefan Eissing
55968fd14b
pytest: add test for use of CURLMOPT_MAX_HOST_CONNECTIONS
Add test_02_33 to run with various values for the multi option
CURLMOPT_MAX_HOST_CONNECTIONS and CURLOPT_FRESH_CONNECT to trigger
connection pool limit handling code.

Closes #15494
2024-12-02 21:16:10 +01:00
Viktor Szakats
9126eb5a8a
cmake: replace CURL_*_DIR with {PROJECT,CMAKE_CURRENT}_*_DIR
It reduces the number of synonym variables in the code.
Makes it easier to grok and grep.

- replace `CURL_SOURCE_DIR`
  with `PROJECT_SOURCE_DIR`.

- replace `CURL_BINARY_DIR`
  with `PROJECT_BINARY_DIR` or `CMAKE_CURRENT_BINARY_DIR`.

- replace a single use of `CMAKE_BINARY_DIR`
  with `PROJECT_BINARY_DIR`.

- replace `CMAKE_CURRENT_*_DIR`
  with `PROJECT_*_DIR` where it makes the code more uniform.

- quote an argument (formatting).

Closes #15331
2024-10-22 19:13:08 +02:00
Daniel Stenberg
8403e5a701
tests: fix callback signatures to please UndefinedBehaviorSanitizer
Make test applications use the correct prototypes for callbacks.

Closes #15289
2024-10-15 14:33:47 +02:00
Stefan Eissing
962097b8dd
TLS: TLSv1.3 earlydata support for curl
Based on #14135, implement TLSv1.3 earlydata support for the curl
command line, libcurl and its implementation in GnuTLS.

If a known TLS session announces early data support, and the feature is
enabled *and* it is not a "connect-only" transfer, delay the TLS
handshake until the first request is being sent.

- Add --tls-earldata as new boolean command line option for curl.
- Add CURLSSLOPT_EARLYDATA to libcurl to enable use of the feature.
- Add CURLINFO_EARLYDATA_SENT_T to libcurl, reporting the amount of
  bytes sent and accepted/rejected by the server.

Implementation details:
- store the ALPN protocol selected at the SSL session.
- When reusing the session and enabling earlydata, use exactly
  that ALPN protocol for negoptiation with the server. When the
  sessions ALPN does not match the connections ALPN, earlydata
  will not be enabled.
- Check that the server selected the correct ALPN protocol for
  an earlydata connect. If the server does not confirm or reports
  something different, the connect fails.
- HTTP/2: delay sending the initial SETTINGS frames during connect,
  if not connect-only.

Verification:
- add test_02_32 to verify earlydata GET with nghttpx.
- add test_07_70 to verify earlydata PUT with nghttpx.
- add support in 'hx-download', 'hx-upload' clients for the feature

Assisted-by: ad-chaos on github
Closes #15211
2024-10-11 12:28:22 +02:00
Daniel Stenberg
d78e129d50
WebSockets: make support official (non-experimental)
Inverts the configure/cmake options to instead provide options that
disable WebSockets and have them (ws + wss) enabled by default.

Closes #14936
2024-09-27 13:20:25 +02:00
Viktor Szakats
d82f9f965c
build: add pytest targets
It enables running pytests in cmake jobs, regardless of underlying build
tool choice (= makes it work with ninja.)

Also:

- drop pytest logic launching `make` and exiting in case of failure.
  Maybe there is a better way and keep this functionality somehow, bind
  it to a command-line option? make it fail softly?

- GHA/linux: invoke pytest via the build, not directly.

- autotools: add missing dummy runtests targets when cross-compiling.

Closes #15034
2024-09-25 09:25:44 +02:00
Viktor Szakats
5cefda1b93
build: tidy up deprecation suppression, enable warnings for clang
Suppress deprecation warnings the closest to the deprecated code, using
`CURL_IGNORE_DEPRECATION()`. Then drop build-specific suppressions, and
file-wide ones. The latter is not compatible with Unity mode. Also
replace manual suppressions with a macro to apply to all compilers with
deprecation warning support. Also enable deprecation warnings for clang.

- curl/curl.h: enable deprecation warnings for clang.

- docs/examples: stop setting `CURL_DISABLE_DEPRECATION` with autotools.
  Suppression moved to C-level earlier. Syncs with cmake.
  Follow-up to 5fc61a37c1 #14123

- tests/http/clients: stop setting `CURL_DISABLE_DEPRECATION` in
  autotools. If it becomes necessary in the future, it can be done in
  C via the macro. Syncs with cmake.

- lib1545: stop setting `CURL_DISABLE_DEPRECATION` in autotools.
  Drop guard from test source.
  Follow-up to 0f10360073 #12444

- libtest, unit: replace `CURL_DISABLE_DEPRECATION` with
  `CURL_IGNORE_DEPRECATION()`.

- docs/examples: replace pragmas with `CURL_IGNORE_DEPRECATION()`.

Closes #14789
2024-09-21 00:59:52 +02:00
Daniel Stenberg
fbf5d507ce
lib/src: white space edits to comply better with code style
... as checksrc now finds and complains about these.

Closes #14921
2024-09-19 14:59:12 +02:00
Stefan Eissing
bef0acaf21
request: correctly reset the eos_sent flag
Add test cases

Bug: https://marc.info/?l=git&m=172620452502747&w=2
Reported-by: Patrick Steinhardt
Closes #14895
2024-09-13 13:28:10 +02:00
Viktor Szakats
e512fbfa67
printf: fix mingw-w64 format checks
Change mingw-w64 printf format checks in public curl headers to use
`__MINGW_PRINTF_FORMAT` instead of `gnu_printf`. This syncs the format
checker with format string macros published via `curl/system.h`. (Also
disable format checks for mingw-w64 older than 3.0.0 (2013-09-20) and
classic-mingw, which do not support this macro.)

This fixes bogus format checker `-Wformat` warnings in 3rd party code
using curl format strings with the curl printf functions, when using
mingw-w64 7.0.0 (2019-11-10) and older (with GCC, MSVCRT).

It also allows to delete two workaounds for this within curl itself:
- setting `-D__USE_MINGW_ANSI_STDIO=1` for mingw-w64 via cmake and
  configure for `docs/examples` and `tests/http/clients`.
  Ref: c730c8549b #14640

The format check macro is incompatible (depending on mingw-w64 version
and configuration) with the C99 `%z` (`size_t`) format string used
internally by curl.

To work around this problem, override the format check style in curl
public headers to use `gnu_printf`. This is compatible with `%z` in all
mingw-w64 versions and allows keeping the C99 format strings internally.

Also:
- lib/ws.c: add missing space to an error message.
- docs/examples/ftpgetinfo.c: fix to use standard printf.

Ref: #14643 (take 1)
Follow-up to 3829759bd0 #12489

Closes #14703
2024-09-02 21:03:01 +02:00
Daniel Stenberg
3af75e18d6
configure: remove USE_EXPLICIT_LIB_DEPS
Added a long time ago for something that libtool should handle for us.

Closes #14697
2024-08-28 13:45:01 +02:00
Viktor Szakats
8b09138083
cmake: minor tidy-ups
- show `OpenSSL v3+` when detected (as in `./configure`).
  (this string also makes its way to `curl-config`.)
- prefer `unset(VAR)` over `set(VAR)`.
  Same effect, but `unset()` tells the intent unambiguously.
  https://cmake.org/cmake/help/latest/command/set.html
- drop "implementation" from an `option()` description.
- FindGSS: replace legacy keyword alias with modern alternative.
  https://cmake.org/cmake/help/latest/command/get_filename_component.html
- move `CURL_STATIC_CRT` logic next to its `option()`.
- improve order of `libcurl.pc`/`curl-config` variable init lines.
- tests: drop/shorten custom target names.
  They inflated generated make files by 550KB.
  Keep target name logic for sync between code snippets.
  Follow-up to a2ef5d36b3 #14660
- clear a variable after use.
- restore `STATUS` for `Features:`/`Protocols:` `message()`s:
  Without it the output goes to stderr, and appears in red in CMake GUI.
  It doesn't seem possible to show a line on stdout without leading
  underscores to match `curl -V` and `./configure` output.
  Partial revert of acbc6b703f #14197
- WindowsCache: move `HAVE_LINUX_TCP_H` into the header group.
- move strings to the same line as their `STRING` keyword.
- formatting in generated code.
- delete bogus comment.
- unfold lines for readability.
- fix a too long line. (for cmakelint)
- missing quotes, whitespace, comments.

Closes #14610
2024-08-26 11:00:08 +02:00
Viktor Szakats
a2ef5d36b3
cmake: sync code between test/example targets
- reuse local variable names.
- sync `PROJECT_LABEL`, add where missing.
- namespace all target names.
- bind header directories to each target.
- tests/server: limit `CURL_STATICLIB` to Windows (as in autotools.)
- drop functions with a single caller.

Closes #14660
2024-08-23 18:42:56 +02:00
Viktor Szakats
42843af0b8
tidy-up: spelling WebSockets
Closes #14646
2024-08-22 17:43:57 +02:00
Viktor Szakats
c730c8549b
build: make CURL_FORMAT_CURL_OFF_T[U] work with mingw-w64 <=7.0.0
Add tweak for mingw-w64 when building tests/http/client programs to
avoid a bogus `-Wformat` warning when using mingw-w64 v7.0.0 or older.
The warning is bogus because these programs use curl's `printf()`
implementation that is guaranteed to support that format spec.

Add this for both CMake and autotools. (But only CMake is CI tested with
an old toolchain.)

Apply the workaround to `docs/examples`, and fix an example to use
curl's `printf()` with `CURL_FORMAT_CURL_OFF_T`.

Reintroduce curl `printf()` calls into `tests/http/client`, via #14625.
Also restore large number masks to a printf, changed earlier in #14382.

Follow-up to 232302f88a #14382
Ref: https://github.com/curl/curl/pull/14625#issuecomment-2302361737

Closes #14640
2024-08-22 10:45:04 +02:00
Viktor Szakats
d222dbe788
build: tidy up internal macro names for libcurl.pc
Rename internal macros to match their `libcurl.pc` metadata counterpart.
Also apply these to the `curl-config.in` template.

- `CPPFLAG_CURL_STATICLIB` -> `LIBCURL_PC_CFLAGS`
- `LIBCURL_LIBS`           -> `LIBCURL_PC_LIBS_PRIVATE`
- `LIBCURL_NO_SHARED`      -> `LIBCURL_PC_LIBS`

Closes #14476
2024-08-12 14:56:41 +02:00
Viktor Szakats
aa3a31ce26
cmake: exclude tests/http/clients builds by default
To limit building them with the testdeps target, like it's done with
the rest of test programs.

Follow-up to 232302f88a #14382
Closes #14477
2024-08-10 10:02:48 +02:00
Viktor Szakats
c2889a7b41
cmake: more syntax tidy-up
- quote string literals.
  In the hope it improves syntax-highlighting and readability.

- use lowercase, underscore-prefixed local var names.
  As a hint for scope, to help readability.

- prefer `pkg_search_module` (over `pkg_check_modules`).
  They are the same, but `pkg_search_module` stops searching
  at the first hit.

- more `IN LISTS` in `foreach()`.

- OtherTests.cmake: clear `CMAKE_EXTRA_INCLUDE_FILES` after use.

- add `PROJECT_LABEL` for http/client and unit test targets.

- sync `Find*` module comments and formatting.

- drop a few local variables.

- drop bogus `CARES_LIBRARIES` from comment.

- unquote numeric literal.

Follow-up to acbc6b703f #14197
Closes #14388
2024-08-07 23:41:27 +02:00
Viktor Szakats
c6fb9895b6
cmake: cleanup header paths
- sync build-dir/source-dir header path order with autotools, by
  including build-dir first, then source-dir.
  This prevents out-of-tree builds breaking due to leftover generated
  headers in the source tree.

- tests/unit: move `src` ahead of `libtest` in header path, syncing with
  autotools.

- stop adding non-existing generated `include` dir to header path.
  There are no generated `include` headers and this directory is either
  missing in out-of-tree builds or the same as the one already added
  globally via the root `CMakeLists.txt`.

- lib: stop adding a duplicate source include directory to the header
  path.
  It's already added globally via the root `CMakeLists.txt`.

- lib: stop adding the project root to the header path.

- docs/examples: drop internal header paths.
  Examples do not and should not use internal headers.

- replace `curl_setup_once.h` in comments with `curl_setup.h`,
  the header actually used, and also referred to in autotools comments.

- add comment why we need `src` in include path for `tests/server`.

- add quotes around header directories.

Closes #14416
2024-08-06 22:33:05 +02:00
Viktor Szakats
232302f88a
cmake: add Linux CI job, fix pytest with cmake
- extend existing Linux workflow with CMake support.
  Including running pytest the first time with CMake.

- cmake: generate `tests/config` and `tests/http/config.ini`.
  Required for pytest tests.
  Uses basic detection logic. Feel free to take it from here.
  Also dump config files in a CI step for debugging purposes.

- cmake: build `tests/http/clients` programs.

- fix portability issues with `tests/http/clients` programs.
  Some of them use `getopt()`, which is not supported by MSVC.
  Fix the rest to compile in CI (old-mingw-w64, MSVC, Windows).

- GHA/linux: add CMake job matching an existing autotools one.

- GHA/linux: test `-DCURL_LIBCURL_VERSIONED_SYMBOLS=ON`
  in the new CMake job.

- reorder testdeps to build server, client tests first and then
  libtests and units, to catch errors in the more complex/unique
  sources earlier.

- sort list in `tests/http/clients/Makefile.inc`.

Closes #14382
2024-08-06 02:40:32 +02:00
Stefan Eissing
35bf766280
http2: improved upload eos handling
- replace the counting of upload lengths with the new eos send flag
- improve frequency of stream draining to happen less on events where it
  is not needed
- this PR is based on #14220

http2, cf-h2-proxy: fix EAGAINed out buffer
- in adjust pollset and shutdown handling, a non-empty `ctx->outbufq`
  must trigger send polling, irregardless of http/2 flow control
- in http2, fix retry handling of blocked GOAWAY frame

test case improvement:
- let client 'upload-pausing' handle http versions

Closes #14253
2024-08-05 08:53:55 +02:00
Viktor Szakats
f81f351b9a
tidy-up: OS names
Use these words and casing more consistently across text, comments and
one curl tool output:
AIX, ALPN, ANSI, BSD, Cygwin, Darwin, FreeBSD, GitHub, HP-UX, Linux,
macOS, MS-DOS, MSYS, MinGW, NTLM, POSIX, Solaris, UNIX, Unix, Unicode,
WINE, WebDAV, Win32, winbind, WinIDN, Windows, Windows CE, Winsock.

Mostly OS names and a few more.

Also a couple of other minor text fixups.

Closes #14360
2024-08-04 19:17:45 +02:00
Stefan Eissing
c9b95c0bb3
lib: graceful connection shutdown
When libcurl discards a connection there are two phases this may go
through: "shutdown" and "closing". If a connection is aborted, the
shutdown phase is skipped and it is closed right away.

The connection filters attached to the connection implement the phases
in their `do_shutdown()` and `do_close()` callbacks. Filters carry now a
`shutdown` flags next to `connected` to keep track of the shutdown
operation.

Filters are shut down from top to bottom. If a filter is not connected,
its shutdown is skipped. Notable filters that *do* something during
shutdown are HTTP/2 and TLS. HTTP/2 sends the GOAWAY frame. TLS sends
its close notify and expects to receive a close notify from the server.

As sends and receives may EAGAIN on the network, a shutdown is often not
successful right away and needs to poll the connection's socket(s). To
facilitate this, such connections are placed on a new shutdown list
inside the connection cache.

Since managing this list requires the cooperation of a multi handle,
only the connection cache belonging to a multi handle is used. If a
connection was in another cache when being discarded, it is removed
there and added to the multi's cache. If no multi handle is available at
that time, the connection is shutdown and closed in a one-time,
best-effort attempt.

When a multi handle is destroyed, all connection still on the shutdown
list are discarded with a final shutdown attempt and close. In curl
debug builds, the environment variable `CURL_GRACEFUL_SHUTDOWN` can be
set to make this graceful with a timeout in milliseconds given by the
variable.

The shutdown list is limited to the max number of connections configured
for a multi cache. Set via CURLMOPT_MAX_TOTAL_CONNECTIONS. When the
limit is reached, the oldest connection on the shutdown list is
discarded.

- In multi_wait() and multi_waitfds(), collect all connection caches
  involved (each transfer might carry its own) into a temporary list.
  Let each connection cache on the list contribute sockets and
  POLLIN/OUT events it's connections are waiting for.

- in multi_perform() collect the connection caches the same way and let
  them peform their maintenance. This will make another non-blocking
  attempt to shutdown all connections on its shutdown list.

- for event based multis (multi->socket_cb set), add the sockets and
  their poll events via the callback. When `multi_socket()` is invoked
  for a socket not known by an active transfer, forward this to the
  multi's cache for processing. On closing a connection, remove its
  socket(s) via the callback.

TLS connection filters MUST NOT send close nofity messages in their
`do_close()` implementation. The reason is that a TLS close notify
signals a success. When a connection is aborted and skips its shutdown
phase, the server needs to see a missing close notify to detect
something has gone wrong.

A graceful shutdown of FTP's data connection is performed implicitly
before regarding the upload/download as complete and continuing on the
control connection. For FTP without TLS, there is just the socket close
happening. But with TLS, the sent/received close notify signals that the
transfer is complete and healthy. Servers like `vsftpd` verify that and
reject uploads without a TLS close notify.

- added test_19_* for shutdown related tests
- test_19_01 and test_19_02 test for TCP RST packets
  which happen without a graceful shutdown and should
  no longer appear otherwise.
- add test_19_03 for handling shutdowns by the server
- add test_19_04 for handling shutdowns by curl
- add test_19_05 for event based shutdowny by server
- add test_30_06/07 and test_31_06/07 for shutdown checks
  on FTP up- and downloads.

Closes #13976
2024-06-26 08:33:17 +02:00
Andy Pan
479858e15e
test: fix CURLOPT_TCP_KEEPCNT typo
Follow up to b77d627d24

Closes #13931
2024-06-12 16:08:05 +02:00
Andy Pan
b77d627d24
tcpkeepalive: add CURLOPT_TCP_KEEPCNT and --keepalive-cnt
Closes #13885
2024-06-12 09:31:17 +02:00
Stefan Eissing
30de937bda
transfer: conn close on paused upload
- add 2 variations on test_07_42 which PAUSEs uploads
  and response connections terminating either right away
  or after the 100-continue response
- when detecting the connection being closed in transfer.c
  readwrite_data(), clear ALL send bits in data->req.keepon.
  It no longer makes send to wait for a KEEP_SEND_PAUSE or HOLD.
- in the protocol client writer add the check for incomplete
  response bodies. When an EOS is seen and the length is known,
  check that and fail if bytes are missing.

Reported-by: Sergey Bronnikov
Fixes #13740
Closes #13750
2024-05-23 23:55:09 +02:00
Stefan Eissing
c8e0cd1de8
http3: quiche+ngtcp2 improvements
- quiche: error transfers that try to receive on a closed
  or draining connection
- ngtcp2: use callback for extending max bidi streams. This
  allows more precise calculation of MAX_CONCURRENT as we
  only can start a new stream when the server acknowledges
  the close - not when we locally have closed it.
- remove a fprintf() from h2-download client to avoid excess
  log files on tests timing out.

Closes #13475
2024-04-26 14:03:05 +02:00
Stefan Eissing
08d10d2a00
http3: extend download abort tests, fixes in ngtcp2
- fix flow handling in ngtcp2 to ACK data on streams
  we abort ourself.
- extend test_02_23* cases to also run for h3
- skip test_02_23* for OpenSSL QUIC as it gets stalled
  on progressing the connection

Closes #13374
2024-04-16 23:48:22 +02:00
Stefan Eissing
270a25c011
cw-out: improved error handling
- remember error encountered in invoking write callback and always fail
  afterwards without further invokes

- check behaviour in test_02_17 with h2-pausing client

Reported-by: Pavel Kropachev
Fixes #13337
Closes #13340
2024-04-16 15:52:10 +02:00
Stefan Eissing
28c5ddf13a
tests: stabilitze test_02_23*
- h2-download now always opens the output file on first write callback
  invocation, if it will pause the transfer or not.
- Checks on output files then does not depend on the amount of data curl
  has collected for the first write.

Closes #13323
2024-04-09 13:27:21 +02:00
Stefan Eissing
3210101088
tls: use shared init code for TCP+QUIC
Closes #13172
2024-04-09 09:08:05 +02:00
Stefan Eissing
1302aa6b50
http2: emit RST when client write fails
- When the writing of response data fails, reset the stream
  and do not return a callback error to nghttp2. That would
  be a fatal error for the connection and harm other requests.
- add test cases for various abort scenarios

Reported-by: Konstantin Kuzov
Fixes #13292
Closes #13298
2024-04-08 15:49:52 +02:00
Gisle Vanem
a42de088a2 tests/http: fix compiler warning
- Init result code variable to fix clang warning that it may be used
  uninitialized.

Fixes https://github.com/curl/curl/issues/13301
Closes https://github.com/curl/curl/pull/13304
2024-04-07 14:21:31 -04:00
Viktor Szakats
dc178faee9
tidy-up: whitespace [ci skip] 2024-04-05 15:50:07 +00:00
Stefan Eissing
cfc65fd1ee
request: paused upload on completed download, assess connection
A transfer with a completed download that is still uploading needs to
check the connection state when it is PAUSEd, since connection
close/errors would otherwise go unnoticed.

Reported-by: Sergey Bronnikov
Fixes #13260
Closes #13271
2024-04-04 11:45:19 +02:00
Viktor Szakats
95a882d268
build: fix -Wconversion/-Wsign-conversion warnings
Fix remaining warnings in examples and tests which are not suppressed
by the pragma in `lib/curl_setup.h`.

Silence a toolchain issue causing warnings in `FD_SET()` calls with
older Cygwin/MSYS2 builds. Likely fixed on 2020-08-03 by:
https://cygwin.com/git/?p=newlib-cygwin.git;a=commitdiff;h=5717262b8ecfed0f7fab63e2c09c78991e36f9dd

Follow-up to 2dbe75bd7f #12492

Closes #12557
2023-12-20 15:38:45 +00:00
Viktor Szakats
3829759bd0
build: enable missing OpenSSF-recommended warnings, with fixes
https://best.openssf.org/Compiler-Hardening-Guides/Compiler-Options-Hardening-Guide-for-C-and-C++.html
as of 2023-11-29 [1].

Enable new recommended warnings (except `-Wsign-conversion`):

- enable `-Wformat=2` for clang (in both cmake and autotools).
- add `CURL_PRINTF()` internal attribute and mark functions accepting
  printf arguments with it. This is a copy of existing
  `CURL_TEMP_PRINTF()` but using `__printf__` to make it compatible
  with redefinting the `printf` symbol:
  https://gcc.gnu.org/onlinedocs/gcc-3.0.4/gcc_5.html#SEC94
- fix `CURL_PRINTF()` and existing `CURL_TEMP_PRINTF()` for
  mingw-w64 and enable it on this platform.
- enable `-Wimplicit-fallthrough`.
- enable `-Wtrampolines`.
- add `-Wsign-conversion` commented with a FIXME.
- cmake: enable `-pedantic-errors` the way we do it with autotools.
  Follow-up to d5c0351055 #2747
- lib/curl_trc.h: use `CURL_FORMAT()`, this also fixes it to enable format
  checks. Previously it was always disabled due to the internal `printf`
  macro.

Fix them:

- fix bug where an `set_ipv6_v6only()` call was missed in builds with
  `--disable-verbose` / `CURL_DISABLE_VERBOSE_STRINGS=ON`.
- add internal `FALLTHROUGH()` macro.
- replace obsolete fall-through comments with `FALLTHROUGH()`.
- fix fallthrough markups: Delete redundant ones (showing up as
  warnings in most cases). Add missing ones. Fix indentation.
- silence `-Wformat-nonliteral` warnings with llvm/clang.
- fix one `-Wformat-nonliteral` warning.
- fix new `-Wformat` and `-Wformat-security` warnings.
- fix `CURL_FORMAT_SOCKET_T` value for mingw-w64. Also move its
  definition to `lib/curl_setup.h` allowing use in `tests/server`.
- lib: fix two wrongly passed string arguments in log outputs.
  Co-authored-by: Jay Satiro
- fix new `-Wformat` warnings on mingw-w64.

[1] 56c0fde389/docs/Compiler-Hardening-Guides/Compiler-Options-Hardening-Guide-for-C-and-C%2B%2B.md

Closes #12489
2023-12-16 13:12:37 +00:00
Viktor Szakats
ede2e812c2
tidy-up: whitespace
Closes #12484
2023-12-08 03:28:50 +00:00
Viktor Szakats
84338c4de2
build: add more picky warnings and fix them
Enable more picky compiler warnings. I've found these options in the
nghttp3 project when implementing the CMake quick picky warning
functionality for it [1].

`-Wunused-macros` was too noisy to keep around, but fixed a few issues
it revealed while testing.

- autotools: reflect the more precisely-versioned clang warnings.
  Follow-up to 033f8e2a08 #12324
- autotools: sync between clang and gcc the way we set `no-multichar`.
- autotools: avoid setting `-Wstrict-aliasing=3` twice.
- autotools: disable `-Wmissing-noreturn` for MSYS gcc targets [2].
  It triggers in libtool-generated stub code.

- lib/timeval: delete a redundant `!MSDOS` guard from a `WIN32` branch.

- lib/curl_setup.h: delete duplicate declaration for `fileno`.
  Added in initial commit ae1912cb0d
  (1999-12-29). This suggests this may not be needed anymore, but if
  it does, we may restore this for those specific (non-Windows) systems.
- lib: delete unused macro `FTP_BUFFER_ALLOCSIZE` since
  c1d6fe2aaa.
- lib: delete unused macro `isxdigit_ascii` since
  f65f750742.
- lib/mqtt: delete unused macro `MQTT_HEADER_LEN`.
- lib/multi: delete unused macro `SH_READ`/`SH_WRITE`.
- lib/hostip: add `noreturn` function attribute via new `CURL_NORETURN`
  macro.
- lib/mprintf: delete duplicate declaration for `Curl_dyn_vprintf`.
- lib/rand: fix `-Wunreachable-code` and related fallouts [3].
- lib/setopt: fix `-Wunreachable-code-break`.
- lib/system_win32 and lib/timeval: fix double declarations for
  `Curl_freq` and `Curl_isVistaOrGreater` in CMake UNITY mode [4].
- lib/warnless: fix double declarations in CMake UNITY mode [5].
  This was due to force-disabling the header guard of `warnless.h` to
  to reapply it to source code coming after `warnless.c` in UNITY
  builds. This reapplied declarations too, causing the warnings.
  Solved by adding a header guard for the lines that actually need
  to be reapplied.
- lib/vauth/digest: fix `-Wunreachable-code-break` [6].
- lib/vssh/libssh2: fix `-Wunreachable-code-break` and delete redundant
  block.
- lib/vtls/sectransp: fix `-Wunreachable-code-break` [7].
- lib/vtls/sectransp: suppress `-Wunreachable-code`.
  Detected in `else` branches of dynamic feature checks, with results
  known at compile-time, e.g.
  ```c
  if(SecCertificateCopySubjectSummary)  /* -> true */
  ```
  Likely fixable as a separate micro-project, but given SecureTransport
  is deprecated anyway, let's just silence these locally.
- src/tool_help: delete duplicate declaration for `helptext`.
- src/tool_xattr: fix `-Wunreachable-code`.
- tests: delete duplicate declaration for `unitfail` [8].
- tests: delete duplicate declaration for `strncasecompare`.
- tests/libtest: delete duplicate declaration for `gethostname`.
  Originally added in 687df5c8c3
  (2010-08-02).
  Got complicated later: c49e9683b8
  If there are still systems around with warnings, we may restore the
  prototype, but limited for those systems.
- tests/lib2305: delete duplicate declaration for
  `libtest_debug_config`.
- tests/h2-download: fix `-Wunreachable-code-break`.

[1] a70edb08e9/cmake/PickyWarningsC.cmake
[2] https://ci.appveyor.com/project/curlorg/curl/builds/48553586/job/3qkgjauiqla5fj45?fullLog=true#L1675
[3] https://github.com/curl/curl/actions/runs/6880886309/job/18716044703?pr=12331#step:7:72
    https://github.com/curl/curl/actions/runs/6883016087/job/18722707368?pr=12331#step:7:109
[4] https://ci.appveyor.com/project/curlorg/curl/builds/48555101/job/9g15qkrriklpf1ut#L204
[5] https://ci.appveyor.com/project/curlorg/curl/builds/48555101/job/9g15qkrriklpf1ut#L218
[6] https://github.com/curl/curl/actions/runs/6880886309/job/18716042927?pr=12331#step:7:290
[7] https://github.com/curl/curl/actions/runs/6891484996/job/18746659406?pr=12331#step:9:1193
[8] https://github.com/curl/curl/actions/runs/6882803986/job/18722082562?pr=12331#step:33:1870

Closes #12331
2023-11-21 16:35:42 +00:00