- update client reader documentation
- client reader, add rewind capabilities
- tell creader to rewind on next start
- Curl_client_reset() will keep reader for future rewind if requested
- add Curl_client_cleanup() for freeing all resources independent of
rewinds
- add Curl_client_start() to trigger rewinds
- move rewind code from multi.c to sendf.c and make part of
"cr-in"'s implementation
- http, move the "resume_from" handling into the client readers
- the setup of a HTTP request is reshuffled to follow:
* determine method, target, auth negotiation
* install the client reader(s) for the request, including crlf
conversions and "chunked" encoding
* apply ranges to client reader
* concat request headers, upgrades, cookies, etc.
* complete request by determining Content-Length of installed
readers in combination with method
* send
- add methods for client readers to
* return the overall length they will generate (or -1 when unknown)
* return the amount of data on the CLIENT level, so that
expect-100 can decide if it want to apply itself
* set a "resume_from" offset or fail if unsupported
- struct HTTP has become largely empty now
- rename `Client_reader_*` to `Curl_creader_*`
Closes#13026
The function that replaces occurances of "--longoption" with "-Z,
--longoption" etc with the proper highlight applied, no longer loops
over the options.
Closes#13041
This fixes miscellaneous typos and duplicated words in the docs, lib
and test comments and a few user facing errorstrings.
Author: RainRat on Github
Reviewed-by: Daniel Gustafsson <daniel@yesql.se>
Reviewed-by: Dan Fandrich <dan@coneharvesters.com>
Closes: #13019
- replace `Curl_read()`, `Curl_write()` and `Curl_nwrite()` to
clarify when and at what level they operate
- send/recv of transfer related data is now done via
`Curl_xfer_send()/Curl_xfer_recv()` which no longer has
socket/socketindex as parameter. It decides on the transfer
setup of `conn->sockfd` and `conn->writesockfd` on which
connection filter chain to operate.
- send/recv on a specific connection filter chain is done via
`Curl_conn_send()/Curl_conn_recv()` which get the socket index
as parameter.
- rename `Curl_setup_transfer()` to `Curl_xfer_setup()` for
naming consistency
- clarify that the special CURLE_AGAIN hangling to return
`CURLE_OK` with length 0 only applies to `Curl_xfer_send()`
and CURLE_AGAIN is returned by all other send() variants.
- fix a bug in websocket `curl_ws_recv()` that mixed up data
when it arrived in more than a single chunk (to be made
into a sperate PR, also)
Added as documented [in
CLIENT-READER.md](5b1f31dfba/docs/CLIENT-READERS.md).
- old `Curl_buffer_send()` completely replaced by new `Curl_req_send()`
- old `Curl_fillreadbuffer()` replaced with `Curl_client_read()`
- HTTP chunked uploads are now formatted in a client reader added when
needed.
- FTP line-end conversions are done in a client reader added when
needed.
- when sending requests headers, remaining buffer space is filled with
body data for sending in "one go". This is independent of the request
body size. Resolves#12938 as now small and large requests have the
same code path.
Changes done to test cases:
- test513: now fails before sending request headers as this initial
"client read" triggers the setup fault. Behaves now the same as in
hyper build
- test547, test555, test1620: fix the length check in the lib code to
only fail for reads *smaller* than expected. This was a bug in the
test code that never triggered in the old implementation.
Closes#12969
The curldown conversion accidentally replaced daniel@haxx.se with
just daniel.se. This reverts back to the proper email address in
the curldown docs as well as in a few other stray places where it
was incorrect (while unrelated to curldown).
Reviewed-by: Daniel Stenberg <daniel@haxx.se>
Closes: #12997
Returns 1 if the previous transfer used a proxy, otherwise 0. Useful to
for example determine if a `NOPROXY` pattern matched the hostname or
not.
Extended test 970 and 972
- refs #12397 where it is dicussed how to en-/disable verbose output
of DoH operations
- introducing `struct curl_trc_feat` to track a curl feature for
tracing
- adding `data->state.feat` optionally pointing to the feature a
transfer belongs to
- adding trace functions and verbosity checks on features
- using trace feature in DoH code
- documenting `doh` as feature for `--trace-config`
Closes#12411
- when data arrived in several chunks, the collection into
the passed buffer always started at offset 0, overwriting
the data already there.
adding test_20_07 to verify fix
- debug environment var CURL_WS_CHUNK_SIZE can be used to
influence the buffer chunk size used for en-/decoding.
Closes#12945
- can be borrowed by transfer during recv-write operation
- needs to be released before borrowing again
- adjustis size to `data->set.buffer_size`
- used in transfer.c readwrite_data()
Closes#12805
- make sure angle brackets are escaped
- remove a lot of superfluous double quotes
- replace several double quotes with backticks
To make nicer-looking markdown.
Closes#12884
Make sure we use \< and \> in markdown all over so that it renders
correctly, on GitHub and elsewhere. cd2nroff now outputs a warning if it
finds an unescaled angle bracket.
Ref: #12854Closes#12869
When setting the CURLOPT_SSLCERT option to a certificate thumprint, it
is required to have a backslash between the "store location", "store
name" and "thumbprint" tokens. These slashes were present in the
previous documentation, but were missed in the transition to markdown
documentation.
Closes#12854
- Turn docs/INSTALL.cmake into a proper markdown file,
docs/INSTALL-CMAKE.md
- Move things around to divide the description into configuration,
building and installing sections
- Mention the more modern cmake options to configure, build and install,
but also retain the older variants as fallbacks
Closes#12772
The previous command line to generate the tool_listhelp.c source file
broke with 2494b8dd51.
Make 'make listhelp' invoked in src/ generate it. Also update the
comment in the file to mention the right procedure.
Closes#12786
Rework CMake options for building/using curl tool and libcurl manuals.
- rename `ENABLE_MANUAL` to `ENABLE_CURL_MANUAL`, meaning:
to build man page and built-in manual for curl tool.
- rename `BUILD_DOCS` to `BUILD_LIBCURL_DOCS`, meaning:
to build man pages for libcurl.
- `BUILD_LIBCURL_DOCS` now works without having to enable
`ENABLE_CURL_MANUAL` too.
- drop support for existing CMake-level `USE_MANUAL` option to avoid
confusion. (It used to work with the effect of current
`ENABLE_CURL_MANUAL`, but only by accident.)
Assisted-by: Richard Levitte
Ref: #12771Closes#12773
This means words, phrases or things we have decided not to use - words that
are spelled right according to the dictionary but we want to avoid. In the
name of consistency and better documentation.
Closes#12764
- cmake: enable `BUILD_DOCS` by default (this controls converting and
installing `.3` files from `.md` sources)
- cmake: speed up generating `.3` files by using a single command per
directory, instead of a single command per file. This reduces external
commands by about a thousand. (There remains some CMake logic kicking
in resulting in 500 -one per file- external `-E touch_nocreate` calls.)
- cd2nroff: add ability to process multiple input files.
- cd2nroff: add `-k` option to use the source filename to form the
output filename. (instead of the default in-file `Title:` line.)
Follow-up to 3f08d80b22
Follow-up to ea0b575dab#12753
Follow-up to eefcc1bda4#12730Closes#12762
- switch all invidual files documenting command line options into .md,
as the documentation is now markdown-looking.
- made the parser treat 4-space indents as quotes
- switch to building the curl.1 manpage using the "mainpage.idx" file,
which lists the files to include to generate it, instead of using the
previous page-footer/headers. Also, those files are now also .md
ones, using the same format. I gave them underscore prefixes to make
them sort separately:
_NAME.md, _SYNOPSIS.md, _DESCRIPTION.md, _URL.md, _GLOBBING.md,
_VARIABLES.md, _OUTPUT.md, _PROTOCOLS.md, _PROGRESS.md, _VERSION.md,
_OPTIONS.md, _FILES.md, _ENVIRONMENT.md, _PROXYPREFIX.md,
_EXITCODES.md, _BUGS.md, _AUTHORS.md, _WWW.md, _SEEALSO.md
- updated test cases accordingly
Closes#12751
curldown is this new file format for libcurl man pages. It is markdown
inspired with differences:
- Each file has a set of leading headers with meta-data
- Supports a small subset of markdown
- Uses .md file extensions for editors/IDE/GitHub to treat them nicely
- Generates man pages very similar to the previous ones
- Generates man pages that still convert nicely to HTML on the website
- Detects and highlights mentions of curl symbols automatically (when
their man page section is specified)
tools:
- cd2nroff: converts from curldown to nroff man page
- nroff2cd: convert an (old) nroff man page to curldown
- cdall: convert many nroff pages to curldown versions
- cd2cd: verifies and updates a curldown to latest curldown
This setup generates .3 versions of all the curldown versions at build time.
CI:
Since the documentation is now technically markdown in the eyes of many
things, the CI runs many more tests and checks on this documentation,
including proselint, link checkers and tests that make sure we capitalize the
first letter after a period...
Closes#12730
- HTTP/3 for curl using OpenSSL's own QUIC stack together
with nghttp3
- configure with `--with-openssl-quic` to enable curl to
build this. This requires the nghttp3 library
- implementation with the following restrictions:
* macOS has to use an unconnected UDP socket due to an
issue in OpenSSL's datagram implementation
See https://github.com/openssl/openssl/issues/23251
This makes connections to non-reponsive servers hang.
* GET requests will send the indicator that they have
no body in a separate QUIC packet. This may result
in processing delays or Transfer-Encodings on proxied
requests
* uploads that encounter blocks will use 100% cpu as
detection of these flow control issue is not working
(we have not figured out to pry that from OpenSSL).
Closes#12734
- remove use of .BI for code snippet
- stop using .br, just do a blank line
- remove use of .PP
- remove use for .sp
- remove backslash in .IP
- use .IP instead of .TP
Closes#12731
... since this funtion has not supported null pointer fd_set arguments since
at least 2006. (That's when I stopped my git blame journey)
Fixes#12691
Reported-by: sfan5 on github
Closes#12692
Previously it would match only on a sequence of non-space, which made it
miss to highlight for example "public suffix list".
Updated the recent cookie.d edit from 5da57193b7 to use bold instead
of italics.
Closes#12689
Returns the time, in microseconds, during which this transfer was held
in a waiting queue before it started "for real". A transfer might be put
in a queue if after getting started, it cannot create a new connection
etc due to set conditions and limits imposed by the application.
Ref: #12293Closes#12368
- Escape inner quotes with two backslashes.
Two backslashes escapes the backslash for the man page and will show as
a single backslash.
eg: "{\\"name\\": \\"daniel\\"}" shows as "{\"name\": \"daniel\"}".
Closes https://github.com/curl/curl/pull/12588
`winsock2.h` pulls in `windows.h`. `ws2tcpip.h` pulls in `winsock2.h`.
`winsock2.h` and `ws2tcpip.h` are also pulled by `curl/curl.h`.
Keep only those headers that are not already included, or the code under
it uses something from that specific header.
Closes#12539
A new error code to be used when an internal field grows too large, like
when a dynbuf reaches its maximum. Previously it would return
CURLE_OUT_OF_MEMORY for this, which is highly misleading.
Ref: #12268Closes#12269
- autotools, cmake: assume that if we detect Windows, `windows.h`,
`winsock2.h` and `ws2tcpip.h` do exist.
- lib: fix 3 outlier `#if` conditions to use `USE_WINSOCK` instead of
looking for `winsock2.h`.
- autotools: merge 3 Windows check methods into one.
- move Watt-32 and lwIP socket support to `setup-win32.h` from
`config-win32.h`. It opens up using these with all build tools. Also
merge logic with Windows Sockets.
- fix to assume Windows sockets with the mingw32ce toolchain.
Follow-up to: 2748c64d60
- cmake: delete unused variable `signature_call_conv` since
eb33ccd533.
- autotools: simplify `CURL_CHECK_WIN32_LARGEFILE` detection.
- examples/externalsocket: fix header order.
- cmake/OtherTests.cmake: delete Windows-specific `_source_epilogue`
that wasn't used anymore.
- cmake/OtherTests.cmake: set `WIN32_LEAN_AND_MEAN` for test
`SIZEOF_STRUCT_SOCKADDR_STORAGE`.
After this patch curl universally uses `_WIN32` to guard
Windows-specific logic. It guards Windows Sockets-specific logic with
`USE_WINSOCK` (this might need further work).
Reviewed-by: Jay Satiro
Closes#12495
https://best.openssf.org/Compiler-Hardening-Guides/Compiler-Options-Hardening-Guide-for-C-and-C++.html
as of 2023-11-29 [1].
Enable new recommended warnings (except `-Wsign-conversion`):
- enable `-Wformat=2` for clang (in both cmake and autotools).
- add `CURL_PRINTF()` internal attribute and mark functions accepting
printf arguments with it. This is a copy of existing
`CURL_TEMP_PRINTF()` but using `__printf__` to make it compatible
with redefinting the `printf` symbol:
https://gcc.gnu.org/onlinedocs/gcc-3.0.4/gcc_5.html#SEC94
- fix `CURL_PRINTF()` and existing `CURL_TEMP_PRINTF()` for
mingw-w64 and enable it on this platform.
- enable `-Wimplicit-fallthrough`.
- enable `-Wtrampolines`.
- add `-Wsign-conversion` commented with a FIXME.
- cmake: enable `-pedantic-errors` the way we do it with autotools.
Follow-up to d5c0351055#2747
- lib/curl_trc.h: use `CURL_FORMAT()`, this also fixes it to enable format
checks. Previously it was always disabled due to the internal `printf`
macro.
Fix them:
- fix bug where an `set_ipv6_v6only()` call was missed in builds with
`--disable-verbose` / `CURL_DISABLE_VERBOSE_STRINGS=ON`.
- add internal `FALLTHROUGH()` macro.
- replace obsolete fall-through comments with `FALLTHROUGH()`.
- fix fallthrough markups: Delete redundant ones (showing up as
warnings in most cases). Add missing ones. Fix indentation.
- silence `-Wformat-nonliteral` warnings with llvm/clang.
- fix one `-Wformat-nonliteral` warning.
- fix new `-Wformat` and `-Wformat-security` warnings.
- fix `CURL_FORMAT_SOCKET_T` value for mingw-w64. Also move its
definition to `lib/curl_setup.h` allowing use in `tests/server`.
- lib: fix two wrongly passed string arguments in log outputs.
Co-authored-by: Jay Satiro
- fix new `-Wformat` warnings on mingw-w64.
[1] 56c0fde389/docs/Compiler-Hardening-Guides/Compiler-Options-Hardening-Guide-for-C-and-C%2B%2B.mdCloses#12489
Remove use of .TP and some .B. The idea is to reduce nroff syntax as
much as possible and to use it consistently. Ultimately, we should be
able to introduce our own easier-to-use-and-read syntax/formatting and
convert on generation time.
Closes#12535
They were previously sorted based on the file names, which use a .d
extension, making "data" get placed after "data-binary" etc. Making the
sort ignore the extention fixes the ordering.
Reported-by: Boris Verkhovskiy
Bug: https://curl.se/mail/archive-2023-12/0014.htmlCloses#12494
- use the correct include file
- make sure they are declared as in the header file
- fix minor nroff syntax mistakes (missing .fi)
These are verified by verify-synopsis.pl, which extracts the SYNPOSIS
code and runs it through gcc.
Closes#12402
Windows compilers define `_WIN32` automatically. Windows SDK headers
or build env defines `WIN32`, or we have to take care of it. The
agreement seems to be that `_WIN32` is the preferred practice here.
Make the source code rely on that to detect we're building for Windows.
Public `curl.h` was using `WIN32`, `__WIN32__` and `CURL_WIN32` for
Windows detection, next to the official `_WIN32`. After this patch it
only uses `_WIN32` for this. Also, make it stop defining `CURL_WIN32`.
There is a slight chance these break compatibility with Windows
compilers that fail to define `_WIN32`. I'm not aware of any obsolete
or modern compiler affected, but in case there is one, one possible
solution is to define this macro manually.
grepping for `WIN32` remains useful to discover Windows-specific code.
Also:
- extend `checksrc` to ensure we're not using `WIN32` anymore.
- apply minor formatting here and there.
- delete unnecessary checks for `!MSDOS` when `_WIN32` is present.
Co-authored-by: Jay Satiro
Reviewed-by: Daniel Stenberg
Closes#12376
The current design of the Hyper integration requires rebuilding the
Hyper clientconn for each request. However, building the clientconn
requires resending the HTTP/2 connection preface, which is incorrect
from a protocol perspective. That in turn causes servers to send GOAWAY
frames, effectively degrading performance to "no connection reuse" in
the best case. It may also be triggering some bugs where requests get
dropped entirely and reconnects take too long.
This doesn't rule out HTTP/2 support with Hyper, but it may take a
redesign of the Hyper integration in order to make things work.
Closes#12191
- Add an explanation of the CURL_BLOB_COPY flag to CURLOPT_CAINFO_BLOB
and CURLOPT_PROXY_CAINFO_BLOB docs.
All the other _BLOB option docs already have the same explanation.
Closes https://github.com/curl/curl/pull/12277
- convert ensure_trailing into ensure_trailing_slash
- strdup the URL string to own it proper
- use shorter variable names
- combine some expressions
- simplify error handling in ipfs_gateway()
- add MAX_GATEWAY_URL_LEN + proper bailout if maximum is reached
- ipfs-gateway.d polish and simplification
- shorten ipfs error message + make them "synthetic"
Closes#12281
* Remove other mention of hyper memory-leaks from `KNOWN_BUGS`.
Should have been removed in 629723ecf2
* Remove mention of aws-sigv4 sort query string from `KNOWN_BUGS`.
Fixed in #11806
* Remove mention of aws-sigv4 query empty value problems
* Remove mention of aws-sigv4 missing amz-content-sha256
Fixed in #9995
The previously built man page "curl.1" must be copied with the original
modification date, otherwise the man page is never updated.
This fixes a bug that has been introduced with commit 2568441cab.
Reviewed-by: Dan Fandrich
Reviewed-by: Daniel Stenberg
Closes#12199
After this patch we assume availability of `getaddrinfo` and
`freeaddrinfo`, first introduced in Windows XP. Meaning curl
now requires building for Windows XP as a minimum.
TODO: assume these also in autotools.
Ref: https://github.com/curl/curl/pull/12221#issuecomment-1783761806Closes#12225
Delete leftovers of the `crypt-auth` `./configure` option and
add the new ones that replaced them.
Follow-up to e92edfbef6#11490
Reviewed-by: Daniel Stenberg
Closes#12194
Remove the CURL_CA_FALLBACK logic. That build option was added to allow
primarily OpenSSL to use the default paths for loading the CA certs. For
GnuTLS it was instead made to load the "system certs", which is
different and not desirable.
The native CA store loading is now asked for with this option.
Follow-up to 7b55279d1d
Co-authored-by: Jay Satiro
Closes#12137
- Move documentation of libcurl environment variables used only in debug
builds from libcurl-env into a separate document libcurl-env-dbg.
- Document more debug environment variables.
Previously undocumented or missing a description:
CURL_ALTSVC_HTTP, CURL_DBG_SOCK_WBLOCK, CURL_DBG_SOCK_WPARTIAL,
CURL_DBG_QUIC_WBLOCK, CURL_DEBUG, CURL_DEBUG_SIZE, CURL_GETHOSTNAME,
CURL_HSTS_HTTP, CURL_FORCETIME, CURL_SMALLREQSEND, CURL_SMALLSENDS,
CURL_TIME.
Closes https://github.com/curl/curl/pull/11811
- Warn that the user's debug callback may be called with the handle
parameter set to an internal handle.
Without this warning the user may assume that the only handles their
debug callback receives are the easy handles on which they set
CURLOPT_DEBUGFUNCTION.
This is a follow-up to f8cee8cc which changed DoH handles to inherit
the debug callback function set in the user's easy handle. As a result
those handles are now passed to the user's debug callback function.
Closes https://github.com/curl/curl/pull/12034
in FAQ and MANUAL.md
- example.com was made for this purpose.
- reduces the risk that one of those domains suddenly start hosting
something nasty and we provide links to them
Closes#11986
Earlier this year we changed our own stderr variable to use the standard
name `stderr` (to avoid bugs where someone is using `stderr` instead of
the curl-tool specific variable). This solution needed to override the
standard `stderr` symbol via the preprocessor. This in turn didn't play
well with unity builds and caused curl tool to crash or stay silent due
to an uninitialized stderr. This was a hard to find issue, fixed by
manually breaking out one file from the unity sources.
To avoid two these two tricks, this patch implements a different
solution: Restore using our own local variable for our stderr output and
leave `stderr` as-is. To avoid using `stderr` by mistake, add a
`checksrc` rule (based on logic we already used in lib for `strerror`)
that detects any `stderr` use in `src` and points to using our own
variable instead: `tool_stderr`.
Follow-up to 06133d3e9b
Follow-up to 2f17a9b654Closes#11958
- Fix netrc info message to use the generic ".netrc" filename if the
user did not specify a netrc location.
- Update --netrc doc to add that recent versions of curl on Windows
prefer .netrc over _netrc.
Before:
* Couldn't find host google.com in the (nil) file; using defaults
After:
* Couldn't find host google.com in the .netrc file; using defaults
Closes https://github.com/curl/curl/pull/11904
Previously it would only stop them from getting started if the size is
known to be too big then.
Update the libcurl and curl docs accordingly.
Fixes#11810
Reported-by: Elliot Killick
Assisted-by: Jay Satiro
Closes#11820
- ipfs://<cid>
- ipns://<cid>
This allows you tu use ipfs in curl like:
curl ipfs://<cid>
and
curl ipns://<cid>
For more information consult the readme at:
https://curl.se/docs/ipfs.htmlCloses#8805
Prior to 7.75.0 response headers were not printed if -f/--fail was used
and an error was reported by server. This was fixed in ab525c0
(precedes 7.75.0).
Closes#11822
References to things that were added or changed in a specific version
should be specified as "(added in [version]) for two reasons:
1 - consistency
2 - to allow gen.pl to strip them out if deemed referring to too old
versions
Closes#11821
ngtcp2 v0.19.0 made size of `ecn` member of `ngtcp2_pkt_info`
an `uint8_t` (was: `uint32_t`). Adjust our local cast accordingly.
Fixes:
```
./curl/lib/vquic/curl_ngtcp2.c:1912:12: warning: implicit conversion loses integer precision: 'uint32_t' (aka 'unsigned int') to 'uint8_t' (aka 'unsigned char') [-Wimplicit-int-conversion]
pi.ecn = (uint32_t)ecn;
~ ^~~~~~~~~~~~~
```
Also bump ngtcp2, nghttp3 and nghttp2 to their latest versions in our
docs and CI.
Ref: 80447281bb
Ref: https://github.com/ngtcp2/ngtcp2/pull/877Closes#11798
At least FreeBSD preserves cwd across makefile lines, so rules
consisting of more than one "cd X; do_something" must be explicitly run
in a subshell to avoid this. This problem caused the Cirrus FreeBSD
build to fail when parallel make jobs were enabled.
- delete completed TODO from `./CMakeLists.txt`.
- convert a C++ comment to C89 in `./CMake/CurlTests.c`.
- delete duplicate EOLs from EOF.
- add missing EOL at EOF.
- delete whitespace at EOL (except from expected test results).
- convert tabs to spaces.
- convert CRLF EOLs to LF in GHA yaml.
- text casing fixes in `./CMakeLists.txt`.
- fix a codespell typo in `packages/OS400/initscript.sh`.
Closes#11772
Store numerical IPv6 addresses in the alt-svc file with the brackets
present.
Verify with test 437 and 438
Fixes#11737
Reported-by: oliverpool on github
Closes#11743
Too excessive escaping made the parsing not find the correct long names
later and instead add "wrong" links.
Follow-up to 439ff2052e
Reported-by: Lukas Tribus
Fixes#11688Closes#11689
- Remove the out-of-date SSL backend list supported by
CURLOPT_SSL_CTX_FUNCTION.
It makes more sense to just refer to that document instead of having
a separate list that has to be kept in sync.
Closes https://github.com/curl/curl/pull/11665
