Allow building with c-ares and yet use threaded resolver for the main
host A/AAAA resolving:
`--with-ares` provides the c-ares install path and defaults to use
c-ares for name resolving
`--with-threaded-resolver` still uses c-ares in the build (for HTTPS)
but uses the threaded resolver for "normal" resolves.
It works similarly for cmake: ENABLE_ARES enables ares, and if
ENABLE_THREADED_RESOLVER also is set, c-ares is used for HTTPS RR and
the threaded resolver for "normal" resolves.
HTTPSRR and c-ares-rr are new features return by curl_version_info() and
thus shown by curl -V.
The c-ares-rr feature bit is there to make it possible to distinguish
between builds using c-ares for all name resolves and builds that use
the threaded resolves for the regular name resolves and c-ares for
HTTPSRR only. "c-ares-rr" means it does not use c-ares for "plain" name
resolves.
HTTPSRR support is EXPERIMENTAL only.
Closes#16054
Adds the experimental feature `ssls-export` to libcurl and curl for
importing and exporting SSL sessions from/to a file.
* add functions to libcurl API
* add command line option `--ssl-sessions <filename>` to curl
* add documenation
* add support in configure
* add support in cmake
+ add pytest case
Closes#15924
We started using codeql for static code analysis in 7183f5acc3,
June 2020.
Since then, not a single commit has been merged into the source code
repository citing codeql as source or reason. Yet, it keeps getting
updated and we get constant reminders to upgrade the pinning it to the
latest hash.
During 4.5 years with intense development and significant code churn.
While Coverity, scan-build and CodeSonar have belped us point out many
mistakes, codeql has remained silent (or had false positives).
For this little gain, I think we spend a disproportionate amount of work
on codeql maintanance.
We can try again in a future if we think it improves.
Assisted-by: Viktor Szakats
Closes#15798
Extend `INSTALL-CMAKE` document with the list of available options,
a short description and default values.
The list may not be 100% complete.
There are no component boundaries in CMake, so the line is blurry
between curl options, CMake options, CMake Find modules options.
I included certain CMake options that seemed useful, and/or have
dedicated use withing curl's CMake source. But, all CMake built-in
options are usable, as documented upstream in CMake.
The naming of the options has a heritage and the inconsistencies with
it, including a lack of clear namespace. This may be subject to future
updates, also after figuring out which name has special meaning within
CMake and/or CMake projects out of unwritten convention or something
more tangible.
CMake allows to initialize any internal variable via `-D`. This may be
useful to pre-initialize/override feature check results. The list
doesn't contain these, and they remain officially undocumented.
Also:
- make adjustments to keep the spellchecker happy.
- retrofit description changes to the cmake sources.
- stop documenting deprecated `Find*` variables.
Reported-by: Daniel Stenberg
Fixes https://github.com/curl/curl/discussions/14885Closes#15388
`Curl_bufq_unwrite()` used the head instead of the tail chunk to shrink
the bufq's content. Fix this and add test case that checks correct
behaviour.
Amended test 2601 accordingly.
Reported-by: Chris Stubbs
Closes#15136
The documented and mandated step has been to not use buildconf but to
invoke 'autoreconf -fi' for four years already.
This change only drops buildconf from the release tarball, it remains
present in git for now.
Follow-up to 85868537d6Closes#14412
Use these words and casing more consistently across text, comments and
one curl tool output:
AIX, ALPN, ANSI, BSD, Cygwin, Darwin, FreeBSD, GitHub, HP-UX, Linux,
macOS, MS-DOS, MSYS, MinGW, NTLM, POSIX, Solaris, UNIX, Unix, Unicode,
WINE, WebDAV, Win32, winbind, WinIDN, Windows, Windows CE, Winsock.
Mostly OS names and a few more.
Also a couple of other minor text fixups.
Closes#14360
As runtests.md and testcurl.md. Very few people actually need these as
manpages anyway.
With this, we have no more nroff formatted documents in git.
Closes#14324
If a malicious server can trigger a NULL dereference in curl or
otherwise cause curl to crash (and nothing worse), chances are big that
we do not consider that a security problem.
Closes#13974
Multipath TCP (MPTCP), standardized in RFC8684 [1], is a TCP extension
that enables a TCP connection to use different paths.
Multipath TCP has been used for several use cases. On smartphones, MPTCP
enables seamless handovers between cellular and Wi-Fi networks while
preserving established connections. This use-case is what pushed Apple
to use MPTCP since 2013 in multiple applications [2]. On dual-stack
hosts, Multipath TCP enables the TCP connection to automatically use the
best performing path, either IPv4 or IPv6. If one path fails, MPTCP
automatically uses the other path.
To benefit from MPTCP, both the client and the server have to support
it. Multipath TCP is a backward-compatible TCP extension that is enabled
by default on recent Linux distributions (Debian, Ubuntu, Redhat, ...).
Multipath TCP is included in the Linux kernel since version 5.6 [3]. To
use it on Linux, an application must explicitly enable it when creating
the socket. No need to change anything else in the application.
This attached patch adds an --mptcp option which allows the creation of
an MPTCP socket instead of TCP on Linux. If Multipath TCP is not
supported on the system, an error will be reported. It is important to
note that if the end server doesn't support MPTCP, the connection will
continue after a seamless fallback to TCP.
Link: https://www.rfc-editor.org/rfc/rfc8684.html [1]
Link: https://www.tessares.net/apples-mptcp-story-so-far/ [2]
Link: https://www.mptcp.dev [3]
Co-developed-by: Dorian Craps (@CrapsDorian) <doriancraps@gmail.com>
Co-developed-by: Olivier Bonaventure (@obonaventure) <Olivier.Bonaventure@uclouvain.be>
Co-developed-by: Matthieu Baerts (@matttbe) <matttbe@kernel.org>
Signed-off-by: Dorian Craps <dorian.craps@student.vinci.be>
Closes#13278
Fixes:
- in uds tests, abort also silently on os errors
- be conservative on the h3 goaway duration
- detect curl debug build and use in checks
- fix caddy version check for slight difference under linux
- set caddy default path fitting for linux
- fix deprecation warnings in valid time checks
FTP tests:
- add '--with-test-vsftpd=path' to configure
- use vsftpd default path suitable for linux
- add test_30 with plain FTP tests
- add test_31 with --ssl-reqd FTP tests
- add vsftpd to linux GHA for pytest workflows
Closes#13661
- Turn docs/INSTALL.cmake into a proper markdown file,
docs/INSTALL-CMAKE.md
- Move things around to divide the description into configuration,
building and installing sections
- Mention the more modern cmake options to configure, build and install,
but also retain the older variants as fallbacks
Closes#12772
This means words, phrases or things we have decided not to use - words that
are spelled right according to the dictionary but we want to avoid. In the
name of consistency and better documentation.
Closes#12764
- switch all invidual files documenting command line options into .md,
as the documentation is now markdown-looking.
- made the parser treat 4-space indents as quotes
- switch to building the curl.1 manpage using the "mainpage.idx" file,
which lists the files to include to generate it, instead of using the
previous page-footer/headers. Also, those files are now also .md
ones, using the same format. I gave them underscore prefixes to make
them sort separately:
_NAME.md, _SYNOPSIS.md, _DESCRIPTION.md, _URL.md, _GLOBBING.md,
_VARIABLES.md, _OUTPUT.md, _PROTOCOLS.md, _PROGRESS.md, _VERSION.md,
_OPTIONS.md, _FILES.md, _ENVIRONMENT.md, _PROXYPREFIX.md,
_EXITCODES.md, _BUGS.md, _AUTHORS.md, _WWW.md, _SEEALSO.md
- updated test cases accordingly
Closes#12751
curldown is this new file format for libcurl man pages. It is markdown
inspired with differences:
- Each file has a set of leading headers with meta-data
- Supports a small subset of markdown
- Uses .md file extensions for editors/IDE/GitHub to treat them nicely
- Generates man pages very similar to the previous ones
- Generates man pages that still convert nicely to HTML on the website
- Detects and highlights mentions of curl symbols automatically (when
their man page section is specified)
tools:
- cd2nroff: converts from curldown to nroff man page
- nroff2cd: convert an (old) nroff man page to curldown
- cdall: convert many nroff pages to curldown versions
- cd2cd: verifies and updates a curldown to latest curldown
This setup generates .3 versions of all the curldown versions at build time.
CI:
Since the documentation is now technically markdown in the eyes of many
things, the CI runs many more tests and checks on this documentation,
including proselint, link checkers and tests that make sure we capitalize the
first letter after a period...
Closes#12730
- use the correct include file
- make sure they are declared as in the header file
- fix minor nroff syntax mistakes (missing .fi)
These are verified by verify-synopsis.pl, which extracts the SYNPOSIS
code and runs it through gcc.
Closes#12402
- Move documentation of libcurl environment variables used only in debug
builds from libcurl-env into a separate document libcurl-env-dbg.
- Document more debug environment variables.
Previously undocumented or missing a description:
CURL_ALTSVC_HTTP, CURL_DBG_SOCK_WBLOCK, CURL_DBG_SOCK_WPARTIAL,
CURL_DBG_QUIC_WBLOCK, CURL_DEBUG, CURL_DEBUG_SIZE, CURL_GETHOSTNAME,
CURL_HSTS_HTTP, CURL_FORCETIME, CURL_SMALLREQSEND, CURL_SMALLSENDS,
CURL_TIME.
Closes https://github.com/curl/curl/pull/11811
- ipfs://<cid>
- ipns://<cid>
This allows you tu use ipfs in curl like:
curl ipfs://<cid>
and
curl ipns://<cid>
For more information consult the readme at:
https://curl.se/docs/ipfs.htmlCloses#8805
* Configure changes to detect AWS-LC
* CMakeLists.txt changes to detect AWS-LC
* Compile-time branches needed to support AWS-LC
* Correctly set OSSL_VERSION and report AWS-LC release number
* GitHub Actions script to build with autoconf and cmake against AWS-LC
AWS-LC is a BoringSSL/OpenSSL derivative
For more information see https://github.com/awslabs/aws-lc/Closes#10320
New cfilter HTTP-CONNECT for h3/h2/http1.1 eyeballing.
- filter is installed when `--http3` in the tool is used (or
the equivalent CURLOPT_ done in the library)
- starts a QUIC/HTTP/3 connect right away. Should that not
succeed after 100ms (subject to change), a parallel attempt
is started for HTTP/2 and HTTP/1.1 via TCP
- both attempts are subject to IPv6/IPv4 eyeballing, same
as happens for other connections
- tie timeout to the ip-version HAPPY_EYEBALLS_TIMEOUT
- use a `soft` timeout at half the value. When the soft timeout
expires, the HTTPS-CONNECT filter checks if the QUIC filter
has received any data from the server. If not, it will start
the HTTP/2 attempt.
HTTP/3(ngtcp2) improvements.
- setting call_data in all cfilter calls similar to http/2 and vtls filters
for use in callback where no stream data is available.
- returning CURLE_PARTIAL_FILE for prematurely terminated transfers
- enabling pytest test_05 for h3
- shifting functionality to "connect" UDP sockets from ngtcp2
implementation into the udp socket cfilter. Because unconnected
UDP sockets are weird. For example they error when adding to a
pollset.
HTTP/3(quiche) improvements.
- fixed upload bug in quiche implementation, now passes 251 and pytest
- error codes on stream RESET
- improved debug logs
- handling of DRAIN during connect
- limiting pending event queue
HTTP/2 cfilter improvements.
- use LOG_CF macros for dynamic logging in debug build
- fix CURLcode on RST streams to be CURLE_PARTIAL_FILE
- enable pytest test_05 for h2
- fix upload pytests and improve parallel transfer performance.
GOAWAY handling for ngtcp2/quiche
- during connect, when the remote server refuses to accept new connections
and closes immediately (so the local conn goes into DRAIN phase), the
connection is torn down and a another attempt is made after a short grace
period.
This is the behaviour observed with nghttpx when we tell it to shut
down gracefully. Tested in pytest test_03_02.
TLS improvements
- ALPN selection for SSL/SSL-PROXY filters in one vtls set of functions, replaces
copy of logic in all tls backends.
- standardized the infof logging of offered ALPNs
- ALPN negotiated: have common function for all backends that sets alpn proprty
and connection related things based on the negotiated protocol (or lack thereof).
- new tests/tests-httpd/scorecard.py for testing h3/h2 protocol implementation.
Invoke:
python3 tests/tests-httpd/scorecard.py --help
for usage.
Improvements on gathering connect statistics and socket access.
- new CF_CTRL_CONN_REPORT_STATS cfilter control for having cfilters
report connection statistics. This is triggered when the connection
has completely connected.
- new void Curl_pgrsTimeWas(..) method to report a timer update with
a timestamp of when it happend. This allows for updating timers
"later", e.g. a connect statistic after full connectivity has been
reached.
- in case of HTTP eyeballing, the previous changes will update
statistics only from the filter chain that "won" the eyeballing.
- new cfilter query CF_QUERY_SOCKET for retrieving the socket used
by a filter chain.
Added methods Curl_conn_cf_get_socket() and Curl_conn_get_socket()
for convenient use of this query.
- Change VTLS backend to query their sub-filters for the socket when
checks during the handshake are made.
HTTP/3 documentation on how https eyeballing works.
TLS improvements
- ALPN selection for SSL/SSL-PROXY filters in one vtls set of functions, replaces
copy of logic in all tls backends.
- standardized the infof logging of offered ALPNs
- ALPN negotiated: have common function for all backends that sets alpn proprty
and connection related things based on the negotiated protocol (or lack thereof).
Scorecard with Caddy.
- configure can be run with `--with-test-caddy=path` to specify which caddy to use for testing
- tests/tests-httpd/scorecard.py now measures download speeds with caddy
pytest improvements
- adding Makfile to clean gen dir
- adding nghttpx rundir creation on start
- checking httpd version 2.4.55 for test_05 cases where it is needed. Skipping with message if too old.
- catch exception when checking for caddy existance on system.
Closes#10349
- adding '--with-test-httpd=<path>' to configure non-standard apache2
install
- python env and base classes for running httpd
- basic tests for connectivity with h1/h2/h3
- adding test cases for truncated responses in http versions.
- adding goaway test for HTTP/3.
- adding "stuttering" tests with parallel downloads in chunks with
varying delays between chunks.
- adding a curltest module to the httpd server, adding GOAWAY test.
- mod_curltest now installs 2 handlers
- 'echo': writing as response body what came as request body
- 'tweak': with query parameters to tweak response behaviour
- marked known fails as skip for now
Closes#10175
- they are mostly pointless in all major jurisdictions
- many big corporations and projects already don't use them
- saves us from pointless churn
- git keeps history for us
- the year range is kept in COPYING
checksrc is updated to allow non-year using copyright statements
Closes#10205
Update bare GNU Make `Makefile.m32` to:
- Move objects into a subdirectory.
- Add support for MS-DOS. Tested with DJGPP.
- Add support for Watt-32 (on MS-DOS).
- Add support for AmigaOS.
- Rename `Makefile.m32` to `Makefile.mk`
- Replace `ARCH` with `TRIPLET`.
- Build `tool_hugehelp.c` proper (when tools are available).
- Drop MS-DOS compatibility macro `USE_ZLIB` (replaced by `HAVE_LIBZ`)
- Add support for `ZLIB_LIBS` to override `-lz`.
- Omit object files when building examples.
- Default `CC` to `gcc` once again, for convenience. (Caveat: compiler
name `cc` cannot be set now.)
- Set `-DCURL_NO_OLDIES` for examples, like autotools does.
- Delete `makefile.dj` files. Notice the configuration details and
defaults are not retained with the new method.
- Delete `makefile.amiga` files. A successful build needs a few custom
options. We're also not retaining all build details from the existing
Amiga make files.
- Rename `Makefile.m32` to `Makefile.mk` to reflect that they are not
Windows/MinGW32-specific anymore.
- Add support for new `CFG` options: `-map`, `-debug`, `-trackmem`
- Set `-DNDEBUG` by default.
- Allow using `-DOS=...` in all `lib/config-*.h` headers, syncing this
with `config-win32.h`.
- Look for zlib parts in `ZLIB_PATH/include` and `ZLIB_PATH/lib`
instead of bare `ZLIB_PATH`.
Note that existing build configurations for MS-DOS and AmigaOS likely
become incompatible with this change.
Example AmigaOS configuration:
```
export CROSSPREFIX=/opt/amiga/bin/m68k-amigaos-
export CC=gcc
export CPPFLAGS='-DHAVE_PROTO_BSDSOCKET_H'
export CFLAGS='-mcrt=clib2'
export LDFLAGS="${CFLAGS}"
export LIBS='-lnet -lm'
make -C lib -f Makefile.mk
make -C src -f Makefile.mk
```
Example MS-DOS configuration:
```
export CROSSPREFIX=/opt/djgpp/bin/i586-pc-msdosdjgpp-
export WATT_PATH=/opt/djgpp/net/watt
export ZLIB_PATH=/opt/djgpp
export OPENSSL_PATH=/opt/djgpp
export OPENSSL_LIBS='-lssl -lcrypt'
export CFG=-zlib-ssl
make -C lib -f Makefile.mk
make -C src -f Makefile.mk
```
Closes#9764
