A command line tool and library for transferring data with URL syntax, supporting DICT, FILE, FTP, FTPS, GOPHER, GOPHERS, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, MQTT, POP3, POP3S, RTMP, RTMPS, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS, TELNET, TFTP, WS and WSS. libcurl offers a myriad of powerful features
cclientcurlftpgopherhacktoberfesthttphttpsimapsldaplibcurllibrarymqttpop3scpsftptransfer-datatransferring-datauser-agentwebsocket
e7adf3e837
TL;DR: Save 10 minutes of CI time for GHA/macos jobs using pre-fills and
add pre-fill verification for Apple and Windows. Also restores Xcode job
and saves 1.5-10 minutes configuring iOS jobs.
Pre-filling feature detection results can bring down the CMake configure
step to ~5 seconds on most GHA runners, ~10 seconds in slow envs like
Cygwin/MSYS2.
The potential savings per job are:
- 5-40 (average 19) seconds on GHA/macos (33 jobs)
- ~10 seconds on GHA for iOS GNU Makefile (1 job)
- 1.5-10 minutes on GHA for iOS Xcode generator (1 job)
- 10 seconds on GHA/linux with native Ubuntu (12 jobs)
- 40 seconds for Cygwin/MSYS2 (2 jobs)
- 5-10 seconds for virtualized BSDs, native CPU (3 jobs)
- ~60 seconds for virtualized BSDs, emulated CPU (1 job)
On native Windows pre-filling has been in place for a long time and
saving 8 minutes (VS2019-VS2015) to 1.5-2 minutes (VS2022), 3 minutes
(VS2022 UWP), and 30-60 seconds (MinGW), per CI job.
The downside is that detection results need to be manually collected and
filtered to those that universally apply to all platforms that they are
enabled on. Another downside is that by using a cache, we're not running
the actual detections, and thus won't catch regressions in them. It
means we must make sure that the cache is solid and matches with actual
detections results. An upside is that it gives a rough overview of which
features are available on which platforms. Another upside is pre-filled
values do work for feature detections skipped for cross-builds, e.g.
`HAVE_WRITABLE_ARGV`.
This PR adds a pre-fill cache that supports all Unixes (except OmniOS)
used in CI, and makes it usable with an internal option. It also enables
it for GHA/macos CI jobs, where the maximum savings are. And also for
the two iOS [1] and two Cygwin/MSYS2 jobs. The latters don't have
pre-fill checks and we can drop them if they turn into a hassle.
Saving:
- 10 minutes of CI time per GHA/macos workflow run. [2]
- ~80 seconds per GHA/windows workflow run with Cygwin/MSYS2.
(offsetting the cost of pre-fill verifications)
- 1.5-10 minutes per GHA/non-native runs with iOS jobs. [3]
You can enable pre-fill locally with `-D_CURL_PREFILL=ON`. It's
experimental, and if you experience a problem, file a PR or an Issue.
This PR also adds a pre-fill checker for macOS and MinGW/MSVC Windows
GHA jobs to catch if the cache diverges from real detections. It also
adds this logic to AppVeyor, but doesn't enable it due to the perf
penalty of 2 minutes mininum.
The pre-fill checker works by configuring out-of-tree with and without
pre-fill, then diffing their `lib/curl_config.h` outputs.
Exceptions are 3 detection results exposed indirectly [4], and missing
to expose 2, of which one is the C89 header `stddef.h`. While we assume
the C99 `stdint.h` available outside iOS. We can expose them in the
future, if necessary.
The pre-fill checks cost in total:
- ~20 seconds for macOS
- ~40 seconds for MinGW on GHA
- ~80 seconds for MSVC on GHA (UWP would be 2x this)
An extra time saving potential is caching type sizes. They are
well-known, and seldom change, esp. in CI. GHA/Windows jobs spend 8-17
seconds per job on these ~12 feature checks. ~5s on Cygwin/MSYS2. Couple
of seconds on other platforms. (This PR doesn't make this optimization.)
Another opportunity is doing the same for autotools, which typically
spends more time in the configuration step than cmake.
[1] Xcode job restored as a
follow-up to
|
||
|---|---|---|
| .circleci | ||
| .github | ||
| CMake | ||
| docs | ||
| include | ||
| lib | ||
| LICENSES | ||
| m4 | ||
| packages | ||
| plan9 | ||
| projects | ||
| scripts | ||
| src | ||
| tests | ||
| winbuild | ||
| .dir-locals.el | ||
| .git-blame-ignore-revs | ||
| .gitattributes | ||
| .gitignore | ||
| .mailmap | ||
| acinclude.m4 | ||
| appveyor.sh | ||
| appveyor.yml | ||
| buildconf | ||
| CHANGES.md | ||
| CMakeLists.txt | ||
| configure.ac | ||
| COPYING | ||
| curl-config.in | ||
| Dockerfile | ||
| GIT-INFO.md | ||
| libcurl.pc.in | ||
| Makefile.am | ||
| README | ||
| README.md | ||
| RELEASE-NOTES | ||
| renovate.json | ||
| REUSE.toml | ||
| SECURITY.md | ||
curl is a command-line tool for transferring data specified with URL syntax. Learn how to use curl by reading the manpage or everything curl.
Find out how to install curl by reading the INSTALL document.
libcurl is the library curl is using to do its job. It is readily available to be used by your software. Read the libcurl manpage to learn how.
Open Source
curl is Open Source and is distributed under an MIT-like license.
Contact
Contact us on a suitable mailing list or use GitHub issues/ pull requests/ discussions.
All contributors to the project are listed in the THANKS document.
Commercial support
For commercial support, maybe private and dedicated help with your problems or applications using (lib)curl visit the support page.
Website
Visit the curl website for the latest news and downloads.
Source code
Download the latest source from the Git server:
git clone https://github.com/curl/curl.git
Security problems
Report suspected security problems via our HackerOne page and not in public.
Notice
curl contains pieces of source code that is Copyright (c) 1998, 1999 Kungliga Tekniska Högskolan. This notice is included here to comply with the distribution terms.
Backers
Thank you to all our backers 🙏 Become a backer.
Sponsors
Support this project by becoming a sponsor.