Commit Graph

33 Commits

Author SHA1 Message Date
Daniel Stenberg
62515e8e9d
scripts/mdlinkcheck: markdown link checker
This script parses all markdown files in the repository, extracts all
links and verifies that they work.

It makes sure to only check the URLs once, even if used in multiple
links. There is a whitelist for URLs we deem unnecessary to check.

It uses curl to do the checks.

As a bonus, this makes it easy to run this check locally.

Closes #15742
2024-12-14 14:22:30 +01:00
Viktor Szakats
0aece8f66d
tidy-up: indent, whitespace, #error in make files
Replace invalid C with `#error`.

Cherry-picked from #14692
Closes #14997
2024-09-22 09:51:15 +02:00
Viktor Szakats
60c3d04465
autotools: add support for 'unity' builds, enable in CI
Implement the "unity" builds as known from CMake, but for autotools.
It's limited to `lib` and `src` (CMake also supports it in `tests`).

Enable with: `--enable-unity` (disabled by default)

Unity builds speed up builds significantly. Cygwin and Windows builds in
particular, but the effect is noticeable on most systems. It also allows
discovering unity issues with autotools, benefitting also CMake when
building the same combination. In CI it makes turnaround times quicker.

This closes build performance with CMake. autotools still lags behind
because it builds shared and static libcurl in two, separate passes.
CMake does it in one. Manpage compilation isn't batched, it is in CMake.
After unity and test bundle support the slowest parts of the build are
the configuration phase (which is effectively a tedious, non-parallel,
compilation and/or linking of 300+ tiny programs. The next bottleneck
is compiling individual examples and finally test servers (only slow
with autotools).

The autotools implementation is slightly less efficient than CMake,
because 3 sources are permanently excluded while in CMake this isn't
necessary and solved more efficiently while building libtests. There is
also no 'unity' support for tests, making them a less efficient also.

Enable it in CI for most `configure` jobs. Except in GHA/dist (though
it works fine there too), to use the default config there. Also skip for
the Linux AWC-LC job where it made builds time a few seconds longer
(reason undiscovered.)

Autotools test suite builds compared between master -> `--enable-unity`:
- GHA/Linux: 32s -> 12s
  https://github.com/curl/curl/actions/runs/10705668823/job/29681617374
  https://github.com/curl/curl/actions/runs/10742978889/job/29796766297
- GHA/macOS: 37s -> 10s
  https://github.com/curl/curl/actions/runs/10705668813/job/29681632885
  https://github.com/curl/curl/actions/runs/10742978699/job/29796768875
- GHA/FreeBSD: 15m25 -> 10m58 (full workflow time, ~qemu)
  https://github.com/curl/curl/actions/runs/10705668811/job/29681607915
  https://github.com/curl/curl/actions/runs/10742978937/job/29796766115
- GHA/Cygwin: 3m32 -> 1m21
  https://github.com/curl/curl/actions/runs/10705668809/job/29681609965
  https://github.com/curl/curl/actions/runs/10742978645/job/29796756933
- GHA/MSYS2: 2m42 -> 50s
  https://github.com/curl/curl/actions/runs/10705668808/job/29681621166
  https://github.com/curl/curl/actions/runs/10742978662/job/29799739289
- GHA/mingw-w64: 5m32 -> 1m23
  https://github.com/curl/curl/actions/runs/10705668808/job/29681628787
  https://github.com/curl/curl/actions/runs/10742978662/job/29799741568

Closes #14815
2024-09-20 23:53:33 +02:00
Daniel Stenberg
5c14d696fe
maketgz: move from / into scripts
Closes #14797
2024-09-05 17:07:41 +02:00
Dan Fandrich
59b419f1a5 CI: add a script and job to run cmakelint
A number of checks don't match our style or are buggy and so are disabled.

Co-authored-by: Viktor Szakats <vszakats@users.noreply.github.com>
Fixes #14580
Closes #14665
2024-08-27 08:33:44 -07:00
Daniel Stenberg
0066d169e8
managen: wordwrap long example lines in ASCII output
The entire ASCII version of the manpage word wraps at a fixed column,
while example command lines can easily go wider than so.

This change now makes manage work on wrapping long example command lines
to make them look nicer. And also to avoid triggering the build error
caused by too long lines in the output.

Quoted lines cannot be wrapped, so managen now errors out if they are
"too long". With this addition, the 'maxline' script is removed as it is
no longer needed.

Closes #14543
2024-08-16 08:57:19 +02:00
Daniel Stenberg
badbd4eb46
manpage: ensure a maximum width for the text version
... using the new script 'maxline' to which we specify the maximum
number of columns we allow any single line to be, or it will cause an
error.

Starting out with a max width at 100 columns.

Bonus: shorten the long line in the --ipfs-gateway section.

Closes #14423
2024-08-08 17:49:37 +02:00
Daniel Stenberg
86039e6e42
verify-release: shell script that verifies a release tarball
This script remakes a provided curl release and verifies that the newly
built version is identical to the original file.

Due to bugs in releases up to and including curl 8.9.1, it does not work
on tarballs generated before commit 754acd1a9d.

Closes #14350
2024-08-02 23:05:41 +02:00
Daniel Stenberg
754acd1a9d
dist: fix reproducible build from release tarball
Make it possible to rebuild an identical copy from a release tarball. It
was previously only possible from a checked out git repository.

- add release-tools.sh to dist
- keep Makefile.dist around to include it in dist
- regenerate tool_huge.c with the new version in dist
- fix the dist CI job to not do make clean like before

Closes #14336
2024-08-02 08:28:56 +02:00
Daniel Stenberg
d28f74913c
dmaketgz: release tarball generation using docker
For easier reproducibility.

Mention using this script in RELEASE-PROCEDURE

Closes #13388
2024-04-16 23:38:55 +02:00
Daniel Stenberg
1ccf1cd993
scripts/managen: the new name and home for the manpage generator
It was previously docs/cmdline-opts/gen.pl

Closes #13089
2024-03-08 17:05:08 +01:00
Daniel Stenberg
f03c85635f
docs: ascii version of manpage without nroff
Create ASCII version of manpage without nroff

 - build src/tool_hugegelp.c from the ascii manpage
 - move the the manpage and the ascii version build to docs/cmdline-opts
 - remove all use of nroff from the build process
 - should make the build entirely reproducible (by avoiding nroff)

 - partly reverts 2620aa9 to build libcurl option man pages one by one
   in cmake because the appveyor builds got all crazy until I did

The ASCII version of the manpage

 - is built with gen.pl, just like the manpage is
 - has a right-justified column making the appearance similar to the previous
   version
 - uses a 4-space indent per level (instead of the old version's 7)
 - does not do hyphenation of words (which nroff does)

History

  We first made the curl build use nroff for building the hugehelp file in
  December 1998, for curl 5.2.

Closes #13047
2024-03-06 15:55:59 +01:00
Dan Fandrich
ab173d1443 configure: Don't build shell completions when disabled
With the recent changes to completion file building, the files were
built always and only installation was selectively disabled.  Now, when
they are disabled they aren't even built, avoiding a build-time error in
environments where it's not possible to run the curl binary that was
just created (e.g. if library paths were not set up correctly).

Follow-up to 0f7aba83c

Reported-by: av223119 on github
Fixes #13027
Closes #13030
2024-03-02 16:36:15 -08:00
Dan Fandrich
0f7aba83cc configure: Don't make shell completions without perl
The code that attempted to skip building the shell completions didn't
work properly and tried to build them even if perl wasn't available.
This step, as well as the install step, is now properly skipped without
perl.

Follow-up to 89733e2dd

Closes #13022
2024-03-01 01:16:24 -08:00
Dan Fandrich
89733e2dd2 configure: build & install shell completions when enabled
The --with-fish-functions-dir and --with-zsh-functions-dir options
currently have no effect on a normal build because the scripts/ directory
where they're used is not built. Add scripts/ to a normal build and
change the completion options to default to off to preserve the existing
behaviour.

Closes: #12906
2024-02-29 16:41:31 -08:00
Daniel Stenberg
eefcc1bda4
docs: introduce "curldown" for libcurl man page format
curldown is this new file format for libcurl man pages. It is markdown
inspired with differences:

- Each file has a set of leading headers with meta-data
- Supports a small subset of markdown
- Uses .md file extensions for editors/IDE/GitHub to treat them nicely
- Generates man pages very similar to the previous ones
- Generates man pages that still convert nicely to HTML on the website
- Detects and highlights mentions of curl symbols automatically (when
  their man page section is specified)

tools:

- cd2nroff: converts from curldown to nroff man page
- nroff2cd: convert an (old) nroff man page to curldown
- cdall: convert many nroff pages to curldown versions
- cd2cd: verifies and updates a curldown to latest curldown

This setup generates .3 versions of all the curldown versions at build time.

CI:

Since the documentation is now technically markdown in the eyes of many
things, the CI runs many more tests and checks on this documentation,
including proselint, link checkers and tests that make sure we capitalize the
first letter after a period...

Closes #12730
2024-01-23 00:29:02 +01:00
Daniel Stenberg
d6447b1bd2
scripts: add schemetable.c
This tool generates a scheme-matching table.

It iterates over a number of different initial and shift values in order
to find the hash algorithm that needs the smallest possible table.

The generated hash function, table and table size then needs to be used
by the url.c:Curl_getn_scheme_handler() function.
2023-11-19 13:57:31 +01:00
Daniel Stenberg
2bc1d775f5
copyright: update all copyright lines and remove year ranges
- they are mostly pointless in all major jurisdictions
- many big corporations and projects already don't use them
- saves us from pointless churn
- git keeps history for us
- the year range is kept in COPYING

checksrc is updated to allow non-year using copyright statements

Closes #10205
2023-01-03 09:19:21 +01:00
max.mehl
ad9bc5976d
copyright: make repository REUSE compliant
Add licensing and copyright information for all files in this repository. This
either happens in the file itself as a comment header or in the file
`.reuse/dep5`.

This commit also adds a Github workflow to check pull requests and adapts
copyright.pl to the changes.

Closes #8869
2022-06-13 09:13:00 +02:00
Daniel Stenberg
8e22fc68e7
scripts: move three scripts from lib/ to scripts/
Move checksrc.pl, firefox-db2pem.sh and mk-ca-bundle.pl since they don't
particularly belong in lib/

Also created an EXTRA_DIST= in scripts/Makefile.am instead of specifying
those files in the root Makefile.am

Closes #8625
2022-03-23 15:26:11 +01:00
Daniel Stenberg
4d2f800677
curl.se: new home
Closes #6172
2020-11-04 23:59:47 +01:00
Daniel Stenberg
9a8b3b3e13
copyright: fix out-of-date copyright ranges and missing headers
Reported by the new script 'scripts/copyright.pl'. The script has a
regex whitelist for the files that don't need copyright headers.

Removed three (mostly usesless) README files from docs/

Closes #5141
2020-03-24 15:05:59 +01:00
Simon Legner
e075b2149b
scripts/completion.pl: also generate fish completion file
This is the renamed script formerly known as zsh.pl

Closes #3545
2019-03-02 11:31:18 +01:00
Dave Reisner
0b4ccc97f2
scripts: include _curl as part of CLEANFILES
Closes #2718
2018-07-08 23:20:11 +02:00
Daniel Stenberg
4cbaee45a9 scripts/make: use $(EXEEXT) for executables
Reported-by: bodop

Fixes #771
2016-04-18 15:51:32 +02:00
Daniel Stenberg
4af40b3646 URLs: change all http:// URLs to https:// 2016-02-03 00:19:02 +01:00
Alessandro Ghedini
ebfe00c050 scripts: don't generate and install zsh completion when cross-compiling 2016-01-11 23:32:30 +01:00
Alessandro Ghedini
fb7cbf75a5 scripts: fix zsh completion generation
The script should use the just-built curl, not the system one. This fixes
zsh completion generation when no system curl is installed.
2016-01-11 23:32:30 +01:00
Dan Fandrich
28533d3277 scripts/Makefile: build zsh script even in an out-of-tree build 2015-12-16 22:13:02 +01:00
Daniel Stenberg
3b51652328 scripts/makefile: add standard header 2015-12-09 00:34:39 +01:00
Daniel Stenberg
f08fea7169 scripts/Makefile: fix GNUism and survive no perl
Closes #555

Reported-by: Thomas Klausner
2015-12-09 00:34:12 +01:00
Daniel Stenberg
19a8afb3c0 zsh install: fix DESTDIR support
Reported-by: Mohammad AlSaleh
2015-11-27 23:24:07 +01:00
Daniel Shahaf
be0d4141af build: Install zsh completion
Fixes #534
Closes #537
2015-11-24 22:22:01 +01:00