curl/.github/workflows/non-native.yml
Viktor Szakats 60c3d04465
autotools: add support for 'unity' builds, enable in CI
Implement the "unity" builds as known from CMake, but for autotools.
It's limited to `lib` and `src` (CMake also supports it in `tests`).

Enable with: `--enable-unity` (disabled by default)

Unity builds speed up builds significantly. Cygwin and Windows builds in
particular, but the effect is noticeable on most systems. It also allows
discovering unity issues with autotools, benefitting also CMake when
building the same combination. In CI it makes turnaround times quicker.

This closes build performance with CMake. autotools still lags behind
because it builds shared and static libcurl in two, separate passes.
CMake does it in one. Manpage compilation isn't batched, it is in CMake.
After unity and test bundle support the slowest parts of the build are
the configuration phase (which is effectively a tedious, non-parallel,
compilation and/or linking of 300+ tiny programs. The next bottleneck
is compiling individual examples and finally test servers (only slow
with autotools).

The autotools implementation is slightly less efficient than CMake,
because 3 sources are permanently excluded while in CMake this isn't
necessary and solved more efficiently while building libtests. There is
also no 'unity' support for tests, making them a less efficient also.

Enable it in CI for most `configure` jobs. Except in GHA/dist (though
it works fine there too), to use the default config there. Also skip for
the Linux AWC-LC job where it made builds time a few seconds longer
(reason undiscovered.)

Autotools test suite builds compared between master -> `--enable-unity`:
- GHA/Linux: 32s -> 12s
  https://github.com/curl/curl/actions/runs/10705668823/job/29681617374
  https://github.com/curl/curl/actions/runs/10742978889/job/29796766297
- GHA/macOS: 37s -> 10s
  https://github.com/curl/curl/actions/runs/10705668813/job/29681632885
  https://github.com/curl/curl/actions/runs/10742978699/job/29796768875
- GHA/FreeBSD: 15m25 -> 10m58 (full workflow time, ~qemu)
  https://github.com/curl/curl/actions/runs/10705668811/job/29681607915
  https://github.com/curl/curl/actions/runs/10742978937/job/29796766115
- GHA/Cygwin: 3m32 -> 1m21
  https://github.com/curl/curl/actions/runs/10705668809/job/29681609965
  https://github.com/curl/curl/actions/runs/10742978645/job/29796756933
- GHA/MSYS2: 2m42 -> 50s
  https://github.com/curl/curl/actions/runs/10705668808/job/29681621166
  https://github.com/curl/curl/actions/runs/10742978662/job/29799739289
- GHA/mingw-w64: 5m32 -> 1m23
  https://github.com/curl/curl/actions/runs/10705668808/job/29681628787
  https://github.com/curl/curl/actions/runs/10742978662/job/29799741568

Closes #14815
2024-09-20 23:53:33 +02:00

213 lines
8.3 KiB
YAML

# Copyright (C) Viktor Szakats
#
# SPDX-License-Identifier: curl
name: non-native
'on':
push:
branches:
- master
- '*/ci'
paths-ignore:
- '**/*.md'
- '.circleci/**'
- 'appveyor.*'
- 'packages/**'
- 'plan9/**'
- 'projects/**'
- 'winbuild/**'
pull_request:
branches:
- master
paths-ignore:
- '**/*.md'
- '.circleci/**'
- 'appveyor.*'
- 'packages/**'
- 'plan9/**'
- 'projects/**'
- 'winbuild/**'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.sha }}
cancel-in-progress: true
permissions: {}
jobs:
netbsd:
name: 'NetBSD (cmake, openssl, clang)'
runs-on: ubuntu-latest
timeout-minutes: 30
strategy:
matrix:
arch: ['x86_64']
steps:
- uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4
- name: 'cmake'
uses: cross-platform-actions/action@cdc9ee69ef84a5f2e59c9058335d9c57bcb4ac86 # v0.25.0
with:
operating_system: 'netbsd'
version: '10.0'
architecture: ${{ matrix.arch }}
run: |
# https://pkgsrc.se/
sudo pkgin -y install cmake ninja-build pkg-config perl brotli heimdal openldap-client libssh2 libidn2 libpsl nghttp2 py311-impacket
cmake -B bld -G Ninja \
-DCMAKE_UNITY_BUILD=ON \
-DCURL_WERROR=ON \
-DENABLE_DEBUG=ON -DCMAKE_BUILD_TYPE=Debug -DCMAKE_RUNTIME_OUTPUT_DIRECTORY_DEBUG= \
-DENABLE_WEBSOCKETS=ON \
-DCURL_USE_OPENSSL=ON \
-DCURL_BROTLI=ON -DCURL_USE_GSSAPI=ON \
|| { cat bld/CMakeFiles/CMake*.yaml; false; }
cmake --build bld --config Debug --parallel 3
bld/src/curl --disable --version
if [ '${{ matrix.arch }}' = 'x86_64' ]; then # Slow on emulated CPU
cmake --build bld --config Debug --parallel 3 --target testdeps
export TFLAGS='-j0' # flakies: ~389 ~392 ~TFTP and more
cmake --build bld --config Debug --target test-ci
fi
echo '::group::build examples'
cmake --build bld --config Debug --parallel 3 --target curl-examples
echo '::endgroup::'
openbsd:
name: 'OpenBSD (cmake, libressl, clang)'
runs-on: ubuntu-latest
timeout-minutes: 30
strategy:
matrix:
arch: ['x86_64']
steps:
- uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4
- name: 'cmake'
uses: cross-platform-actions/action@cdc9ee69ef84a5f2e59c9058335d9c57bcb4ac86 # v0.25.0
with:
operating_system: 'openbsd'
version: '7.5'
architecture: ${{ matrix.arch }}
run: |
# https://openbsd.app/
sudo pkg_add cmake ninja perl brotli openldap-client libssh2 libidn2 libpsl nghttp2 python3 py3-impacket
cmake -B bld -G Ninja \
-DCMAKE_UNITY_BUILD=ON \
-DCURL_WERROR=ON \
-DENABLE_DEBUG=ON -DCMAKE_BUILD_TYPE=Debug -DCMAKE_RUNTIME_OUTPUT_DIRECTORY_DEBUG= \
-DENABLE_WEBSOCKETS=ON \
-DCURL_USE_OPENSSL=ON \
-DCURL_BROTLI=ON \
|| { cat bld/CMakeFiles/CMake*.yaml; false; }
cmake --build bld --config Debug --parallel 3
bld/src/curl --disable --version
if [ '${{ matrix.arch }}' = 'x86_64' ]; then # Slow on emulated CPU
cmake --build bld --config Debug --parallel 3 --target testdeps
export TFLAGS='-j8 ~3017 ~TFTP ~FTP' # FIXME: TFTP requests executed twice? Related: `curl: (69) TFTP: Access Violation`?
cmake --build bld --config Debug --target test-ci
fi
echo '::group::build examples'
cmake --build bld --config Debug --parallel 3 --target curl-examples
echo '::endgroup::'
freebsd:
name: 'FreeBSD (${{ matrix.build }}, openssl, ${{ matrix.compiler }}, ${{ matrix.arch }})'
runs-on: ubuntu-latest
timeout-minutes: 60
strategy:
matrix:
include:
- { build: 'autotools', arch: 'x86_64', compiler: 'clang' }
- { build: 'autotools', arch: 'arm64', compiler: 'clang' }
- { build: 'cmake' , arch: 'arm64', compiler: 'clang' }
fail-fast: false
steps:
- uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4
- name: 'autotools'
if: ${{ matrix.build == 'autotools' }}
uses: cross-platform-actions/action@cdc9ee69ef84a5f2e59c9058335d9c57bcb4ac86 # v0.25.0
with:
operating_system: 'freebsd'
version: '14.1'
architecture: ${{ matrix.arch }}
run: |
# https://ports.freebsd.org/
sudo pkg install -y autoconf automake libtool \
pkgconf brotli openldap26-client libidn2 libnghttp2 nghttp2 stunnel py311-impacket
autoreconf -fi
export CC='${{ matrix.compiler }}'
mkdir bld && cd bld && ../configure --enable-unity --enable-debug --enable-warnings --enable-werror \
--prefix="${HOME}"/install \
--enable-websockets \
--with-openssl \
--with-brotli --enable-ldap --enable-ldaps --with-libidn2 --with-libssh2 --with-nghttp2 --with-gssapi \
--disable-dependency-tracking || { tail -n 1000 config.log; false; }
make -j3 install
src/curl --disable --version
if [ '${{ matrix.arch }}' = 'x86_64' ]; then # Slow on emulated CPU
make -j3 -C tests
make test-ci V=1 TFLAGS='-j8 ~FTP'
fi
echo '::group::build examples'
make -j3 examples
echo '::endgroup::'
- name: 'cmake'
if: ${{ matrix.build == 'cmake' }}
uses: cross-platform-actions/action@cdc9ee69ef84a5f2e59c9058335d9c57bcb4ac86 # v0.25.0
with:
operating_system: 'freebsd'
version: '14.1'
architecture: ${{ matrix.arch }}
run: |
# https://ports.freebsd.org/
sudo pkg install -y cmake ninja perl5 \
pkgconf brotli openldap26-client libidn2 libnghttp2 nghttp2 stunnel py311-impacket
cmake -B bld -G Ninja \
'-DCMAKE_C_COMPILER=${{ matrix.compiler }}' \
-DCMAKE_UNITY_BUILD=ON \
-DCURL_WERROR=ON \
-DENABLE_DEBUG=ON -DCMAKE_BUILD_TYPE=Debug -DCMAKE_RUNTIME_OUTPUT_DIRECTORY_DEBUG= \
-DENABLE_WEBSOCKETS=ON \
-DCURL_USE_OPENSSL=ON \
-DCURL_BROTLI=ON -DCURL_USE_GSSAPI=ON \
|| { cat bld/CMakeFiles/CMake*.yaml; false; }
cmake --build bld --config Debug --parallel 3
bld/src/curl --disable --version
if [ '${{ matrix.arch }}' = 'x86_64' ]; then # Slow on emulated CPU
cmake --build bld --config Debug --parallel 3 --target testdeps
export TFLAGS='-j8 ~FTP'
cmake --build bld --config Debug --target test-ci
fi
echo '::group::build examples'
cmake --build bld --config Debug --parallel 3 --target curl-examples
echo '::endgroup::'
omnios:
name: 'OmniOS (autotools, openssl, gcc, amd64)'
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4
- name: 'autotools'
uses: vmactions/omnios-vm@2f97978b477e0a014767f65ecf3656b933541460 # v1
with:
usesh: true
# https://pkg.omnios.org/r151050/core/en/index.shtml
prepare: pkg install build-essential libtool
run: |
ln -s /usr/bin/gcpp /usr/bin/cpp # Some tests expect `cpp`, which is named `gcpp` in this env.
autoreconf -fi
mkdir bld && cd bld && ../configure --enable-unity --enable-debug --enable-warnings --enable-werror \
--prefix="${HOME}"/install \
--enable-websockets \
--with-openssl \
--disable-dependency-tracking || { tail -n 1000 config.log; false; }
gmake -j3 install
src/curl --disable --version
gmake -j3 -C tests
gmake test-ci V=1 TFLAGS='-j12 ~MQTT ~FTP'
echo '::group::build examples'
gmake -j3 examples
echo '::endgroup::'