They use Visual Studio generators, which are multi-target.
The build command does the Release/Debug selection via `--config`.
Also:
- appveyor: drop unnecessary conditional for 3 options.
To sync with GHA.
- appveyor: drop unused `-DCMAKE_INSTALL_PREFIX=`.
To sync with GHA.
- sync cmake option order between GHA and appveyor.
Closes#16372
TL;DR: Save 10 minutes of CI time for GHA/macos jobs using pre-fills and
add pre-fill verification for Apple and Windows. Also restores Xcode job
and saves 1.5-10 minutes configuring iOS jobs.
Pre-filling feature detection results can bring down the CMake configure
step to ~5 seconds on most GHA runners, ~10 seconds in slow envs like
Cygwin/MSYS2.
The potential savings per job are:
- 5-40 (average 19) seconds on GHA/macos (33 jobs)
- ~10 seconds on GHA for iOS GNU Makefile (1 job)
- 1.5-10 minutes on GHA for iOS Xcode generator (1 job)
- 10 seconds on GHA/linux with native Ubuntu (12 jobs)
- 40 seconds for Cygwin/MSYS2 (2 jobs)
- 5-10 seconds for virtualized BSDs, native CPU (3 jobs)
- ~60 seconds for virtualized BSDs, emulated CPU (1 job)
On native Windows pre-filling has been in place for a long time and
saving 8 minutes (VS2019-VS2015) to 1.5-2 minutes (VS2022), 3 minutes
(VS2022 UWP), and 30-60 seconds (MinGW), per CI job.
The downside is that detection results need to be manually collected and
filtered to those that universally apply to all platforms that they are
enabled on. Another downside is that by using a cache, we're not running
the actual detections, and thus won't catch regressions in them. It
means we must make sure that the cache is solid and matches with actual
detections results. An upside is that it gives a rough overview of which
features are available on which platforms. Another upside is pre-filled
values do work for feature detections skipped for cross-builds, e.g.
`HAVE_WRITABLE_ARGV`.
This PR adds a pre-fill cache that supports all Unixes (except OmniOS)
used in CI, and makes it usable with an internal option. It also enables
it for GHA/macos CI jobs, where the maximum savings are. And also for
the two iOS [1] and two Cygwin/MSYS2 jobs. The latters don't have
pre-fill checks and we can drop them if they turn into a hassle.
Saving:
- 10 minutes of CI time per GHA/macos workflow run. [2]
- ~80 seconds per GHA/windows workflow run with Cygwin/MSYS2.
(offsetting the cost of pre-fill verifications)
- 1.5-10 minutes per GHA/non-native runs with iOS jobs. [3]
You can enable pre-fill locally with `-D_CURL_PREFILL=ON`. It's
experimental, and if you experience a problem, file a PR or an Issue.
This PR also adds a pre-fill checker for macOS and MinGW/MSVC Windows
GHA jobs to catch if the cache diverges from real detections. It also
adds this logic to AppVeyor, but doesn't enable it due to the perf
penalty of 2 minutes mininum.
The pre-fill checker works by configuring out-of-tree with and without
pre-fill, then diffing their `lib/curl_config.h` outputs.
Exceptions are 3 detection results exposed indirectly [4], and missing
to expose 2, of which one is the C89 header `stddef.h`. While we assume
the C99 `stdint.h` available outside iOS. We can expose them in the
future, if necessary.
The pre-fill checks cost in total:
- ~20 seconds for macOS
- ~40 seconds for MinGW on GHA
- ~80 seconds for MSVC on GHA (UWP would be 2x this)
An extra time saving potential is caching type sizes. They are
well-known, and seldom change, esp. in CI. GHA/Windows jobs spend 8-17
seconds per job on these ~12 feature checks. ~5s on Cygwin/MSYS2. Couple
of seconds on other platforms. (This PR doesn't make this optimization.)
Another opportunity is doing the same for autotools, which typically
spends more time in the configuration step than cmake.
[1] Xcode job restored as a
follow-up to be5f20202c#16302
[2] GHA/macos cmake configure times in seconds:
Job | Bef. | After | Gain
:----------------------------------------------- | ----: | ----: | ----:
CM clang GnuTLS !ldap krb5 | 21.2 | 4.5 | 16.7
CM clang LibreSSL !ldap heimdal c-ares +examples | 13.3 | 3.9 | 9.4
CM clang OpenSSL +static libssh +examples | 20.0 | 4.6 | 15.4
CM clang OpenSSL IDN clang-tidy~ (w/chkprefill) | 15.7 | 18.6 | -2.9
CM clang OpenSSL gsasl rtmp AppleIDN | 25.0 | 4.7 | 20.3
CM clang OpenSSL torture !FTP | 15.3 | 4.5 | 10.8
CM clang OpenSSL torture FTP | 25.0 | 5.9 | 19.1
CM clang SecureTransport debug | 18.0 | 3.8 | 14.2
CM clang macos-13 SecureTransport | 45.8 | 12.4 | 33.4
CM clang macos-14 SecureTransport | 15.8 | 4.6 | 11.2
CM clang macos-15 SecureTransport | 26.8 | 6.1 | 20.7
CM clang mbedTLS openldap brotli zstd | 15.1 | 6.5 | 8.6
CM clang wolfSSL !ldap brotli zstd | 27.0 | 4.4 | 22.6
CM gcc-12 GnuTLS !ldap krb5 | 39.1 | 8.7 | 30.4
CM gcc-12 LibreSSL !ldap heimdal c-ares +examples| 23.8 | 7.2 | 16.6
CM gcc-12 OpenSSL +static libssh +examples | 20.7 | 8.5 | 12.2
CM gcc-12 OpenSSL gsasl rtmp AppleIDN | 23.1 | 10.1 | 13.0
CM gcc-12 SecureTransport debug | 21.1 | 4.8 | 16.3
CM gcc-12 mbedTLS openldap brotli zstd | 21.4 | 5.8 | 15.6
CM gcc-12 wolfSSL !ldap brotli zstd | 21.1 | 6.9 | 14.2
CM gcc-14 macos-13 SecureTransport | 61.9 | 18.7 | 43.2
CM gcc-14 macos-14 SecureTransport | 30.5 | 6.4 | 24.1
CM gcc-14 macos-15 SecureTransport | 32.7 | 8.4 | 24.3
CM llvm@15 GnuTLS !ldap krb5 | 21.1 | 7.5 | 13.6
CM llvm@15 LibreSSL !ldap heimdal c-ares +exampl~| 24.6 | 6.8 | 17.8
CM llvm@15 OpenSSL +static libssh +examples | 19.0 | 6.4 | 12.6
CM llvm@15 OpenSSL gsasl rtmp AppleIDN | 19.0 | 8.2 | 10.8
CM llvm@15 SecureTransport debug | 18.0 | 5.4 | 12.6
CM llvm@15 macos-13 SecureTransport | 66.2 | 25.7 | 40.5
CM llvm@15 macos-14 SecureTransport | 31.9 | 6.1 | 25.8
CM llvm@15 mbedTLS openldap brotli zstd | 19.5 | 8.9 | 10.6
CM llvm@15 wolfSSL !ldap brotli zstd | 24.3 | 5.9 | 18.4
CM llvm@18 macos-15 SecureTransport | 33.8 | 6.4 | 27.4
Total | 856.8 | 257.3 | 599.5
Before: https://github.com/curl/curl/actions/runs/13311042735/job/37173478424
After: https://github.com/curl/curl/actions/runs/13313927119/job/37183206426?pr=15841
[3] iOS:
Before: https://github.com/curl/curl/actions/runs/13326401704?pr=15841
After: https://github.com/curl/curl/actions/runs/13332177764?pr=15841
[4] detection results exposed indirectly in `curl_config.h`:
- `HAVE_FILE_OFFSET_BITS` via `_FILE_OFFSET_BITS`
- `HAVE_GETHOSTBYNAME_R_*_REENTRANT` via `NEED_REENTRANT`
- `HAVE_SOCKADDR_IN6_SIN6_ADDR` via `USE_IPV6`
Closes#15841
Before this patch curl code was redefining `getaddrinfo` and
`freeaddrinfo` system symbols to plug in its debug wrappers. This was
causing pains to avoid applying the redefinitions to system headers
defining these functions, and to the local debug wrappers. Especially
in unity builds. It also required workarounds for systems where these
symbols are already macros.
Introduce curl-namespaced macros for these functions and use them.
This allows to drop all workarounds and makes it work in all envs,
local targets and unity/bundle combinations.
Also drop GHA/windows workaround and use the same unity batch across
all jobs. Follow-up to 29e4eda631#16272
Ref: #16272
Ref: 71cf0d1fca#14772
Ref: 3efba94f77#14765
Ref: f7d5f47059#14399Closes#16274
Default curl unity builds make a single unit for each target. It means
all target sources are batched together and built in a single compiler
invocation. With multi-core CPUs this doesn't always result in the best
possible performance. This patch enables smaller batches for jobs where
this resulted in shorter build times. These jobs are Cygwin, MSYS2,
MinGW, running on the Windows runners.
Use batch of 30 (meaning 30 sources batched into units), and 32 for
Cygwin/MSYS2 to avoid a unity fallout that's subject to a different PR.
(CMake allows to set the number of sources per unit, not the number
of units, though the latter may be more practical to max out CPU cores.)
Also override to not batch the `curlu` target because batching lost
a little bit of time there, due to the already existing parallelism when
building the `testdeps` targets.
For jobs on the macOS and Linux runners jobs were already mostly single
digit or below teen seconds, and batching didn't improve on them
noticeably. On VM jobs, the virtual CPUs are limited, so I didn't
make a try. In AppVeyor and GHA vcpkg jobs (using msbuild), batching
didn't result in conclusive or any gains.
Build times in seconds (curl + testdeps):
Job | Before | After w curlu=0 | Gain
:--------------------| :-------------- | :-------------- | :---
cygwin, CM | 19 + 32 = 51 | 12 + 32 = 44 | 7
msys2, CM | 7 + 15 = 22 | 5 + 14 = 19 | 3
mingw gcc U, CM | 19 + 30 = 49 | 13 + 32 = 45 | 4
mingw ucrt, CM | 32 + 42 = 74 | 15 + 43 = 58 | 16
mingw clang, CM | 15 + 21 = 36 | 8 + 21 = 29 | 7
mingw uwp, CM | 30 + 40 = 70 | 14 + 40 = 54 | 16
mingw gcc, CM | 20 + 31 = 51 | 12 + 31 = 43 | 8
mingw x86, CM | 35 + 40 = 75 | 15 + 38 = 53 | 22
dl-mingw, CM 9.5.0 | 88 + 99 = 187 | 42 + 101 = 143 | 44
dl-mingw, CM 7.3.0 U | 24 + 32 = 56 | 17 + 35 = 52 | 4
Total | | | 131
Total gain per GHA/windows workflow runs: 2m11s
Runs:
Before: https://github.com/curl/curl/actions/runs/13220256084/job/36904342259
After: https://github.com/curl/curl/actions/runs/13220383702/job/36904602981https://github.com/curl/curl/actions/runs/13220613141/job/36905170104https://github.com/curl/curl/actions/runs/13222019443/job/36908358550
With curlu tweak: https://github.com/curl/curl/actions/runs/13222239255/job/36908782462
Ref: 116950a250#16265Closes#16272
To match the rest of Windows jobs.
dl-mingw, CM 9.5.0-x86_64 schannel: 4m24s -> 2m56s
dl-mingw, CM 7.3.0-x86_64 schannel U: 4m37s -> 3m10s
(based on a few runs.)
Closes#16271
Use the last known good release of Git for Windows by installing it
manually. It restores `runtests.pl` performance to the levels before
the October 2024 and this week's fallouts which gradually deployed
MSYS2 runtimes with pipe/signal/concurrency issues.
Also:
- restore vcpkg job's test parallelism to `-j8` (from `-j4`).
- keep using the default shell for jobs not running tests.
To avoid the unnecessary Git for Windows install overhead.
Upsides:
- good performance again.
- easy to experiment with any version.
Downsides:
- installing the Git for Windows package takes 15-30 seconds.
- we're pinned to an old package version.
- no canary to tell when the issue is fixed on the runner images.
Unknown:
- stability. (no MSYS2 runtimes were ever stable and it's difficult
to quantify if a version improves or worsens stability/flakiness, and
intermittent env failures.
Follow-up to 1bf774df57#16217
Follow-up to 5f9411f953#15380Closes#16265
- drop `--quiet 2` option where used, to have uniform output.
- replace `apt` with `apt-get` in one job. sync options with rest.
- replace deprecated `apt-key` command with the alternative recommended
by `apt-key(8)`.
- drop stray `cd /tmp`, no longer needed after migrating to GHA.
- shorten `--option Dpkg::Use-Pty=0` to `-o Dpkg::Use-Pty=0`.
- add `-o Dpkg::Use-Pty=0` to hide `apt-get` progress bars taking
vertical log space, where missing.
- drop `-y --no-install-suggests --no-install-recommends` `apt-get`
options. They are the default in the ubuntu-24.04 image.
- GHA/distcheck: move `name:` to top in steps where not there.
- scripts/cijobs.pl: catch `apt-get` lines with the `-o` option.
Closes#16127
RFC 6455 Section 5.2 notes that for bits RSV1, RSV2, and RSV3 of the
framing header, a non-zero value that is not defined by a negotiated
extension MUST Fail the WebSocket connection.
Test 2310 verifies
Closes#16069
They play better with Unixy shells. The compiler has been supporting
dash options since its early versions.
Also fix to detect warnings options passed in dash-style.
Closes#16063
- move checkout step right before use.
- quotes in yaml and bash.
- unfold single-line run commands.
- set `CMAKE_WARN_DEPRECATED=OFF` for CMake with Android NDK to avoid
a wall of useless deprecation warnings hiding useful output.
These warnings happen in 3rd-party scripts, and unfixable in curl.
Closes#16042
To match cmake builds.
- GHA/windows: allow autotools UWP builds.
- detect UWP and add to `buildinfo.txt`.
Consider it enabled if `CPPFLAGS` contains `-DWINSTORECOMPAT`.
- disable telnet with UWP.
- enable Unicode with UWP.
- do not use `wldap32` with UWP.
- do not enable `USE_WIN32_CRYPTO` with UWP.
- make sure to link to `ws2_32` in UWP builds.
To fix `undefined reference to `in6addr_any'` when linking
`tests/server` programs. More in the comment.
Closes#16020
CMake (as of 3.31.2) doesn't fully recognize mingw-w64 with
`CMAKE_SYSTEM_NAME=WindowsStore`.
The manual logic works around it.
Also move existing DJGPP workaround to the same block.
Closes#16019
- fix `find` commands to not miss items.
- call `file` on the built files in `curl -V` steps.
To give more feedback on what was built.
- add `curl info` step for cross-jobs that can't do a `curl -V`.
It lists the files built and calls `file` on them.
- appveyor: make a VS2010 32-bit to match the VS2008 job it replaced.
Follow-up to d34aeecb08#15934
- GHA/windows: drop the word "old" from standalone mingw-w64 jobs to not
conflate it with "old mingw" we no longer support (while also keeping
it short).
Cherry-picked from #15975Closes#16001
- make `curl_dependency_option()` more generic.
- extend `CURL_BROTLI` and `CURL_ZSTD` options to accept
`AUTO` in addition to existing `ON` and `OFF`.
- change `CURL_BROTLI` and `CURL_ZSTD` option default
to `AUTO`. Was: `OFF`.
It brings cmake behavior closer to `./configure`.
Still different:
- `./configure` defaults to `off` which means to check default
locations. cmake checks more locations by default.
(Also tried `NO_CMAKE_PATH`, but then it checked less locations.)
- cmake returns both `brotlicommon` and `brotlidec` libs,
while `./configure` only returns the latter.
- ci: drop explicit cmake options, that are now unnecessary.
- GHA/configure-vs-cmake: make adjustments to make tests pass.
Closes#15431
This makes `runtests.pl` run the final executables directly.
Before this patch it called the autotools/libtool wrapper tool, which
then called the final executables.
This solution was already used for `curl.exe`.
Applies to tests run in the `mingw, AM x86_64 c-ares U` job, which still
shows unexplained flakiness.
Also makes tests finish 45 seconds faster.
Ref: #14854
Follow-up to 1a2d38c47c#15437Closes#15662
The MSVC UWP job in CI did not actually enable UWP. Fix this and
the fallouts discovered after enabling it.
- GHA/windows: make sure to enable UWP in MSVC vcpkg UWP job.
Use the CMake options and C flags already used for mingw-w64, but use
`WINAPI_FAMILY_PC_APP` instead of the deprecated `WINAPI_FAMILY_APP`.
(The former is not supported by mingw-w64, so leave it there as-is.)
Follow-up to cb22cfca69#14077
- GHA/windows: by default the MSVC UWP job became 2x-3x slower than
others after actually enabling UWP. Most of it is caused by
CMake/MSBuild automatically building full APPX containers for each
`.exe` target. This includes 21 CMake feature detections. Each
detection app is built into a 15MB APPX project, with code signing,
logos, etc. Example:
https://github.com/curl/curl/actions/runs/12056968170/job/33620610958
Disable this overhead for curl build targets via custom
`CMAKE_VS_GLOBALS` options. I've found no way to apply them to feature
detection targets, so those remain slow.
- cmake: automatically enable Unicode for UWP builds. It's required.
Also stop enabling it manually in the existing CI job.
- tests: fix `getpid()` use for Windows UWP:
```
tests\server\util.c(281,21): warning C4013: 'getpid' undefined; assuming extern returning int
```
Ref: https://github.com/curl/curl/actions/runs/12061215311/job/33632904249#step:11:38
- src/tool_doswin: disable `GetLoadedModulePaths()` for UWP.
mingw-w64 UWP was okay with this, but MS SDK headers are not.
This makes `--dump-module-paths` return empty for UWP builds.
```
src\tool_doswin.c(620,3): error C2065: 'MODULEENTRY32': undeclared identifier
src\tool_doswin.c(626,11): warning C4013: 'CreateToolhelp32Snapshot' undefined; assuming extern returning int
src\tool_doswin.c(626,36): error C2065: 'TH32CS_SNAPMODULE': undeclared identifier
src\tool_doswin.c(632,7): warning C4013: 'Module32First' undefined; assuming extern returning int
```
Ref: https://github.com/curl/curl/actions/runs/12055081933/job/33614629930#step:9:35
- examples: fix `websocket.c` to include `winsock2.h` before `windows.h`
to make it build with MSVC UWP:
```
include\curl\curl.h(143,16): error C2061: syntax error: identifier 'curl_socket_t'
include\curl\curl.h(143,16): error C2059: syntax error: ';'
include\curl\curl.h(417,52): error C2146: syntax error: missing ')' before identifier 'curlfd'
include\curl\curl.h(417,38): error C2081: 'curl_socket_t': name in formal parameter list illegal
```
Ref: https://github.com/curl/curl/actions/runs/12055317910/job/33615644427#step:14:126
- GHA/windows: silence linker warning with MSVC UWP builds:
```
LINK : warning LNK4075: ignoring '/INCREMENTAL' due to '/OPT:ICF' specification
```
Ref: https://github.com/curl/curl/actions/runs/12055696808/job/33616629610#step:11:38
- GHA/windows: set `/INCREMENTAL:NO` for all MSVC jobs to improve
performance a little.
- cmake: show `UWP` platform flag.
Ref: #15652Closes#15657
The ECH feature cannot be built without HTTPS RR.
ECH automatically implied HTTPS RR in `./configure` but not in CMake,
winbuild, documentation.
Also update documentation and CI configs.
Follow-up to a362962b72#11922Closes#15648
Merge cmake and autotools build steps for cygwin, msys2 and
cross-linux jobs.
Advantages:
- makes it easier to keep the two build tracks in sync.
- uses the same steps across jobs.
- avoids scrolling through greyed out steps.
- syncs steps with other workflows already merged like this.
- less code.
Also:
- stop ignoring WebSockets tests results for msys2/mingw-w64 cmake jobs,
except for 2301 2302 that were also ignored for autotools. Syncing the
two build methods.
- drop 'cmake' from step names where cmake was the only build tool.
This was redundant as "CM" already indicates it in the job name.
Closes#15643
- GHA/windows: switch mingw-w64 UWP CI job to use UCRT.
`msvcr120_app` was missing `getch()` for example.
Follow-up to f988842d85#15637
This job tests compiling for UWP correctly, but the the resulting
`curl.exe` still doesn't look like a correct UWP app, now exiting
on startup with: `curl: error initializing curl library`.
- tool_getpass: restore `getch()` for UWP builds.
Follow-up to f988842d85#15637
- schannel: silence `-Werror=null-dereference` warning in mingw-w64 UWP:
```
lib/vtls/schannel_verify.c: In function 'Curl_verify_host':
lib/vtls/schannel_verify.c:558:33: error: null pointer dereference [-Werror=null-dereference]
558 | for(i = 0; i < alt_name_info->cAltEntry; ++i) {
| ~~~~~~~~~~~~~^~~~~~~~~~~
lib/vtls/schannel_verify.c:559:50: error: null pointer dereference [-Werror=null-dereference]
559 | PCERT_ALT_NAME_ENTRY entry = &alt_name_info->rgAltEntry[i];
| ~~~~~~~~~~~~~^~~~~~~~~~~~
```
Ref: https://github.com/curl/curl/actions/runs/12022656065/job/33515255397?pr=15638#step:19:27
Follow-up to 9640a8ef6f#15421
- GHA/windows: fix `find` command in MSVC job step.
Follow-up to 5f9411f953#15380
- GHA/windows: drop unnecessary `windowsappcompat` lib from mingw-w64
UWP job. Also drop related MSYS2 package.
- GHA/windows: cmake 3.31.0 still invokes `windres` with wrong options
with mingw-w64 UPW. Update curl version in comment accordingly.
- GHA/windows: tidy up mingw-w64 UWP spec logic, limit it to gcc.
- GHA/windows: update comments on `curl.exe` UWP startup errors.
Closes#15638
Avoid the `curl.exe` wrapper binary created by libtool, and run the real
`curl.exe` directly for tests and version information.
This solution was used in Azure jobs. I missed it when migrating jobs
to GHA.
Applies to tests run in the `mingw, AM x86_64 c-ares U` job, which has
seen unexplained flakiness.
Ref: 354afc891d#6049
Follow-up to e53523fef0#14859Closes#15437
- Create a new macro SCH_DEV() to manage verbose debug messages that are
only useful for debugging Schannel recv decryption.
schannel_recv contains a lot of useful debug messages to help debug the
function, however in practice they are not otherwise useful and showing
them in debug builds adds a lot of noise.
To show these messages curl must now be built with
CURL_SCHANNEL_DEV_DEBUG defined.
Prior to this change many, but not all, extra-verbose messages were
wrapped in DEBUGF() so they were only shown in debug builds.
Ref: https://github.com/curl/curl/issues/14807Closes#14826
Fix the significant perf regression for vcpkg jobs by switching to the
MSYS2 shell environment from Git for Windows. This env is already used
for old-mingw-w64 job that remained unaffected by this issue.
The issue began with the windows-runner update 20241015.1.0. It bumped
Git for Windows from Git 2.46.2.windows.1 to Git 2.47.0.windows.1. GfW
bumped its MSYS2 components, including `msys-2.0.dll`. That's Cygwin
code, which may have contributed to this. Pipes were involved and
`runtests.pl` relies on pipes heavily in parallel mode. (The issue was
not seen with parallel tests disabled, in retrospect.)
This is useful as a permanent solution too. It drop GfW as a dependency
and makes Windows jobs use one less shell/env flavour.
Long term it might help to use native Windows Perl to avoid the MSYS
layer completely, if there is a way to make that work.
Assortment of possibly related links:
https://cygwin.com/pipermail/cygwin/2024-August/256398.htmlf78009cb1c7f3c225325https://github.com/actions/runner-images/issues/10843https://github.com/git-for-windows/git/issues/5199https://github.com/git-for-windows/msys2-runtime/pull/757913a41703555afcb2f31c5f4dcdc5
Follow-up to c33174d42f#15364
Follow-up to 1e0305973c#15356Closes#15380
- Linux: move test and pytest prereqs right before test run.
- returns build phase results faster.
- allows skipping steps for jobs that don't need them.
- makes dependencies more transparent.
- sync prereq install step names.
- use `tests/requirements.txt` more.
Closes#15275
- bump github/codeql-action from 3.26.10 to 3.26.11
- bump vmactions/omnios-vm from 1.0.7 to 1.0.8
- bump actions/cache from 4.0.2 to 4.1.1
Closes#15178Closes#15179Closes#15244
