Before this patch curl code was redefining `getaddrinfo` and
`freeaddrinfo` system symbols to plug in its debug wrappers. This was
causing pains to avoid applying the redefinitions to system headers
defining these functions, and to the local debug wrappers. Especially
in unity builds. It also required workarounds for systems where these
symbols are already macros.
Introduce curl-namespaced macros for these functions and use them.
This allows to drop all workarounds and makes it work in all envs,
local targets and unity/bundle combinations.
Also drop GHA/windows workaround and use the same unity batch across
all jobs. Follow-up to 29e4eda631#16272
Ref: #16272
Ref: 71cf0d1fca#14772
Ref: 3efba94f77#14765
Ref: f7d5f47059#14399Closes#16274
Default curl unity builds make a single unit for each target. It means
all target sources are batched together and built in a single compiler
invocation. With multi-core CPUs this doesn't always result in the best
possible performance. This patch enables smaller batches for jobs where
this resulted in shorter build times. These jobs are Cygwin, MSYS2,
MinGW, running on the Windows runners.
Use batch of 30 (meaning 30 sources batched into units), and 32 for
Cygwin/MSYS2 to avoid a unity fallout that's subject to a different PR.
(CMake allows to set the number of sources per unit, not the number
of units, though the latter may be more practical to max out CPU cores.)
Also override to not batch the `curlu` target because batching lost
a little bit of time there, due to the already existing parallelism when
building the `testdeps` targets.
For jobs on the macOS and Linux runners jobs were already mostly single
digit or below teen seconds, and batching didn't improve on them
noticeably. On VM jobs, the virtual CPUs are limited, so I didn't
make a try. In AppVeyor and GHA vcpkg jobs (using msbuild), batching
didn't result in conclusive or any gains.
Build times in seconds (curl + testdeps):
Job | Before | After w curlu=0 | Gain
:--------------------| :-------------- | :-------------- | :---
cygwin, CM | 19 + 32 = 51 | 12 + 32 = 44 | 7
msys2, CM | 7 + 15 = 22 | 5 + 14 = 19 | 3
mingw gcc U, CM | 19 + 30 = 49 | 13 + 32 = 45 | 4
mingw ucrt, CM | 32 + 42 = 74 | 15 + 43 = 58 | 16
mingw clang, CM | 15 + 21 = 36 | 8 + 21 = 29 | 7
mingw uwp, CM | 30 + 40 = 70 | 14 + 40 = 54 | 16
mingw gcc, CM | 20 + 31 = 51 | 12 + 31 = 43 | 8
mingw x86, CM | 35 + 40 = 75 | 15 + 38 = 53 | 22
dl-mingw, CM 9.5.0 | 88 + 99 = 187 | 42 + 101 = 143 | 44
dl-mingw, CM 7.3.0 U | 24 + 32 = 56 | 17 + 35 = 52 | 4
Total | | | 131
Total gain per GHA/windows workflow runs: 2m11s
Runs:
Before: https://github.com/curl/curl/actions/runs/13220256084/job/36904342259
After: https://github.com/curl/curl/actions/runs/13220383702/job/36904602981https://github.com/curl/curl/actions/runs/13220613141/job/36905170104https://github.com/curl/curl/actions/runs/13222019443/job/36908358550
With curlu tweak: https://github.com/curl/curl/actions/runs/13222239255/job/36908782462
Ref: 116950a250#16265Closes#16272
To match the rest of Windows jobs.
dl-mingw, CM 9.5.0-x86_64 schannel: 4m24s -> 2m56s
dl-mingw, CM 7.3.0-x86_64 schannel U: 4m37s -> 3m10s
(based on a few runs.)
Closes#16271
Use the last known good release of Git for Windows by installing it
manually. It restores `runtests.pl` performance to the levels before
the October 2024 and this week's fallouts which gradually deployed
MSYS2 runtimes with pipe/signal/concurrency issues.
Also:
- restore vcpkg job's test parallelism to `-j8` (from `-j4`).
- keep using the default shell for jobs not running tests.
To avoid the unnecessary Git for Windows install overhead.
Upsides:
- good performance again.
- easy to experiment with any version.
Downsides:
- installing the Git for Windows package takes 15-30 seconds.
- we're pinned to an old package version.
- no canary to tell when the issue is fixed on the runner images.
Unknown:
- stability. (no MSYS2 runtimes were ever stable and it's difficult
to quantify if a version improves or worsens stability/flakiness, and
intermittent env failures.
Follow-up to 1bf774df57#16217
Follow-up to 5f9411f953#15380Closes#16265
- drop `--quiet 2` option where used, to have uniform output.
- replace `apt` with `apt-get` in one job. sync options with rest.
- replace deprecated `apt-key` command with the alternative recommended
by `apt-key(8)`.
- drop stray `cd /tmp`, no longer needed after migrating to GHA.
- shorten `--option Dpkg::Use-Pty=0` to `-o Dpkg::Use-Pty=0`.
- add `-o Dpkg::Use-Pty=0` to hide `apt-get` progress bars taking
vertical log space, where missing.
- drop `-y --no-install-suggests --no-install-recommends` `apt-get`
options. They are the default in the ubuntu-24.04 image.
- GHA/distcheck: move `name:` to top in steps where not there.
- scripts/cijobs.pl: catch `apt-get` lines with the `-o` option.
Closes#16127
RFC 6455 Section 5.2 notes that for bits RSV1, RSV2, and RSV3 of the
framing header, a non-zero value that is not defined by a negotiated
extension MUST Fail the WebSocket connection.
Test 2310 verifies
Closes#16069
They play better with Unixy shells. The compiler has been supporting
dash options since its early versions.
Also fix to detect warnings options passed in dash-style.
Closes#16063
- move checkout step right before use.
- quotes in yaml and bash.
- unfold single-line run commands.
- set `CMAKE_WARN_DEPRECATED=OFF` for CMake with Android NDK to avoid
a wall of useless deprecation warnings hiding useful output.
These warnings happen in 3rd-party scripts, and unfixable in curl.
Closes#16042
To match cmake builds.
- GHA/windows: allow autotools UWP builds.
- detect UWP and add to `buildinfo.txt`.
Consider it enabled if `CPPFLAGS` contains `-DWINSTORECOMPAT`.
- disable telnet with UWP.
- enable Unicode with UWP.
- do not use `wldap32` with UWP.
- do not enable `USE_WIN32_CRYPTO` with UWP.
- make sure to link to `ws2_32` in UWP builds.
To fix `undefined reference to `in6addr_any'` when linking
`tests/server` programs. More in the comment.
Closes#16020
CMake (as of 3.31.2) doesn't fully recognize mingw-w64 with
`CMAKE_SYSTEM_NAME=WindowsStore`.
The manual logic works around it.
Also move existing DJGPP workaround to the same block.
Closes#16019
- fix `find` commands to not miss items.
- call `file` on the built files in `curl -V` steps.
To give more feedback on what was built.
- add `curl info` step for cross-jobs that can't do a `curl -V`.
It lists the files built and calls `file` on them.
- appveyor: make a VS2010 32-bit to match the VS2008 job it replaced.
Follow-up to d34aeecb08#15934
- GHA/windows: drop the word "old" from standalone mingw-w64 jobs to not
conflate it with "old mingw" we no longer support (while also keeping
it short).
Cherry-picked from #15975Closes#16001
- make `curl_dependency_option()` more generic.
- extend `CURL_BROTLI` and `CURL_ZSTD` options to accept
`AUTO` in addition to existing `ON` and `OFF`.
- change `CURL_BROTLI` and `CURL_ZSTD` option default
to `AUTO`. Was: `OFF`.
It brings cmake behavior closer to `./configure`.
Still different:
- `./configure` defaults to `off` which means to check default
locations. cmake checks more locations by default.
(Also tried `NO_CMAKE_PATH`, but then it checked less locations.)
- cmake returns both `brotlicommon` and `brotlidec` libs,
while `./configure` only returns the latter.
- ci: drop explicit cmake options, that are now unnecessary.
- GHA/configure-vs-cmake: make adjustments to make tests pass.
Closes#15431
This makes `runtests.pl` run the final executables directly.
Before this patch it called the autotools/libtool wrapper tool, which
then called the final executables.
This solution was already used for `curl.exe`.
Applies to tests run in the `mingw, AM x86_64 c-ares U` job, which still
shows unexplained flakiness.
Also makes tests finish 45 seconds faster.
Ref: #14854
Follow-up to 1a2d38c47c#15437Closes#15662
The MSVC UWP job in CI did not actually enable UWP. Fix this and
the fallouts discovered after enabling it.
- GHA/windows: make sure to enable UWP in MSVC vcpkg UWP job.
Use the CMake options and C flags already used for mingw-w64, but use
`WINAPI_FAMILY_PC_APP` instead of the deprecated `WINAPI_FAMILY_APP`.
(The former is not supported by mingw-w64, so leave it there as-is.)
Follow-up to cb22cfca69#14077
- GHA/windows: by default the MSVC UWP job became 2x-3x slower than
others after actually enabling UWP. Most of it is caused by
CMake/MSBuild automatically building full APPX containers for each
`.exe` target. This includes 21 CMake feature detections. Each
detection app is built into a 15MB APPX project, with code signing,
logos, etc. Example:
https://github.com/curl/curl/actions/runs/12056968170/job/33620610958
Disable this overhead for curl build targets via custom
`CMAKE_VS_GLOBALS` options. I've found no way to apply them to feature
detection targets, so those remain slow.
- cmake: automatically enable Unicode for UWP builds. It's required.
Also stop enabling it manually in the existing CI job.
- tests: fix `getpid()` use for Windows UWP:
```
tests\server\util.c(281,21): warning C4013: 'getpid' undefined; assuming extern returning int
```
Ref: https://github.com/curl/curl/actions/runs/12061215311/job/33632904249#step:11:38
- src/tool_doswin: disable `GetLoadedModulePaths()` for UWP.
mingw-w64 UWP was okay with this, but MS SDK headers are not.
This makes `--dump-module-paths` return empty for UWP builds.
```
src\tool_doswin.c(620,3): error C2065: 'MODULEENTRY32': undeclared identifier
src\tool_doswin.c(626,11): warning C4013: 'CreateToolhelp32Snapshot' undefined; assuming extern returning int
src\tool_doswin.c(626,36): error C2065: 'TH32CS_SNAPMODULE': undeclared identifier
src\tool_doswin.c(632,7): warning C4013: 'Module32First' undefined; assuming extern returning int
```
Ref: https://github.com/curl/curl/actions/runs/12055081933/job/33614629930#step:9:35
- examples: fix `websocket.c` to include `winsock2.h` before `windows.h`
to make it build with MSVC UWP:
```
include\curl\curl.h(143,16): error C2061: syntax error: identifier 'curl_socket_t'
include\curl\curl.h(143,16): error C2059: syntax error: ';'
include\curl\curl.h(417,52): error C2146: syntax error: missing ')' before identifier 'curlfd'
include\curl\curl.h(417,38): error C2081: 'curl_socket_t': name in formal parameter list illegal
```
Ref: https://github.com/curl/curl/actions/runs/12055317910/job/33615644427#step:14:126
- GHA/windows: silence linker warning with MSVC UWP builds:
```
LINK : warning LNK4075: ignoring '/INCREMENTAL' due to '/OPT:ICF' specification
```
Ref: https://github.com/curl/curl/actions/runs/12055696808/job/33616629610#step:11:38
- GHA/windows: set `/INCREMENTAL:NO` for all MSVC jobs to improve
performance a little.
- cmake: show `UWP` platform flag.
Ref: #15652Closes#15657
The ECH feature cannot be built without HTTPS RR.
ECH automatically implied HTTPS RR in `./configure` but not in CMake,
winbuild, documentation.
Also update documentation and CI configs.
Follow-up to a362962b72#11922Closes#15648
Merge cmake and autotools build steps for cygwin, msys2 and
cross-linux jobs.
Advantages:
- makes it easier to keep the two build tracks in sync.
- uses the same steps across jobs.
- avoids scrolling through greyed out steps.
- syncs steps with other workflows already merged like this.
- less code.
Also:
- stop ignoring WebSockets tests results for msys2/mingw-w64 cmake jobs,
except for 2301 2302 that were also ignored for autotools. Syncing the
two build methods.
- drop 'cmake' from step names where cmake was the only build tool.
This was redundant as "CM" already indicates it in the job name.
Closes#15643
- GHA/windows: switch mingw-w64 UWP CI job to use UCRT.
`msvcr120_app` was missing `getch()` for example.
Follow-up to f988842d85#15637
This job tests compiling for UWP correctly, but the the resulting
`curl.exe` still doesn't look like a correct UWP app, now exiting
on startup with: `curl: error initializing curl library`.
- tool_getpass: restore `getch()` for UWP builds.
Follow-up to f988842d85#15637
- schannel: silence `-Werror=null-dereference` warning in mingw-w64 UWP:
```
lib/vtls/schannel_verify.c: In function 'Curl_verify_host':
lib/vtls/schannel_verify.c:558:33: error: null pointer dereference [-Werror=null-dereference]
558 | for(i = 0; i < alt_name_info->cAltEntry; ++i) {
| ~~~~~~~~~~~~~^~~~~~~~~~~
lib/vtls/schannel_verify.c:559:50: error: null pointer dereference [-Werror=null-dereference]
559 | PCERT_ALT_NAME_ENTRY entry = &alt_name_info->rgAltEntry[i];
| ~~~~~~~~~~~~~^~~~~~~~~~~~
```
Ref: https://github.com/curl/curl/actions/runs/12022656065/job/33515255397?pr=15638#step:19:27
Follow-up to 9640a8ef6f#15421
- GHA/windows: fix `find` command in MSVC job step.
Follow-up to 5f9411f953#15380
- GHA/windows: drop unnecessary `windowsappcompat` lib from mingw-w64
UWP job. Also drop related MSYS2 package.
- GHA/windows: cmake 3.31.0 still invokes `windres` with wrong options
with mingw-w64 UPW. Update curl version in comment accordingly.
- GHA/windows: tidy up mingw-w64 UWP spec logic, limit it to gcc.
- GHA/windows: update comments on `curl.exe` UWP startup errors.
Closes#15638
Avoid the `curl.exe` wrapper binary created by libtool, and run the real
`curl.exe` directly for tests and version information.
This solution was used in Azure jobs. I missed it when migrating jobs
to GHA.
Applies to tests run in the `mingw, AM x86_64 c-ares U` job, which has
seen unexplained flakiness.
Ref: 354afc891d#6049
Follow-up to e53523fef0#14859Closes#15437
- Create a new macro SCH_DEV() to manage verbose debug messages that are
only useful for debugging Schannel recv decryption.
schannel_recv contains a lot of useful debug messages to help debug the
function, however in practice they are not otherwise useful and showing
them in debug builds adds a lot of noise.
To show these messages curl must now be built with
CURL_SCHANNEL_DEV_DEBUG defined.
Prior to this change many, but not all, extra-verbose messages were
wrapped in DEBUGF() so they were only shown in debug builds.
Ref: https://github.com/curl/curl/issues/14807Closes#14826
Fix the significant perf regression for vcpkg jobs by switching to the
MSYS2 shell environment from Git for Windows. This env is already used
for old-mingw-w64 job that remained unaffected by this issue.
The issue began with the windows-runner update 20241015.1.0. It bumped
Git for Windows from Git 2.46.2.windows.1 to Git 2.47.0.windows.1. GfW
bumped its MSYS2 components, including `msys-2.0.dll`. That's Cygwin
code, which may have contributed to this. Pipes were involved and
`runtests.pl` relies on pipes heavily in parallel mode. (The issue was
not seen with parallel tests disabled, in retrospect.)
This is useful as a permanent solution too. It drop GfW as a dependency
and makes Windows jobs use one less shell/env flavour.
Long term it might help to use native Windows Perl to avoid the MSYS
layer completely, if there is a way to make that work.
Assortment of possibly related links:
https://cygwin.com/pipermail/cygwin/2024-August/256398.htmlf78009cb1c7f3c225325https://github.com/actions/runner-images/issues/10843https://github.com/git-for-windows/git/issues/5199https://github.com/git-for-windows/msys2-runtime/pull/757913a41703555afcb2f31c5f4dcdc5
Follow-up to c33174d42f#15364
Follow-up to 1e0305973c#15356Closes#15380
- Linux: move test and pytest prereqs right before test run.
- returns build phase results faster.
- allows skipping steps for jobs that don't need them.
- makes dependencies more transparent.
- sync prereq install step names.
- use `tests/requirements.txt` more.
Closes#15275
- bump github/codeql-action from 3.26.10 to 3.26.11
- bump vmactions/omnios-vm from 1.0.7 to 1.0.8
- bump actions/cache from 4.0.2 to 4.1.1
Closes#15178Closes#15179Closes#15244
Cygwin jobs took 30 minutes and was the reason for moving them to their
own workflow. After recent changes, they finish in 8 minutes, which is
on par or lower than the longest Windows jobs. Thus, Cygwin can now be
moved back to the Windows workflow without causing extra wait for all
the workflow jobs to complete (and allowing a manual retry for the
remaining flaky jobs).
Notice that Cygwin still uses the cygwin/cygwin-install-action action to
install Cygwin itself. This action uses no caching and the time it takes
fluctuates widely depending on mirror speed, congestion and other
things. If Cygwin tends to hold up jobs again for this or other reasons,
this patch may be reverted.
Follow-up to 80a8e24956#15083
Follow-up to 2d8464c4cb#14366Closes#15086
