Replace autotools with cmake to avoid libtool wrappers that are changing
`LD_LIBRARY_PATH` in a way incompatible with the thread sanitizer.
To fix the output when the sanitizier is finding something:
```
==51718==WARNING: Can't write to symbolizer at fd 7
/usr/bin/llvm-symbolizer-18: /home/runner/work/curl/curl/bld/lib/.libs/libcurl.so.4: no version information available (required by /usr/bin/llvm-symbolizer-18)
/usr/bin/llvm-symbolizer-18: symbol lookup error: /home/runner/openssl/lib/libcrypto.so.3: undefined symbol: __tsan_func_entry
```
Ref: https://github.com/curl/curl/actions/runs/16911402500/job/47913783729#step:39:4466
After:
```
13:50:04.117885 == Info:ThreadSanitizer: thread T1 finished with ignores enabled, created at:
closing connection #0#0 pthread_create <null> (libtests+0x6bc0f) (BuildId: 4fe889446291259934205ac03931c397aa0210d3)
#1 Curl_thread_create /home/runner/work/curl/curl/lib/curl_threads.c:73:6 (libcurl.so.4+0x55a76) (BuildId: cb0f14ba2ad68c9cab0c980d9a5d7a53cc0782da)
#2 async_thrdd_init /home/runner/work/curl/curl/lib/asyn-thrdd.c:500:26 (libcurl.so.4+0x1c153) (BuildId: cb0f14ba2ad68c9cab0c980d9a5d7a53cc0782da)
[...]
```
Ref: https://github.com/curl/curl/actions/runs/16939193922/job/48003405272?pr=18274#step:39:4018
Also:
- disable memory tracker which turned out to be incompatible with
the thread sanitizer and detaching threads.
Ref: #18263 and #curl IRC.
- the job is ~30 seconds faster after this patch.
Reported-by: Stefan Eissing
Bug: https://github.com/curl/curl/pull/18263#issuecomment-3179279440
Follow-up to a2bcec0ee0#14751Closes#18274
actions/checkout@v5 requires a newer libstdc++ than the container
contains. Update it to a backwards-compatible version just like we
already do for libc6.
Ref: #18250Closes#18255
In CI we want to ensure that examples build cleanly, but we don't want
to actually run them there. Meaning it's enough to just compile, but not
link them in CI. Saving time up to 2-4x (MSVC), and disk space up
to 1.2GB (or 8-70x).
Add a new cmake target that compiles all examples without linking them
into runnable binaries. Keep a full build for a single example to test
if it links correctly.
Also:
- CI: switch over all `curl-examples` targets to `curl-examples-build`
- GHA/linux-old: build examples in one of the cmake builds.
Result highlights:
Job | Bef. | Bef. | Aft. | Aft. |
:------------------ | ---: | ----: | ---: |----: |
cygwin | 15s | 9MB | 10s | 1MB |
msys | 13s | 8MB | 7s | 1MB |
dl-mingw 15 | 39s | 113M | 34s | 2MB |
dl-mingw 9.5.0 | 49s | 115MB | 42s | 2MB |
dl-mingw 7.3.0 | 19s | 113MB | 14s | 2MB |
dl-mingw 6.4.0 | 9s | 12MB | 7s | 4MB |
Linux cross | 19s | 28MB | 19s | 2MB |
MSVC UWP | 65s | 374MB | 9s | 17MB |
MSVC x64 | 22s | 846MB | 9s | 17MB |
VS2010 | 48s | 105MB | 15s | 9MB |
VS2022 clang-cl | 195s | 1.2GB | 51s | 20MB |
iOS Xcode | 8s | | 5s | |
macOS LibreSSL | 16s | | 11s | |
Linux aws-lc | 3s | | 1s | |
Follow-up to dda251ef10#18232Closes#18209
By micromanaging the project dependency and its inclusion into the test
project. It feels like an awkward construct, but perhaps better than
nothing.
It's also fragile because it's a static build with no assistance from
the external project (curl in this case). Mitigated in test by disabling
all dependencies and some features.
Since there is no special core cmake logic to be tested here, in CI
the test is tested really. To keep CI jobs at minimum, only add 3 of
them, taking 42s in total. (All 6 would take 270s.)
Follow-up to e2a23d5d0d#17203Closes#18208
Recently sometimes the Ubuntu package repository is very slow to access.
Remove the time limit for the install step, and bump the total limit,
aligning with Linux jobs.
In most cases the `install packages` steps takes 15-25 seconds.
Sometimes this goes up to 30-45 minutes.
Reported-by: nevakrien on github
Bug: https://github.com/curl/curl/discussions/14854#discussioncomment-13988574Closes#18163
Also:
- GHA/linux, macos: test `install` with autotools too.
- GHA/linux, macos: enable fish and zsh completion in an autotools job.
- scripts: drop dynamic perl checks redundant after this patch.
- scripts: drop two interim variables.
- autotools: make `build-certs` target check perl first.
- autotools: replace `$(PERL)` with `@PERL@` to match the pattern used
in most automake scripts. For consistency. This makes `PERL` be
defined at configure-time, as opposed to make-time, for these cases
now.
Closes#18141
- cmake: Before this patch a missing perl disabled the curl manual.
After this patch, it automatically picks up a pre-built hugehelp,
if present (= when building from a release tarball).
Follow-up to 0035ff45c5#16081
- autotools: Rework behavior when perl is missing. Before this patch
it caused a hard error when docs/manual/ca-embed were enabled.
Of these, docs were enabled by default. After this patch, doc
generation is automatically skipped, with a warning. Manual generation
falls back to using a pre-built hugehelp, or a stub if that's missing.
CA-embed is automatically skipped, with a warning.
Slight difference from cmake: When built with no perl and no pre-built
hugehelp, the manual is enabled, but the content is empty; with cmake
it's disabled proper.
Follow-up to 137aecfbf1#13514
Follow-up to 541321507e#12857
Also:
- GHA/distcheck: verify if the pre-built hugehelp makes it into curl.
- GHA/distcheck: streamline disabling perl with autotools.
- GHA/distcheck: `--disable-docs` no longer required to build without perl.
Passing it also implicitly disables the curl manual, which is
undesired in these tests.
- cmake: drop redundant `HAVE_MANUAL_TOOLS` interim variable.
- cmake: move two `set()` lines close to their counterparts.
Follow-up to 25119fbaaa#18104Closes#18118
Schannel is not supported by UWP. SSPI is also required by Schannel in
curl, and SSPI also isn't supported by UWP.
mingw-w64 is able to create such build regardless (my guess: due to API
parts not accurately marked as UWP-only), but the binary is unlikely
to work. With MSVC the failure happens at build-time.
Ref: https://learn.microsoft.com/windows/win32/api/sspi/nf-sspi-initsecurityinterfacea#requirements
Ref: https://learn.microsoft.com/windows/win32/secauthn/initializesecuritycontext--schannel#requirements
Drop all UWP-related logic, including two related feature checks, that
can now be permanently enabled.
Also:
- build: show fatal error for Schannel in UWP mode.
- build: do not allow enabling SSPI in UWP mode.
- drop undocumented option `DISABLE_SCHANNEL_CLIENT_CERT`. Added without
mention in an unrelated commit. The PR text says to save size. On x64
this is 0.3%, or 4KB out of 1.3MB. The tiny gain doesn't justify
an extra build variant. Ref: 8beff43559
- move `MPROTO_SCHANNEL_CERT_SHARE_KEY` closer to its use.
- replace commented block with `#if 0`.
Reviewed-by: Jay Satiro
Follow-up to cd0ec4784c#17089Closes#18116
- add AM out-of-tree no perl job.
- add AM in-tree no perl job.
- make CM out-of-tree job use no perl.
- add CM in-tree no perl job.
- run `curl -V` after builds.
- show the number of `--manual` lines.
- set `--enable-werror` in autotools jobs.
Ref: https://github.com/curl/curl/issues/18088#issuecomment-3135112176Closes#18104
libtests and clients were built the same way after recent overhauls.
libtests are used by runtests, clients by pytests.
Merge clients into libtests, aligning their entry function signature,
dropping common utility functions, and simplifying the build.
Note: After this patch `CURLDEBUG` applies to cli tests, when enabled.
Also:
- lib552: drop local copy-paste debug callback in favor of testtrace.
- lib552: drop local copy-paste dump function in favor of testtrace.
- clients: use `long` for HTTP version, drop casts.
- clients: replace local dump function in favor of testrace clone.
- sync cli test entry function prototype with libtests'.
- h2_serverpush: replace local trace callback with testtrace.
- de-duplicate 3 websocket close, ping, ping, functions. Kept the pong
iteration from `ws_pingpong`. Note: the pong clone in `lib2304` was
returning an error when `curl_ws_recv()` returned non-zero and
the payload matched the expected one anyway. After this patch, this
case returns success, as it does in `ws_pingpong`.
`lib2304` keeps passing, but I'm not sure if the previous behavior
was intentional.
- display full value in websocket close, ping, pong, drop casts.
Closes#18079
- add startup check for 'danted' to avoid fails on low cpu
- rename 'sockd' to 'danted' everywhere to clarify what we use
- add proper defaults for 'danted' for debian
- install 'dante-server' in pytest ci runs
Closes#18075
Fixing:
- HTTPS-RR builds with c-ares and Linux MUSL.
- curl-for-win minimal builds with Linux MUSL.
It should fix all other kinds of entaglement between curl's redefintions
of system symbols and system (or 3rd-party) headers sensitive to that.
It also syncs memory override behavior between unity & non-unity builds,
thus reducing build variations.
The idea is to define and declare everything once in `curl_setup.h`,
without overriding any system symbols with curl ones yet. Then, like
before this patch, override them, if necessary, in each source file via
`curl_memory.h` and `memdebug.h`, after including system headers.
To ensure a clean slate with no overrides at the beginning of each
source file, reset all of them unconditionally at the end of
`curl_setup.h`, by including `curl_mem_undef.h`. (This assumes
`curl_setup.h` is always included first, which is already the case
throughout the codebase.)
`curl_mem_undef.h` can also be included explicitly wherever overrides
are causing problems. E.g. in tests which use unity-style builds and
a previously included `curl_memory.h`/`memdebug.h` can be spilling into
other source files.
The simplified role of the two override headers:
- `curl_memory.h`: overrides system memory allocator functions to
libcurl ones, when memory tracing (aka `CURLDEBUG`) is disabled.
- `memdebug.h`: overrides system memory allocator and some other
functions to curl debug functions, when memory tracing is enabled.
Changed made in this patch, step-by-step:
- curl_memory.h: move allocator typedefs and protos to `curl_setup.h`.
- memdebug.h: move `ALLOC_*` macros to `curl_setup.h`.
- memdebug.h: move allocator protos to `curl_setup.h`.
- memdebug.h: move `Curl_safefree()` macro to `curl_setup.h`.
(it's a regular macro, with a one-time, global, definition.)
- curl_memory.h: move system symbol undefs to a new, separate header:
`curl_mem_undef.h`.
- curl_setup.h: include `curl_mem_undef.h` at the end, unconditionally,
to reset system symbol macros after each inclusion.
- handle `sclose()` and `fake_sclose()` in `curl_setup.h`. They are not
system symbols, a one-time definition does the job.
Also:
- GHA/linux: enable unity mode for the HTTP-RR c-ares MUSL job.
Follow-up to 17ab4d62e6#16413
That said, I'd still find it better to avoid redefining system macros.
To communicate clearly the fact that they are not the original system
calls and they do behave differently. And, it would allow dropping the
undef/redef dance in each source file, and maintaining the logic with
it. The "last #include files should be in this order" comments in each
source would also become unnecessary. Also the trick of using
`(func)` (or interim macros) to call the non-overridden function where
required. This method works for printf and most everything else already.
For `_tcsdup`, socket and fopen functions this could work without
disturbing the codebase much.
Ref: #16428 (clean reboot of)
Closes#17827
- GHA/windows: disable building certs in the MSVC job that's not running
tests. Saves 4-5 seconds for MSVC, makes logs shorter for the rests.
- GHA/linux: build tests in two more jobs (LTO, CM Rustls), 5s each.
- GHA/linux: skip 'install test prereqs' for `skiprun` jobs.
(there were no such jobs before this patch.)
Closes#18034
ngtcp2 1.14.0 added crypto dependencies to the ngtcp2 crypto `.pc`
files. It broke GHA builds, because how curl's `configure` is setting up
the per-dependency custom prefixes for pkg-config.
`configure` uses `PKG_CONFIG_LIBDIR` to set per-dependency custom
prefixes, as specified via `--with-ngtcp2=<custom-dir>`. In classic
`pkg-config` this overrides any previously configured `PKG_CONFIG_DIR`.
This in turn break detecting transitive pkg-config modules unless they
are found at locations `pkg-config` is searching by default. This
doesn't affect `pkgconf` because it appends `PKG_CONFIG_LIBDIR` to
the custom `PKG_CONFIG_DIR`, according to its man page.
It may make sense to fix this in`acinclude.m4`, to make sure to honor
global custom pkg-config paths while detecting components at custom
locations, regardless of pkg-config implementation. But this PR doesn't
do this.
Instead it drops the ngtcp2 custom path and lets detection rely on
`PKG_CONFIG_DIR` that's already set up for all custom-built dependencies
anyway.
Also:
- fix `openssl-quic` job to use the custom-built nghttp2 (like other
jobs do) instead of the system default.
- configure nghttp3 via `PKG_CONFIG_DIR` in the `openssl-quic` job,
to sync with other jobs. And drop `--with-nghttp3` option.
cb9b1a4c4e/acinclude.m4 (L1376-L1381)https://manpages.debian.org/unstable/pkg-config/pkg-config.1.en.htmlhttps://man.archlinux.org/man/pkgconf.1.en
Ref: https://github.com/ngtcp2/ngtcp2/pull/1689#issuecomment-3121576712Closes#18022Closes#18028
- runtests: fix `codeset-utf8` feature detection. Before this patch it
detected if the calling environment had UTF-8 enabled. If not, UTF-8
tests were all skipped. After this patch, it detects if UTF-8 is
supported by the calling environment regardless of what's currently
enabled.
Follow-up to 0b70b23ef4#15039
- GHA/linux: sync `codeset-test` to also reset `LC_CTYPE` and
`LC_NUMBER`. To give it more spin.
Follow-up to c221c0ee59#17938
- GHA/macos: fix to actually enable `codeset-test`. Also set `LC_ALL`,
which seems necessary to trigger issues.
Follow-up to c221c0ee59#17938
- tests/data: replace `LC_CTYPE` env with `LC_ALL` in all tests
requiring a locale. Also to avoid potential issues with a blank or
unset `LC_ALL`, as seen earlier. And to ensure that the override works
on all platforms (as tested in CI.)
Slight downside is that this now resets the language/culture to `C`.
Ref: b4c9982382#4743
Ref: 23208e330a#4738
- replace `en_US.UTF-8` with `C.UTF-8` to be language/culture-agnostic.
- TEST-SUITE.md: drop `UTF-8` as a requirement for tests.
Tests shall work (or least be skipped) without UTF-8 support.
Tests requiring UTF-8 locale:
165, 962, 963, 964, 965, 966, 967, 1448, 1560, 2046, 2047
Tests requiring UTF-8 locale, but passing without one anyway:
955, 956, 957, 958, 959, 960, 961, 968, 1034, 1035
Spec 1997: https://pubs.opengroup.org/onlinepubs/7908799/xbd/envvar.html
Spec 2008: https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html
Ref: c221c0ee59#17938
Ref: 7cf8414fab
Ref: 4c140a5628
Ref: 28faaacee2#2436
Ref: ecd1d020abCloses#17988
To fix running test 1560 when `LC_ALL` is set to something unexpected
(e.g. `C`). Also syncing it with the rest of tests.
Also:
- GHA/linux: enable `libidn2` in more jobs.
Also to enable test 1560 reproducing this issue in more jobs.
- GHA/linux: run tests with `LC_ALL=C` in one of the jobs.
- GHA/linux: switch to the non-deprecated package name for libidn2.
- GHA/macos: run tests with non-default locale settings in one job.
- GHA/macos: enable AppleIDN in that job.
Ref: https://github.com/curl/curl/pull/17933#issuecomment-3074582840
Follow-up to f27262b179#10196Closes#17938
- codespell: break logic out into its own runnable script. Allowing
to run it on local machines.
- codespell: install via `pip`, bump to latest version.
- codespell: show version number in CI log.
- codespell: drop no longer needed word exception: `msdos`.
- codespell: include all curl source tree, except `packages` and
`winbuild`. Drop an obsolete file exclusion.
- add new spellchecker job using the `typos` tool. It includes
the codespell dictionary and a couple more. Use linuxbrew to install
it. This takes 10 seconds, while installing via `cargo` from source
would take over a minute.
- codespell: introduce an inline ignore filter compatible with `cspell`
Make `typos` recognize it, too. Move single exceptions inline.
Fix new typos found. Also rename variables and words to keep
spellchecking exceptions at minumum. This involves touching some tests.
Also switch base64 strings to `%b64[]` to avoid false positives.
Ref: https://github.com/crate-ci/typos/blob/master/docs/reference.md
Ref: https://github.com/codespell-project/codespell?tab=readme-ov-file#inline-ignore
Ref: https://github.com/codespell-project/codespell/issues/1212#issuecomment-1721152455
Ref: https://cspell.org/docs/Configuration/document-settingsCloses#17905
- autotools: fix auto-detection on the Windows platform.
It was mis-detected when targeting Windows XP/2003 64-bit.
It was permanently undetected when building for Windows 32-bit.
```
lib/url.c: In function 'zonefrom_url':
lib/url.c:1802:18: error: implicit declaration of function 'if_nametoindex' [-Wimplicit-function-declaration]
1802 | scopeidx = if_nametoindex(zoneid);
| ^~~~~~~~~~~~~~
lib/url.c:1802:18: error: nested extern declaration of 'if_nametoindex' [-Werror=nested-externs]
```
Ref: https://github.com/curl/curl/actions/runs/16405598782/job/46351023138?pr=17982#step:10:29
Reported-by: LoRd_MuldeR
Fixes#17979
Without this patch the workaround for the 8.15.0 release is:
`export ac_cv_func_if_nametoindex=0` for Windows XP/2003 64-bit.
Background: Checking for the `if_nametoindex()` function via
`AC_CHECK_FUNCS()` (autotools) or `check_function_exists()` (cmake) do
not work on Windows, for two reasons:
- the function may be disabled at compile-time in Windows headers
when targeting old Windows versions (XP or WS2003 in curl context)
via `_WIN32_WINNT`. But it's always present in the system implib
`iphlpapi` where these checks are looking.
- for 32-bit Windows the function signature in the implib requires
a 4-byte argument, while these checks always use no arguments,
making them always fail.
- cmake: call `if_nametoindex` dynamically with mingw-w64 v1.0.
This mingw-w64 version lacks prototype and implib entry for it.
- cmake: add auto-detection for Windows and use as a fallback for
non-pre-fill cases.
- cmake: disable pre-fill with `_CURL_PREFILL=OFF`. (for testing)
- cmake: disable pre-fill for untested compilers. (i.e. non-MSVC,
non-mingw64)
- GHA/windows: make an autotools job build for Windows XP.
Follow-up to 0d71b18153#17413Closes#17982
To fix test 1308 in MultiSSL builds.
Failure was caused by the random number generator virtual function being
NULL, instead of pointing to the implementation in the runtime-selected
TLS backend. This could happen in MultiSSL builds when a functionality
was asking for a random number without triggering a VTLS function table
initialization first. Such functionality is MIME, or form data via MIME.
The reason CI did not catch it in an earlier MultiSSL GHA/windows job,
is that it was a debug-enabled one. In debug-enabled builds the test
runner was overriding the random number generator for all tests.
Fixed this by moving the override to the tests requiring it, via
1fcf22585f#17971, enabling debug builds
to catch this issue.
Enable MultiSSL in two CI jobs, to verify this patch.
Fixing:
```
test 1308...[formpost tests]
libtests returned 44, when expecting 0
1308: exit FAILED
[...]
=== Start of file stderr1308
URL: log/3/test-1308
tests/libtest/lib1308.c:70 Assertion 'res == 0' FAILED: curl_formget returned error
tests/libtest/lib1308.c:72 Assertion 'total_size == 518' FAILED: curl_formget got wrong size back
tests/libtest/lib1308.c:88 Assertion 'res == 0' FAILED: curl_formget returned error
tests/libtest/lib1308.c:89 Assertion 'total_size == 899' FAILED: curl_formget got wrong size back
```
Ref: https://github.com/curl/curl/actions/runs/16387693424/job/46309536359?pr=17963#step:16:2515
Bug: https://github.com/curl/curl/pull/17963#issuecomment-3092282057Closes#17970
Bind them to the (arbitrary choice of) Apple clang and gcc compilers,
respectively.
Also:
- bind existing mbedTLS job to the llvm compiler, to keep the number of
jobs the same as before this patch.
- move OpenLDAP from mbedTLS over to LibreSSL to keep testing it with
all 3 compilers.
- simplify exclusions for clang-tidy and torture jobs.
- tag clang-tidy and torture jobs via `install_steps`.
To avoid keeping around special bool fields.
Closes#17963
