macOS was chosen because xmllint comes preinstalled, saving the prereq
install step. But, macOS's xmllint jobs sometimes doesn't finish in 1m
(instead of under 1 second) and gets cancelled, causing flaky failures.
Go with Linux and an install phase (of 15s) instead.
Examples:
https://github.com/curl/curl/actions/runs/19558021722/job/56004334495Closes#19634
There are 58 non-compliant files. Mark them with the `notxml` keyword.
Also include the compliant ones in the GHA/checksrc xmllint CI job.
Also:
- delete XML prolog from the 3 test data files that had them.
- FILEFORMAT.md: document the `notxml` keyword.
- FILEFORMAT.md: fix a typo.
Follow-up to de49cc89ab#19470
Follow-up to f3095f0dbd#19528
Follow-up to 87ba80a6dfCloses#19595
- badwords.pl: add `-a` option to check all lines in source code files.
Before this patch indented lines were skipped (to avoid Markdown code
fences.)
- GHA/checksrc: use `-a` when verifying the source code.
- GHA/checksrc: disable `So` and `But` rules for source code.
- GHA/checksrc: add docs/examples to the verified sources.
- badwords.txt: delete 4 duplicates.
- badwords.txt: group and sort contractions.
- badwords.txt: allow ` url = `, `DIR`, `<file name`.
Closes#19536
Fix SC2046: "Quote this to prevent word splitting."
Ref: https://www.shellcheck.net/wiki/SC2046
Also:
- shellcheck.sh: add `set -eu`.
- shellcheck.sh, yamlcheck.sh: always run from repo root.
- pass `--` before passing the list of files, where missing.
- badwords.pl, cleancmd.pl: rework to accept `git ls-files` arguments.
Requires Perl 5.22+ (2015-Jun-01) on Windows.
Ref: https://perldoc.perl.org/functions/open
- INTERNALS.md: require Perl 5.22 on Windows.
- spacecheck.pl: formatting.
- GHA/http3-linux: rework command to avoid SC2046.
- stop suppressing SC2046 warnings.
The yamlcheck.sh issue reported-by: Stanislav Fort (Aisle Research)
Ref: 20251109163515_6eb31da3-deb2-4f4d-8327-935904f27da5
Closes#19432
It also does shellcheck on `run:` elements, overlapping with
the homegrown `shellcheck-ci.sh` with the same purpose. But it also does
more and perhaps could replace the script too, especially in curl
sub-repos.
Also:
- GHA/macos: delete potentially useful, but commented, and ultimately
unused, non-default Xcode-testing logic. It's causing unused matrix
exceptions, upsetting actionlint.
Ref: https://github.com/rhysd/actionlintCloses#19373
To silence zizmor 1.16.0 warnings.
Also:
- http3-linux: replace hard-coded workflow name with variable.
Follow-up to a8174176b5#13841
- codeql: set `cancel-in-progress: true`.
zizmor apparently does not allow `false` in pedantic mode anymore:
https://github.com/zizmorcore/zizmor/pull/1227
- codeql: sync concurrency setting with the rest of the jobs.
(I'm not sure this is correct, or why it was previously special-cased.)
Expressions used (before and after this patch):
- `group: ${{ github.workflow }}-${{ github.event.sha }}-${{ github.event.target_url }}`
for GHA/appveyor-status.
- `group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.sha }}`
for all the rest.
Ref: https://github.com/curl/curl/actions/runs/18776245057/job/53571438139?pr=19209Closes#19215
For a complete, online, check.
After this patch the check takes 30s, up from a fraction of a second.
Also bump CodeQL actions to their latest version.
Closes#18827
Add script to run all Perl sources through `perl -c` to ensure no
issues, and run this script via GHA/checksrc in CI.
Fallouts:
- fix two repeated declarations.
- move `shell_quote()` from `testutil.pm` to `pathhelp.pm`, to
avoid circular dependency in `globalconfig.pm`.
Closes#18745
In an attempt to make `brew install` commands initialize faster.
Often this command started with 20-50 seconds of delay before this
patch. This is an attempt to make it launch faster.
Cherry-picked from #18736
- requirements.txt: shorten copyright headers.
- requirements.txt: pin packages to versions.
- GHA/windows: use `tests/requirements.txt`.
Pick a `cryptography` package version that satifies both `impacket`
and pytests dependencies.
- GHA/checksrc: move pip deps into a new `requirements.txt`.
To make Dependabot detect and bump them.
- GHA/checksrc: replace apt packages for python test deps with pip
install `tests/**/requirements.txt` to a venv.
- GHA/checksrc: use venv and drop `--break-system-packages`.
- GHA/linux: fix to actually activate venvs.
Follow-up to 2638570241#15578
- GHA/linux: fixup (did not cause an issue)
Follow-up to d75785c7de#18660
- GHA: create venvs later, simplify commands.
- GHA: sync pip command-line options, e.g. drop progress-bar,
everywhere.
Assisted-by: Dan Fandrich
Closes#18708
- codespell: break logic out into its own runnable script. Allowing
to run it on local machines.
- codespell: install via `pip`, bump to latest version.
- codespell: show version number in CI log.
- codespell: drop no longer needed word exception: `msdos`.
- codespell: include all curl source tree, except `packages` and
`winbuild`. Drop an obsolete file exclusion.
- add new spellchecker job using the `typos` tool. It includes
the codespell dictionary and a couple more. Use linuxbrew to install
it. This takes 10 seconds, while installing via `cargo` from source
would take over a minute.
- codespell: introduce an inline ignore filter compatible with `cspell`
Make `typos` recognize it, too. Move single exceptions inline.
Fix new typos found. Also rename variables and words to keep
spellchecking exceptions at minumum. This involves touching some tests.
Also switch base64 strings to `%b64[]` to avoid false positives.
Ref: https://github.com/crate-ci/typos/blob/master/docs/reference.md
Ref: https://github.com/codespell-project/codespell?tab=readme-ov-file#inline-ignore
Ref: https://github.com/codespell-project/codespell/issues/1212#issuecomment-1721152455
Ref: https://cspell.org/docs/Configuration/document-settingsCloses#17905
- GHA/checksrc: make a step name more accurate.
- cmakelint.sh: make sure to run from project root.
To make it easier to run locally.
- cmakelint.sh: use `set -eu`.
Closes#17915
CI check used macOS before this patch, but with the help of Linuxbrew,
latest zizmor and shellcheck are also available on Linux.
Also:
- migrate CI checks to the misc check workflow, to make both shellcheck
use the same, latest, shellcheck version, and to save the overhead of
an extra workflow.
Closes#17911
`checksrc.bat` was outdated and required Perl for `checksrc.pl` anyway.
Rewrite `checksrc-all.sh` in Perl, making it usable in envs without
a POSIX shell.
Closes#17882
Job `name:` now mandatory in zizmor.
Also:
- enclose `name:` values in single-quotes, for uniformity.
- drop `name: checkout` where set, for uniformity.
- dist: also install with cmake.
- dist: replace `make` with `cmake --build` for cmake.
(to make this make-tool agnostic)
- appveyor-status: double-quote shell arguments.
- tweak existing names to be shorter, to sync terms and style across
jobs and steps.
Ref: https://github.com/zizmorcore/zizmor/releases/tag/v1.10.0Closes#17773
zizmor 1.9.0 effectively bans using GHA macros within shell script
snippets. Rework them to pass GHA macro values via envs and use those
envs in shell snippets. `${{ env.* }}` macros could be converted
to native env references after making their names shell-compatible.
Envs and shell commands can no longer be used in GHA macro values. Most
cases could be fixed by using literals. Passing quoted values with
spaces combined with other args also doesn't work anymore. This was
replaced by passing them separately.
Despite the initial complications, avoiding GHA macros in scripts does
seems to make the CI code reasonable cleaner. It also makes it possible
to analyze the scripts with shellcheck, finding subtle issues that went
unnoticed till now.
Also:
- un-ignore and fix three existing zizmor `template-injection` issues.
- add script to extract and shellcheck all shell code stored within GHA
and Circle CI YAML files.
- add CI job to run this script.
- fix shellcheck issues discovered.
- fix minor differences between cmake and autotools FreeBSD jobs.
- merge cmake/autotools FreeBSD jobs to avoid developing unwanted
differences again.
- fix/sync quote use across shell code.
- replace `$HOME` with `~` or literal where it made sense.
- replace most `brew --prefix` with literals.
- move all curl install tests to the `curl-install*` prefix.
- add missing curl install tests to cygwin/msys/mingw/*bsd.
- pipe to `tar` instead of storing downloads on disk.
- drop unnecessary `PKG_CONFIG_PATH` when building nghttp3.
Ref: https://github.com/curl/curl/actions/runs/15461461371/job/43523546041
Ref: https://github.com/zizmorcore/zizmor/releases/tag/v1.9.0
Follow-up to e522f47986#17278Closes#17537
- Done with the new top-complexity script which uses the pmccabe tool.
- Any function scoring over 100 makes the test fail
- The script outputs all functions scoring over 70
- Two >100 functions are whitelisted by name, but they are not allowed
to increase their scores.
Closes#17398
To avoid redundant work in CI and to avoid a single checksrc issue make
all autotools jobs fail. After this patch checksrc issues make fail
the checksrc job, the `dist / verify-out-of-tree-autotools-debug`,
`dist / maketgz-and-verify-in-tree` jobs and the fuzzer job (if run).
Of these, the `dist` jobs replicate local builds, also testing the build
logic.
Also add a script to check the complete local repository, optionally
with the build tree to verify generated C files.
Also:
- automatically run checksrc in subdirectories having a `checksrc`
target. (examples, OS400, tests http/client, unit and tunit)
- tests/libtest: make sure to run `checksrc` on generated `lib1521.c`.
(requires in-tree autotools build.)
- tests: run `checksrc` on targets also for non-`DEBUGBUILD`
builds. It ensures to check `lib1521.c` in CI via job
`dist / maketgz-and-verify-in-tree`.
- src: drop redundant `$(builddir)` in autotools builds.
- scripts: add `checksrc-all.sh` script to check all C sources and
the build directory as an option.
- use the above from CI, also make it verify all generated sources.
- silence `checksrc` issues in generated C sources.
- checksrc: add `-v` option to enable verbose mode.
- checksrc: make verbose mode show checked filename and fix to only
return error on failure.
- make sure that generated C files pass `checksrc`.
Assisted-by: Daniel Stenberg
Closes#17376
Require now pytest-xdist from tests/http/requirements.txt and
run pytest in 'auto' parallel mode (counts cpu cores).
For CI runs, set the worker count to 4, overriding the
core count of 2 exposed in the images.
- use Filelock to generate allocated ports at start for all
workers and have subsequent workers just read the file and
take the ports for their slot
- make httpd config clearing a function fixture so every test
starts with a clean httpd config
- have fixture `configures_httpd` as parameter of test cases
that configure httpd anyway, saving one reload
- add pytest-xdist and filelock to required pyhton modules
- add installs to ruff CI
- give live checks waiting for a server to start up longer time
- add fixtures to tests that rely on a server
- do not stop servers unnecessarily. failures may not start them
properly again, leading to unexpected fails in whatever follows
- add a https: port to httpd that is *not* back by QUIC to allow
failover tests without stopping the QUIC server
Closes#17295
This makes it easier for users to run it with the correct options
locally. Also, update the ruff, pytype and cmakelint versions to the
latest current ones.
Closes#17343
The pedantic level is experimental. If it causes issues, we may just
disable it alongside the ignore comments.
Also:
- silence error:
```
INFO audit: zizmor: completed label.yml
error[dangerous-triggers]: use of fundamentally insecure workflow trigger
--> label.yml:13:1
|
13 | 'on': [pull_request_target]
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^ pull_request_target is almost always used insecurely
|
= note: audit confidence -> Medium
```
- fix pedantic warning:
```
INFO audit: zizmor: completed label.yml
warning[excessive-permissions]: overly broad permissions
--> label.yml:1:1
... |
24 | | with:
25 | | repo-token: '${{ secrets.GITHUB_TOKEN }}'
| |____________________________________________________- default permissions used due to no permissions: block
|
= note: audit confidence -> Medium
```
- silence `template-injection` false positives like:
```
- note: ${{ matrix.build.torture && 'test-torture' || 'test-ci' }} may expand into attacker-controllable code
- note: ${{ contains(matrix.build.install_steps, 'pytest') && 'caddy httpd vsftpd' || '' }} may expand into attacker-controllable code
```
It doesn't seem like these could be controlled by an attacker.
Let me know if I'm missing something.
Closes#17278
This step could take from 5 seconds to 5 minutes, sometimes making it
run out of its time slot. It affected 60 CI jobs.
Saving an estimated minimum of 5 minutes per CI run.
Also fixing:
```
Fri, 25 Apr 2025 06:19:14 GMT
Processing triggers for man-db (2.12.0-4build2) ...
Fri, 25 Apr 2025 06:23:40 GMT
Running kernel seems to be up-to-date.
[...]
Error: The action 'install packages' has timed out after 5 minutes.
```
Ref: https://github.com/curl/curl/actions/runs/14658212268/job/41136971525?pr=17180#step:2:169Closes#17181
Build/install the man page. Install the script.
This an "import" rather than just adding the script into the release tarball
with maketgz to make sure the tarball build is completely reproducible.
wcurl development, issues etc is still to be done at
https://github.com/curl/wcurl
Assisted-by: Viktor Szakats
Assisted-by: Samuel Henrique
Closes#17035
Before this patch the curl repository and source tarball distribution
contained test certificates as binary blobs. Used by runtests.
Drop these certificates in favor of generating them dynamically as
part of the build process. Both via autotools and CMake.
As part of this, improve certificates, the generator script and process,
file layout, and fix any issue to make it work fast and smooth both in
CI and local builds.
Note, cert generator scripts require OpenSSL >=1.0.2
(or LibreSSL >=3.1.0). Generation requires POSIX shell, also with CMake.
Without a POSIX shell tests relying on TLS (and stunnel) will fail.
Details:
- build: generate certs as part of the test run process.
- build, tests: generate certs in the build directory.
- binarycheck: drop concept of known binary files with hashes.
- binarycheck: move binary check logic into spacecheck and drop this
separate checker tool.
- build: fix to clean all cert files.
- autotools: fix to not run leaf cert generators in parallel. To avoid
confusion when updating the revocation database and counter.
- scripts: drop `scripts` subdir, merge two scripts into one,
auto-generate root cert, allow generating multiple leafs at once.
- scripts: switch to EC-256 keys (was: RSA-2048). For key size and perf.
- scripts: drop `-x` echo, text dumps, most other output. To avoid log
noise and make it quicker in CI.
- scripts: make it non-RSA-specific.
- scripts: delete unused code.
- scripts: use POSIX shell shebang. Some envs don't have bash (Alpine).
- scripts: pass test pseudo-secrets via the command-line. To avoid:
```
+ openssl genrsa -out test-ca.key -passout fd:0 2048
Invalid password argument, starting with "fd:"
```
- cmake: fix to launch generator scripts via the detected POSIX shell.
- cmake: fix `build-certs` rule to not depend on `SRPFILES`
(`srp-verifier-*`).
- cmake: drop `EXCLUDE_FROM_ALL` for the cert subdir. It makes
the Visual Studio generator miss to create the `clean-certs`,
`build-certs` targets. No target depend on them, so they don't execute
implicitly anyway. Fixes:
```
MSBUILD : error MSB1009: Project file does not exist.
Switch: clean-certs.vcxproj
```
- cmake: add `VERBATIM USES_TERMINAL` to `build-certs` target.
- GHA/linux: install openssl on Alpine, for the cert generator scripts.
Follow-up to 556f722fe3#16593
Follow-up to fa461b4eff#14486Closes#16824
- drop `--quiet 2` option where used, to have uniform output.
- replace `apt` with `apt-get` in one job. sync options with rest.
- replace deprecated `apt-key` command with the alternative recommended
by `apt-key(8)`.
- drop stray `cd /tmp`, no longer needed after migrating to GHA.
- shorten `--option Dpkg::Use-Pty=0` to `-o Dpkg::Use-Pty=0`.
- add `-o Dpkg::Use-Pty=0` to hide `apt-get` progress bars taking
vertical log space, where missing.
- drop `-y --no-install-suggests --no-install-recommends` `apt-get`
options. They are the default in the ubuntu-24.04 image.
- GHA/distcheck: move `name:` to top in steps where not there.
- scripts/cijobs.pl: catch `apt-get` lines with the `-o` option.
Closes#16127
These ensure a more consistent style and can find some errors statically
that would otherwise only be seen at run-time. Also, bump the Ubuntu
version of some other checks to get newer versions of some linters.
Closes#15067
