runtests: add options to set minimum number of tests, use them

To detect mistakes made in the runtests framework that reduce
the number of test runs. Before this patch it could go undetected with
a green CI.

The minimum thresholds will need light maintenance going forward (either
bumping them periodically, or adjust if some may fell below minimums for
justified reasons). We may also make minimums tighter or looser, or more
job-specific.

Latest number of test runs for each job can be seen at Test Clutch:
https://testclutch.curl.se/static/reports/feature-matrix.html

Also:
- GHA: set minimums.

Assisted-by: Dan Fandrich

Follow-up to f2a75a14dd
Follow-up to bb1391f943 #19510

Closes #19942
This commit is contained in:
Viktor Szakats 2025-12-12 01:55:30 +01:00
parent eb39fee40b
commit 3f1cd809ee
No known key found for this signature in database
GPG key ID: B5ABD165E2AEF201
9 changed files with 66 additions and 22 deletions

View file

@ -44,6 +44,7 @@ permissions: {}
env:
CURL_CI: github
CURL_TEST_MIN: 1700
MAKEFLAGS: -j 4
LDFLAGS: -w # suppress 'object file was built for newer macOS version than being linked' warnings
@ -225,16 +226,20 @@ jobs:
compiler: clang
install: brotli zstd
configure: --without-ssl --with-brotli --with-zstd
tflags: '--min=1450'
xcode: '' # default Xcode. Set it once to silence actionlint.
- name: '!ssl !debug'
compiler: gcc-13
configure: --without-ssl
tflags: '--min=950'
- name: '!ssl'
compiler: clang
configure: --enable-debug --without-ssl
tflags: '--min=1500'
- name: '!ssl libssh2 AppleIDN'
compiler: clang
configure: --enable-debug --with-libssh2=/opt/homebrew/opt/libssh2 --without-ssl --with-apple-idn
tflags: '--min=1550'
- name: 'OpenSSL libssh c-ares'
compiler: clang
install: libssh
@ -246,8 +251,10 @@ jobs:
- name: '!ssl c-ares'
compiler: clang
configure: --enable-debug --enable-ares --without-ssl
tflags: '--min=1500'
- name: '!ssl HTTP-only'
compiler: clang
tflags: '--min=950'
configure: >-
--enable-debug
--disable-alt-svc --disable-dict --disable-file --disable-ftp --disable-gopher --disable-imap
@ -277,7 +284,7 @@ jobs:
- name: 'OpenSSL event-based'
compiler: clang
configure: --enable-debug --with-openssl=/opt/homebrew/opt/openssl
tflags: --test-event
tflags: '--test-event --min=1300'
# cmake
- name: 'OpenSSL gsasl rtmp AppleIDN SecTrust +examples'
install: libnghttp3 libngtcp2 gsasl rtmpdump
@ -328,18 +335,19 @@ jobs:
compiler: clang
install: rustls-ffi
generate: -DENABLE_DEBUG=ON -DCURL_USE_RUSTLS=ON -DUSE_ECH=ON -DCURL_DISABLE_LDAP=ON
tflags: '--min=1650'
- name: 'OpenSSL torture !FTP'
compiler: clang
install: libnghttp3
install_steps: torture
generate: -DENABLE_DEBUG=ON -DBUILD_SHARED_LIBS=OFF -DENABLE_THREADED_RESOLVER=OFF -DOPENSSL_ROOT_DIR=/opt/homebrew/opt/openssl -DUSE_OPENSSL_QUIC=ON
tflags: -t --shallow=25 !FTP
tflags: '-t --shallow=25 !FTP --min=1450'
- name: 'OpenSSL torture FTP'
compiler: clang
install: libnghttp3
install_steps: torture
generate: -DENABLE_DEBUG=ON -DBUILD_SHARED_LIBS=OFF -DENABLE_THREADED_RESOLVER=OFF -DOPENSSL_ROOT_DIR=/opt/homebrew/opt/openssl -DUSE_OPENSSL_QUIC=ON
tflags: -t --shallow=20 FTP
tflags: '-t --shallow=20 FTP --min=260'
exclude:
# opt out jobs from combinations that have the compiler set manually
- { compiler: llvm@18, build: { compiler: 'clang' } }