runtests: add options to set minimum number of tests, use them

To detect mistakes made in the runtests framework that reduce
the number of test runs. Before this patch it could go undetected with
a green CI.

The minimum thresholds will need light maintenance going forward (either
bumping them periodically, or adjust if some may fell below minimums for
justified reasons). We may also make minimums tighter or looser, or more
job-specific.

Latest number of test runs for each job can be seen at Test Clutch:
https://testclutch.curl.se/static/reports/feature-matrix.html

Also:
- GHA: set minimums.

Assisted-by: Dan Fandrich

Follow-up to f2a75a14dd
Follow-up to bb1391f943 #19510

Closes #19942
This commit is contained in:
Viktor Szakats 2025-12-12 01:55:30 +01:00
parent eb39fee40b
commit 3f1cd809ee
No known key found for this signature in database
GPG key ID: B5ABD165E2AEF201
9 changed files with 66 additions and 22 deletions

View file

@ -36,6 +36,7 @@ permissions: {}
env:
MAKEFLAGS: -j 5
CURL_CI: github
CURL_TEST_MIN: 1600
CURL_CLANG_TIDYFLAGS: '-checks=-clang-analyzer-security.insecureAPI.bzero,-clang-analyzer-security.insecureAPI.strcpy,-clang-analyzer-optin.performance.Padding,-clang-analyzer-security.ArrayBound,-clang-analyzer-security.insecureAPI.DeprecatedOrUnsafeBufferHandling,-clang-analyzer-valist.Uninitialized'
# renovate: datasource=github-tags depName=libressl/portable versioning=semver registryUrl=https://github.com
LIBRESSL_VERSION: 4.2.1
@ -184,6 +185,7 @@ jobs:
configure: --with-openssl --disable-ipv6 --enable-debug --disable-unity --disable-libcurl-option --disable-digest-auth
- name: 'openssl https-only'
tflags: '--min=1150'
configure: >-
--with-openssl --enable-debug --disable-unity
--disable-dict --disable-gopher --disable-ldap --disable-telnet
@ -195,13 +197,13 @@ jobs:
- name: 'openssl torture !FTP'
install_packages: libnghttp2-dev libssh2-1-dev libc-ares-dev
generate: -DCURL_USE_OPENSSL=ON -DENABLE_DEBUG=ON -DENABLE_ARES=ON
tflags: '-t --shallow=25 !FTP'
tflags: '-t --shallow=25 !FTP --min=1450'
torture: true
- name: 'openssl torture FTP'
install_packages: libnghttp2-dev libssh2-1-dev libc-ares-dev
generate: -DCURL_USE_OPENSSL=ON -DENABLE_DEBUG=ON -DENABLE_ARES=ON
tflags: '-t --shallow=20 FTP'
tflags: '-t --shallow=20 FTP --min=260'
torture: true
- name: 'openssl i686'
@ -215,6 +217,7 @@ jobs:
--with-openssl --with-librtmp --with-libssh2 --with-libidn2 --enable-ares --enable-debug
- name: '!ssl !http !smtp !imap'
tflags: '--min=475'
configure: --without-ssl --enable-debug --disable-http --disable-smtp --disable-imap --disable-unity
- name: 'libressl Fil-C'
@ -303,11 +306,12 @@ jobs:
LDFLAGS: -fsanitize=memory
LIBS: -ldl
configure: CC=clang --without-ssl --without-zlib --without-brotli --without-zstd --without-libpsl --without-nghttp2 --enable-debug
tflags: '--min=1480'
- name: 'event-based'
install_packages: libssh-dev
configure: --enable-debug --disable-shared --disable-threaded-resolver --with-libssh --with-openssl
tflags: '-n --test-event'
tflags: '-n --test-event --min=1350'
- name: 'duphandle'
install_packages: libssh-dev