To allow replacing `&` characters in `tests/data/test*` files for
XML-compliance.
Also:
- document `%GT`, `%LT`
Follow-up to de49cc89ab#19470Closes#19824
Address issues listed in #19770:
- allow for ngttpx to successfully shut down on last attempt that might
extend beyond the finish timestamp
- timeline checks: allos `time_starttransfer` to appear anywhere in
the timeline as a slow client might seen response data before setting
the other counters
- dump logs on test_05_02 as it was not reproduced locally
Fixes#19970Closes#19783
A fix for the tests that took the longest:
- test_05: make the server close the HTTP/1.1 connection when
simulating an error during a download. This eliminates waiting
for a keepalive timeout
- test_02: pause tests with slightly smaller documents, eliminate
special setup for HTTP/2. We test stream window handling now
elsewhere already
- cli_hx_download: run look in 500ms steps instead of 1sec, resuming
paused tranfers earlier.
Closes#19809
To avoid potentially executing a different Perl than used by the rest
of the build and tests.
Also to be more portable by not relying on shebang support, though these
particular tests require POSIX shell anyway.
Closes#19767
Move check logic from postchecks to `test613.pl`.
To make these test data files XML-compliant.
Also to avoid POSIX/bash-shellisms in postcheck.
Closes#19766
Rework the way curl's custom Find modules advertise their properties.
Before this patch, Find modules returned detected dependency properties
(header dirs, libs, libdirs, C flags, etc.) via global variables. curl's
main `CMakeLists.txt` copied their values into global lists, which it
later applied to targets. This solution worked internally, but it was
unsuited for the public, distributed `CURLConfig.cmake` and publishing
curl's Find modules with it, due to polluting the namespace of consumer
projects. It's also impractical to apply the many individual variables
to every targets depending on libcurl.
To allow using Find modules in consumer projects, this patch makes them
define as imported interface targets, named `CURL::<dependency>`. Then
store dependency information as target properties. It avoids namespace
pollution and makes the dependency information apply automatically
to all targets using `CURL::libcurl_static`.
Find modules continue to return `*_FOUND` and `*_VERSION` variables.
For dependencies detected via `pkg-config`, CMake 3.16+ is recommended.
Older CMake versions have a varying degree of support for
propagating/handling library directories. This may cause issues in envs
where dependencies reside in non-system locations and detected via
`pkg-config` (e.g. macOS + Homebrew). Use `CURL_USE_PKGCONFIG=OFF`
to fix these issues. Or upgrade to newer CMake, or link libcurl
dynamically.
Also:
- re-enable `pkg-config` for old cmake `find_library()` integration
tests.
- make `curlinfo` build after these changes.
- distribute local Find modules.
- export the raw list of lib dependencies via `CURL_LIBRARIES_PRIVATE`.
- `CURLconfig.cmake`: use curl's Find modules to detect dependencies in
the consumer env.
- add custom property to target property debug function.
- the curl build process no longer modifies `CMAKE_C_FLAGS`.
Follow-up to e86542038d#17047
Ref: #14930
Ref: https://github.com/libssh2/libssh2/pull/1535
Ref: https://github.com/libssh2/libssh2/pull/1571
Ref: https://github.com/libssh2/libssh2/pull/1581
Ref: https://github.com/libssh2/libssh2/pull/1623Closes#16973
Before this patch curl used the C preprocessor to override standard
memory allocation symbols: malloc, calloc, strdup, realloc, free.
The goal of these is to replace them with curl's debug wrappers in
`CURLDEBUG` builds, another was to replace them with the wrappers
calling user-defined allocators in libcurl. This solution needed a bunch
of workarounds to avoid breaking external headers: it relied on include
order to do the overriding last. For "unity" builds it needed to reset
overrides before external includes. Also in test apps, which are always
built as single source files. It also needed the `(symbol)` trick
to avoid overrides in some places. This would still not fix cases where
the standard symbols were macros. It was also fragile and difficult
to figure out which was the actual function behind an alloc or free call
in a specific piece of code. This in turn caused bugs where the wrong
allocator was accidentally called.
To avoid these problems, this patch replaces this solution with
`curlx_`-prefixed allocator macros, and mapping them _once_ to either
the libcurl wrappers, the debug wrappers or the standard ones, matching
the rest of the code in libtests.
This concludes the long journey to avoid redefining standard functions
in the curl codebase.
Note: I did not update `packages/OS400/*.c` sources. They did not
`#include` `curl_setup.h`, `curl_memory.h` or `memdebug.h`, meaning
the overrides were never applied to them. This may or may not have been
correct. For now I suppressed the direct use of standard allocators
via a local `.checksrc`. Probably they (except for `curlcl.c`) should be
updated to include `curl_setup.h` and use the `curlx_` macros.
This patch changes mappings in two places:
- `lib/curl_threads.c` in libtests: Before this patch it mapped to
libcurl allocators. After, it maps to standard allocators, like
the rest of libtests code.
- `units`: before this patch it mapped to standard allocators. After, it
maps to libcurl allocators.
Also:
- drop all position-dependent `curl_memory.h` and `memdebug.h` includes,
and delete the now unnecessary headers.
- rename `Curl_tcsdup` macro to `curlx_tcsdup` and define like the other
allocators.
- map `curlx_strdup()` to `_strdup()` on Windows (was: `strdup()`).
To fix warnings silenced via `_CRT_NONSTDC_NO_DEPRECATE`.
- multibyte: map `curlx_convert_*()` to `_strdup()` on Windows
(was: `strdup()`).
- src: do not reuse the `strdup` name for the local replacement.
- lib509: call `_strdup()` on Windows (was: `strdup()`).
- test1132: delete test obsoleted by this patch.
- CHECKSRC.md: update text for `SNPRINTF`.
- checksrc: ban standard allocator symbols.
Follow-up to b12da22db1#18866
Follow-up to db98daab05#18844
Follow-up to 4deea9396b#18814
Follow-up to 9678ff5b1b#18776
Follow-up to 10bac43b87#18774
Follow-up to 20142f5d06#18634
Follow-up to bf7375ecc5#18503
Follow-up to 9863599d69#18502
Follow-up to 3bb5e58c10#17827Closes#19626
Change `inputbuff` parameter from `const char *` to `const uint8_t *` to
reflect the binary nature of the input bytes. Half the code was casting
unsigned char to signed already in calling.
Closes#19722
Make `port` member in these struct of type `uint16_t`.
add `uint8_t transport` to `struct ip_quadruple
Define TRNSPRT_NONE as 0. By assigning a valid transport only on a
successful connection, it is clear when the ip_quadruple members are
valid. Also, for transports not involving ports, the getinfos for
`CURLINFO_PRIMARY_PORT` and `CURLINFO_LOCAL_PORT` will now always return
-1.
Make all `transport` members and parameters of type `uint8_t`.
Document the return value of `CURLINFO_LOCAL_PORT` and
`CURLINFO_PRIMARY_PORT` in this regard. Add tests that writeout stats
report ports correctly.
Closes#19708
* move the TIMER_POSTQUEUE to the time a connection is chosen,
so that TIMER_NAMELOOKUP always happens afterwards
* client writer: do not trigger TIMER_STARTTRANSFER on CLIENTWRITE_INFO
as ftp and other pingpong protocols write that before starting anything
that is the tranfer itself
* Elimnating debug trancing of "closed stream/connection - bailing"
as confusing, as connection is not really closed on most cases.
* Setting 'data->req.upload_done` correctly, so that no "abort upload"
is happening at the end of a perfectly fine download.
* Adding test cases with up-/download of 0-length files.
* pytest: add a "timeline" of timer value checks to Resulst in curl.py,
so that this can be used in several test cases, replacing the local
stuff in test_16
* add timeline checks to ftp test cases
Closes#19269
- Curl_async_getaddrinfo() always returned NULL so it was pointless.
Return proper curlcode instead to distinguish between errors. Same for
Curl_doh().
- simplify the IP address handling
- make Curl_str2addr() function return CURLcode
Closes#19669
Replace:
- `open()` with `curlx_open()` (1 call).
- `fopen()` with `curlx_fopen()`.
- `fclose()` with `curlx_fclose()`.
To centralize interacting with the CRT in preparation for using "safe"
alternatives on Windows. This also adds long-filename and Unicode
support for these operations on Windows.
Keep using `open()` in the signal handler to avoid any issues with
calling code not allowed in signal handlers.
Cherry-picked from #19643Closes#19679
Windows CRTs have a `share.h`. Before this patch when trying to
`#include <share.h>` it, the compiler picked up curl's internal
`lib/share.h` instead. Rename it to avoid this issue.
CRT `share.h` has constants necessary for using safe open CRT functions.
Also rename `lib/share.c` to keep matching the header.
Ref: https://learn.microsoft.com/cpp/c-runtime-library/sharing-constants
Ref: 625f2c1644#16949#16991
Cherry-picked from #19643Closes#19676
Description of how this works in `docs/internal/RATELIMITS.ms`.
Notable implementation changes:
- KEEP_SEND_PAUSE/KEEP_SEND_HOLD and KEEP_RECV_PAUSE/KEEP_RECV_HOLD
no longer exist. Pausing is down via blocked the new rlimits.
- KEEP_SEND_TIMED no longer exists. Pausing "100-continue" transfers
is done in the new `Curl_http_perform_pollset()` method.
- HTTP/2 rate limiting implemented via window updates. When
transfer initiaiting connection has a ratelimit, adjust the
initial window size
- HTTP/3 ngtcp2 rate limitin implemnented via ack updates
- HTTP/3 quiche does not seem to support this via its API
- the default progress-meter has been improved for accuracy
in "current speed" results.
pytest speed tests have been improved.
Closes#19384
- adjust cipher list in infof() statement for min/max TLS version
- skip test_17_07 for wolfSSL 5.8.4 when CHACHA20 is negotiated
due to regression with homebrew build on ARM systems.
Fixes#19644
Reported-by: Viktor Szakats
Closes#19662
Usage:
- autotools: `--disable-typecheck` (or `--enable-typecheck` (default))
- cmake: `-DCURL_DISABLE_TYPECHECK=ON`.
To disable `curl_easy_setopt()`/`curl_easy_getinfo()` type checking with
supported (new) gcc and clang compilers. It is useful to improve build
performance for the `tests/libtest` target. In particular the CodeQL
analyzer may take above an hour to compile with type checking enabled,
and disabling it brings it down to seconds. On local machines it may
also cut build times in half when build testdeps, depending on platform
and compiler.
Other than these cases, we recommend leaving type checking enabled.
Ref: fdacf34aae#19632
Also:
- GHA/codeql: use it.
- test1165: check in `include/curl`.
- lib1912: delete stray todo comment.
- spelling and comment nits.
Closes#19637
To override the curl default of 5 minutes (300000 ms).
Sometimes a simple test data change can result in a stuck test, this
option makes it exit with an error early. Possible future use in CI
or fast machines to prevent a single test taking 5 minutes and failing
the whole job.
Example hangers:
tests/data/test65:
```diff
-<data1000 crlf="yes">
+<data1000 crlf="headers">
```
tests/data/tests993:
```diff
-%repeat[1000 x 95 328485%0d%0a]%</data>
+%repeat[1000 x 95 328485%0d%0a]%
+</data>
```
Closes#19319
There are 58 non-compliant files. Mark them with the `notxml` keyword.
Also include the compliant ones in the GHA/checksrc xmllint CI job.
Also:
- delete XML prolog from the 3 test data files that had them.
- FILEFORMAT.md: document the `notxml` keyword.
- FILEFORMAT.md: fix a typo.
Follow-up to de49cc89ab#19470
Follow-up to f3095f0dbd#19528
Follow-up to 87ba80a6dfCloses#19595
Reduce number of files failing `xmllint --format` from 133 to 57 (-76)
(3% of 1894), by replacing `<` and `>` with new macro `%LT` and `%GT`,
in most places, which is in email addresses (192 lines).
Follow-up to a9ec2a676c#19491Closes#19470
Passing the option as-is to libcurl is fine, but checking that the file
exists allows the tool to better provide a helpful message.
This now done for the following options:
--cacert, --crlfile, --knownhosts, --netrc-file, --proxy-cacert amd
--proxy-crlfile
Bonus: bail out properly on OOM errors in the --cert parser.
Reported-by: Wesley Moore
Fixes#19583Closes#19585
The parsing of the HTTP/1.1 formatted request into the h2/h3 header
structures should detect CURLOPT_CUSTOMREQUEST methods and forward them
correctly.
Add test_01_20 to verify
Fixes#19543
Reported-by: Omdahake on github
Closes#19563
There remain some false positives, hits in test data, and `dir` use,
around 100 issues in total.
There is no plan to enforce badwords on tests.
Also:
- badwords.txt: let a few `manpage[s]` occurrences through
(in Perl code).
Closes#19541
- fix test_17_20 flakiness: the test case did not have `nghttpx` in
its parameters, causing it to no check if a reload was necessary.
When that test ran behind one that gave nghttpx another certificate,
eg. in parallel mode, it used the wrong pinned pubkey.
- Have `env` provide lists of HTTP protocol versions available for
testing. Replace parameterized tests on a fixed protocol list with
the dynamic one from env. This makes checks for protocol availability
in the test function bodies superfluous.
refs #19489Closes#19540
test 1459 "SFTP with corrupted known_hosts" was seen failing in the past.
To fix it, the test was automatically disabled when detecting libssh
0.9.3 or older, as in the curl CircleCI job, running on Ubuntu 20.04.
This work for a long time, until bumping the CircleCI runner to Ubuntu
22.04 (to have OpenSSL 3), where the test was running again, and failing
with the isssue seen in the past.
- Test skipped with Ubuntu 20.04 (libssh 0.9.3):
https://app.circleci.com/pipelines/github/curl/curl/16445/workflows/7f198763-e0b0-4037-9245-4c4b40ab8726/jobs/155164
- Failure seen with Ubuntu 22.04 (libssh 0.9.6):
https://app.circleci.com/pipelines/github/curl/curl/16452/workflows/b817a808-0fd4-40b0-8eb0-d064926efe12/jobs/155206?invite=true#step-107-211709_45
- Failure seen with Ubuntu 24.04 (libssh 0.10.6):
https://app.circleci.com/pipelines/github/curl/curl/16455/workflows/86c631f1-3c5f-4438-b398-3df2bdab5d20/jobs/155218
Turns out the issue issue isn't libssh 0.9.3 itself, but
a CircleCI-specific default configuration in `/etc/ssh/ssh_config`:
```
# BEGIN ANSIBLE MANAGED BLOCK
Host *
StrictHostKeyChecking no <------ this particular line
HashKnownHosts no
SendEnv LANG LC_*
# END ANSIBLE MANAGED BLOCK
```
libssh will consult configuration files on hard-coded default system
locations and alter its behavior based on settings found in them.
This libssh behavior is present in all supported versions:
5a2abd34cehttps://gitlab.com/libssh/libssh-mirror/-/tags/libssh-0.9.0
It means the existing disable logic based on libssh version worked by
coincidence, and what needs to be checked is these configurations
to decide if it's safe to run the test. Another, simpler option is
to also accept the result code 67, though in that case the test
wouldn't actually test what we want, but would pass anyway.
With the old `oldlibssh` workaround deleted, and the problematic setting
manually overridden (`StrictHostKeyChecking yes`):
- CircleCI Ubuntu 20.04 passes with 1459 enabled:
https://app.circleci.com/pipelines/github/curl/curl/16483/workflows/87a9f389-76a2-4a32-acde-c0b411a4c842/jobs/155302
- CircleCI Ubuntu 22.04 does too:
https://app.circleci.com/pipelines/github/curl/curl/16483/workflows/87a9f389-76a2-4a32-acde-c0b411a4c842/jobs/155303
To fix, replace the `runtests` `oldlibssh` detection logic to parse
libssh config files (instead of checking for libssh version) and disable
test 1459 based on that. Notice the detection is making a light attempt
to parse these files, and does not implement most config file features
(such as includes, quoted values and `=` operator.)
The new runtests workaround tests OK with the:
- default CircleCI configuration, disabling 1459 automatically.
- a sudoless configuration fix, with 1459 run successfully.
Also keep setting this option in CircleCI jobs.
- a sudo configuration fix, with 1459 run successfully.
Ref: https://app.circleci.com/pipelines/github/curl/curl/16492/workflows/56f39335-97ba-412c-9a9b-3d662694375a
GHA jobs are not affected and they work fine, with 1459 running successfully
before and after this patch.
It's possible the libssh API offers ways to control config file use
and/or set the strict host checking option programatically. Maybe
to enable in debug mode (albeit CircleCI job are not debug-enabled),
or offer an option for them. It may be something for a future patch.
Follow-up to 23540923e1#8622
Follow-up to 4b01a57c95#8548
Follow-up to bdc664a640#8490
Follow-up to 7c140f6b2d#8444
Ref: 6d9c5c91b9#19549Closes#19557
