Commit f2ce6c46 among other things added the use of own library context
instead of the default context. Default context has access to OpenSSL
configuration file, own context doesn't have it.
Therefore if a pkcs11 provider is loaded via config file, the function
OSSL_PROVIDER_available() incorrectly detects the provider as
unavailable.
Fix this by loading the OpenSSL config to the library context according
to OpenSSL documentation:
"OSSL_LIB_CTX_load_config() loads a configuration file using the given
ctx. This can be used to associate a library context with providers that
are loaded from a configuration."
Moreover use the provider_loaded flag instead of provider pointer to
determine if a provider is available, as the latter is not set when the
provider is loaded from a configuration.
Closes#17804
Replace the `libstubgss.so`-based overload solution with one built into
libcurl at compile-time.
The previous, `LD_PRELOAD`-based, solution was non-portable, allowlisted
for Linux, BSD and Solaris. It also required non-debug builds, which
turned out to be an accidental condition:
7d342c723c. It also required a curl tool
built against a shared libcurl. Detecting this condition wasn't always
accurate, e.g. with certain cmake configurations.
The overload solution also didn't work on macOS, though it theoretically
should have:
- #17653
- #2394
Experiments on making the overload solution work in more envs:
- #17759
That revealed that it also did not work on NetBSD, in CI.
The replacement solution is overloading the necessary GSS-API functions
for test 2056 and 2057 at compile time. It requires a debug-enabled curl
build (due to its insecure nature).
This makes these tests run on all platforms. Including most GSS jobs in
CI, that are running tests. (the exception is old-linux, non-debug jobs,
where it felt overkill to enable debug for this.)
The refactored GSS stub code needs to overload less than before because
it's free to use the official GSS API. (This didn't work with
the overload solution on Alpine for example). It can also use libcurl
functions, allowing to replace `snprintf()` with `msnprintf()`.
OS/400 is also overloading GSS API functions. I haven't tested how this
works after this PR. In theory it should, because this PR doesn't rely
on preprocessor overrides.
Note that for future GSS tests, it may be necessary to stub these GSS
API functions: `gss_inquire_context()`, `gss_unwrap()`, `gss_wrap()`.
They are on codepaths not (yet) touched by tests.
Also:
- stub-gss: check for token buffer overrun.
- stub-gss: replace size macros with `sizeof()`.
- GHA: enable debug for some jobs with GSS.
- GHA/linux: ignore results for 2056 and 2057 in the valgrind job.
They leak the same way as seen with 2077 and 2078.
Ref: 7020ba7979#17462
Ref: 146759716c#14430
- GHA/linux: fix to ignore `gss_import_name()` leaks in valgrind builds.
only.
- lib/vauth/krb5_gssapi: reduce variable scope.
- lib/vauth/spnego_gssapi: reduce variable scope.
- tests/libtest: drop code and build logic dealing with `libstubgss`.
- runtests:
- drop `ld_preload` feature.
- drop special handling of `LD_PRELOAD` env in tests.
- drop logic dealing with shared curl tool detection.
- drop `LD_PRELOAD` envs from tests.
Follow-up to 56d949d31a#1687Closes#17752
In order to do unit tests for private functions, functions that are
marked UNITTEST but without a global scope in the library, functions
that do not have prototypes in their corresponding header file, unit
tests previously brought their own private prototype *copy* into the
unit test.
This was error-prone when the internal function changes but the change
might be missed in the unit test which then uses an outdated prototype
copy for testing.
This change removes the private prototypes from unit tests and instead
introduces a C file parser that parses the specific C files and extracts
the necessary unit test prototypes into a generated header file for unit
tests to use. This geneated lib/unitprotos.h header is then included by
unit tests that need private prototypes.
Assisted-by: Viktor Szakats
Closes#17750
Drop `strcasecompare` and `strncasecompare` in favor of libcurl API
calls `curl_strequal` and `curl_strnequal` respectively.
Also drop unnecessary `strcase.h` includes. Include `curl/curl.h`
instead where it wasn't included before.
Closes#17772
Eliminating the socket readability check in the socket connection
filters for the 'data_pending' callback. Improves performance of
handling of transfers, up to ~30%, depending on parallelism and response
size.
Whatever `data_pending()` once was, its semantics are now:
"Is there anything buffered in the connection filters that needs
receive?"
Any checks of the socket's readability are done via `multi_wait()`
and friends.
Fix the one place in HTTP/1 proxy code that checked `data_pending()` and
did an early return if false. Remove that check and actually try to
receive data every time.
Closes#17785
- tests/libtest: move exception to `stub_gssapi.h`.
- tests/libtest: move remaining exception to `testtrace.c`.
- tests/server: drop obsolete exception.
- docs/examples: move `BANNEDFUNC` exceptions to local files (3 lines).
- docs/examples: move `ERRNOVAR` exception to `ephiperfifo.c`.
- docs/examples: drop `typedef struct` (8 files).
- lib/curlx: add `.checksrc` with banned funcs copied from lib.
- checksrc: ban `strncpy`, `strtok_r`, `strtoul` by default.
Drop local bans. Add exception for `strtoul` to `tests/server'.
- lib, src: sync banned funcs.
Also:
- REUSE: drop `stunnel.pem`, it no longer exists.
- docs/examples: formatting.
- docs/examples: simplify some `sizeof()`s.
Closes#17764
Change the signature of `bufq` functions from
* `ssize_t Curl_bufq_*(..., CURLcode *err)` to
* `CURLcode Curl_bufq_*(..., size_t *pn)`
This allows us to write slightly less code and avoids the ssize_t/size_t
conversions in many cases. Also, it gets the function in line with all
the other send/recv signatures.
Added helper functions in `cfilters.h` for sending from/receving into
a bufq.
Fuzzer now fails to build due to these changes and its testing of
the bufq API.
Closes#17396
All arguments and local variables in `http_header_s` were unused when
both `CURL_DISABLE_COOKIES` and `CURL_DISABLE_HSTS` were defined.
Closes https://github.com/curl/curl/pull/17753
- markup some functions UNITTEST, so that they are static unless in a
unit test build
- make some functions #ifdef UNITTESTS as they are only used from unit
tests
- adjusted unit tests accordingly to use local prototypes for functions
not global in the library
Closes#17734
For consistency. RFC 3986 section 2.1 says:
"URI producers and normalizers should use uppercase hexadecimal digits
for all percent-encodings."
Reported-by: Jeroen Ooms
Fixes#17685Closes#17739
* calculate capacity growth on multi's xfer table and bitsets to
work correctly when approaching UINT_MAX
* uint-bset: track the first 64bit slot used. This avoids slot scans
on empty sets.
* uint-tbl: remove restriction to grow ot UINT_MAX, it is multi's
job to enforce limits suitable for its use
* test751: use curl_mfprintf() for error messages
Closes#17731
Replace existing `mk-unity.pl` `--embed` workaround with running
`clang-tidy` manually on individual test source instead. This aligns
with how clang-tidy works and removes `mk-unity.pl` from the solution.
Also:
- mqttd: fix potentially uninitialized buffer by zero filling it.
```
tests/server/mqttd.c:484:41: error: The left operand of '<<' is a garbage value
[clang-analyzer-core.UndefinedBinaryOperatorResult,-warnings-as-errors]
484 | payload_len = (size_t)(buffer[10] << 8) | buffer[11];
| ^
[...]
tests/server/mqttd.c:606:45: error: The left operand of '<<' is a garbage value
[clang-analyzer-core.UndefinedBinaryOperatorResult,-warnings-as-errors]
606 | topiclen = (size_t)(buffer[1 + bytes] << 8) | buffer[2 + bytes];
| ^
```
- sockfilt: fix potential out-of-bound pointer:
```
tests/server/sockfilt.c:1128:33: error: The 2nd argument to 'send' is a buffer
with size 17010 but should be a buffer with size equal to or greater than
the value of the 3rd argument (which is 18446744073709551615)
[clang-analyzer-unix.StdCLibraryFunctions,-warnings-as-errors]
1128 | ssize_t bytes_written = swrite(sockfd, buffer, buffer_len);
| ^
```
- clang-tidy: suppress bogus `bzero()` warnings that happens
inside the notorious `FD_ZERO()` macros, on macOS.
Ref: https://github.com/curl/curl/pull/17680#issuecomment-2991730158Closes#17705
Caught by gcc (Ubuntu 12.3.0-1ubuntu1~22.04) 12.3.0:
```
vtls/rustls.c: In function ‘cr_connect’:
vtls/rustls.c:857:61: warning: ‘%s’ directive argument is null [-Wformat-overflow=]
857 | failf(data, "rustls: must provide certificate with key '%s'",
| ^~
```
Closes#17704
The c-ares header directory was added to the header path within `lib`,
as opposed to every other dependency which added them in the root
`CMakeLists.txt`. Such exception is no longer necessary. This patch
aligns c-ares header setup with the rest of dependencies. And also with
autotools, which also makes no exception here.
Cherry-picked from #17705
Cherry-picked from #16973Closes#17707
Connection filters had a method `get_host()` which had not really been
documented. Since then, the cf had the `query()` method added. Replace
the separate get_host with query.
Add `CF_QUERY_HOST_PORT` as query to connection filters to retrieve
which remote hostname and port the filter (or its sub-filter) is talking
to. The query is implemented by HTTP and SOCKS filters, all others pass
it through.
Add `Curl_conn_get_current_host()` to retrieve the remote host and port
for a connection. During connect, this will return the host the
connection is talking to right now. Before/After connect, this will
return `conn->host.name`.
This is used by SASL authentication.
Closes#17419
Add a bitset `dirty` to the multi handle. The presence of a transfer int
he "dirty" set means: this transfer has something to do ASAP.
"dirty" is set by multiplexing protocols like HTTP/2 and 3 when
encountering response data for another transfer than the current one.
"dirty" is set by protocols that want to be called.
Implementation:
* just an additional `uint_bset` in the multi handle
* `Curl_multi_mark_dirty()` to add a transfer to the dirty set.
* `multi_runsingle()` clears the dirty bit of the transfer at
start. Without new dirty marks, this empties the set after
al dirty transfers have been run.
* `multi_timeout()` immediately gives the current time and
timeout_ms == 0 when dirty transfers are present.
* multi_event: marks all transfers tracked for a socket as dirty.
Then marks all expired transfers as dirty. Then it runs
all dirty transfers.
With this mechanism:
* Most uses of `EXPIRE_RUN_NOW` are replaced by `Curl_multi_mark_dirty()`
* `Curl_multi_mark_dirty()` is cheaper than querying if a transfer is
already dirty or set for timeout. There is no need to check, just do it.
* `data->state.select_bits` is eliminated. We need no longer to
simulate a poll event to make a transfer run.
Closes#17662
- cmake: use `CURL_RCFILES` instead of literal.
- cmake: use `LIB_RCFILES` instead of literal.
- cmake: fix comments.
- autotools: use `CURL_RCFILES` in `EXTRA_DIST`.
- autotools: use `LIB_RCFILES` in `EXTRA_DIST`.
- autotools: fix comments.
- autotools: fix indentation.
Closes#17694
In the unlikely case that no SSH auth methods are supported, the
previous code would return 0 from myssh_in_AUTH_PKEY_INIT. However,
following the code path, it seems like it should be returning SSH_ERROR,
as set in myssh_to_ERROR (through myssh_to_GSSAPI_AUTH,
myssh_to_KEY_AUTH and myssh_to_PASSWD_AUTH).
In actuality, this is unlikely to occur, as the similar code in
myssh_in_AUTHLIST would have already returned an error in this scenario.
However setting a return value and then ignoring it is a bit fishy and
should be documented if this is intended.
I believe this used to return an error, but was changed in the recent
re-factoring of this code.
Closes#17691
Allow prequotes to be sent after curl has changed the working directory,
just before the listing command if the URL is a directory.
FTP state machine is updated with the new FTP_LIST_PREQUOTE state and
FTP_RETR_LIST_TYPE type.
Test 754 verifies
Fixes#8602Closes#17616
Concentrate the handling of KEEP_RECV_PAUSE and KEEP_SEND_PAUSE into
common transfer functions. Setting or clearing these bits requires
subsequent actions involving connection events and client reader/writer
notifications. Have it in one place.
Closes#17650
Used to be a pointer set (and cleared) by the socket connection filters
to a struct in their contexts. Instead, add a filter query method to
obtain the pointer when needed.
Closes#17385
The `#undef` hack is no longer necessary after changing the redifitions
to not map back to the original symbols.
This makes it unnecessary to repeat the redefinitions after compiling
`warnless.c` itself (in unity mode).
Which in turns makes it unnecessary to include `warnless.h` again, to
trigger such redefinition.
This also means that `read`/`write` are now redefined on Windows from
the first inclusion of `warnless.h`.
Also:
- tests/server: drop a repeat `warnless.h` include, that is unnecessary
after this patch.
- tests/unit: drop repeat `warnless.h` include.
- tests/libtest: drop repeat `warnless.h` includes.
- tests/libtest: formatting.
Follow-up to 2f312a14da#17619
Follow-up to 84338c4de2#12331
Follow-up to 6239146e93Closes#17673
Use it from libtests' `first.c` and thus also from units, and tunits.
Also:
- cmake: drop stray `curltool` lib dependency for units.
- units: stop depending on `src` headers.
- tests/server: drop depending on `src` headers.
(the remaining one listed in the comments, `tool_xattr.h`, was not
actually used from servers.)
- tests/server: drop duplicate curlx headers.
(Except `warnless.h`, which is tricky on Windows.)
Closes#17672
To pass `-municode` to the linker. Before this patch we passed this via
`target_link_libraries()` which is designed to pass libraries. Keep
using it for old CMake versions, where no better alternative existed.
https://cmake.org/cmake/help/latest/command/target_link_options.html
Also:
- also pass `-municode` as `PRIVATE` for old cmake versions.
(it should not make a difference because no target depends on the curl
tool, but this seem to be the modern, non-ambiguous syntax.)
- unfold a bunch of split lines for greppability of `add_library()` and
`add_executable()` commands.
- quote a string.
Closes#17670
When checking to expire a transfer with input data pending, also assess
the blocked status and do not EXPIRE_RUN_NOW a transfer that is blocked
on READ/WRITE.
Follow-up to 62349e45a8#17636Closes#17639
When a transfer is done prematurely, a RST is generated to the server,
but was not send right away due to send buffering.
Flush the send buffer in a best effort when transfer is done.
Reported-by: Michael Kaufmann
Fixes#17611Closes#17638
When multi creates the pollset of a transfer, it checks now if
a connection (FIRST/SECONDARY) socket waits on POLLIN and has input data
pending in filters (relevant to OpenSSL's new read ahead). If so, it
triggers a timeout on the transfer via EXPIRE_RUN_NOW.
This fixes sporadic stalls in test 988 when running event based.
Closes#17636
Introduce an immutable `CURL_FOPEN()` macro to store the `fopen()`
mapping on Windows. Then use that instead `(fopen)` from `memdebug.c`.
It makes CURLDEBUG builds use the correct `fopen` wrapper on Windows.
This macro is only defined on Windows, as of this patch.
This is necessary after cde81e4398,
which no longer applies the default `fopen()` override to `memdebug.c`.
Also:
- curl_setup.h: de-dupe, simplify Windows file I/O function overrides.
- curl_memory.h: fix to reset `fopen` to `curlx_win32_fopen()` on
Windows. Before this patch it reset it to stock `fopen()`.
Follow-up to cde81e4398#17631Closes#16747
Before this patch `memdebug.c` was compiled as a separate source in
unity builds. This was necessary because `memdebug.c` failed to compile
if `memdebug.h` was included before it, in `CURLDEBUG` mode. This patch
fixes this issue and allows to compile `memdebug.c` as part of the unity
source batch. This removes an exception and makes builds perform a notch
better.
- introduce `CURL_SCLOSE()` macro as an immutable synonym of `sclose()`.
- memdebug: replace `sclose()` reference with `CURL_SCLOSE()` to compile
as expected when `sclose()` is overridden by `memdebug.h`.
- memdebug: make it not break when including `memdebug.h` before it in
`CURLDEBUG` mode. Do this by calling low-level functions as
`(function)`.
- autotools, cmake: drop memdebug exception, include it like any other
source file. This is now possible because `memdebug.c` doesn't break
if `memdebug.h` was included before it, in `CURLDEBUG` builds.
- mk-unity: drop `--exclude` option. No longer used after this patch.
- drop `MEMDEBUG_NODEFINES` macro hack. No longer necessary.
Ref: #16747Closes#16746Closes#16738Closes#17631
Instead of relying on CMake's built-in unity feature, use `mk-unity.pl`,
as already done with autotools. It simplified the build, shortens logs
and makes debugging easier because of the fewer build variations.
It also allows testing / fixing with cmake and those automatically apply
to autotools builds too. cmake builds can be much-much faster, esp.
when working the builds themselves.
It also enables "unity" in old cmake versions. Basically every test
target is a single generated .c source.
Also:
- drop a `lib` unity workaround for libtests with autotools after fixing
the issue in libtests itself. It drops a few exceptions and makes
libcurl build faster (in autotools unity).
- fix another `lib` autotools unity issue and drop the workaround for it
from `mk-unity.pl`. `srcdir` was missing from the header path.
- simplify `mk-unity.pl` command-lines, drop exclusions.
Follow-up to 2c27a67daa#17590Closes#17628
Both are available with well-known conditions, under non-Windows, and
`curl/curl.h` already uses them. `sys/time.h` is also necessary for
mingw-w64 for `gettimeofday()`.
Follow-up to 56d5982312#17522Closes#17581
