With the new addition of OpenSSL QUIC API support and the support in
ngtcp2 main branch, make the necessary adjustments in curl to support
this combination.
- add support in configure.ac to detect the feature OPENSSL_QUIC_API2 in
openssl
- initialise ngtcp2 properly in this combination
- add a Curl_vquic_init() for global initialisation that ngtcp2 likes
for performance reasons
- add documentation on how to build in docs/HTTP3.md
- add CI testing in http3-linux.yml
Assisted-by: Viktor Szakats
Closes#17027
"asyn" is the internal name under which both c-ares and threaded
resolver operate. Make the naming more consistent. Implement the c-ares
resolver in `asyn-ares.*` and the threaded resolver in `asyn-thrdd.*`.
The common functions are in `asyn-base.c`.
When `CURLRES_ASYNCH` is defined, either of the two is used and
`data->state.async` exists. Members of that struct vary for the selected
implementation, but have the fields `hostname`, `port` and `ip_version`
always present. This are populated when the async resolving starts and
eliminate the need to pass them again when checking on the status and
processing the results of the resolving.
Add a `Curl_resolv_blocking()` to `hostip.h` that relieves FTP and SOCKS
from having to repeat the same code.
`Curl_resolv_check()` remains the function to check for status of
ongoing resolving. Now it also performs internally the check if the
needed DNS entry exists in the dnscache and if so, aborts the asnyc
operation. (libcurl right now does not check for duplicate resolve
attempts. an area for future improvements).
The number of functions in `asyn.h` has been reduced. There were subtle
difference in "cancel()" and "kill()" calls, both replaced by
`Curl_async_shutdown()` now. This changes behaviour for threaded
resolver insofar as the resolving thread is now always joined unless
`data->set.quick_exit` is set. Before this was only done on some code
paths. A future improvement would be a thread pool that keeps a limit
and also could handle joins more gracefully.
DoH, not previously tagged under "asny", has its struct `doh_probes` now
also in `data->state.async`, moved there from `data->req` because it
makes more sense. Further integration of DoH underneath the "asyn"
umbrella seems like a good idea.
Closes#16963
The SChannel code uses the CertFindCertificateInStore function to
retrieve the client certificate from a pkcs12 certificate store.
However, when called with the CERT_FIND_ANY flag, this function does not
provide any guarantees on the order in which certificates are retrieved.
If a pkcs12 file contains an entire certificate chain instead of a
single client certificate, the CertFindCertificateInStore function may
return the CA or an intermediate certificate instead of the desired
client certificate. Since there is no associated private key for such a
certificate, the TLS handshake fails.
With this change, we now pass the CERT_FIND_HAS_PRIVATE_KEY flag. This
ensures that the CertFindCertificateInStore function will return a
certificate which has a corresponding private key. This will stop the CA
and intermediate certificates from being selected. I don't think there
would be much use in a client certificate which has no associated
private key, so this should ensure the client certificate is selected. I
suppose it may be possible for a pkcs12 file to contain multiple
certificates with private keys and the new behaviour may not guarantee
which is selected. However, this is no worse that the previous behaviour
in which any certificate may been selected.
The CERT_FIND_HAS_PRIVATE_KEY is only available in Windows 8 / Server
2012 (aka Windows NT6.2). For older versions, we will fall back to using
the CERT_FIND_ANY flag.
Closes#16825
wolfSSL headers publish the `HAVE_ALPN` macro to tell if it has ALPN
support compiled in. Use that instead of `HAS_ALPN`, which was never
set.
Follow-up to edd573d980#16167Closes#17056
by including headers using "../[header]" when done from C files in
subdirectories, we do not need to specify the lib source dir as an
include path and we reduce the risk of header name collisions with
headers in the SDK using the same file names.
Idea-by: Kai Pastor
Ref: #16949Closes#16991
Curl_hexbyte - output a byte as a two-digit ASCII hex number
Curl_hexval - convert an ASCII hex digit to its binary value
... instead of duplicating similar code and hexdigit strings in numerous
places.
Closes#16888
The condition required to reach this call could not happen, because
cf_ssl_scache_get() already checks the same condition and returns NULL
for 'scache' prior to this.
Found by CodeSonar
Closes#16896
With rustls-ffi 0.15+ we can set up a callback for writing TLS secrets
hooked up to call Curl_tls_keylog_write. To make sure the associated
file is cleaned up we update the Curl_ssl struct for the rustls-ffi vtls
backend to have a cleanup callback.
Closes#16828
- Return 0 for password length if OpenSSL is expecting a certificate
password but the user did not provide one.
Prior to this change libcurl would crash if OpenSSL called the certificate
password callback in libcurl but no password was provided (NULL).
Reported-by: Roman Zharkov
Fixes https://github.com/curl/curl/issues/16806
Closes https://github.com/curl/curl/pull/16807
While adding support for key blobs, a check and error code update moved
after some logic, resulting in the updated code not checked anymore.
Detected by clang-tidy:
```
lib/vtls/mbedtls.c:768:7: error: Value stored to 'ret' is never read [clang-analyzer-deadcode.DeadStores,-warnings-as-errors]
768 | ret = MBEDTLS_ERR_PK_TYPE_MISMATCH;
| ^
```
Ref: https://github.com/curl/curl/actions/runs/13953249156/job/39057979349?pr=16764#step:12:178
Regression from 05e0453050#7157
Cherry-picked from #16764Closes#16766
Allowing 4GB on a 32-bit system is just asking for problems and could in
theory cause integer overflow in the dynbuf code.
The dynbuf now has an assert to catch code trying to set a max larger
than half SIZE_T_MAX.
Reported-by: Rinku Das
Closes#16716
Before this patch, standard `E*` errno codes were redefined on Windows,
onto matching winsock2 `WSA*` error codes, which have different values.
This broke uses where using the `E*` value in non-socket context, or
other places expecting a POSIX `errno`, e.g. file I/O, threads, IDN or
interfacing with dependencies.
Fix it by introducing a curl-specific `SOCKE*` set of macros that map to
`WSA*` on Windows and standard POSIX codes on other platforms. Then
verify and update the code to use `SOCKE*` or `E*` macro depending on
context.
- Add `SOCKE*` macros that map to either winsock2 or POSIX error codes.
And use them with `SOCKERRNO` or in contexts requiring
platform-dependent socket error codes.
This fixes `E*` uses which were supposed be POSIX values, not `WSA*`
socket errors, on Windows:
- lib/curl_multibyte.c
- lib/curl_threads.c
- lib/idn.c
- lib/vtls/gtls.c
- lib/vtls/rustls.c
- src/tool_cb_wrt.c
- src/tool_dirhie.c
- Ban `E*` codes having a `SOCKE*` mapping, via checksrc.
Authored-by: Daniel Stenberg
- Add exceptions for `E*` codes used in file I/O, or other contexts
requiring POSIX error codes.
Also:
- ftp: fix missing `SOCKEACCES` mapping for Windows.
- add `SOCKENOMEM` for `Curl_getaddrinfo()` via `asyn-thread.c`.
- tests/server/sockfilt: fix to set `SOCKERRNO` in local `select()`
override on Windows.
- lib/inet_ntop: fix to return `WSAEINVAL` on Windows, where `ENOSPC` is
used on other platforms. To simulate Windows' built-in `inet_ntop()`,
as tested on a Win10 machine.
Note:
- WINE returns `STATUS_INVALID_PARAMETER` = `0xC000000D`.
- Microsoft documentation says it returns `WSA_INVALID_PARAMETER`
(= `ERROR_INVALID_PARAMETER`) 87:
https://learn.microsoft.com/windows/win32/api/ws2tcpip/nf-ws2tcpip-inet_ntop#return-value
- lib/inet_ntop: drop redundant `CURL_SETERRNO(ENOSPC)`.
`inet_ntop4()` already sets it before returning `NULL`.
- replace stray `WSAEWOULDBLOCK` with `USE_WINSOCK` macro to detect
winsock2.
- move existing `SOCKE*` mappings from `tests/server` to
`curl_setup_once.h`.
- add missing `EINTR`, `EINVAL` constants for WinCE.
Follow-up to abf80aae38#16612
Follow-up to d69425ed7d#16615
Bug: https://github.com/curl/curl/pull/16553#issuecomment-2704679377Closes#16621
The issues found fell into these categories, with the applied fixes:
- const was accidentally stripped.
Adjust code to not cast or cast with const.
- const/volatile missing from arguments, local variables.
Constify arguments or variables, adjust/delete casts. Small code
changes in a few places.
- const must be stripped because an API dependency requires it.
Strip `const` with `CURL_UNCONST()` macro to silence the warning out
of our control. These happen at API boundaries. Sometimes they depend
on dependency version, which this patch handles as necessary. Also
enable const support for the zlib API, using `ZLIB_CONST`. Supported
by zlib 1.2.5.2 and newer.
- const must be stripped because a curl API requires it.
Strip `const` with `CURL_UNCONST()` macro to silence the warning out
of our immediate control. For example we promise to send a non-const
argument to a callback, though the data is const internally.
- other cases where we may avoid const stripping by code changes.
Also silenced with `CURL_UNCONST()`.
- there are 3 places where `CURL_UNCONST()` is cast again to const.
To silence this type of warning:
```
lib/vquic/curl_osslq.c:1015:29: error: to be safe all intermediate
pointers in cast from 'unsigned char **' to 'const unsigned char **'
must be 'const' qualified [-Werror=cast-qual]
lib/cf-socket.c:734:32: error: to be safe all intermediate pointers in
cast from 'char **' to 'const char **' must be 'const' qualified
[-Werror=cast-qual]
```
There may be a better solution, but I couldn't find it.
These cases are handled in separate subcommits, but without further
markup.
If you see a `-Wcast-qual` warning in curl, we appreciate your report
about it.
Closes#16142
Seen with downloaded mingw 7.3.0 when built against MSYS2 mbedTLS 3.6.2:
```
lib/vtls/cipher_suite.c: In function 'cs_zip_to_str':
lib/vtls/cipher_suite.c:789:16: error: conversion to 'uint8_t {aka unsigned char}' from 'int' may alter its value [-Werror=conversion]
indexes[1] = ((zip[0] << 4) & 0x3F) | zip[1] >> 4;
^
lib/vtls/cipher_suite.c:790:16: error: conversion to 'uint8_t {aka unsigned char}' from 'int' may alter its value [-Werror=conversion]
indexes[2] = ((zip[1] << 2) & 0x3F) | zip[2] >> 6;
^
lib/vtls/cipher_suite.c:793:16: error: conversion to 'uint8_t {aka unsigned char}' from 'int' may alter its value [-Werror=conversion]
indexes[5] = ((zip[3] << 4) & 0x3F) | zip[4] >> 4;
^
lib/vtls/cipher_suite.c:794:16: error: conversion to 'uint8_t {aka unsigned char}' from 'int' may alter its value [-Werror=conversion]
indexes[6] = ((zip[4] << 2) & 0x3F) | zip[5] >> 6;
^
```
Ref: https://github.com/curl/curl/actions/runs/13719756989/job/38372409927?pr=16429#step:10:21
Cherry-picked from #16429Closes#16614
Before this patch `--http2` did not work in gcc builds with Secure
Transport, because ALPN relied on a compiler supporting the
`HAVE_BUILTIN_AVAILABLE` aka `__builtin_available()` feature. This
is clang-specific and missing from gcc (as of gcc v14).
Add support for ALPN and HTTP/2 when this compiler feature is missing.
Also drop test exceptions from GHA/macos in CI.
Follow-up to 092f6815c8
Ref: c349bd668c#14097 (issue 15.)
Ref: #4314Closes#16581
No longer ignore the `--ciphers` argument in gnutls curl builds, but use
it to set the gnutls priority string.
When the set ciphers start with '+', '-' or '!', it is *appended* to the
curl generated priority string. Otherwise it replaces the curl one
completely.
Add test_17_18 to check various combinations.
Closes#16557
Further testing with timeouts in event based processing revealed that
our current shutdown handling in the connection pool was not clear
enough. Graceful shutdowns can only happen inside a multi handle and it
was confusing to track in the code which situation actually applies. It
seems better to split the shutdown handling off and have that code
always be part of a multi handle.
Add `cshutdn.[ch]` with its own struct to maintain connections being
shut down. A `cshutdn` always belongs to a multi handle and uses that
for socket/timeout monitoring.
The `cpool`, which can be part of a multi or share, either passes
connections to a `cshutdn` or terminates them with a one-time, best
effort.
Add an `admin` easy handle to each multi and share. This is used to
perform all maintenance operations where no "real" easy handle is
available. This solves the problem that the multi admin handle requires
some additional initialisation (e.g. timeout list).
The share needs its admin handle as it is often cleaned up when no other
transfer or multi handle exists any more. But we need a `data` in almost
every call.
Fix file:// handling of errors when adding a new connection to the pool.
Changes in `curl` itself:
- for parallel transfers, do not set a connection pool in the share,
rely on the multi's connection pool instead. While not a requirement
for the new `cshutdn` to work, this is
a) helpful in testing to trigger graceful shutdowns
b) a broader code coverage of libcurl via the curl tool
- on test_event with uv, cleanup the multi handle before returning from
parallel_event(). The uv struct is on the stack, cleanup of the multi
later will crash when it tries to register sockets. This is a "eat
your own dogfood" related fix.
Closes#16508
Fixes#16249
Forwarded-to-us-by: Carlos Henrique Lima Melara
Always use `gnutls_certificate_set_x509_key_file2()` for loading keys
and certificates, even without a password, since this function support
pkcs11 urls.
Thanks to @tatsuhiro-t for finding this out.
Help-by: Tatsuhiro Tsujikawa
Closes#16472
- GHA/windows/WinCE:
- set `-O3 -DNDEBUG` C flags manually for the CMake mingw32ce build.
CMake doesn't recognize the platform and fails to add them. To match
autotools (using `-O2`), and hit similar compiler warnings.
- enable parallel builds for cmake.
- tune parallelism for cmake using unity batches.
- tune parallelism for autotools.
Follow-up to 2a292c3984#15975
- tests: fix potentially uninitialized value in `readline()` in
`getpart.c`. Detected by gcc 4.4.0 `-O2` (Windows CE) jobs:
```
tests/server/getpart.c: In function 'getpart':
tests/server/getpart.c:298: error: 'datalen' may be used uninitialized in this function
```
Ref: https://github.com/curl/curl/actions/runs/13522595237/job/37785147505?pr=16476#step:11:25
Follow-up to 592880a3ca
- vtls_scache: rework returning pointer to avoid compiler warning seen
with `-O3` gcc 4.4.0 builds (Windows CE/schannel):
```
lib/vtls/schannel.c: In function 'schannel_connect_step1':
lib/vtls/vtls_scache.c:975: error: dereferencing pointer 'old_cred.4474' does break strict-aliasing rules
lib/vtls/vtls_scache.c:985: error: dereferencing pointer 'old_cred.4474' does break strict-aliasing rules
lib/vtls/schannel.c:959: note: initialized from here
```
Ref: https://github.com/curl/curl/actions/runs/13523868335/job/37789610845#step:9:25
Follow-up to fa0ccd9f1f#15774Closes#16476
- Show verbose message if the CA native import option is set but
the wolfSSL build does not support it.
wolfSSL has to be built with WOLFSSL_SYS_CA_CERTS to import native
CA certificates and that may not be common.
Closes https://github.com/curl/curl/pull/16417
This allows you to use the `certs` and `num_certs` writeout variables in
the curl tool, and getting information about the server certificates
using CURLINFO_CERTINFO.
Closes#16459
Enable TLS Early Data for wolfSSL:
- merge WOLFSSL_CTX and WOLFSSL setup from ngtcp2 with the general
implemenation in wolfssl.c
- enable for QUIC via ngtcp2
- give Curl_vquic_tls_init() a `struct alpn_spec` like used for the TCP
case. Adapt gnutls and other users.
- enable pytest test cases for early data with wolfSSL
and while this messes up wolfssl.c anyway, do
- rename all struct/functions with prefix 'wolfssl_' to 'wssl_' to not
pollute that name prefix
- rename `ctx/handle` to `ssl_ctx/ssl`, as used in openssl case
Closes#16167
When running curl event based, connect attempts stalled as the 'done'
check was using the wrong state in gnutls.
Add event based pytest runs to all http3 jobs and the openssl and
mbedtls ones on linux.
Closes#16423
Make it possible to build curl for Windows CE using the CeGCC toolchain.
With both CMake and autotools, including tests and examples, also in CI.
The build configuration is the default one with Schannel enabled. No
3rd-party dependencies have been tested.
Also revive old code to make Schannel build with Windows CE, including
certificate verification.
Builds have been throughougly tested. But, I've made no functional tests
for this PR. Some parts (esp. file operations, like truncate and seek)
are stubbed out and likely broken as a result. Test servers build, but
they do not work on Windows CE. This patch substitutes `fstat()` calls
with `stat()`, which operate on filenames, not file handles. This may or
may not work and/or may not be secure.
About CeGCC: I used the latest available macOS binary build v0.59.1
r1397 from 2009, in native `mingw32ce` build mode. CeGCC is in effect
MinGW + GCC 4.4.0 + old/classic-mingw Windows headers. It targets
Windows CE v3.0 according to its `_WIN32_WCE` value. It means this PR
restores portions of old/classic-mingw support. It makes the Windows CE
codepath compatible with GCC 4.4.0. It also adds workaround for CMake,
which cannot identify and configure this toolchain out of the box.
Notes:
- CMake doesn't recognize CeGCC/mingw32ce, necessitating tricks as seen
with Amiga and MS-DOS.
- CMake doesn't set `MINGW` for mingw32ce. Set it and `MINGW32CE`
manually as a helper variable, in addition to `WINCE` which CMake sets
based on `CMAKE_SYSTEM_NAME`.
- CMake fails to create an implib for `libcurl.dll`, due to not
recognizing the platform as a Windowsy one. This patch adds the
necessary workaround to make it work.
- headers shipping with CeGCC miss some things curl needs for Schannel
support. Fixed by restoring and renovating code previously deleted
old-mingw code.
- it's sometime non-trivial to figure out if a fallout is WinCE,
mingw32ce, old-mingw, or GCC version-specific.
- WinCE is always Unicode. With exceptions: no `wmain`,
`GetProcAddress()`.
- `_fileno()` is said to convert from `FILE *` to `void *` which is
a Win32 file `HANDLE`. (This patch doesn't use this, but with further
effort it probably could be.)
https://stackoverflow.com/questions/3989545/how-do-i-get-the-file-handle-from-the-fopen-file-structure
- WinCE has no signals, current directory, stdio/CRT file handles, no
`_get_osfhandle()`, no `errno`, no `errno.h`. Some of this stuff is
standard C89, yet missing from this platform. Microsoft expects
Windows CE apps to use Win32 file API and `FILE *` exclusively.
- revived CeGCC here (not tested for this PR):
https://building.enlyze.com/posts/a-new-windows-ce-x86-compiler-in-2024/
On `UNDER_CE` vs. `_WIN32_WCE`: (This patch settled on `UNDER_CE`)
- A custom VS2008 WinCE toolchain does not set any of these.
The compiler binaries don't contain these strings, and has no compiler
option for targeting WinCE, hinting that a vanilla toolchain isn't
setting any of them either.
- `UNDER_CE` is automatically defined by the CeGCC compiler.
https://cegcc.sourceforge.net/docs/details.html
- `UNDER_CE` is similar to `_WIN32`, except it's not set automatically
by all compilers. It's not supposed to have any value, like a version.
(Though e.g. OpenSSL sets it to a version)
- `_WIN32_WCE` is the CE counterpart of the non-CE `_WIN32_WINNT` macro.
That does return the targeted Windows CE version.
- `_WIN32_WCE` is not defined by compilers, and relies on a header
setting it to a default, or the build to set it to the desired target
version. This is also how `_WIN32_WINNT` works.
- `_WIN32_WCE` default is set by `windef.h` in CeGCC.
- `_WIN32_WCE` isn't set to a default by MSVC Windows CE headers (the
ones I checked at least).
- CMake sets `_WIN32_WCE=<ver>`, `UNDER_CE`, `WINCE` for MSVC WinCE.
- `_WIN32_WCE` seems more popular in other projects, including CeGCC
itself. `zlib` is a notable exception amongst curl dependencies,
which uses `UNDER_CE`.
- Since `_WIN32_WCE` needs "certain" headers to have it defined, it's
undefined depending on headers included beforehand.
- `curl/curl.h` re-uses `_WIN32_WCE`'s as a self-guard, relying on
its not-(necessarily)-defined-by-default property:
25b445e479/include/curl/curl.h (L77)
Toolchain downloads:
- Windows:
https://downloads.sourceforge.net/cegcc/cegcc/0.59.1/cegcc_mingw32ce_cygwin1.7_r1399.tar.bz2
- macOS Intel:
https://downloads.sourceforge.net/cegcc/cegcc/0.59.1/cegcc_mingw32ce_snowleopard_r1397.tar.bz2Closes#15975
curl detects Vista as part of its global initialization. Use that result
instead of detecting it again in Schannel, to save some cycles and API
calls.
Follow-up to 46e97b10ba#16400Closes#16408
- Do not set x509_store_setup until after all CA certificate imports.
- If CA blob import fails then return error unconditionally.
This is to sync the behavior with OpenSSL and other backends.
Prior to this change, setting CA native and/or CA blob were treated as
mutually exclusive from setting CA file and/or CA path.
CA blob import still overrides CA file import as documented.
Closes https://github.com/curl/curl/pull/16391
Add a 'wanted' major HTTP version bitmask next to the 'allowed' bitmask
in HTTP version negotiation. This will try connections as specified in
'wanted', but enabled Alt-Svc and HTTPS-RR to redirect to other major
HTTP versions, if those are 'allowed'.
Changes libcurl internal default to `CURL_HTTP_VERSION_NONE` and removes
the code in curl that sets `CURL_HTTP_VERSION_2TLS` if the command line
does not say anything else.
Closes#16117
Remove `blocking` argument from cfilter's connect method.
Implement blocking behaviour in Curl_conn_connect() instead for all
filter chains.
Update filters implementations. Several of which did never use the
paramter (QUIC for example). Simplifies connect handling in TLS filters
that no longer need to loop
Fixed a blocking connect call in FTP when waiting on a socket accept()
which only worked because the filter did not implement it.
Closes#16397
ALPN requires mingw-w64 9.0 or newer.
Also fix ALPN-enabled builds for UWP. This assumes that WINE doesn't
support UWP, which seems to be the case when writing this.
Closes#16385
