In OpenSSL < 3.0, the modularity was provided by mechanism called
"engines". This is supported in curl, but the engines got deprecated
with OpenSSL 3.0 in favor of more versatile providers.
This adds a support for OpenSSL Providers, to use PKCS#11 keys, namely
through the pkcs11 provider. This is done using similar approach as the
engines and this is automatically built in when the OpenSSL 3 and newer
is used.
Signed-off-by: Jakub Jelen <jjelen@redhat.com>
Closes#15587
When a QUIC TLS session announced early data support and
'CURLSSLOPT_EARLYDATA' is set for the transfer, send initial request and
body (up to the 128k we buffer) as 0RTT when curl is built with
ngtcp2+gnutls.
QUIC 0RTT needs not only the TLS session but the QUIC transport
paramters as well. Store those and the earlydata max value together with
the session in the cache.
Add test case for h3 use of this. Enable quic early data in nghttpx for
testing.
Closes#15667
Described in detail in internal doc TLS-SESSIONS.md
Main points:
- use a new `ssl_peer_key` for cache lookups by connection filters
- recognize differences between TLSv1.3 and other tickets
* TLSv1.3 tickets are single-use, cache can hold several of them for a peer
* TLSv1.2 are reused, keep only a single one per peer
- differentiate between ticket BLOB to store (that could be persisted) and object instances
- use put/take/return pattern for cache access
- remember TLS version, ALPN protocol, time received and lifetime of ticket
- auto-expire tickets after their lifetime
Closes#15774
For TLS backends that don't need these functions, they now use plain
NULL pointers instead of setting a function that does nothing.
Helps making it clearer that a specific TLS handler does not provide
anything specific for that action.
Closes#15772
- make colliding vtls static function names unique.
- wolfssl: stop including an unused compatibility header.
- cmake: adapt detection logic for openssl+wolfssl coexist.
- wolfssl: fix to use native wolfSSL API in ECH codepath.
- openssl+wolfssl: fix ECH code to coexist.
Requires a post wolfSSL v5.7.4, recent master for `OPENSSL_COEXIST`
feature, and `CPPFLAGS=-DOPENSSL_COEXIST`.
Ref: https://github.com/wolfSSL/wolfssl/issues/8194Closes#15596
Fix regression that no longer printed the error messages about expired
certificates in openssl. Add test case for openssl/gnutls/wolfssl.
Fixes#15612
Reported-by: hiimmat on github
Closes#15613
Previously there were three error situations that only added the (unhelpful)
OpenSSL error strings, now we prefix those with a short explanation for the
error situation.
Reported-by: Jeroen Ooms
Fixes#15473Closes#15474
That error code was introduced and has been used for OpenSSL ENGINE
things and not others, so switch the other use cases over to other TLS
related error codes.
Closes#15430
Use session cache for QUIC when built with quictls or wolfSSL.
Add test_017_10 for verifying QUIC TLS session reuse when built with
quictls, gnutls or wolfssl.
Closes#15358
Once SSL_shutdown() has been called, OpenSSL does not really seem to
like it when it is called again and the other side has some finally data
to deliver.
Instead SSL_read() needs to be used solely, once the close notify has
been sent from curl's side.
Closes#15321
Based on #14135, implement TLSv1.3 earlydata support for the curl
command line, libcurl and its implementation in GnuTLS.
If a known TLS session announces early data support, and the feature is
enabled *and* it is not a "connect-only" transfer, delay the TLS
handshake until the first request is being sent.
- Add --tls-earldata as new boolean command line option for curl.
- Add CURLSSLOPT_EARLYDATA to libcurl to enable use of the feature.
- Add CURLINFO_EARLYDATA_SENT_T to libcurl, reporting the amount of
bytes sent and accepted/rejected by the server.
Implementation details:
- store the ALPN protocol selected at the SSL session.
- When reusing the session and enabling earlydata, use exactly
that ALPN protocol for negoptiation with the server. When the
sessions ALPN does not match the connections ALPN, earlydata
will not be enabled.
- Check that the server selected the correct ALPN protocol for
an earlydata connect. If the server does not confirm or reports
something different, the connect fails.
- HTTP/2: delay sending the initial SETTINGS frames during connect,
if not connect-only.
Verification:
- add test_02_32 to verify earlydata GET with nghttpx.
- add test_07_70 to verify earlydata PUT with nghttpx.
- add support in 'hx-download', 'hx-upload' clients for the feature
Assisted-by: ad-chaos on github
Closes#15211
booleans should use the type 'bool' and set the value to TRUE/FALSE
non-booleans should not be 'bool' and should not set the value to
TRUE/FALSE
Closes#15123
If there is a (memory) error when creating the certinfo data, the code
would previously continue which could lead to a partial/broken response.
Now, the first error aborts and cleans up the entire thing.
A certinfo "collection" error is however still not considered an error
big enough to stop the handshake.
Bonus 1: made two functions static (and removed the Curl_ prefix) that
were not used outside of openssl.c
Bonus 2: removed the unused function Curl_ossl_set_client_cert
Closes#14780
The SSL_Session object is mutated during connection inside openssl,
and it might not be thread-safe. Besides, according to documentation
of openssl:
```
SSL_SESSION objects keep internal link information about the session
cache list, when being inserted into one SSL_CTX object's session
cache. One SSL_SESSION object, regardless of its reference count,
must therefore only be used with one SSL_CTX object (and the SSL
objects created from this SSL_CTX object).
```
If I understand correctly, it is not safe to share it even in a
single thread.
Instead, serialize the SSL_SESSION before adding it to the cache,
and deserialize it after retrieving it from the cache, so that no
concurrent write to the same object is infeasible.
Also
- add a ci test for thread sanitizer
- add a test for sharing ssl sessions concurrently
- avoid redefining memory functions when not building libcurl, but
including the soruce in libtest
- increase the concurrent connections limit in sws
Notice that there are fix for a global data race for openssl which
is not yet release. The fix is cherry pick for the ci test with
thread sanitizer.
d8def79838Closes#14751
It could previously be set with configure/cmake and used in rare cases
for reading randomness: with ancient mbedTLS or rustls without
arc4random.
We now get randomness in this order:
1. The TLS library's way to provide random
2. On Windows: Curl_win32_random
3. if arc4random exists, use that
4. weak non-crytographically strong pseudo-random
Closes#14749
When using CMake Unity build.
- use unique name for `set_ssl_version_min_max()`
Fixes collision between GnuTLS, mbedTLS and SecureTransport.
```
lib\vtls\mbedtls.c(317,1): error C2084: function 'CURLcode set_ssl_version_min_max(Curl_easy *,ssl_peer *,ssl_primary_config *,const char **,const char *)' already has a body
lib\vtls\mbedtls.c(837,49): warning C4133: 'function': incompatible types - from 'Curl_cfilter *' to 'Curl_easy *'
lib\vtls\mbedtls.c(837,53): warning C4133: 'function': incompatible types - from 'Curl_easy *' to 'ssl_peer *'
lib\vtls\mbedtls.c(837,25): error C2198: 'set_ssl_version_min_max': too few arguments for call
```
- use unique name for `do_file_type()`
Fixes collision between GnuTLS, OpenSSL and wolfSSL.
```
lib\vtls\openssl.c(1053,12): error C2084: function 'gnutls_x509_crt_fmt_t do_file_type(const char *)' already has a body
```
Ref: https://github.com/curl/curl/actions/runs/10341162641/job/28622681573?pr=14484#step:10:31
Cherry-picked from #14495Closes#14516
Channel Bindings are used to tie the session context to a specific TLS
channel. This is to provide additional proof of valid identity,
mitigating authentication relay attacks.
Major web servers have the ability to require (None/Accept/Require)
GSSAPI channel binding, rendering Curl unable to connect to such
websites unless support for channel bindings is implemented.
IIS calls this feature Extended Protection (EPA), which is used in
Enterprise environments using Kerberos for authentication.
This change require krb5 >= 1.19, otherwise channel bindings won't be
forwarded through SPNEGO.
Co-Authored-By: Steffen Kieß <947515+steffen-kiess@users.noreply.github.com>
Closes#13098
Make sure that `io_need` is cleared and set at the filter operations.
Add some more tracing for shutdown situations.
Improve shutdown handling for blocked sends. OpenSSL is a bit tricksy
here that it only reports WANT_WRITE on SSL_shutdown(), but never on
SSL_read() on blocked sends. So we need to use both.
At last, set SSL_MODE_ACCEPT_MOVING_WRITE_BUFFER when available since we
are not always retrying sends from the very same address, as testing
showed.
Closes#14375
Adds a `bool eos` flag to send methods to indicate that the data
is the last chunk the invovled transfer wants to send to the server.
This will help protocol filters like HTTP/2 and 3 to forward the
stream's EOF flag and also allow to EAGAIN such calls when buffers
are not yet fully flushed.
Closes#14220
Replace Curl_resolv_unlock() with Curl_resolv_unlink():
-replace inuse member with refcount in Curl_dns_entry
- pass Curl_dns_entry ** to unlink, so it gets always cleared
- solve potential (but unlikley) UAF in FTP's handling of looked up
Curl_dns_entry. Esp. do not use addr information after unlinking an entry.
In reality, the unlink will not free memory, as the dns entry is still
referenced by the hostcache. But this is not safe and relying on no other
code pruning the cache in the meantime.
- pass permanent flag when adding a dns entry instead of fixing timestamp
afterwards.
url.c: fold several static *resolve_* functions into one.
Closes#14195
Adds a `bool eos` flag to send methods to indicate that the data is the
last chunk the invovled transfer wants to send to the server.
This will help protocol filters like HTTP/2 and 3 to forward the
stream's EOF flag and also allow to EAGAIN such calls when buffers are
not yet fully flushed.
Closes#14220
- deduplicate the code in many tls backends that check
for an existing id and delete it before adding the new one
- rename ssl_primary_config's `sessionid` bool to `cache_session`
Closes#14121
Based on the standards and guidelines we use for our documentation.
- expand contractions (they're => they are etc)
- host name = > hostname
- file name => filename
- user name = username
- man page => manpage
- run-time => runtime
- set-up => setup
- back-end => backend
- a HTTP => an HTTP
- Two spaces after a period => one space after period
Closes#14073
When libcurl discards a connection there are two phases this may go
through: "shutdown" and "closing". If a connection is aborted, the
shutdown phase is skipped and it is closed right away.
The connection filters attached to the connection implement the phases
in their `do_shutdown()` and `do_close()` callbacks. Filters carry now a
`shutdown` flags next to `connected` to keep track of the shutdown
operation.
Filters are shut down from top to bottom. If a filter is not connected,
its shutdown is skipped. Notable filters that *do* something during
shutdown are HTTP/2 and TLS. HTTP/2 sends the GOAWAY frame. TLS sends
its close notify and expects to receive a close notify from the server.
As sends and receives may EAGAIN on the network, a shutdown is often not
successful right away and needs to poll the connection's socket(s). To
facilitate this, such connections are placed on a new shutdown list
inside the connection cache.
Since managing this list requires the cooperation of a multi handle,
only the connection cache belonging to a multi handle is used. If a
connection was in another cache when being discarded, it is removed
there and added to the multi's cache. If no multi handle is available at
that time, the connection is shutdown and closed in a one-time,
best-effort attempt.
When a multi handle is destroyed, all connection still on the shutdown
list are discarded with a final shutdown attempt and close. In curl
debug builds, the environment variable `CURL_GRACEFUL_SHUTDOWN` can be
set to make this graceful with a timeout in milliseconds given by the
variable.
The shutdown list is limited to the max number of connections configured
for a multi cache. Set via CURLMOPT_MAX_TOTAL_CONNECTIONS. When the
limit is reached, the oldest connection on the shutdown list is
discarded.
- In multi_wait() and multi_waitfds(), collect all connection caches
involved (each transfer might carry its own) into a temporary list.
Let each connection cache on the list contribute sockets and
POLLIN/OUT events it's connections are waiting for.
- in multi_perform() collect the connection caches the same way and let
them peform their maintenance. This will make another non-blocking
attempt to shutdown all connections on its shutdown list.
- for event based multis (multi->socket_cb set), add the sockets and
their poll events via the callback. When `multi_socket()` is invoked
for a socket not known by an active transfer, forward this to the
multi's cache for processing. On closing a connection, remove its
socket(s) via the callback.
TLS connection filters MUST NOT send close nofity messages in their
`do_close()` implementation. The reason is that a TLS close notify
signals a success. When a connection is aborted and skips its shutdown
phase, the server needs to see a missing close notify to detect
something has gone wrong.
A graceful shutdown of FTP's data connection is performed implicitly
before regarding the upload/download as complete and continuing on the
control connection. For FTP without TLS, there is just the socket close
happening. But with TLS, the sent/received close notify signals that the
transfer is complete and healthy. Servers like `vsftpd` verify that and
reject uploads without a TLS close notify.
- added test_19_* for shutdown related tests
- test_19_01 and test_19_02 test for TCP RST packets
which happen without a graceful shutdown and should
no longer appear otherwise.
- add test_19_03 for handling shutdowns by the server
- add test_19_04 for handling shutdowns by curl
- add test_19_05 for event based shutdowny by server
- add test_30_06/07 and test_31_06/07 for shutdown checks
on FTP up- and downloads.
Closes#13976
This adds connection shutdown infrastructure and first use for FTP. FTP
data connections, when not encountering an error, are now shut down in a
blocking way with a 2sec timeout.
- add cfilter `Curl_cft_shutdown` callback
- keep a shutdown start timestamp and timeout at connectdata
- provide shutdown timeout default and member in
`data->set.shutdowntimeout`.
- provide methods for starting, interrogating and clearing
shutdown timers
- provide `Curl_conn_shutdown_blocking()` to shutdown the
`sockindex` filter chain in a blocking way. Use that in FTP.
- add `Curl_conn_cf_poll()` to wait for socket events during
shutdown of a connection filter chain.
This gets the monitoring sockets and events via the filters
"adjust_pollset()" methods. This gives correct behaviour when
shutting down a TLS connection through a HTTP/2 proxy.
- Implement shutdown for all socket filters
- for HTTP/2 and h2 proxying to send GOAWAY
- for TLS backends to the best of their capabilities
- for tcp socket filter to make a final, nonblocking
receive to avoid unwanted RST states
- add shutdown forwarding to happy eyeballers and
https connect ballers when applicable.
Closes#13904
