- Split WINSOCK and POSIX code in `multi_wait()` as the ifdef'ery
was becoming unreadable
- define `ENABLE_WAKEUP` to mean the wakeup socketpair is enabled,
no additional USE_WINSOCK check needed. Under WINSOCK
`ENABLE_WAKEUP` is not defined, so it's availability is as before
under the double defined() checks
- When the multi handle has "alive" transfers, the admin handle's
pollset include the wakeup receive socket. This results in the
admin handle running when someone uses `curl_multi_wakeup()`.
- Without any "alive" transfers, the wakeup socket is removed from
the pollset. Otherwise, event based processing would never finish,
eg. leave the event loop.
- The wakeup socket was never registered for event processing before,
e.g. `curl_multi_wakeup()` never worked in that mode.
- Adjust test exepectations on socket callback invocations and
number of sockets appearing in waitfds sets.
Closes#20832
OpenSSL records its peer verification status inside its SSL_SESSION
objects. When a session is later reused, the SSL connection inherits
this verify status.
Session keys prevent reuse of sessions between connections that verify
the peer and those who do not. However, when Apple SecTrust is used
to verify a connection, this does not update the Sessions verify
status (and there is no setter). On session reuse, OpenSSL fails
the verification and Apple SecTrust cannot verify either since the
certificate peer chain is not available.
Fix this by checking the verification status on session reuse and
remove the session again if the peer needs to be verified, but the
session is not.
Reported-by: Christian Schmitza
Fixes#20435Closes#20446
- fix test_17_20 flakiness: the test case did not have `nghttpx` in
its parameters, causing it to no check if a reload was necessary.
When that test ran behind one that gave nghttpx another certificate,
eg. in parallel mode, it used the wrong pinned pubkey.
- Have `env` provide lists of HTTP protocol versions available for
testing. Replace parameterized tests on a fixed protocol list with
the dynamic one from env. This makes checks for protocol availability
in the test function bodies superfluous.
refs #19489Closes#19540
To allow running pytests on more curl configurations.
Also delete a redundant H3 feature check from test_17_14_expired_cert.
Cherry-picked from #19407Closes#19412
Detect via curlinfo if curl has verbose strings disabled, and skip
tests that require it.
Also:
- cmake: make pytests depend on curlinfo.
Cherry-picked from #18797Closes#18801
libtests and clients were built the same way after recent overhauls.
libtests are used by runtests, clients by pytests.
Merge clients into libtests, aligning their entry function signature,
dropping common utility functions, and simplifying the build.
Note: After this patch `CURLDEBUG` applies to cli tests, when enabled.
Also:
- lib552: drop local copy-paste debug callback in favor of testtrace.
- lib552: drop local copy-paste dump function in favor of testtrace.
- clients: use `long` for HTTP version, drop casts.
- clients: replace local dump function in favor of testrace clone.
- sync cli test entry function prototype with libtests'.
- h2_serverpush: replace local trace callback with testtrace.
- de-duplicate 3 websocket close, ping, ping, functions. Kept the pong
iteration from `ws_pingpong`. Note: the pong clone in `lib2304` was
returning an error when `curl_ws_recv()` returned non-zero and
the payload matched the expected one anyway. After this patch, this
case returns success, as it does in `ws_pingpong`.
`lib2304` keeps passing, but I'm not sure if the previous behavior
was intentional.
- display full value in websocket close, ping, pong, drop casts.
Closes#18079
To make building the http client tests faster, with no duplication, by
using the build method that other test binaries already use.
The difference compared to other tests is that these don't use internal
libcurl headers or code. With the exception of `curl_config.h`, for
a feature macro.
Before this patch, these tests were built like examples.
Also:
- de-duplicate code and give unique names to colliding symbols.
- add local getopt implementation and enable all code for MSVC.
Adapted for curl via Public Domain source:
4e618ef782/getopt.h
Credits-to: Christopher Wellons
Thanks!
Closes#17627
Require now pytest-xdist from tests/http/requirements.txt and
run pytest in 'auto' parallel mode (counts cpu cores).
For CI runs, set the worker count to 4, overriding the
core count of 2 exposed in the images.
- use Filelock to generate allocated ports at start for all
workers and have subsequent workers just read the file and
take the ports for their slot
- make httpd config clearing a function fixture so every test
starts with a clean httpd config
- have fixture `configures_httpd` as parameter of test cases
that configure httpd anyway, saving one reload
- add pytest-xdist and filelock to required pyhton modules
- add installs to ruff CI
- give live checks waiting for a server to start up longer time
- add fixtures to tests that rely on a server
- do not stop servers unnecessarily. failures may not start them
properly again, leading to unexpected fails in whatever follows
- add a https: port to httpd that is *not* back by QUIC to allow
failover tests without stopping the QUIC server
Closes#17295
when CURLMOPT_MAX_HOST_CONNECTIONS or CURLMOPT_MAX_TOTAL_CONNECTIONS
limits are reached, force close connections in shutdown to go below
limit when possible.
Fixes#17020
Reported-by: Fujii Hironori
Closes#17022
Further testing with timeouts in event based processing revealed that
our current shutdown handling in the connection pool was not clear
enough. Graceful shutdowns can only happen inside a multi handle and it
was confusing to track in the code which situation actually applies. It
seems better to split the shutdown handling off and have that code
always be part of a multi handle.
Add `cshutdn.[ch]` with its own struct to maintain connections being
shut down. A `cshutdn` always belongs to a multi handle and uses that
for socket/timeout monitoring.
The `cpool`, which can be part of a multi or share, either passes
connections to a `cshutdn` or terminates them with a one-time, best
effort.
Add an `admin` easy handle to each multi and share. This is used to
perform all maintenance operations where no "real" easy handle is
available. This solves the problem that the multi admin handle requires
some additional initialisation (e.g. timeout list).
The share needs its admin handle as it is often cleaned up when no other
transfer or multi handle exists any more. But we need a `data` in almost
every call.
Fix file:// handling of errors when adding a new connection to the pool.
Changes in `curl` itself:
- for parallel transfers, do not set a connection pool in the share,
rely on the multi's connection pool instead. While not a requirement
for the new `cshutdn` to work, this is
a) helpful in testing to trigger graceful shutdowns
b) a broader code coverage of libcurl via the curl tool
- on test_event with uv, cleanup the multi handle before returning from
parallel_event(). The uv struct is on the stack, cleanup of the multi
later will crash when it tries to register sockets. This is a "eat
your own dogfood" related fix.
Closes#16508
Enable TLS Early Data for wolfSSL:
- merge WOLFSSL_CTX and WOLFSSL setup from ngtcp2 with the general
implemenation in wolfssl.c
- enable for QUIC via ngtcp2
- give Curl_vquic_tls_init() a `struct alpn_spec` like used for the TCP
case. Adapt gnutls and other users.
- enable pytest test cases for early data with wolfSSL
and while this messes up wolfssl.c anyway, do
- rename all struct/functions with prefix 'wolfssl_' to 'wssl_' to not
pollute that name prefix
- rename `ctx/handle` to `ssl_ctx/ssl`, as used in openssl case
Closes#16167
Rework the event based handling of transfers and connections to
be "localized" into a single source file with clearer dependencies.
- add multi_ev.c and multi_ev.h
- add docs/internal/MULTI-EV.md to explain the overall workings
- only do event handling book keeping when the socket callback
is set
- add handling for "connection only" event tracking, when internal
easy handles are used that are not really tied to a connection.
Used in connection pool.
- remove transfer member "last_poll" and connections "shutdown_poll"
and keep all that internal to multi_ev.c
- add CURL_TRC_M() for tracing of "multi" related things, including
event handling and connection pool operations. Add new trace
feature "multi" for trace config.
multi traces will show exactly what is going on in regard to
event handling.
- multi: trace transfers "mstate" in every CURL_TRC_M() call
- make internal trace buffer 2048 bytes and end the silliness
with +n here -m there. Adjust test 1652 expectations of resulting
length and input edge cases.
- add trace feature "lib-ids" to perfix libcurl traces with transfer
and connection ids. Useful for debugging libcurl applications.
Closes#16308
Once SSL_shutdown() has been called, OpenSSL does not really seem to
like it when it is called again and the other side has some finally data
to deliver.
Instead SSL_read() needs to be used solely, once the close notify has
been sent from curl's side.
Closes#15321
Fix issues detected.
Also:
- One of the `.vc` files used LF EOLs, while the other didn't.
Make that one also use LF EOLs, as this is apparently supported by
`nmake`.
- Drop `.dsw` and `.btn` types from `.gitattributes`.
The repository doesn't use them.
- Sync section order with the rest of files in
`tests/certs/EdelCurlRoot-ca.prm`.
- Indent/align `.prm` and `.pem` files.
- Delete dummy `[something]` section from `.prm` and `.pem` files.
Mental note:
MSVC `.sln` files seem to accept spaces for indentation and also support
LF line-endings. I cannot test this and I don't know what's more
convenient when updating them, so left them as-is, with specific
exclusions.
Closes#14031
When libcurl discards a connection there are two phases this may go
through: "shutdown" and "closing". If a connection is aborted, the
shutdown phase is skipped and it is closed right away.
The connection filters attached to the connection implement the phases
in their `do_shutdown()` and `do_close()` callbacks. Filters carry now a
`shutdown` flags next to `connected` to keep track of the shutdown
operation.
Filters are shut down from top to bottom. If a filter is not connected,
its shutdown is skipped. Notable filters that *do* something during
shutdown are HTTP/2 and TLS. HTTP/2 sends the GOAWAY frame. TLS sends
its close notify and expects to receive a close notify from the server.
As sends and receives may EAGAIN on the network, a shutdown is often not
successful right away and needs to poll the connection's socket(s). To
facilitate this, such connections are placed on a new shutdown list
inside the connection cache.
Since managing this list requires the cooperation of a multi handle,
only the connection cache belonging to a multi handle is used. If a
connection was in another cache when being discarded, it is removed
there and added to the multi's cache. If no multi handle is available at
that time, the connection is shutdown and closed in a one-time,
best-effort attempt.
When a multi handle is destroyed, all connection still on the shutdown
list are discarded with a final shutdown attempt and close. In curl
debug builds, the environment variable `CURL_GRACEFUL_SHUTDOWN` can be
set to make this graceful with a timeout in milliseconds given by the
variable.
The shutdown list is limited to the max number of connections configured
for a multi cache. Set via CURLMOPT_MAX_TOTAL_CONNECTIONS. When the
limit is reached, the oldest connection on the shutdown list is
discarded.
- In multi_wait() and multi_waitfds(), collect all connection caches
involved (each transfer might carry its own) into a temporary list.
Let each connection cache on the list contribute sockets and
POLLIN/OUT events it's connections are waiting for.
- in multi_perform() collect the connection caches the same way and let
them peform their maintenance. This will make another non-blocking
attempt to shutdown all connections on its shutdown list.
- for event based multis (multi->socket_cb set), add the sockets and
their poll events via the callback. When `multi_socket()` is invoked
for a socket not known by an active transfer, forward this to the
multi's cache for processing. On closing a connection, remove its
socket(s) via the callback.
TLS connection filters MUST NOT send close nofity messages in their
`do_close()` implementation. The reason is that a TLS close notify
signals a success. When a connection is aborted and skips its shutdown
phase, the server needs to see a missing close notify to detect
something has gone wrong.
A graceful shutdown of FTP's data connection is performed implicitly
before regarding the upload/download as complete and continuing on the
control connection. For FTP without TLS, there is just the socket close
happening. But with TLS, the sent/received close notify signals that the
transfer is complete and healthy. Servers like `vsftpd` verify that and
reject uploads without a TLS close notify.
- added test_19_* for shutdown related tests
- test_19_01 and test_19_02 test for TCP RST packets
which happen without a graceful shutdown and should
no longer appear otherwise.
- add test_19_03 for handling shutdowns by the server
- add test_19_04 for handling shutdowns by curl
- add test_19_05 for event based shutdowny by server
- add test_30_06/07 and test_31_06/07 for shutdown checks
on FTP up- and downloads.
Closes#13976
