On platforms where neither accept4 nor fcntl was available, an
EPRT connection did not send the accepted socket as non-blocking.
This became apparent when TLS was in use and the test receive
on shutdown did simply hang.
Reported-by: Denis Goleshchikhin
Fixes#19753Closes#19851
Address issues listed in #19770:
- allow for ngttpx to successfully shut down on last attempt that might
extend beyond the finish timestamp
- timeline checks: allos `time_starttransfer` to appear anywhere in
the timeline as a slow client might seen response data before setting
the other counters
- dump logs on test_05_02 as it was not reproduced locally
Fixes#19970Closes#19783
A fix for the tests that took the longest:
- test_05: make the server close the HTTP/1.1 connection when
simulating an error during a download. This eliminates waiting
for a keepalive timeout
- test_02: pause tests with slightly smaller documents, eliminate
special setup for HTTP/2. We test stream window handling now
elsewhere already
- cli_hx_download: run look in 500ms steps instead of 1sec, resuming
paused tranfers earlier.
Closes#19809
Make `port` member in these struct of type `uint16_t`.
add `uint8_t transport` to `struct ip_quadruple
Define TRNSPRT_NONE as 0. By assigning a valid transport only on a
successful connection, it is clear when the ip_quadruple members are
valid. Also, for transports not involving ports, the getinfos for
`CURLINFO_PRIMARY_PORT` and `CURLINFO_LOCAL_PORT` will now always return
-1.
Make all `transport` members and parameters of type `uint8_t`.
Document the return value of `CURLINFO_LOCAL_PORT` and
`CURLINFO_PRIMARY_PORT` in this regard. Add tests that writeout stats
report ports correctly.
Closes#19708
* move the TIMER_POSTQUEUE to the time a connection is chosen,
so that TIMER_NAMELOOKUP always happens afterwards
* client writer: do not trigger TIMER_STARTTRANSFER on CLIENTWRITE_INFO
as ftp and other pingpong protocols write that before starting anything
that is the tranfer itself
* Elimnating debug trancing of "closed stream/connection - bailing"
as confusing, as connection is not really closed on most cases.
* Setting 'data->req.upload_done` correctly, so that no "abort upload"
is happening at the end of a perfectly fine download.
* Adding test cases with up-/download of 0-length files.
* pytest: add a "timeline" of timer value checks to Resulst in curl.py,
so that this can be used in several test cases, replacing the local
stuff in test_16
* add timeline checks to ftp test cases
Closes#19269
Description of how this works in `docs/internal/RATELIMITS.ms`.
Notable implementation changes:
- KEEP_SEND_PAUSE/KEEP_SEND_HOLD and KEEP_RECV_PAUSE/KEEP_RECV_HOLD
no longer exist. Pausing is down via blocked the new rlimits.
- KEEP_SEND_TIMED no longer exists. Pausing "100-continue" transfers
is done in the new `Curl_http_perform_pollset()` method.
- HTTP/2 rate limiting implemented via window updates. When
transfer initiaiting connection has a ratelimit, adjust the
initial window size
- HTTP/3 ngtcp2 rate limitin implemnented via ack updates
- HTTP/3 quiche does not seem to support this via its API
- the default progress-meter has been improved for accuracy
in "current speed" results.
pytest speed tests have been improved.
Closes#19384
- adjust cipher list in infof() statement for min/max TLS version
- skip test_17_07 for wolfSSL 5.8.4 when CHACHA20 is negotiated
due to regression with homebrew build on ARM systems.
Fixes#19644
Reported-by: Viktor Szakats
Closes#19662
The parsing of the HTTP/1.1 formatted request into the h2/h3 header
structures should detect CURLOPT_CUSTOMREQUEST methods and forward them
correctly.
Add test_01_20 to verify
Fixes#19543
Reported-by: Omdahake on github
Closes#19563
There remain some false positives, hits in test data, and `dir` use,
around 100 issues in total.
There is no plan to enforce badwords on tests.
Also:
- badwords.txt: let a few `manpage[s]` occurrences through
(in Perl code).
Closes#19541
- fix test_17_20 flakiness: the test case did not have `nghttpx` in
its parameters, causing it to no check if a reload was necessary.
When that test ran behind one that gave nghttpx another certificate,
eg. in parallel mode, it used the wrong pinned pubkey.
- Have `env` provide lists of HTTP protocol versions available for
testing. Replace parameterized tests on a fixed protocol list with
the dynamic one from env. This makes checks for protocol availability
in the test function bodies superfluous.
refs #19489Closes#19540
The HTTP/3 tests did send 20 transfers against nghttpx with a backend
that failed the uploads with a 400 and an incomplete response body. This
causes stream resets.
Apache keeps the connection open, but newer nghttpx closes the front
connection after "too many" reset. When that bites, it depends on the
number of transfers ongoing how the test case fails. This led to flaky
outcomes.
Reduce the transfers to just a single one and check the result of
that one. Parallelism is not important here.
refs #19489Closes#19530
- allow 02_28 to run in HTTP/1.1 without H2 support again.
Follow-up to 3752de465d#19412
- fix to skip 02_28 for all protocols for curl without compression
support (either zlib, brotli or ztsd).
Closes#19458
To allow running pytests on more curl configurations.
Also delete a redundant H3 feature check from test_17_14_expired_cert.
Cherry-picked from #19407Closes#19412
A regression in curl 8.17.0 led to a customer CAPATH set by the
application (or the curl command) to be ignored unless licurl was built
with a default CAPATH.
Add test cases using `--capath` on the custom pytest CA, generated with
the help of the openssl command when available.
Fixes#19401
Reported-by: Brad King
Closes#19308
In pytests test_08 with the Caddy server, the new rate-limiting in
ngtcp2 did close the connection because it found "too many" stream data
packet repeats.
It is unclear if this is some Caddy issue or if the ngtcp2 implementaton
is wrong. Or if curl can do anything here.
Reported as https://github.com/ngtcp2/ngtcp2/issues/1850
This PR overwrites the ratelimit defaults in ngtcp2 with ten times
increased values. This makes the errors disappear on macOS.
Enable test_08_04/05 in CI again to see if there are any issues
to be found there. (We had those disabled before having parallel
pytests.)
Closes#19274
Change the test certificate to carry a altname 'dns:127.0.0.1' which
should *not* match in test_17_05_bad_ip_addr.
wolfSSL: since `wolfSSL_check_domain_name()` does not differentiate
between DNS and IP names, use if only for DNS names. For IP addresses,
get the peer certificate after the handshake and check that using
wolfSSL_X509_check_ip_asc().
Unfortunately, this succeeds where it should not, as wolfSSL internally
used the same check code for both cases. So, skip the test case until
wolfSSL fixes that.
Reported-by: Joshua Rogers
Closes#19252
- cryptography from 46.0.2 to 46.0.3 in /tests
- filelock from 3.19.1 to 3.20.0 in /tests
- psutil from 7.1.0 to 7.1.1 in /tests
Closes#19217Closes#19218Closes#19219
- honor request id (`id=<number>`) in `curltest/put` and
`curltest/sslinfo` handlers.
- do not truncate `max_upload` input parameter.
- delete unused variables.
- formatting.
Inspired by Joshua's report on tests.
Closes#19061
In MOST protocols and runs, the 'pretransfer' time is less than the
'starttransfer'. E.g. request being sent before response comes in.
However, when curl is starved of cpu a server response might start
streaming in before the multi-state transitioned to DID (and recorded
the 'pretransfer' time).
Do no longer check that 'pretransfer' is less or equal 'starttransfer'.
Check that is is less or equal to the total time instead.
Closes#19096
When calling scorecard with --flame to produce a flamegraph, use
"perf" on linux platforms to do the measurements. Update the scorecard
documentation about it.
Closes#19058
Protocol handlers not flagging PROTOPT_SSL that allow reuse of existing
SSL connections now need to carry the flag PROTOPT_SSL_REUSE.
Add PROTOPT_SSL_REUSE to imap, ldap, pop3, smtp and ftp.
Add tests the http: urls do not reuse https: connections and vice versa.
Reported-by: Sakthi SK
Fixes#19006Closes#19007
- cryptography from 44.0.1 to 46.0.2 in tests/http
- ruff from 0.13.2 to 0.14.0 in .github/scripts
- reuse from 6.0.0 to 6.1.2 in .github/scripts
- github/codeql-action from 3.30.5 to 4.30.7
Closes#18941Closes#18942Closes#18943Closes#18945Closes#18947
Add infrastructure to colled and dispatch notifications for transfers
and the multi handle in general. Applications can register a callback
and en-/disable notification type the are interested in.
Without a callback installed, notifications are not collected. Same when
a notification type has not been enabled.
Memory allocation failures on adding notifications lead to a general
multi failure state and result in CURLM_OUT_OF_MEMORY returned from
curl_multi_perform() and curl_multi_socket*() invocations.
Closes#18432
- configure/cmake support for enabling the option
- supported in OpenSSL and GnuTLS backends
- when configured, Apple SecTrust is the default trust store
for peer verification. When one of the CURLOPT_* for adding
certificates is used, that default does not apply.
- add documentation of build options and SSL use
Closes#18703
Detect via curlinfo if curl has verbose strings disabled, and skip
tests that require it.
Also:
- cmake: make pytests depend on curlinfo.
Cherry-picked from #18797Closes#18801
When attempts on all addresses have been started, do no longer set any
EXPIRE_HAPPY_EYEBALLS timeouts.
Fixes#18767
Reported-by: Johannes Schindelin
Closes#18768
Fail on errors from SSL_handle_events().
Force quit Caddy test instance that is left hanging longer with
openssl-quic tests for unknown reasons.
Reported in Joshua's sarif data
Closes#18720
- requirements.txt: shorten copyright headers.
- requirements.txt: pin packages to versions.
- GHA/windows: use `tests/requirements.txt`.
Pick a `cryptography` package version that satifies both `impacket`
and pytests dependencies.
- GHA/checksrc: move pip deps into a new `requirements.txt`.
To make Dependabot detect and bump them.
- GHA/checksrc: replace apt packages for python test deps with pip
install `tests/**/requirements.txt` to a venv.
- GHA/checksrc: use venv and drop `--break-system-packages`.
- GHA/linux: fix to actually activate venvs.
Follow-up to 2638570241#15578
- GHA/linux: fixup (did not cause an issue)
Follow-up to d75785c7de#18660
- GHA: create venvs later, simplify commands.
- GHA: sync pip command-line options, e.g. drop progress-bar,
everywhere.
Assisted-by: Dan Fandrich
Closes#18708
Access the error codes of ngtcp2 when a connect attempt failes. Trace
the information for analysis. Treat errors as permanent failure by
default, trigger retrying only when the server refused without
indicating an error.
Closes#18521
Tweaks around handling of --limit-rate:
* tracing: trace outstanding timeouts by name
* multi: do not mark transfer as dirty that have
an EXPIRE_TOOFAST set
* multi: have one static function to asses speed limits
* multi: when setting EXPIRE_TOOFAST remove the transfers
from the dirty set
* progress: rename vars and comment on how speed limit
timeouts are calculated, for clarity
* transfer: when speed limiting, exit the receive loop
after a quarter of the limit has been received, not
on the first chunk received.
* cf-ip-happy.c: clear EXPIRE_HAPPY_EYEBALLS on connect
* scorecard: add --limit-rate parameter to test with
speed limits in effect
Closes#18454
test_10_08, uploading larger files for a h2 proxy, sporadically fails
with a decrpytion error on received data in AWS-LC. The frequency can
be increased by simulated network receive blocks.
Not setting a 4 * TLS record sized buffer, leaving AWS-LC at its
default buffer size seems to mitigate this problem.
Closes#18434
Write out 9-length frames to client's WRITEFUNCTION
Read 0-length frames from READFUNCTION *if* the function
started a new frame via `curl_ws_start_frame()`.
Fixes#18286Closes#18332
Reported-by: Andriy Druk
Add support for CURLOPT_READFUNCTION with WebSocket urls when *not* in
connect-only mode, e.g. when using curl_multi_perform.
Install the callback function and set CURLOPT_UPLOAD. Return
CURL_READFUNC_PAUSE when having nothing more to send and unpause the
transfer when more data is ready.
This will send the read bytes in a WebSocket BINARY frame.
Add support for this mode in the pytest "ws_data" client and have all
tests run in 'curl_ws_send/recv' and 'peform' mode as well.
Add `curl_ws_start_frame()`. Document, cover in libcurl-ws.md and
explain the READFUNCTION mode for websockets.
Add example `websocket-updown` for this.
Closes#17683
When an SSL_write() blocks we need to retry it with the
same length as before or stupid OpenSSL freaks out. Remember
it, limit any longer sends and fail shorter ones.
Fixes#18121
Reported-by: adamse on github
Closes#18132
test_02_36 tests h1/h2/h3 with urls longer than 1/16/32/64K.
Protocols behave the same until the size exceed 64k when h2 frame limits
bite and h3 exhibits a different http status.
Failed attempt to reproduce #18121Closes#18129
Since the content varies during connection setup and while doing it
(eyeballing), remove these strcut from `connectdata` and replace use
with querying the connection filters. Those keep that information
already.
Change the info logging of established connections to also give the
local address and port.
Closes#17960
libtests and clients were built the same way after recent overhauls.
libtests are used by runtests, clients by pytests.
Merge clients into libtests, aligning their entry function signature,
dropping common utility functions, and simplifying the build.
Note: After this patch `CURLDEBUG` applies to cli tests, when enabled.
Also:
- lib552: drop local copy-paste debug callback in favor of testtrace.
- lib552: drop local copy-paste dump function in favor of testtrace.
- clients: use `long` for HTTP version, drop casts.
- clients: replace local dump function in favor of testrace clone.
- sync cli test entry function prototype with libtests'.
- h2_serverpush: replace local trace callback with testtrace.
- de-duplicate 3 websocket close, ping, ping, functions. Kept the pong
iteration from `ws_pingpong`. Note: the pong clone in `lib2304` was
returning an error when `curl_ws_recv()` returned non-zero and
the payload matched the expected one anyway. After this patch, this
case returns success, as it does in `ws_pingpong`.
`lib2304` keeps passing, but I'm not sure if the previous behavior
was intentional.
- display full value in websocket close, ping, pong, drop casts.
Closes#18079
- add startup check for 'danted' to avoid fails on low cpu
- rename 'sockd' to 'danted' everywhere to clarify what we use
- add proper defaults for 'danted' for debian
- install 'dante-server' in pytest ci runs
Closes#18075
Add a new commandline option --out-null that discards all
response bytes into the void. Replaces non-portable use of
'-o /dev/null' with more efficiency.
Feature added in 8.16.0
Closes#17800
