This version is already two releases out of support, but is "only" 7
years old so is probably still being used in the real world. Document
this version along with some other testing dependencies. Remove code
support for earlier versions. Disable ruff checks that need a newer
version.
Mark Python mutable class variables with ClassVar, to denote that the
danger this can cause has been considered. Since any change made to
these in any object affects all other objects, this can cause locality
errors. However, as used in the test suite, they are are never modified
and so they are annotated as being intended.
Always set a timezone in datetime objects, as mixing naive and
timezone-aware object can cause errors.
These fix ruff rules DTZ005, RUF012.
* call super() without arguments
* mark an unused variable as such
* simplify by using dict getter for default values
* use writelines() when possible
* use dedent to simplify some text formatting
* avoid items() on dict in a loop when unnecessary
* replace most Python format() calls with f-strings
* use capture_output in subprocess.run
This fixes ruff rules FLY002, FURB122, PERF102, RUF059, SIM401, UP008,
UP022, UP030.
* combine separate if statements
* remove an unneeded encode() call
* remove unneeded returns
* simplify code when returning early
This fixes ruff rules SIM102, SIM114, UP012, PLR1711.
* remove an unneeded ruff warning disable
* remove coding: utf-8 from Python code; PEP 3120 makes UTF-8 the
default encoding
* remove unusable shebang lines from Python code
* remove empty print strings
* disable warnings when file objects are stored; these instances can't
be handled with context managers
* use more consistent whitespace in Python code, fixing flake8 warnings
* set the executable bit on scorecard.py, making it easier to run
These fix ruff rules EXE001, FURB105, UP009, SIM115.
* Explicitly set "check" in subprocess.run() to raise an exception
automatically, where it was done manually before
* Use contextlib.suppress to ignore exceptions
* Use custom exceptions for test errors for clarity and flexibility.
* Replace IOError with OSError
This fixes ruff rules BLE001, PLW1510, S110, TRY201, TRY203, TRY002,
UP024.
- make `const` data `static`, where missing and possible.
- replace `strlen()` on literal or const strings with `sizeof()`.
While the latter is optimized by popular C compiler, e.g. MSVC only
does it with `/O2`.
- replace magic numbers with `sizeof()`, where missing.
- introduce `CURL_CSTRLEN()` macro for `sizeof(char[]) - 1`.
- use `CURL_CSTRLEN()` macro.
- move `const` before integer types, where missing.
- replace `char *var` with `var[]`, where missing and possible.
- use double const, where missing.
`static const char *` -> `static const char * const`.
- lib1514: constify pointers.
- unit3205: drop redundant cast, avoid another one.
- unit1666: map `OID()` macro to identical `STRCONST()`.
Closes#22406
Forward only the app error code from the receiving side to the h3 layer.
This only takes effect when building against ngtcp2 v1.25.0 or higher.
Fixes#22270Closes#22356
- In vsftpd ignore unrelated RST by matching tcpdump RSTs to the data
connection port pair.
Prior to this change an unrelated RST on a recycled ephemeral port
could cause test failure.
Closes https://github.com/curl/curl/pull/22305
When the threaded resolver gets AAAA results that carry a link-local
address without scope-id, it now re-queues a query with AF_UNSPEC and
strips ipv4 addresses from that result. Whatever the resulting addresses
and scope-ids are, this becomes the result of the resolve.
Fixes#22330
Reported-by: Bartel Sielski
Closes#22368
- CMake/CurlTests.c: sync indent with rest of file and most of code.
- cmake: sync term in `_CURL_PREFILL` description with rest of code.
- curl_setup.h: document function relying on `_CRT_SECURE_NO_WARNINGS`.
- tests/http/scorecard.py: typo in message.
- 'Quiche' -> 'quiche'.
- fix comment and formatting nits.
Closes#22378
Use separate dns cache entries for addresses (A+AAAA) and HTTPS-RR
results. That makes also "negative" results independent of each other.
Dns cache entries, once in use, can no longer be modified safely, as
concurrent use would require each access then to be done under lock. By
using separate entries, we can update a HTTPS entry without needing to
duplicate an existing address entry for the same host+port.
Connection filters can now ask for DNS resolves. This works at any time
during connection setup and while connect is going on.
`Curl_conn_dns_add_addr_resolve()` and
`Curl_conn_dns_add_https_resolve()` are used for that. They check if the
resolve is already ongoing, can be added to a resolve not started yet
or, as last resort, create a new filter instance and add it to the
connection (it's easier to add more filters than making the same filter
handle multiple resolves. Since DNS filters are removed once the
connection is established, there is no later penalty).
HTTPS-RR queries are added by the `HTTPS-CONNECT`, `SSL` and `QUIC`
filters. The latter will only do that when ECH is configured and
supported. That means we trigger HTTPS-RR queries only when the results
matter.
Add test_22_06 for ALPN influenced via HTTPS-RR when tunneling through a
proxy. This did not work before.
Adjust test2100 to use https: as the previous http: URL no longer
triggers HTTPS-RR resolves.
Closes#22216
- keep pause state at request
- keep presense of content decoding writers at request
- only add cw-pause writer when content decoding happens
- define macro for passing data down the writer chain
scorecard: add option to test with h2o server
Closes#22335
As reported in the bug:
> On a 32-bit architecture (i386):
> 1. `chunks * chunk_size` evaluates to `unsigned int` (32-bit
> unsigned).
> 2. Due to Usual Arithmetic Conversions in C, the signed `-1` (int)
> operand of the ternary operator is promoted to `unsigned int`,
> yielding `4294967295` (0xFFFFFFFF).
> 3. The ternary operator returns `4294967295` as an `unsigned int`.
> 4. This unsigned value is then assigned to `r->clength` (apr_off_t,
> 64-bit signed). Since the source is unsigned, it is zero-extended,
> resulting in `r->clength` becoming `+4294967295`.
> 5. The subsequently executed check `if(r->clength >= 0)` evaluates to
> true.
> 6. Inside the block, `apr_ltoa(r->pool, (long)r->clength)` casts it to
> a 32-bit signed `long` (on 32-bit platforms), which truncates it
> back to `-1`, formatting it as "-1" and sending the
> "Content-Length: -1" header.
>
> On a 64-bit architecture (amd64), `size_t` is 64-bit, and `-1` (32-bit
> int) is promoted to `unsigned long` (64-bit), yielding
> `18446744073709551615`. When assigned to `r->clength` (64-bit signed),
> it wraps back to `-1`, which correctly skips the Content-Length
> generation.
>
> This type promotion mismatch can be safely fixed by avoiding the
> signed/unsigned mixture in the ternary operator.
Origin: debian, https://bugs.debian.org/1140793
Bug-Debian: https://bugs.debian.org/1140793
These are standard Debian patches' trailers, I can get rid of then if you prefer.
I've tested the fix using [debusine on i386 runner](https://debusine.debian.net/debian/developers/artifact/4083252/raw/log):
> 782s tests/http/test_05_errors.py::TestErrors::test_05_04_unclean_tls_shutdown[http/1.0] PASSED [ 17%]
> 782s tests/http/test_05_errors.py::TestErrors::test_05_04_unclean_tls_shutdown[http/1.1] PASSED [ 18%]
> 782s tests/http/test_05_errors.py::TestErrors::test_05_04_unclean_tls_shutdown[h2] PASSED [ 18%]
Versus the [i386 run in Debian's infrastructure](https://debusine.debian.net/debian/developers/artifact/4084220/raw/log):
> 2690s tests/http/test_05_errors.py::TestErrors::test_05_04_unclean_tls_shutdown[http/1.0] FAILED [ 17%]
> 2690s tests/http/test_05_errors.py::TestErrors::test_05_04_unclean_tls_shutdown[http/1.1] PASSED [ 18%]
> 2690s tests/http/test_05_errors.py::TestErrors::test_05_04_unclean_tls_shutdown[h2] PASSED [ 18%]
Closes#22210
- Adjust pollset during connection shutdown.
- Separate the FIRSTSOCKET and SECONDSOCKET check so that one being in
connect or shutdown no longer adds poll events for the other.
- Fix shutdown state evaluation (as detailed below).
- Add a unit test for Curl_conn_adjust_pollset.
- Add a client for event testing.
- Test that shutdown sockets stay with the socket callback until done.
The pollset predicate now reads the connection's own shutdown state
rather than going through data->conn, and the EXPIRE_SHUTDOWN arming in
cshutdn_perform() works again: next_expire_ms started at 0 and could
never be set, so a peer that never sends a close_notify would still park
its connection forever, timeout or not.
Reported-by: pszemus@users.noreply.github.com
Fixes https://github.com/curl/curl/issues/22282
Closes https://github.com/curl/curl/pull/22304
Remove the timestamp passed for ratelimit checks. Let the limit
calculation obtain a timestamp when it needs it. Most transfers run
without active ratelimits and getting a fresh timestamp is unnecessary.
Closes#22292
- Determine max host and total connection limits from the transfer's own
multi handle and test that it works for shared connections.
Prior to this change, since df67269 (precedes 8.13.0),
Curl_cpool_check_limits() took the limits from the multi handle that
owns the pool. A share-owned pool is not owned by any multi, so the
limit check always passed.
Prior to df67269 the limits came from the transfer's own multi handle.
Fixes https://github.com/curl/curl/issues/22265
Closes https://github.com/curl/curl/pull/22266
Websocket frames need to be written individualy, so that applications
can access the frame meta data correctly. This worked until the write
function triggered a pause. Remaining frames accumulated in the "out"
writer's buffer and on unpaused were written in one chunk.
ws decode writer will now stop writing frames when the client writer
is paused. To handle the writing of buffered raw data after an unpause,
client writers have gotten a new "flush" method.
Add pytest test_20_12 with a new client to check handling of pauses
and websocket frames.
Reported-by: Hendrik Hübner
Fixes#22273Closes#22283
The curl tool was creating a new transfer every time it checked if
it needed to add one to reach max parallelism. This led to eventually
all configured transfers to have easy handles created.
Limit the creation again to the ones needed for max parallelism.
scorecard.py: set --out-null only for curl versions that support it
Closes#22277
FTP servers using SSL can be configured to check TLS session reuse on
the DATA connection. They hand out a new session on every CONTROL
connect and require to see the client using exactly that one when
up-/downloading on DATA.
This means:
1. We have to configure the SSL filter on the DATA connection with
exactly the same peers.
2. We have to remember the SSL session on the CONTROL connection -
separately from the session cache. The SSL filter on the DATA
connection then looks for a session at the CONTROL filter and, if
present, uses that.
Tests:
Enable `require_ssl_reuse` in our pytest setup for vsftpd. This
reproduces the problem reported in #22225 and verifies the fix.
Skip ftp+SSL pytests for rustls, as we have no possibility to reuse
sessions in that backend.
Schannel: we do not run these tests with the backend. I expect it has
similar problems but am not able to verify.
Reported-by: Laurent Sabourin
Fixes#22225Closes#22246
Add classic tests 3223-3226 and extend tests/http/test_12_reuse.py so that
connection reuse decisions are explicit:
- partial/aborted HTTP/1.1 responses must not reuse the connection
(premature multi_done closes non-multiplexed conns)
- different target hostnames must not share a connection even when they
resolve to the same address (url_match_destination)
- HTTP Basic credentials are per-request (PROTOPT_CREDSPERREQUEST), so
different -u values still reuse; assert that with num_connects
- same host positive control reuses (num_connects 1 then 0)
Closes#22224
- GHA/windows: mention `IgnoreStandardErrorWarningFormat=true`
in comment.
- cmake: fix MIT/GNU GSS order in messages.
- drop some exclamation marks from messages.
- drop redundant ending newlines from messages.
- fold/unfold where possible.
- fix indent, whitespace, typos and other nits.
Closes#22185
- Makefile.am: add root `test-quiet` target that maps to
`tests/quiet-test`, for completeness.
Ref: #22098
- tests/Makefile.am: drop unused remains of `MANFILES`.
Follow-up to fa3f889752#17463
Also replace the `distclean` target to delete `CLEANFILES`, otherwise
the CI dist job fails. (I'm not grokking why)
- tests/Makefile.am: sync up `tests` target with CMake, to pass `TFLAGS`
`-a`.
Follow-up to 904e7ecb66#19347
- tests/certs/Makefile.am: fix to clean generated certs via `CLEANFILES`
to comply with autotools documentation, which says the previously used
variable `DISTCLEANFILES` is meant for files created by `./configure`.
Ref: https://www.gnu.org/software/automake/manual/html_node/Clean.html
Follow-up to 44341e736a#16824
- tests/http/Makefile.am: add `./` prefix to glob passed to `rm -rf` on
clean, to avoid deleting unintented files, or pass unintended options.
Follow-up to 671158242d#10349
Cherry-picked from #22102
Cherry-picked from #22150Closes#22154
When talking to a forwarding proxy, do not start HTTPS Eyeballing.
We might support this in the future, but for now, the --httpx.x
arguments to do not apply to such a setup.
Add a test case for forward proxying without use of ALPN.
Closes#22033
Add `data->state.origin` as the origin the transfer is sending the
current request to/gets the response from. Use it for request specific
properties like authentication, hsts and cookie handling, etc.
Unless talking to a forwarding HTTP proxy (e.g. not tunneling),
`data->state.origin` and `conn->origin` are the same.
With a forwarding HTTP proxy in play, `conn->origin` is set to
`conn->http_proxy.peer` and `conn->bits.origin_is_proxy` (a new bit) is
set.
Remove the connection bits, now replaced with:
* `conn->bits.socksproxy` -> `conn->socks_proy.peer`
* `conn->bits.httpproxy` -> `conn->http_proy.peer`
* `conn->bits.proxy` -> `(conn->socks_proy.peer || conn->http_proy.peer`)
* `conn->bits.tunnel_proxy` -> (`conn->http_proy.peer && !conn->bits.origin_is_proxy`)
* `(conn->bits.httpproxy && !conn->bits.tunnel_proxy)` -> `conn->bits.origin_is_proxy`
Rename `noproxy.[ch]` to `proxy.[ch]`. Move the connection proxy setup
code from `url.c` to `proxy.c`.
Remove `data->info.conn_remote_port` as no one uses it.
Add test_40_02b for a SOCKS connection to a forwarding HTTPS proxy.
Update internal documentation about peers and creds.
Closes#21967
Share common functions/structs between ngtcp2 HTTP/3 and the proxy
version.
Fix bugs in proxy implementation when it comes to stream and pollset
handling and transfer lifetimes.
Curl_multi_xfer_sockbuf_borrow: work without multi
When a connection gets shutdown by a share, the easy handle used is
share->admin and it does not have a multi handle. In that case let
Curl_multi_xfer_sockbuf_borrow() allocate a buffer to be freed on
release.
This happens when a TLS filter sends its last notify through a HTTP/3
proxy tunnel.
Closes#21871
Do not send PONG frames unless there is sufficient space left in the
websocket send buffer. A server might be lazy in reading our data and
intermediary PONG frames can be skipped by a client (RFC 6455, ch.
5.5.3).
Add test case measuring no real RSS increase on a server blasting with
PING frames.
Closes#21911
- dante.py, dnsd.py, sshd.py: drop redundant conditions.
Spotted in sshd by GitHub Code Quality.
- curl.py: comment out `if` to silence CodeQL warning.
Reported by GitHub CodeQL
Follow-up to 8145476d5d#21916Closes#21917
Also:
- drop two unreachable return statements.
- test_17_ssl_use: avoid implicit string concatenations in lists.
Reported by GitHub CodeQL
Closes#21916
Whack the times reported for a transfer (see
https://curl.se/libcurl/c/curl_easy_getinfo.html#TIMES) into order for
all variations of up-/download, http/ftp etc. Make sure they are
reported in the documented order.
There is still the *possibility* of PRETRANSFER being longer then
POSTTRANSFER, if a server sends a response before an upload is done.
POST is the time the first response byte is received, and PRE is the
time the last byte was sent by curl.
This may happen with more likelihood on HTTP/2 and 3 for a server
rejected upload. But for successful uploads, the answer will almost over
come afterwards.
Undo the previous twists in lib500.c tests, adjust pytest timeline
checks.
Fixes#21828
Reported-by: BazaarAcc32 on github
Closes#21843
- drop more uses of the word "just". (not enforced here)
- drop some uses of the "will" word.
- "then" -> "than".
- tests/http/testenv/curl.py: fix copy-paste typo in error message.
- pytest: replace `shutdownh` with `shutdown` in test names.
Spotted by GitHub Code Quality.
- comment typos.
- whitespace and newlines fixes.
Closes#21830
- merge tests into a single class.
For shorter names, to fix sort order by test number, and to align with
other tests.
- fix preconditions to make `test_60_04_guard_proxy_http3_unsupported`
actually run.
- replace local precondition with constant of the same effect.
- drop redundant non-`ngtcp2` requirement for
`test_60_04_guard_proxy_http3_unsupported`.
(seemed relevant for no longer supported openssl-quic builds.)
- drop unused `NGTCP2_ONLY_MSG` constant.
Follow-up to e4139a73c8#21798
- avoid creating unnecessary test data blobs, and minimize their scopes.
Follow-up to 91facd7bb3#21791
Follow-up to e78b1b3ecc#21153Closes#21811
Add test_31_13 to check connection reuse on mixed --ssl-reqd setting.
For that add debug env var CURL_DBG_NO_USE_SSL_ON_FIRST to disable
--ssl-reqd for the first url. Check that the connection without SSL
from the first url is not reused on the second URL that requires it.
Tweak special ftp: protocol check to fail a DEBUGASSERT on mismatched
`use_ssl` settings as that should have been caught before in the
connection reuse matching (imap/smtp etc. do not have this extra check
and rely on the general part doing its job).
Closes#21665
- test_60_02_connect_tunnel_fail[fail_h1_over_h3_proxytunnel]
- test_60_02_connect_tunnel_fail[fail_h3_over_h2_proxytunnel]
- test_60_02_connect_tunnel_fail[fail_h3_over_h3_proxytunnel]
- test_60_03_h3_target_auto_connect_udp[proxy_h3]
- test_60_15_connect_timeout
Further flaky ones may be disabled in future commits.
All to be re-enabled after stabilizing them.
Follow-up to 59213f8248#21789
Follow-up to e78b1b3ecc#21153Closes#21803
- replace literal -1 with `NGHTTP3_ERR_CALLBACK_FAILURE` in nghttp3
callback.
- replace `NGHTTP3_ERR_CALLBACK_FAILURE` with
`NGTCP2_ERR_CALLBACK_FAILURE` in ngtcp2 callbacks.
- test_60_h3_proxy: fix non-critical typo in symbol.
Spotted by GitHub Code Quality
Follow-up to e78b1b3ecc#21153Closes#21802
code:
- less exception handling in existing code
- true ip happy eyeballing
- enable certificate verification
- cf-h2-proxy: abort connection when server closed connection
tests:
- remove all --insecure and --proxy-insecure args
- make session reuse test_60_12 a working one
- resolve port conflicts between h2o and nghttpx
- use proxy args better
- make test_60_06 run shorter
- kill h2o at the end of tests, normal stop takes too long
Ref: 59213f8248#21789
Follow-up to e78b1b3ecc#21153Closes#21798
