- decode results when individual requests are done
- makes happy eyeballing start asap
- remove doh_resp structures as no longer needed
- remove CURL_DNS_TYPE_NS, CURL_DNS_TYPE_CNAME and
CURL_DNS_TYPE_DNAME from DoH
- DoH: do not set PIPEWAIT and SSL OPTS when url starts with http:
- mark Doh master handle as dirty after every sub-request, not only the last
- Doh: start probe on AAAA before A (was the other way).
cf-dns: set EXPIRE_HAPPY_EYEBALLS timer when waiting 50ms on AAAA
result or progress will not be triggered in time.
Add debug env var CURL_DBG_HE_AAAA_AWAIT_MS to override the
default 50ms on waiting for the AAAA result to arrive.
test2100: set CURL_DBG_HE_AAAA_AWAIT_MS to 60 seconds to
provide enough time for slow CI runs to sent all DoH requests.
closes#22514
dnsd now opens UDP+TCP sockets and accepts http: DoH requests to obtain
the same, configured answers (records, delays, error codes) as over UDP.
DoH: use `async->queries_ongoing` like all other resolvers instead of
the internal `pending` counter. Fixes waiting for results.
Tests: in pytest, parameterize dnsd tests to use both DNS and DoH.
Closes#22506
The fallback path for cryptography < 42 was broken by commit e13362c2
that caused a comparison between offset-naive and offset-aware
datetimes. Use the positional form of tz in datetime.now() everywhere.
Ref: #22396
Pointed out by Codex Security
Closes#22426
When writing a decoded chunk of websocket data, always flush the writer
chain so that buffered data gets delivered before the ws meta data gets
updated.
Add client writer flags CURL_CW_FLAG_BLOWUP for writer types that may
significantly enlarge write sizes. This flag causes the pause writer to
be added and shrinks the write chunk sizes. We do not want that for
content decoders like WS that do not change the size.
Add test_20_13 to check that large frames are paused/unpaused correctly
with the matching meta data.
Fixes#22413
Reported-by: Hendrik Hübner
Closes#22416
This version is already two releases out of support, but is "only" 7
years old so is probably still being used in the real world. Document
this version along with some other testing dependencies. Remove code
support for earlier versions. Disable ruff checks that need a newer
version.
Mark Python mutable class variables with ClassVar, to denote that the
danger this can cause has been considered. Since any change made to
these in any object affects all other objects, this can cause locality
errors. However, as used in the test suite, they are are never modified
and so they are annotated as being intended.
Always set a timezone in datetime objects, as mixing naive and
timezone-aware object can cause errors.
These fix ruff rules DTZ005, RUF012.
* call super() without arguments
* mark an unused variable as such
* simplify by using dict getter for default values
* use writelines() when possible
* use dedent to simplify some text formatting
* avoid items() on dict in a loop when unnecessary
* replace most Python format() calls with f-strings
* use capture_output in subprocess.run
This fixes ruff rules FLY002, FURB122, PERF102, RUF059, SIM401, UP008,
UP022, UP030.
* combine separate if statements
* remove an unneeded encode() call
* remove unneeded returns
* simplify code when returning early
This fixes ruff rules SIM102, SIM114, UP012, PLR1711.
* remove an unneeded ruff warning disable
* remove coding: utf-8 from Python code; PEP 3120 makes UTF-8 the
default encoding
* remove unusable shebang lines from Python code
* remove empty print strings
* disable warnings when file objects are stored; these instances can't
be handled with context managers
* use more consistent whitespace in Python code, fixing flake8 warnings
* set the executable bit on scorecard.py, making it easier to run
These fix ruff rules EXE001, FURB105, UP009, SIM115.
* Explicitly set "check" in subprocess.run() to raise an exception
automatically, where it was done manually before
* Use contextlib.suppress to ignore exceptions
* Use custom exceptions for test errors for clarity and flexibility.
* Replace IOError with OSError
This fixes ruff rules BLE001, PLW1510, S110, TRY201, TRY203, TRY002,
UP024.
- make `const` data `static`, where missing and possible.
- replace `strlen()` on literal or const strings with `sizeof()`.
While the latter is optimized by popular C compiler, e.g. MSVC only
does it with `/O2`.
- replace magic numbers with `sizeof()`, where missing.
- introduce `CURL_CSTRLEN()` macro for `sizeof(char[]) - 1`.
- use `CURL_CSTRLEN()` macro.
- move `const` before integer types, where missing.
- replace `char *var` with `var[]`, where missing and possible.
- use double const, where missing.
`static const char *` -> `static const char * const`.
- lib1514: constify pointers.
- unit3205: drop redundant cast, avoid another one.
- unit1666: map `OID()` macro to identical `STRCONST()`.
Closes#22406
- In vsftpd ignore unrelated RST by matching tcpdump RSTs to the data
connection port pair.
Prior to this change an unrelated RST on a recycled ephemeral port
could cause test failure.
Closes https://github.com/curl/curl/pull/22305
- CMake/CurlTests.c: sync indent with rest of file and most of code.
- cmake: sync term in `_CURL_PREFILL` description with rest of code.
- curl_setup.h: document function relying on `_CRT_SECURE_NO_WARNINGS`.
- tests/http/scorecard.py: typo in message.
- 'Quiche' -> 'quiche'.
- fix comment and formatting nits.
Closes#22378
- keep pause state at request
- keep presense of content decoding writers at request
- only add cw-pause writer when content decoding happens
- define macro for passing data down the writer chain
scorecard: add option to test with h2o server
Closes#22335
As reported in the bug:
> On a 32-bit architecture (i386):
> 1. `chunks * chunk_size` evaluates to `unsigned int` (32-bit
> unsigned).
> 2. Due to Usual Arithmetic Conversions in C, the signed `-1` (int)
> operand of the ternary operator is promoted to `unsigned int`,
> yielding `4294967295` (0xFFFFFFFF).
> 3. The ternary operator returns `4294967295` as an `unsigned int`.
> 4. This unsigned value is then assigned to `r->clength` (apr_off_t,
> 64-bit signed). Since the source is unsigned, it is zero-extended,
> resulting in `r->clength` becoming `+4294967295`.
> 5. The subsequently executed check `if(r->clength >= 0)` evaluates to
> true.
> 6. Inside the block, `apr_ltoa(r->pool, (long)r->clength)` casts it to
> a 32-bit signed `long` (on 32-bit platforms), which truncates it
> back to `-1`, formatting it as "-1" and sending the
> "Content-Length: -1" header.
>
> On a 64-bit architecture (amd64), `size_t` is 64-bit, and `-1` (32-bit
> int) is promoted to `unsigned long` (64-bit), yielding
> `18446744073709551615`. When assigned to `r->clength` (64-bit signed),
> it wraps back to `-1`, which correctly skips the Content-Length
> generation.
>
> This type promotion mismatch can be safely fixed by avoiding the
> signed/unsigned mixture in the ternary operator.
Origin: debian, https://bugs.debian.org/1140793
Bug-Debian: https://bugs.debian.org/1140793
These are standard Debian patches' trailers, I can get rid of then if you prefer.
I've tested the fix using [debusine on i386 runner](https://debusine.debian.net/debian/developers/artifact/4083252/raw/log):
> 782s tests/http/test_05_errors.py::TestErrors::test_05_04_unclean_tls_shutdown[http/1.0] PASSED [ 17%]
> 782s tests/http/test_05_errors.py::TestErrors::test_05_04_unclean_tls_shutdown[http/1.1] PASSED [ 18%]
> 782s tests/http/test_05_errors.py::TestErrors::test_05_04_unclean_tls_shutdown[h2] PASSED [ 18%]
Versus the [i386 run in Debian's infrastructure](https://debusine.debian.net/debian/developers/artifact/4084220/raw/log):
> 2690s tests/http/test_05_errors.py::TestErrors::test_05_04_unclean_tls_shutdown[http/1.0] FAILED [ 17%]
> 2690s tests/http/test_05_errors.py::TestErrors::test_05_04_unclean_tls_shutdown[http/1.1] PASSED [ 18%]
> 2690s tests/http/test_05_errors.py::TestErrors::test_05_04_unclean_tls_shutdown[h2] PASSED [ 18%]
Closes#22210
Websocket frames need to be written individualy, so that applications
can access the frame meta data correctly. This worked until the write
function triggered a pause. Remaining frames accumulated in the "out"
writer's buffer and on unpaused were written in one chunk.
ws decode writer will now stop writing frames when the client writer
is paused. To handle the writing of buffered raw data after an unpause,
client writers have gotten a new "flush" method.
Add pytest test_20_12 with a new client to check handling of pauses
and websocket frames.
Reported-by: Hendrik Hübner
Fixes#22273Closes#22283
The curl tool was creating a new transfer every time it checked if
it needed to add one to reach max parallelism. This led to eventually
all configured transfers to have easy handles created.
Limit the creation again to the ones needed for max parallelism.
scorecard.py: set --out-null only for curl versions that support it
Closes#22277
FTP servers using SSL can be configured to check TLS session reuse on
the DATA connection. They hand out a new session on every CONTROL
connect and require to see the client using exactly that one when
up-/downloading on DATA.
This means:
1. We have to configure the SSL filter on the DATA connection with
exactly the same peers.
2. We have to remember the SSL session on the CONTROL connection -
separately from the session cache. The SSL filter on the DATA
connection then looks for a session at the CONTROL filter and, if
present, uses that.
Tests:
Enable `require_ssl_reuse` in our pytest setup for vsftpd. This
reproduces the problem reported in #22225 and verifies the fix.
Skip ftp+SSL pytests for rustls, as we have no possibility to reuse
sessions in that backend.
Schannel: we do not run these tests with the backend. I expect it has
similar problems but am not able to verify.
Reported-by: Laurent Sabourin
Fixes#22225Closes#22246
- dante.py, dnsd.py, sshd.py: drop redundant conditions.
Spotted in sshd by GitHub Code Quality.
- curl.py: comment out `if` to silence CodeQL warning.
Reported by GitHub CodeQL
Follow-up to 8145476d5d#21916Closes#21917
Also:
- drop two unreachable return statements.
- test_17_ssl_use: avoid implicit string concatenations in lists.
Reported by GitHub CodeQL
Closes#21916
Whack the times reported for a transfer (see
https://curl.se/libcurl/c/curl_easy_getinfo.html#TIMES) into order for
all variations of up-/download, http/ftp etc. Make sure they are
reported in the documented order.
There is still the *possibility* of PRETRANSFER being longer then
POSTTRANSFER, if a server sends a response before an upload is done.
POST is the time the first response byte is received, and PRE is the
time the last byte was sent by curl.
This may happen with more likelihood on HTTP/2 and 3 for a server
rejected upload. But for successful uploads, the answer will almost over
come afterwards.
Undo the previous twists in lib500.c tests, adjust pytest timeline
checks.
Fixes#21828
Reported-by: BazaarAcc32 on github
Closes#21843
- drop more uses of the word "just". (not enforced here)
- drop some uses of the "will" word.
- "then" -> "than".
- tests/http/testenv/curl.py: fix copy-paste typo in error message.
- pytest: replace `shutdownh` with `shutdown` in test names.
Spotted by GitHub Code Quality.
- comment typos.
- whitespace and newlines fixes.
Closes#21830
code:
- less exception handling in existing code
- true ip happy eyeballing
- enable certificate verification
- cf-h2-proxy: abort connection when server closed connection
tests:
- remove all --insecure and --proxy-insecure args
- make session reuse test_60_12 a working one
- resolve port conflicts between h2o and nghttpx
- use proxy args better
- make test_60_06 run shorter
- kill h2o at the end of tests, normal stop takes too long
Ref: 59213f8248#21789
Follow-up to e78b1b3ecc#21153Closes#21798
- H3 proxy: re-sync code with original source `curl_ngtcp2.c` to reduce
differences, and to apply missed minor fixes. Also apply clang-format.
Drop redundant `#undef`s, casts, `#endif` comments, includes, drop
intermediate variables, sync include and macro order.
Follow-up to e78b1b3ecc#21153
- INSTALL-CMAKE.md: move `CURL_ENABLE_SMB` to the enable section.
- tests/http/env: rename `tcpdmp` to `tcpdump` to match object variable.
- mbedtls: drop incorrect `mbedTLS 4+` comments.
(features are also supported by 3+, meaning it's always supported.)
- lib1648: rename a variable to match purpose.
- CIPHERS.md: alpha-sort link list.
- replace rare `X''` hex markup with `0x`.
- `IP v4/6` -> `IPv4/6`.
- 'version X.Y' -> 'vX.Y', where sensible.
- 'VX.Y' -> 'vX.Y', where sensible.
- fix indents, casing, newlines, typos.
Closes#21772
This patch adds two major proxy capabilities to curl (ngtcp2 QUIC):
- HTTP/3 Proxy CONNECT: Tunnel HTTP/1.1 or HTTP/2 traffic through an
HTTPS proxy that speaks HTTP/3 (QUIC) using the standard CONNECT
method over an HTTP/3 connection.
- MASQUE CONNECT-UDP: Tunnel HTTP/3 (QUIC) traffic through an HTTP
proxy (speaking HTTP/1.1, HTTP/2, or HTTP/3) using the extended
CONNECT method with the CONNECT-UDP protocol (RFC9297 & RFC9298).
Public API additions:
- `CURLPROXY_HTTPS3`: new proxy type constant for HTTP/3 proxy
- `--proxy-http3`: new CLI flag to negotiate HTTP/3 with HTTPS proxy
The implementation adds two new filters:
- `H3-PROXY` - enables negotiating HTTP/3 (QUIC) to the proxy and
running CONNECT/CONNECT-UDP through that proxy transport.
- `CAPSULE` - dedicated filter inserted between QUIC transport and
HTTP-PROXY to handle datagram capsule encapsulation/decapsulation.
Here is how the curl filter chaining looks in different scenarios:
- HTTP/3 Proxy CONNECT (tunneling TCP protocols over QUIC proxy):
conn -> HTTP/1.1 or HTTP/2 -> SSL -> HTTP-PROXY ->
H3-PROXY -> HAPPY-EYEBALLS -> UDP
- MASQUE CONNECT-UDP (tunneling QUIC over any proxy):
conn -> HTTP/3 -> CAPSULE -> HTTP-PROXY -> H3-PROXY ->
HAPPY-EYEBALLS -> UDP
conn -> HTTP/3 -> CAPSULE -> HTTP-PROXY -> H1-PROXY or H2-PROXY ->
SSL -> HAPPY-EYEBALLS -> TCP
- Both features currently require the ngtcp2 QUIC backend.
- Both features are experimental (disabled by default). Enable with
`--enable-proxy-http3`(autotools) or `-DUSE_PROXY_HTTP3=ON`(CMake).
Tests:
- tests/unit/unit3400.c: Unit tests for capsule protocol encode/decode
- tests/http/test_60_h3_proxy.py: Comprehensive pytest integration suite
- tests/http/testenv/h2o.py: Managing h2o instances with HTTP/1.1, HTTP/2,
and HTTP/3 (QUIC) listeners, proxy.connect and proxy.connect-udp enabled.
References:
RFC 9297 - HTTP Datagrams and the Capsule Protocol
RFC 9298 - Proxying UDP in HTTP
RFC 9000 §16 — Variable-Length Integer Encoding
Signed-off-by: Aritra Basu <aritrbas+gh@cisco.com>
Closes#21153
Since all API features we need for TLSv1.3 earlydata support do exist
only from version 3.7.2 onwards, make that the minimal version required.
Fixes#21750
Reported-by: Johannes Schlatow
Closes#21751
Add a "limit" handler to httpd that responds 429 after 5 requests and
then 429s all requests for 2 seconds. After which another 4 requests are
served before 429 strikes again.
Closes#21357
Support HTTPS records in dnsd that have ALPN preferences.
Add pytest test_22_* to verify that HTTPS record ALPN
preferencces take effect if the HTTPS resolve arrives in time.
Fix HTTPS eyeballing use of timeouts for second attempt. Also,
make an initial HTTP/1.1 attempt switch off HTTP/2.
Closes#21329
Add "Delay-A: ms", "Delay-AAAA: ms" and "Delay-HTTPS: ms" to the test
dnsd config and implement delayed response handling.
Add test_21_09 and test_21_10 to check that delayed responses connect
using the undelayed address family.
Closes#21299
When a user supplied an ip address in a URL as hostname, use that even
when address family restrictions like -4 or -6 are set.
Add test_10_15/16 to verify with a local proxy server.
Fixes#21146
Reported-by: Terrance Wong
How:
- cf-dns: on see the hostname is an ip(v6) address, add the respective
A/AAAA to the dns query bits
- cf-dns/hostip: only hand out addrinfos for a family if that family
is part of the DNS queries. That prevents for example ipv6 addresses
to show up from dns cache entries
- change cf-ip-happy to no longer check for "ip_version" and instead
use all addresses that cf-dns hands out
Closes#21295
Add `resolv-threaded` to curlinfo to detect use of the threaded resolver
correctly even with c-ares linked to https-rr.
Run test_21_05 exactly when threaded resolver is built.
Closes#21287
- Checks for missing explicit `return` statements at the end of functions
that can return non-`None` values.
- Checks for classes that inherit from `object`.
- Checks for useless expressions.
- Within an `except*` clause, raise exceptions with `raise ... from err`
or `raise ... from None` to distinguish them from errors in exception
handling
- Checks for variable assignments that immediately precede a `return` of the
assigned variable.
- Checks for `else` statements with a `return` statement in the preceding
`if` block.
- Checks for unnecessary parentheses on raised exceptions.
Closes: #21258
Use a thread queue and pool for asnyc threaded DNS resolves.
Add pytest test_21_* for verification.
Add `CURLMOPT_RESOLVE_THREADS_MAX` to allow applications to
resize the thread pool used.
Add `CURLMOPT_QUICK_EXIT` to allow applications to skip thread
joins when cleaning up a multi handle. Multi handles in
`curl_easy_perform()` inherit this from `CURLOPT_QUICK_EXIT`.
Add several debug environment variables for testing.
Closes#20936
- It is slower and uses more memory than the alternatives and is only
experimental in curl.
- We disable a few tests for OpenSSL-QUIC because of flakiness
- It gets little attention from OpenSSL and we have no expectation of the
major flaws getting corrected anytime soon.
- No one has spoken up for keeping it
- curl users building with vanilla OpenSSL can still use QUIC through the
means of ngtcp2
Closes#20226
