- keep pause state at request
- keep presense of content decoding writers at request
- only add cw-pause writer when content decoding happens
- define macro for passing data down the writer chain
scorecard: add option to test with h2o server
Closes#22335
As reported in the bug:
> On a 32-bit architecture (i386):
> 1. `chunks * chunk_size` evaluates to `unsigned int` (32-bit
> unsigned).
> 2. Due to Usual Arithmetic Conversions in C, the signed `-1` (int)
> operand of the ternary operator is promoted to `unsigned int`,
> yielding `4294967295` (0xFFFFFFFF).
> 3. The ternary operator returns `4294967295` as an `unsigned int`.
> 4. This unsigned value is then assigned to `r->clength` (apr_off_t,
> 64-bit signed). Since the source is unsigned, it is zero-extended,
> resulting in `r->clength` becoming `+4294967295`.
> 5. The subsequently executed check `if(r->clength >= 0)` evaluates to
> true.
> 6. Inside the block, `apr_ltoa(r->pool, (long)r->clength)` casts it to
> a 32-bit signed `long` (on 32-bit platforms), which truncates it
> back to `-1`, formatting it as "-1" and sending the
> "Content-Length: -1" header.
>
> On a 64-bit architecture (amd64), `size_t` is 64-bit, and `-1` (32-bit
> int) is promoted to `unsigned long` (64-bit), yielding
> `18446744073709551615`. When assigned to `r->clength` (64-bit signed),
> it wraps back to `-1`, which correctly skips the Content-Length
> generation.
>
> This type promotion mismatch can be safely fixed by avoiding the
> signed/unsigned mixture in the ternary operator.
Origin: debian, https://bugs.debian.org/1140793
Bug-Debian: https://bugs.debian.org/1140793
These are standard Debian patches' trailers, I can get rid of then if you prefer.
I've tested the fix using [debusine on i386 runner](https://debusine.debian.net/debian/developers/artifact/4083252/raw/log):
> 782s tests/http/test_05_errors.py::TestErrors::test_05_04_unclean_tls_shutdown[http/1.0] PASSED [ 17%]
> 782s tests/http/test_05_errors.py::TestErrors::test_05_04_unclean_tls_shutdown[http/1.1] PASSED [ 18%]
> 782s tests/http/test_05_errors.py::TestErrors::test_05_04_unclean_tls_shutdown[h2] PASSED [ 18%]
Versus the [i386 run in Debian's infrastructure](https://debusine.debian.net/debian/developers/artifact/4084220/raw/log):
> 2690s tests/http/test_05_errors.py::TestErrors::test_05_04_unclean_tls_shutdown[http/1.0] FAILED [ 17%]
> 2690s tests/http/test_05_errors.py::TestErrors::test_05_04_unclean_tls_shutdown[http/1.1] PASSED [ 18%]
> 2690s tests/http/test_05_errors.py::TestErrors::test_05_04_unclean_tls_shutdown[h2] PASSED [ 18%]
Closes#22210
- Adjust pollset during connection shutdown.
- Separate the FIRSTSOCKET and SECONDSOCKET check so that one being in
connect or shutdown no longer adds poll events for the other.
- Fix shutdown state evaluation (as detailed below).
- Add a unit test for Curl_conn_adjust_pollset.
- Add a client for event testing.
- Test that shutdown sockets stay with the socket callback until done.
The pollset predicate now reads the connection's own shutdown state
rather than going through data->conn, and the EXPIRE_SHUTDOWN arming in
cshutdn_perform() works again: next_expire_ms started at 0 and could
never be set, so a peer that never sends a close_notify would still park
its connection forever, timeout or not.
Reported-by: pszemus@users.noreply.github.com
Fixes https://github.com/curl/curl/issues/22282
Closes https://github.com/curl/curl/pull/22304
Remove the timestamp passed for ratelimit checks. Let the limit
calculation obtain a timestamp when it needs it. Most transfers run
without active ratelimits and getting a fresh timestamp is unnecessary.
Closes#22292
- Determine max host and total connection limits from the transfer's own
multi handle and test that it works for shared connections.
Prior to this change, since df67269 (precedes 8.13.0),
Curl_cpool_check_limits() took the limits from the multi handle that
owns the pool. A share-owned pool is not owned by any multi, so the
limit check always passed.
Prior to df67269 the limits came from the transfer's own multi handle.
Fixes https://github.com/curl/curl/issues/22265
Closes https://github.com/curl/curl/pull/22266
Websocket frames need to be written individualy, so that applications
can access the frame meta data correctly. This worked until the write
function triggered a pause. Remaining frames accumulated in the "out"
writer's buffer and on unpaused were written in one chunk.
ws decode writer will now stop writing frames when the client writer
is paused. To handle the writing of buffered raw data after an unpause,
client writers have gotten a new "flush" method.
Add pytest test_20_12 with a new client to check handling of pauses
and websocket frames.
Reported-by: Hendrik Hübner
Fixes#22273Closes#22283
The curl tool was creating a new transfer every time it checked if
it needed to add one to reach max parallelism. This led to eventually
all configured transfers to have easy handles created.
Limit the creation again to the ones needed for max parallelism.
scorecard.py: set --out-null only for curl versions that support it
Closes#22277
FTP servers using SSL can be configured to check TLS session reuse on
the DATA connection. They hand out a new session on every CONTROL
connect and require to see the client using exactly that one when
up-/downloading on DATA.
This means:
1. We have to configure the SSL filter on the DATA connection with
exactly the same peers.
2. We have to remember the SSL session on the CONTROL connection -
separately from the session cache. The SSL filter on the DATA
connection then looks for a session at the CONTROL filter and, if
present, uses that.
Tests:
Enable `require_ssl_reuse` in our pytest setup for vsftpd. This
reproduces the problem reported in #22225 and verifies the fix.
Skip ftp+SSL pytests for rustls, as we have no possibility to reuse
sessions in that backend.
Schannel: we do not run these tests with the backend. I expect it has
similar problems but am not able to verify.
Reported-by: Laurent Sabourin
Fixes#22225Closes#22246
Add classic tests 3223-3226 and extend tests/http/test_12_reuse.py so that
connection reuse decisions are explicit:
- partial/aborted HTTP/1.1 responses must not reuse the connection
(premature multi_done closes non-multiplexed conns)
- different target hostnames must not share a connection even when they
resolve to the same address (url_match_destination)
- HTTP Basic credentials are per-request (PROTOPT_CREDSPERREQUEST), so
different -u values still reuse; assert that with num_connects
- same host positive control reuses (num_connects 1 then 0)
Closes#22224
- GHA/windows: mention `IgnoreStandardErrorWarningFormat=true`
in comment.
- cmake: fix MIT/GNU GSS order in messages.
- drop some exclamation marks from messages.
- drop redundant ending newlines from messages.
- fold/unfold where possible.
- fix indent, whitespace, typos and other nits.
Closes#22185
- Makefile.am: add root `test-quiet` target that maps to
`tests/quiet-test`, for completeness.
Ref: #22098
- tests/Makefile.am: drop unused remains of `MANFILES`.
Follow-up to fa3f889752#17463
Also replace the `distclean` target to delete `CLEANFILES`, otherwise
the CI dist job fails. (I'm not grokking why)
- tests/Makefile.am: sync up `tests` target with CMake, to pass `TFLAGS`
`-a`.
Follow-up to 904e7ecb66#19347
- tests/certs/Makefile.am: fix to clean generated certs via `CLEANFILES`
to comply with autotools documentation, which says the previously used
variable `DISTCLEANFILES` is meant for files created by `./configure`.
Ref: https://www.gnu.org/software/automake/manual/html_node/Clean.html
Follow-up to 44341e736a#16824
- tests/http/Makefile.am: add `./` prefix to glob passed to `rm -rf` on
clean, to avoid deleting unintented files, or pass unintended options.
Follow-up to 671158242d#10349
Cherry-picked from #22102
Cherry-picked from #22150Closes#22154
When talking to a forwarding proxy, do not start HTTPS Eyeballing.
We might support this in the future, but for now, the --httpx.x
arguments to do not apply to such a setup.
Add a test case for forward proxying without use of ALPN.
Closes#22033
Add `data->state.origin` as the origin the transfer is sending the
current request to/gets the response from. Use it for request specific
properties like authentication, hsts and cookie handling, etc.
Unless talking to a forwarding HTTP proxy (e.g. not tunneling),
`data->state.origin` and `conn->origin` are the same.
With a forwarding HTTP proxy in play, `conn->origin` is set to
`conn->http_proxy.peer` and `conn->bits.origin_is_proxy` (a new bit) is
set.
Remove the connection bits, now replaced with:
* `conn->bits.socksproxy` -> `conn->socks_proy.peer`
* `conn->bits.httpproxy` -> `conn->http_proy.peer`
* `conn->bits.proxy` -> `(conn->socks_proy.peer || conn->http_proy.peer`)
* `conn->bits.tunnel_proxy` -> (`conn->http_proy.peer && !conn->bits.origin_is_proxy`)
* `(conn->bits.httpproxy && !conn->bits.tunnel_proxy)` -> `conn->bits.origin_is_proxy`
Rename `noproxy.[ch]` to `proxy.[ch]`. Move the connection proxy setup
code from `url.c` to `proxy.c`.
Remove `data->info.conn_remote_port` as no one uses it.
Add test_40_02b for a SOCKS connection to a forwarding HTTPS proxy.
Update internal documentation about peers and creds.
Closes#21967
Share common functions/structs between ngtcp2 HTTP/3 and the proxy
version.
Fix bugs in proxy implementation when it comes to stream and pollset
handling and transfer lifetimes.
Curl_multi_xfer_sockbuf_borrow: work without multi
When a connection gets shutdown by a share, the easy handle used is
share->admin and it does not have a multi handle. In that case let
Curl_multi_xfer_sockbuf_borrow() allocate a buffer to be freed on
release.
This happens when a TLS filter sends its last notify through a HTTP/3
proxy tunnel.
Closes#21871
Do not send PONG frames unless there is sufficient space left in the
websocket send buffer. A server might be lazy in reading our data and
intermediary PONG frames can be skipped by a client (RFC 6455, ch.
5.5.3).
Add test case measuring no real RSS increase on a server blasting with
PING frames.
Closes#21911
- dante.py, dnsd.py, sshd.py: drop redundant conditions.
Spotted in sshd by GitHub Code Quality.
- curl.py: comment out `if` to silence CodeQL warning.
Reported by GitHub CodeQL
Follow-up to 8145476d5d#21916Closes#21917
Also:
- drop two unreachable return statements.
- test_17_ssl_use: avoid implicit string concatenations in lists.
Reported by GitHub CodeQL
Closes#21916
Whack the times reported for a transfer (see
https://curl.se/libcurl/c/curl_easy_getinfo.html#TIMES) into order for
all variations of up-/download, http/ftp etc. Make sure they are
reported in the documented order.
There is still the *possibility* of PRETRANSFER being longer then
POSTTRANSFER, if a server sends a response before an upload is done.
POST is the time the first response byte is received, and PRE is the
time the last byte was sent by curl.
This may happen with more likelihood on HTTP/2 and 3 for a server
rejected upload. But for successful uploads, the answer will almost over
come afterwards.
Undo the previous twists in lib500.c tests, adjust pytest timeline
checks.
Fixes#21828
Reported-by: BazaarAcc32 on github
Closes#21843
- drop more uses of the word "just". (not enforced here)
- drop some uses of the "will" word.
- "then" -> "than".
- tests/http/testenv/curl.py: fix copy-paste typo in error message.
- pytest: replace `shutdownh` with `shutdown` in test names.
Spotted by GitHub Code Quality.
- comment typos.
- whitespace and newlines fixes.
Closes#21830
- merge tests into a single class.
For shorter names, to fix sort order by test number, and to align with
other tests.
- fix preconditions to make `test_60_04_guard_proxy_http3_unsupported`
actually run.
- replace local precondition with constant of the same effect.
- drop redundant non-`ngtcp2` requirement for
`test_60_04_guard_proxy_http3_unsupported`.
(seemed relevant for no longer supported openssl-quic builds.)
- drop unused `NGTCP2_ONLY_MSG` constant.
Follow-up to e4139a73c8#21798
- avoid creating unnecessary test data blobs, and minimize their scopes.
Follow-up to 91facd7bb3#21791
Follow-up to e78b1b3ecc#21153Closes#21811
Add test_31_13 to check connection reuse on mixed --ssl-reqd setting.
For that add debug env var CURL_DBG_NO_USE_SSL_ON_FIRST to disable
--ssl-reqd for the first url. Check that the connection without SSL
from the first url is not reused on the second URL that requires it.
Tweak special ftp: protocol check to fail a DEBUGASSERT on mismatched
`use_ssl` settings as that should have been caught before in the
connection reuse matching (imap/smtp etc. do not have this extra check
and rely on the general part doing its job).
Closes#21665
- test_60_02_connect_tunnel_fail[fail_h1_over_h3_proxytunnel]
- test_60_02_connect_tunnel_fail[fail_h3_over_h2_proxytunnel]
- test_60_02_connect_tunnel_fail[fail_h3_over_h3_proxytunnel]
- test_60_03_h3_target_auto_connect_udp[proxy_h3]
- test_60_15_connect_timeout
Further flaky ones may be disabled in future commits.
All to be re-enabled after stabilizing them.
Follow-up to 59213f8248#21789
Follow-up to e78b1b3ecc#21153Closes#21803
- replace literal -1 with `NGHTTP3_ERR_CALLBACK_FAILURE` in nghttp3
callback.
- replace `NGHTTP3_ERR_CALLBACK_FAILURE` with
`NGTCP2_ERR_CALLBACK_FAILURE` in ngtcp2 callbacks.
- test_60_h3_proxy: fix non-critical typo in symbol.
Spotted by GitHub Code Quality
Follow-up to e78b1b3ecc#21153Closes#21802
code:
- less exception handling in existing code
- true ip happy eyeballing
- enable certificate verification
- cf-h2-proxy: abort connection when server closed connection
tests:
- remove all --insecure and --proxy-insecure args
- make session reuse test_60_12 a working one
- resolve port conflicts between h2o and nghttpx
- use proxy args better
- make test_60_06 run shorter
- kill h2o at the end of tests, normal stop takes too long
Ref: 59213f8248#21789
Follow-up to e78b1b3ecc#21153Closes#21798
- H3 proxy: re-sync code with original source `curl_ngtcp2.c` to reduce
differences, and to apply missed minor fixes. Also apply clang-format.
Drop redundant `#undef`s, casts, `#endif` comments, includes, drop
intermediate variables, sync include and macro order.
Follow-up to e78b1b3ecc#21153
- INSTALL-CMAKE.md: move `CURL_ENABLE_SMB` to the enable section.
- tests/http/env: rename `tcpdmp` to `tcpdump` to match object variable.
- mbedtls: drop incorrect `mbedTLS 4+` comments.
(features are also supported by 3+, meaning it's always supported.)
- lib1648: rename a variable to match purpose.
- CIPHERS.md: alpha-sort link list.
- replace rare `X''` hex markup with `0x`.
- `IP v4/6` -> `IPv4/6`.
- 'version X.Y' -> 'vX.Y', where sensible.
- 'VX.Y' -> 'vX.Y', where sensible.
- fix indents, casing, newlines, typos.
Closes#21772
This patch adds two major proxy capabilities to curl (ngtcp2 QUIC):
- HTTP/3 Proxy CONNECT: Tunnel HTTP/1.1 or HTTP/2 traffic through an
HTTPS proxy that speaks HTTP/3 (QUIC) using the standard CONNECT
method over an HTTP/3 connection.
- MASQUE CONNECT-UDP: Tunnel HTTP/3 (QUIC) traffic through an HTTP
proxy (speaking HTTP/1.1, HTTP/2, or HTTP/3) using the extended
CONNECT method with the CONNECT-UDP protocol (RFC9297 & RFC9298).
Public API additions:
- `CURLPROXY_HTTPS3`: new proxy type constant for HTTP/3 proxy
- `--proxy-http3`: new CLI flag to negotiate HTTP/3 with HTTPS proxy
The implementation adds two new filters:
- `H3-PROXY` - enables negotiating HTTP/3 (QUIC) to the proxy and
running CONNECT/CONNECT-UDP through that proxy transport.
- `CAPSULE` - dedicated filter inserted between QUIC transport and
HTTP-PROXY to handle datagram capsule encapsulation/decapsulation.
Here is how the curl filter chaining looks in different scenarios:
- HTTP/3 Proxy CONNECT (tunneling TCP protocols over QUIC proxy):
conn -> HTTP/1.1 or HTTP/2 -> SSL -> HTTP-PROXY ->
H3-PROXY -> HAPPY-EYEBALLS -> UDP
- MASQUE CONNECT-UDP (tunneling QUIC over any proxy):
conn -> HTTP/3 -> CAPSULE -> HTTP-PROXY -> H3-PROXY ->
HAPPY-EYEBALLS -> UDP
conn -> HTTP/3 -> CAPSULE -> HTTP-PROXY -> H1-PROXY or H2-PROXY ->
SSL -> HAPPY-EYEBALLS -> TCP
- Both features currently require the ngtcp2 QUIC backend.
- Both features are experimental (disabled by default). Enable with
`--enable-proxy-http3`(autotools) or `-DUSE_PROXY_HTTP3=ON`(CMake).
Tests:
- tests/unit/unit3400.c: Unit tests for capsule protocol encode/decode
- tests/http/test_60_h3_proxy.py: Comprehensive pytest integration suite
- tests/http/testenv/h2o.py: Managing h2o instances with HTTP/1.1, HTTP/2,
and HTTP/3 (QUIC) listeners, proxy.connect and proxy.connect-udp enabled.
References:
RFC 9297 - HTTP Datagrams and the Capsule Protocol
RFC 9298 - Proxying UDP in HTTP
RFC 9000 §16 — Variable-Length Integer Encoding
Signed-off-by: Aritra Basu <aritrbas+gh@cisco.com>
Closes#21153
Since all API features we need for TLSv1.3 earlydata support do exist
only from version 3.7.2 onwards, make that the minimal version required.
Fixes#21750
Reported-by: Johannes Schlatow
Closes#21751
Fix a recent regression: when a unix-socket is configured, all proxy
settings must be ignored. The `via_peer` had been checked correctly,
but the connections proxy bits were not cleared.
Add test_11_04 to verify.
Reported-by: Fabian Keil (libcurl mailing list)
Closes#21630
Or consisting of just a single dot.
Such names cannot be resolved with DNS.
While they *can* still be resolved with /etc/hosts or --resolve tricks,
they easily cause internal problems because their trailing dots.
Let's not allow them anymore.
Closes#21622
- update action `actions/cache` from 5.0.4 to 5.0.5
- update action `actions/upload-artifact` from 7.0.0 to 7.0.1
- update action `github/codeql-action` from 4.32.4 to 4.35.2
- update action `msys2/setup-msys2` from 2.31.0 to 2.31.1
- update pip `filelock` from 3.25.2 to 3.29.0
- update pip `impacket` to 0.13.0
- update pip `ruff` from 0.15.10 to 0.15.12
Closes#21483Closes#21482
