- Adjust pollset during connection shutdown.
- Separate the FIRSTSOCKET and SECONDSOCKET check so that one being in
connect or shutdown no longer adds poll events for the other.
- Fix shutdown state evaluation (as detailed below).
- Add a unit test for Curl_conn_adjust_pollset.
- Add a client for event testing.
- Test that shutdown sockets stay with the socket callback until done.
The pollset predicate now reads the connection's own shutdown state
rather than going through data->conn, and the EXPIRE_SHUTDOWN arming in
cshutdn_perform() works again: next_expire_ms started at 0 and could
never be set, so a peer that never sends a close_notify would still park
its connection forever, timeout or not.
Reported-by: pszemus@users.noreply.github.com
Fixes https://github.com/curl/curl/issues/22282
Closes https://github.com/curl/curl/pull/22304
E.g. on CHERI pointers are 128 bits [1]. This causes the unit3214 test
to fail, which was written with more traditional platforms in mind. Here
is the output of log/stderr3214:
```
URL: -
BAD: struct Curl_easy is 7984 bytes, allowed to be 5370: 2614 bytes too big
BAD: struct connectdata is 1408 bytes, allowed to be 1300: 108 bytes too big
BAD: struct Curl_multi is 1248 bytes, allowed to be 850: 398 bytes too big
BAD: struct curl_httppost is 224 bytes, allowed to be 112: 112 bytes too big
BAD: struct curl_slist is 32 bytes, allowed to be 16: 16 bytes too big
BAD: struct curl_khkey is 32 bytes, allowed to be 24: 8 bytes too big
BAD: struct curl_hstsentry is 48 bytes, allowed to be 40: 8 bytes too big
BAD: struct curl_mime is 144 bytes, allowed to be 96: 48 bytes too big
BAD: struct curl_mimepart is 592 bytes, allowed to be 440: 152 bytes too big
BAD: struct curl_certinfo is 32 bytes, allowed to be 16: 16 bytes too big
BAD: struct curl_tlssessioninfo is 32 bytes, allowed to be 16: 16 bytes too big
BAD: struct curl_blob is 32 bytes, allowed to be 24: 8 bytes too big
BAD: struct CURLMsg is 48 bytes, allowed to be 24: 24 bytes too big
BAD: struct curl_header is 80 bytes, allowed to be 48: 32 bytes too big
Test ended with result 14
```
Multiply the allowed size on systems with larger than 64-bit pointers.
[1] https://www.cl.cam.ac.uk/research/security/ctsrd/pdfs/20171017a-cheri-poster.pdfCloses#22299
Also repurpose existing build-time feature checks into unit test 1961,
to verify.
Prior to this patch these functions were auto-detected with both
autotools and cmake. In case of autotools there was an extra
verification phase ensuring the functions work as expected. This step
required running the function, thus was limited to non-cross-builds. For
cross-builds and CMake it always used the system implementation if
present. On Windows it always used the local implementation, because
availability/use is complicated there.
After this patch all platforms, always use the local implementation,
which is known to be accurate. This makes curl behave more consistently,
and simplifies the build process, a fixes cross-builds and CMake
auto-detection differences.
Also:
- test1960: enable unconditionally.
- checksrc: disallow globally, allowlist in `block_ip.c` example.
- dnsd: verify ntop result for NULL before passing to printf.
Ref: https://github.com/curl/curl/pull/22137#issuecomment-4797440983
Ref: #22137
Ref: 8537a5b0bc#16577Closes#22170
- cmp-pkg-config.sh: replace `-r -f` with `-rf` to match rest of repo.
- configure.ac: add double quotes for robustness (not a bug).
- curl-openssl.m4: merge nested `if`s.
- CurlTests.c: drop `!= 0`, also to sync with m4.
- CurlTests.c: replace `example.com` with `localhost` in
`gethostbyname()` feature test code. (compile-only, not a bug)
- GHA/http3-linux: drop literal `true` from bool expression.
- lib650: drop redundant `&`.
- move variable/call to left-hand side of equality checks, where
missing.
- perl: detach `<`/`>` from filename in `open()`, where missing.
- schannel: apply two nit fixes lost in rebase.
- scripts/verify-release: drop redundant double quotes.
- scripts/verify-release: exit with error code on error.
- synctime: replace magic numbers with `sizeof()`.
- telnet: add missing parentheses to macro value.
- tests/Makefile.am: use single quotes.
- tool_operate: drop redundant `break` after `return` in VMS code.
- unit2413: drop unused NULL pointer + free call.
- unit2413: fix duplicate test case name.
- urlapi: drop redundant parentheses.
- urlapi: drop `CURL_UNCONST()` that became redundant.
Closes#22186
- GHA/windows: mention `IgnoreStandardErrorWarningFormat=true`
in comment.
- cmake: fix MIT/GNU GSS order in messages.
- drop some exclamation marks from messages.
- drop redundant ending newlines from messages.
- fold/unfold where possible.
- fix indent, whitespace, typos and other nits.
Closes#22185
If any sublabel is longer than 63 octets, abort. This then also catches
compression attempts.
Verified in test 1658
Reported-by: netspacer.research
Closes#22124
To avoid mistakes or abuse to cause problems. Many public DNS providers
cap their cache times to this.
Verify in test 1650
Reported-by: netspacer.research
Closes#22122
- `N byte` -> `N-byte` or `N bytes`.
- INTERNALS.md: language tweaks.
- schannel: language tweak in comment/error message.
- socks_gssapi, socks_sspi: simplify composing an error message.
(at a cost of 8 extra constant string bytes.)
- m4/curl-compilers.m4: fix typo in link (in comment).
- contrithanks.sh: fix indent, drop stray `;` terminator.
- lib, src, tests: drop/fix a bunch of badwords.
- fix typos in comments.
- fix indent, stray spaces.
Some of these spotted by GitHub Code Quality and Copilot
Closes#22009
Add `data->state.origin` as the origin the transfer is sending the
current request to/gets the response from. Use it for request specific
properties like authentication, hsts and cookie handling, etc.
Unless talking to a forwarding HTTP proxy (e.g. not tunneling),
`data->state.origin` and `conn->origin` are the same.
With a forwarding HTTP proxy in play, `conn->origin` is set to
`conn->http_proxy.peer` and `conn->bits.origin_is_proxy` (a new bit) is
set.
Remove the connection bits, now replaced with:
* `conn->bits.socksproxy` -> `conn->socks_proy.peer`
* `conn->bits.httpproxy` -> `conn->http_proy.peer`
* `conn->bits.proxy` -> `(conn->socks_proy.peer || conn->http_proy.peer`)
* `conn->bits.tunnel_proxy` -> (`conn->http_proy.peer && !conn->bits.origin_is_proxy`)
* `(conn->bits.httpproxy && !conn->bits.tunnel_proxy)` -> `conn->bits.origin_is_proxy`
Rename `noproxy.[ch]` to `proxy.[ch]`. Move the connection proxy setup
code from `url.c` to `proxy.c`.
Remove `data->info.conn_remote_port` as no one uses it.
Add test_40_02b for a SOCKS connection to a forwarding HTTPS proxy.
Update internal documentation about peers and creds.
Closes#21967
Move as sockaddr2string() into cf-socket.c where its only callers are.
Mark as UNITTEST for unit1609.
Move "struct Curl_sockaddr_ex" into sockaddr.h, so connect.h and
cf-socket.h can be included without all the system headers needed.
Closes#21946
Adjust code to avoid `-Wformat-signedness` warnings, while making sure
that enums are always cast to a known type when passing them to `printf`
functions, to support compilers and compiler settings where enums are
not default-size signed ints.
- cast integers printed as hex to `unsigned`. (63 times, 20 of them in
`mbedtls.c`)
- cast misc enums to `int` for printing. (31 times)
- cast `CURL_LOCK_DATA_*` enums to `int`. (4 times)
- cast `CURL_FORMADD_*` enums to `int`. (13 times)
- cast `CURLSHE_*` enums to `int`. (3 times)
- cast `CURLUE_*` enums to `int`. (33 times)
- cast `CURLMSG_*` enums to `int`. (6 times)
- cast `CURLE_*` enums to `int`. (~380 times)
- unit1675: fix mask.
Follow-up to 7c34365cce#21879
Ref: #18343 (initial attempt)
Closes#20848
Share common functions/structs between ngtcp2 HTTP/3 and the proxy
version.
Fix bugs in proxy implementation when it comes to stream and pollset
handling and transfer lifetimes.
Curl_multi_xfer_sockbuf_borrow: work without multi
When a connection gets shutdown by a share, the easy handle used is
share->admin and it does not have a multi handle. In that case let
Curl_multi_xfer_sockbuf_borrow() allocate a buffer to be freed on
release.
This happens when a TLS filter sends its last notify through a HTTP/3
proxy tunnel.
Closes#21871
Whack the times reported for a transfer (see
https://curl.se/libcurl/c/curl_easy_getinfo.html#TIMES) into order for
all variations of up-/download, http/ftp etc. Make sure they are
reported in the documented order.
There is still the *possibility* of PRETRANSFER being longer then
POSTTRANSFER, if a server sends a response before an upload is done.
POST is the time the first response byte is received, and PRE is the
time the last byte was sent by curl.
This may happen with more likelihood on HTTP/2 and 3 for a server
rejected upload. But for successful uploads, the answer will almost over
come afterwards.
Undo the previous twists in lib500.c tests, adjust pytest timeline
checks.
Fixes#21828
Reported-by: BazaarAcc32 on github
Closes#21843
closing a filter chain and reconnecting it again is a complication
that only the HTTP/1.x proxy filter used. Remove it from all filters.
Instead, a filter can return CURLE_AGAIN during the connect phase
and the cf-setup filter will tear down all "sub filters" and restart
over.
With this, a filter never resets to the initial phase but progresses
through connect -> connected -> shutdown -> destroy once.
Closes#21831
All users of the `verify_memory()` macro used a fixed-length buffer for
the test output, which then needed a workaround to silence GCC
`-Waddress` warnings.
```
tests/unit/unit1615.c: In function 'test_unit1615':
tests/libtest/unitcheck.h:51:8: error: the address of 'output_buf' will always evaluate as 'true' [-Werror=address]
51 | if((dynamic) && memcmp(dynamic, check, len)) { \
| ^
tests/unit/unit1615.c:114:3: note: in expansion of macro 'verify_memory'
114 | verify_memory(output_buf, precomp_hash1, CURL_SHA512_256_DIGEST_LENGTH);
| ^~~~~~~~~~~~~
```
Drop redundant address check and the workarounds with it.
Closes#21833
- drop more uses of the word "just". (not enforced here)
- drop some uses of the "will" word.
- "then" -> "than".
- tests/http/testenv/curl.py: fix copy-paste typo in error message.
- pytest: replace `shutdownh` with `shutdown` in test names.
Spotted by GitHub Code Quality.
- comment typos.
- whitespace and newlines fixes.
Closes#21830
When a transfer goes against another origin than the initial one, do not
add the following to the ssl configuration: client cert, client key, srp
user/pass, pinned key.
Closes#21695
code:
- less exception handling in existing code
- true ip happy eyeballing
- enable certificate verification
- cf-h2-proxy: abort connection when server closed connection
tests:
- remove all --insecure and --proxy-insecure args
- make session reuse test_60_12 a working one
- resolve port conflicts between h2o and nghttpx
- use proxy args better
- make test_60_06 run shorter
- kill h2o at the end of tests, normal stop takes too long
Ref: 59213f8248#21789
Follow-up to e78b1b3ecc#21153Closes#21798
This patch adds two major proxy capabilities to curl (ngtcp2 QUIC):
- HTTP/3 Proxy CONNECT: Tunnel HTTP/1.1 or HTTP/2 traffic through an
HTTPS proxy that speaks HTTP/3 (QUIC) using the standard CONNECT
method over an HTTP/3 connection.
- MASQUE CONNECT-UDP: Tunnel HTTP/3 (QUIC) traffic through an HTTP
proxy (speaking HTTP/1.1, HTTP/2, or HTTP/3) using the extended
CONNECT method with the CONNECT-UDP protocol (RFC9297 & RFC9298).
Public API additions:
- `CURLPROXY_HTTPS3`: new proxy type constant for HTTP/3 proxy
- `--proxy-http3`: new CLI flag to negotiate HTTP/3 with HTTPS proxy
The implementation adds two new filters:
- `H3-PROXY` - enables negotiating HTTP/3 (QUIC) to the proxy and
running CONNECT/CONNECT-UDP through that proxy transport.
- `CAPSULE` - dedicated filter inserted between QUIC transport and
HTTP-PROXY to handle datagram capsule encapsulation/decapsulation.
Here is how the curl filter chaining looks in different scenarios:
- HTTP/3 Proxy CONNECT (tunneling TCP protocols over QUIC proxy):
conn -> HTTP/1.1 or HTTP/2 -> SSL -> HTTP-PROXY ->
H3-PROXY -> HAPPY-EYEBALLS -> UDP
- MASQUE CONNECT-UDP (tunneling QUIC over any proxy):
conn -> HTTP/3 -> CAPSULE -> HTTP-PROXY -> H3-PROXY ->
HAPPY-EYEBALLS -> UDP
conn -> HTTP/3 -> CAPSULE -> HTTP-PROXY -> H1-PROXY or H2-PROXY ->
SSL -> HAPPY-EYEBALLS -> TCP
- Both features currently require the ngtcp2 QUIC backend.
- Both features are experimental (disabled by default). Enable with
`--enable-proxy-http3`(autotools) or `-DUSE_PROXY_HTTP3=ON`(CMake).
Tests:
- tests/unit/unit3400.c: Unit tests for capsule protocol encode/decode
- tests/http/test_60_h3_proxy.py: Comprehensive pytest integration suite
- tests/http/testenv/h2o.py: Managing h2o instances with HTTP/1.1, HTTP/2,
and HTTP/3 (QUIC) listeners, proxy.connect and proxy.connect-udp enabled.
References:
RFC 9297 - HTTP Datagrams and the Capsule Protocol
RFC 9298 - Proxying UDP in HTTP
RFC 9000 §16 — Variable-Length Integer Encoding
Signed-off-by: Aritra Basu <aritrbas+gh@cisco.com>
Closes#21153
- use `UNITTEST_BEGIN_SIMPLE`/`UNITTEST_END_SIMPLE` where missing.
- drop redundant `(void)arg;` where using `UNITTEST_BEGIN*`.
- unit1636: drop redundant block after `UNITTEST_BEGIN*`.
- unit1609: fix typo in comment.
- unit1627: merge to `if`s.
Closes#21715
cert_type, key, key_type, key_passwd and key_blob lived in
ssl_config_data but not in ssl_primary_config, so they were invisible to
match_ssl_primary_config() and to the TLS session cache peer key.
Two easy handles sharing a connection pool could reuse each other's
authenticated connections when they differed only on SSLKEY, SSLKEYTYPE,
KEYPASSWD, SSLCERTTYPE or SSLKEYBLOB. The second handle would silently
inherit the first handle's authenticated identity.
Promote all five fields into ssl_primary_config so the conn-reuse
predicate and session cache key cover the complete client credential
set. Also replace the fixed ":CCERT" session cache marker with the
actual clientcert path so sessions are not shared across different
client certificates.
Verified by test 3303 and 3304
Reported-By: Joshua Rogers (AISLE Research)
Closes#21667
Refactor the netrc scanner. Add test case for checking that the last
matched machine with unmatched login does not return the password as
success (unit1304).
Closes#21624
Also fix fallouts found.
Windows clang-tidy CI job is a little pickier than I'd prefer due to the
`_CURL_TESTS_CONCAT=ON` option used there, and all macros considered
local, thus checked by the compiler. Upside: it revealed macro usage
dynamics in tests. If too annoying, `first.h` may be opted-out from the
concat logic. Some macros may also be deleted instead of `#if 0`-ing.
Follow-up to e0e56e9ae4#21550
Follow-up to 5fa5cb3825#20593Closes#21554
