* format `data->state.aptr.host` without crlf, so signatures do not need
to strip it again. Add the crlf when adding the header to the request
dynbuf
* check `connect-to` strings on normalized hostname and user supplied
hostname (when those differ)
* libssh: always use the peer for setting ssh option SSH_OPTIONS_HOST,
preserve ipv6 [] enclosure, use IDN converted hostname otherwise. This
is the libssh documented expectation.
Do NOT use strings from URL parsing.
Closes#22128
Remove members of `struct urlpieces` that now live in peers/creds.
Change all remaining uses of those urlpieces.
When comparing schemes, use the protocol constants.
Closes#22171
Do not check if a particular connection is alive or not more than once
every second. We did this on every connection reuse which is overkill
when sending many requests to the same host.
Closes#22169
- GHA/windows: mention `IgnoreStandardErrorWarningFormat=true`
in comment.
- cmake: fix MIT/GNU GSS order in messages.
- drop some exclamation marks from messages.
- drop redundant ending newlines from messages.
- fold/unfold where possible.
- fix indent, whitespace, typos and other nits.
Closes#22185
Fix a regression in curl 8.21.0-rc3: Check whether the host and the "connect
to host" are equal before initializing the proxy. If they are equal, switching
to tunnel mode is not necessary.
Follow-up to 73daec6Closes#22110
When tying credentials to a connection (NTLM, Negotiate) also link the
origin the credentials are for. This prevents a connection reuse with
the same credentials, but intended for another origin.
The mis-reuse could happen for a forwarding proxy and NTLM (although, in
the mind of the person writing this, it is an insane setup).
Closes#22040
Add `data->state.origin` as the origin the transfer is sending the
current request to/gets the response from. Use it for request specific
properties like authentication, hsts and cookie handling, etc.
Unless talking to a forwarding HTTP proxy (e.g. not tunneling),
`data->state.origin` and `conn->origin` are the same.
With a forwarding HTTP proxy in play, `conn->origin` is set to
`conn->http_proxy.peer` and `conn->bits.origin_is_proxy` (a new bit) is
set.
Remove the connection bits, now replaced with:
* `conn->bits.socksproxy` -> `conn->socks_proy.peer`
* `conn->bits.httpproxy` -> `conn->http_proy.peer`
* `conn->bits.proxy` -> `(conn->socks_proy.peer || conn->http_proy.peer`)
* `conn->bits.tunnel_proxy` -> (`conn->http_proy.peer && !conn->bits.origin_is_proxy`)
* `(conn->bits.httpproxy && !conn->bits.tunnel_proxy)` -> `conn->bits.origin_is_proxy`
Rename `noproxy.[ch]` to `proxy.[ch]`. Move the connection proxy setup
code from `url.c` to `proxy.c`.
Remove `data->info.conn_remote_port` as no one uses it.
Add test_40_02b for a SOCKS connection to a forwarding HTTPS proxy.
Update internal documentation about peers and creds.
Closes#21967
Adjust code to avoid `-Wformat-signedness` warnings, while making sure
that enums are always cast to a known type when passing them to `printf`
functions, to support compilers and compiler settings where enums are
not default-size signed ints.
- cast integers printed as hex to `unsigned`. (63 times, 20 of them in
`mbedtls.c`)
- cast misc enums to `int` for printing. (31 times)
- cast `CURL_LOCK_DATA_*` enums to `int`. (4 times)
- cast `CURL_FORMADD_*` enums to `int`. (13 times)
- cast `CURLSHE_*` enums to `int`. (3 times)
- cast `CURLUE_*` enums to `int`. (33 times)
- cast `CURLMSG_*` enums to `int`. (6 times)
- cast `CURLE_*` enums to `int`. (~380 times)
- unit1675: fix mask.
Follow-up to 7c34365cce#21879
Ref: #18343 (initial attempt)
Closes#20848
This reverts commit 3e9817cd1b.
The change was incorrect as the check was not for the options the commit
message mentions.
Reported-by: ByteRay on hackerone
Closes#21899
- drop more uses of the word "just". (not enforced here)
- drop some uses of the "will" word.
- "then" -> "than".
- tests/http/testenv/curl.py: fix copy-paste typo in error message.
- pytest: replace `shutdownh` with `shutdown` in test names.
Spotted by GitHub Code Quality.
- comment typos.
- whitespace and newlines fixes.
Closes#21830
Reshuffle code a little to detect when the transfer's credentials
actually change. Otherwise, leave the existing creds in place.
This gives the precise location where we may want to reset other
states that become invalid with change credentials. Also, by
keeping a creds instance as long as it is valid, we can associate
meta data with it.
Closes#21755
When a transfer goes against another origin than the initial one, do not
add the following to the ssl configuration: client cert, client key, srp
user/pass, pinned key.
Closes#21695
Add test_31_13 to check connection reuse on mixed --ssl-reqd setting.
For that add debug env var CURL_DBG_NO_USE_SSL_ON_FIRST to disable
--ssl-reqd for the first url. Check that the connection without SSL
from the first url is not reused on the second URL that requires it.
Tweak special ftp: protocol check to fail a DEBUGASSERT on mismatched
`use_ssl` settings as that should have been caught before in the
connection reuse matching (imap/smtp etc. do not have this extra check
and rely on the general part doing its job).
Closes#21665
code:
- less exception handling in existing code
- true ip happy eyeballing
- enable certificate verification
- cf-h2-proxy: abort connection when server closed connection
tests:
- remove all --insecure and --proxy-insecure args
- make session reuse test_60_12 a working one
- resolve port conflicts between h2o and nghttpx
- use proxy args better
- make test_60_06 run shorter
- kill h2o at the end of tests, normal stop takes too long
Ref: 59213f8248#21789
Follow-up to e78b1b3ecc#21153Closes#21798
This patch adds two major proxy capabilities to curl (ngtcp2 QUIC):
- HTTP/3 Proxy CONNECT: Tunnel HTTP/1.1 or HTTP/2 traffic through an
HTTPS proxy that speaks HTTP/3 (QUIC) using the standard CONNECT
method over an HTTP/3 connection.
- MASQUE CONNECT-UDP: Tunnel HTTP/3 (QUIC) traffic through an HTTP
proxy (speaking HTTP/1.1, HTTP/2, or HTTP/3) using the extended
CONNECT method with the CONNECT-UDP protocol (RFC9297 & RFC9298).
Public API additions:
- `CURLPROXY_HTTPS3`: new proxy type constant for HTTP/3 proxy
- `--proxy-http3`: new CLI flag to negotiate HTTP/3 with HTTPS proxy
The implementation adds two new filters:
- `H3-PROXY` - enables negotiating HTTP/3 (QUIC) to the proxy and
running CONNECT/CONNECT-UDP through that proxy transport.
- `CAPSULE` - dedicated filter inserted between QUIC transport and
HTTP-PROXY to handle datagram capsule encapsulation/decapsulation.
Here is how the curl filter chaining looks in different scenarios:
- HTTP/3 Proxy CONNECT (tunneling TCP protocols over QUIC proxy):
conn -> HTTP/1.1 or HTTP/2 -> SSL -> HTTP-PROXY ->
H3-PROXY -> HAPPY-EYEBALLS -> UDP
- MASQUE CONNECT-UDP (tunneling QUIC over any proxy):
conn -> HTTP/3 -> CAPSULE -> HTTP-PROXY -> H3-PROXY ->
HAPPY-EYEBALLS -> UDP
conn -> HTTP/3 -> CAPSULE -> HTTP-PROXY -> H1-PROXY or H2-PROXY ->
SSL -> HAPPY-EYEBALLS -> TCP
- Both features currently require the ngtcp2 QUIC backend.
- Both features are experimental (disabled by default). Enable with
`--enable-proxy-http3`(autotools) or `-DUSE_PROXY_HTTP3=ON`(CMake).
Tests:
- tests/unit/unit3400.c: Unit tests for capsule protocol encode/decode
- tests/http/test_60_h3_proxy.py: Comprehensive pytest integration suite
- tests/http/testenv/h2o.py: Managing h2o instances with HTTP/1.1, HTTP/2,
and HTTP/3 (QUIC) listeners, proxy.connect and proxy.connect-udp enabled.
References:
RFC 9297 - HTTP Datagrams and the Capsule Protocol
RFC 9298 - Proxying UDP in HTTP
RFC 9000 §16 — Variable-Length Integer Encoding
Signed-off-by: Aritra Basu <aritrbas+gh@cisco.com>
Closes#21153
The HTTP/2 feature is deprecated, few servers implement it and our
implementation is complicated by its state management. Make the two
CURLOPT_* involved a nop and deprecate them.
Closes#21723
When a proxy is set from an environment variable, detect if that proxy
is not the same as previously and flush state.
Verified by test1647: verify changing proxy with env variables and make
sure Digest state is flushed in the second use
Closes#21666
Refactor the netrc scanner. Add test case for checking that the last
matched machine with unmatched login does not return the password as
success (unit1304).
Closes#21624
Fix a recent regression: when a unix-socket is configured, all proxy
settings must be ignored. The `via_peer` had been checked correctly,
but the connections proxy bits were not cleared.
Add test_11_04 to verify.
Reported-by: Fabian Keil (libcurl mailing list)
Closes#21630
CURLOPT_GSSAPI_DELEGATION exposes distinct modes:
CURLGSSAPI_DELEGATION_POLICY_FLAG is documented as delegating only when
OK-AS-DELEGATE policy permits it, while CURLGSSAPI_DELEGATION_FLAG is
unconditional. The new SSPI implementation checks for either bit and
sets ISC_REQ_DELEGATE, so a caller requesting policy-limited delegation
is put on the same SSPI path as unconditional delegation.
In addition, curl's existing protection that avoids reusing a connection
when the GSS delegation setting differs was guarded only by HAVE_GSSAPI;
SSPI-only builds now have an effective delegation option, but the
connection's delegation setting was neither copied nor compared. This
would cause Windows SSPI Negotiate/Kerberos authentication to delegate
credentials contrary to the caller's selected policy or reuse an
already-delegated authenticated connection for a transfer that requested
no delegation.
Follow-up to cc6777d939
Reported by Codex Security
Closes#21583
The SASL service name, used in authentication, is part of curl's credentials
when authenticating to a server/proxy. Make it part of `struct Curl_creds`.
Change code to use `creds` to obtain a service name. By tying creds used
to the connection, connection reuse is also only allowed when the service
name matches.
Closes#21585
Authorizdation credentials are kept in `struct Curl_creds`. This contains:
* `user`: the username, maybe the empty string
* `passwd`: the password, maybe the empty string
* `sasl_authzid`: the SASL authz value, maybe the empty string
* `oauth_bearer`: the OAUTH bearer token, maybe the empty string
* `source`: where the credentials from from
* `refcount`: a reference counter to link/unkink creds
A `creds` with all values empty is equivalent to NULL, e.g. no `creds`
instance. With reference counting, `creds` can be linked/unlinked
in several places.
See docs/internals/CREDENTIALS.md for use.
Closes#21548
When a connection is tested for reuse in a transfer that *may* upgrade
to TLS (commonly via STARTTLS), the SSL configuration must match the
existing connection.
Reported-by: Andrew Nesbit
Closes#21522
The CURLOPT_SSH_HOST_PUBLIC_KEY_* options are documented to verify the
host at connect time and not for connection reuse. Once the SSH host has
been deemed okay, it remains okay as long as the connection survives.
In addition: this function currently always returned TRUE since the
pointers have been NULLed in the SSH backend code before this function
is called.
Follow-up to c31fcf2dec
Reported-by: Andrew Nesbit
Closes#21519
`struct Curl_peer` keeps information about a communication endpoint
together. It will replace `conn->host` and `conn->conn_to_host` and
proxyinfo host. It will also become part of `struct ssl_peer`.
It has a reference counter, so an instance can be shared between
connections and filters.
Elminiates `conn->host` and `conn->connect_to_host`, used in the
proxyinfo structures. Passed to DNS resolution and socks filters, etc.
Pass peer to http proxy and socks tunnel filters. Use peer in dns filter
and resolving. Make `Curl_peer` a member in the `struct ssl_peer`.
Add `docs/internals/PEERS.md` for documentation.
Closes#21472
So that CURLcode failure is correct and not figured out later via
`conn->bits.proxy`. Add the flag to the async struct.
`for_proxy` is figured out by the caller of Curl_resolv() when it
figures out which host/proxy name it wants the addresses for.
Remove CONN_IS_PROXIED macro as no longer needed.
Closes#21423
- make sure all UNITTEST prototypes mark in which unit test they are used,
with "@unittest" markup
- make sure all UNITTEST functions do not use Curl_ prefix, as that is a
prefix we use for global private functions and these functions are static
and therefore not global and the prefix is wrong
- drop UNITTEST for functions not used in unit tests
- make the extract-unit-protos script highlight the above issues if found
- extract-unit-protos now also outputs the unit test number for all the
generated protos in lib/unitprotos.h to aid readers. It also adds the source
file and line number where the proto originates from.
- extract-unit-protos now exits with a non-zero value if any of the above
warnings are triggered
- cf-dns: Curl_cf_dns_result => static cf_dns_result
- hostip: Curl_ipv6works => static ipv6works
- url: remove Curl_setup_conn() - not used anymore
- connect: Curl_timeleft_now_ms => UNITTEST timeleft_now_ms
Closes#21330
To make it scoped for the single request appropriately.
Reported-by: Muhamad Arga Reksapati
Verify with libtest 2504: a custom Host *disabled* on reused handle
Closes#21312
- examples: sync debug output printf masks.
- INSTALL-CMAKE.md: tidy up section for some options.
- curl_sha512_256: delete comment suggesting an optimization.
- vtls/keylog: scope a variable.
- vtls/openssl: make a source code URL a permalink.
- vtls/schannel: drop redundant parentheses.
- test1119.pl: robustify `$1` -> `$s`.
- sync arg names in comments to match the code.
- tidy up and minor fixes to comments.
- fix formatting/indenting/comment/newline/include nits.
- move `UNITTEST` protos next to definitions, sync their argument names.
- make variables static.
- add parentheses to Perl `open()` calls.
- drop unnecessary double quotes in Perl.
- clang-format.
Closes#21000
