Parse extension names and values before deciding whether a parameter is
known. This prevents long names and token or quoted-string values from
hiding later ma or persist parameters, as required by RFC 7838 section
3.
Add unit coverage for long extension names and both value forms,
including an escaped quote and semicolon inside a quoted string.
Fixes#22644Closes#22676
Iterating unit test 1627 50,000 times show the new version to be 31%
faster on my machine.
- unit1627: add more test strings, In particular three, five and six letter
non-existing schemes.
- remove scripts/schemetable.c, not used anymore
Closes#22658
The previous 4 entries were exhausted on simple command lines, depending
on environment/build. Grow the initial size to 8 to avoid early allocs.
Closes#22633
Change the storage of easy handle option strings from an array sized for
all possible options to a hash set to reduce memory footprint.
Give the hash set initially room for 4 strings, with first allocation
happening when it goes beyond that. Measurements without test suite
and a forced fail on growing the set gives:
Size Result
2 1261 tests out of 1951 reported OK: 64%
4 1792 tests out of 1951 reported OK: 91%
8 1944 tests out of 1951 reported OK: 99%
16 1949 tests out of 1951 reported OK: 99%
32 single fail of 3211, unit test for u8_strset
Add u8_strset that keeps the tuples (uint8_t id, char *str)
and allows set/unset by `id`. Add that as data->set.strings.
Define MACROS
* CURL_EASY_STR(data, id) for access
* CURL_EASY_STR_SET(data, id, s) for setting, making a copy
* CURL_EASY_STR_SETN(data, id, s) for setting, no copy
* CURL_EASY_STR_CLEAR(data, id) for unsetting
* CURL_EASY_STR_CLEAR0(data, id) for unsetting and zero-ing value
Add `data->set.str_copypostfields` to handle former `STRING_COPYPOSTFIELDS`
string that was not always a string and could carry NUL bytes.
Add unit tests to test3211.
Closes#22628
To simplify and to remove an exception where `errno` was reused to
return a socket error codes on Windows.
The error was used by one call site (`sockaddr2string()` in
`cf-socket.c`), but it was in practice always propagated as
`SOCKEAFNOSUPPORT` to callers.
Also:
- cf-socket: update 3 error messages to show `CURLcode` accordingly.
- if2ip: handle `curlx_inet_ntop()` error in `Curl_if2ip()`.
- dnsd: display `CURLcode` on two errors.
Follow-up to 39dec13ec0#22170Closes#22229
- Move expire timeout code from multi into splay.c
- keep a "time_base" timestamp to calculate timediff_t for
actual timeout values. Unfortunately this means our
timeouts will go wrong after ~500,000 years of continuous
operations...
- use timediff_t as key in splay instead of curltime
- use timediff_t in transfers expire times instead of curltime
- re-comment splay.c for better understanding how it works
- replace splay nodes double-linked "same" list with a single link,
we almost never have duplicate keys
- keep transfer `mid` in splay nodes instead of the transfer pointer
- keep registered bit in splay node for tracking instead of separate
bit in transfer
- adapt unit1309.c to changes in timediff_t and mid
Closes#22584
Keep only a single `start` time and calculate the rest in delta
microseconds since then. Separate timestamps and durations into separate
sub-structures for clarity.
remove progress.t_acceptdata
It was recorded in FTP accept but never used anywhere.
Closes#22547
The dedotdotify() function that removes ./ and ../ sequences from paths
juggles memory and is slow. Now needs_dedotdot() is called first to
determine if the removal process is necessary and otherwise avoids doing
it. Avoids unnecessary memory operations.
Adjusted unit test 1395 accordingly because now a lot of input strings
return NULL for "no change necessary".
Suggested-by: Max Dymond
Closes#22557
- decode results when individual requests are done
- makes happy eyeballing start asap
- remove doh_resp structures as no longer needed
- remove CURL_DNS_TYPE_NS, CURL_DNS_TYPE_CNAME and
CURL_DNS_TYPE_DNAME from DoH
- DoH: do not set PIPEWAIT and SSL OPTS when url starts with http:
- mark Doh master handle as dirty after every sub-request, not only the last
- Doh: start probe on AAAA before A (was the other way).
cf-dns: set EXPIRE_HAPPY_EYEBALLS timer when waiting 50ms on AAAA
result or progress will not be triggered in time.
Add debug env var CURL_DBG_HE_AAAA_AWAIT_MS to override the
default 50ms on waiting for the AAAA result to arrive.
test2100: set CURL_DBG_HE_AAAA_AWAIT_MS to 60 seconds to
provide enough time for slow CI runs to sent all DoH requests.
closes#22514
- make `const` data `static`, where missing and possible.
- replace `strlen()` on literal or const strings with `sizeof()`.
While the latter is optimized by popular C compiler, e.g. MSVC only
does it with `/O2`.
- replace magic numbers with `sizeof()`, where missing.
- introduce `CURL_CSTRLEN()` macro for `sizeof(char[]) - 1`.
- use `CURL_CSTRLEN()` macro.
- move `const` before integer types, where missing.
- replace `char *var` with `var[]`, where missing and possible.
- use double const, where missing.
`static const char *` -> `static const char * const`.
- lib1514: constify pointers.
- unit3205: drop redundant cast, avoid another one.
- unit1666: map `OID()` macro to identical `STRCONST()`.
Closes#22406
`sys/types.h` and `sys/socket.h` (non-Win32). They are included via
`curl/curl.h` and `curl_setup.h`.
This drops `HAVE_SYS_TYPES_H` guards from the codebase. It's safe
because `sys/types.h` (POSIX) is already required unconditionally by
`curl/curl.h`. It remains used in feature checks by both autotools and
cmake; to be reviewed in a future step.
Closes#22374
When the threaded resolver gets AAAA results that carry a link-local
address without scope-id, it now re-queues a query with AF_UNSPEC and
strips ipv4 addresses from that result. Whatever the resulting addresses
and scope-ids are, this becomes the result of the resolve.
Fixes#22330
Reported-by: Bartel Sielski
Closes#22368
Add call stacks to easy and multi instances. Record ongoing API calls
and callback invocations there to detect recursion and not allowed
invocations.
Define enums for easy, multi and callbacks in `api.h`. In `api.c` define
properties for these functions:
- can they recurse
- is the easy/multi handle destroyed during the call or should it be
good afterwards
- is the call allowed when a multi event callback is ongoing
- is the call allowed when a notification callback is ongoing
Entering a guard
- checks that passed CURL*/CURLM* are GOOD on entering
- checks that easy handle's `mid` is correct and it is known
for it in the multi.
- checks that call properties are obeyed (recursion, callback checks)
- checks that passed CURL*/CURLM* are GOOD on leaving, unless call is
known to kill it
Checks for ongoing callbacks inspect the whole call stack and catches
nested invocations (which our current flags can not).
Call stacks in easy/multi handle are fixed size and will deny recursion
when the limit is reached. The current limits are 7 for easy and 15 for
multi now.
Removes:
- multi->in_callback, check is done via call stack
- multi->in_ntfy_cb, check is done via call stack
The overhead in my tests seems minimal, if noticeable at all.
Closes#22237
Use separate dns cache entries for addresses (A+AAAA) and HTTPS-RR
results. That makes also "negative" results independent of each other.
Dns cache entries, once in use, can no longer be modified safely, as
concurrent use would require each access then to be done under lock. By
using separate entries, we can update a HTTPS entry without needing to
duplicate an existing address entry for the same host+port.
Connection filters can now ask for DNS resolves. This works at any time
during connection setup and while connect is going on.
`Curl_conn_dns_add_addr_resolve()` and
`Curl_conn_dns_add_https_resolve()` are used for that. They check if the
resolve is already ongoing, can be added to a resolve not started yet
or, as last resort, create a new filter instance and add it to the
connection (it's easier to add more filters than making the same filter
handle multiple resolves. Since DNS filters are removed once the
connection is established, there is no later penalty).
HTTPS-RR queries are added by the `HTTPS-CONNECT`, `SSL` and `QUIC`
filters. The latter will only do that when ECH is configured and
supported. That means we trigger HTTPS-RR queries only when the results
matter.
Add test_22_06 for ALPN influenced via HTTPS-RR when tunneling through a
proxy. This did not work before.
Adjust test2100 to use https: as the previous http: URL no longer
triggers HTTPS-RR resolves.
Closes#22216
- Adjust pollset during connection shutdown.
- Separate the FIRSTSOCKET and SECONDSOCKET check so that one being in
connect or shutdown no longer adds poll events for the other.
- Fix shutdown state evaluation (as detailed below).
- Add a unit test for Curl_conn_adjust_pollset.
- Add a client for event testing.
- Test that shutdown sockets stay with the socket callback until done.
The pollset predicate now reads the connection's own shutdown state
rather than going through data->conn, and the EXPIRE_SHUTDOWN arming in
cshutdn_perform() works again: next_expire_ms started at 0 and could
never be set, so a peer that never sends a close_notify would still park
its connection forever, timeout or not.
Reported-by: pszemus@users.noreply.github.com
Fixes https://github.com/curl/curl/issues/22282
Closes https://github.com/curl/curl/pull/22304
E.g. on CHERI pointers are 128 bits [1]. This causes the unit3214 test
to fail, which was written with more traditional platforms in mind. Here
is the output of log/stderr3214:
```
URL: -
BAD: struct Curl_easy is 7984 bytes, allowed to be 5370: 2614 bytes too big
BAD: struct connectdata is 1408 bytes, allowed to be 1300: 108 bytes too big
BAD: struct Curl_multi is 1248 bytes, allowed to be 850: 398 bytes too big
BAD: struct curl_httppost is 224 bytes, allowed to be 112: 112 bytes too big
BAD: struct curl_slist is 32 bytes, allowed to be 16: 16 bytes too big
BAD: struct curl_khkey is 32 bytes, allowed to be 24: 8 bytes too big
BAD: struct curl_hstsentry is 48 bytes, allowed to be 40: 8 bytes too big
BAD: struct curl_mime is 144 bytes, allowed to be 96: 48 bytes too big
BAD: struct curl_mimepart is 592 bytes, allowed to be 440: 152 bytes too big
BAD: struct curl_certinfo is 32 bytes, allowed to be 16: 16 bytes too big
BAD: struct curl_tlssessioninfo is 32 bytes, allowed to be 16: 16 bytes too big
BAD: struct curl_blob is 32 bytes, allowed to be 24: 8 bytes too big
BAD: struct CURLMsg is 48 bytes, allowed to be 24: 24 bytes too big
BAD: struct curl_header is 80 bytes, allowed to be 48: 32 bytes too big
Test ended with result 14
```
Multiply the allowed size on systems with larger than 64-bit pointers.
[1] https://www.cl.cam.ac.uk/research/security/ctsrd/pdfs/20171017a-cheri-poster.pdfCloses#22299
Also repurpose existing build-time feature checks into unit test 1961,
to verify.
Prior to this patch these functions were auto-detected with both
autotools and cmake. In case of autotools there was an extra
verification phase ensuring the functions work as expected. This step
required running the function, thus was limited to non-cross-builds. For
cross-builds and CMake it always used the system implementation if
present. On Windows it always used the local implementation, because
availability/use is complicated there.
After this patch all platforms, always use the local implementation,
which is known to be accurate. This makes curl behave more consistently,
and simplifies the build process, a fixes cross-builds and CMake
auto-detection differences.
Also:
- test1960: enable unconditionally.
- checksrc: disallow globally, allowlist in `block_ip.c` example.
- dnsd: verify ntop result for NULL before passing to printf.
Ref: https://github.com/curl/curl/pull/22137#issuecomment-4797440983
Ref: #22137
Ref: 8537a5b0bc#16577Closes#22170
- cmp-pkg-config.sh: replace `-r -f` with `-rf` to match rest of repo.
- configure.ac: add double quotes for robustness (not a bug).
- curl-openssl.m4: merge nested `if`s.
- CurlTests.c: drop `!= 0`, also to sync with m4.
- CurlTests.c: replace `example.com` with `localhost` in
`gethostbyname()` feature test code. (compile-only, not a bug)
- GHA/http3-linux: drop literal `true` from bool expression.
- lib650: drop redundant `&`.
- move variable/call to left-hand side of equality checks, where
missing.
- perl: detach `<`/`>` from filename in `open()`, where missing.
- schannel: apply two nit fixes lost in rebase.
- scripts/verify-release: drop redundant double quotes.
- scripts/verify-release: exit with error code on error.
- synctime: replace magic numbers with `sizeof()`.
- telnet: add missing parentheses to macro value.
- tests/Makefile.am: use single quotes.
- tool_operate: drop redundant `break` after `return` in VMS code.
- unit2413: drop unused NULL pointer + free call.
- unit2413: fix duplicate test case name.
- urlapi: drop redundant parentheses.
- urlapi: drop `CURL_UNCONST()` that became redundant.
Closes#22186
- GHA/windows: mention `IgnoreStandardErrorWarningFormat=true`
in comment.
- cmake: fix MIT/GNU GSS order in messages.
- drop some exclamation marks from messages.
- drop redundant ending newlines from messages.
- fold/unfold where possible.
- fix indent, whitespace, typos and other nits.
Closes#22185
If any sublabel is longer than 63 octets, abort. This then also catches
compression attempts.
Verified in test 1658
Reported-by: netspacer.research
Closes#22124
To avoid mistakes or abuse to cause problems. Many public DNS providers
cap their cache times to this.
Verify in test 1650
Reported-by: netspacer.research
Closes#22122
- `N byte` -> `N-byte` or `N bytes`.
- INTERNALS.md: language tweaks.
- schannel: language tweak in comment/error message.
- socks_gssapi, socks_sspi: simplify composing an error message.
(at a cost of 8 extra constant string bytes.)
- m4/curl-compilers.m4: fix typo in link (in comment).
- contrithanks.sh: fix indent, drop stray `;` terminator.
- lib, src, tests: drop/fix a bunch of badwords.
- fix typos in comments.
- fix indent, stray spaces.
Some of these spotted by GitHub Code Quality and Copilot
Closes#22009
Add `data->state.origin` as the origin the transfer is sending the
current request to/gets the response from. Use it for request specific
properties like authentication, hsts and cookie handling, etc.
Unless talking to a forwarding HTTP proxy (e.g. not tunneling),
`data->state.origin` and `conn->origin` are the same.
With a forwarding HTTP proxy in play, `conn->origin` is set to
`conn->http_proxy.peer` and `conn->bits.origin_is_proxy` (a new bit) is
set.
Remove the connection bits, now replaced with:
* `conn->bits.socksproxy` -> `conn->socks_proy.peer`
* `conn->bits.httpproxy` -> `conn->http_proy.peer`
* `conn->bits.proxy` -> `(conn->socks_proy.peer || conn->http_proy.peer`)
* `conn->bits.tunnel_proxy` -> (`conn->http_proy.peer && !conn->bits.origin_is_proxy`)
* `(conn->bits.httpproxy && !conn->bits.tunnel_proxy)` -> `conn->bits.origin_is_proxy`
Rename `noproxy.[ch]` to `proxy.[ch]`. Move the connection proxy setup
code from `url.c` to `proxy.c`.
Remove `data->info.conn_remote_port` as no one uses it.
Add test_40_02b for a SOCKS connection to a forwarding HTTPS proxy.
Update internal documentation about peers and creds.
Closes#21967
Move as sockaddr2string() into cf-socket.c where its only callers are.
Mark as UNITTEST for unit1609.
Move "struct Curl_sockaddr_ex" into sockaddr.h, so connect.h and
cf-socket.h can be included without all the system headers needed.
Closes#21946
Adjust code to avoid `-Wformat-signedness` warnings, while making sure
that enums are always cast to a known type when passing them to `printf`
functions, to support compilers and compiler settings where enums are
not default-size signed ints.
- cast integers printed as hex to `unsigned`. (63 times, 20 of them in
`mbedtls.c`)
- cast misc enums to `int` for printing. (31 times)
- cast `CURL_LOCK_DATA_*` enums to `int`. (4 times)
- cast `CURL_FORMADD_*` enums to `int`. (13 times)
- cast `CURLSHE_*` enums to `int`. (3 times)
- cast `CURLUE_*` enums to `int`. (33 times)
- cast `CURLMSG_*` enums to `int`. (6 times)
- cast `CURLE_*` enums to `int`. (~380 times)
- unit1675: fix mask.
Follow-up to 7c34365cce#21879
Ref: #18343 (initial attempt)
Closes#20848
Share common functions/structs between ngtcp2 HTTP/3 and the proxy
version.
Fix bugs in proxy implementation when it comes to stream and pollset
handling and transfer lifetimes.
Curl_multi_xfer_sockbuf_borrow: work without multi
When a connection gets shutdown by a share, the easy handle used is
share->admin and it does not have a multi handle. In that case let
Curl_multi_xfer_sockbuf_borrow() allocate a buffer to be freed on
release.
This happens when a TLS filter sends its last notify through a HTTP/3
proxy tunnel.
Closes#21871
