Move curlx_ functions into its own subdir.
The idea is to use the curlx_ prefix proper on these functions, and use
these same function names both in tool, lib and test suite source code.
Stop the previous special #define setup for curlx_ names.
The printf defines are now done for the library alone. Tests no longer
use the printf defines. The tool code sets its own defines. The printf
functions are not curlx, they are publicly available.
The strcase defines are not curlx_ functions and should not be used by
tool or server code.
dynbuf, warnless, base64, strparse, timeval, timediff are now proper
curlx functions.
When libcurl is built statically, the functions from the library can be
used as-is. The key is then that the functions must work as-is, without
having to be recompiled for use in tool/tests. This avoids symbol
collisions - when libcurl is built statically, we use those functions
directly when building the tool/tests. When libcurl is shared, we
build/link them separately for the tool/tests.
Assisted-by: Jay Satiro
Closes#17253
With a meta_hash at each connection (similar to easy handle, let
multi_ev.c store its pollsets as meta data, no longer needing its own
hashes.
This eliminates the last use of Curl_hash_offt. Remove it.
Closes#17095
Change multi's book keeping of transfers to no longer use lists, but a
special table and bitsets for unsigned int values.
`multi-xfers` is the `uint_tbl` where `multi_add_handle()` inserts a new
transfer which assigns it a unique identifier `mid`. Use bitsets to keep
track of transfers that are in state "process" or "pending" or
"msgsent".
Use sparse bitsets to replace `conn->easyq` and event handlings tracking
of transfers per socket. Instead of pointers, keep the mids involved.
Provide base data structures and document them in docs/internal:
* `uint_tbl`: a table of transfers with `mid` as lookup key,
handing out a mid for adds between 0 - capacity.
* `uint_bset`: a bitset keeping unsigned ints from 0 - capacity.
* `uint_spbset`: a sparse bitset for keeping a small number of
unsigned int values
* `uint_hash`: for associating `mid`s with a pointer.
This makes the `mid` the recommended way to refer to transfers inside
the same multi without risk of running into a UAF.
Modifying table and bitsets is safe while iterating over them. Overall
memory requirements are lower as with the double linked list apprach.
Closes#16761
Adds a "meta_hash" to each easy handle for keeping special data during
operations. All meta data set needs to add its destructor callback, so
that meta data gets destroyed properly when the easy handle is cleaned
up or reset.
Add data->master_mid for "sub" transfers that belong to a "master" easy
handle. When a "sub" transfer is done, the corresponding "master" can
add a callback to be invoked. Used in DoH name resolution.
DoH: use easy meta hash to add internal structs for DoH name resolution.
One in each in each probe easy handle. When probes are done, response
data is copied from the probe to the initiating easy.
This allows DoH using transfers and their probes to be cleaned up in any
sequence correctly.
Fold DoH cleanup into the Curl_async_shutdown() and Curl_async_destroy()
functions.
Closes#16384
"asyn" is the internal name under which both c-ares and threaded
resolver operate. Make the naming more consistent. Implement the c-ares
resolver in `asyn-ares.*` and the threaded resolver in `asyn-thrdd.*`.
The common functions are in `asyn-base.c`.
When `CURLRES_ASYNCH` is defined, either of the two is used and
`data->state.async` exists. Members of that struct vary for the selected
implementation, but have the fields `hostname`, `port` and `ip_version`
always present. This are populated when the async resolving starts and
eliminate the need to pass them again when checking on the status and
processing the results of the resolving.
Add a `Curl_resolv_blocking()` to `hostip.h` that relieves FTP and SOCKS
from having to repeat the same code.
`Curl_resolv_check()` remains the function to check for status of
ongoing resolving. Now it also performs internally the check if the
needed DNS entry exists in the dnscache and if so, aborts the asnyc
operation. (libcurl right now does not check for duplicate resolve
attempts. an area for future improvements).
The number of functions in `asyn.h` has been reduced. There were subtle
difference in "cancel()" and "kill()" calls, both replaced by
`Curl_async_shutdown()` now. This changes behaviour for threaded
resolver insofar as the resolving thread is now always joined unless
`data->set.quick_exit` is set. Before this was only done on some code
paths. A future improvement would be a thread pool that keeps a limit
and also could handle joins more gracefully.
DoH, not previously tagged under "asny", has its struct `doh_probes` now
also in `data->state.async`, moved there from `data->req` because it
makes more sense. Further integration of DoH underneath the "asyn"
umbrella seems like a good idea.
Closes#16963
Made the HTTPS-RR parser a little stricter while at it.
Drop the ALPN escape handling, that was not needed.
Make the hode handle (and ignore) duplicate ALPN entries.
Closes#16972
Slight refactoring around dnscache, e.g. hostcache
- eliminate `data->state.hostcache`. Always look up
relevant dnscache at share/multi.
- unify naming to "dnscache", replacing "hostcache"
- use `struct Curl_dnscache`, even though it just
contains a `Curl_hash` for now.
- add `Curl_dnscache_destroy()` for cleanup in
share/multi.
Closes#16941
The condition could not happen, as the function is only called from a
single place where the caller already made sure it can't happen. This
change still removes the flawed logic.
Reported-by: Ronald Crane
Closes#16710
Fixed a heap read overflow when parsing the HTTP RR svcparams. Also the
code failed to enforce the requirements of SvcParamKey order specified
in section 2.2 of the RFC 9460.
Closes#16598
- fold DoH and async HTTPS-RR handling into common code.
have common cleanups, etc. Have a CURLcode result in async
handling to allow HTTPS RR parsing to fail.
- keep target, ipv4hints, ipv6hints, port and echconfig also
when resolving via cares. We need to know `target` and `port`
when evaluating possible ALPN candidates to not go astray.
- add CURL_TRC_DNS for tracing DNS operations
- replace DoH specific tracing with DNS, use doh as alias
for dns in curl_global_tracea()
Closes#16132
To reduce the number `-Wunused-macro` compiler warnings:
- delete unused macros.
- comment out unused macro that are part of a set.
- move macros into the scope they are used.
This may be useful to enable by default, but there are tricky cases that
I didn't manage to fix and paused the effort. E.g. internal features
checks in `openssl.c`. There is more, once those are fixed.
Closes#16279
Improved the filter implementation to be flexible in which order h3 and
h2/h1 are attempted. When HTTPSRR is enabled, look at the ALPNs it found
and use the order given for connecting in default setups.
Closes#16012
When enabled in the build.
Update test2100: verify with HTTPS RR included
Adjust runtests and server/disabled.c to include "HTTPSRR" as a feature
in the test suite.
Also, decode the ALPN list in HTTPS records straight into IDs. There's
no point in storing everything in string format. Skip ALPNs we do not
support.
Closes#16007
`Makefile.mk` supported MS-DOS and Amiga, but `./configure` also
supported them in a better tested and more flexible way.
This patch also adds CMake support for MS-DOS/DJGPP and Amiga OS 3.
`Makefile.mk` was not maintained. Delete it in favour of first-tier
build methods.
Also include some non-MS-DOS/AmigaOS-specific tidy-up, see details at
the end of this message.
Details:
- fix/silence all MS-DOS/DJGPP build warnings and issues.
- add MS-DOS support to cmake.
- default to `ENABLE_THREADED_RESOLVER=OFF` for MS-DOS.
- add support for `WATT_ROOT`.
- use static libcurl with MS-DOS.
- fixup default CMake suffixes/prefixes for DJGPP.
- disable hidden symbols for MS-DOS. Not supported on MS-DOS.
- opt-in MS-DOS into `USE_UNIX_SOCKETS`.
- improve MS-DOS support in autotools.
- default to `--disable-threaded-resolver` for MS-DOS.
- make sure to use `close_s()` (from Watt-32) with autotools and cmake.
`Makefile.mk` used it before this patch.
- GHA: add DJGPP cmake (~30s) and autotools (~60s) build jobs.
Also build tests and examples with cmake.
- improve AmigaOS support in autotools:
- configure: detect `CloseSocket()` when it's a macro.
- configure: fix `IoctlSocket` detection on AmigaOS.
- curl-amissl.m4: pass AmiSSL libs to tests/servers.
- add AmigaOS3 support to cmake:
- cmake: fix `HAVE_IOCTLSOCKET_CAMEL` and
`HAVE_IOCTLSOCKET_CAMEL_FIONBIO` detections.
- set necessary system libs.
- add AmiSSL support.
- inet_ntop, inet_pton: fix using it for AmigaOS. cmake detects them,
and they did not compile with AmigaOS.
- cmake: better sync `gethostname` detection with autotools.
Fixes detection for AmigaOS, where `gethostname` is a macro.
- cmake: fix `sys/utime.h` detection on AmigaOS.
- cmake: force-disable `getaddrinfo` for AmigaOS.
- cmake: tweak threading and static/shared default for AmigaOS.
- cmake: rely on manual variable `AMIGA` to enable the platform.
- GHA: add AmigaOS cmake and autotools (~45s) jobs.
Also build tests and examples with cmake.
- INSTALL: update MS-DOS and AmigaOS build instructions.
- amigaos: fix `-Wpointer-sign` and
`zero or negative size array '_args'` in `Printf()`.
- amigaos: fix `-Wpointer-sign`
- amigaos: fix `-Wredundant-decls` `errno` and `h_errno`.
- amigaos: brute-force silence `lseek()` size warnings.
- amigaos: server/resolve: silence `-Wdiscarded-qualifiers`.
- amigaos: server/resolve: fix `-Wpointer-sign`.
- amigaos: fix `CURL_SA_FAMILY_T` type.
- nonblock: prefer `HAVE_IOCTLSOCKET_CAMEL_FIONBIO` for AmigaOS.
`ioctl` is also detected, but fails when used. Make the above override
it for a successful build.
Authored-by: Darren Banfi
Fixes#15537Closes#15603
- tftpd: prefer `HAVE_IOCTLSOCKET_CAMEL_FIONBIO` for AmigaOS.
- tftpd: tidy-up conditional code.
- curl: set stack size to 16384 for AmigaOS3/4
Overriding the default 4096.
Suggested-by: Darren Banfi
Ref: https://github.com/curl/curl/pull/15543#issuecomment-2498783123
Ref: https://wiki.amigaos.net/wiki/Controlling_Application_Stack
- functypes.h: fix `SEND_QUAL_ARG2` for AmigaOS.
- tftp: add missing cast in sendto() call for AmigaOS.
- getinfo: fix warning with AmigaOS.
- tool_operate: silence warning with AmigaOS
- amigaos: fix building libtests due to missing `RLIMIT_NOFILE`.
- curl_gethostname: silence warning for AmigaOS.
- ftp: silence `-Wtype-limits` for AmigaOS.
- libtest: fix timeval initialization for AmigaOS.
- examples: fix `timeval` initialization for AmigaOS.
- examples: silence warning for AmigaOS.
- configure: fix IPv6 detection for cross-builds.
- netrc: fix to build with AmigaOS cleanly.
- buildinfo: detect and add `DOS` tag for MS-DOS builds.
- buildinfo: add `AMIGA` to buildinfo.txt in auttools.
- build: move `USE_WATT32` macro definition to cmake/configure.
Non-MS-DOS/AmigeOS-specific tidy-ups:
- configure: sync `sa_family_t` detection with cmake.
- configure: sync `ADDRESS_FAMILY` detection signals with cmake.
- doh: use `CURL_SA_FAMILY_T`.
- lib: drop mingw-specific `CURL_SA_FAMILY_T` workaround.
- cmake: extend instead of override check-specific
configurations/requirements.
This allows to honor global requirements added earlier.
Necessary for AmigaOS for example.
- cmake: omit warning on disabled IPv6 for MS-DOS and AmigaOS.
No IPv6 support on these platforms. Also sync with autotools.
- lib1960: use libcurl `inet_pton()` wrapper.
- cmake: detect LibreSSL (to match autotools).
- cmake: say the specific OpenSSL flavour detected.
- hostip: add missing `HAVE_SOCKADDR_IN6_SIN6_SCOPE_ID` guard.
- lib: simplify classic mac feature guards.
Follow-up to a8861b6ccd#9764Closes#15543
It makes the callbacks get different signnatures when used from within
libcurl vs outside of it by libcurl-using applications (such as the
libtests) and this triggers UndefinedBehaviorSanitizer errors.
Closes#15289
booleans should use the type 'bool' and set the value to TRUE/FALSE
non-booleans should not be 'bool' and should not set the value to
TRUE/FALSE
Closes#15123
`data->id` is unique in *most* situations, but not in all. If a libcurl
application uses more than one connection cache, they will overlap. This
is a rare situations, but libcurl apps do crazy things. However, for
informative things, like tracing, `data->id` is superior, since it
assigns new ids in curl's serial curl_easy_perform() use.
Introduce `data->mid` which is a unique identifer inside one multi
instance, assigned on multi_add_handle() and cleared on
multi_remove_handle().
Use the `mid` in DoH operations and also in h2/h3 stream hashes.
Reported-by: 罗朝辉
Fixes#14414Closes#14499
Replace Curl_resolv_unlock() with Curl_resolv_unlink():
-replace inuse member with refcount in Curl_dns_entry
- pass Curl_dns_entry ** to unlink, so it gets always cleared
- solve potential (but unlikley) UAF in FTP's handling of looked up
Curl_dns_entry. Esp. do not use addr information after unlinking an entry.
In reality, the unlink will not free memory, as the dns entry is still
referenced by the hostcache. But this is not safe and relying on no other
code pruning the cache in the meantime.
- pass permanent flag when adding a dns entry instead of fixing timestamp
afterwards.
url.c: fold several static *resolve_* functions into one.
Closes#14195
When removing an easy handle that had DoH sub-easy handles going, those
were not removed from the multi handle. Their memory was reclaimed on
curl_easy_cleanup() of the owning handle, but multi still had them in
their list.
Add `Curl_doh_close()` and `Curl_doh_cleanup()` as common point for
handling the DoH resource management. Use the `multi` present in the doh
handles (if so), for removal, as the `data->multi` might already have
been NULLed at this time.
Reported-by: 罗朝辉
Fixes#14207Closes#14212
This PR fixes a leak and a crash that can happen when curl encounters
bad HTTPS RR values in DNS. We're starting to do better testing of that
kind of thing and e.g. have published bad HTTPS RR values at
dodgy.test.defo.ie.
Closes#14151
Based on the standards and guidelines we use for our documentation.
- expand contractions (they're => they are etc)
- host name = > hostname
- file name => filename
- user name = username
- man page => manpage
- run-time => runtime
- set-up => setup
- back-end => backend
- a HTTP => an HTTP
- Two spaces after a period => one space after period
Closes#14073
`CURLDEBUG` is meant to enable memory tracking, but in a bunch of cases,
it was protecting debug features that were supposed to be guarded with
`DEBUGBUILD`.
Replace these uses with `DEBUGBUILD`.
This leaves `CURLDEBUG` uses solely for its intended purpose: to enable
the memory tracking debug feature.
Also:
- autotools: rely on `DEBUGBUILD` to enable `checksrc`.
Instead of `CURLDEBUG`, which worked in most cases because debug
builds enable `CURLDEBUG` by default, but it's not accurate.
- include `lib/easyif.h` instead of keeping a copy of a declaration.
- add CI test jobs for the build issues discovered.
Ref: https://github.com/curl/curl/pull/13694#issuecomment-2120311894Closes#13718
- make local_decode_rdata_name use dynbuf instead of calloc + memcpy
- avoid extra memdup in local_decode_rdata_alpn
- no need to if() before free()
- use memdup instead of calloc + memcpy in Curl_doh_decode_httpsrr
Reviewed-by: Stephen Farrell
Closes#13526
Before this patch, two macros were used to guard IPv6 features in curl
sources: `ENABLE_IPV6` and `USE_IPV6`. This patch makes the source use
the latter for consistency with other similar switches.
`-DENABLE_IPV6` remains accepted for compatibility as a synonym for
`-DUSE_IPV6`, when passed to the compiler.
`ENABLE_IPV6` also remains the name of the CMake and `Makefile.vc`
options to control this feature.
Closes#13349
- refs #12397 where it is dicussed how to en-/disable verbose output
of DoH operations
- introducing `struct curl_trc_feat` to track a curl feature for
tracing
- adding `data->state.feat` optionally pointing to the feature a
transfer belongs to
- adding trace functions and verbosity checks on features
- using trace feature in DoH code
- documenting `doh` as feature for `--trace-config`
Closes#12411
- enable `-Wsign-conversion` warnings, but also setting them to not
raise errors.
- fix `-Warith-conversion` warnings seen in CI.
These are triggered by `-Wsign-converion` and causing errors unless
explicitly silenced. It makes more sense to fix them, there just a few
of them.
- fix some `-Wsign-conversion` warnings.
- hide `-Wsign-conversion` warnings with a `#pragma`.
- add macro `CURL_WARN_SIGN_CONVERSION` to unhide them on a per-build
basis.
- update a CI job to unhide them with the above macro:
https://github.com/curl/curl/actions/workflows/linux.yml -> OpenSSL -O3
Closes#12492
Previously it was wrongly returning CURLE_OUT_OF_MEMORY when the
response did not contain any addresses. Now it more accurately returns
CURLE_COULDNT_RESOLVE_HOST.
Reported-by: lRoccoon on github
Fixes#12365Closes#12366
GCC 14 introduces a new -Walloc-size included in -Wextra which gives:
```
src/tool_operate.c: In function ‘add_per_transfer’:
src/tool_operate.c:213:5: warning: allocation of insufficient size ‘1’ for type ‘struct per_transfer’ with size ‘480’ [-Walloc-size]
213 | p = calloc(sizeof(struct per_transfer), 1);
| ^
src/var.c: In function ‘addvariable’:
src/var.c:361:5: warning: allocation of insufficient size ‘1’ for type ‘struct var’ with size ‘32’ [-Walloc-size]
361 | p = calloc(sizeof(struct var), 1);
| ^
```
The calloc prototype is:
```
void *calloc(size_t nmemb, size_t size);
```
So, just swap the number of members and size arguments to match the
prototype, as we're initialising 1 struct of size `sizeof(struct
...)`. GCC then sees we're not doing anything wrong.
Closes#12292
- test for a simplified C99 variadic check
- args to infof() in --disable-verbose are no longer disregarded but
must compile.
Closes#12167Fixes#12083Fixes#11880Fixes#11891
- resolving is done for a connection, not for every transfer
- save create/dup/free of a cares channel for each transfer
- check values of setopt calls against a local channel if no
connection has been attached yet, when needed.
Closes#12198
