This function REQUIRES the size of the target buffer as well as the
length of the source string. Meant to make it harder to do a bad
strcpy().
Removes 23 calls to strcpy().
Closes#20067
- replace `sendf.h` with `curl_trc.h` where it was included just for it.
- drop unused `curl_trc.h` includes.
- easy: delete obsolete comment about `send.h` include reason.
Also:
- move out `curl_trc.h` include from `sendf.h` and include it directly
in users, where not done already. To flatten the include tree and
to less rely on indirect includes.
- stop including `sendf.h` from other headers, replace it with forward
declaration of `Curl_easy`, as done already elsewhere.
Verified with an all non-unity CI run.
Closes#20061
To make it available for all files. Drop includes from individual
sources. This header was already included from most sources and not
specific to any internal subsystem.
Also to ensure that two system symbol redefines on Windows (`read()` and
`write()`) get applied to all sources. Move them to `curl_setup.h`.
Closes#20056
Always use curlx_now() when calling Curl_pgrs_now(data). Tests with the
"manual" updates to now proved differ more then 100ms in parallel testing.
Add `curlx_nowp()` to set current time into a struct curltime.
Add `curlx_ptimediff_ms() and friends, passing pointers.
Update documentation.
Closes#19998
Use `data->progress.now` as the timestamp of proecssing a transfer.
Update it on significant events and refrain from calling `curlx_now()`
in many places.
The problem this addresses is
a) calling curlx_now() has costs, depending on platform. Calling it
every time results in 25% increase `./runtest` duration on macOS.
b) we used to pass a `struct curltime *` around to save on calls, but
when some method directly use `curx_now()` and some use the passed
pointer, the transfer experienes non-linear time. This results in
timeline checks to report events in the wrong order.
By keeping a timestamp in the easy handle and updating it there, no
longer invoking `curlx_now()` in the "lower" methods, the transfer
can observer a steady clock progression.
Add documentation in docs/internals/TIME-KEEPING.md
Reported-by: Viktor Szakats
Fixes#19935Closes#19961
Before this patch curl used the C preprocessor to override standard
memory allocation symbols: malloc, calloc, strdup, realloc, free.
The goal of these is to replace them with curl's debug wrappers in
`CURLDEBUG` builds, another was to replace them with the wrappers
calling user-defined allocators in libcurl. This solution needed a bunch
of workarounds to avoid breaking external headers: it relied on include
order to do the overriding last. For "unity" builds it needed to reset
overrides before external includes. Also in test apps, which are always
built as single source files. It also needed the `(symbol)` trick
to avoid overrides in some places. This would still not fix cases where
the standard symbols were macros. It was also fragile and difficult
to figure out which was the actual function behind an alloc or free call
in a specific piece of code. This in turn caused bugs where the wrong
allocator was accidentally called.
To avoid these problems, this patch replaces this solution with
`curlx_`-prefixed allocator macros, and mapping them _once_ to either
the libcurl wrappers, the debug wrappers or the standard ones, matching
the rest of the code in libtests.
This concludes the long journey to avoid redefining standard functions
in the curl codebase.
Note: I did not update `packages/OS400/*.c` sources. They did not
`#include` `curl_setup.h`, `curl_memory.h` or `memdebug.h`, meaning
the overrides were never applied to them. This may or may not have been
correct. For now I suppressed the direct use of standard allocators
via a local `.checksrc`. Probably they (except for `curlcl.c`) should be
updated to include `curl_setup.h` and use the `curlx_` macros.
This patch changes mappings in two places:
- `lib/curl_threads.c` in libtests: Before this patch it mapped to
libcurl allocators. After, it maps to standard allocators, like
the rest of libtests code.
- `units`: before this patch it mapped to standard allocators. After, it
maps to libcurl allocators.
Also:
- drop all position-dependent `curl_memory.h` and `memdebug.h` includes,
and delete the now unnecessary headers.
- rename `Curl_tcsdup` macro to `curlx_tcsdup` and define like the other
allocators.
- map `curlx_strdup()` to `_strdup()` on Windows (was: `strdup()`).
To fix warnings silenced via `_CRT_NONSTDC_NO_DEPRECATE`.
- multibyte: map `curlx_convert_*()` to `_strdup()` on Windows
(was: `strdup()`).
- src: do not reuse the `strdup` name for the local replacement.
- lib509: call `_strdup()` on Windows (was: `strdup()`).
- test1132: delete test obsoleted by this patch.
- CHECKSRC.md: update text for `SNPRINTF`.
- checksrc: ban standard allocator symbols.
Follow-up to b12da22db1#18866
Follow-up to db98daab05#18844
Follow-up to 4deea9396b#18814
Follow-up to 9678ff5b1b#18776
Follow-up to 10bac43b87#18774
Follow-up to 20142f5d06#18634
Follow-up to bf7375ecc5#18503
Follow-up to 9863599d69#18502
Follow-up to 3bb5e58c10#17827Closes#19626
- Curl_async_getaddrinfo() always returned NULL so it was pointless.
Return proper curlcode instead to distinguish between errors. Same for
Curl_doh().
- simplify the IP address handling
- make Curl_str2addr() function return CURLcode
Closes#19669
Windows CRTs have a `share.h`. Before this patch when trying to
`#include <share.h>` it, the compiler picked up curl's internal
`lib/share.h` instead. Rename it to avoid this issue.
CRT `share.h` has constants necessary for using safe open CRT functions.
Also rename `lib/share.c` to keep matching the header.
Ref: https://learn.microsoft.com/cpp/c-runtime-library/sharing-constants
Ref: 625f2c1644#16949#16991
Cherry-picked from #19643Closes#19676
Windows CE support was limited to successful builds with ming32ce
(a toolchain that hasn't seen an update since 2009, using an ancient gcc
version and "old mingw"-style SDK headers, that curl deprecated earlier).
Builds with MSVC were broken for a long time. mingw32ce builds were never
actually tested and runtime and unlikely to work due to missing stubs.
Windows CE toolchains also miss to comply with C89. Paired with lack of
demand and support for the platform, curl deprecated it earlier.
This patch removes support from the codebase to ease maintaining Windows
codepaths.
Follow-up to f98c0ba834#17924
Follow-up to 8491e6574c#17379
Follow-up to 2a292c3984#15975Closes#17927
- badwords.pl: add `-a` option to check all lines in source code files.
Before this patch indented lines were skipped (to avoid Markdown code
fences.)
- GHA/checksrc: use `-a` when verifying the source code.
- GHA/checksrc: disable `So` and `But` rules for source code.
- GHA/checksrc: add docs/examples to the verified sources.
- badwords.txt: delete 4 duplicates.
- badwords.txt: group and sort contractions.
- badwords.txt: allow ` url = `, `DIR`, `<file name`.
Closes#19536
Rename `Curl_timeleft()` to `Curl_timeleft_ms()` to make the units in
the returned `timediff_t` clear. (We used to always have ms there, but
with QUIC started to sometimes calc ns as well).
Rename some assigned vars without `_ms` suffix for clarity as well.
Closes#19486
For all network related protocols there must be a non-blank hostname
used. This change adds a few asserts in some places to make debug/tests
catch mistakes if any such would slip in.
Closes#19146
After this patch, the codebase no longer overrides system printf
functions. Instead it explicitly calls either the curl printf functions
`curl_m*printf()` or the system ones using their original names.
Also:
- drop unused `curl_printf.h` includes.
- checksrc: ban system printf functions, allow where necessary.
Follow-up to db98daab05#18844
Follow-up to 4deea9396b#18814Closes#18866
Before this patch `accept4()`, `socket()`, `socketpair()`, `send()` and
`recv()` system symbols were remapped via macros, using the same name,
to local curl debug wrappers. This patch replaces these overrides by
introducing curl-namespaced macros that map either to the system symbols
or to their curl debug wrappers in `CURLDEBUG` (TrackMemory) builds.
This follows a patch that implemented the same for `accept()`.
The old method required tricks to make these redefines work in unity
builds, and avoid them interfering with system headers. These tricks
did not work for system symbols implemented as macros.
The new method allows to setup these mappings once, without interfering
with system headers, upstream macros, or unity builds. It makes builds
more robust.
Also:
- checksrc: ban all mapped functions.
- docs/examples: tidy up checksrc rules.
Follow-up to 9863599d69#18502
Follow-up to 3bb5e58c10#17827Closes#18503
Remove use of pthread_cancel in asnyc threaded resolving. While there
are system where this works, others might leak to resource leakage
(memory, files, etc.). The popular nsswitch is one example where resolve
code can be dragged in that is not prepared.
The overall promise and mechanism of pthread_cancel() is just too
brittle and the historcal design of getaddrinfo() continues to haunt us.
Fixes#18532
Reported-by: Javier Blazquez
Closes#18540
Add optional detail to `Curl_resolver_error()` to add to failure message
where available. This makes, for c-ares, the reason for a failed
resource available to the user without extra trace config.
When "dns" tracing enabled, print the c-ares server config at the start
of a resolve.
Closes#18251
Changed strategy to start up and terminate resolver thread.
When starting up:
Start the thread with mutex acquired, wait for signal from thread that
it started and has incremented the ref counter. Thread set
pthread_cancel() to disabled before that and only enables cancelling
during resolving itself. This assure that the ref counter is correct and
the unlinking of the resolve context always happens.
When shutting down resolving:
If ref counting shows thread has finished, join it, free everything. If
thread has not finished, try pthread_cancel() (non Windows), but keep
the thread handle around.
When destroying resolving:
Shutdown first, then, if the thread is still there and 'quick_exit' is
not set, join it and free everything. This might occur a delay if
getaddrinfo() hangs and cannot be interrupted by pthread_cancel().
Destroying resolving happens when another resolve is started on an
easy handle or when the easy handle is closed.
Add test795 to check that connect timeout triggers correctly
when resolving is delayed. Add debug env var `CURL_DNS_DELAY_MS`
to simulate delays in resolving.
Fix test1557 to set `quick_exit` and use `xxx.invalid` as domain
instead of `nothing` that was leading to hangers in CI.
Closes#18263
* Add more tracing information to c-ares errors.
* remove CURL_ASYNC_SUCCESS, rename `ares->last_status` to
`ares->ares_status`. Give trace explanation for "common"
errors
* add ares "csv" information to tracing on failure
* DoH: invoke `Curl_resolver_error()` on failure to populate
error buf
Closes#18247
- Consistently keep options within ranges
- Reduce the maximum maxredirs value to fit a signed short
- Removed comments as the place to document the options is not here
Closes#18174
Hold them for half the normal lifetime. Helps when told to transfer N
URLs in quick succession that all use the same non-resolving hostname.
Done by storing a DNS entry with a NULL pointer for 'addr'.
Previously an attempt was made in #12406 by Björn Stenberg that was
ultimately never merged.
Closes#18157
`getsock()` calls operated on a global limit that could
not be configure beyond 16 sockets. This is no longer adequate
with the new happy eyeballing strategy.
Instead, do the following:
- make `struct easy_pollset` dynamic. Starting with
a minimal room for two sockets, the very common case,
allow it to grow on demand.
- replace all protocol handler getsock() calls with pollsets
and a CURLcode to return failures
- add CURLcode return for all connection filter `adjust_pollset()`
callbacks, since they too can now fail.
- use appropriately in multi.c and multi_ev.c
- fix unit2600 to trigger pollset growth
Closes#18164
Instead of using integer seconds. Also: if the cache contains over
30,000 entries after first pruning, it makes anoter round and removes
all entries that are older than half the age of the oldest entry until
it goes below 30,000.
Closes#18160
New multi option CURLMOPT_NETWORK_CHANGED with a long bitmask value:
- CURLM_NWCOPT_CLEAR_CONNS: do not reuse existing connections, close all
idle connections.
- CURLM_NWCOPT_CLEAR_DNS: clear the multi's DNS cache.
All other bits reserved for future extensions.
Fixes#17225
Reported-by: ウさん
Closes#17613
- hostip: fix unused variable with `CURL_DISABLE_SHUFFLE_DNS`
```
lib/hostip.c: In function 'Curl_dnscache_mk_entry':
lib/hostip.c:490:42: warning: unused parameter 'data' [-Wunused-parameter]
490 | Curl_dnscache_mk_entry(struct Curl_easy *data,
| ~~~~~~~~~~~~~~~~~~^~~~
```
- setopt: fix unused function with `CURL_DISABLE_HTTP`
```
lib/setopt.c:214:17: warning: 'httpauth' defined but not used [-Wunused-function]
214 | static CURLcode httpauth(struct Curl_easy *data, bool proxy,
| ^~~~~~~~
```
- url: fix unused function with `CURL_DISABLE_NETRC`
```
lib/url.c:2760:13: warning: 'str_has_ctrl' defined but not used [-Wunused-function]
2760 | static bool str_has_ctrl(const char *input)
| ^~~~~~~~~~~~
```
Seen with a minimal curl-for-win build:
```
CW_CONFIG=dev-x64-zero-osnotls-osnoidn-nohttp-nocurltool-linux-unity
```
Closes#17818
Drop `strcasecompare` and `strncasecompare` in favor of libcurl API
calls `curl_strequal` and `curl_strnequal` respectively.
Also drop unnecessary `strcase.h` includes. Include `curl/curl.h`
instead where it wasn't included before.
Closes#17772
The `struct Curl_dns_entry *` used to established a connection
do not have the connection's lifetime, but the transfer's lifetime
(of the transfer that initiates the connect).
`Curl_dns_entry *` is reference counted with the "dns cache". That
cache might be owned by the multi or the transfer's share. In the
share, the reference count needs updating under lock.
Therefore, the dns entry can only be kept *and* released using the
same transfer it was initially looked up from. But a connection is
often discarded using another transfer.
So far, the problem of this has been avoided in clearing the connection's
dns entries in the "multi_don()" handling. So, connections had NULL
dns entries after the initial transfers and its connect had been handled.
Keeping the dns entries in data->state seems therefore a better choice.
Also: remove the `struct Curl_dns_entry *` from the connect filters
contexts. Use `data->state.dns` every time instead and fail correctly
when not present and needed.
Closes#17383
Move curlx_ functions into its own subdir.
The idea is to use the curlx_ prefix proper on these functions, and use
these same function names both in tool, lib and test suite source code.
Stop the previous special #define setup for curlx_ names.
The printf defines are now done for the library alone. Tests no longer
use the printf defines. The tool code sets its own defines. The printf
functions are not curlx, they are publicly available.
The strcase defines are not curlx_ functions and should not be used by
tool or server code.
dynbuf, warnless, base64, strparse, timeval, timediff are now proper
curlx functions.
When libcurl is built statically, the functions from the library can be
used as-is. The key is then that the functions must work as-is, without
having to be recompiled for use in tool/tests. This avoids symbol
collisions - when libcurl is built statically, we use those functions
directly when building the tool/tests. When libcurl is shared, we
build/link them separately for the tool/tests.
Assisted-by: Jay Satiro
Closes#17253
Adds a "meta_hash" to each easy handle for keeping special data during
operations. All meta data set needs to add its destructor callback, so
that meta data gets destroyed properly when the easy handle is cleaned
up or reset.
Add data->master_mid for "sub" transfers that belong to a "master" easy
handle. When a "sub" transfer is done, the corresponding "master" can
add a callback to be invoked. Used in DoH name resolution.
DoH: use easy meta hash to add internal structs for DoH name resolution.
One in each in each probe easy handle. When probes are done, response
data is copied from the probe to the initiating easy.
This allows DoH using transfers and their probes to be cleaned up in any
sequence correctly.
Fold DoH cleanup into the Curl_async_shutdown() and Curl_async_destroy()
functions.
Closes#16384
"asyn" is the internal name under which both c-ares and threaded
resolver operate. Make the naming more consistent. Implement the c-ares
resolver in `asyn-ares.*` and the threaded resolver in `asyn-thrdd.*`.
The common functions are in `asyn-base.c`.
When `CURLRES_ASYNCH` is defined, either of the two is used and
`data->state.async` exists. Members of that struct vary for the selected
implementation, but have the fields `hostname`, `port` and `ip_version`
always present. This are populated when the async resolving starts and
eliminate the need to pass them again when checking on the status and
processing the results of the resolving.
Add a `Curl_resolv_blocking()` to `hostip.h` that relieves FTP and SOCKS
from having to repeat the same code.
`Curl_resolv_check()` remains the function to check for status of
ongoing resolving. Now it also performs internally the check if the
needed DNS entry exists in the dnscache and if so, aborts the asnyc
operation. (libcurl right now does not check for duplicate resolve
attempts. an area for future improvements).
The number of functions in `asyn.h` has been reduced. There were subtle
difference in "cancel()" and "kill()" calls, both replaced by
`Curl_async_shutdown()` now. This changes behaviour for threaded
resolver insofar as the resolving thread is now always joined unless
`data->set.quick_exit` is set. Before this was only done on some code
paths. A future improvement would be a thread pool that keeps a limit
and also could handle joins more gracefully.
DoH, not previously tagged under "asny", has its struct `doh_probes` now
also in `data->state.async`, moved there from `data->req` because it
makes more sense. Further integration of DoH underneath the "asyn"
umbrella seems like a good idea.
Closes#16963
Slight refactoring around dnscache, e.g. hostcache
- eliminate `data->state.hostcache`. Always look up
relevant dnscache at share/multi.
- unify naming to "dnscache", replacing "hostcache"
- use `struct Curl_dnscache`, even though it just
contains a `Curl_hash` for now.
- add `Curl_dnscache_destroy()` for cleanup in
share/multi.
Closes#16941
