This function REQUIRES the size of the target buffer as well as the
length of the source string. Meant to make it harder to do a bad
strcpy().
Removes 23 calls to strcpy().
Closes#20067
- replace `sendf.h` with `curl_trc.h` where it was included just for it.
- drop unused `curl_trc.h` includes.
- easy: delete obsolete comment about `send.h` include reason.
Also:
- move out `curl_trc.h` include from `sendf.h` and include it directly
in users, where not done already. To flatten the include tree and
to less rely on indirect includes.
- stop including `sendf.h` from other headers, replace it with forward
declaration of `Curl_easy`, as done already elsewhere.
Verified with an all non-unity CI run.
Closes#20061
To make it available for all files. Drop includes from individual
sources. This header was already included from most sources and not
specific to any internal subsystem.
Also to ensure that two system symbol redefines on Windows (`read()` and
`write()`) get applied to all sources. Move them to `curl_setup.h`.
Closes#20056
Delete where unused, include where to used to avoid relying on
side-effect of other headers.
Also: delete "for curlx_nonblock" comments. That's the only symbol
offered by this header.
Closes#20055
- curl_range: replace `sendf.h` with direct header dependency
`curl_trc.h`.
- drop `curl/curl.h` includes from internal sourcees in favor of the
include made from `curl_setup.h`. Replace it with the latter where
it's the only include.
- include `curl_setup.h` before using macros, where missing.
- drop redundant `stdlib.h`, `string.h` includes, in favor of
`curl_setup_once.h` including them.
- drop redundant `limits.h` in favor of `curl_setup.h` including it.
- fake_addrinfo.h: fix typo in comment.
- curl_setup_once.h: drop `stdio.h` in favor of earlier include in
`curl_setup.h`.
- drop stray, unused, `stddef.h` includes.
- memdebug.h: add missing `stddef.h` include. (relying on accidental
includes via other headers before this patch.)
- stddef.h: document why it's included.
- strerr: drop `curl/mprintf.h` in favor of `curl/curl.h` including it
via `curl_setup.h`.
Closes#20027
- apply more clang-format.
- lib/version: use `CURL_ARRAYSIZE()`.
- INSTALL-CMAKE.md: sync-up an option description with others.
- examples: delete unused main args.
- examples/ftpgetinfo: document `_CRT_SECURE_NO_WARNINGS` symbol.
- delete remaining stray duplicate lines.
- acinclude.m4: drop an unnecessary x-hack.
- vtls/mbedtls: join a URL split into two lines.
- src/tool_cb_see: add parentheses around macro expressions.
- src/tool_operate: move literals to the right side of comparisons.
- libtests: sync up fopen/fstat error messages between tests.
- curl_setup.h: replace `if ! defined __LP64` with `ifndef __LP64`.
I assume it makes no difference on Tandem systems, as the latter form
is already used in `include/curl/system.h`.
Closes#20018
Use `data->progress.now` as the timestamp of proecssing a transfer.
Update it on significant events and refrain from calling `curlx_now()`
in many places.
The problem this addresses is
a) calling curlx_now() has costs, depending on platform. Calling it
every time results in 25% increase `./runtest` duration on macOS.
b) we used to pass a `struct curltime *` around to save on calls, but
when some method directly use `curx_now()` and some use the passed
pointer, the transfer experienes non-linear time. This results in
timeline checks to report events in the wrong order.
By keeping a timestamp in the easy handle and updating it there, no
longer invoking `curlx_now()` in the "lower" methods, the transfer
can observer a steady clock progression.
Add documentation in docs/internals/TIME-KEEPING.md
Reported-by: Viktor Szakats
Fixes#19935Closes#19961
MSTATE_TUNNELING is no longer in use now that we have proxy connection
filters. Remove the state.
Remove the http handler `connect_it` method as it was merely a NOP.
Closes#19894
Before this patch curl used the C preprocessor to override standard
memory allocation symbols: malloc, calloc, strdup, realloc, free.
The goal of these is to replace them with curl's debug wrappers in
`CURLDEBUG` builds, another was to replace them with the wrappers
calling user-defined allocators in libcurl. This solution needed a bunch
of workarounds to avoid breaking external headers: it relied on include
order to do the overriding last. For "unity" builds it needed to reset
overrides before external includes. Also in test apps, which are always
built as single source files. It also needed the `(symbol)` trick
to avoid overrides in some places. This would still not fix cases where
the standard symbols were macros. It was also fragile and difficult
to figure out which was the actual function behind an alloc or free call
in a specific piece of code. This in turn caused bugs where the wrong
allocator was accidentally called.
To avoid these problems, this patch replaces this solution with
`curlx_`-prefixed allocator macros, and mapping them _once_ to either
the libcurl wrappers, the debug wrappers or the standard ones, matching
the rest of the code in libtests.
This concludes the long journey to avoid redefining standard functions
in the curl codebase.
Note: I did not update `packages/OS400/*.c` sources. They did not
`#include` `curl_setup.h`, `curl_memory.h` or `memdebug.h`, meaning
the overrides were never applied to them. This may or may not have been
correct. For now I suppressed the direct use of standard allocators
via a local `.checksrc`. Probably they (except for `curlcl.c`) should be
updated to include `curl_setup.h` and use the `curlx_` macros.
This patch changes mappings in two places:
- `lib/curl_threads.c` in libtests: Before this patch it mapped to
libcurl allocators. After, it maps to standard allocators, like
the rest of libtests code.
- `units`: before this patch it mapped to standard allocators. After, it
maps to libcurl allocators.
Also:
- drop all position-dependent `curl_memory.h` and `memdebug.h` includes,
and delete the now unnecessary headers.
- rename `Curl_tcsdup` macro to `curlx_tcsdup` and define like the other
allocators.
- map `curlx_strdup()` to `_strdup()` on Windows (was: `strdup()`).
To fix warnings silenced via `_CRT_NONSTDC_NO_DEPRECATE`.
- multibyte: map `curlx_convert_*()` to `_strdup()` on Windows
(was: `strdup()`).
- src: do not reuse the `strdup` name for the local replacement.
- lib509: call `_strdup()` on Windows (was: `strdup()`).
- test1132: delete test obsoleted by this patch.
- CHECKSRC.md: update text for `SNPRINTF`.
- checksrc: ban standard allocator symbols.
Follow-up to b12da22db1#18866
Follow-up to db98daab05#18844
Follow-up to 4deea9396b#18814
Follow-up to 9678ff5b1b#18776
Follow-up to 10bac43b87#18774
Follow-up to 20142f5d06#18634
Follow-up to bf7375ecc5#18503
Follow-up to 9863599d69#18502
Follow-up to 3bb5e58c10#17827Closes#19626
Change `inputbuff` parameter from `const char *` to `const uint8_t *` to
reflect the binary nature of the input bytes. Half the code was casting
unsigned char to signed already in calling.
Closes#19722
Description of how this works in `docs/internal/RATELIMITS.ms`.
Notable implementation changes:
- KEEP_SEND_PAUSE/KEEP_SEND_HOLD and KEEP_RECV_PAUSE/KEEP_RECV_HOLD
no longer exist. Pausing is down via blocked the new rlimits.
- KEEP_SEND_TIMED no longer exists. Pausing "100-continue" transfers
is done in the new `Curl_http_perform_pollset()` method.
- HTTP/2 rate limiting implemented via window updates. When
transfer initiaiting connection has a ratelimit, adjust the
initial window size
- HTTP/3 ngtcp2 rate limitin implemnented via ack updates
- HTTP/3 quiche does not seem to support this via its API
- the default progress-meter has been improved for accuracy
in "current speed" results.
pytest speed tests have been improved.
Closes#19384
- badwords.pl: add `-a` option to check all lines in source code files.
Before this patch indented lines were skipped (to avoid Markdown code
fences.)
- GHA/checksrc: use `-a` when verifying the source code.
- GHA/checksrc: disable `So` and `But` rules for source code.
- GHA/checksrc: add docs/examples to the verified sources.
- badwords.txt: delete 4 duplicates.
- badwords.txt: group and sort contractions.
- badwords.txt: allow ` url = `, `DIR`, `<file name`.
Closes#19536
Add new functions in `curlx/warnless.h` for controlled type
conversions:
* curlx_uitouz, convert unsigned into to size_t (should always work)
* curlx_uztoso, convert size_t to curl_off_t, capping at CURL_OFF_T_MAX
* curlx_sztouz, convert ssize_t to size_t, return TRUE when ok
* curlx_sotouz_range, convert curl_off_t to size_t interval, capping
values to interval bounds
Remove some unnecesary casts, convert some internal recv functions
to the "return result, have size_t* arg" pattern.
Closes#19495
Rename `Curl_timeleft()` to `Curl_timeleft_ms()` to make the units in
the returned `timediff_t` clear. (We used to always have ms there, but
with QUIC started to sometimes calc ns as well).
Rename some assigned vars without `_ms` suffix for clarity as well.
Closes#19486
Fix edge cases around handling of pending send frames and encoding
frames with size_t/curl_off_t possible flowy things.
Reported-by: Joshua Rogers
Closes#18965
After this patch, the codebase no longer overrides system printf
functions. Instead it explicitly calls either the curl printf functions
`curl_m*printf()` or the system ones using their original names.
Also:
- drop unused `curl_printf.h` includes.
- checksrc: ban system printf functions, allow where necessary.
Follow-up to db98daab05#18844
Follow-up to 4deea9396b#18814Closes#18866
- GHA/checkdocs: rename `spellcheck` job to `pyspelling` to say
the exact tool used.
- GHA/checkdocs: restore a comment.
- GHA/linux: add `-B .` to a cmake configure to avoid warning, and
future breakage.
- autotools: use correct casing for `Schannel`.
- doh: update RFC URL.
- drop redundant parenthesis.
- fix indentation, whitespace.
Closes#18756
Improvements around HTTP Upgrade: and multiplex hanndling:
* add `Curl_conn_set_multiplex()` to set connection's multiplex
bit and trigger "connchanged" events
* call `Curl_conn_set_multiplex()` in filters' `CF_CTRL_CONN_INFO_UPDATE`
implementation where other connection properties are updated.
This prevents connection updates before the final filter chain
is chosen.
* rename enum `UPGR101_INIT` to `UPGR101_NONE`
* rename connection bit `asks_multiplex` to `upgrade_in_progress`
* trigger "connchanged" when `upgrade_in_progress` clears
* rename `WebSockets` to `WebSocket` as it is the common term
used in documentation
Closes#18227
The auto PONG frames were inserted into the connection at the time
a PING had been decoded, irregardless if an upstream frame was just
in the middle of being assembled.
Add PONG frames only to the buffer if there is no frame currently
assemebled and, if it is, set the control frame aside. This control
frame is then added on the first opportunity of a "clean" send buffer.
There is only a single control frame set aside at a time. This means
a double PING will, when the PONG cannot be sent right away, only
send the last PONG.
I imagine this is fine. We want to prevent the endless buffering of
PONG frames on a connection where the server sends but does no receives.
Reported-by: Calvin Ruocco
Fixes#16706Closes#18479
Sending websocket data did not clear the "upload_done" flag of
the initial HTTP Upgrade request, leading to KEEP_SEND never be
cleared. This caused the socket to be polled for INOUT after all
the websocket data had been sent. A busy loop.
Closes#18476
Split out adding of individual request headers into a switch. Check
the connection http version only on fresh connections, use separate
methods.
Add TE: header directly without allocation. Add bit for indicating
Connection: header has been added and custom headers should not do
that again.
Closes#18444
Write out 9-length frames to client's WRITEFUNCTION
Read 0-length frames from READFUNCTION *if* the function
started a new frame via `curl_ws_start_frame()`.
Fixes#18286Closes#18332
Reported-by: Andriy Druk
Add support for CURLOPT_READFUNCTION with WebSocket urls when *not* in
connect-only mode, e.g. when using curl_multi_perform.
Install the callback function and set CURLOPT_UPLOAD. Return
CURL_READFUNC_PAUSE when having nothing more to send and unpause the
transfer when more data is ready.
This will send the read bytes in a WebSocket BINARY frame.
Add support for this mode in the pytest "ws_data" client and have all
tests run in 'curl_ws_send/recv' and 'peform' mode as well.
Add `curl_ws_start_frame()`. Document, cover in libcurl-ws.md and
explain the READFUNCTION mode for websockets.
Add example `websocket-updown` for this.
Closes#17683
`getsock()` calls operated on a global limit that could
not be configure beyond 16 sockets. This is no longer adequate
with the new happy eyeballing strategy.
Instead, do the following:
- make `struct easy_pollset` dynamic. Starting with
a minimal room for two sockets, the very common case,
allow it to grow on demand.
- replace all protocol handler getsock() calls with pollsets
and a CURLcode to return failures
- add CURLcode return for all connection filter `adjust_pollset()`
callbacks, since they too can now fail.
- use appropriately in multi.c and multi_ev.c
- fix unit2600 to trigger pollset growth
Closes#18164
Change the signature of `bufq` functions from
* `ssize_t Curl_bufq_*(..., CURLcode *err)` to
* `CURLcode Curl_bufq_*(..., size_t *pn)`
This allows us to write slightly less code and avoids the ssize_t/size_t
conversions in many cases. Also, it gets the function in line with all
the other send/recv signatures.
Added helper functions in `cfilters.h` for sending from/receving into
a bufq.
Fuzzer now fails to build due to these changes and its testing of
the bufq API.
Closes#17396
This started out as regression tests for the `curl_ws_recv()` and
`curl_ws_send()` implementation and ended up with a bugfix, additional
protocol validation and minor logging improvements.
- Fix reset of fragmented message decoder state when a PING/PONG is
received in between message fragments.
- Fix undefined behavior (applying zero offset to null pointer) in
curl_ws_send() when the given buffer is NULL.
- Detect invalid overlong PING/PONG/CLOSE frames.
- Detect invalid fragmented PING/PONG/CLOSE frames.
- Detect invalid sequences of fragmented frames.
- a) A continuation frame (0x80...) is received without any ongoing
fragmented message.
- b) A new fragmented message is started (0x81/0x01/0x82/0x02...)
before the ongoing fragmented message has terminated.
- Made logs for invalid opcodes easier to understand.
- Moved noisy logs to the `CURL_TRC_WS` log level.
- Unified the prefixes for WebSocket log messages: `[WS] ...`
- Add env var `CURL_WS_FORCE_ZERO_MASK` in debug builds.
- If set, it forces the bit mask applied to outgoing payloads to
0x00000000, which effectively means the payload is not masked at
all. This drastically simplifies defining the expected `<protocol>`
data in test cases.
- 2700: Frame types
- 2701: Invalid opcode 0x3
- 2702: Invalid opcode 0xB
- 2703: Invalid reserved bit RSV1 _(replaces 2310)_
- 2704: Invalid reserved bit RSV2
- 2705: Invalid reserved bit RSV3
- 2706: Invalid masked server message
- 2707: Peculiar frame sizes _(part. replaces 2311)_
- 2708: Automatic PONG
- 2709: No automatic PONG _(replaces 2312)_
- 2710: Unsolicited PONG
- 2711: Empty PING/PONG/CLOSE
- 2712: Max sized PING/PONG/CLOSE
- 2713: Invalid oversized PING _(replaces 2307)_
- 2714: Invalid oversized PONG
- 2715: Invalid oversized CLOSE
- 2716: Invalid fragmented PING
- 2717: Invalid fragmented PONG
- 2718: Invalid fragmented CLOSE
- 2719: Fragmented messages _(part. replaces 2311)_
- 2720: Fragmented messages with empty fragments
- 2721: Fragmented messages with interleaved pong
- 2722: Invalid fragmented message without initial frame
- 2723: Invalid fragmented message without final frame
- 2305: curl_ws_recv() loop reading three larger frames
- This test involuntarily sent an invalid sequence of opcodes (0x01...,0x01...,0x81...) , but neither libcurl nor the test caught this! The correct sequence was tested in 2311 (0x01...,0x00...,0x80...). See below for 2311.
- Validation of the opcode sequence was added to libcurl and is now tested in 2723.
- Superseded by 2719 (fragmented message) and 2707 (large frames).
- 2307: overlong PING payload
- The tested PING payload length check was actually missing, but the test didn't catch this since it involuntarily sent an invalid opcode (0x19... instead of 0x89...) so that the expected error occurred, but for the wrong reason.
- Superseded by 2713.
- 2310: unknown reserved bit set in frame header
- Superseded by 2703 and extended by 2704 and 2705.
- 2311: curl_ws_recv() read fragmented message
- Superseded by 2719 (fragmented message) and 2707 (large frames).
- 2312: WebSockets no auto ping
- Superseded by 2709.
- No tests for `CURLOPT_WRITEFUNCTION`.
- No tests for sending of invalid frames/fragments.
Closes#17136
Move curlx_ functions into its own subdir.
The idea is to use the curlx_ prefix proper on these functions, and use
these same function names both in tool, lib and test suite source code.
Stop the previous special #define setup for curlx_ names.
The printf defines are now done for the library alone. Tests no longer
use the printf defines. The tool code sets its own defines. The printf
functions are not curlx, they are publicly available.
The strcase defines are not curlx_ functions and should not be used by
tool or server code.
dynbuf, warnless, base64, strparse, timeval, timediff are now proper
curlx functions.
When libcurl is built statically, the functions from the library can be
used as-is. The key is then that the functions must work as-is, without
having to be recompiled for use in tool/tests. This avoids symbol
collisions - when libcurl is built statically, we use those functions
directly when building the tool/tests. When libcurl is shared, we
build/link them separately for the tool/tests.
Assisted-by: Jay Satiro
Closes#17253
Eliminates union member on struct connectdata. Sample of how
other procotols can handle their connection related data.
This avoids potention mix-ups of the `proto` union of a
connection with other protocol instances.
Removed ws "disconnect" callback as meta data is automatically
destroyed when a connection is destroyed.
Closes#17146
It passed in the wrong header length to the check function, which made
it do duplicated headers in cases where the user provides its own set.
Reported-by: sbernatsky on github
Fixes#17170Closes#17194Closes#16178
This adds another bitflag on CURLOPT_WS_OPTIONS (CURLWS_NOAUTOPONG) that
disables the default and automatic PONG reply in the WebSocket layer.
Assisted-by: Calvin Ruocco
Closes#16744
