Commit graph

587 commits

Author SHA1 Message Date
Daniel Stenberg
da27db068f
tool_operate: cleanups
- move the state struct from config to global. It is used as a single
  instance anyway so might as well be a single one to save memory.
- simplify and combine several conditions
- set default retry delay inititally
- use better struct field names to make it easier to understand their
  purposes
- remove the state->outfiles field as it was not necessary
- remove superfluous glob cleanup call
- move conditions around to remove an indent level
- move the ->url NULL check

Takes single_transfer()'s complexity score down from 78 to 68.

Closes #18226
2025-08-08 11:43:28 +02:00
Daniel Stenberg
02b22ee4ea
curl: remove first argument from helpf()
It was always the same and it is a global already!

Closes #18221
2025-08-07 16:55:39 +02:00
Daniel Stenberg
3b40128b0f
curl: make global truly global
The GlobalConfig only exists in a single instance and it has worked like
this since the dawn of time. It is about time we stop passing around
pointers to what was already essentially a global object and instead
just use a... global.

It simplifies things.

Closes #18213
2025-08-07 10:43:06 +02:00
Daniel Stenberg
4b61e44e3d
tool_operate: use the correct config pointer
Triggered when doing 100+ parallel globbed upload transfers.

Reported-by: letshack9707 on hackerone

Closes #18200
2025-08-06 16:47:52 +02:00
Daniel Stenberg
034612cd51
tool_urlglob: polish, cleanups, improvements
- assert instead of printing "internal error" for unlikely events
- avoid allocating the main struct
- convert globerror() from macro to function
- renames to shorter and clearer names
- malloc + copy => memdup0
- change buffer handling to dynbuf
- realloc to handle more globs, but use less memory for few

Closes #18198
2025-08-06 08:47:53 +02:00
Daniel Stenberg
952117c032
tool_cb_hrd: remove global pointer from 'struct HdrCbData'
It is not necessary

Closes #18184
2025-08-05 15:47:56 +02:00
Stefan Eissing
1ad2009ad6
multi: add new information extraction method
Adds `curl_off_t curl_multi_get_offt(CURLM *multi_handle, CURLMinfo_offt
info)` to the multi interface with enums:

* CURLMINFO_XFERS_CURRENT: current number of transfers
* CURLMINFO_XFERS_RUNNING: number of running transfers
* CURLMINFO_XFERS_PENDING: number of pending transfers
* CURLMINFO_XFERS_DONE: number of finished transfers to read
* CURLMINFO_XFERS_ADDED: total number of transfers added, ever

Add documentation for functions and info enums.

Add use in the curl command line tool to replace two static
variables counting the same "from the outside".

refs #17870
Closes #17992
2025-08-04 23:48:57 +02:00
Daniel Stenberg
4654493fed
curl: add --parallel-max-host to limit concurrent connections per host
Where 'host' is protocol + hostname + portnumber.

Closes #18052
2025-07-31 23:30:40 +02:00
Daniel Stenberg
4d025fd912
curl: make --retry-delay and --retry-max-time accept decimal seconds
Like other time options already do.

Reported-by: Alice Lee Poetics
Fixes #18109
Closes #18111
2025-07-31 08:55:58 +02:00
Daniel Stenberg
98bcb885fe
curl: make the URL indexes 64 bit
Otherwise we could misbehave already at 2 billion URLs and we can't have
that. A few of the counters are already correctly using the right type.

Closes #18096
2025-07-30 23:13:45 +02:00
Daniel Stenberg
b2ccfbf2fb
tool_operate: simplify single_transfer
- let the caller do the cleanup on fail
- avoid gotos and use direct returns more
- use while() loop

Closes #18077
2025-07-29 16:15:05 +02:00
Stefan Eissing
6845533e24
curl: add long option '--out-null'
Add a new commandline option --out-null that discards all
response bytes into the void. Replaces non-portable use of
'-o /dev/null' with more efficiency.

Feature added in 8.16.0

Closes #17800
2025-07-28 14:57:38 +02:00
Viktor Szakats
89771d19d5
tidy-up: prefer ifdef/ifndef for single checks
Closes #18018
2025-07-27 22:35:17 +02:00
Viktor Szakats
b2bccdc257
tidy-up: move literal to the right side of comparisons
Closes #17876
2025-07-27 22:35:17 +02:00
Daniel Stenberg
6e44631d9c
tool_operate: avoid superfluous strdup'ing output
Closes #17946
2025-07-18 00:32:13 +02:00
Daniel Stenberg
f0b8137c1c
curl: remove the global argument from many functions
Since the config struct has a pointer to it, both pointers don't need to
be passed on.

Closes #17890
2025-07-10 23:28:31 +02:00
Stefan Eissing
2db8ae480f
quic: implement CURLINFO_TLS_SSL_PTR
Replace the old Curl_ssl_get_internals() with a new connection filter
query to retrieve the information. Implement that filter query for TCP
and QUIC TLS filter types.

Add tests in client tls_session_reuse to use the info option and check
that pointers are returned.

Reported-by: Larry Campbell
Fixes #17801
Closes #17809
2025-07-06 20:29:54 +02:00
Viktor Szakats
c4f9487089
tidy-up: replace <memdebug.h> with "memdebug.h" (src, units)
Closes #17722
2025-06-24 09:44:28 +02:00
Viktor Szakats
1a70977a83
build: drop explicit curlx from hdr paths, refer headers with curlx/ prefix
To make all src and test code refer to curlx headers the same way.

Also:
- src: move `curlx.h` include to `tool_setup.h`.
- src/tool_setup.h: drop stray `curlx/timeval.h`.
- servers: de-duplicate `curlx.h` and `curl_setup.h` includes.
- libtests, units: drop stray curlx sub-headers in favor of
  `<curlx/curlx.h>`.
- tests: include `curlx.h` with `<>` instead of `""`. To match
  other parts of the codebase.

Closes #17680
2025-06-23 17:02:43 +02:00
DoI
9a2663322c
curl: implement non-blocking STDIN read on Windows
Implements a seperate read thread for STDIN on Windows when curl is run
with -T/--upload-file .

This uses a similar technique to the nmap/ncat project, spawning a
seperate thread which creates a loop-back bound socket, sending STDIN
into this socket, and reading from the other end of said TCP socket in a
non-blocking way in the rest of curl.

Fixes #17451
Closes #17572
2025-06-21 11:04:29 +02:00
Viktor Szakats
916f241f2f
lib: make CURLX_SET_BINMODE() and use it
Use it from libtests' `first.c` and thus also from units, and tunits.

Also:
- cmake: drop stray `curltool` lib dependency for units.
- units: stop depending on `src` headers.
- tests/server: drop depending on `src` headers.
  (the remaining one listed in the comments, `tool_xattr.h`, was not
  actually used from servers.)
- tests/server: drop duplicate curlx headers.
  (Except `warnless.h`, which is tricky on Windows.)

Closes #17672
2025-06-19 15:57:37 +02:00
Viktor Szakats
35d0c047ce
lib: make curlx_wait_ms() and use it
Move function to curlx/, change all callers.

Also:
- src: replace local implementation.
- tests/client: replace local ad-hoc sleep code.
- tests/libtest: replace local `wait_ms()` implementation.
- lib1531: replace local ad-hoc sleep code.
- tests/server: replace local, simplified copy.
- tests/server: formatting, drop some unused headers.

Closes #17641
2025-06-19 15:57:37 +02:00
Daniel Stenberg
b42776b4f4
tool_operate: fix return code when --retry is used but not triggered
Verify with test 752

Reported-by: fjaell on github
Fixes #17554
Closes #17559
2025-06-09 09:10:06 +02:00
Daniel Stenberg
9b89fd6db7
curl: only warn once for --manual in manual-disabled build
It would previously say it twice.

Closes #17441
2025-05-24 22:48:24 +02:00
Daniel Stenberg
ed07f59841
tool_operate: split up single_transfer
Complexity reduced from 124 to 83

Remove whitelisting of this function from the complexity script.

Closes #17437
2025-05-24 15:43:08 +02:00
Daniel Stenberg
65e4444d67
curl: change the struct getout flags field into bitfields
As the flags were use mostly as individual booleans anyway, the code
gets simpler when we use bitfields instead of manual bitwise operations.

Closes #17436
2025-05-24 13:41:32 +02:00
Daniel Stenberg
dd22442e3b
tool_operate: make retrycheck() a separate function
Simplifies post_per_transfer()

Closes #17381
2025-05-19 13:27:54 +02:00
Daniel Stenberg
99f5c5c794
tool_operate: when retrying, only truncate regular files
If /dev/null or another character device etc is used for output, trying
to truncate that only causes errors.

Add test 1497 to verify

Fixes #17371
Reported-by: Brendan Dolan-Gavitt
Closes #17374
2025-05-19 09:42:54 +02:00
Daniel Stenberg
f97d372703
tool_operate: move config2setopts to separate file, split into subs
To decrease size and complexity. Complexity taken down from 190 to 80.

Bonus:

- remove leftover HTTP/0.9 warning never triggered since hyper was dropped
- remove the ftp-skip-ip option unless FTP is used
- only set HTTP options if HTTP(S) is used
- remove use of the pointless SETOPT_CHECK macro

Side-effect:

- The order of the options in --libcurl is modified

Closes #17352
2025-05-15 14:32:45 +02:00
Daniel Stenberg
13032ff75c
curl: fix memory leak when -h is used in config file
Reported-by: antypanty on hackerone

Add test 748 to reproduce and verify fix

Closes #17306
2025-05-10 11:16:07 +02:00
Daniel Stenberg
c30c7d2170
src: include memdebug.h consistently with angle brackets <>
Not double quotes. Since the file is never local.

Closes #17284
2025-05-08 10:21:02 +02:00
Daniel Stenberg
d689bd915e
src: rename curlx_safefree to tool_safefree
It is not part of the curlx function collection, just a macro that might
as well be a local version.

Closes #17270
2025-05-08 08:16:44 +02:00
Daniel Stenberg
255aac56f9
curlx: move into to curlx/
Move curlx_ functions into its own subdir.

The idea is to use the curlx_ prefix proper on these functions, and use
these same function names both in tool, lib and test suite source code.
Stop the previous special #define setup for curlx_ names.

The printf defines are now done for the library alone. Tests no longer
use the printf defines. The tool code sets its own defines. The printf
functions are not curlx, they are publicly available.

The strcase defines are not curlx_ functions and should not be used by
tool or server code.

dynbuf, warnless, base64, strparse, timeval, timediff are now proper
curlx functions.

When libcurl is built statically, the functions from the library can be
used as-is. The key is then that the functions must work as-is, without
having to be recompiled for use in tool/tests. This avoids symbol
collisions - when libcurl is built statically, we use those functions
directly when building the tool/tests. When libcurl is shared, we
build/link them separately for the tool/tests.

Assisted-by: Jay Satiro

Closes #17253
2025-05-07 11:01:15 +02:00
Andrei Florea
a638828c88
TLS: add CURLOPT_SSL_SIGNATURE_ALGORITHMS and --sigalgs
Fixes #12982
Closes #16964
2025-04-30 17:47:22 +02:00
Dave Nicolson
ed0966643d
tool_operate: fix pluralization of seconds
Closes #16751
2025-03-24 22:44:09 +01:00
Viktor Szakats
51d8213579
core: stop redefining E* macros on Windows, map EACCES, related fixes
Before this patch, standard `E*` errno codes were redefined on Windows,
onto matching winsock2 `WSA*` error codes, which have different values.
This broke uses where using the `E*` value in non-socket context, or
other places expecting a POSIX `errno`, e.g. file I/O, threads, IDN or
interfacing with dependencies.

Fix it by introducing a curl-specific `SOCKE*` set of macros that map to
`WSA*` on Windows and standard POSIX codes on other platforms. Then
verify and update the code to use `SOCKE*` or `E*` macro depending on
context.

- Add `SOCKE*` macros that map to either winsock2 or POSIX error codes.
  And use them with `SOCKERRNO` or in contexts requiring
  platform-dependent socket error codes.

  This fixes `E*` uses which were supposed be POSIX values, not `WSA*`
  socket errors, on Windows:
  - lib/curl_multibyte.c
  - lib/curl_threads.c
  - lib/idn.c
  - lib/vtls/gtls.c
  - lib/vtls/rustls.c
  - src/tool_cb_wrt.c
  - src/tool_dirhie.c

- Ban `E*` codes having a `SOCKE*` mapping, via checksrc.
  Authored-by: Daniel Stenberg

- Add exceptions for `E*` codes used in file I/O, or other contexts
  requiring POSIX error codes.

Also:
- ftp: fix missing `SOCKEACCES` mapping for Windows.
- add `SOCKENOMEM` for `Curl_getaddrinfo()` via `asyn-thread.c`.
- tests/server/sockfilt: fix to set `SOCKERRNO` in local `select()`
  override on Windows.
- lib/inet_ntop: fix to return `WSAEINVAL` on Windows, where `ENOSPC` is
  used on other platforms. To simulate Windows' built-in `inet_ntop()`,
  as tested on a Win10 machine.
  Note:
  - WINE returns `STATUS_INVALID_PARAMETER` = `0xC000000D`.
  - Microsoft documentation says it returns `WSA_INVALID_PARAMETER`
    (= `ERROR_INVALID_PARAMETER`) 87:
    https://learn.microsoft.com/windows/win32/api/ws2tcpip/nf-ws2tcpip-inet_ntop#return-value
- lib/inet_ntop: drop redundant `CURL_SETERRNO(ENOSPC)`.
  `inet_ntop4()` already sets it before returning `NULL`.
- replace stray `WSAEWOULDBLOCK` with `USE_WINSOCK` macro to detect
  winsock2.
- move existing `SOCKE*` mappings from `tests/server` to
  `curl_setup_once.h`.
- add missing `EINTR`, `EINVAL` constants for WinCE.

Follow-up to abf80aae38 #16612
Follow-up to d69425ed7d #16615
Bug: https://github.com/curl/curl/pull/16553#issuecomment-2704679377

Closes #16621
2025-03-13 00:03:25 +01:00
Daniel Stenberg
af3fe375b3
src: remove final uses of Curl_ symbol prefixes in tool code
Closes #16678
2025-03-12 14:37:08 +01:00
Daniel Stenberg
fdd97148e8
tool_operate: fix build with --libcurl support disabled
A compiler warning for unused argument.

Reported-by: Marcel Raad

Closes #16684
2025-03-12 13:12:23 +01:00
Viktor Szakats
436d4a360a
curltime: use libcurl time functions in src and tests/server
The curl tool and tests/server used 2 parallel implementations
of libcurl's `Curl_now()` and `Curl_timediff()` functions.

Make them use the libcurl one.

Closes #16653
2025-03-12 11:33:26 +01:00
Daniel Stenberg
dc12ecd5db
curl: add my_setopt_long() and _offt()
Two new dedicated functions for setting long and curl_off_t options with
curl_easy_setopt(). These make it easier to make sure we pass on the
right option (types) so that the --libcurl code also gets right.

Corrected a few errors.

Closes #16669
2025-03-11 14:58:13 +01:00
Daniel Stenberg
64a79fdd7a
tool_operate: fix a stray Curl_ symbol use 2025-03-11 07:25:12 +01:00
Daniel Stenberg
438dd08b54
src: s/Curl_safefree/curlx_safefree
Towards using curlx_ prefix for all libcurl code that is used in the
tool outside of the "real" API.

Closes #16664
2025-03-11 07:25:01 +01:00
Daniel Stenberg
8df315d5b3
src: remove Curl_ prefix from tool-specific function
Since that's a prefix we use for library-private functions

Curl_execpath is now tool_execpath

Closes #16657
2025-03-10 22:54:28 +01:00
Viktor Szakats
f4e23950c7
build: enable -Wcast-qual, fix or silence compiler warnings
The issues found fell into these categories, with the applied fixes:

- const was accidentally stripped.
  Adjust code to not cast or cast with const.

- const/volatile missing from arguments, local variables.
  Constify arguments or variables, adjust/delete casts. Small code
  changes in a few places.

- const must be stripped because an API dependency requires it.
  Strip `const` with `CURL_UNCONST()` macro to silence the warning out
  of our control. These happen at API boundaries. Sometimes they depend
  on dependency version, which this patch handles as necessary. Also
  enable const support for the zlib API, using `ZLIB_CONST`. Supported
  by zlib 1.2.5.2 and newer.

- const must be stripped because a curl API requires it.
  Strip `const` with `CURL_UNCONST()` macro to silence the warning out
  of our immediate control. For example we promise to send a non-const
  argument to a callback, though the data is const internally.

- other cases where we may avoid const stripping by code changes.
  Also silenced with `CURL_UNCONST()`.

- there are 3 places where `CURL_UNCONST()` is cast again to const.
  To silence this type of warning:
  ```
  lib/vquic/curl_osslq.c:1015:29: error: to be safe all intermediate
    pointers in cast from 'unsigned char **' to 'const unsigned char **'
    must be 'const' qualified [-Werror=cast-qual]
  lib/cf-socket.c:734:32: error: to be safe all intermediate pointers in
    cast from 'char **' to 'const char **' must be 'const' qualified
    [-Werror=cast-qual]
  ```
  There may be a better solution, but I couldn't find it.

These cases are handled in separate subcommits, but without further
markup.

If you see a `-Wcast-qual` warning in curl, we appreciate your report
about it.

Closes #16142
2025-03-10 22:30:15 +01:00
Viktor Szakats
57218d5327
tidy-up: replace Curl_safefree with free before re-assignment
Also drop `NULL` assignments after `Curl_safefree()`.

Closes #16640
2025-03-10 20:58:05 +01:00
Daniel Stenberg
8dca3b0656
src: replace strto[u][ld] with curlx_str_ parsers
- Better error handling (no errno mess), better limit checks.

- Also removed all uses of curlx_strtoofft()

Closes #16634
2025-03-10 08:09:41 +01:00
Zhang Wen
e51c494ed1
tool_operate: change "1 retries" to "1 retry"
Closes #16586
2025-03-06 16:04:39 +01:00
Daniel Stenberg
ed15fce1fd
src: avoid strdup on platforms not doing UTF8 conversions
... and use more const strings.

Closes #16560
2025-03-05 09:19:09 +01:00
tiymat
6758aa722d
IMAP: add CURLOPT_UPLOAD_FLAGS and --upload-flags
Set properties on the uploaded resource.

Test 3209 and 3210 verify.

Closes #15970
2025-03-04 15:21:16 +01:00
Daniel Stenberg
48ba23070a
tool_operate: remove unnecessary (long) typecasts
Closes #16540
2025-03-03 23:08:12 +01:00