Commit graph

458 commits

Author SHA1 Message Date
Daniel Stenberg
1dba44b2f1
tool_getparam: replace malloc + copy by dynbuf for --data 2024-01-08 22:38:22 +01:00
Viktor Szakats
e9a7d4a1c8
windows: use built-in _WIN32 macro to detect Windows
Windows compilers define `_WIN32` automatically. Windows SDK headers
or build env defines `WIN32`, or we have to take care of it. The
agreement seems to be that `_WIN32` is the preferred practice here.
Make the source code rely on that to detect we're building for Windows.

Public `curl.h` was using `WIN32`, `__WIN32__` and `CURL_WIN32` for
Windows detection, next to the official `_WIN32`. After this patch it
only uses `_WIN32` for this. Also, make it stop defining `CURL_WIN32`.

There is a slight chance these break compatibility with Windows
compilers that fail to define `_WIN32`. I'm not aware of any obsolete
or modern compiler affected, but in case there is one, one possible
solution is to define this macro manually.

grepping for `WIN32` remains useful to discover Windows-specific code.

Also:

- extend `checksrc` to ensure we're not using `WIN32` anymore.

- apply minor formatting here and there.

- delete unnecessary checks for `!MSDOS` when `_WIN32` is present.

Co-authored-by: Jay Satiro
Reviewed-by: Daniel Stenberg

Closes #12376
2023-11-22 15:42:25 +00:00
Sam James
bc8509a748
misc: fix -Walloc-size warnings
GCC 14 introduces a new -Walloc-size included in -Wextra which gives:

```
src/tool_operate.c: In function ‘add_per_transfer’:
src/tool_operate.c:213:5: warning: allocation of insufficient size ‘1’ for type ‘struct per_transfer’ with size ‘480’ [-Walloc-size]
  213 |   p = calloc(sizeof(struct per_transfer), 1);
      |     ^
src/var.c: In function ‘addvariable’:
src/var.c:361:5: warning: allocation of insufficient size ‘1’ for type ‘struct var’ with size ‘32’ [-Walloc-size]
  361 |   p = calloc(sizeof(struct var), 1);
      |     ^
```

The calloc prototype is:
```
void *calloc(size_t nmemb, size_t size);
    ```

So, just swap the number of members and size arguments to match the
prototype, as we're initialising 1 struct of size `sizeof(struct
...)`. GCC then sees we're not doing anything wrong.

Closes #12292
2023-11-11 23:35:47 +01:00
Daniel Stenberg
01d9b8bc7c
curl: move IPFS code into src/tool_ipfs.[ch]
- convert ensure_trailing into ensure_trailing_slash
- strdup the URL string to own it proper
- use shorter variable names
- combine some expressions
- simplify error handling in ipfs_gateway()
- add MAX_GATEWAY_URL_LEN + proper bailout if maximum is reached
- ipfs-gateway.d polish and simplification
- shorten ipfs error message + make them "synthetic"

Closes #12281
2023-11-07 07:42:28 +01:00
Daniel Stenberg
ba281e5c72
tool_operate: do not mix memory models
Make sure 'inputpath' only points to memory allocated by libcurl so that
curl_free works correctly.

Pointed out by Coverity

Follow-up to 859e88f653

Closes #12280
2023-11-06 14:29:53 +01:00
Mark Gaiser
859e88f653
curl: improved IPFS and IPNS URL support
Previously just ipfs://<cid> and ipns://<cid> was supported, which is
too strict for some usecases.

This patch allows paths and query arguments to be used too.
Making this work according to normal http semantics:

 ipfs://<cid>/foo/bar?key=val
 ipns://<cid>/foo/bar?key=val

The gateway url support is changed.
It now only supports gateways in the form of:

 http://<gateway>/foo/bar
 http://<gateway>

Query arguments here are explicitly not allowed and trigger an intended
malformed url error.

There also was a crash when IPFS_PATH was set with a non trailing
forward slash. This has been fixed.

Lastly, a load of test cases have been added to verify the above.

Reported-by: Steven Allen
Fixes #12148
Closes #12152
2023-11-05 10:59:20 +01:00
Loïc Yhuel
014ce7c0d0
tool: fix --capath when proxy support is disabled
After 95e8515ca0, --capath always sets CURLOPT_PROXY_CAPATH, which fails
with CURLE_UNKNOWN_OPTION when proxy support is disabled.

Closes #12089
2023-10-21 14:52:47 +02:00
Marcin Rataj
7f7fa2fd67
tool_operate: fix links in ipfs errors
URL fragment links generated from headers in
https://curl.se/docs/ipfs.html are lowercase.

Closes #12133
2023-10-16 08:42:19 +02:00
Viktor Szakats
e5bb88b8f8
tool: use our own stderr variable
Earlier this year we changed our own stderr variable to use the standard
name `stderr` (to avoid bugs where someone is using `stderr` instead of
the curl-tool specific variable). This solution needed to override the
standard `stderr` symbol via the preprocessor. This in turn didn't play
well with unity builds and caused curl tool to crash or stay silent due
to an uninitialized stderr. This was a hard to find issue, fixed by
manually breaking out one file from the unity sources.

To avoid two these two tricks, this patch implements a different
solution: Restore using our own local variable for our stderr output and
leave `stderr` as-is. To avoid using `stderr` by mistake, add a
`checksrc` rule (based on logic we already used in lib for `strerror`)
that detects any `stderr` use in `src` and points to using our own
variable instead: `tool_stderr`.

Follow-up to 06133d3e9b
Follow-up to 2f17a9b654

Closes #11958
2023-09-28 10:50:56 +00:00
Daniel Stenberg
a35e2dd69c
tool_operate: free 'gateway' correctly
Pointed out by Coverity. The fix in 93885cf3a8 was incomplete.

Also removed repeated wording in IPFS related error messages.

Closes #11969
2023-09-28 10:14:38 +02:00
Daniel Stenberg
c50cbac0f0
tool_operate: avoid strlen() -1 on zero length content from file
Follow-up to 65b563a96a

Closes #11959
2023-09-28 09:24:57 +02:00
Daniel Stenberg
93885cf3a8
tool_operate: fix memory mixups
Switch to plain getenv() from curl_getenv() to avoid the allocation and
having to keep track of which free() or curl_free() that need to be
used.

Coverity found issues and a memory leak.

Follow-up to 65b563a96a

Closes #11959
2023-09-28 09:24:57 +02:00
Mark Gaiser
65b563a96a
curl: add support for the IPFS protocols:
- ipfs://<cid>
- ipns://<cid>

This allows you tu use ipfs in curl like:
curl ipfs://<cid>
and
curl ipns://<cid>

For more information consult the readme at:
https://curl.se/docs/ipfs.html

Closes #8805
2023-09-23 11:02:10 +02:00
Daniel Stenberg
c5a9630739
tool_operate: make aws-sigv4 not require TLS to be used
Maybe not used too often, but we want it for testing and it should work.
2023-09-07 17:50:27 +02:00
Daniel Stenberg
139ecfb869
tool_operate: fix memory leak when SSL_CERT_DIR is used
Detected by Coverity

Follow-up to 29bce9857a

Closes #11577
2023-08-02 16:10:49 +02:00
Jay Satiro
af3f4e419b tool_cb_wrt: fix invalid unicode for windows console
- Suppress an incomplete UTF-8 sequence at the end of the buffer.

- Attempt to reconstruct incomplete UTF-8 sequence from prior call(s)
  in current call.

Prior to this change, in Windows console UTF-8 sequences split between
two or more calls to the write callback would cause invalid "replacement
characters" U+FFFD to be printed instead of the actual Unicode
character. This is because in Windows only UTF-16 encoded characters are
printed to the console, therefore we convert the UTF-8 contents to
UTF-16, which cannot be done with partial UTF-8 sequences.

Reported-by: Maksim Arhipov

Fixes https://github.com/curl/curl/issues/9841
Closes https://github.com/curl/curl/pull/10890
2023-08-01 03:28:12 -04:00
Daniel Stenberg
2e160c9c65
tool: add "variable" support
Add support for command line variables. Set variables with --variable
name=content or --variable name@file (where "file" can be stdin if set
to a single dash (-)).

Variable content is expanded in option parameters using "{{name}}"
(without the quotes) if the option name is prefixed with
"--expand-". This gets the contents of the variable "name" inserted, or
a blank if the name does not exist as a variable. Insert "{{" verbatim
in the string by prefixing it with a backslash, like "\\{{".

Import an environment variable with --variable %name. It makes curl exit
with an error if the environment variable is not set. It can also rather
get a default value if the variable does not exist, using =content or
@file like shown above.

Example: get the USER environment variable into the URL:

 --variable %USER
 --expand-url = "https://example.com/api/{{USER}}/method"

When expanding variables, curl supports a set of functions that can make
the variable contents more convenient to use. It can trim leading and
trailing white space with "trim", output the contents as a JSON quoted
string with "json", URL encode it with "url" and base 64 encode it with
"b64". To apply functions to a variable expansion, add them colon
separated to the right side of the variable. They are then performed in
a left to right order.

Example: get the contents of a file called $HOME/.secret into a variable
called "fix". Make sure that the content is trimmed and percent-encoded
sent as POST data:

  --variable %HOME=/home/default
  --expand-variable fix@{{HOME}}/.secret
  --expand-data "{{fix:trim:url}}"
  https://example.com/

Documented. Many new test cases.

Co-brainstormed-by: Emanuele Torre
Assisted-by: Jat Satiro
Closes #11346
2023-07-31 11:51:34 +02:00
Daniel Stenberg
29bce9857a
tool_operate: allow SSL_CERT_FILE and SSL_CERT_DIR
... used at once.

Reported-by: Gabriel Corona
Fixes #11325
Closes #11531
2023-07-29 23:41:42 +02:00
Daniel Stenberg
9ec099a2bc
tool_operate: allow cookie lines up to 8200 bytes
Since this option might set multiple cookies in the same line, it does
not make total sense to cap this at 4096 bytes, which is the limit for a
single cookie name or value.

Closes #11303
2023-06-12 14:15:37 +02:00
Raito Bezarius
0a75964d0d
haproxy: add --haproxy-clientip flag to spoof client IPs
CURLOPT_HAPROXY_CLIENT_IP in the library

Closes #10779
2023-06-05 20:08:37 +02:00
Daniel Stenberg
9ad23c38e5
curl: add --ca-native and --proxy-ca-native
These are two boolean options to ask curl to use the native OS's CA
store when verifying TLS servers. For peers and for proxies
respectively.

They currently only have an effect for curl on Windows when built to use
OpenSSL for TLS.

Closes #11049
2023-06-05 20:07:11 +02:00
Daniel Stenberg
fff65550bc
CURLOPT_MAIL_RCPT_ALLOWFAILS: replace CURLOPT_MAIL_RCPT_ALLLOWFAILS
Deprecate the name using three Ls and prefer the name with two.

Replaces #10047
Closes #11218
2023-06-03 23:29:04 +02:00
Daniel Stenberg
bfa7006424
test447: test PUTting a file that grows
... and have curl trim the end when it reaches the expected total amount
of bytes instead of over-sending.

Reported-by: JustAnotherArchivist on github
Closes #11223
2023-06-01 13:43:31 +02:00
Daniel Stenberg
1f85420a28
curl: count uploaded data to stop at the originally given size
Closes #11223
Fixes #11222
Reported-by: JustAnotherArchivist on github
2023-06-01 13:43:28 +02:00
Daniel Stenberg
6661bd588d
tool: remove exclamation marks from error/warning messages 2023-06-01 08:19:21 +02:00
Daniel Stenberg
741f7ed4bc
tool: use errorf() for error output
Convert a number of fprintf() calls.
2023-06-01 08:19:11 +02:00
Daniel Stenberg
6d45b9ca9c
tool: remove newlines from all helpf/notef/warnf/errorf calls
Make voutf() always add one.

Closes #11226
2023-06-01 08:18:21 +02:00
Daniel Stenberg
0807fd72f9
tool_urlglob: use curl_off_t instead of longs
To handle more globs better (especially on Windows)

Closes #11224
2023-05-31 08:14:41 +02:00
Emanuele Torre
f198d33e8d
checksrc: disallow spaces before labels
Out of 415 labels throughout the code base, 86 of those labels were
not at the start of the line. Which means labels always at the start of
the line is the favoured style overall with 329 instances.

Out of the 86 labels not at the start of the line:
* 75 were indented with the same indentation level of the following line
* 8 were indented with exactly one space
* 2 were indented with one fewer indentation level then the following
  line
* 1 was indented with the indentation level of the following line minus
  three space (probably unintentional)

Co-Authored-By: Viktor Szakats

Closes #11134
2023-05-18 20:45:04 +02:00
Daniel Stenberg
39a33fcac0
tool_operate: refuse (--data or --form) and --continue-at combo
libcurl assumes that a --continue-at resumption is done to continue an
upload using the read callback and neither --data nor --form use
that and thus won't do what the user wants. Whatever the user wants
with this strange combination.

Add test 426 to verify.

Reported-by: Smackd0wn on github
Fixes #11081
Closes #11083
2023-05-08 14:10:44 +02:00
Daniel Stenberg
9ce7eee070
checksrc: find bad indentation in conditions without open brace
If the previous line starts with if/while/for AND ends with a closed
parenthesis and there's an equal number of open and closed parentheses
on that line, verify that this line is indented $indent more steps, if
not a cpp line.

Also adjust the fall-out from this fix.

Closes #11054
2023-04-28 23:11:00 +02:00
Daniel Stenberg
c0a9f905e6
tool_operate: pass a long as CURLOPT_HEADEROPT argument
Closes #10798
2023-03-22 13:51:37 +01:00
Marcel Raad
ff5c3455ce
tool_operate: silence unused parameter warning
`global` is only used in the `my_setopt` macro version without
`CURL_DISABLE_LIBCURL_OPTION` since commit 4774decf10a.

Closes https://github.com/curl/curl/pull/10752
2023-03-13 13:43:26 +01:00
Jay Satiro
2f17a9b654 tool: improve --stderr handling
- freopen stderr with the user-specified file (--stderr file) instead of
  using a separate 'errors' stream.

- In tool_setup.h override stdio.h's stderr macro as global variable
  tool_stderr.

Both freopen and overriding the stderr macro are necessary because if
the user-specified filename is "-" then stdout is assigned to
tool_stderr and no freopen takes place. See the PR for more information.

Ref: https://github.com/curl/curl/issues/10491

Closes https://github.com/curl/curl/pull/10673
2023-03-12 00:58:40 -05:00
Grisha Levit
1dd929674e
tool: dump headers even if file is write-only
The fixes in #10079 brought a (seemingly unrelated) change of open mode
from `wb`/`ab` to `wb+`/`ab+` for the headerfile. This makes it no
longer possible to write the header file to e.g. a pipe, like:

    curl -D >(grep ...) file:///dev/null

Which presently results in `Warning: Failed to open /dev/fd/63`

See #10079
Closes #10675
2023-03-06 11:14:37 +01:00
Jay Satiro
d9b7f6e750
curl: make -w's %{stderr} use the file set with --stderr
Reported-by: u20221022 on github
Fixes #10491
Closes #10569
2023-02-23 10:05:13 +01:00
Daniel Stenberg
60e13311f4
tool_operate: propagate error codes for missing URL after --next
Fixes #10558
Reported-by: u20221022 on github
Closes #10580
2023-02-22 08:44:45 +01:00
Daniel Stenberg
1c9cfb7af3
tool_operate: avoid fclose(NULL) on bad header dump file
Fixes #10570
Reported-by: Jérémy Rabasco
Closes #10571
2023-02-21 10:34:29 +01:00
Jay Satiro
41dfb7f516 tool_operate: fix scanbuild compiler warning
Prior to this change Azure CI scanbuild warned of a potential NULL
pointer string passed to strtol when CURLDEBUG enabled, even though the
way the code was written it wouldn't have happened.

Bug: 5479d991 (r101159711)
Reported-by: Marcel Raad

Closes https://github.com/curl/curl/pull/10559
2023-02-18 19:02:40 -05:00
Daniel Stenberg
6841f2ed5f
curl: make --silent work stand-alone
- renamed the struct field to 'silent' to match the cmdline option
- make --show-error toggle independently of --silent
- make --silent independent of ->noprogress as well

By doing this, the three options --silent, --no-progress-meter and
--show-error should work independently of each other and also work with
and without '--no-' prefix as documented.

Reported-by: u20221022 on github
Fixes #10535
Closes #10536
2023-02-17 09:22:49 +01:00
Daniel Stenberg
5479d9916e
tool_operate: allow debug builds to set buffersize
Using the CURL_BUFFERSIZE environment variable.

Closes #10532
2023-02-16 14:36:24 +01:00
Daniel Stenberg
95fe2bba74
tool_operate: move the 'updated' variable
This was already done by Dan Fandrich in the previous PR but somehow I
lost that fixup.

Follow-up to 349c5391f2
2023-02-07 12:16:13 +01:00
Dan Fandrich
349c5391f2
tool_operate: Fix error codes on bad URL & OOM
curl would erroneously report CURLE_OUT_OF_MEMORY in some cases instead
of CURLE_URL_MALFORMAT. In other cases, it would erroneously return
CURLE_URL_MALFORMAT instead of CURLE_OUT_OF_MEMORY.  Add a test case to
test the former condition.

Fixes #10130
Closes #10414
2023-02-07 08:20:04 +01:00
Daniel Stenberg
e7512e78c2
tool_operate: repair --rate
Regression from a55256cfb2 (7.87.0)
Reported-by: highmtworks on github
Fixes #10357
Closes #10358
2023-01-29 16:06:06 +01:00
Stefan Eissing
24e4e57cf3
tool_operate: fix headerfile writing
Do not rely on the first transfer started to be the first to get a
response (remember -Z). All transfers now write the headefile (-D) in
append mode, making sure that the order of transfer responses does not
lead to overwrites of previous data.

Closes #10224
2023-01-04 23:11:54 +01:00
Daniel Stenberg
2bc1d775f5
copyright: update all copyright lines and remove year ranges
- they are mostly pointless in all major jurisdictions
- many big corporations and projects already don't use them
- saves us from pointless churn
- git keeps history for us
- the year range is kept in COPYING

checksrc is updated to allow non-year using copyright statements

Closes #10205
2023-01-03 09:19:21 +01:00
Daniel Stenberg
c6aa19c1da
writeout: add %{certs} and %{num_certs}
Let users get the server certificate chain using the command line

Closes #10019
2022-12-27 22:41:17 +01:00
Daniel Stenberg
0bf8b796a0
tool_operate: share HSTS between handles 2022-12-27 15:22:01 +01:00
John Bampton
80efd3bc61
misc: fix grammar and spelling
Closes #10137
2022-12-22 18:07:07 +01:00
Marcel Raad
03ac7ab4ad
tool_operate: fix CURLOPT_SOCKS5_GSSAPI_NEC type
`CURLOPT_SOCKS5_GSSAPI_NEC` is a long, while `socks5_gssapi_nec` was
made a bool in commit 4ac64eadf6.

Closes https://github.com/curl/curl/pull/10124
2022-12-21 15:37:54 +01:00