Commit graph

291 commits

Author SHA1 Message Date
Stefan Eissing
e12b39e133
trace: make tracing available in non-debug builds
Add --trace-config to curl

Add curl_global_trace() to libcurl

Closes #11421
2023-08-03 17:32:25 +02:00
Daniel Stenberg
2e160c9c65
tool: add "variable" support
Add support for command line variables. Set variables with --variable
name=content or --variable name@file (where "file" can be stdin if set
to a single dash (-)).

Variable content is expanded in option parameters using "{{name}}"
(without the quotes) if the option name is prefixed with
"--expand-". This gets the contents of the variable "name" inserted, or
a blank if the name does not exist as a variable. Insert "{{" verbatim
in the string by prefixing it with a backslash, like "\\{{".

Import an environment variable with --variable %name. It makes curl exit
with an error if the environment variable is not set. It can also rather
get a default value if the variable does not exist, using =content or
@file like shown above.

Example: get the USER environment variable into the URL:

 --variable %USER
 --expand-url = "https://example.com/api/{{USER}}/method"

When expanding variables, curl supports a set of functions that can make
the variable contents more convenient to use. It can trim leading and
trailing white space with "trim", output the contents as a JSON quoted
string with "json", URL encode it with "url" and base 64 encode it with
"b64". To apply functions to a variable expansion, add them colon
separated to the right side of the variable. They are then performed in
a left to right order.

Example: get the contents of a file called $HOME/.secret into a variable
called "fix". Make sure that the content is trimmed and percent-encoded
sent as POST data:

  --variable %HOME=/home/default
  --expand-variable fix@{{HOME}}/.secret
  --expand-data "{{fix:trim:url}}"
  https://example.com/

Documented. Many new test cases.

Co-brainstormed-by: Emanuele Torre
Assisted-by: Jat Satiro
Closes #11346
2023-07-31 11:51:34 +02:00
Daniel Stenberg
8d7cda1f92
curl: return error when asked to use an unsupported HTTP version
When one of the following options are used but the libcurl in use does
not support it:

--http2
--http2-prior-knowledge
--proxy-http2

Closes #11440
2023-07-15 19:02:20 +02:00
Stefan Eissing
f06cc4f85e
tool: add curl command line option --trace-ids
- added and documented --trace-ids to prepend (after the timestamp)
  the transfer and connection identifiers to each verbose log line
- format is [n-m] with `n` being the transfer id and `m` being the
  connection id. In case there is not valid connection id, print 'x'.
- Log calls with a handle that has no transfer id yet, are written
  without any ids.

Closes #11185
2023-06-12 23:53:09 +02:00
Boris Verkhovskiy
b832cab112
tool_getparam: fix comment
Closes #11253
2023-06-05 20:10:36 +02:00
Raito Bezarius
0a75964d0d
haproxy: add --haproxy-clientip flag to spoof client IPs
CURLOPT_HAPROXY_CLIENT_IP in the library

Closes #10779
2023-06-05 20:08:37 +02:00
Daniel Stenberg
9ad23c38e5
curl: add --ca-native and --proxy-ca-native
These are two boolean options to ask curl to use the native OS's CA
store when verifying TLS servers. For peers and for proxies
respectively.

They currently only have an effect for curl on Windows when built to use
OpenSSL for TLS.

Closes #11049
2023-06-05 20:07:11 +02:00
Daniel Stenberg
6661bd588d
tool: remove exclamation marks from error/warning messages 2023-06-01 08:19:21 +02:00
Daniel Stenberg
741f7ed4bc
tool: use errorf() for error output
Convert a number of fprintf() calls.
2023-06-01 08:19:11 +02:00
Daniel Stenberg
6d45b9ca9c
tool: remove newlines from all helpf/notef/warnf/errorf calls
Make voutf() always add one.

Closes #11226
2023-06-01 08:18:21 +02:00
Daniel Stenberg
0807fd72f9
tool_urlglob: use curl_off_t instead of longs
To handle more globs better (especially on Windows)

Closes #11224
2023-05-31 08:14:41 +02:00
Daniel Stenberg
9ce7eee070
checksrc: find bad indentation in conditions without open brace
If the previous line starts with if/while/for AND ends with a closed
parenthesis and there's an equal number of open and closed parentheses
on that line, verify that this line is indented $indent more steps, if
not a cpp line.

Also adjust the fall-out from this fix.

Closes #11054
2023-04-28 23:11:00 +02:00
Daniel Stenberg
e39754f6a2
curl: add --proxy-http2
For trying HTTP/2 with an HTTPS proxy.

Closes #10926
2023-04-14 10:39:23 +02:00
Daniel Stenberg
e2452cf60e
tool_getparam: error if --next is used without a prior URL
Reported-by: 積丹尼 Dan Jacobson
Ref: https://github.com/curl/curl/pull/10769#discussion_r1137895629

Closes #10782
2023-03-17 14:07:10 +01:00
Jay Satiro
2f17a9b654 tool: improve --stderr handling
- freopen stderr with the user-specified file (--stderr file) instead of
  using a separate 'errors' stream.

- In tool_setup.h override stdio.h's stderr macro as global variable
  tool_stderr.

Both freopen and overriding the stderr macro are necessary because if
the user-specified filename is "-" then stdout is assigned to
tool_stderr and no freopen takes place. See the PR for more information.

Ref: https://github.com/curl/curl/issues/10491

Closes https://github.com/curl/curl/pull/10673
2023-03-12 00:58:40 -05:00
Daniel Stenberg
780ff2c95c
tool_getparam: don't add a new node for just --no-remote-name
Unless --remote-name-all is used.

Fixes #10564
Reported-by: u20221022 on github
Closes #10582
2023-02-22 08:43:13 +01:00
Daniel Stenberg
6841f2ed5f
curl: make --silent work stand-alone
- renamed the struct field to 'silent' to match the cmdline option
- make --show-error toggle independently of --silent
- make --silent independent of ->noprogress as well

By doing this, the three options --silent, --no-progress-meter and
--show-error should work independently of each other and also work with
and without '--no-' prefix as documented.

Reported-by: u20221022 on github
Fixes #10535
Closes #10536
2023-02-17 09:22:49 +01:00
Daniel Stenberg
ce24ea373f
tool_getparam: make --get a true boolean
To match how it is documented in the man page.

Fixes #10527
Reported-by: u20221022 on github
Closes #10531
2023-02-16 10:53:59 +01:00
Daniel Stenberg
e6b72644c2
tool_getparam: fix compiler warning when !HAVE_WRITABLE_ARGV
Follow-up to 2ed0e1f70e

Closes #10286
2023-01-13 10:39:23 +01:00
Daniel Stenberg
13991d60ee
src: add --http3-only
Warning: --http3 and --http3-only are subject to change again (or be
removed) before HTTP/3 support goes non-experimental.

Closes #10264
2023-01-12 16:18:05 +01:00
Harry Sintonen
2ed0e1f70e
tool_getparam: fix hiding of command line secrets
Closes #10276
2023-01-12 14:09:23 +01:00
Daniel Stenberg
2bc1d775f5
copyright: update all copyright lines and remove year ranges
- they are mostly pointless in all major jurisdictions
- many big corporations and projects already don't use them
- saves us from pointless churn
- git keeps history for us
- the year range is kept in COPYING

checksrc is updated to allow non-year using copyright statements

Closes #10205
2023-01-03 09:19:21 +01:00
Daniel Stenberg
e9c580de4e
lib: parse numbers with fixed known base 10
... instead of using 0 argument that allows decimal, hex or octal when
the number is documented and assumed to use base 10.

Closes #9933
2022-11-17 23:15:37 +01:00
Patrick Monnerat
184fc6f07a
tool: use feature names instead of bit mask, when possible
If the run-time libcurl is too old to support feature names, the name
array is created locally from the bit masks. This is the only sequence
left that uses feature bit masks.

Closes #9583
2022-11-14 09:19:10 +01:00
Daniel Stenberg
0f860da4b6
tool_getparam: make --no-get work as the opposite of --get
... as documented.

Closes #9891
2022-11-12 00:13:29 +01:00
Daniel Stenberg
b6e1afd069
curl: add --url-query
This option adds a piece of data, usually a name + value pair, to the
end of the URL query part. The syntax is identical to that used for
--data-urlencode with one extension:

If the argument starts with a '+' (plus), the rest of the string is
provided as-is unencoded.

This allows users to "build" query parts with options and URL encoding
even when not doing GET requests, which the already provided option -G
(--get) is limited to.

This idea was born in a Twitter thread.

Closes #9691
2022-11-08 14:37:34 +01:00
Daniel Stenberg
52cc4a85fd
style: use space after comment start and before comment end
/* like this */

/*not this*/

checksrc is updated accordingly

Closes #9828
2022-10-30 22:31:29 +01:00
Daniel Stenberg
a55256cfb2
curl: timeout in the read callback
The read callback can timeout if there's nothing to read within the
given maximum period. Example use case is when doing "curl -m 3
telnet://example.com" or anything else that expects input on stdin or
similar that otherwise would "hang" until something happens and then not
respect the timeout.

This fixes KNOWN_BUG 8.1, first filed in July 2009.

Bug: https://sourceforge.net/p/curl/bugs/846/

Closes #9815
2022-10-28 17:57:14 +02:00
Rickard Hallerbäck
703efb3379
tool_paramhlp: make the max argument a 'double'
To fix compiler warnings "Implicit conversion from 'long' to 'double'
may lose precision"

Closes #9700
2022-10-16 00:46:52 +02:00
Daniel Stenberg
1527dd07f1
tool_getparam: pass in the snprintf("%.*s") string length as 'int'
Reported by Coverity CID 1515928

Closes #9679
2022-10-10 08:42:48 +02:00
Daniel Stenberg
b90f857fab
tool_getparam: split out data_urlencode() into its own function
Closes #9673
2022-10-08 11:48:20 +02:00
Patrick Monnerat
677266c769
tool: remove protocol count limitation
Replace bit mask protocol sets by null-terminated arrays of protocol
tokens. These are the addresses of the protocol names returned by
curl_version_info().

Protocol names are sorted case-insensitively before output to satisfy CI
tests matches consistency.

The protocol list returned by curl_version_info() is augmented with all
RTMP protocol variants.

Test 1401 adjusted for new alpha ordered output.

Closes #9546
2022-09-22 13:49:10 +02:00
Patrick Monnerat
dd2a024323
cli tool: do not use disabled protocols
As they are now rejected by the library, take care of not passing
disabled protocol names to CURLOPT_PROTOCOLS_STR and
CURLOPT_REDIR_PROTOCOLS_STR.

Rather than using the CURLPROTO_* constants, dynamically assign protocol
numbers based on the order they are listed by curl_version_info().

New type proto_set_t implements prototype bit masks: it should therefore
be large enough to accomodate all library-enabled protocols. If not,
protocol numbers beyond the bit count of proto_set_t are recognized but
"inaccessible": when used, a warning is displayed and the value is
ignored. Should proto_set_t overflows, enabled protocols are reordered to
force those having a public CURLPROTO_* representation to be accessible.

Code has been added to subordinate RTMP?* protocols to the presence of
RTMP in the enabled protocol list, being returned by curl_version_info()
or not.
2022-09-16 23:29:08 +02:00
Daniel Stenberg
79e404d039
curl: warn for --ssl use, considered insecure
Closes #9519
2022-09-16 21:27:20 +02:00
Daniel Stenberg
a850b9edb9
getparameter: return PARAM_MANUAL_REQUESTED for -M even when disabled
... to improve the output in this situation. Now it doesn't say "option
unknown" anymore.

Closes #9485
2022-09-14 08:04:39 +02:00
Daniel Stenberg
472f1cbe7e
NPN: remove support for and use of
Next Protocol Negotiation is a TLS extension that was created and used
for agreeing to use the SPDY protocol (the precursor to HTTP/2) for
HTTPS. In the early days of HTTP/2, before the spec was finalized and
shipped, the protocol could be enabled using this extension with some
servers.

curl supports the NPN extension with some TLS backends since then, with
a command line option `--npn` and in libcurl with
`CURLOPT_SSL_ENABLE_NPN`.

HTTP/2 proper is made to use the ALPN (Application-Layer Protocol
Negotiation) extension and the NPN extension has no purposes
anymore. The HTTP/2 spec was published in May 2015.

Today, use of NPN in the wild should be extremely rare and most likely
totally extinct. Chrome removed NPN support in Chrome 51, shipped in
June 2016. Removed in Firefox 53, April 2017.

Closes #9307
2022-09-05 07:39:02 +02:00
Daniel Stenberg
5b059ba895
getparam: correctly clean args
Follow-up to bf7e887b24

The previous fix for #9128 was incomplete and caused #9397.

Fixes #9397
Closes #9399
2022-08-31 01:03:36 +02:00
Daniel Stenberg
f7e14fee68
tool_getparam: make --doh-url "" switch it off
A possible future addition could be to parse the URL first too to verify
that it is valid before trying to use it.

Assisted-by: Jay Satiro
Closes #9207
2022-07-27 10:56:13 +02:00
Daniel Stenberg
206550a9c2
tool_getparam: fix cleanarg() for unicode builds
Use the correct type, and make cleanarg an empty macro if the cleaning
ability is absent.

Fixes #9195
Closes #9196

Reviewed-by: Jay Satiro
Reviewed-by: Marcel Raad
2022-07-25 23:58:43 +02:00
Daniel Stenberg
bf7e887b24
tool_getparam: repair cleanarg
Regression since 9e5669f.

Make sure the "cleaning" of command line arguments is done on the
original argv[] pointers. As a bonus, it also exits better on out of
memory error.

Reported-by: Litter White
Fixes #9128
Closes #9130
2022-07-10 15:49:14 +02:00
Daniel Stenberg
e6f8445ede
setopt: add CURLOPT_PROTOCOLS_STR and CURLOPT_REDIR_PROTOCOLS_STR
... as replacements for deprecated CURLOPT_PROTOCOLS and
CURLOPT_REDIR_PROTOCOLS as these new ones do not risk running into the
32 bit limit the old ones are facing.

CURLINFO_PROTCOOL is now deprecated.

The curl tool is updated to use the new options.

Added test 1597 to verify the libcurl protocol parser.

Closes #8992
2022-07-04 08:35:09 +02:00
max.mehl
ad9bc5976d
copyright: make repository REUSE compliant
Add licensing and copyright information for all files in this repository. This
either happens in the file itself as a comment header or in the file
`.reuse/dep5`.

This commit also adds a Github workflow to check pull requests and adapts
copyright.pl to the changes.

Closes #8869
2022-06-13 09:13:00 +02:00
Boris Verkhovskiy
83ee5c428d
curl: re-enable --no-remote-name
Closes #8931
2022-05-31 13:23:22 +02:00
JustAnotherArchivist
10cd69623a tool_getparam: fix --parallel-max maximum value constraint
- Clamp --parallel-max to MAX_PARALLEL (300) instead of resetting to
  default value.

Previously, --parallel-max 300 would use 300 concurrent transfers, but
--parallel-max 301 would unexpectedly use only 50. This change clamps
higher values to the maximum (ie --parallel-max 301 would use 300).

Closes https://github.com/curl/curl/pull/8930
2022-05-28 03:23:57 -04:00
Daniel Stenberg
8f48b5d783
curl: add --rate to set max request rate per time unit
--rate "12/m" - for 12 per minute or
--rate "5/h" - for 5 per hour

Removed from TODO

Closes #8671
2022-05-23 17:59:56 +02:00
Daniel Stenberg
84052154f7
curl: deprecate --random-file and --egd-file
As libcurl no longer has any functionality for them, the tool now does
nothing with them.

Closes #8670
2022-05-17 11:06:07 +02:00
Daniel Gustafsson
79bf349a50 tool_getparam: simplify conditional statement
param_place cannot be NULL here since we immediately efter this block
perform arithmetic on it (and use it in order to get here) so there is
little reason to check.

Closes: #8786
Reviewed-by: Daniel Stenberg <daniel@haxx.se>
2022-05-02 14:04:59 +02:00
Daniel Stenberg
a09a12806e
tool_getparam: error out on missing -K file
Add test 411 to verify.

Reported-by: Median Median Stride
Bug: https://hackerone.com/reports/1542881
Closes #8731
2022-04-25 11:38:50 +02:00
Ian Blanes
9e5bd9ba19
curl: fix segmentation fault for empty output file names.
Function glob_match_url set *result to NULL when called with filename =
"", producing an indirect NULL pointer dereference.

Closes #8606
2022-03-28 09:39:09 +02:00
HexTheDragon
1831a6e7f1
curl: add --no-clobber
Does not overwrite output files if they already exist

Closes #7708
Co-authored-by: Daniel Stenberg
2022-03-11 08:38:01 +01:00