- By making retry_sleep() a separate funtion that determines how long to
wait until the next retry.
- switch the retry timer to uint32 to make it uniform across platforms
Closes#21279
Also:
- support per-directory and per-upper-directory whitelist entries.
- convert badlist input grep tweak into the above format.
(except for 'And' which had just a few hits.)
- fix many code exceptions, but do not enforce.
(there also remain about 350 'will' uses in lib)
- fix badwords in example code, drop exceptions.
- badwords-all: convert to Perl.
To make it usable from CMake.
- FAQ: reword to not use 'will'. Drop exception.
Closes#20886
- asyn-thrdd.c: scope an include.
- apply more clang-format suggestions.
- tidy-up PP guard comments.
- delete empty line from the top of headers.
- add empty line after `curl_setup.h` include where missing.
- fix indent.
- CODE_STYLE.md: add `strcpy`.
Follow-up to 8636ad55df#20088
- lib1901.c: drop unnecessary line.
Follow-up to 436e67f65b#20076Closes#20070
- move the state struct from config to global. It is used as a single
instance anyway so might as well be a single one to save memory.
- simplify and combine several conditions
- set default retry delay inititally
- use better struct field names to make it easier to understand their
purposes
- remove the state->outfiles field as it was not necessary
- remove superfluous glob cleanup call
- move conditions around to remove an indent level
- move the ->url NULL check
Takes single_transfer()'s complexity score down from 78 to 68.
Closes#18226
The GlobalConfig only exists in a single instance and it has worked like
this since the dawn of time. It is about time we stop passing around
pointers to what was already essentially a global object and instead
just use a... global.
It simplifies things.
Closes#18213
Otherwise we could misbehave already at 2 billion URLs and we can't have
that. A few of the counters are already correctly using the right type.
Closes#18096
Use 'config' for pointing to a OperationConfig
Use 'global' for pointing to GlobalConfig
Bonus: add config_alloc(), an easier way to allocate + init a new
OperationConfig struct.
Closes#17888
The curl tool and tests/server used 2 parallel implementations
of libcurl's `Curl_now()` and `Curl_timediff()` functions.
Make them use the libcurl one.
Closes#16653
- split up in a few smaller and easier to read functions
- simplify several sections
- avoid superfluous extra allocations
- remove unused debug code
Closes#15385
With this option, the entire download is skipped if the selected target
filename already exists when the opertion is about to begin.
Test 994, 995 and 996 verify.
Ref: #11012Closes#13993
- they are mostly pointless in all major jurisdictions
- many big corporations and projects already don't use them
- saves us from pointless churn
- git keeps history for us
- the year range is kept in COPYING
checksrc is updated to allow non-year using copyright statements
Closes#10205
The read callback can timeout if there's nothing to read within the
given maximum period. Example use case is when doing "curl -m 3
telnet://example.com" or anything else that expects input on stdin or
similar that otherwise would "hang" until something happens and then not
respect the timeout.
This fixes KNOWN_BUG 8.1, first filed in July 2009.
Bug: https://sourceforge.net/p/curl/bugs/846/Closes#9815
In some circumstances when doing parallel transfers, the
single_transfer_cleanup() would not be called and then 'inglob' could
leak.
Test 496 verifies
Reported-by: Trail of Bits
Closes#9749
- parallel transfers: only alloc and keep errorbuffers in memory for
actual "live" transfers and not for the ones in the pending queue
- serial transfers: reuse the same fixed buffer for all transfers, not
allocated at all.
Closes#9394
Add licensing and copyright information for all files in this repository. This
either happens in the file itself as a comment header or in the file
`.reuse/dep5`.
This commit also adds a Github workflow to check pull requests and adapts
copyright.pl to the changes.
Closes#8869
Unless muted (with -s) When doing globbing, curl would output mime-like
separators between the separate transfers. This is not documented
anywhere, surprises users and clobbers the output. Gone now.
Updated test 18 and 1235
Reported-by: jonny112 on github
Bug: https://github.com/curl/curl/discussions/8257Closes#8278
- Abort via progress callback to fail early during parallel transfers.
When a critical error occurs during a transfer (eg --fail-early
constraint) then other running transfers will be aborted via progress
callback and finish with error CURLE_ABORTED_BY_CALLBACK (42). In this
case, the callback error does not become the most recent error and a
custom error message is used for those transfers:
curld --fail --fail-early --parallel
https://httpbin.org/status/404https://httpbin.org/delay/10
curl: (22) The requested URL returned error: 404
curl: (42) Transfer aborted due to critical error in another transfer
> echo %ERRORLEVEL%
22
Fixes https://github.com/curl/curl/issues/6939
Closes https://github.com/curl/curl/pull/6984
Warning: this will make existing curl command lines that use metalink to
stop working.
Reasons for removal:
1. We've found several security problems and issues involving the
metalink support in curl. The issues are not detailed here. When
working on those, it become apparent to the team that several of the
problems are due to the system design, metalink library API and what
the metalink RFC says. They are very hard to fix on the curl side
only.
2. The metalink usage with curl was only very briefly documented and was
not following the "normal" curl usage pattern in several ways, making
it surprising and non-intuitive which could lead to further security
issues.
3. The metalink library was last updated 6 years ago and wasn't so
active the years before that either. An unmaintained library means
there's a security problem waiting to happen. This is probably reason
enough.
4. Metalink requires an XML parsing library, which is complex code (even
the smaller alternatives) and to this day often gets security
updates.
5. Metalink is not a widely used curl feature. In the 2020 curl user
survey, only 1.4% of the responders said that they'd are using it. In
2021 that number was 1.2%. Searching the web also show very few
traces of it being used, even with other tools.
6. The torrent format and associated technology clearly won for
downloading large files from multiple sources in parallel.
Cloes #7176
In particular, these ones can help a user to create its own error
message when one or transfers fail.
writeout: add 'onerror', 'url', 'urlnum', 'exitcode', 'errormsg'
onerror - lets a user only show the rest on non-zero exit codes
url - the input URL used for this transfer
urlnum - the numerical URL counter (0 indexed) for this transfer
exitcode - the numerical exit code for the transfer
errormsg - obvious
Reported-by: Earnestly on github
Fixes#6199Closes#6207
The previous sleep for retries would block all other concurrent
transfers. Starting now, the retry will instead be properly marked to
not get restarted until after the delay time but other transfers can
still continue in the mean time.
Closes#5917
- Stick to a single unified way to use structs
- Make checksrc complain on 'typedef struct {'
- Allow them in tests, public headers and examples
- Let MD4_CTX, MD5_CTX, and SHA256_CTX typedefs remain as they actually
typedef different types/structs depending on build conditions.
Closes#5338
Attempt to unpause a busy read in the CURLOPT_XFERINFOFUNCTION.
When uploading from stdin in non-blocking mode, a delay in reading
the stream (EAGAIN) causes curl to pause sending data
(CURL_READFUNC_PAUSE). Prior to this change, a busy read was
detected and unpaused only in the CURLOPT_WRITEFUNCTION handler.
This change performs the same busy read handling in a
CURLOPT_XFERINFOFUNCTION handler.
Fixes#2051Closes#4599
Reported-by: bdry on github
This is done by making sure each individual transfer is first added to a
linked list as then they can be performed serially, or at will, in
parallel.
Closes#3804
- replace tabs with spaces where possible
- remove line ending spaces
- remove double/triple newlines at EOF
- fix a non-UTF-8 character
- cleanup a few indentations/line continuations
in manual examples
Closes https://github.com/curl/curl/pull/3037
To help assist with the detection of incorrect return codes, as per
commits ee23d13a79, 33b8960dc8 and aba98991a5, updated the operate
based functions to return CURLcode error codes.
Configuration files such as curl_config.h and all config-*.h no longer exist
nor are generated/copied into 'src' directory, now these only exist in 'lib'
directory from where curl tool sources uses them.
Additionally old src/setup.h has been refactored into src/tool_setup.h which
now pulls lib/setup.h
The possibility of a makefile needing an include path adjustment exists.
