This should return a SSL_CTX pointer but it was accidentally broken.
Verify with test 1587
Follow-up to 2db8ae480f
Spotted by Codex Security
Closes#21290
Connection filters can now carry the flag CF_TYPE_SETUP, indicating that
they are only needed during connection setup, e.g. connect.
Once the connection is fully established, those filter are removed
again. This frees resources and also makes the filter (call) chains
shorter.
Closes#21269
New connection filter `cf-dns` that manages DNS queries. If hands
out addresses and HTTPS-RR records to anyone interested. Used by
HTTPS and IP happy eyeballing.
Information may become available *before* the libcurl "dns entry"
is complete, e.g. all queries have been answered. The cf-ip-happy
filter uses this information to start connection attempts as soon
as the first address is available.
The multi MSTATE_RESOLVING was removed. A new connection always
goes to MSTATE_CONNECTING. The connectdata bit `dns_resolved`
indicates when DNS information is complete. This is used for
error reporting and starting the progress meter.
Removed dns entries `data->state.dns[i]`, as the `cf-dns` filter
now keeps the reference now.
Many minor tweaks for making this work and pass address information
around safely.
Closes#21027
Use a thread queue and pool for asnyc threaded DNS resolves.
Add pytest test_21_* for verification.
Add `CURLMOPT_RESOLVE_THREADS_MAX` to allow applications to
resize the thread pool used.
Add `CURLMOPT_QUICK_EXIT` to allow applications to skip thread
joins when cleaning up a multi handle. Multi handles in
`curl_easy_perform()` inherit this from `CURLOPT_QUICK_EXIT`.
Add several debug environment variables for testing.
Closes#20936
Also:
- support per-directory and per-upper-directory whitelist entries.
- convert badlist input grep tweak into the above format.
(except for 'And' which had just a few hits.)
- fix many code exceptions, but do not enforce.
(there also remain about 350 'will' uses in lib)
- fix badwords in example code, drop exceptions.
- badwords-all: convert to Perl.
To make it usable from CMake.
- FAQ: reword to not use 'will'. Drop exception.
Closes#20886
- vms/curlmsg_vms.h: delete unused/commented code.
- vtls/schannel_verify: sort includes.
- typecheck-gcc.h: fix indent and alignment.
- lib/config-win32.h: drop idle `#undef`.
- spacecheck: check for stray empty lines before after curly braces.
- make literals more readable: 1048576 -> 1024 * 1024
- scope variables.
- use ISO date in a comment.
- drop redundant parentheses.
- drop empty comments.
- unfold lines.
- duplicate/stray spaces in comments.
- fix indent, whitespace, minor typos.
Closes#20690
- asyn-thrdd.c: scope an include.
- apply more clang-format suggestions.
- tidy-up PP guard comments.
- delete empty line from the top of headers.
- add empty line after `curl_setup.h` include where missing.
- fix indent.
- CODE_STYLE.md: add `strcpy`.
Follow-up to 8636ad55df#20088
- lib1901.c: drop unnecessary line.
Follow-up to 436e67f65b#20076Closes#20070
- drop stray duplicate empty lines in docs, scripts, test data, include,
examples, tests.
- drop duplicate PP parenthesis.
- curl-functions.m4: move literals to the right side in if expressions,
to match rest of the source code.
- FAQ.md: delete language designator from an URL.
- packages: apply clang-format (OS400, VMS).
- scripts/schemetable.c: apply clang-format.
- data320: delete duplicate empty line that doesn't change the outcome.
- spacecheck: extend to check for duplicate empty lines
(with exceptions.)
- fix whitespace nits
Closes#19936
To conclude changing the send buffer type from `const void *` to `const
uint8_t *`, change the top level send function and its implementations.
Closes#19743
Change the send parameter from `const void *` to `const uint8_t *` and
adapt calling code. Several had already unsigned chars and were casting.
Closes#19729
The transfer loop used to check the socket and if no poll events
were seen, triggered a "DATA_IDLE" event into the filters to let
them schedule times/do things anyway.
Since we no longer check the socket, the filters have been called
already and the DATA_IDLE event is unnecessary work. Remove it.
Closes#19060
Rewrite the code that removes a filter from the connection and discards
it. Always look at the connection, otherwise it will not work of the
filter is at the top of the chain.
Change QUIC filter setup code to always tear down the chain in
construction when an error occured.
HTTP proxy, do not remove the h1/h2 sub filter on close. Leave it to be
discarded with the connection. Avoids keeping an additional pointer that
might become dangling.
Triggered by a reported on a code bug in discard method.
Reported-by: Joshua Rogers
Closes#18596
`getsock()` calls operated on a global limit that could
not be configure beyond 16 sockets. This is no longer adequate
with the new happy eyeballing strategy.
Instead, do the following:
- make `struct easy_pollset` dynamic. Starting with
a minimal room for two sockets, the very common case,
allow it to grow on demand.
- replace all protocol handler getsock() calls with pollsets
and a CURLcode to return failures
- add CURLcode return for all connection filter `adjust_pollset()`
callbacks, since they too can now fail.
- use appropriately in multi.c and multi_ev.c
- fix unit2600 to trigger pollset growth
Closes#18164
Since the content varies during connection setup and while doing it
(eyeballing), remove these strcut from `connectdata` and replace use
with querying the connection filters. Those keep that information
already.
Change the info logging of established connections to also give the
local address and port.
Closes#17960
Add a connection filter query to obtained the negotiated ALPN
protocol to check in setup/protocols how the connection needs
to behave.
Remove the members `alpn` and `proxy_alpn` from `connectdata`.
Closes#17947
The `transport` to use for a transfer, e.g. TCP/QUIC/UNIX/UDP, is
initially selected by options and protocol used. This is set at the
`struct connectdata` as `transport` member.
During connection establishment, this transport may change due to
Alt-Svc or Happy-Eyeballing. Most common is the switch from TCP to QUIC.
Rename the connection member to `transport_wanted` and add a way to
query the connection for the transport in use via a new connection
filter query.
The filter query can also be used in the happy eyeballing attempts when
code needs to know which transport is used by the "filter below". This
happens in wolfssl initialization, as one example.
Closes#17923
Replace the old Curl_ssl_get_internals() with a new connection filter
query to retrieve the information. Implement that filter query for TCP
and QUIC TLS filter types.
Add tests in client tls_session_reuse to use the info option and check
that pointers are returned.
Reported-by: Larry Campbell
Fixes#17801Closes#17809
Change the signature of `bufq` functions from
* `ssize_t Curl_bufq_*(..., CURLcode *err)` to
* `CURLcode Curl_bufq_*(..., size_t *pn)`
This allows us to write slightly less code and avoids the ssize_t/size_t
conversions in many cases. Also, it gets the function in line with all
the other send/recv signatures.
Added helper functions in `cfilters.h` for sending from/receving into
a bufq.
Fuzzer now fails to build due to these changes and its testing of
the bufq API.
Closes#17396
Connection filters had a method `get_host()` which had not really been
documented. Since then, the cf had the `query()` method added. Replace
the separate get_host with query.
Add `CF_QUERY_HOST_PORT` as query to connection filters to retrieve
which remote hostname and port the filter (or its sub-filter) is talking
to. The query is implemented by HTTP and SOCKS filters, all others pass
it through.
Add `Curl_conn_get_current_host()` to retrieve the remote host and port
for a connection. During connect, this will return the host the
connection is talking to right now. Before/After connect, this will
return `conn->host.name`.
This is used by SASL authentication.
Closes#17419
Used to be a pointer set (and cleared) by the socket connection filters
to a struct in their contexts. Instead, add a filter query method to
obtain the pointer when needed.
Closes#17385
When multi creates the pollset of a transfer, it checks now if
a connection (FIRST/SECONDARY) socket waits on POLLIN and has input data
pending in filters (relevant to OpenSSL's new read ahead). If so, it
triggers a timeout on the transfer via EXPIRE_RUN_NOW.
This fixes sporadic stalls in test 988 when running event based.
Closes#17636
cfilter/conn: change send/recv function signatures. Unify the
calling/return conventions in our send/receive handling.
Curl_conn_recv(), adjust pnread type
Parameter `pnread` was a `ssize_t *`, but `size_t *` is better since the
function returns any error in its `CURLcode` return value.
Closes#17546
When ftp_done() is called to terminate the transfer, it needs to tear
down any open SECONDARY filter chain. The condition on when to do that
was relying on there to be a valid socket. This is not sufficient as the
socket is only set *after* happy eyeballing has decided on one.
Instead of checking for a valid conn->sock, check if any connection
filter is installed.
Fixes#17482
Reported-by: Rasmus Melchior Jacobsen
Closes#17491
Move curlx_ functions into its own subdir.
The idea is to use the curlx_ prefix proper on these functions, and use
these same function names both in tool, lib and test suite source code.
Stop the previous special #define setup for curlx_ names.
The printf defines are now done for the library alone. Tests no longer
use the printf defines. The tool code sets its own defines. The printf
functions are not curlx, they are publicly available.
The strcase defines are not curlx_ functions and should not be used by
tool or server code.
dynbuf, warnless, base64, strparse, timeval, timediff are now proper
curlx functions.
When libcurl is built statically, the functions from the library can be
used as-is. The key is then that the functions must work as-is, without
having to be recompiled for use in tool/tests. This avoids symbol
collisions - when libcurl is built statically, we use those functions
directly when building the tool/tests. When libcurl is shared, we
build/link them separately for the tool/tests.
Assisted-by: Jay Satiro
Closes#17253
The issues found fell into these categories, with the applied fixes:
- const was accidentally stripped.
Adjust code to not cast or cast with const.
- const/volatile missing from arguments, local variables.
Constify arguments or variables, adjust/delete casts. Small code
changes in a few places.
- const must be stripped because an API dependency requires it.
Strip `const` with `CURL_UNCONST()` macro to silence the warning out
of our control. These happen at API boundaries. Sometimes they depend
on dependency version, which this patch handles as necessary. Also
enable const support for the zlib API, using `ZLIB_CONST`. Supported
by zlib 1.2.5.2 and newer.
- const must be stripped because a curl API requires it.
Strip `const` with `CURL_UNCONST()` macro to silence the warning out
of our immediate control. For example we promise to send a non-const
argument to a callback, though the data is const internally.
- other cases where we may avoid const stripping by code changes.
Also silenced with `CURL_UNCONST()`.
- there are 3 places where `CURL_UNCONST()` is cast again to const.
To silence this type of warning:
```
lib/vquic/curl_osslq.c:1015:29: error: to be safe all intermediate
pointers in cast from 'unsigned char **' to 'const unsigned char **'
must be 'const' qualified [-Werror=cast-qual]
lib/cf-socket.c:734:32: error: to be safe all intermediate pointers in
cast from 'char **' to 'const char **' must be 'const' qualified
[-Werror=cast-qual]
```
There may be a better solution, but I couldn't find it.
These cases are handled in separate subcommits, but without further
markup.
If you see a `-Wcast-qual` warning in curl, we appreciate your report
about it.
Closes#16142
Rework the event based handling of transfers and connections to
be "localized" into a single source file with clearer dependencies.
- add multi_ev.c and multi_ev.h
- add docs/internal/MULTI-EV.md to explain the overall workings
- only do event handling book keeping when the socket callback
is set
- add handling for "connection only" event tracking, when internal
easy handles are used that are not really tied to a connection.
Used in connection pool.
- remove transfer member "last_poll" and connections "shutdown_poll"
and keep all that internal to multi_ev.c
- add CURL_TRC_M() for tracing of "multi" related things, including
event handling and connection pool operations. Add new trace
feature "multi" for trace config.
multi traces will show exactly what is going on in regard to
event handling.
- multi: trace transfers "mstate" in every CURL_TRC_M() call
- make internal trace buffer 2048 bytes and end the silliness
with +n here -m there. Adjust test 1652 expectations of resulting
length and input edge cases.
- add trace feature "lib-ids" to perfix libcurl traces with transfer
and connection ids. Useful for debugging libcurl applications.
Closes#16308
Remove `blocking` argument from cfilter's connect method.
Implement blocking behaviour in Curl_conn_connect() instead for all
filter chains.
Update filters implementations. Several of which did never use the
paramter (QUIC for example). Simplifies connect handling in TLS filters
that no longer need to loop
Fixed a blocking connect call in FTP when waiting on a socket accept()
which only worked because the filter did not implement it.
Closes#16397
Translate the `data->set.httpwant` which is one of the consts from the
public API (CURL_HTTP_VERSION_*) into a major version mask plus
additional flags for internal handling.
`Curl_http_neg_init()` does the translation and flags setting in http.c,
using new internal consts CURL_HTTP_V1x, CURL_HTTP_V2x and CURL_HTTP_V3x
for the major versions. The flags are
- only_10: when the application explicity asked fro HTTP/1.0
- h2_upgrade: when the application asks for upgrading 1.1 to 2.
- h2_prior_knowledge: when directly talking h2 without ALPN
- accept_09: when a HTTP/0.9 response is acceptable.
The Alt-Svc and HTTPS RR redirections from one ALPN to another obey the
allowed major versions. If a transfer has only h3 enabled, Alt-Svc
redirection to h2 is ignored.
This is the current implementation. It can be debated if Alt-Svc should
be able to override the allowed major versions. Added test_12_06 to
verify the current restriction.
Closes#16100
The variable `conn->httpversion` was used for several purposes and it
was unclear at which time the value represents what.
- rename `conn->httpversion` to `conn->httpversion_seen`
This makes clear that the variable only records the last
HTTP version seen on the connection - if any. And that it
no longer is an indication of what version to use.
- Change Alt-Svc handling to no longer modify `conn->httpversion`
but set `data->state.httpwant` for influencing the HTTP version
to use on a transfer.
- Add `data->req.httpversion_sent` to have a record of what
HTTP version was sent in a request
- Add connection filter type CF_TYPE_HTTP
- Add filter query `CF_QUERY_HTTP_VERSION` to ask what HTTP
filter version is in place
- Lookup filters HTTP version instead of using `conn->httpversion`
Test test_12_05 now switches to HTTP/1.1 correctly and the
expectations have been fixed.
Removed the connection fitler "is_httpN()" checks and using
the version query instead.
Closes#16073
Make transfer attach/detach to/from connections chepaer.
- the "attach" event was no longer implemented by any filter
- the "detach" did the same as the "done" event for the filters
who still implemented it. It should be superfluous as the "done"
must always happen.
Closes#16067
Update IP related information at the connection and the transfer in two
places only: once the filter chain connects and when a transfer is added
to a connection. The latter only updates on reuse when the filters
already are connected.
The only user of that information before a full connect is the HAProxy
filter. Add cfilter CF_QUERY_IP_INFO query to let it find the
information from the filters "below".
This solves two issues with the previous version:
- updates where often done twice with the same info
- happy eyeballing filter "forks" could overwrite each others
updates before the full winner was determined.
Closes#14699
- replace the counting of upload lengths with the new eos send flag
- improve frequency of stream draining to happen less on events where it
is not needed
- this PR is based on #14220
http2, cf-h2-proxy: fix EAGAINed out buffer
- in adjust pollset and shutdown handling, a non-empty `ctx->outbufq`
must trigger send polling, irregardless of http/2 flow control
- in http2, fix retry handling of blocked GOAWAY frame
test case improvement:
- let client 'upload-pausing' handle http versions
Closes#14253
Since data can be held in connection filter buffers when sending gives
EAGAIN, add methods to query this and perform flushing of those buffers.
The transfer loop will continue sending until all upload data is
processed and the connection is flushed.
- add `CF_QUERY_SEND_PENDING` to query filters
- add `CF_CTRL_DATA_SEND_FLUSH` to flush filters
- change `Curl_req_want_send()` to query the connection
if it needs flushing
- use `Curl_req_want_send()` to determine the POLLOUT
in the PERFORMING multi state
- implement flush handling in the HTTP/2 connection filter
Closes#14271
Adds a `bool eos` flag to send methods to indicate that the data
is the last chunk the invovled transfer wants to send to the server.
This will help protocol filters like HTTP/2 and 3 to forward the
stream's EOF flag and also allow to EAGAIN such calls when buffers
are not yet fully flushed.
Closes#14220
Adds a `bool eos` flag to send methods to indicate that the data is the
last chunk the invovled transfer wants to send to the server.
This will help protocol filters like HTTP/2 and 3 to forward the
stream's EOF flag and also allow to EAGAIN such calls when buffers are
not yet fully flushed.
Closes#14220
Based on the standards and guidelines we use for our documentation.
- expand contractions (they're => they are etc)
- host name = > hostname
- file name => filename
- user name = username
- man page => manpage
- run-time => runtime
- set-up => setup
- back-end => backend
- a HTTP => an HTTP
- Two spaces after a period => one space after period
Closes#14073
When libcurl discards a connection there are two phases this may go
through: "shutdown" and "closing". If a connection is aborted, the
shutdown phase is skipped and it is closed right away.
The connection filters attached to the connection implement the phases
in their `do_shutdown()` and `do_close()` callbacks. Filters carry now a
`shutdown` flags next to `connected` to keep track of the shutdown
operation.
Filters are shut down from top to bottom. If a filter is not connected,
its shutdown is skipped. Notable filters that *do* something during
shutdown are HTTP/2 and TLS. HTTP/2 sends the GOAWAY frame. TLS sends
its close notify and expects to receive a close notify from the server.
As sends and receives may EAGAIN on the network, a shutdown is often not
successful right away and needs to poll the connection's socket(s). To
facilitate this, such connections are placed on a new shutdown list
inside the connection cache.
Since managing this list requires the cooperation of a multi handle,
only the connection cache belonging to a multi handle is used. If a
connection was in another cache when being discarded, it is removed
there and added to the multi's cache. If no multi handle is available at
that time, the connection is shutdown and closed in a one-time,
best-effort attempt.
When a multi handle is destroyed, all connection still on the shutdown
list are discarded with a final shutdown attempt and close. In curl
debug builds, the environment variable `CURL_GRACEFUL_SHUTDOWN` can be
set to make this graceful with a timeout in milliseconds given by the
variable.
The shutdown list is limited to the max number of connections configured
for a multi cache. Set via CURLMOPT_MAX_TOTAL_CONNECTIONS. When the
limit is reached, the oldest connection on the shutdown list is
discarded.
- In multi_wait() and multi_waitfds(), collect all connection caches
involved (each transfer might carry its own) into a temporary list.
Let each connection cache on the list contribute sockets and
POLLIN/OUT events it's connections are waiting for.
- in multi_perform() collect the connection caches the same way and let
them peform their maintenance. This will make another non-blocking
attempt to shutdown all connections on its shutdown list.
- for event based multis (multi->socket_cb set), add the sockets and
their poll events via the callback. When `multi_socket()` is invoked
for a socket not known by an active transfer, forward this to the
multi's cache for processing. On closing a connection, remove its
socket(s) via the callback.
TLS connection filters MUST NOT send close nofity messages in their
`do_close()` implementation. The reason is that a TLS close notify
signals a success. When a connection is aborted and skips its shutdown
phase, the server needs to see a missing close notify to detect
something has gone wrong.
A graceful shutdown of FTP's data connection is performed implicitly
before regarding the upload/download as complete and continuing on the
control connection. For FTP without TLS, there is just the socket close
happening. But with TLS, the sent/received close notify signals that the
transfer is complete and healthy. Servers like `vsftpd` verify that and
reject uploads without a TLS close notify.
- added test_19_* for shutdown related tests
- test_19_01 and test_19_02 test for TCP RST packets
which happen without a graceful shutdown and should
no longer appear otherwise.
- add test_19_03 for handling shutdowns by the server
- add test_19_04 for handling shutdowns by curl
- add test_19_05 for event based shutdowny by server
- add test_30_06/07 and test_31_06/07 for shutdown checks
on FTP up- and downloads.
Closes#13976
- clarify Curl_xfer_setup() with RECV/SEND flags and different calls for
which socket they operate on. Add a shutdown flag for secondary
sockets
- change Curl_xfer_setup() calls to new functions
- implement non-blocking connection shutdown at the end of receiving or
sending a transfer
Closes#13913
This adds connection shutdown infrastructure and first use for FTP. FTP
data connections, when not encountering an error, are now shut down in a
blocking way with a 2sec timeout.
- add cfilter `Curl_cft_shutdown` callback
- keep a shutdown start timestamp and timeout at connectdata
- provide shutdown timeout default and member in
`data->set.shutdowntimeout`.
- provide methods for starting, interrogating and clearing
shutdown timers
- provide `Curl_conn_shutdown_blocking()` to shutdown the
`sockindex` filter chain in a blocking way. Use that in FTP.
- add `Curl_conn_cf_poll()` to wait for socket events during
shutdown of a connection filter chain.
This gets the monitoring sockets and events via the filters
"adjust_pollset()" methods. This gives correct behaviour when
shutting down a TLS connection through a HTTP/2 proxy.
- Implement shutdown for all socket filters
- for HTTP/2 and h2 proxying to send GOAWAY
- for TLS backends to the best of their capabilities
- for tcp socket filter to make a final, nonblocking
receive to avoid unwanted RST states
- add shutdown forwarding to happy eyeballers and
https connect ballers when applicable.
Closes#13904
- add `Curl_hash_offt` as hashmap between a `curl_off_t` and
an object. Use this in h2+h3 connection filters to associate
`data->id` with the internal stream state.
- changed implementations of all affected connection filters
- removed `h2_ctx*` and `h3_ctx*` from `struct HTTP` and thus
the easy handle
- solves the problem of attaching "foreign protocol" easy handles
during connection shutdown
Test 1616 verifies the new hash functions.
Closes#13204
