Commit graph

24 commits

Author SHA1 Message Date
Stefan Eissing
2b3dfb4ad4
lib: make resolving HTTPS DNS records reliable:
- allow to specify when they are wanted on starting a resolve
- match dns cache entries accordingly. An entry which never
  tried to get HTTPS-RRs is no answer for a resolve that wants
  it.
- fix late arrivals of resolve answers to match the "async"
  records that started them - if it still exists.
- provide for multiple "async" resolves in a transfer at the
  same time. We may need to resolve an IP interface while the
  main connection resolve has not finished yet.
- allow lookup of HTTPS-RR information as soon as it is
  available, even if A/AAAA queries are still ongoing.

For this, the "async" infrastructure is changed:

- Defined bits for DNS queries `CURL_DNSQ_A`, `CURL_DNSQ_AAAA`
  and `CURL_DNSQ_HTTPS`. These replace `ip_version` which says
  nothing about HTTPS.
  Use them in dns cache entries for matching.
- enhance the `async->id` to be a unique `uint32_t` for
  resolves inside one multi. This is weak, as the id may
  wrap around. However it is combined with the `mid` of
  the easy handle, making collisions highly unlikely.
  `data->state.async` is only accessed in few places where
  the mid/async-id match is performed.
- vtls: for ECH supporting TLS backends (openssl, rustls, wolfssl),
  retrieve the HTTPS-RR information from the dns connection filter.
  Delay the connect if the HTTPS-RR is needed, but has not
  been resolved yet.

The implementation of all this is complete for the threaded
resolver. c-ares resolver and DoH do not take advantage of
all new async features yet. To be done in separate PRs.

Details:

c-ares: cleanup settings and initialisation. Any ares channel
is only being created on starting a resolve and propagating
operations in setopt.c to the channel are not helpful.

Changed threaded+ares pollset handling so that they do not
overwrite each others `ASYNC_NAME` timeouts.

Add trace name 'threads' for tracing thread queue and
pool used by threaded resolver.

Closes #21175
2026-04-01 15:36:31 +02:00
Daniel Stenberg
b4dba346cd
stop using the word 'just'
Everywhere. In documentation and code comments.

It is almost never a good word and almost always a filler that should be
avoided.

Closes #20793
2026-03-03 15:30:22 +01:00
Viktor Szakats
61273f5812
badwords: catch and fix threading-related words
Also:
- sync newlines between the two threaded examples.

Closes #20001
2025-12-16 21:26:58 +01:00
Stefan Eissing
12a3182fc3
ssh: tracing and better pollset handling
Remove connection member `waitfor` and keep it in the SSH connection
meta. Add `ssh` to supported tracing features, convert many DEBUGF
printgs to traces.

Closes #19745
2025-11-28 16:05:43 +01:00
Stefan Eissing
b022389757
ip-happy: do not set unnecessary timeout
When attempts on all addresses have been started, do no longer set any
EXPIRE_HAPPY_EYEBALLS timeouts.

Fixes #18767
Reported-by: Johannes Schindelin
Closes #18768
2025-10-01 08:01:47 +02:00
Daniel Stenberg
daa8693619
VERSIONS: list all past releases
This document now lists all previous releases.

This allows us to verify that documentation refers to actual release
versions.

Test 971 now verifies options-in-versions and all command line options
documentation individually. Fixed a few discrepancies.

Test 1488 verifies libcurl options "Added-in" to exist. Fixed a few
discrepancies there as well.

Closes #16907
2025-04-03 08:26:33 +02:00
Stefan Eissing
cfc657a48d
multi: event based rework
Rework the event based handling of transfers and connections to
be "localized" into a single source file with clearer dependencies.

- add multi_ev.c and multi_ev.h
- add docs/internal/MULTI-EV.md to explain the overall workings
- only do event handling book keeping when the socket callback
  is set
- add handling for "connection only" event tracking, when internal
  easy handles are used that are not really tied to a connection.
  Used in connection pool.
- remove transfer member "last_poll" and connections "shutdown_poll"
  and keep all that internal to multi_ev.c
- add CURL_TRC_M() for tracing of "multi" related things, including
  event handling and connection pool operations. Add new trace
  feature "multi" for trace config.
  multi traces will show exactly what is going on in regard to
  event handling.
- multi: trace transfers "mstate" in every CURL_TRC_M() call
- make internal trace buffer 2048 bytes and end the silliness
  with +n here -m there. Adjust test 1652 expectations of resulting
  length and input edge cases.
- add trace feature "lib-ids" to perfix libcurl traces with transfer
  and connection ids. Useful for debugging libcurl applications.

Closes #16308
2025-02-22 14:47:40 +01:00
Stefan Eissing
1b710381ca
https-rr: implementation improvements
- fold DoH and async HTTPS-RR handling into common code.
  have common cleanups, etc. Have a CURLcode result in async
  handling to allow HTTPS RR parsing to fail.
- keep target, ipv4hints, ipv6hints, port and echconfig also
  when resolving via cares. We need to know `target` and `port`
  when evaluating possible ALPN candidates to not go astray.
- add CURL_TRC_DNS for tracing DNS operations
- replace DoH specific tracing with DNS, use doh as alias
  for dns in curl_global_tracea()

Closes #16132
2025-02-18 16:12:26 +01:00
Stefan Eissing
515a21f350
vtls: feature ssls-export for SSL session im-/export
Adds the experimental feature `ssls-export` to libcurl and curl for
importing and exporting SSL sessions from/to a file.

* add functions to libcurl API
* add command line option `--ssl-sessions <filename>` to curl
* add documenation
* add support in configure
* add support in cmake
+ add pytest case

Closes #15924
2025-01-08 23:32:07 +01:00
Stefan Eissing
1c42ea4066
smtp: add tracing feature
Add `smtp` as tracing feature, use CURL_TRC_SMTP() in code to
trace operations.

Closes #14531
2024-08-14 11:51:55 +02:00
Stefan Eissing
3e64569a9e
websocket: introduce blocking sends
When using `curl_ws_send()`, perform a blocking send of the data under
the following conditions:

- the websocket is in raw mode and the call is done from within a curl
  callback. A partial write of the data could subsequently mess up the
  ws framing, as a callback has a hard time handling this.

- the websocket is encoding the data itself, has added it to its
  internal sendbuf. A partial flush of the buffer has unclear semantics
  for the caller, as they will have no idea what to send again.

Fixes WebSockets tests with CURL_DBG_SOCK_WBLOCK=90 set.
Closes #14458
2024-08-12 19:19:28 +02:00
Stefan Eissing
06c5829dab
curl: support repeated use of the verbose option; -vv etc
- make mentioning `-v` on the curl command line increase the
  verbosity of the trace output
- related discussion https://github.com/curl/curl/discussions/13810
- make a single -v revert all previous -v+ changes
- make --no-verbose also reset all trace configs

Closes #13977
2024-08-07 08:14:36 +02:00
Daniel Stenberg
5a488251f7
curldown: fixups
- make DEFAULT sections less repetitive

- make historic mentions use HISTORY

- generate the protocols section on `# %PROTOCOLS%` instead of guessing
  where to put it

- generate the availability section on `# %AVAILABILITY%` instead of
  guessing where to put it

- make the protocols section more verbose

Closes #14227
2024-07-19 17:03:25 +02:00
Daniel Stenberg
8c1d9378ac
curldown: make 'added-in:' a mandatory header field
- generate AVAILABILITY manpage sections automatically - for consistent
  wording

- allows us to double-check against other documumentation (symbols-in-versions
  etc)

- enables proper automation/scripting based on this data

- lots of them were wrong or missing in the manpages

- several of them repeated (sometimes mismatching) backend support info

Add test 1488 to verify "added-in" version numbers against
symbols-in-versions.

Closes #14217
2024-07-18 18:04:09 +02:00
Stefan Eissing
b7c7dffe35
ftp: add tracing support
- add `Curl_trc_feat_ftp` for tracing via trace config
- add macro CURL_TRC_FTP(data, fmt, ...)
- replace DEBUGF(infof()) statements in ftp.c by CURL_TRC_FTP()
- always trace FTP connection state

Closes #13580
2024-05-10 23:39:48 +02:00
Stefan Eissing
0b28ece657
lib: add trace support for client reads and writes
- add `CURL_TRC_READ()` and `CURL_TRC_WRITE()`
- use in generic client writers and readers, as well
  as http headers, chunking and websockets

Closes #13223
2024-04-05 16:08:10 +02:00
Daniel Stenberg
df3d5ef25e
curl_global_trace.md: shorten the description
Closes #13263
2024-04-02 23:39:39 +02:00
Daniel Stenberg
e3fe020089
docs/libcurl: generate PROTOCOLS from meta-data
Remove the PROTOCOLS section from the source files completely and
instead generate them based on the header data in the curldown files.

It also generates TLS backend information for options marked for TLS as
protocol.

Closes #13175
2024-03-23 18:13:03 +01:00
Daniel Stenberg
b935fd4a07
docs: make each libcurl man specify protocol(s)
The mandatory header now has a mandatory list of protocols for which the
manpage is relevant.

Most man pages already has a "PROTOCOLS" section, but this introduces a
stricter way to specify the relevant protocols.

cd2nroff verifies that at least one protocol is mentioned (which can be
`*`).

This information is not used just yet, but A) the PROTOCOLS section can
now instead get generated and get a unified wording across all manpages
and B) this allows us to more reliably filter/search for protocol
specific manpages/options.

Closes #13166
2024-03-21 15:27:06 +01:00
Daniel Gustafsson
8d67c61c47 curldown: Fix email address in Copyright
The curldown conversion accidentally replaced daniel@haxx.se with
just daniel.se.  This reverts back to the proper email address in
the curldown docs as well as in a few other stray places where it
was incorrect (while unrelated to curldown).

Reviewed-by: Daniel Stenberg <daniel@haxx.se>
Closes: #12997
2024-02-28 11:28:10 +01:00
Stefan Eissing
f7e598791f
DoH: add trace configuration
- refs #12397 where it is dicussed how to en-/disable verbose output
  of DoH operations
- introducing `struct curl_trc_feat` to track a curl feature for
  tracing
- adding `data->state.feat` optionally pointing to the feature a
  transfer belongs to
- adding trace functions and verbosity checks on features
- using trace feature in DoH code
- documenting `doh` as feature for `--trace-config`

Closes #12411
2024-02-20 14:01:59 +01:00
Daniel Stenberg
96af350661
libcurl-docs: cleanups
CURLMOPT_SOCKETDATA.md: fix typo
 CURLMOPT_TIMERDATA.md: fix typo
 CURLOPT_COOKIELIST.m: quote strings
 CURLOPT_PREREQFUNCTION.md: quote variable names
 CURLOPT_TCP_NODELAY.md: rephrased to please spell checker
 CURLOPT_WILDCARDMATCH.md: rephrased
 libcurl-tutorial.md: use correct option name
 curl_global_init_mem.md: quote headers
 curl_easy_getinfo.md: use correct symbol names in headers
 curl_global_trace.md: quote some headers
 curl_ws_meta.md: quote struct field names
 libcurl-env.md: quote headers
2024-02-19 11:41:12 +01:00
Daniel Stenberg
e5000e797f
GHA: add a job scanning for "bad words" in markdown
This means words, phrases or things we have decided not to use - words that
are spelled right according to the dictionary but we want to avoid. In the
name of consistency and better documentation.

Closes #12764
2024-01-24 08:44:34 +01:00
Daniel Stenberg
eefcc1bda4
docs: introduce "curldown" for libcurl man page format
curldown is this new file format for libcurl man pages. It is markdown
inspired with differences:

- Each file has a set of leading headers with meta-data
- Supports a small subset of markdown
- Uses .md file extensions for editors/IDE/GitHub to treat them nicely
- Generates man pages very similar to the previous ones
- Generates man pages that still convert nicely to HTML on the website
- Detects and highlights mentions of curl symbols automatically (when
  their man page section is specified)

tools:

- cd2nroff: converts from curldown to nroff man page
- nroff2cd: convert an (old) nroff man page to curldown
- cdall: convert many nroff pages to curldown versions
- cd2cd: verifies and updates a curldown to latest curldown

This setup generates .3 versions of all the curldown versions at build time.

CI:

Since the documentation is now technically markdown in the eyes of many
things, the CI runs many more tests and checks on this documentation,
including proselint, link checkers and tests that make sure we capitalize the
first letter after a period...

Closes #12730
2024-01-23 00:29:02 +01:00
Renamed from docs/libcurl/curl_global_trace.3 (Browse further)