mirror of
https://github.com/curl/curl.git
synced 2026-08-24 13:43:32 +03:00
INTERNALS: cat lib/README* >> INTERNALS
and a conversion to markdown. Removed the lib/README.* files. The idea being to move toward having INTERNALS as the one and only "book" of internals documentation. Added a TOC to top of the document.
This commit is contained in:
Daniel Stenberg
parent
cbf2920d02
commit
55f3eb588d
10 changed files with 528 additions and 627 deletions
|
|
@ -21,9 +21,6 @@
|
|||
###########################################################################
|
||||
AUTOMAKE_OPTIONS = foreign nostdinc
|
||||
|
||||
DOCS = README.encoding README.memoryleak README.ares README.curlx \
|
||||
README.hostip README.multi_socket README.httpauth README.curl_off_t
|
||||
|
||||
CMAKE_DIST = CMakeLists.txt curl_config.h.cmake
|
||||
|
||||
EXTRA_DIST = Makefile.b32 Makefile.m32 Makefile.vc6 config-win32.h \
|
||||
|
|
@ -31,7 +28,7 @@ EXTRA_DIST = Makefile.b32 Makefile.m32 Makefile.vc6 config-win32.h \
|
|||
makefile.dj config-dos.h libcurl.plist libcurl.rc config-amigaos.h \
|
||||
makefile.amiga Makefile.netware nwlib.c nwos.c config-win32ce.h \
|
||||
config-os400.h setup-os400.h config-symbian.h Makefile.Watcom \
|
||||
config-tpf.h $(DOCS) mk-ca-bundle.pl mk-ca-bundle.vbs $(CMAKE_DIST) \
|
||||
config-tpf.h mk-ca-bundle.pl mk-ca-bundle.vbs $(CMAKE_DIST) \
|
||||
firefox-db2pem.sh config-vxworks.h Makefile.vxworks checksrc.pl \
|
||||
objnames-test08.sh objnames-test10.sh objnames.inc checksrc.whitelist
|
||||
|
||||
|
|
|
|||
|
|
@ -1,69 +0,0 @@
|
|||
_ _ ____ _
|
||||
___| | | | _ \| |
|
||||
/ __| | | | |_) | |
|
||||
| (__| |_| | _ <| |___
|
||||
\___|\___/|_| \_\_____|
|
||||
|
||||
How To Build libcurl to Use c-ares For Asynch Name Resolves
|
||||
===========================================================
|
||||
|
||||
c-ares:
|
||||
http://c-ares.haxx.se/
|
||||
|
||||
NOTE
|
||||
The latest libcurl version requires c-ares 1.6.0 or later.
|
||||
|
||||
Once upon the time libcurl built fine with the "original" ares. That is no
|
||||
longer true. You need to use c-ares.
|
||||
|
||||
Build c-ares
|
||||
============
|
||||
|
||||
1. unpack the c-ares archive
|
||||
2. cd c-ares-dir
|
||||
3. ./configure
|
||||
4. make
|
||||
5. make install
|
||||
|
||||
Build libcurl to use c-ares in the curl source tree
|
||||
===================================================
|
||||
|
||||
1. name or symlink the c-ares source directory 'ares' in the curl source
|
||||
directory
|
||||
2. ./configure --enable-ares
|
||||
|
||||
Optionally, you can point out the c-ares install tree root with the the
|
||||
--enable-ares option.
|
||||
|
||||
3. make
|
||||
|
||||
Build libcurl to use an installed c-ares
|
||||
========================================
|
||||
|
||||
1. ./configure --enable-ares=/path/to/ares/install
|
||||
2. make
|
||||
|
||||
c-ares on win32
|
||||
===============
|
||||
(description brought by Dominick Meglio)
|
||||
|
||||
First I compiled c-ares. I changed the default C runtime library to be the
|
||||
single-threaded rather than the multi-threaded (this seems to be required to
|
||||
prevent linking errors later on). Then I simply build the areslib project (the
|
||||
other projects adig/ahost seem to fail under MSVC).
|
||||
|
||||
Next was libcurl. I opened lib/config-win32.h and I added a:
|
||||
#define USE_ARES 1
|
||||
|
||||
Next thing I did was I added the path for the ares includes to the include
|
||||
path, and the libares.lib to the libraries.
|
||||
|
||||
Lastly, I also changed libcurl to be single-threaded rather than
|
||||
multi-threaded, again this was to prevent some duplicate symbol errors. I'm
|
||||
not sure why I needed to change everything to single-threaded, but when I
|
||||
didn't I got redefinition errors for several CRT functions (malloc, stricmp,
|
||||
etc.)
|
||||
|
||||
I would have modified the MSVC++ project files, but I only have VC.NET and it
|
||||
uses a different format than VC6.0 so I didn't want to go and change
|
||||
everything and remove VC6.0 support from libcurl.
|
||||
|
|
@ -1,68 +0,0 @@
|
|||
|
||||
curl_off_t explained
|
||||
====================
|
||||
|
||||
curl_off_t is a data type provided by the external libcurl include headers. It
|
||||
is the type meant to be used for the curl_easy_setopt() options that end with
|
||||
LARGE. The type is 64bit large on most modern platforms.
|
||||
|
||||
Transition from < 7.19.0 to >= 7.19.0
|
||||
-------------------------------------
|
||||
|
||||
Applications that used libcurl before 7.19.0 that are rebuilt with a libcurl
|
||||
that is 7.19.0 or later may or may not have to worry about anything of
|
||||
this. We have made a significant effort to make the transition really seamless
|
||||
and transparent.
|
||||
|
||||
You have have to take notice if you are in one of the following situations:
|
||||
|
||||
o Your app is using or will after the transition use a libcurl that is built
|
||||
with LFS (large file support) disabled even though your system otherwise
|
||||
supports it.
|
||||
|
||||
o Your app is using or will after the transition use a libcurl that doesn't
|
||||
support LFS at all, but your system and compiler support 64bit data types.
|
||||
|
||||
In both these cases, the curl_off_t type will now (after the transition) be
|
||||
64bit where it previously was 32bit. This will cause a binary incompatibility
|
||||
that you MAY need to deal with.
|
||||
|
||||
Benefits
|
||||
--------
|
||||
|
||||
This new way has several benefits:
|
||||
|
||||
o Platforms without LFS support can still use libcurl to do >32 bit file
|
||||
transfers and range operations etc as long as they have >32 bit data-types
|
||||
supported.
|
||||
|
||||
o Applications will no longer easily build with the curl_off_t size
|
||||
mismatched, which has been a very frequent (and annoying) problem with
|
||||
libcurl <= 7.18.2
|
||||
|
||||
Historically
|
||||
------------
|
||||
|
||||
Previously, before 7.19.0, the curl_off_t type would be rather strongly
|
||||
connected to the size of the system off_t type, where currently curl_off_t is
|
||||
independent of that.
|
||||
|
||||
The strong connection to off_t made it troublesome for application authors
|
||||
since when they did mistakes, they could get curl_off_t type of different
|
||||
sizes in the app vs libcurl, and that caused strange effects that were hard to
|
||||
track and detect by users of libcurl.
|
||||
|
||||
SONAME
|
||||
------
|
||||
|
||||
We opted to not bump the soname for the library unconditionally, simply
|
||||
because soname bumping is causing a lot of grief and moaning all over the
|
||||
community so we try to keep that at minimum. Also, our selected design path
|
||||
should be 100% backwards compatible for the vast majority of all libcurl
|
||||
users.
|
||||
|
||||
Enforce SONAME bump
|
||||
-------------------
|
||||
|
||||
If configure doesn't detect your case where a bump is necessary, re-run it
|
||||
with the --enable-soname-bump command line option!
|
||||
|
|
@ -1,61 +0,0 @@
|
|||
_ _ ____ _
|
||||
___| | | | _ \| |
|
||||
/ __| | | | |_) | |
|
||||
| (__| |_| | _ <| |___
|
||||
\___|\___/|_| \_\_____|
|
||||
|
||||
Source Code Functions Apps Might Use
|
||||
====================================
|
||||
|
||||
The libcurl source code offers a few functions by source only. They are not
|
||||
part of the official libcurl API, but the source files might be useful for
|
||||
others so apps can optionally compile/build with these sources to gain
|
||||
additional functions.
|
||||
|
||||
We provide them through a single header file for easy access for apps:
|
||||
"curlx.h"
|
||||
|
||||
curlx_strtoofft()
|
||||
|
||||
A macro that converts a string containing a number to a curl_off_t number.
|
||||
This might use the curlx_strtoll() function which is provided as source
|
||||
code in strtoofft.c. Note that the function is only provided if no
|
||||
strtoll() (or equivalent) function exist on your platform. If curl_off_t
|
||||
is only a 32 bit number on your platform, this macro uses strtol().
|
||||
|
||||
curlx_tvnow()
|
||||
|
||||
returns a struct timeval for the current time.
|
||||
|
||||
curlx_tvdiff()
|
||||
|
||||
returns the difference between two timeval structs, in number of
|
||||
milliseconds.
|
||||
|
||||
curlx_tvdiff_secs()
|
||||
|
||||
returns the same as curlx_tvdiff but with full usec resolution (as a
|
||||
double)
|
||||
|
||||
FUTURE
|
||||
======
|
||||
|
||||
Several functions will be removed from the public curl_ name space in a
|
||||
future libcurl release. They will then only become available as curlx_
|
||||
functions instead. To make the transition easier, we already today provide
|
||||
these functions with the curlx_ prefix to allow sources to get built properly
|
||||
with the new function names. The functions this concerns are:
|
||||
|
||||
curlx_getenv
|
||||
curlx_strequal
|
||||
curlx_strnequal
|
||||
curlx_mvsnprintf
|
||||
curlx_msnprintf
|
||||
curlx_maprintf
|
||||
curlx_mvaprintf
|
||||
curlx_msprintf
|
||||
curlx_mprintf
|
||||
curlx_mfprintf
|
||||
curlx_mvsprintf
|
||||
curlx_mvprintf
|
||||
curlx_mvfprintf
|
||||
|
|
@ -1,60 +0,0 @@
|
|||
|
||||
Content Encoding Support for libcurl
|
||||
|
||||
* About content encodings:
|
||||
|
||||
HTTP/1.1 [RFC 2616] specifies that a client may request that a server encode
|
||||
its response. This is usually used to compress a response using one of a set
|
||||
of commonly available compression techniques. These schemes are `deflate' (the
|
||||
zlib algorithm), `gzip' and `compress' [sec 3.5, RFC 2616]. A client requests
|
||||
that the sever perform an encoding by including an Accept-Encoding header in
|
||||
the request document. The value of the header should be one of the recognized
|
||||
tokens `deflate', ... (there's a way to register new schemes/tokens, see sec
|
||||
3.5 of the spec). A server MAY honor the client's encoding request. When a
|
||||
response is encoded, the server includes a Content-Encoding header in the
|
||||
response. The value of the Content-Encoding header indicates which scheme was
|
||||
used to encode the data.
|
||||
|
||||
A client may tell a server that it can understand several different encoding
|
||||
schemes. In this case the server may choose any one of those and use it to
|
||||
encode the response (indicating which one using the Content-Encoding header).
|
||||
It's also possible for a client to attach priorities to different schemes so
|
||||
that the server knows which it prefers. See sec 14.3 of RFC 2616 for more
|
||||
information on the Accept-Encoding header.
|
||||
|
||||
* Current support for content encoding:
|
||||
|
||||
Support for the 'deflate' and 'gzip' content encoding are supported by
|
||||
libcurl. Both regular and chunked transfers should work fine. The library
|
||||
zlib is required for this feature. 'deflate' support was added by James
|
||||
Gallagher, and support for the 'gzip' encoding was added by Dan Fandrich.
|
||||
|
||||
* The libcurl interface:
|
||||
|
||||
To cause libcurl to request a content encoding use:
|
||||
|
||||
curl_easy_setopt(curl, CURLOPT_ACCEPT_ENCODING, <string>)
|
||||
|
||||
where <string> is the intended value of the Accept-Encoding header.
|
||||
|
||||
Currently, libcurl only understands how to process responses that use the
|
||||
"deflate" or "gzip" Content-Encoding, so the only values for
|
||||
CURLOPT_ACCEPT_ENCODING that will work (besides "identity," which does
|
||||
nothing) are "deflate" and "gzip" If a response is encoded using the
|
||||
"compress" or methods, libcurl will return an error indicating that the
|
||||
response could not be decoded. If <string> is NULL no Accept-Encoding header
|
||||
is generated. If <string> is a zero-length string, then an Accept-Encoding
|
||||
header containing all supported encodings will be generated.
|
||||
|
||||
The CURLOPT_ACCEPT_ENCODING must be set to any non-NULL value for content to
|
||||
be automatically decoded. If it is not set and the server still sends encoded
|
||||
content (despite not having been asked), the data is returned in its raw form
|
||||
and the Content-Encoding type is not checked.
|
||||
|
||||
* The curl interface:
|
||||
|
||||
Use the --compressed option with curl to cause it to ask servers to compress
|
||||
responses using any format supported by curl.
|
||||
|
||||
James Gallagher <jgallagher@gso.uri.edu>
|
||||
Dan Fandrich <dan@coneharvesters.com>
|
||||
|
|
@ -1,35 +0,0 @@
|
|||
hostip.c explained
|
||||
==================
|
||||
|
||||
The main COMPILE-TIME DEFINES to keep in mind when reading the host*.c
|
||||
source file are these:
|
||||
|
||||
CURLRES_IPV6 - this host has getaddrinfo() and family, and thus we use
|
||||
that. The host may not be able to resolve IPv6, but we don't really have to
|
||||
take that into account. Hosts that aren't IPv6-enabled have CURLRES_IPV4
|
||||
defined.
|
||||
|
||||
CURLRES_ARES - is defined if libcurl is built to use c-ares for asynchronous
|
||||
name resolves. This can be Windows or *nix.
|
||||
|
||||
CURLRES_THREADED - is defined if libcurl is built to use threading for
|
||||
asynchronous name resolves. The name resolve will be done in a new thread,
|
||||
and the supported asynch API will be the same as for ares-builds. This is
|
||||
the default under (native) Windows.
|
||||
|
||||
If any of the two previous are defined, CURLRES_ASYNCH is defined too. If
|
||||
libcurl is not built to use an asynchronous resolver, CURLRES_SYNCH is
|
||||
defined.
|
||||
|
||||
The host*.c sources files are split up like this:
|
||||
|
||||
hostip.c - method-independent resolver functions and utility functions
|
||||
hostasyn.c - functions for asynchronous name resolves
|
||||
hostsyn.c - functions for synchronous name resolves
|
||||
asyn-ares.c - functions for asynchronous name resolves using c-ares
|
||||
asyn-thread.c - functions for asynchronous name resolves using threads
|
||||
hostip4.c - IPv4 specific functions
|
||||
hostip6.c - IPv6 specific functions
|
||||
|
||||
The hostip.h is the single united header file for all this. It defines the
|
||||
CURLRES_* defines based on the config*.h and curl_setup.h defines.
|
||||
|
|
@ -1,74 +0,0 @@
|
|||
|
||||
1. PUT/POST without a known auth to use (possibly no auth required):
|
||||
|
||||
(When explicitly set to use a multi-pass auth when doing a POST/PUT,
|
||||
libcurl should immediately go the Content-Length: 0 bytes route to avoid
|
||||
the first send all data phase, step 2. If told to use a single-pass auth,
|
||||
goto step 3.)
|
||||
|
||||
Issue the proper PUT/POST request immediately, with the correct
|
||||
Content-Length and Expect: headers.
|
||||
|
||||
If a 100 response is received or the wait for one times out, start sending
|
||||
the request-body.
|
||||
|
||||
If a 401 (or 407 when talking through a proxy) is received, then:
|
||||
|
||||
If we have "more than just a little" data left to send, close the
|
||||
connection. Exactly what "more than just a little" means will have to be
|
||||
determined. Possibly the current transfer speed should be taken into
|
||||
account as well.
|
||||
|
||||
NOTE: if the size of the POST data is less than MAX_INITIAL_POST_SIZE (when
|
||||
CURLOPT_POSTFIELDS is used), libcurl will send everything in one single
|
||||
write() (all request-headers and request-body) and thus it will
|
||||
unconditionally send the full post data here.
|
||||
|
||||
2. PUT/POST with multi-pass auth but not yet completely negotiated:
|
||||
|
||||
Send a PUT/POST request, we know that it will be rejected and thus we claim
|
||||
Content-Length zero to avoid having to send the request-body. (This seems
|
||||
to be what IE does.)
|
||||
|
||||
3. PUT/POST as the last step in the auth negotiation, that is when we have
|
||||
what we believe is a completed negotiation:
|
||||
|
||||
Send a full and proper PUT/POST request (again) with the proper
|
||||
Content-Length and a following request-body.
|
||||
|
||||
NOTE: this may very well be the second (or even third) time the whole or at
|
||||
least parts of the request body is sent to the server. Since the data may
|
||||
be provided to libcurl with a callback, we need a way to tell the app that
|
||||
the upload is to be restarted so that the callback will provide data from
|
||||
the start again. This requires an API method/mechanism that libcurl
|
||||
doesn't have today. See below.
|
||||
|
||||
Data Rewind
|
||||
|
||||
It will be troublesome for some apps to deal with a rewind like this in all
|
||||
circumstances. I'm thinking for example when using 'curl' to upload data
|
||||
from stdin. If libcurl ends up having to rewind the reading for a request
|
||||
to succeed, of course a lack of this callback or if it returns failure, will
|
||||
cause the request to fail completely.
|
||||
|
||||
The new callback is set with CURLOPT_IOCTLFUNCTION (in an attempt to add a
|
||||
more generic function that might be used for other IO-related controls in
|
||||
the future):
|
||||
|
||||
curlioerr curl_ioctl(CURL *handle, curliocmd cmd, void *clientp);
|
||||
|
||||
And in the case where the read is to be rewinded, it would be called with a
|
||||
cmd named CURLIOCMD_RESTARTREAD. The callback would then return CURLIOE_OK,
|
||||
if things are fine, or CURLIOE_FAILRESTART if not.
|
||||
|
||||
Backwards Compatibility
|
||||
|
||||
The approach used until now, that issues a HEAD on the given URL to trigger
|
||||
the auth negotiation could still be supported and encouraged, but it would
|
||||
be up to the app to first fetch a URL with GET/HEAD to negotiate on, since
|
||||
then a following PUT/POST wouldn't need to negotiate authentication and
|
||||
thus avoid double-sending data.
|
||||
|
||||
Optionally, we keep the current approach if some option is set
|
||||
(CURLOPT_HEADBEFOREAUTH or similar), since it seems to work fairly well for
|
||||
POST on most servers.
|
||||
|
|
@ -1,55 +0,0 @@
|
|||
_ _ ____ _
|
||||
___| | | | _ \| |
|
||||
/ __| | | | |_) | |
|
||||
| (__| |_| | _ <| |___
|
||||
\___|\___/|_| \_\_____|
|
||||
|
||||
How To Track Down Suspected Memory Leaks in libcurl
|
||||
===================================================
|
||||
|
||||
Single-threaded
|
||||
|
||||
Please note that this memory leak system is not adjusted to work in more
|
||||
than one thread. If you want/need to use it in a multi-threaded app. Please
|
||||
adjust accordingly.
|
||||
|
||||
|
||||
Build
|
||||
|
||||
Rebuild libcurl with -DCURLDEBUG (usually, rerunning configure with
|
||||
--enable-debug fixes this). 'make clean' first, then 'make' so that all
|
||||
files actually are rebuilt properly. It will also make sense to build
|
||||
libcurl with the debug option (usually -g to the compiler) so that debugging
|
||||
it will be easier if you actually do find a leak in the library.
|
||||
|
||||
This will create a library that has memory debugging enabled.
|
||||
|
||||
Modify Your Application
|
||||
|
||||
Add a line in your application code:
|
||||
|
||||
curl_memdebug("dump");
|
||||
|
||||
This will make the malloc debug system output a full trace of all resource
|
||||
using functions to the given file name. Make sure you rebuild your program
|
||||
and that you link with the same libcurl you built for this purpose as
|
||||
described above.
|
||||
|
||||
Run Your Application
|
||||
|
||||
Run your program as usual. Watch the specified memory trace file grow.
|
||||
|
||||
Make your program exit and use the proper libcurl cleanup functions etc. So
|
||||
that all non-leaks are returned/freed properly.
|
||||
|
||||
Analyze the Flow
|
||||
|
||||
Use the tests/memanalyze.pl perl script to analyze the dump file:
|
||||
|
||||
tests/memanalyze.pl dump
|
||||
|
||||
This now outputs a report on what resources that were allocated but never
|
||||
freed etc. This report is very fine for posting to the list!
|
||||
|
||||
If this doesn't produce any output, no leak was detected in libcurl. Then
|
||||
the leak is mostly likely to be in your code.
|
||||
|
|
@ -1,53 +0,0 @@
|
|||
Implementation of the curl_multi_socket API
|
||||
|
||||
The main ideas of the new API are simply:
|
||||
|
||||
1 - The application can use whatever event system it likes as it gets info
|
||||
from libcurl about what file descriptors libcurl waits for what action
|
||||
on. (The previous API returns fd_sets which is very select()-centric).
|
||||
|
||||
2 - When the application discovers action on a single socket, it calls
|
||||
libcurl and informs that there was action on this particular socket and
|
||||
libcurl can then act on that socket/transfer only and not care about
|
||||
any other transfers. (The previous API always had to scan through all
|
||||
the existing transfers.)
|
||||
|
||||
The idea is that curl_multi_socket_action() calls a given callback with
|
||||
information about what socket to wait for what action on, and the callback
|
||||
only gets called if the status of that socket has changed.
|
||||
|
||||
We also added a timer callback that makes libcurl call the application when
|
||||
the timeout value changes, and you set that with curl_multi_setopt() and the
|
||||
CURLMOPT_TIMERFUNCTION option. To get this to work, Internally, there's an
|
||||
added a struct to each easy handle in which we store an "expire time" (if
|
||||
any). The structs are then "splay sorted" so that we can add and remove
|
||||
times from the linked list and yet somewhat swiftly figure out both how long
|
||||
time there is until the next nearest timer expires and which timer (handle)
|
||||
we should take care of now. Of course, the upside of all this is that we get
|
||||
a curl_multi_timeout() that should also work with old-style applications
|
||||
that use curl_multi_perform().
|
||||
|
||||
We created an internal "socket to easy handles" hash table that given
|
||||
a socket (file descriptor) return the easy handle that waits for action on
|
||||
that socket. This hash is made using the already existing hash code
|
||||
(previously only used for the DNS cache).
|
||||
|
||||
To make libcurl able to report plain sockets in the socket callback, we had
|
||||
to re-organize the internals of the curl_multi_fdset() etc so that the
|
||||
conversion from sockets to fd_sets for that function is only done in the
|
||||
last step before the data is returned. I also had to extend c-ares to get a
|
||||
function that can return plain sockets, as that library too returned only
|
||||
fd_sets and that is no longer good enough. The changes done to c-ares are
|
||||
available in c-ares 1.3.1 and later.
|
||||
|
||||
We have done a test runs with up to 9000 connections (with a single active
|
||||
one). The curl_multi_socket_action() invoke then takes less than 10
|
||||
microseconds in average (using the read-only-1-byte-at-a-time hack). We are
|
||||
now below the 60 microseconds "per socket action" goal (the extra 50 is the
|
||||
time libevent needs).
|
||||
|
||||
Documentation
|
||||
|
||||
http://curl.haxx.se/libcurl/c/curl_multi_socket_action.html
|
||||
http://curl.haxx.se/libcurl/c/curl_multi_timeout.html
|
||||
http://curl.haxx.se/libcurl/c/curl_multi_setopt.html
|
||||
Loading…
Add table
Add a link
Reference in a new issue