mime: do not perform more than one read in a row

Input buffer filling may delay the data sending if data reads are slow.
To overcome this problem, file and callback data reads do not accumulate
in buffer anymore. All other data (memory data and mime framing) are
considered as fast and still concatenated in buffer.
As this may highly impact performance in terms of data overhead, an early
end of part data check is added to spare a read call.
When encoding a part's data, an encoder may require more bytes than made
available by a single read. In this case, the above rule does not apply
and reads are performed until the encoder is able to deliver some data.

Tests 643, 644, 645, 650 and 654 have been adapted to the output data
changes, with test data size reduced to avoid the boredom of long lists of
1-byte chunks in verification data.
New test 667 checks mimepost using single-byte read callback with encoder.
New test 668 checks the end of part data early detection.

Fixes #4826
Reported-by: MrdUkk on github
This commit is contained in:
Patrick Monnerat 2020-03-07 11:51:00 +01:00 committed by Daniel Stenberg
parent 96972ec1c0
commit 1e4cb333ef
No known key found for this signature in database
GPG key ID: 5CC908FDB71E12C2
15 changed files with 679 additions and 77 deletions

View file

@ -85,7 +85,7 @@ test626 test627 test628 test629 test630 test631 test632 test633 test634 \
test635 test636 test637 test638 test639 test640 test641 test642 \
test643 test644 test645 test646 test647 test648 test649 test650 test651 \
test652 test653 test654 test655 test656 test658 test659 test660 test661 \
test662 test663 test664 test665 \
test662 test663 test664 test665 test667 test668 \
test670 test671 test672 test673 \
\
test700 test701 test702 test703 test704 test705 test706 test707 test708 \

View file

@ -69,18 +69,18 @@ s/boundary=------------------------[a-z0-9]*/boundary=--------------------------
POST /643 HTTP/1.1
Host: %HOSTIP:%HTTPPORT
Accept: */*
Content-Length: 718
Content-Length: 640
Content-Type: multipart/form-data; boundary=----------------------------
------------------------------
Content-Disposition: form-data; name="sendfile"; filename="postit2.c"
this is what we post to the silly web server
dummy
------------------------------
Content-Disposition: form-data; name="callbackdata"
this is what we post to the silly web server
dummy
------------------------------
Content-Disposition: form-data; name="filename"
@ -99,18 +99,18 @@ blah blah
POST /643 HTTP/1.1
Host: %HOSTIP:%HTTPPORT
Accept: */*
Content-Length: 732
Content-Length: 654
Content-Type: multipart/form-data; boundary=----------------------------
------------------------------
Content-Disposition: form-data; name="sendfile alternative"; filename="file name 2"
this is what we post to the silly web server
dummy
------------------------------
Content-Disposition: form-data; name="callbackdata"
this is what we post to the silly web server
dummy
------------------------------
Content-Disposition: form-data; name="filename"

View file

@ -44,7 +44,7 @@ s/boundary=------------------------[a-z0-9]*/boundary=--------------------------
POST /644 HTTP/1.1
Host: %HOSTIP:%HTTPPORT
Accept: */*
Content-Length: 718
Content-Length: 640
Content-Type: multipart/form-data; boundary=----------------------------
------------------------------

View file

@ -73,16 +73,42 @@ Transfer-Encoding: chunked
Content-Type: multipart/form-data; boundary=----------------------------
Expect: 100-continue
2ce
76
------------------------------
Content-Disposition: form-data; name="sendfile"; filename="postit2.c"
this is what we post to the silly web server
d
1
u
1
m
1
m
1
y
1
65
------------------------------
Content-Disposition: form-data; name="callbackdata"
this is what we post to the silly web server
1
d
1
u
1
m
1
m
1
y
1
19a
------------------------------
Content-Disposition: form-data; name="filename"
@ -108,16 +134,42 @@ Transfer-Encoding: chunked
Content-Type: multipart/form-data; boundary=----------------------------
Expect: 100-continue
2dc
84
------------------------------
Content-Disposition: form-data; name="sendfile alternative"; filename="file name 2"
this is what we post to the silly web server
d
1
u
1
m
1
m
1
y
1
65
------------------------------
Content-Disposition: form-data; name="callbackdata"
this is what we post to the silly web server
1
d
1
u
1
m
1
m
1
y
1
19a
------------------------------
Content-Disposition: form-data; name="filename"

View file

@ -63,7 +63,7 @@ Transfer-Encoding: chunked
Content-Type: multipart/form-data; boundary=----------------------------
Expect: 100-continue
60a
361
------------------------------
Content-Disposition: form-data; name="fieldname"
Content-Type: text/plain
@ -89,12 +89,16 @@ This is data from a file.
Content-Disposition: attachment; filename="test650.filedata"
Content-Type: text/whatever
a5
This is data from a file.
------------------------------
Content-Disposition: attachment; filename="test650.filedata"
Content-Type: text/whatever
af
This is data from a file.
--------------------------------
@ -102,6 +106,8 @@ This is data from a file.
------------------------------
Content-Disposition: form-data; name="filecontents"
10f
This is data from a file.
------------------------------
@ -112,8 +118,12 @@ Content-Disposition: form-data; name="formlength"
Content-Disposition: form-data; name="standardinput"
Content-Type: application/octet-stream
16
Some data from stdin
30
--------------------------------
0

View file

@ -81,7 +81,7 @@ Transfer-Encoding: chunked
Content-Type: multipart/form-data; boundary=----------------------------
Expect: 100-continue
20c
1af
------------------------------
Content-Disposition: form-data; name="greeting"
Content-Type: application/X-Greeting
@ -98,7 +98,21 @@ This is data from a file
------------------------------
Content-Disposition: form-data
this is what we post to the silly web server
1
d
1
u
1
m
1
m
1
y
1
30
--------------------------------

85
tests/data/test667 Normal file
View file

@ -0,0 +1,85 @@
<testcase>
<info>
<keywords>
HTTP
HTTP POST
HTTP MIME POST
</keywords>
</info>
#
# Server-side
<reply>
<data>
HTTP/1.1 200 OK
Date: Thu, 09 Nov 2010 14:49:00 GMT
Server: test-server/fake swsclose
Connection: close
Content-Type: text/html
hello
</data>
<datacheck>
HTTP/1.1 200 OK
Date: Thu, 09 Nov 2010 14:49:00 GMT
Server: test-server/fake swsclose
Connection: close
Content-Type: text/html
hello
</datacheck>
</reply>
# Client-side
<client>
<server>
http
</server>
# tool is what to use instead of 'curl'
<tool>
lib667
</tool>
<name>
HTTP chunked mimepost using single-byte read callback with encoder
</name>
<command>
http://%HOSTIP:%HTTPPORT/667
</command>
</client>
#
# Verify data after the test has been "shot"
<verify>
<strippart>
s/^--------------------------[a-z0-9]*/------------------------------/
s/boundary=------------------------[a-z0-9]*/boundary=----------------------------/
</strippart>
# Note that the stripping above removes 12 bytes from every occurrence of the
# boundary string and since 5 of them are in the body contents, we see
# (5*12) == 60 bytes less
<protocol>
POST /667 HTTP/1.1
Host: %HOSTIP:%HTTPPORT
Accept: */*
Transfer-Encoding: chunked
Content-Type: multipart/form-data; boundary=----------------------------
Expect: 100-continue
7f
------------------------------
Content-Disposition: form-data; name="field"
Content-Transfer-Encoding: base64
4
ZHVt
34
bXk=
--------------------------------
0
</protocol>
</verify>
</testcase>

99
tests/data/test668 Normal file
View file

@ -0,0 +1,99 @@
<testcase>
<info>
<keywords>
HTTP
HTTP POST
HTTP MIME POST
</keywords>
</info>
#
# Server-side
<reply>
<data>
HTTP/1.1 200 OK
Date: Thu, 09 Nov 2010 14:49:00 GMT
Server: test-server/fake swsclose
Connection: close
Content-Type: text/html
hello
</data>
<datacheck>
HTTP/1.1 200 OK
Date: Thu, 09 Nov 2010 14:49:00 GMT
Server: test-server/fake swsclose
Connection: close
Content-Type: text/html
hello
</datacheck>
</reply>
# Client-side
<client>
<server>
http
</server>
# tool is what to use instead of 'curl'
<tool>
lib668
</tool>
<name>
HTTP mimepost early end of data detection
</name>
<command>
http://%HOSTIP:%HTTPPORT/668
</command>
<file name="log/file668.txt">
This is data from a file
</file>
</client>
#
# Verify data after the test has been "shot"
<verify>
<strippart>
s/^--------------------------[a-z0-9]*/------------------------------/
s/boundary=------------------------[a-z0-9]*/boundary=----------------------------/
</strippart>
# Note that the stripping above removes 12 bytes from every occurrence of the
# boundary string and since 5 of them are in the body contents, we see
# (5*12) == 60 bytes less
<protocol>
POST /668 HTTP/1.1
Host: %HOSTIP:%HTTPPORT
Accept: */*
Transfer-Encoding: chunked
Content-Type: multipart/form-data; boundary=----------------------------
Expect: 100-continue
c1
------------------------------
Content-Disposition: form-data; name="field1"
dummy
------------------------------
Content-Disposition: form-data; name="field2"
5
dummy
91
------------------------------
Content-Disposition: form-data; name="field3"; filename="file668.txt"
Content-Type: text/plain
49
This is data from a file
--------------------------------
0
</protocol>
</verify>
</testcase>

View file

@ -22,7 +22,7 @@ noinst_PROGRAMS = chkhostname libauthretry libntlmconnect \
lib571 lib572 lib573 lib574 lib575 lib576 lib578 lib579 lib582 \
lib583 lib585 lib586 lib587 lib589 lib590 lib591 lib597 lib598 lib599 \
lib643 lib644 lib645 lib650 lib651 lib652 lib653 lib654 lib655 lib658 \
lib659 lib661 \
lib659 lib661 lib667 lib668 \
lib670 lib671 lib672 lib673 \
lib1156 \
lib1500 lib1501 lib1502 lib1503 lib1504 lib1505 lib1506 lib1507 lib1508 \
@ -349,6 +349,14 @@ lib659_CPPFLAGS = $(AM_CPPFLAGS)
lib661_SOURCES = lib661.c $(SUPPORTFILES)
lib661_CPPFLAGS = $(AM_CPPFLAGS)
lib667_SOURCES = lib667.c $(SUPPORTFILES) $(TESTUTIL) $(WARNLESS)
lib667_LDADD = $(TESTUTIL_LIBS)
lib667_CPPFLAGS = $(AM_CPPFLAGS)
lib668_SOURCES = lib668.c $(SUPPORTFILES) $(TESTUTIL) $(WARNLESS)
lib668_LDADD = $(TESTUTIL_LIBS)
lib668_CPPFLAGS = $(AM_CPPFLAGS)
lib670_SOURCES = lib670.c $(SUPPORTFILES) $(TESTUTIL) $(WARNLESS)
lib670_LDADD = $(TESTUTIL_LIBS)
lib670_CPPFLAGS = $(AM_CPPFLAGS) -DLIB670

View file

@ -26,11 +26,9 @@
static char data[]=
#ifdef CURL_DOES_CONVERSIONS
/* ASCII representation with escape sequences for non-ASCII platforms */
"\x74\x68\x69\x73\x20\x69\x73\x20\x77\x68\x61\x74\x20\x77\x65\x20\x70"
"\x6f\x73\x74\x20\x74\x6f\x20\x74\x68\x65\x20\x73\x69\x6c\x6c\x79\x20"
"\x77\x65\x62\x20\x73\x65\x72\x76\x65\x72\x0a";
"\x64\x75\x6d\x6d\x79\x0a";
#else
"this is what we post to the silly web server\n";
"dummy\n";
#endif
struct WriteThis {

View file

@ -5,7 +5,7 @@
* | (__| |_| | _ <| |___
* \___|\___/|_| \_\_____|
*
* Copyright (C) 1998 - 2018, Daniel Stenberg, <daniel@haxx.se>, et al.
* Copyright (C) 1998 - 2020, Daniel Stenberg, <daniel@haxx.se>, et al.
*
* This software is licensed as described in the file COPYING, which
* you should have received as part of this distribution. The terms
@ -26,11 +26,9 @@
static char data[]=
#ifdef CURL_DOES_CONVERSIONS
/* ASCII representation with escape sequences for non-ASCII platforms */
"\x74\x68\x69\x73\x20\x69\x73\x20\x77\x68\x61\x74\x20\x77\x65\x20\x70"
"\x6f\x73\x74\x20\x74\x6f\x20\x74\x68\x65\x20\x73\x69\x6c\x6c\x79\x20"
"\x77\x65\x62\x20\x73\x65\x72\x76\x65\x72\x0a";
"\x64\x75\x6d\x6d\x79\x0a";
#else
"this is what we post to the silly web server\n";
"dummy\n";
#endif
struct WriteThis {

117
tests/libtest/lib667.c Normal file
View file

@ -0,0 +1,117 @@
/***************************************************************************
* _ _ ____ _
* Project ___| | | | _ \| |
* / __| | | | |_) | |
* | (__| |_| | _ <| |___
* \___|\___/|_| \_\_____|
*
* Copyright (C) 1998 - 2020, Daniel Stenberg, <daniel@haxx.se>, et al.
*
* This software is licensed as described in the file COPYING, which
* you should have received as part of this distribution. The terms
* are also available at https://curl.haxx.se/docs/copyright.html.
*
* You may opt to use, copy, modify, merge, publish, distribute and/or sell
* copies of the Software, and permit persons to whom the Software is
* furnished to do so, under the terms of the COPYING file.
*
* This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
* KIND, either express or implied.
*
***************************************************************************/
#include "test.h"
#include "memdebug.h"
static char data[]=
#ifdef CURL_DOES_CONVERSIONS
/* ASCII representation with escape sequences for non-ASCII platforms */
"\x64\x75\x6d\x6d\x79";
#else
"dummy";
#endif
struct WriteThis {
char *readptr;
curl_off_t sizeleft;
};
static size_t read_callback(char *ptr, size_t size, size_t nmemb, void *userp)
{
struct WriteThis *pooh = (struct WriteThis *)userp;
int eof = !*pooh->readptr;
if(size*nmemb < 1)
return 0;
eof = pooh->sizeleft <= 0;
if(!eof)
pooh->sizeleft--;
if(!eof) {
*ptr = *pooh->readptr; /* copy one single byte */
pooh->readptr++; /* advance pointer */
return 1; /* we return 1 byte at a time! */
}
return 0; /* no more data left to deliver */
}
int test(char *URL)
{
CURL *easy = NULL;
curl_mime *mime = NULL;
curl_mimepart *part;
CURLcode result;
int res = TEST_ERR_FAILURE;
struct WriteThis pooh;
/*
* Check proper handling of mime encoder feature when the part read callback
* delivers data bytes one at a time. Use chunked encoding for accurate test.
*/
if(curl_global_init(CURL_GLOBAL_ALL) != CURLE_OK) {
fprintf(stderr, "curl_global_init() failed\n");
return TEST_ERR_MAJOR_BAD;
}
easy = curl_easy_init();
/* First set the URL that is about to receive our POST. */
test_setopt(easy, CURLOPT_URL, URL);
/* get verbose debug output please */
test_setopt(easy, CURLOPT_VERBOSE, 1L);
/* include headers in the output */
test_setopt(easy, CURLOPT_HEADER, 1L);
/* Prepare the callback structure. */
pooh.readptr = data;
pooh.sizeleft = (curl_off_t) strlen(data);
/* Build the mime tree. */
mime = curl_mime_init(easy);
part = curl_mime_addpart(mime);
curl_mime_name(part, "field");
curl_mime_encoder(part, "base64");
/* Using an undefined length forces chunked transfer. */
curl_mime_data_cb(part, (curl_off_t) -1, read_callback, NULL, NULL, &pooh);
/* Bind mime data to its easy handle. */
test_setopt(easy, CURLOPT_MIMEPOST, mime);
/* Send data. */
result = curl_easy_perform(easy);
if(result) {
fprintf(stderr, "curl_easy_perform() failed\n");
res = (int) result;
}
test_cleanup:
curl_easy_cleanup(easy);
curl_mime_free(mime);
curl_global_cleanup();
return res;
}

122
tests/libtest/lib668.c Normal file
View file

@ -0,0 +1,122 @@
/***************************************************************************
* _ _ ____ _
* Project ___| | | | _ \| |
* / __| | | | |_) | |
* | (__| |_| | _ <| |___
* \___|\___/|_| \_\_____|
*
* Copyright (C) 1998 - 2020, Daniel Stenberg, <daniel@haxx.se>, et al.
*
* This software is licensed as described in the file COPYING, which
* you should have received as part of this distribution. The terms
* are also available at https://curl.haxx.se/docs/copyright.html.
*
* You may opt to use, copy, modify, merge, publish, distribute and/or sell
* copies of the Software, and permit persons to whom the Software is
* furnished to do so, under the terms of the COPYING file.
*
* This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
* KIND, either express or implied.
*
***************************************************************************/
#include "test.h"
#include "memdebug.h"
static char data[]=
#ifdef CURL_DOES_CONVERSIONS
/* ASCII representation with escape sequences for non-ASCII platforms */
"\x64\x75\x6d\x6d\x79";
#else
"dummy";
#endif
struct WriteThis {
char *readptr;
curl_off_t sizeleft;
};
static size_t read_callback(char *ptr, size_t size, size_t nmemb, void *userp)
{
struct WriteThis *pooh = (struct WriteThis *)userp;
size_t len = strlen(pooh->readptr);
(void) size; /* Always 1.*/
if(len > nmemb)
len = nmemb;
if(len) {
memcpy(ptr, pooh->readptr, len);
pooh->readptr += len;
}
return len;
}
int test(char *URL)
{
CURL *easy = NULL;
curl_mime *mime = NULL;
curl_mimepart *part;
CURLcode result;
int res = TEST_ERR_FAILURE;
struct WriteThis pooh1, pooh2;
/*
* Check early end of part data detection.
*/
if(curl_global_init(CURL_GLOBAL_ALL) != CURLE_OK) {
fprintf(stderr, "curl_global_init() failed\n");
return TEST_ERR_MAJOR_BAD;
}
easy = curl_easy_init();
/* First set the URL that is about to receive our POST. */
test_setopt(easy, CURLOPT_URL, URL);
/* get verbose debug output please */
test_setopt(easy, CURLOPT_VERBOSE, 1L);
/* include headers in the output */
test_setopt(easy, CURLOPT_HEADER, 1L);
/* Prepare the callback structures. */
pooh1.readptr = data;
pooh1.sizeleft = (curl_off_t) strlen(data);
pooh2 = pooh1;
/* Build the mime tree. */
mime = curl_mime_init(easy);
part = curl_mime_addpart(mime);
curl_mime_name(part, "field1");
/* Early end of data detection can be done because the data size is known. */
curl_mime_data_cb(part, (curl_off_t) strlen(data),
read_callback, NULL, NULL, &pooh1);
part = curl_mime_addpart(mime);
curl_mime_name(part, "field2");
/* Using an undefined length forces chunked transfer and disables early
end of data detection for this part. */
curl_mime_data_cb(part, (curl_off_t) -1, read_callback, NULL, NULL, &pooh2);
part = curl_mime_addpart(mime);
curl_mime_name(part, "field3");
/* Regular file part sources early end of data can be detected because
the file size is known. In addition, and EOF test is performed. */
curl_mime_filedata(part, "log/file668.txt");
/* Bind mime data to its easy handle. */
test_setopt(easy, CURLOPT_MIMEPOST, mime);
/* Send data. */
result = curl_easy_perform(easy);
if(result) {
fprintf(stderr, "curl_easy_perform() failed\n");
res = (int) result;
}
test_cleanup:
curl_easy_cleanup(easy);
curl_mime_free(mime);
curl_global_cleanup();
return res;
}