runtests: do CRLF replacements per section only

The `crlf="yes"` attribute and "hyper mode" are now only applied on a
subset of dedicated sections: data, datacheck, stdout and protocol.

Updated test 2500 accordingly.

Also made test1 use crlf="yes" for <protocol>, mostly because it is
often used as a template test case. Going forward, using this attribute
we should be able to write test cases using linefeeds only and avoid
mixed line ending encodings.

Follow-up to ca15b7512e

Fixes #10009
Closes #10010

tests/FILEFORMAT.md

@@ -196,7 +196,7 @@ When using curl built with Hyper, the keywords must include HTTP or HTTPS for
 'hyper mode' to kick in and make line ending checks work for tests.
 ## `<reply>`
 
-### `<data [nocheck="yes"] [sendzero="yes"] [base64="yes"] [hex="yes"] [nonewline="yes"]>`
+### `<data [nocheck="yes"] [sendzero="yes"] [base64="yes"] [hex="yes"] [nonewline="yes"] [crlf="yes"]>`
 
 data to be sent to the client on its request and later verified that it
 arrived safely. Set `nocheck="yes"` to prevent the test script from verifying
@@ -225,12 +225,16 @@ and used as "raw" data.
 `nonewline=yes` means that the last byte (the trailing newline character)
 should be cut off from the data before sending or comparing it.
 
+`crlf=yes` forces *header* newlines to become CRLF even if not written so in
+the source file. Note that this makes runtests.pl parse and "guess" what is a
+header and what is not in order to apply the CRLF line endings appropriately.
+
 For FTP file listings, the `<data>` section will be used *only* if you make
 sure that there has been a CWD done first to a directory named `test-[NUM]`
 where `NUM` is the test case number. Otherwise the ftp server can't know from
 which test file to load the list content.
 
-### `<dataNUM>`
+### `<dataNUM [crlf="yes"]>`
 
 Send back this contents instead of the <data> one. The `NUM` is set by:
 
@@ -257,7 +261,7 @@ a connect prefix.
 ### `<socks>`
 Address type and address details as logged by the SOCKS proxy.
 
-### `<datacheck [mode="text"] [nonewline="yes"]>`
+### `<datacheck [mode="text"] [nonewline="yes"] [crlf="yes"]>`
 if the data is sent but this is what should be checked afterwards. If
 `nonewline=yes` is set, runtests will cut off the trailing newline from the
 data before comparing with the one actually received by the client.
@@ -265,7 +269,7 @@ data before comparing with the one actually received by the client.
 Use the `mode="text"` attribute if the output is in text mode on platforms
 that have a text/binary difference.
 
-### `<datacheckNUM [nonewline="yes"] [mode="text"]>`
+### `<datacheckNUM [nonewline="yes"] [mode="text"] [crlf="yes"]>`
 The contents of numbered `datacheck` sections are appended to the non-numbered
 one.
 
@@ -562,13 +566,16 @@ changing protocol data such as port numbers or user-agent strings.
 One perl op per line that operates on the protocol dump. This is pretty
 advanced. Example: `s/^EPRT .*/EPRT stripped/`.
 
-### `<protocol [nonewline="yes"]>`
+### `<protocol [nonewline="yes"] crlf="yes">`
 
 the protocol dump curl should transmit, if `nonewline` is set, we will cut off
 the trailing newline of this given data before comparing with the one actually
 sent by the client The `<strip>` and `<strippart>` rules are applied before
 comparisons are made.
 
+`crlf=yes` forces the newlines to become CRLF even if not written so in the
+test.
+
 ### `<proxy [nonewline="yes"]>`
 
 The protocol dump curl should transmit to an HTTP proxy (when the http-proxy
@@ -585,7 +592,7 @@ have a text/binary difference.
 If `nonewline` is set, we will cut off the trailing newline of this given data
 before comparing with the one actually received by the client
 
-### `<stdout [mode="text"] [nonewline="yes"]>`
+### `<stdout [mode="text"] [nonewline="yes"] [crlf="yes"]>`
 This verifies that this data was passed to stdout.
 
 Use the mode="text" attribute if the output is in text mode on platforms that
@@ -594,6 +601,9 @@ have a text/binary difference.
 If `nonewline` is set, we will cut off the trailing newline of this given data
 before comparing with the one actually received by the client
 
+`crlf=yes` forces the newlines to become CRLF even if not written so in the
+test.
+
 ### `<file name="log/filename" [mode="text"]>`
 The file's contents must be identical to this after the test is complete. Use
 the mode="text" attribute if the output is in text mode on platforms that have

tests/data/test1

@@ -9,7 +9,7 @@ HTTP GET
 #
 # Server-side
 <reply>
-<data>
+<data crlf="yes">
 HTTP/1.1 200 OK
 Date: Tue, 09 Nov 2010 14:49:00 GMT
 Server: test-server/fake
@@ -42,12 +42,12 @@ http://%HOSTIP:%HTTPPORT/%TESTNUMBER
 #
 # Verify data after the test has been "shot"
 <verify>
-<protocol>
+<protocol crlf="yes">
-GET /%TESTNUMBER HTTP/1.1
+GET /%TESTNUMBER HTTP/1.1
-Host: %HOSTIP:%HTTPPORT
+Host: %HOSTIP:%HTTPPORT
-User-Agent: curl/%VERSION
+User-Agent: curl/%VERSION
-Accept: */*
+Accept: */*
-
+
 </protocol>
 </verify>
 </testcase>

tests/data/test2500

@@ -1,4 +1,4 @@
-<testcase http-crlf="yes">
+<testcase>
 <info>
 <keywords>
 HTTP
@@ -10,7 +10,7 @@ HTTP/3
 #
 # Server-side
 <reply>
-<data nocheck="yes">
+<data nocheck="yes" crlf="yes">
 HTTP/1.1 200 OK
 Date: Tue, 09 Nov 2010 14:49:00 GMT
 Last-Modified: Tue, 13 Jun 2000 12:10:00 GMT
@@ -51,7 +51,7 @@ HTTP/3 GET:
 ^X-Forwarded-Proto:.*
 ^Via:.*
 </strip>
-<protocol>
+<protocol crlf="yes">
 GET https://localhost:%HTTP3PORT/%TESTNUMBER HTTP/1.1
 Host: localhost:%HTTP3PORT
 User-Agent: curl/%VERSION

tests/runtests.pl

@@ -317,7 +317,6 @@ my %ignored_keywords;   # key words of tests to ignore results
 my %enabled_keywords;   # key words of tests to run
 my %disabled;           # disabled test cases
 my %ignored;            # ignored results of test cases
-my $crlf_http = 0;      # always convert HTTP heaaders to cr+lf
 my $sshdid;      # for socks server, ssh daemon version id
 my $sshdvernum;  # for socks server, ssh daemon version number
 my $sshdverstr;  # for socks server, ssh daemon version string
@@ -3609,7 +3608,13 @@ sub subBase64 {
 
 my $prevupdate;
 sub subNewlines {
-    my ($thing) = @_;
+    my ($force, $thing) = @_;
+
+    if($force) {
+        # enforce CRLF newline
+        $$thing =~ s/\x0d*\x0a/\x0d\x0a/;
+        return;
+    }
 
     # When curl is built with Hyper, it gets all response headers delivered as
     # name/value pairs and curl "invents" the newlines when it saves the
@@ -3624,7 +3629,7 @@ sub subNewlines {
         # skip curl error messages
         ($$thing !~ /^curl: \(\d+\) /))) {
         # enforce CRLF newline
-        $$thing =~ s/\x0a/\x0d\x0a/;
+        $$thing =~ s/\x0d*\x0a/\x0d\x0a/;
         $prevupdate = 1;
     }
     else {
@@ -3696,8 +3701,7 @@ sub prepro {
     my (@entiretest) = @_;
     my $show = 1;
     my @out;
-    my $crlf_header = ($crlf_http || ($has_hyper && ($keywords{"HTTP"}
+    my $data_crlf;
-                                                     || $keywords{"HTTPS"})));
     for my $s (@entiretest) {
         my $f = $s;
         if($s =~ /^ *%if (.*)/) {
@@ -3721,9 +3725,19 @@ sub prepro {
             next;
         }
         if($show) {
+            # The processor does CRLF replacements in the <data*> sections if
+            # necessary since those parts might be read by separate servers.
+            if($s =~ /^ *<data(.*)\>/) {
+                if($1 =~ /crlf="yes"/ || $has_hyper) {
+                    $data_crlf = 1;
+                }
+            }
+            elsif(($s =~ /^ *<\/data/) && $data_crlf) {
+                $data_crlf = 0;
+            }
             subVariables(\$s, $testnum, "%");
             subBase64(\$s);
-            subNewlines(\$s) if($crlf_header);
+            subNewlines(0, \$s) if($data_crlf);
             push @out, $s;
         }
     }
@@ -3917,12 +3931,6 @@ sub singletest {
         $why = serverfortest($testnum);
     }
 
-    $crlf_http = 0;
-    my %hash = testcaseattr();
-    if($hash{'http-crlf'}) {
-        $crlf_http = 1;
-    }
-
     # Save a preprocessed version of the entire test file. This allows more
     # "basic" test case readers to enjoy variable replacements.
     my @entiretest = fulltest();
@@ -4045,6 +4053,11 @@ sub singletest {
                     # of the datacheck
                     chomp($replycheckpart[$#replycheckpart]);
                 }
+                if($replycheckpartattr{'crlf'} ||
+                   ($has_hyper && ($keywords{"HTTP"}
+                                   || $keywords{"HTTPS"}))) {
+                    map subNewlines(0, \$_), @replycheckpart;
+                }
                 push(@reply, @replycheckpart);
             }
         }
@@ -4066,6 +4079,11 @@ sub singletest {
             map s/\r\n/\n/g, @reply;
             map s/\n/\r\n/g, @reply;
         }
+        if($replyattr{'crlf'} ||
+           ($has_hyper && ($keywords{"HTTP"}
+                           || $keywords{"HTTPS"}))) {
+            map subNewlines(0, \$_), @reply;
+        }
     }
 
     # this is the valid protocol blurb curl should generate
@@ -4522,6 +4540,12 @@ sub singletest {
             chomp($validstdout[$#validstdout]);
         }
 
+        if($hash{'crlf'} ||
+           ($has_hyper && ($keywords{"HTTP"}
+                           || $keywords{"HTTPS"}))) {
+            map subNewlines(0, \$_), @validstdout;
+        }
+
         $res = compare($testnum, $testname, "stdout", \@actual, \@validstdout);
         if($res) {
             return $errorreturncode;
@@ -4622,6 +4646,10 @@ sub singletest {
             }
         }
 
+        if($hash{'crlf'}) {
+            map subNewlines(1, \$_), @protstrip;
+        }
+
         if((!$out[0] || ($out[0] eq "")) && $protstrip[0]) {
             logmsg "\n $testnum: protocol FAILED!\n".
                 " There was no content at all in the file $SERVERIN.\n".
@@ -4753,6 +4781,11 @@ sub singletest {
                 map s/\r\n/\n/g, @outfile;
                 map s/\n/\r\n/g, @outfile;
             }
+            if($hash{'crlf'} ||
+               ($has_hyper && ($keywords{"HTTP"}
+                               || $keywords{"HTTPS"}))) {
+                map subNewlines(0, \$_), @outfile;
+            }
 
             my $strip;
             for $strip (@stripfile) {