tool_cb_wrt: fix invalid unicode for windows console

- Suppress an incomplete UTF-8 sequence at the end of the buffer.

- Attempt to reconstruct incomplete UTF-8 sequence from prior call(s)
  in current call.

Prior to this change, in Windows console UTF-8 sequences split between
two or more calls to the write callback would cause invalid "replacement
characters" U+FFFD to be printed instead of the actual Unicode
character. This is because in Windows only UTF-16 encoded characters are
printed to the console, therefore we convert the UTF-8 contents to
UTF-16, which cannot be done with partial UTF-8 sequences.

Reported-by: Maksim Arhipov

Fixes https://github.com/curl/curl/issues/9841
Closes https://github.com/curl/curl/pull/10890
This commit is contained in:
Jay Satiro 2023-04-04 05:10:52 -04:00
parent 0b947e8ca2
commit af3f4e419b
4 changed files with 136 additions and 21 deletions

View file

@ -464,6 +464,12 @@ static CURLcode post_per_transfer(struct GlobalConfig *global,
}
}
#ifdef WIN32
/* Discard incomplete UTF-8 sequence buffered from body */
if(outs->utf8seq[0])
memset(outs->utf8seq, 0, sizeof(outs->utf8seq));
#endif
/* if retry-max-time is non-zero, make sure we haven't exceeded the
time */
if(per->retry_numretries &&