build: tidy up and simplify setmode() detection and use

- move macro to `curl_setup.h` (from curlx), and rename.
  It's required by src, test servers, libtests. Also used by unit/tunit,
  (which is fixable but this patch doesn't touch it.)
- special-case it for Windows/Cygwin/MS-DOS.
- build: drop `setmode()`/`_setmode()` detection.
  This also avoids detecting the different `setmode()` on BSDs,
  and a lot of complexity and overhead.
- use `CURL_O_BINARY`.

Follow-up to 250d613763 #15787
Follow-up to 5e70566094 #15169

Closes #20539

lib/curl_setup.h

@@ -868,14 +868,23 @@
 
 /* Since O_BINARY is used in bitmasks, setting it to zero makes it usable in
    source code but yet it does not ruin anything */
-#ifdef _O_BINARY  /* for _WIN32 */
+#ifdef _O_BINARY  /* for _WIN32 || MSDOS */
 #define CURL_O_BINARY _O_BINARY
-#elif defined(O_BINARY)
+#elif defined(O_BINARY)  /* __CYGWIN__ */
 #define CURL_O_BINARY O_BINARY
 #else
 #define CURL_O_BINARY 0
 #endif
 
+/* Requires io.h when available */
+#ifdef MSDOS
+#define CURL_BINMODE(stream) (void)setmode(fileno(stream), CURL_O_BINARY)
+#elif defined(_WIN32) || defined(__CYGWIN__)
+#define CURL_BINMODE(stream) (void)_setmode(fileno(stream), CURL_O_BINARY)
+#else
+#define CURL_BINMODE(stream) (void)stream
+#endif
+
 /* In Windows the default file mode is text but an application can override it.
 Therefore we specify it explicitly. https://github.com/curl/curl/pull/258
 */