Re: MinGW vs. setlocale

[Eli Zaretskii]

> From: address@hidden (Ludovic Courtès)
> Date: Mon, 09 Jun 2014 21:30:46 +0200
> 
> > 1. i18n.test completely fails, because it depends on the ability to
> >    change the program's locale at run time.  I wish this whole test
> >    were skipped on Windows.  (I'm quite sure I reported this last
> >    year.)
> 
> What does ‘setlocale’ return when called more than once on Windows?  Is
> there an exception thrown or something that would allow i18n.test to
> determine that tests should be skipped?

A very good question, thank you for asking it.  The short answer is
that yes, it threw an exception.  The long answer is that while
looking into the reasons of these exceptions, I found a few snafus,
and succeeded to fix some, so that most of the i18n test now works on
Windows.

Here are the details.

First, make-locale threw an exception, because it tried to call
'setlocale' with LC_MESSAGES, which the Windows runtime doesn't
support.  locale-categories.h tried to avoid that by conditioning that
call by LC_MESSAGES being defined, but the oh-so-helpful libintl.h
header file just happens to define it to some arbitrary large
constant.  So the ifdef didn't work, and setlocale barfed.  Here's the
suggested solution:

--- libguile/locale-categories.h~0      2010-12-14 21:15:17 +0200
+++ libguile/locale-categories.h        2014-06-10 18:54:06 +0300
@@ -23,8 +23,10 @@
 SCM_DEFINE_LOCALE_CATEGORY (COLLATE)
 SCM_DEFINE_LOCALE_CATEGORY (CTYPE)
 
-#ifdef LC_MESSAGES
-/* MinGW doesn't have `LC_MESSAGES'.  */
+#if defined(LC_MESSAGES) && !(defined(LC_MAX) && LC_MESSAGES > LC_MAX)
+/* MinGW doesn't have `LC_MESSAGES'.  libintl.h might define
+   `LC_MESSAGES' for MinGW to an arbitrary large value which we cannot
+   use in a call to `setlocale'.  */
 SCM_DEFINE_LOCALE_CATEGORY (MESSAGES)
 #endif
 

The next problem is that i18n.test uses Posix locale strings, whereas
the Windows runtime names the same locales by different names.
Moreover, Windows 'setlocale' doesn't support UTF-8 encoding (even
though a Windows UTF-8 codepage exists).  So every test for a locale
other than "C" was failing, because setlocale failed.  I replaced
Posix locales with similar Windows ones; see the following patch, in
which I also emoved all but one LC_MESSAGES, because these always fail
on Windows:

--- test-suite/tests/i18n.test~0        2014-01-22 00:20:53 +0200
+++ test-suite/tests/i18n.test  2014-06-10 11:24:15 +0300
@@ -40,16 +40,19 @@
   (pass-if "make-locale (2 args, list)"
     (not (not (make-locale (list LC_COLLATE LC_MESSAGES) "C"))))
 
+  (pass-if "make-locale (2 args, list)"
+    (not (not (make-locale (list LC_COLLATE LC_NUMERIC) "C"))))
+
   (pass-if "make-locale (3 args)"
     (not (not (make-locale (list LC_COLLATE) "C"
-                           (make-locale (list LC_MESSAGES) "C")))))
+                           (make-locale (list LC_NUMERIC) "C")))))
 
   (pass-if-exception "make-locale with unknown locale" exception:locale-error
     (make-locale LC_ALL "does-not-exist"))
 
   (pass-if "locale?"
     (and (locale? (make-locale (list LC_ALL) "C"))
-         (locale? (make-locale (list LC_MESSAGES LC_NUMERIC) "C"
+         (locale? (make-locale (list LC_TIME LC_NUMERIC) "C"
                                (make-locale (list LC_CTYPE) "C")))))
 
   (pass-if "%global-locale"
@@ -82,19 +85,29 @@
 
 
 (define %french-locale-name
-  "fr_FR.ISO-8859-1")
+  (if (string-contains %host-type "-mingw32")
+      "fra_FRA.850"
+      "fr_FR.ISO-8859-1"))
 
 (define %french-utf8-locale-name
-  "fr_FR.UTF-8")
+  (if (string-contains %host-type "-mingw32")
+      "fra_FRA.1252"
+      "fr_FR.UTF-8"))
 
 (define %turkish-utf8-locale-name
-  "tr_TR.UTF-8")
+  (if (string-contains %host-type "-mingw32")
+      "tur_TRK.1254"
+      "tr_TR.UTF-8"))
 
 (define %german-utf8-locale-name
-  "de_DE.UTF-8")
+  (if (string-contains %host-type "-mingw32")
+      "deu_DEU.1252"
+      "de_DE.UTF-8"))
 
 (define %greek-utf8-locale-name
-  "el_GR.UTF-8")
+  (if (string-contains %host-type "-mingw32")
+      "grc_ELL.1253"
+      "el_GR.UTF-8"))
 
 (define %american-english-locale-name
   "en_US")
@@ -148,13 +161,14 @@
   (under-locale-or-unresolved %french-utf8-locale thunk))
 
 (define (under-turkish-utf8-locale-or-unresolved thunk)
-  ;; FreeBSD 8.2 and 9.1, Solaris 2.10, and Darwin 8.11.0 have a broken
-  ;; tr_TR locale where `i' is mapped to uppercase `I' instead of `Ä°',
-  ;; so disable tests on that platform.
+  ;; FreeBSD 8.2 and 9.1, Solaris 2.10, Darwin 8.11.0, and MinGW have
+  ;; a broken tr_TR locale where `i' is mapped to uppercase `I'
+  ;; instead of `İ', so disable tests on that platform.
   (if (or (string-contains %host-type "freebsd8")
           (string-contains %host-type "freebsd9")
           (string-contains %host-type "solaris2.10")
-          (string-contains %host-type "darwin8"))
+          (string-contains %host-type "darwin8")
+         (string-contains %host-type "-mingw32"))
       (throw 'unresolved)
       (under-locale-or-unresolved %turkish-utf8-locale thunk)))
 
@@ -192,7 +206,10 @@
         ;; strings.
         (dynamic-wind
           (lambda ()
-            (setlocale LC_ALL "fr_FR.UTF-8"))
+           (setlocale LC_ALL
+                      (if (string-contains %host-type "-mingw32")
+                          "fra_FRA.1252"
+                          "fr_FR.UTF-8")))
           (lambda ()
             (string-locale-ci=? "œuf" "ŒUF"))
           (lambda ()
             (setlocale LC_ALL "C"))))))

After all these changes, some tests still fail or throw exceptions:

  UNRESOLVED: i18n.test: text collation (French): string-locale-ci=?
  UNRESOLVED: i18n.test: text collation (French): string-locale-ci=? (2 args, 
wide strings)
  UNRESOLVED: i18n.test: text collation (French): string-locale-ci=? (3 args, 
wide strings)
  UNRESOLVED: i18n.test: text collation (French): string-locale-ci<>?
  UNRESOLVED: i18n.test: text collation (French): string-locale-ci<>? (wide 
strings)
  UNRESOLVED: i18n.test: text collation (French): string-locale-ci<>? (wide and 
narrow strings)
  UNRESOLVED: i18n.test: text collation (French): char-locale-ci<>?
  UNRESOLVED: i18n.test: text collation (French): char-locale-ci<>? (wide)
  UNRESOLVED: i18n.test: text collation (Greek): string-locale-ci=?
  UNRESOLVED: i18n.test: character mapping: char-locale-upcase Turkish
  UNRESOLVED: i18n.test: character mapping: char-locale-downcase Turkish
  UNRESOLVED: i18n.test: string mapping: string-locale-upcase Greek
  UNRESOLVED: i18n.test: string mapping: string-locale-upcase Greek (two sigmas)
  UNRESOLVED: i18n.test: string mapping: string-locale-downcase Greek
  UNRESOLVED: i18n.test: string mapping: string-locale-downcase Greek (two 
sigmas)
  UNRESOLVED: i18n.test: string mapping: string-locale-upcase Turkish
  UNRESOLVED: i18n.test: string mapping: string-locale-downcase Turkish

I don't know why these fail.  Is it possible that the underlying
functions assume that the string arguments are encoded according to
the locale's codeset?  If so, since the source file is encoded in
UTF-8, that won't work on Windows, and the strings need to be recoded
before they are passed to libunistring functions.  Any ideas for
debugging this are welcome.

  FAIL: i18n.test: nl-langinfo et al.: locale-day (French)
  FAIL: i18n.test: nl-langinfo et al.: locale-day (French, using 
`%global-locale')

This is because gnulib's nl_langinfo only supports C locale for the
day names.  I'm taking this up with gnulib maintainers.

  FAIL: i18n.test: number->locale-string: French: integer
  FAIL: i18n.test: number->locale-string: French: fraction
  FAIL: i18n.test: number->locale-string: French: fraction, 1 digit
  FAIL: i18n.test: monetary-amount->locale-string: French: integer
  FAIL: i18n.test: monetary-amount->locale-string: French: fraction

There's no blank after the 7th digit, where the test expects it.  Not
sure what kind of problem is that, perhaps again due to gnulib's
nl_langinfo.

  UNRESOLVED: i18n.test: format ~h: French: 12345.5678
  UNRESOLVED: i18n.test: format ~h: English: 12345.5678

~h is not supported on Windows.