[Bug binutils/10165] New: windres crashes when processing resource files with UTF-8 (cp 65001) encoding.

[tjb at postpro dot net]

OS: Windows XP SP2

windres, version 2.18 and up, when processing an rc file that contains at least
one quoted string, will crash with an access violation (on dereference of NULL
pointer) if the code page is explicitly set to 65001 (UTF-8).  

Here is an example input that causes the issue:

======================== BEGIN =========================
#define IDS_STRING1 1
#pragma code_page(65001)

STRINGTABLE
BEGIN
        IDS_STRING1 "Any string will do..."
END
========================= END ==========================

The cause:

When calling the MultiByteToWideChar function with a code page of 65000, 65001,
or others (see http://msdn.microsoft.com/en-us/library/dd319072(VS.85).aspx),
the dwFlags parameter must be 0, unless when using Windows XP and later, in
which it may be 0 or MB_ERR_INVALID_CHARS.  If the value given for dwFlags is in
fact set to a value other than the aforementioned valid cases, the function will
set the last error to ERROR_INVALID_FLAGS.  The wind_MultiByteToWideChar
function always passes MB_PRECOMPOSED as the dwFlags parameter to
MultiByteToWideChar, causing the function to fail and not convert the UTF-7/8
string to UTF-16.  The wind_MultiByteToWideChar function assumes that the
MultiByteToWideChar function is successful, when in reality, the wide character
buffer still contains uninitialized data.


What worked for me:

======================== BEGIN =========================
--- binutils-2.19.1/binutils/winduni.c  2007-07-05 12:54:44 -0400
+++ binutils-2.19.1-new/binutils/winduni.c      2009-05-17 17:52:33 -0400
@@ -661,7 +661,15 @@ wind_MultiByteToWideChar (rc_uint_type c
   rc_uint_type ret = 0;
 
 #if defined (_WIN32) || defined (__CYGWIN__)
-  ret = (rc_uint_type) MultiByteToWideChar (cp, MB_PRECOMPOSED,
+  rc_uint_type conv_flags = MB_PRECOMPOSED;
+
+  /* MB_PRECOMPOSED is not allowed for UTF-7 or UTF-8.  
+     MultiByteToWideChar will set the last error to ERROR_INVALID_FLAGS
+     if we do anyways. */
+  if (cp == CP_UTF8 || cp == CP_UTF7)
+      conv_flags = 0;
+
+  ret = (rc_uint_type) MultiByteToWideChar (cp, conv_flags,
                                            mb, -1, u, u_len);
   /* Convert to bytes. */
   ret *= sizeof (unichar);
======================== END =========================

-- 
           Summary: windres crashes when processing resource files with UTF-
                    8 (cp 65001) encoding.
           Product: binutils
           Version: 2.19
            Status: NEW
          Severity: normal
          Priority: P2
         Component: binutils
        AssignedTo: unassigned at sources dot redhat dot com
        ReportedBy: tjb at postpro dot net
                CC: bug-binutils at gnu dot org
 GCC build triplet: i686-pc-mingw32
  GCC host triplet: i686-pc-mingw32
GCC target triplet: i686-pc-mingw32


http://sourceware.org/bugzilla/show_bug.cgi?id=10165

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.