Full unicode support for back-xlib

[Kazunobu Kuriyama]

Hi, all.

Attached are the patches to make the xlib backend fully supports unicode.
Accordingly, if the patches are applied, the backend should handle all 
characters
represented with NSGlyph.

If the X Window System in use is already set up properly for your language
environment (i.e., locale, fonts, an input method), GNUstep with back-xlib
should work in cooperation with your native language only by setting the
GNUSTEP_STRING_ENCODING to an appropriate value. For example, you can browse
and edit text files written in your native language with GNUstep 
applications.
(NOTE: This may not be true if a language in use is one of those in which
text is drawn from right to left, such as Arabic or Hebrew. )

The underlying window system must be XFree86 4.0.2 or higher to enable
the new function provided by the patches.

Moreover, to enable it, you need to pass -DUSE_MULITIBYTE to a compiler when
you build -back.  When this macro is not defined, the implementation
reverts to the original.

To ease this task, I prepared the patch for configure.ac. To use this, go to
the top directory of the source tree of -back, and do
   $ cp configure.ac configurea.ac.orig
   $ patch < configure.ac.patch
   $ autoheader
   $ autoconf
Then you will find the new option --enable-multibyte in the newly generated
configure script.  You can use this option to pass the above macro to a
compiler automatically.

The new configure script also checks if the underlying window system is
XFree86 4.0.2 or higher if --enable-multibyte is specified.

The patches were made against the corresponding files stored in the CVS
repository of July 10.

Any feedback is welcome.

- KK

2003-07-10  Kazunobu Kuriyama  <kazunobu.kuriyama@nifty.com>

        * Source/xlib/XGFont.m: Add full unicode support.
        ([XGFontInfo -dealloc]): new implementation (alternative)
        ([XGFontInfo -drawGlyphs::::]): ditto
        ([XGFontInfo -widthOfGlyphs::]): ditto
        ([XGFontInfo (Private) -setupAttributes::]): ditto
        ([XGFontInfo (Private) -xCharStructForGlyph:]): ditto
        ([static glyph2utf8]): new procedure
        ([static setup_font_info_for_utf8]): new procedure
        
        * configure.ac: Add --multi-byte option.

--- XGFont.m.orig       2003-07-10 01:48:40.000000000 +0900
+++ XGFont.m    2003-07-10 22:42:24.000000000 +0900
@@ -33,6 +33,43 @@
    59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
 */
 
+/*
+    Note on a UTF8 Extension for the xlib backend
+    ---------------------------------------------
+
+    This extension is for those who want to use multibyte characters with
+    text widgets such as NSTextView, but don't want to use (or don't know how
+    to use) freetype2 + iso10646, i.e. the backend art.
+
+    The underlying X Window System must be XFree86 4.0.2 or higher because
+    the extension is based on the XFree86's extension that defines
+    Xutf8DrawString(), Xutf8TextExtent() and so on.  To see if your X Window
+    System has the latter extension, search for the macro X_HAVE_UTF8_STING
+    in /usr/X11R6/include/X11/Xlib.h.  If you find it, you can go further.
+
+    The modification I made is completely limited to the implementation.
+    Therefore the interface remains the same as before, and hence there's
+    no need to modify the source code of the client programs to enjoy the
+    utf8 extension.
+
+    If the X Window System in use already works properly for your language
+    environment (i.e. locale, fonts, and X Input Method), you can exploits
+    the extension only by setting
+       
+       $ export GNUSTEP_STRING_ENCODING=<encoding you want to use>
+
+    For possible values, refer to base/Headers/gnustep/base/NSString.h.
+    So far only NSJapaneseEUCStringEncoding has been tested.
+
+    To enable the extension, run the configure script with the option
+    --enable-graphics=xlib and define the constant macro USE_MULTIBYTE
+    near the top of this file.  Then follow the familiar command sequence:
+    make; make install.  If the macro USE_MULTIBYTE is not defined,
+    the implementation reverts to the original one.
+
+    2003-06-14 Kazunobu Kuriyama <kazunobu.kuriyama@nifty.com>
+ */
+
 #include <config.h>
 
 #include "xlib/XGContext.h"
@@ -45,6 +82,36 @@
 // For the encoding functions
 #include <base/Unicode.h>
 
+
+// UTF8 extension: Uncomment the following line if you want to enable an
+// experimental UTF8 extension so that you can use mulibyte characters with 
+// NSTextView etc.
+//#define USE_MULTIBYTE
+
+#if defined(X_HAVE_UTF8_STRING) && defined(USE_MULTIBYTE)
+#define USE_X_UTF8_STRING 1
+#endif
+
+#ifdef USE_X_UTF8_STRING
+
+#define MAX_FONT_SETS 16
+#define CurrentFontSet font_sets[current_font_set].set
+#define CurrentFontInfo        font_sets[current_font_set].info
+typedef struct FontSet_ {
+    char           init_font_name[128];
+    XFontSet       set;
+    XFontStruct*    info;
+} FontSet;
+
+static char* glyph2utf8(const NSGlyph* glyphs, int length, int *num_bytes);
+static XFontStruct* setup_font_info_for_utf8(Display *xdpy,
+                                            NSString *xfontname);
+
+static const int max_font_sets = MAX_FONT_SETS;
+static FontSet font_sets[MAX_FONT_SETS];    // Never use the first element.
+static int current_font_set = 0;
+#endif // USE_X_UTF8_STRING defined
+
 static Atom XA_SLANT = (Atom)0;
 static Atom XA_SETWIDTH_NAME = (Atom)0;
 static Atom XA_CHARSET_REGISTRY = (Atom)0;
@@ -125,10 +192,27 @@
 
 - (void) dealloc
 {
+#ifdef USE_X_UTF8_STRING
+  //
+  // 2003-06-14
+  //
+  // We can't call XFreeFontSet here due to an X syncronization problem,
+  // i.e, it frees the font set before Xutf8DrawString draws a string,
+  // thus getting the application to crash.  Though the original
+  // implementation potentially has had the same problem, it has not been
+  // manifested because XDrawString is far lighter than Xutf8DrawString.
+  // If some GNUstep applications result in segfault when X is busy,
+  // this may be a possible cause.
+  //
+  // Currently, the memory leek resulting from not calling XFreeFontSet
+  // is avoided by using the circular list 'font_sets'. 
+  //
+#else
   if (font_info != NULL)
     {
       XFreeFont([XGServer currentXDisplay], font_info);
     }
+#endif // USE_X_UTF8_STRING not defined
   [super dealloc];
 }
 
@@ -214,6 +298,31 @@
          onDisplay: (Display*) xdpy drawable: (Drawable) draw
               with: (GC) xgcntxt at: (XPoint) xp
 {
+#ifdef USE_X_UTF8_STRING
+  const char   *x_font_name = NULL;
+  char         *utf8str = NULL;
+  int          num_bytes = 0;
+  int          i;
+
+  if ((x_font_name = [XGXFontName(fontName, matrix[0]) cString]) == NULL)
+    return;
+
+  for (i = MAX_FONT_SETS - 1; i > 0; --i)
+    {
+      if (!strcmp(font_sets[i].init_font_name, x_font_name))
+       break;
+    }
+  if (i == 0)
+    return;
+
+  if ((utf8str = glyph2utf8(glyphs, len, &num_bytes)) != NULL)
+    {
+      Xutf8DrawString(xdpy, draw, font_sets[i].set,
+                     xgcntxt, xp.x, xp.y,
+                     utf8str, num_bytes);
+      free(utf8str);
+    }
+#else
   // This font must already be active!
   unsigned char buf[len];
   int i;
@@ -222,6 +331,7 @@
       buf[i] = glyphs[i];
     }
   XDrawString(xdpy, draw, xgcntxt, xp.x, xp.y, buf, len);
+#endif // USE_X_UTF8_STRING not defined
 }
 
 - (float) widthOfString: (NSString*)string
@@ -242,6 +352,24 @@
 
 - (float) widthOfGlyphs: (const NSGlyph *) glyphs lenght: (int) len
 {
+#ifdef USE_X_UTF8_STRING
+  char *utf8str = NULL;
+  int  num_bytes;
+  float        val;
+
+  if ((utf8str = glyph2utf8(glyphs, len, &num_bytes)) == NULL)
+    {
+      val = 0.0;
+    }
+  else
+    {
+      XRectangle logical;
+      Xutf8TextExtents(CurrentFontSet, utf8str, num_bytes, NULL, &logical);
+      val = (float)logical.width;
+      free(utf8str);
+    }
+  return val;
+#else // USE_X_UTF8_STRING not defined
   unsigned char buf[len];
   int i;
   for (i = 0; i < len; i++)
@@ -249,6 +377,7 @@
       buf[i] = glyphs[i];
     }
   return XTextWidth(font_info, buf, len);
+#endif // USE_X_UTF8_STRING not defined
 }
 
 - (void) setActiveFor: (Display*) xdpy gc: (GC) xgcntxt
@@ -280,6 +409,15 @@
   // Retrieve the XLFD matching the given fontName. DPS->X.
   xfontname = XGXFontName(fontName, matrix[0]);
 
+#ifdef USE_X_UTF8_STRING
+  if (!setup_font_info_for_utf8(xdpy, xfontname))
+    {
+      NSLog(@"XGFont: selected font: %@ at %f (%@) is not available.\n", 
+           fontName, matrix[0], xfontname);
+      return NO;
+    }
+  font_info = CurrentFontInfo;
+#else // USE_X_UTF8_STRING not defined
   // Load X font and get font info structure.
   if ((xfontname == nil) ||
       (font_info = XLoadQueryFont(xdpy, [xfontname cString])) == NULL)
@@ -290,6 +428,7 @@
     }
   else
     NSDebugLog(@"Loaded font: %@", xfontname);
+#endif // USE_X_UTF8_STRING not defined
 
   // Fill the ivars
   ASSIGN(familyName, XGFontFamily(xdpy, font_info));
@@ -346,6 +485,49 @@
 
 - (XCharStruct *)xCharStructForGlyph: (NSGlyph) glyph
 {
+#ifdef USE_X_UTF8_STRING 
+  char                 *utf8char = NULL;
+  int                  bytes = 0;
+  int                  num_chars = 0;
+  static XCharStruct   c = { 0 };
+  XRectangle           ink, logical;
+  const char           *x_font_name;
+  int                  i;
+
+  if ((x_font_name = [XGXFontName(fontName, matrix[0]) cString]) == NULL)
+    return &c;
+
+  for (i = MAX_FONT_SETS - 1; i > 0; --i)
+    {
+      if (!strcmp(font_sets[i].init_font_name, x_font_name))
+       break;
+    }
+  if (i == 0)
+    return &c;
+
+  if ((utf8char = glyph2utf8(&glyph, 1, &bytes)) == NULL)
+    {
+      return &c;
+    }
+  Xutf8TextPerCharExtents(CurrentFontSet, utf8char, bytes,
+                         &ink, &logical, 1,
+                         &num_chars,
+                         NULL, NULL);
+
+  // This is almost OK for Chinese, Japanese, and Korean.  How about other
+  // languages other than European ones?
+  c.lbearing = 0;
+  c.rbearing = 0;
+  c.width = logical.width;
+  c.ascent = CurrentFontInfo->max_bounds.ascent;
+  c.descent = CurrentFontInfo->max_bounds.descent;
+  c.attributes = 0;
+
+  free(utf8char);
+
+  return &c;
+
+#else // USE_X_UTF8_STRING not defined
   XCharStruct *pc = NULL;
 
   if (font_info->per_char)
@@ -392,8 +574,238 @@
                                     b2 - min2]);
         }
     }
-
   return pc;
+#endif // USE_X_UTF8_STRING not defined
 }
 
 @end
+
+
+#ifdef USE_X_UTF8_STRING
+//
+//  glyph2utf8
+//
+//  Converts a given NSGlyph sequence to a byte string encoded in utf8.
+//  Returns NULL if failed, or a pointer to the byte string obtained.
+//  Also, the length of the byte string is returned through the parameter
+//  'num_bytes'.  It is the user's respoinsibility to release the allocated
+//  space pointed to by the return value.
+//
+//  Currently, the implemtation is based on the fact that NSGlyph is a typedef
+//  of unsigned int, disobeying the OO programming discipline.
+//
+static char *
+glyph2utf8(const NSGlyph *glyphs, int length, int *num_bytes)
+{
+  char     *utf8str = NULL;
+  int      utf8str_len = 0;
+  int      utf8str_alloc = 0;
+  NSGlyph   *g = NULL;
+
+  // Theoretically possible maximum, assuming a utf8 character per NSGlyph
+  // instance.
+  utf8str_alloc = 6 * length;
+
+  if (!glyphs || !num_bytes) return NULL;
+
+  if ((utf8str = malloc(utf8str_alloc)) == NULL)
+    {
+      *num_bytes = 0;
+      return NULL;
+    }
+
+  // Referring to http://www.cl.cam.ac.uk/~mgk25/unicode.html,
+  utf8str_len = 0;
+  for (g = (NSGlyph *)glyphs; g < glyphs + length; ++g)
+    {
+      if (*g < 0x00000080)
+       {
+         utf8str[utf8str_len++] = *g;
+       }
+      else if (*g < 0x00000800)
+       {
+         utf8str[utf8str_len++] = 0xc0 | ((*g >>  6) & 0x1f);
+         utf8str[utf8str_len++] = 0x80 | ( *g        & 0x3f);
+       }
+      else if (*g < 0x00010000)
+       {
+         utf8str[utf8str_len++] = 0xe0 | ((*g >> 12) & 0x0f);
+         utf8str[utf8str_len++] = 0x80 | ((*g >>  6) & 0x3f);
+         utf8str[utf8str_len++] = 0x80 | ( *g        & 0x3f);
+       }
+      else if (*g < 0x00200000)
+       {
+         utf8str[utf8str_len++] = 0xf0 | ((*g >> 18) & 0x07);
+         utf8str[utf8str_len++] = 0x80 | ((*g >> 12) & 0x3f);
+         utf8str[utf8str_len++] = 0x80 | ((*g >>  6) & 0x3f);
+         utf8str[utf8str_len++] = 0x80 | ( *g        & 0x3f);
+       }
+      else if (*g < 0x04000000)
+       {
+         utf8str[utf8str_len++] = 0xf8 | ((*g >> 24) & 0x03);
+         utf8str[utf8str_len++] = 0x80 | ((*g >> 18) & 0x3f);
+         utf8str[utf8str_len++] = 0x80 | ((*g >> 12) & 0x3f);
+         utf8str[utf8str_len++] = 0x80 | ((*g >>  6) & 0x3f);
+         utf8str[utf8str_len++] = 0x80 | ( *g        & 0x3f);
+       }
+      else if (*g < 0x80000000)
+       {
+         utf8str[utf8str_len++] = 0xfc | ((*g >> 30) & 0x01);
+         utf8str[utf8str_len++] = 0x80 | ((*g >> 24) & 0x3f);
+         utf8str[utf8str_len++] = 0x80 | ((*g >> 16) & 0x3f);
+         utf8str[utf8str_len++] = 0x80 | ((*g >> 12) & 0x3f);
+         utf8str[utf8str_len++] = 0x80 | ((*g >>  6) & 0x3f);
+         utf8str[utf8str_len++] = 0x80 | ( *g        & 0x3f);
+       }
+      else
+       {
+         // Out of range.
+         free(utf8str);
+         *num_bytes = 0;
+         return NULL;
+       }
+    }
+  *num_bytes = utf8str_len;
+  return utf8str;
+}
+
+//
+//  setup_font_info_for_utf8
+//
+//  Sets XFontSet based on the the parameter 'xfontname'.  Returns NULL
+//  if failed, or a pointer to XFontStruct of a font that is chosen as
+//  the base one.  The returned value is used as the instance variable
+//  'font_info'.
+//
+static XFontStruct *
+setup_font_info_for_utf8(Display *xdpy, NSString *xfontname)
+{
+  char         xlfd[256];
+  char         base_font_name[256];
+  int          xlfd_num_elms = 14;
+  char         *xlfd_elms[14];
+  int          i;
+  BOOL         has_add_style;
+
+  XFontSet     font_set;
+  char         **missing_charsets;
+  int          num_missing_charsets;
+  char         *def_string;
+
+  int          num_fonts;
+  XFontStruct  **font_structs;
+  char         **font_names;
+  XFontStruct  *info;
+
+  if (!xdpy || !xfontname)
+    {
+      return NULL;
+    }
+
+  strcpy(xlfd, [xfontname cString]);
+  has_add_style = YES;
+  if (strstr(xlfd, "--")) 
+    {
+      has_add_style = NO;
+      --xlfd_num_elms;
+    }
+
+  i = 0;
+  xlfd_elms[i++] = strtok(xlfd, "-");
+  while (i < xlfd_num_elms)
+    {
+      xlfd_elms[i] = strtok(NULL, "-");
+      ++i;
+    }
+
+  // To let the X server determine a font set automatically, some elements
+  // of the XLFD should be replaced with the wild card.
+  if (has_add_style) 
+    {
+      sprintf(base_font_name,
+             "-%s-%s-%s-%s-%s-%s-%s-%s-%s-%s-%s-%s-%s-%s",
+             xlfd_elms[0],                 // foundry
+             "*",                          // family
+             xlfd_elms[2],                 // weight
+             xlfd_elms[3],                 // slant
+             xlfd_elms[4],                 // set width
+             xlfd_elms[5],                 // add style
+             xlfd_elms[6],                 // pixel size
+             xlfd_elms[7],                 // point size
+             xlfd_elms[8],                 // resolutionX
+             xlfd_elms[9],                 // resolutionY
+             "*",                          // spacing
+             xlfd_elms[11],                // avg width
+             "*",                          // registry
+             "*"                           // encoding
+      );
+    }
+  else
+    {
+      sprintf(base_font_name,
+             "-%s-%s-%s-%s-%s--%s-%s-%s-%s-%s-%s-%s-%s",
+             xlfd_elms[0],                 // foundry
+             "*",                          // family
+             xlfd_elms[2],                 // weight
+             xlfd_elms[3],                 // slant
+             xlfd_elms[4],                 // set width
+             xlfd_elms[5],                 // pixel size 
+             xlfd_elms[6],                 // point size
+             xlfd_elms[7],                 // resolutionX
+             xlfd_elms[8],                 // resolutionY
+             "*",                          // spacing
+             xlfd_elms[10],                // avg width
+             "*",                          // registry
+             "*"                           // encoding
+      );
+    }
+
+  // N.B. def_string is owned by the X server.
+  font_set = XCreateFontSet(xdpy, base_font_name,
+                           &missing_charsets,
+                           &num_missing_charsets,
+                           &def_string);
+  if (!font_set)
+    {
+      NSLog(@"XGFont: Can't create a font set.\n");
+      return NULL;
+    }
+  if (num_missing_charsets > 0)
+    {
+      for (i = 0; i < num_missing_charsets; ++i)
+       {
+         NSLog(@"XGFont: Charset %s is not available.\n",
+               missing_charsets[i]);
+       }
+      XFreeStringList(missing_charsets);
+    }
+
+  // N.B. font_structs and font_names are owned by the X server.
+  num_fonts = XFontsOfFontSet(font_set, &font_structs, &font_names);
+  if (!num_fonts)
+    {
+      NSLog(@"XGFont: Can't get any information from the font set.\n");
+      return NULL;
+    }
+
+  // FIXME: Is this always the base font?
+  info = font_structs[0];
+
+  // Add the font set to the circular list 'font_sets'.
+  if (current_font_set == MAX_FONT_SETS - 1) 
+    {
+      // Since the list is full, release the oldest font set.
+      current_font_set = 1;
+      XFreeFontSet(xdpy, font_sets[current_font_set].set);
+    }
+  else
+    {
+      ++current_font_set;
+    }
+  strcpy(font_sets[current_font_set].init_font_name, [xfontname cString]);
+  font_sets[current_font_set].set = font_set;
+  font_sets[current_font_set].info = info;
+
+  return info;
+}
+#endif // USE_X_UTF8_STRING

--- configure.ac.orig   2003-07-10 01:48:39.000000000 +0900
+++ configure.ac        2003-07-10 22:34:13.000000000 +0900
@@ -203,6 +203,40 @@
 AC_SUBST(WITH_XFT)
 
 #--------------------------------------------------------------------
+# Yet another extended font support & UTF8 support for back-xlib
+#
+# - This section is relevent only if --enable-graphics=xlib.
+# - Defines USE_MULTIBYTE in config.h.in.
+# - USE_MULTIBYTE is to set to 1 in config.h if --enable-multibyte is
+#   given and X11/Xlib.h defines X_HAVE_UTF8_STRING.
+# - Otherwise, this section does nothing. 
+#--------------------------------------------------------------------
+AC_ARG_ENABLE(multibyte,
+    [  --enable-multibyte      Enable multibyte char support for xlib 
graphics],
+    enable_multibyte=yes, enable_multibyte=no)
+
+if test "x$enable_multibyte" = "xyes"; then
+    AC_MSG_CHECKING([for XFree86 >= 4.0.2])
+    enable_multibyte_save_header=${CPPFLAGS}
+    CPPFLAGS="${GRAPHIC_CFLAGS} ${GRAPHIC_LFLAGS} ${CPPFLAGS}"
+    AC_EGREP_CPP(yes,
+                [
+                   #include <X11/Xlib.h>
+                   #ifdef X_HAVE_UTF8_STRING
+                   yes
+                   #endif
+                ], have_x_have_utf8_string=yes, have_x_have_utf8_string=no)
+    AC_MSG_RESULT([$have_x_have_utf8_string])
+    if test "x$have_x_have_utf8_string" = "xyes"; then
+       AC_DEFINE([USE_MULTIBYTE], 1,
+                 [Define to enable multibyte char support for xlib graphics])
+    else
+       AC_MSG_WARN([You need XFree86 >= 4.0.2 for --enable-multibyte])
+    fi
+    CPPFLAGS="${enable_multibyte_save_header}"
+fi
+
+#--------------------------------------------------------------------
 # GLX support
 #--------------------------------------------------------------------
 WITH_GLX=no