[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[PATCH] implement --enable-encoding for UTF-8 info files
|
From: |
Bruno Haible |
|
Subject: |
[PATCH] implement --enable-encoding for UTF-8 info files |
|
Date: |
Sat, 6 Oct 2007 05:20:03 +0200 |
|
User-agent: |
KMail/1.5.4 |
Hi,
When I try to format the gettext.info file, I get warnings like this:
$ cd gettext-tools/doc
$ LANG= LC_MESSAGES= LC_ALL= LANGUAGE= makeinfo -I ../../gettext-runtime/doc
--no-split -I . -o gettext.info --enable-encoding gettext.texi
/packages/GETTEXT/work/gettext-work/gettext-tools/doc//msgfilter.texi:228:
warning: invalid encoded character `szlig'.
/packages/GETTEXT/work/gettext-work/gettext-tools/doc//iso-639.texi:228:
warning: invalid encoded character `aring'.
And the resulting info file contains transliterated forms of ß and similar,
_although_ I have explicitly specified --enable-encoding and
@documentencoding UTF-8 .
The code has a "fixxme", so apparently this behaviour of makeinfo is
undesired also by other poeple.
This fixes it for me.
2007-10-05 Bruno Haible <address@hidden>
* makeinfo/lang.c (unicode_map): New variable.
(cm_search_iso_map): In case of UTF-8 encoding, return a Unicode code
point.
(add_encoded_char_from_code): New function.
(add_encoded_char, cm_accent_generic_no_headers): Use it.
*** makeinfo/lang.c.bak 2007-08-16 19:42:20.000000000 +0200
--- makeinfo/lang.c 2007-10-06 05:09:53.000000000 +0200
***************
*** 486,495 ****
{ ISO_8859_15, "iso-8859-15", (iso_map_type *) iso8859_15_map },
{ KOI8_R, "koi8-r", (iso_map_type *) koi8_map },
{ KOI8_U, "koi8-u", (iso_map_type *) koi8_map },
! { UTF_8, "utf-8", asis_map }, /* fixxme: much more needed */
{ last_encoding_code, NULL, NULL }
};
/* To update this list, download the current language table from
http://www.loc.gov/standards/iso639-2; specifically,
--- 486,754 ----
{ ISO_8859_15, "iso-8859-15", (iso_map_type *) iso8859_15_map },
{ KOI8_R, "koi8-r", (iso_map_type *) koi8_map },
{ KOI8_U, "koi8-u", (iso_map_type *) koi8_map },
! { UTF_8, "utf-8", asis_map }, /* handled particularly in code
*/
{ last_encoding_code, NULL, NULL }
};
+ /* List of HTML entities. */
+ static struct { const char *html; unsigned int unicode; } unicode_map[] = {
+ /* Extracted from http://www.w3.org/TR/html401/sgml/entities.html through
+ sed -n -e 's|<!ENTITY \([^ ][^ ]*\) *CDATA "[&]#\([0-9][0-9]*\);".*| {
"\1", \2 },|p'
+ | sort -k3 -n */
+ { "quot", 34 },
+ { "amp", 38 },
+ { "lt", 60 },
+ { "gt", 62 },
+ { "nbsp", 160 },
+ { "iexcl", 161 },
+ { "cent", 162 },
+ { "pound", 163 },
+ { "curren", 164 },
+ { "yen", 165 },
+ { "brvbar", 166 },
+ { "sect", 167 },
+ { "uml", 168 },
+ { "copy", 169 },
+ { "ordf", 170 },
+ { "laquo", 171 },
+ { "not", 172 },
+ { "shy", 173 },
+ { "reg", 174 },
+ { "macr", 175 },
+ { "deg", 176 },
+ { "plusmn", 177 },
+ { "sup2", 178 },
+ { "sup3", 179 },
+ { "acute", 180 },
+ { "micro", 181 },
+ { "para", 182 },
+ { "middot", 183 },
+ { "cedil", 184 },
+ { "sup1", 185 },
+ { "ordm", 186 },
+ { "raquo", 187 },
+ { "frac14", 188 },
+ { "frac12", 189 },
+ { "frac34", 190 },
+ { "iquest", 191 },
+ { "Agrave", 192 },
+ { "Aacute", 193 },
+ { "Acirc", 194 },
+ { "Atilde", 195 },
+ { "Auml", 196 },
+ { "Aring", 197 },
+ { "AElig", 198 },
+ { "Ccedil", 199 },
+ { "Egrave", 200 },
+ { "Eacute", 201 },
+ { "Ecirc", 202 },
+ { "Euml", 203 },
+ { "Igrave", 204 },
+ { "Iacute", 205 },
+ { "Icirc", 206 },
+ { "Iuml", 207 },
+ { "ETH", 208 },
+ { "Ntilde", 209 },
+ { "Ograve", 210 },
+ { "Oacute", 211 },
+ { "Ocirc", 212 },
+ { "Otilde", 213 },
+ { "Ouml", 214 },
+ { "times", 215 },
+ { "Oslash", 216 },
+ { "Ugrave", 217 },
+ { "Uacute", 218 },
+ { "Ucirc", 219 },
+ { "Uuml", 220 },
+ { "Yacute", 221 },
+ { "THORN", 222 },
+ { "szlig", 223 },
+ { "agrave", 224 },
+ { "aacute", 225 },
+ { "acirc", 226 },
+ { "atilde", 227 },
+ { "auml", 228 },
+ { "aring", 229 },
+ { "aelig", 230 },
+ { "ccedil", 231 },
+ { "egrave", 232 },
+ { "eacute", 233 },
+ { "ecirc", 234 },
+ { "euml", 235 },
+ { "igrave", 236 },
+ { "iacute", 237 },
+ { "icirc", 238 },
+ { "iuml", 239 },
+ { "eth", 240 },
+ { "ntilde", 241 },
+ { "ograve", 242 },
+ { "oacute", 243 },
+ { "ocirc", 244 },
+ { "otilde", 245 },
+ { "ouml", 246 },
+ { "divide", 247 },
+ { "oslash", 248 },
+ { "ugrave", 249 },
+ { "uacute", 250 },
+ { "ucirc", 251 },
+ { "uuml", 252 },
+ { "yacute", 253 },
+ { "thorn", 254 },
+ { "yuml", 255 },
+ { "OElig", 338 },
+ { "oelig", 339 },
+ { "Scaron", 352 },
+ { "scaron", 353 },
+ { "Yuml", 376 },
+ { "fnof", 402 },
+ { "circ", 710 },
+ { "tilde", 732 },
+ { "Alpha", 913 },
+ { "Beta", 914 },
+ { "Gamma", 915 },
+ { "Delta", 916 },
+ { "Epsilon", 917 },
+ { "Zeta", 918 },
+ { "Eta", 919 },
+ { "Theta", 920 },
+ { "Iota", 921 },
+ { "Kappa", 922 },
+ { "Lambda", 923 },
+ { "Mu", 924 },
+ { "Nu", 925 },
+ { "Xi", 926 },
+ { "Omicron", 927 },
+ { "Pi", 928 },
+ { "Rho", 929 },
+ { "Sigma", 931 },
+ { "Tau", 932 },
+ { "Upsilon", 933 },
+ { "Phi", 934 },
+ { "Chi", 935 },
+ { "Psi", 936 },
+ { "Omega", 937 },
+ { "alpha", 945 },
+ { "beta", 946 },
+ { "gamma", 947 },
+ { "delta", 948 },
+ { "epsilon", 949 },
+ { "zeta", 950 },
+ { "eta", 951 },
+ { "theta", 952 },
+ { "iota", 953 },
+ { "kappa", 954 },
+ { "lambda", 955 },
+ { "mu", 956 },
+ { "nu", 957 },
+ { "xi", 958 },
+ { "omicron", 959 },
+ { "pi", 960 },
+ { "rho", 961 },
+ { "sigmaf", 962 },
+ { "sigma", 963 },
+ { "tau", 964 },
+ { "upsilon", 965 },
+ { "phi", 966 },
+ { "chi", 967 },
+ { "psi", 968 },
+ { "omega", 969 },
+ { "thetasym", 977 },
+ { "upsih", 978 },
+ { "piv", 982 },
+ { "ensp", 8194 },
+ { "emsp", 8195 },
+ { "thinsp", 8201 },
+ { "zwnj", 8204 },
+ { "zwj", 8205 },
+ { "lrm", 8206 },
+ { "rlm", 8207 },
+ { "ndash", 8211 },
+ { "mdash", 8212 },
+ { "lsquo", 8216 },
+ { "rsquo", 8217 },
+ { "sbquo", 8218 },
+ { "ldquo", 8220 },
+ { "rdquo", 8221 },
+ { "bdquo", 8222 },
+ { "dagger", 8224 },
+ { "Dagger", 8225 },
+ { "bull", 8226 },
+ { "hellip", 8230 },
+ { "permil", 8240 },
+ { "prime", 8242 },
+ { "Prime", 8243 },
+ { "lsaquo", 8249 },
+ { "rsaquo", 8250 },
+ { "oline", 8254 },
+ { "frasl", 8260 },
+ { "euro", 8364 },
+ { "image", 8465 },
+ { "weierp", 8472 },
+ { "real", 8476 },
+ { "trade", 8482 },
+ { "alefsym", 8501 },
+ { "larr", 8592 },
+ { "uarr", 8593 },
+ { "rarr", 8594 },
+ { "darr", 8595 },
+ { "harr", 8596 },
+ { "crarr", 8629 },
+ { "lArr", 8656 },
+ { "uArr", 8657 },
+ { "rArr", 8658 },
+ { "dArr", 8659 },
+ { "hArr", 8660 },
+ { "forall", 8704 },
+ { "part", 8706 },
+ { "exist", 8707 },
+ { "empty", 8709 },
+ { "nabla", 8711 },
+ { "isin", 8712 },
+ { "notin", 8713 },
+ { "ni", 8715 },
+ { "prod", 8719 },
+ { "sum", 8721 },
+ { "minus", 8722 },
+ { "lowast", 8727 },
+ { "radic", 8730 },
+ { "prop", 8733 },
+ { "infin", 8734 },
+ { "ang", 8736 },
+ { "and", 8743 },
+ { "or", 8744 },
+ { "cap", 8745 },
+ { "cup", 8746 },
+ { "int", 8747 },
+ { "there4", 8756 },
+ { "sim", 8764 },
+ { "cong", 8773 },
+ { "asymp", 8776 },
+ { "ne", 8800 },
+ { "equiv", 8801 },
+ { "le", 8804 },
+ { "ge", 8805 },
+ { "sub", 8834 },
+ { "sup", 8835 },
+ { "nsub", 8836 },
+ { "sube", 8838 },
+ { "supe", 8839 },
+ { "oplus", 8853 },
+ { "otimes", 8855 },
+ { "perp", 8869 },
+ { "sdot", 8901 },
+ { "lceil", 8968 },
+ { "rceil", 8969 },
+ { "lfloor", 8970 },
+ { "rfloor", 8971 },
+ { "lang", 9001 },
+ { "rang", 9002 },
+ { "loz", 9674 },
+ { "spades", 9824 },
+ { "clubs", 9827 },
+ { "hearts", 9829 },
+ { "diams", 9830 }
+ };
+
/* To update this list, download the current language table from
http://www.loc.gov/standards/iso639-2; specifically,
***************
*** 1051,1070 ****
static int
cm_search_iso_map (char *html)
{
! int i;
! iso_map_type *iso = encoding_table[document_encoding_code].isotab;
! /* If no conversion table for this encoding, quit. */
! if (!iso)
! return -1;
! for (i = 0; iso[i].html; i++)
! {
! if (strcmp (html, iso[i].html) == 0)
! return i;
}
! return -1;
}
--- 1310,1342 ----
static int
cm_search_iso_map (char *html)
{
! if (document_encoding_code == UTF_8)
! {
! size_t i;
! for (i = 0; i < sizeof (unicode_map) / sizeof (unicode_map[0]); i++)
! if (strcmp (html, unicode_map[i].html) == 0)
! return unicode_map[i].unicode;
! return -1;
}
+ else
+ {
+ int i;
+ iso_map_type *iso = encoding_table[document_encoding_code].isotab;
+
+ /* If no conversion table for this encoding, quit. */
+ if (!iso)
+ return -1;
+
+ for (i = 0; iso[i].html; i++)
+ {
+ if (strcmp (html, iso[i].html) == 0)
+ return i;
+ }
! return -1;
! }
}
***************
*** 1146,1151 ****
--- 1418,1454 ----
}
+ static void
+ add_encoded_char_from_code (int rc)
+ {
+ if (document_encoding_code == UTF_8)
+ {
+ if (rc < 0x80)
+ add_char (rc);
+ else if (rc < 0x800)
+ {
+ add_char (0xc0 | (rc >> 6));
+ add_char (0x80 | (rc & 0x3f));
+ }
+ else if (rc < 0x10000)
+ {
+ add_char (0xe0 | (rc >> 12));
+ add_char (0x80 | ((rc >> 6) & 0x3f));
+ add_char (0x80 | (rc & 0x3f));
+ }
+ else
+ {
+ add_char (0xf0 | (rc >> 18));
+ add_char (0x80 | ((rc >> 12) & 0x3f));
+ add_char (0x80 | ((rc >> 6) & 0x3f));
+ add_char (0x80 | (rc & 0x3f));
+ }
+ }
+ else
+ add_char (encoding_table[document_encoding_code].isotab[rc].bytecode);
+ }
+
+
/* If html or xml output, add &HTML_STR; to the output. If not html and
the user requested encoded output, add the real 8-bit character
corresponding to HTML_STR from the translation tables. Otherwise,
***************
*** 1164,1170 ****
int rc = cm_search_iso_map (html_str);
if (rc >= 0)
/* We found it, add the real character. */
! add_char (encoding_table[document_encoding_code].isotab[rc].bytecode);
else
{ /* We didn't find it, that seems bad. */
warning (_("invalid encoded character `%s'"), html_str);
--- 1467,1473 ----
int rc = cm_search_iso_map (html_str);
if (rc >= 0)
/* We found it, add the real character. */
! add_encoded_char_from_code (rc);
else
{ /* We didn't find it, that seems bad. */
warning (_("invalid encoded character `%s'"), html_str);
***************
*** 1268,1280 ****
rc = cm_search_iso_map (buffer);
if (rc >= 0)
! /* A little bit tricky ;-)
! Here we replace the character which has
! been inserted in read_command with
! the value we have found in converting table
! Does there exist a better way to do this? kama. */
! output_paragraph[end - 1]
! = encoding_table[document_encoding_code].isotab[rc].bytecode;
else
{ /* If we didn't find a translation for this character,
put the single instead. E.g., &Xuml; does not exist so X¨
--- 1571,1587 ----
rc = cm_search_iso_map (buffer);
if (rc >= 0)
! {
! /* A little bit tricky ;-)
! Here we replace the character which has
! been inserted in read_command with
! the value we have found in converting table.
! For a multibyte character we use the first byte to
! overwrite the character, then we append the remaining bytes.
! Does there exist a better way to do this? kama. */
! output_paragraph_offset--;
! add_encoded_char_from_code (rc);
! }
else
{ /* If we didn't find a translation for this character,
put the single instead. E.g., &Xuml; does not exist so X¨
- [PATCH] implement --enable-encoding for UTF-8 info files,
Bruno Haible <=