Re: new modules for grapheme cluster breaking

[Bruno Haible]

Hello Ben,

> (I also had to convert GraphemeBreakTest.txt from DOS to Unix
> line-ends and remove trailing white space, because the Git hook
> refused to accept it otherwise.

This is good; I convert these files from ftp.unicode.org to Unix
line-ends myself when I retrieve them, because these DOS line-ends
have more than once brought trouble.

> > The next modules will have a higher-level API, I imagine. You're welcome
> > to discuss the new API with me, before you implement it.
> 
> I am thinking of something like:
>         size_t u<#>_grapheme_len (const uint<#>_t *s, size_t n);
> which would return the number of units in the first grapheme
> cluster in S.

"grapheme" or "grapheme cluster"? I'm a bit confused: The Unicode 3.0
book uses the term "grapheme" to denote the entity that users consider
to be a single character, but UAX #29 nowadays calls it "grapheme cluster".

OK for this kind of API, if the grapheme break determination is context free
(unlike the word break determination). Can you confirm that?

A function for iterating backwards, i.e. returning the grapheme
bounds before a certain point in a string, would be useful too
then. Like u#_next and u#_prev in <unistr.h>:

  const uint#_t * u#_grapheme_next (const uint#_t *s, const uint#_t *end);
  const uint#_t * u#_grapheme_prev (const uint#_t *s, const uint#_t *start);

And for convenience, I would suggest an API that operates on an
entire string, like done for the word breaks:

/* Determine the grapheme [cluster?] break points in S, and store the result
   at p[0..n-1].
   p[i] = 1 means that there is a grapheme [cluster?] boundary between s[i-1] 
and s[i].
   p[i] = 0 means that s[i-1] and s[i] must not be separated.
 */
extern void
       u8_grapheme_breaks (const uint8_t *s, size_t n, char *p);
extern void
       u16_grapheme_breaks (const uint16_t *s, size_t n, char *p);
extern void
       u32_grapheme_breaks (const uint32_t *s, size_t n, char *p);
extern void
       ulc_grapheme_breaks (const char *s, size_t n, char *p);

Bruno