Re: [Bug-gnulib] Re: strtok_r

[Bruno Haible]

Simon Josefsson wrote:
> considering
> that, e.g., UCS-4 is a widely used multibyte encoding that is not
> compatible with ASCII for any character.

UCS-4 is not in the game here. A sequence of UCS-4 code points is not a
char*, because
  1) uint32_t[] and char[] have different alignment restrictions,
  2) Even if you were to cast an uint32_t* to char*, strlen() of it is
     always <= 3, so it makes no sense to use the str* functions on them.

> Can't we say:
>
>     Caveat: It only support one-octet delimiters.  With many character
>             sets, non-ASCII characters cannot be used as delimiters.

No. The point I'm making is: ONLY the ASCII characters from 0x00..0x2F are
usable as delimiters in a locale-independent way. Even ASCII delimiters
such as '@', '\' or '_' are not usable with strtok_r, strsep etc. !

Bruno