Re: [Texmacs-dev] Adding TeXmacs symbols to FriCAS math-mode input

[Bill Page]

On 28 March 2014 10:40, François Poulain <address@hidden> wrote:
> Hi,
>
> [I remove fricas-devel from CC, since it seems impossible to email them
> without be part of the group...]
>

OK no problem.  I will continue to copy this to the fricas-devel group
in case there is still some interest.

> Le Thu, 27 Mar 2014 08:42:08 -0400,
> Bill Page <address@hidden> a écrit :
>
>> > Most of the doc is in Help -> Interfacing.
>>
>> Yes. :( Unfortunately unicode does not seem to be mentioned anywhere
>> there.
>
> Actually, charset is a confusing thing into TeXmacs (due to Cork which
> is almost-but-not-completely ascii compliant, often confused with
> latin1, and other weirds things like TS1 (or T2A in recent past...)).
>
> As far as strings are concerned, cork->utf8 will generally make the job.
>

OK, understood.  I found that I could change my fricas-input.scm to
just pass the cork symbols to display and then TeXmacs automatically
does the conversion to utf-8 without being told.

(define (fricas-send-identifier arg)
  (display arg)
)

I think this deserves to be mentioned in the TeXmacs documentation
somewhere. That way it seems both text-mode input and math-mode input
are treated the same way and FriCAS consistently gets utf-8 characters
as input.

> At document level (stree), a serialisation procedure has to be
> written. The definition via plugin-input-converters allows you to
> tweak the "generic" serializer for fricas specificities.
>
> I am not sure that there is much more to say about it.
>

A few more pictures and examples might help... ;)  I find the
documentation rather sketchy and it still seems difficult to me to
distill the general design into specific requirements. But your
comments are very helpful.

>> But my problem is how to ensure that TeXmacs converts the utf-8
>> characters in scheme: output format generated by FirCAS into math
>> symbols inside TeXmacs.
>
> Actually, utf8 characters in a scheme: session will be badly
> interpreted, because your fricas output should send scheme valid code
> (so with TeXmacs universal encoding).
>

Yes they are interpreted in a useful way in the scheme session. I
guess scheme (at least the version used in TeXmacs) does not support
unicode. So since we are generating scheme as an output from FriCAS,
really we should be doing the conversion from utf-8 to cork in FriCAS
itself, right?  Certainly this is possible and we can probably borrow
some of the source code from TeXmacs to help with that.  Note: I did
find a large table of cork -utf-8 equivalences in scheme/lisp format
in the TeXmacs source but from what I understand this table is
actually loaded into a fast hash table lookup routine written in C++.

> A way to workaround this without rewriting a converter from scratch is
> to call a TeXmacs function at the rendering level via "extern". Try
> e.g. to paste this from "TeXmacs scheme" into TeXmacs :
>
> (math (extern "(lambda (x) (utf8-<gtr>cork (tree-<gtr>stree x)))" "ℝ"))
>
> I used this in the Scilab converter. Be carefull to properly escape the
> brackets.
>

Nice. Yes, this does work!  Thank you very much for explaining this. I
can see however that what this really amounts to is extending the
definition of the scheme format itself.  We can do this but it seems
to have some serious limitations when it comes to cut-and-paste,
particularly between FriCAS and other types of sessions in TeXmacs.

>>  The examples in the documentation only
>> consider the case of latex: format which FriCAS no longer uses for the
>> TeXmacs interface.  FriCAS now interfaces directly with TeXmacs
>> without any intermediate conversion program in the pipe.
>
> So, you have to comply a minimum with the TeXmacs format (charset and
> structure).
>

OK I understand a little better now the implications of generating
scheme format in FriCAS.

>> In experiments I found that TeXmacs does sometimes recognize and
>> convert utf-8 characters to the Cork
>
> This may be possible. There is heuristic detection of charset at
> different input level.
>
>> form but it seems this happens
>> only is parts of the scheme markup that is supposed to be interpreted
>> as literal text and for example the result for char(945) is shown
>> literally as <alpha> in TeXmacs - not the Greek symbol alpha.
>
> If this append, you're quite right : the charset is good. *But*, you
> should use this string in a math environment.
>

I think it is a bit more subtle.  In math environment a single
character string is typeset differently (italized) than a
multi-character string (roman). This conforms to common math usage of
symbols but it looks odd in most other cases where strings of
characters occur.

>> In other cases char(945) results in
>> two unexpected non-ascii characters which seem related to the internal
>> utf-8 encoding so that TeXmacs has not performed the conversion.
>> Again, I do not know where this happens in the TeXmacs source code.
>> It is not something that I do in the code.
>
> I guess there is no conversion, then there is nowhere to search in the
> code. char(945) should result in the string "\xCE\xB1" which,
> interpreted as Cork encoding, means "Îś". Isn't it?
>

Yes.  This works now by incorporating your 'extern' function into the
scheme output generated by FriCAS.

> But it is uneasy to properly understand what happen since I don't have
> fricas installed. It should be simpler with a document containing an
> executed session and, optionnaly, the log (with Debug -> io
> activated).
>

After all of this testing and although it now works, I am convinced
that having chosen to use scheme, it is really our responsibility in
FriCAS to generate proper scheme coding with symbols encoded in cork.
It is just going to be a little more work.

Regards,
Bill Page.