Re: How to use-modules within macro?

[pelzflorian (Florian Pelz)]

Thank you for giving my approach a kind of “trial by fire”.
Considering what you wrote, I still believe in my datum->syntax
approach in my special case.  After all, datum->syntax exists for a
reason, I suppose.  I am interested in your further comments.  They
have been very helpful.

On Wed, Sep 04, 2019 at 03:54:13PM -0400, Mark H Weaver wrote:
> If you could give me a birds-eye view of what you're trying to do here,
> I might be able to suggest other approaches.
> 

I am trying to i18n the Guix website by writing a gettext that accepts
s-expresions.

For example, the about page of Guix <https://guix.gnu.org/about/>
would use this SHTML:

      ;; TRANSLATORS: Features and Defining Packages are section names
      ;; in the English (en) manual.
      ,(G_
        `(p
          "GNU Guix provides "
          ,(G_ (manual-href "state-of-the-art package management features"
                            (G_ "en")
                            (G_ "Features.html")))
          " such as transactional upgrades and roll-backs, reproducible
            build environments, unprivileged package management, and
            per-user profiles.  It uses low-level mechanisms from the "
          ,(G_ `(a (@ (href "https://nixos.org/nix/";)) "Nix"))
          " package manager, but packages are "
          ,(G_ (manual-href "defined" (G_ "en") (G_ "Defining-Packages.html")))
          " as native "
          ,(G_ `(a (@ (href ,(gnu-url "software/guile"))) "Guile"))
          " modules, using extensions to the "
          ,(G_ `(a (@ (href "http://schemers.org";)) "Scheme"))
          " language—which makes it nicely hackable."))


Another program I wrote converts it to a POT file from which I make
this German translation PO file:


#. TRANSLATORS: Features and Defining Packages are section names
#. in the English (en) manual.
#: apps/base/templates/about.scm:64
msgid ""
"GNU Guix provides <1>state-of-the-art package management "
"features<1.1>en</1.1><1.2>Features.html</1.2></1> such as transactional "
"upgrades and roll-backs, reproducible\n"
"            build environments, unprivileged package management, and\n"
"            per-user profiles.  It uses low-level mechanisms from the "
"<2>Nix</2> package manager, but packages are "
"<3>defined<3.1>en</3.1><3.2>Defining-Packages.html</3.2></3> as native "
"<4>Guile</4> modules, using extensions to the <5>Scheme</5> language—which "
"makes it nicely hackable."
msgstr ""
"GNU Guix bietet <1>Paketverwaltungsfunktionalitäten auf dem Stand der "
"Technik<1.1>de</1.1><1.2>Funktionalitaten.html</1.2></1>, wie etwa "
"transaktionelle Aktualisierungen und Rücksetzungen, reproduzierbare "
"Erstellungsumgebungen, eine „unprivilegierte“ Paketverwaltung für Nutzer "
"ohne besondere Berechtigungen sowie ein eigenes Paketprofil für jeden "
"Nutzer. Dazu verwendet es dieselben Mechanismen, die dem "
"Paketverwaltungsprogramm <2>Nix</2> zu Grunde liegen, jedoch werden Pakete "
"als reine <4>Guile</4>-Module <3>definiert<3.1>de</3.1><3.2>Pakete-"
"definieren.html</3.2></3>. Dazu erweitert Guix die <5>Scheme</5>-"
"Programmiersprache, wodurch es leicht ist, selbst an diesen zu hacken."


Then the G_ macro takes apart the original SHTML and, using the
translation, reorders and recombines it automatically to this (I
manually added the indentation now):

(quasiquote
 (p "GNU Guix bietet "
    (unquote
     (manual-href "Paketverwaltungsfunktionalitäten auf dem Stand der Technik"
                  "de"
                  ""
                  "Funktionalitaten.html"
                  ""))
    ", wie etwa transaktionelle Aktualisierungen und Rücksetzungen, 
reproduzierbare Erstellungsumgebungen, eine „unprivilegierte“ Paketverwaltung 
für Nutzer ohne besondere Berechtigungen sowie ein eigenes Paketprofil für 
jeden Nutzer. Dazu verwendet es dieselben Mechanismen, die dem 
Paketverwaltungsprogramm "
    (unquote (quasiquote (a (@ (href "https://nixos.org/nix/";))
                            "Nix")))
    " zu Grunde liegen, jedoch werden Pakete als reine "
    (unquote (quasiquote (a (@ (href (unquote (gnu-url "software/guile"))))
                            "Guile")))
    "-Module "
    (unquote
     (manual-href "definiert"
                  "de"
                  ""
                  "Pakete-definieren.html"
                  ""))
    ". Dazu erweitert Guix die "
    (unquote (quasiquote (a (@ (href "http://schemers.org";))
                            "Scheme")))
    "-Programmiersprache, wodurch es leicht ist, selbst an diesen zu hacken."))



The reason I argued for such a gettext-like approach is that it is
less effort to mark existing SHTML for translation with G_ than if we
were to rewrite everything using something like sirgazil’s format
strings
<https://gitlab.com/sirgazil/guile-lab/blob/master/glab/i18n.scm>, I
think, and that the resulting msgid text in the POT file seems easier
to work with for translators.  Ricardo Wurmus proposed manual XML
without macros that would be easy for translators too, but a G_ macro
is still easier for SHTML authors, I think.  I believe this gettext
approach could be used easily for very many webpages and on other
websites; currently I have a not-yet-submitted completed
internationalization for the entire Guix website but without its very
many blog posts.


Now let me look at your comments:

On Wed, Sep 04, 2019 at 03:54:13PM -0400, Mark H Weaver wrote:
> Hi Florian,
> 
> "pelzflorian (Florian Pelz)" <address@hidden> writes:
> 
> > […]
> 
> There are some problems above:
> 
> (1) The first argument to 'datum->syntax' must be an identifier, which
>     is the syntax object corresponding to a symbol.  Here, you are
>     passing an entire expression, and in the example usage above, #'exp
>     will be the syntax object corresponding to (* 2 3 hour).  Guile
>     should ideally raise an error in this case.
>

Ahh so in the example above it should be

(define-syntax one-more
  (lambda (x)
    (syntax-case x ()
      ((id exp)
       (datum->syntax
        #'id

instead of

> > (define-syntax one-more
> >   (lambda (x)
> >     (syntax-case x ()
> >       ((_ exp)
> >        (datum->syntax
> >         #'exp

I will change this.



> (2) The way you are doing things here destroys hygiene within the
>     expression that you are rewriting.  You convert the entire
>     expression with 'syntax->datum', process the datum, and then convert
>     the rewritten expression using 'datum->syntax'.  The problem here is
>     that 'syntax->datum' discards all of the extra information about
>     lexical environments of identifiers that were kept in the syntax
>     object.  This will cause severe problems when 'one-more' is used in
>     combination with other macros, including unintended variable
>     capture.
>

Yes, but I do not think capturing everything named G_ is a problem for
this special use case.  If someone were to use the ordinary gettext
for pure strings (let’s say its keyword is G_), it would be just as
much of a problem to use the chosen gettext keyword G_ for anything
other than gettext anywhere in the program, because xgettext would
include that other thing in the POT file for translators.

I acknowledge that error messages for errors within the marked
s-expression may be less precise, I suppose.




> To do this properly, you must do the rewriting on the syntax objects
> themselves.  It's okay to convert a syntax object to a datum to test
> whether it's a literal number, but the important thing is that all
> *identifiers* in the rewritten code should be preserved.
> 
> So, instead of using 'match' on the result of 'syntax->datum', you
> should instead use 'syntax-case' on the syntax object itself, like this
> (untested):
> 
>   (let loop ((e #'exp))
>     (syntax-case e ()
>       (num
>        (number? (syntax->datum #'num))
>        #'(1+ num))
>       ((x ...)
>        (map loop #'(x ...)))
>       (y
>        #'y)))
>

When I look at your second mail:

On Wed, Sep 04, 2019 at 05:55:58PM -0400, Mark H Weaver wrote:
> […]
> I should mention that the use of 'map' directly on a syntax object is
> only allowable in certain cases.
> […]
> As a result, there are only a few cases when you can safely assume that
> the top structure of a syntax object is a normal list or pair, and they
> are spelled out in the documentation for 'syntax' in the R6RS Standard
> Libraries specification:
> […]
>   * the copy of (<t> <ellipsis>) is a list if <t> contains any pattern
>     variables,
> […]

I come to believe that the recommended matching functionality of
syntax-case is not powerful enough for what I want the G_ macro to do.

Another consideration is what you mention last in your first mail:

> Finally, I should mention that macro expansion is always done from the
> outside in, meaning that when 'one-more' is expanded, its operand will
> not yet have been expanded.

This outside-in aspect is vital for what I want to do, and it is the
reason I use macros and not procedures, which I would understand much
better.  For my example above, the inner G_ “macros” within the outer
G_ s-expressions are removed from the output by the outer G_ on
purpose and are only used to distinguish sub-s-expressions with
translatable text from sub-s-expressions without translatable text.
If the inner G_ were evaluated before the outer G_ is evaluated, no
distinction would be possible.




> In general, this means that it's impossible
> to comprehend the code within a macro's operands unless the parsing code
> knows about every macro that might be used within the operands.  It's
> not even possible to know which subparts are expressions and which are
> other things like variable binding lists.
> 
> For this reason, I think it's generally a mistake to try to parse code
> within a macro's operands.  It normally only makes sense for macros to
> inspect the parts of operands that are considered part of the macro's
> syntax.  For example, it makes sense for a 'let' macro to parse its
> binding list, or for a 'match' macro to parse its patterns and
> templates, but it does *not* make sense for a macro to try to parse
> general subexpressions passed to the macro.
> 
> If you could give me a birds-eye view of what you're trying to do here,
> I might be able to suggest other approaches.
> 
>        Best,
>         Mark


I will add a comment to my code that my use of datum->syntax is a very
special use case warranting its use, and that in most cases, one
should not try this at home.

Regards,
Florian