Re: M4 syntax $11 vs. ${11}

[Eric Blake]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

According to Paul Eggert on 1/29/2007 10:39 AM:
> Eric Blake <address@hidden> writes:
> 
>> Because I was hoping that M4 2.0 could borrow a leaf from shell parsing,
>> and ALSO implement things like ${1-default} that is more compact than the
>> current ifelse(`$1',,`default',`$1') in M4 parlance.
> 
> Sure, but doesn't that just delay the conversion problem?  The price
> will eventually have to be paid for having a syntax that is so close
> to the shell's.

M4 parameter parsing is already close to the shell's - witness $1, $#, $*,
address@hidden  Which is why the Austin group approved ${ as the
implementation-defined sequence that could be used as the start of an
extended argument, as opposed to inventing something new like $<.  And
people using m4 to produce shell output already have to be aware of how to
use nested quoting to differentiate between m4's $1 and the shell's $[1]
(or [$]1 or $[]1, whatever style we recommend in a single-quoted context).

> 
> How about if M4 uses a slightly-different syntax, so that the two uses
> are never confused?  That way, the conversion problem won't ever have
> to be addressed, and Autoconf won't have to disable the extension
> during a transition period.
> 
> So, for example, suppose M4 uses the syntax ${{11}} to refer to
> argument 11?  Or (if you prefer something shorter) ${+11}?  An
> advantage to this sort of approach is that it gives you more leeway
> for future extensions.

You have a reasonable point there.  My current plan was to add new syntax
categories that could be controlled through the changesyntax builtin
(available in beta since at least 1.4o several years ago).  So using $<1>
instead of ${1} is easy - just use changesyntax([{<], [}>]) up front in
m4sugar.  On the other hand, supporting a multi-character delimiter that
is still selectable by the m4 user is a bit tougher; I would need to
introduce a new macro similar to changequote and changecom (perhaps
changeextarg({{, }}) for choosing ${{1}} to mean an m4-parsed extended
argument, leaving ${1} for literal output).

However, there is still the matter of what 1.4.9 --warn-syntax should warn
about.  1.4.9 doesn't support changesyntax, so I don't see an easy way to
warn about anything other than ${ at worst, or a more specific
${[character set] at best, corresponding to the default behavior of M4 2.0
in the absence of changesyntax.  The best 1.4.9 --warn-syntax can do is
warn about out-of-the box behavior changes in 2.0, rather than worrying
about how the user could apply changesyntax or the possible changeextarg.

Suppose we decide for autoconf to go with changesyntax to make $<1> be for
m4 and ${1} for the shell.  Then we pretty much have to tell users that
'm4 --warn-syntax' is incompatible with autom4te: it won't catch
autom4te's syntax, so don't use it.  The burden for detecting non-portable
constructs in autoconf snippets would then shift away from m4 1.4.9 and
into m4sugar.

On the other hand, if I follow your advice and make M4 2.0's default
behavior for extended arguments use a multi-character delimiter, let's say
${{1}} for the sake of discussion, then the 1.4.9 --warn-syntax problem
becomes trivial.  Simply warn about $<digit><digit> and ${{, since those
are the only sequences that will change semantics in out-of-the-box
behavior with 2.0; and since they are unlikely to exist in many existing
m4 scripts, we could even make autom4te turn on --warn-syntax by default.
 I would still need to implement something like changeextarg({,}) to allow
m4 to accept ${1} outside the context of autom4te, but that is no longer
autoconf's problem.  And having m4 recognize only ${{1}} out of the box is
still compatible with the Austin group's decision that reserved ${ for the
implementation.

Or we could still go with out-of-the-box behavior of M4 2.0 to recognize
${1}, but have m4sugar use changeextarg({{,}}), meaning that --warn-syntax
is incompatible with autoconf but that autoconf 2.62 can document that
${{}} is reserved for the day when autoconf 3.0 requires m4 2.0, rather
than disable m4 extended arguments altogether.

Or maybe I could make --warn-syntax take an optional argument.  Right now,
I hard-coded the regex \$[{0-9][0-9] that triggers the warning, but if
- --warn-syntax were to accept a regular expression of what it should warn
about, then m4sugar could use changeextarg({{,}}), and we could teach
autom4te to use --warn-syntax='\$\([0-9][0-9]\|{{\)' to detect sequences
that will not work with both 1.4.x and 2.0, up until the day that autoconf
3.0 requires m4 2.0.  And is --warn-syntax still the best name?  By taking
a regular expression argument, it can no longer warn on unrelated other
classes of syntax changes; maybe --warn-define-sequence would be better?

One contrasting thought, however - right now, the shorter syntax is
consumed by m4 leaving longer syntax for shell output ($1 vs. $[]1).
Introducing a multi-character extended argument as the default in 2.0
makes this inconsistent - now you have a short sequence ${1} used by the
shell and a long sequence ${{1}} used by 2.0 (at least, once we require m4
2.0 or later, which would probably be worthy of autoconf 3.0).  Whereas if
we just document now that any time you want shell argument expansion, use
empty quotes, as in $[]{1+"$[]@"}, we have achieved a consistent rule of
thumb that will be easier to remember, and I don't have to change
- --warn-syntax, although it does mean that existing scripts will need to be
converted or the extended argument feature of m4 2.0 disabled until we can
require m4 2.0.

Perhaps it is also worth investigating whether autoupdate can easily be
taught how to recognize problem syntax?  When using autoupdate, we could
then assume that any snippet that doesn't already specify that it requires
autoconf 2.62 or later must have intended for ${1} to be for the shell, so
rewrite it as $[]{1}, then teach autom4te to look for any problematic
overquoted $[]{ in the final output.

Let me think on this for a couple more days (as it is, I still haven't
completely implemented extended arguments in CVS m4; partly because I want
to do it right, so I want this thread to come to a consensus before I
commit to coding something), and get more feedback.  But after typing and
re-reading this email, I'm starting to lean towards the following:
- - in m4 1.4.9: make --warn-syntax (or whatever name we decide on) take an
optional argument of a regular expression to warn about if encountered in
macro definitions, with the default regex being \$[{0-9][0-9].
- - in m4 2.0: have ${1} be the out-of-the-box syntax for extended
arguments, but add a new macro changeextarg that can be used to recognize
${{1}} instead
- - in autoconf 2.62: if --warn-syntax is supported by the installed m4,
then turn it on with a regular expression to reject $10 and ${{1}},
neither of which seem to be in many current autoconf snippets; and in
m4sugar, if changesyntax is supported, then disable extended arguments;
that way, ${1} is reserved for the shell no matter what
- - in autoconf 3.0: require m4 2.0, no longer use --warn-syntax, and use
changeextarg to support ${{1}} m4 extended arguments

- --
Don't work too hard, make some time for fun as well!

Eric Blake             address@hidden
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Cygwin)
Comment: Public key at home.comcast.net/~ericblake/eblake.gpg
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFFv1TG84KuGfSFAYARAp1DAJ9DWe97TLG6Q0haeX2IgnRehNCM6gCeMx1p
wq8kfo85df6tbx5ZrwWvTpU=
=BgKE
-----END PGP SIGNATURE-----