keep information of texinfo source

[Patrice Dumas]

Hello,

I think that there a many potential uses of trying to keep all the information
available in the original texinfo source, even the information that is not 
relevant for conversion.  There is already some of this information in the
tree, like @ignore and @macro blocks, @c and @comment, and some specially
flagged text (mostly spaces) that should be ignorable, like 
empty_line_after_command, empty_spaces_after_command, spaces_at_end, 
empty_spaces_before_argument...

But there is also some information lost, such as protection of end of lines
in @def* commands with @, or @include file beginning or end, user defined
macro and value expansion start and end, cpp synchronisation lines, \x{7F}
comments, @if* commands both for kept and removed blocks.

I think this would be important to keep all this information in the parsed
tree. It would also certainly simplify code to flag all those constructs
that are ignorable, for codes that checks empty arguments, for instance.

So I think it would be nice to add a new key in the tree, that would be set
for all those constructs.  It could be 'hidden', 'marksource', 'underlying'.
Right now 'marksource' has my preference, so I'll use that in the following,
but tell me what you'd prefer.

For implementation reasons, it will certainly be required to add a marker
for end of expansion of @value, @include file and user defined macro.  
Currently, my proposal would be to use a new @-command, like 
@marksource{endmacro, mymacro, 3} but which would not embed all the 
information, more like a command that can be used by the Parser to do some
texinfo source rewrite, for internal use.  One could imagine in the future 
generating such commands for the cases above and additional information in a 
separate file to help other parsers, but at first the @marksource
would never appear in the final output, even when generating Texinfo.

In the documentation, the only reference to @marksource would be to say
that this command exist but only internally and should not be used by users.

If this information is embedded in the tree, it would certainly be possible to
recreate include files, or to fold user macro call or @value while having
a tree at the same time.

Opinions?

-- 
Pat