Re: test results differents between the perl and XS parsers

[Gavin Smith]

On Mon, Nov 16, 2020 at 11:28:23AM +0100, Patrice Dumas wrote:
> Hello,
> 
> It is not very important, but would be practical.  Right now the perl
> parser and XS parser outputs are different in a non important way but
> the difference makes the Data::Dumper dump of their results different
> which makes looking at their difference somewhat impractical.  Not a big
> deal, as it does not have any effect on the tests failing or passing.
> So it is possible to regenerate the tests with the perl parser before
> working on a change.  Still, could be nice if it could be fixed.  Here
> is what I investigated.  The differences are like 
> 
> -      'line_nr' => {
> -        'file_name' => '',
> -        'line_nr' => 3,
> -        'macro' => ''
> -      },
> +      'line_nr' => {},
> 
> +$result_trees{'alias_table_command'}{'contents'}[2]{'line_nr'} = 
> $result_trees{'alias_table_command'}{'contents'}[2]{'args'}[0]{'contents'}[0]{'line_nr'};
> 
> In the perl parser, the two elements share the same 'line_nr' object, in
> the XS parser they are duplicate.  One is in the 'extra' field of the
> parent command.  Looking at the trees, it seems to happen with @*table*
> (and maybe in other case, I haven't fully investicated as there are too
> many).  I think that the perl parser is better in that case, the
> whole 'extra' in the parent command should point to the element and avoid 
> duplicating anything.

It is because the line number is part of the ELEMENT type (defined
in tree_types.c).  Note that this is defined as LINE_NR line_nr,
and not as LINE_NR *line_nr.  Thus, there is no scope for sharing
of line number objects.  While this might save memory, it would make
managing memory and avoiding memory leaks much more difficult and just
copying the object is much simpler.

In the Perl code, however, there is always a current line number object
available and when this is referenced the line number is stored as
a reference to the object.  It looks like

          $current->{'contents'}->[-1]->{'line_nr'} = $line_nr;

Hence the object can easily be shared.  Perl's garbage collector
takes care of memory leaks.

The simplest solution to this issue seems to be a "normalisation" phase
(for the tests only) where the line number information is duplicated.

> I tried to look at the C code to find where the reference to the 
> extra in the parent is done in the case of @table and @vtable, 
> and I found in parsetexi/parser.c l 1405
>           add_extra_element (current->parent->parent,
>                                  "command_as_argument", current);
> Looking at extra.c, add_extra_element calls add_extra_key which does not
> seem to duplicate the value but rather points to it.  So I am not sure
> what's going on which leads to the duplication.

There isn't enough information in your email to say for certain what
is going on, but add_extra_element will reference the element, and not
copy it.  There will only be one element with one line number object,
but for the pure Perl output that line number object is being shared with
something else.