Re: Buffer overrun when syntax fragment contains no terminating newline.

[Ben Pfaff]

On Sat, Sep 29, 2018 at 04:26:28PM +0200, John Darrington wrote:
> I've just pushed a change fixing some sporadic crashes in the gui.   
> 
> The bug (which took a bit of tracking down) turned out to be caused by
> a buffer overrun in lexer.c (lex_source_get_).   In particular, we have
> the code:
> 
>  const char *newline = rawmemchr (line, '\n');
> 
> But the documentation for rawmemchr says that it's unpredictable if
> line does not contain a '\n'.
> 
> So this means our syntax parser can crash if we present it with a
> fragment which is not newline terminated.   I wasn't aware that we
> had such a limitation.     Does this need to be fixed, or at least
> explicitly documented ?

Until recently, the lexer and its lower level infrastructure required
source files to end in \n.  Because of this limitation, all the code
that read source files added a trailing newline if one wasn't already
present.  I fixed the limitation in commit e0f9210e814d ("lexer: Add
support for embedded \0 bytes and missing trailing new-line.") because
it made null bytes hard to handle properly.  At the same time, I removed
the code to automatically add a trailing newline, because it was no
longer necessary.

In my code review, I missed this code that still assumed a trailing
newline, and none of the tests caught it for me.  I pushed what I
believe to be a fix now; I don't have enough time right at the moment to
add some more tests, but I'll try to go back and add them later.