Readdelim (was Re: Examples of concurrent coproc usage)

[Martin D Kealey]

On Sun, 28 Apr 2024, 05:03 Carl Edquist, <edquist@cs.wisc.edu> wrote:

>
> > I would hope that mapfile/readarray could do better, since it's not
> > obligated to leave anything in the input stream.
>
> That is an interesting thought, although mapfile seems to read a byte at a
> time also.
>
> [I'm not suggesting this should be changed though.  And frankly I have no
> strong desire for a faster 'read' builtin in the shell, either.  A byte at
> a time is relatively slow, but that's generally fine in most limited
> contexts where it's actually needed.]
>

I'm encountering contexts where the lack of speed is annoying though not
critical.

My thought is that (performance considerations aside), the real functional
> improvement with a new "readd" call would be with _competing_ readers
> (more than one read call waiting on the same pipe at the same time).
>

That's a very good point; I'll mention that when I write to the Linux
kernel team.

In that case a length-prefixed or type-tagged record wouldn't seem to work
> with the regular read(2), because a single reader would not be able to
> read the length/type _and_ the corresponding record together.  You can't
> work around this by reading a byte at a time either.  That's why I said it
> would only seem to work (with read(2)) if the records have a fixed size.
> (In order to grab a whole record atomically.)
>

This makes me wonder whether readv() could be encouraged to handle
length-prefix simply by targeting the length element of a subsequent
element of the iovec array? iov[0].ptr = &iov[1].len; iov[0].len = sizeof
iov[1].len; iov[1].ptr = malloc(MAX_RECORD_SIZE); I suspect this won't work
today because the kernel probably computes the total read size up front.
But maybe some day?

-Martin