Re: "here strings" and tmpfiles

[Chet Ramey]

On 4/9/19 7:48 PM, L A Walsh wrote:


> ----
>     I am aware of that, however, if a pipe implementation
> *stops* on reaching a full condition from some 'tmp-storage-space'
> and awaits for space to become available, a similar dynamic would
> apply.  That's all. 

Is there a pipe implementation out there that uses temporary files? And
if there is, should I worry about that less-than-the-lowest-common-
denominator case?

> 
> Example:  Suppose output from a program
> was buffered to disk files 64k in size.  The reader
> process would get input from those buffers on disk and
> free the files as they are read.  If the writer ran out of
> space, then sleeping and retrying the operation would make
> since, as it would be expected that the reader would be
> freeing blocks on disk as it read them.  It's not always
> a safe assumption, but what else can it do?

This doesn't have anything to do with the issue being discussed.

> 
>>   | Using a file doesn't sequence -- the writer can still continue
>>   | execution pass the point of bash possibly flagging an internal
>>   | error for a non-existent tmp file (writable media) and the
>>   | reader won't get that the "pipe" (file) had no successful writer,
>>   | but instead get an EOF indication and continue, not knowing that
>>   | a fatal error had just occurred.
>>
>> I doubt that is what happens.
>>   
> ----
>     That is what appeared to happen in the post mentioned by Chet.
> The boot process got a /dev/df/99 not found and continued on
> seemingly as though though there had been no input.

I'm not sure what this means. Failing to open a file isn't a fatal script
error.

>>   | However, that would
>>   | be code in the pipe implementation or an IO library on top
>>   | of some StdIO implementation using such.
>>
>> Pipes are implemented in the kernel - userland does nothing different
>> at all (except the way they are created.)
>>   
> ----
>     They usually are.  That doesn't prevent a stdlib implementation
> putting a wrapper around some "non-compliant" kernel call
> to implement a different 'view' to the users of that lib.

If we're going to go off into hypotheticals and speculation, it would be
nice if memfd_create were available universally.

> 
>>   | W/pipes, there is the race condition of the reader not being able
>>   | to read in the condition where the writer has already gone away.
>>
>> Huh?   That's nonsense.   It is perfectly normal for a reader
>> to read long after the writer has finished and exited.   Try this
>>
>>      printf %s\\n hello | { sleep 5; cat; }
>>   
> ===
>     It may be normal in some cases, but:
> 
> https://superuser.com/questions/554855/how-can-i-fix-a-broken-pipe-error

That case is exactly the opposite of the one being discussed here: it's a
write on a pipe when the reader has exited.

>> You are still missing Chet's point.   There is no "< <()" operator.
>> That is two bash syntax elements being combined.  "<" (redirect stdin)
>> and "<()" (create a name to refer to the output from the command).
>>   
> ----
>     I've never seen <() used without '<', so I thought it was
> part of the syntax '< <()'.  

You can't claim ignorance here. You've been told many times that these
are two separate elements.



>     We are talking tradeoffs of using pipes to communicate
> heredocs vs. using a temporary file (presumably in/on /tmp), no?
> The statement reflected my thinking about how, currently,
> the entire contents of the pipe is being "spilled to disk"
> (spilled in the sense of their being insufficient room in
> memory -- or, in this case of there being no 'in-memory'
> implementation at all).

This makes no sense the way you wrote it. If you mean the current
implementation of using temporary files for here documents, you're
correct.


>>   | If bash uses /tmp, it can have a pipe of size 4.7G.  If
>>   | it uses memory, it would have pipe of 79G.
>>
>> That's gibberish.
>>   
> Oh please, its not that obtuse.  If bash currently writes the
> entire contents of "whatever" it is (the here doc), to a temporary
> file, then it is limited by the space on the temporary file system.

If you want to make that point, use `file' instead of `pipe'. What you
wrote doesn't make any sense otherwise. The word `pipe' has a specific
meaning in this context.


> 
>     But the implementation of process substitution in bash
> isn't implemented that way in the currently released version.  It
> uses a tmp file on a disk of fixed size to store *all* of the output
> of the 'writer' before the reader is called.

This is absolutely not true. If you believe this, it might be a reason
you have made incorrect assumptions about other things. Process
substitution uses pipes: anonymous pipes exposed through /dev/fd or FIFOs.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
                 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/