bug#25549: 25.1; eshell grep gives inconsistent output

[npostavs]

tags 25549 patch
quit

npostavs@users.sourceforge.net writes:
>
> Tijs Mallaerts <tijs.mallaerts@gmail.com> writes:
>
>> On my machine following steps seem to be a reproducible test case:
>>
>> - create a file "test-file.txt" with 20.000 identical lines with content 
>> "This is a line."
>> - open eshell and insert the command: cat test-file.txt | grep line | wc
>>
>> This seems to return a different result every time it's run.
>
> Thanks, I can reproduce with this.

The problem can be more easily reproduced (i.e., more often, and only
3000 lines) by

    cat test-file.txt | sleepy-cat | wc

where sleepy-cat is

    #!/bin/sh
    while read line ; do
        echo "$line"
        sleep 0.000001
    done

The problem happens when one of the commands in the pipeline sends its
output to Emacs quickly and the next command in the pipeline is slower.
On receiving data from the first command in eshell-insertion-filter we
call eshell-output-object to send it to the next command, but since
sending might block, Emacs can run other process filters and sentinels
instead.  In this case, while sending a data chunk from cmd1 to cmd2, we
actually end up reading all the data from cmd1 until it terminates and
we call its sentinel.  The sentinel closes the pipes and sends EOF to
cmd2, but we still haven't sent the data from cmd1 to cmd2 yet.

Closing the pipes in a timer, as in the patch below, seems to fix it for
me.

>From a1757a7114fbc20733554d8aa44cea0fa8a991e2 Mon Sep 17 00:00:00 2001
From: Noam Postavsky <npostavs@gmail.com>
Date: Thu, 2 Feb 2017 09:19:43 -0500
Subject: [PATCH v1] Make sure eshell pipelines don't drop data

* lisp/eshell/esh-proc.el (eshell-sentinel): If called while still
handling output of the process, make sure to close the pipes only later,
so that the next process in the pipeline recieves EOF only after getting
all its input (Bug#25549).
---
 lisp/eshell/esh-proc.el | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)

diff --git a/lisp/eshell/esh-proc.el b/lisp/eshell/esh-proc.el
index b0dbb22..ba5cb5c 100644
--- a/lisp/eshell/esh-proc.el
+++ b/lisp/eshell/esh-proc.el
@@ -393,8 +393,20 @@ eshell-sentinel
                    (unless (string= string "run")
                      (unless (string-match "^\\(finished\\|exited\\)" string)
                        (eshell-insertion-filter proc string))
-                     (eshell-close-handles (process-exit-status proc) 'nil
-                                           (cadr entry))))
+                      (let ((handles (nth 1 entry))
+                            (str (prog1 (nth 3 entry)
+                                   (setf (nth 3 entry) nil)))
+                            (status (process-exit-status proc)))
+                        ;; If we're in the middle of handling output
+                        ;; from this process then schedule the EOF for
+                        ;; later.
+                        (letrec ((finish-io
+                                  (lambda ()
+                                    (if (nth 4 entry)
+                                        (run-at-time 0 nil finish-io)
+                                      (when str (eshell-output-object str nil 
handles))
+                                      (eshell-close-handles status 'nil 
handles)))))
+                          (funcall finish-io)))))
                (eshell-remove-process-entry entry))))
        (eshell-kill-process-function proc string)))))
 
-- 
2.9.3