bug#55737: [PATCH] 28.1; Values of `read-process-output-max' above 64k not taken into account on GNU/Linux

[Frédéric Giquel]

Hi,

While trying to understand the reason for slow completion with lsp-mode
in some situations, I noticed that the language server response is
split into 64k chunks despite setting `read-process-output-max' to 1M
(as recommended by lsp-mode maintainers).

The problem is not lsp-mode specific and can be reproduced without
additionnal packages by using the two scripts attached to this email:
- "write-server.py" is the server: it waits for a number
(requested_size) on its input and write a string of "requested_size"
bytes on its output. The file must be executable and located in the
PATH.
- "real-pipe-size.el" launch "write-server.py" and communicate with the
process using different values for `read-process-output-max'. It prints
the maximum chunk size used for each value.

On my system, I got the following output executing 'emacs -Q --script
~/Projets/emacs-pipe/real-pipe-size.el':
Real pipe size (with read-process-output-max = 4096): 4096
Real pipe size (with read-process-output-max = 32768): 32768
Real pipe size (with read-process-output-max = 65536): 65536
Real pipe size (with read-process-output-max = 65537): 65536
Real pipe size (with read-process-output-max = 204800): 65536
Real pipe size (with read-process-output-max = 1048576): 65536
Real pipe size (with read-process-output-max = 2097152): 65536

This 64k limit is GNU/Linux specific and is explained in "Pipe
capacity" section of pipe(7) manpage and can be changed calling
`fcntl'. So I added a call to `fcntl' for `F_SETPIPE_SZ' in
`create_process' function just after other uses of `fcntl' (I'm not
really sure if it's the right place). The patch is attached to this
email.

With the patch, the limit is now the value of `/proc/sys/fs/pipe-max-
size' (which default to 1M and can be change by users with the
CAP_SYS_RESOURCE capability). Output of 'emacs -Q --script
~/Projets/emacs-pipe/real-pipe-size.el' is now:
Real pipe size (with read-process-output-max = 4096): 4096
Real pipe size (with read-process-output-max = 32768): 32768
Real pipe size (with read-process-output-max = 65536): 65536
Real pipe size (with read-process-output-max = 65537): 65537
Real pipe size (with read-process-output-max = 204800): 204800
Real pipe size (with read-process-output-max = 1048576): 1048576
Real pipe size (with read-process-output-max = 2097152): 65536

And my initial problem of slow completion with lsp-mode is fixed.

P.S. I signed the FSF copyright assignment agreement for Emacs last
year.

In GNU Emacs 28.1 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.33,
cairo version 1.16.0)
 of 2022-05-30 built on guiriden
Repository revision: 5a223c7f2ef4c31abbd46367b6ea83cd19d30aa7
Repository branch: heads/emacs-28.1
Windowing system distributor 'The X.Org Foundation', version
11.0.12201000
System Description: Debian GNU/Linux bookworm/sid

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES NOTIFY
INOTIFY PDUMPER PNG RSVG SECCOMP SOUND THREADS TIFF TOOLKIT_SCROLL_BARS
X11 XDBE XIM XPM GTK3 ZLIB

Important settings:
  value of $LANG: fr_FR.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny dired dired-loaddefs
rfc822 mml mml-sec epa derived epg rfc6068 epg-config gnus-util rmail
rmail-loaddefs auth-source cl-seq eieio eieio-core cl-macs
eieio-loaddefs password-cache json map text-property-search time-date
subr-x seq byte-opt gv bytecomp byte-compile cconv mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader cl-
loaddefs
cl-lib sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-
utils
iso-transl tooltip eldoc paren electric uniquify ediff-hook vc-hooks
lisp-float-type elisp-mode mwheel term/x-win x-win term/common-win x-
dnd
tool-bar dnd fontset image regexp-opt fringe tabulated-list replace
newcomment text-mode lisp-mode prog-mode register page tab-bar menu-bar
rfn-eshadow isearch easymenu timer select scroll-bar mouse jit-lock
font-lock syntax font-core term/tty-colors frame minibuffer cl-generic
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite emoji-zwj
charscript
charprop case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice button loaddefs faces cus-face macroexp files
window text-properties overlay sha1 md5 base64 format env code-pages
mule custom widget hashtable-print-readable backquote threads dbusbind
inotify lcms2 dynamic-setting system-font-setting font-render-setting
cairo move-toolbar gtk x-toolkit x multi-tty make-network-process
emacs)

Memory information:
((conses 16 50603 6602)
 (symbols 48 6632 1)
 (strings 32 18327 1513)
 (string-bytes 1 607344)
 (vectors 16 13715)
 (vector-slots 8 184638 11282)
 (floats 8 21 47)
 (intervals 56 195 0)
 (buffers 992 10))

write-server.py
Description: Text Data

real-pipe-size.el
Description: Text Data

pipe-size.patch
Description: Text Data