bug#46326: 27.1.50; Excessive memory allocations with minibuffer-with-setup-hook

[mail]

Dear Emacs developers,

I have an issue on 27.1.50 with excessive memory allocations when using
minibuffer-with-setup-hook with large closures and :append. This issue 
has been
observed in my Consult package, which is similar to Counsel and which 
provides
commands based on completing-read. Naturally I have to perform some 
setup in the
minibuffer-setup-hook. Most of the memory allocations are due to 
add-hook and in
particular the sort function. The sort functions is called because of 
:append
which sets a hook priority. The reason seems to be that large closures 
are
copied, but I didn't fully investigate the reasons for the issue.

Example profile:

    140,068,687  92%       - consult-buffer
    140,068,687  92%        - consult--buffer
    140,068,687  92%         - consult--multi
    138,933,527  91%          - consult--read
    138,933,527  91%           - apply
    138,933,527  91%            - #<subr consult--read>
     99,916,347  65%             - consult--with-async-1
     99,916,347  65%              - #<compiled 0x3e02774d5b1a3f7>
     99,912,203  65%               - consult--with-preview-1
     80,735,735  53%                - #<compiled -0xbfb98ad3a8adf1c>
     80,735,735  53%                 - completing-read
     80,735,735  53%                  - selectrum-completing-read
     62,909,765  41%                   - selectrum-read
> > >  58,826,659  38%                    + add-hook
      1,077,206   0%                    + 
selectrum--minibuffer-post-command-hook
      ....
> > >  19,176,468  12%                + add-hook
> > >  39,017,180  25%             + add-hook

The issue can be mitigated by using a modified version of
minibuffer-with-setup-hook, where I am creating a symbol and fsetting 
instead of
adding a lambda directly via add-hook. This technique is also used in
set-transient-map in order to avoid problems with letrec-closures and
add-hook/remove-hook.

    (defmacro consult--minibuffer-with-setup-hook (fun &rest body)
      (declare (indent 1) (debug t))
      (let ((hook (make-symbol "hook"))
            (append))
        (when (eq (car-safe fun) :append)
          (setq append '(t) fun (cadr fun)))
        `(let ((,hook (make-symbol "consult--minibuffer-setup")))
           (fset ,hook (lambda ()
                         (remove-hook 'minibuffer-setup-hook ,hook)
                         (funcall ,fun)))
           (unwind-protect
               (progn
                 (add-hook 'minibuffer-setup-hook ,hook ,@append)
                 ,@body)
             (remove-hook 'minibuffer-setup-hook ,hook)))))

See the issue https://github.com/minad/consult/issues/193 for reference.

Maybe there exists an alternative solution to the problem and the 
problem should
be fixed on a different level than by replacing the 
minibuffer-with-setup-hook
macro?

Daniel Mendler

In GNU Emacs 27.1.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.5, 
cairo version 1.16.0) of 2020-10-24
Repository revision: c847d5998f588dbf3eca5ea1ec573a2d64a97607
Repository branch: emacs-27
Windowing system distributor 'The X.Org Foundation', version 
11.0.12004000
System Description: Debian GNU/Linux 10 (buster)