[Orgmode] Re: Having (too) many files in org-agenda-files

[Sébastien Vauban]

Hi Matt,

Matt Lundin wrote:
> Sébastien Vauban writes:

First thing: no prob' for my name. The reply is important, my name much
less... isn't it, Matt (or Mattew? ;-))


>> Of course, I have many, many files in Org mode. All files I write (or
>> touch) in fact.
>>
>> Of course, I would like to search through my files at some point in time. I
>> even would like to search through your files at some point in time, I mean
>> through =org-mode/contrib/babel= and =Worg= for example.
>
> The drawback of org-mode's plain text format is that org-mode needs to load,
> parse, and fontify, and preserve in memory all files it queries (i.e.,
> agenda files). Too many agenda files can thus cause a pretty big performance
> hit (both when loading agenda files and constructing agenda views). To avoid
> this, I would recommend including only important files in org-agenda-files.
> Possible solutions for searching other files include:
>
> 1. loading the other directories when needed (e.g., through a function that
>    changes the value of org-agenda-files)

Good idea.


> 2. using the variable org-agenda-text-search-extra-files (see below)

In fact, more or less the same as the above... and already foreseen. Nice.


> 3. searching other directories with external tools (grep, perl script, etc.)

Not really interested by this. I love some much the powerful and very quick
search capabilities offered by Org.


>> The problem is the load-time of my Emacs, now:
>>
>> Emacs Init startup time: 221 seconds.
>
> Do you invoke an agenda command in your emacs, such as org-agenda-to-appt?
> That would cause org-mode to load all org files.

As seen in my reply to Carsten today, yes, I have an org-agenda-list. I know
that it is looking in every pointed file. The problem is that it really takes
much time... fontifying and so.


>> coming from 20 seconds before the heavy use of Org...
>>
>> with tens of times such lines in my *Messages* buffer:
>>
>> OVERVIEW
>> Checking for library `filladapt'...
>> +-> Requiring `filladapt' (already loaded)
>> Checking for library `filladapt'... Found
>> Fontifying Axa.org... (regexps............................................)
>> Checking for library `filladapt'... Found
>> Fontifying Axa.org... (regexps.............................................)
>> Checking for library `filladapt'... Found
>> Fontifying Axa.org... (regexps..............................................)
>> Checking for library `filladapt'... Found
>> +-> Requiring `outline-mode-easy-bindings' (already loaded)
>> +-> Requiring `ispell' (already loaded)
>> Evaluate code AFTER HAVING LOADED `flyspell'... [2 times]
>> Starting new Ispell process [en_US] ...
>> Checking for library `filladapt'...
>> +-> Requiring `filladapt' (already loaded)
>> Checking for library `filladapt'... Found
>
> It seems that filladapt is causing some of the verbosity (and possible
> slowness) there. When loading agenda files, my *Messages* buffer only shows:
>
> OVERVIEW [29 times]

OK, I have to tell that, in order to debug some problems, I've changed some
basic functions in order to get more verbosity in the Messages buffer. Here
they are:

--8<---------------cut here---------------start------------->8---
      ;; REPLACES ORIGINAL in `C source code' (dumped)
      ;; redefine require to leave a trace of packages being loaded
      (if (not (fboundp 'orig-require))
          (fset 'orig-require (symbol-function 'require))
        (message "The code to redefine `require' should not be loaded twice"))

      (defvar my-require-depth 0)

      (defun require (feature &optional filename noerror)
        "Leave a trace of packages being loaded."
        (cond ((member feature features)
               (message "%sRequiring `%s' (already loaded)"
                        (concat (make-string (* 2 my-require-depth) ? ) "+-> ")
                        feature))
              (t
               (message "%sRequiring `%s'"
                        (concat (make-string (* 2 my-require-depth) ? ) "+-> ")
                        feature)
               (let ((my-require-depth (+ 1 my-require-depth)))
                 (orig-require feature filename noerror))
               (message "%sRequiring `%s'...done"
                        (concat (make-string (* 2 my-require-depth) ? ) "+-> ")
                        feature))))

      (defvar missing-packages-list nil
        "List of packages that `try-require' can't find.")

      ;; attempt to load a feature/library, failing silently
      (defun try-require (feature)
        "Attempt to load a library or module. Return true if the
library given as argument is successfully loaded. If not, instead
of an error, just add the package to a list of missing packages."
        (condition-case err
            ;; protected form
            (progn
              (message "Checking for library `%s'..." feature)
              (if (stringp feature)
                  (load-library feature)
                (require feature))
              (message "Checking for library `%s'... Found" feature))
          ;; error handler
          (file-error  ; condition
           (progn
             (message "Checking for library `%s'... Missing" feature)
             (add-to-list 'missing-packages-list feature 'append))
           nil)))
--8<---------------cut here---------------end--------------->8---


>> You'll tell me: not a problem, you do that only once a day, and you use
>> Emacs client/server for the rest of the time. True. A bit, because I
>> sometimes have to restart Emacs for testing a fresh one (not impacted by
>> defvars already defined, or deffaces, etc.).
>>
>> Having to wait almost 4 minutes is a real pain. So, here my
>> comments/questions:
>>
>> - Isn't it possible to delay the fontification/ispell/etc. to when we
>>   really display (i.e., pop up) the buffer? I guess this must be a major
>>   component of the time this takes.
>
> I imagine the fontification is necessary for the agenda to function
> properly.

This assumption should be checked. Does parsing really mean fontifying?  From
what I understood about jit-lock-mode, I'd say no... but...


> Thus, org files are parsed en masse when the agenda is called for the first
> time. (Hence my question about whether you were calling the agenda from
> ~/.emacs.)
>
>> - Couldn't we have 2 vars: =org-agenda-files= for the files you know you
>>   want have scanned for the agenda construction, and an extra list such as
>>   =org-search-files= for files not containing any dates? Then, some time
>>   would have to be taken when =C-c a s=, but not before. And not if you
>>   don't search for anything in your Org files during that Emacs session...
>
> This already exists. See org-agenda-text-search-extra-files.

Thanks for this, did not know about that (up to today)...

Best regards,
  Seb

-- 
Sébastien Vauban