bug#58837: 27.1; Gnus: nnfolder-generate-active-file fails if folder layout not exactly correct

[Henrik Ahlgren]

I am trying to migrate a fairly large archive of email (spanning over 30
years, saved from numerous different email programs, lately Evolution)
stored as traditional mbox type files, to a directory to be accessed
from Gnus using the nnfolder method.

One would assume this is a trivial (and rather common) task, but I have
already spent hours trying to get it working, and getting bit
frustrated.

So, if I am using just the default configuration, with just (nnfolder
"") added to gnus-secondary-select-methods, and copy a few files to
~Mail/archive, quit Gnus, and run nnfolder-generate-active-file, it does
seem to work as expected and finds the new files.

However, if I create a subdirectory under ~Mail/archive and copy some
more files there, nnfolder-generate-active-file fails with:

   nnmail-find-file: Wrong type argument: stringp, nil

If I remove the offending file (and subdirectory) and attempt to
generate-active-file again, it still fails even when ~Mail/archive is in
the same state as previously when it worked, so apparently this leaves
Gnus in some state that requires Emacs to be restarted? How to recover?

The Gnus manual section "6.4.13.10 Mail Folders" has this to say:

     If you have lots of ‘nnfolder’-like files you’d like to read with
  ‘nnfolder’, you can use the ‘M-x nnfolder-generate-active-file’ command
  to make ‘nnfolder’ aware of all likely files in ‘nnfolder-directory’.
  This only works if you use long file names, though.

The obvious question is: How long exactly do the file names have to be,
are my names too short? :-) I guess "long names" refer to newsgroup
style names with dots (something like "sports.bicycling"), but it is
unclear should I flatten the folder structure by replacing / with . or
what is the correct way.

Furthermore, for someone younger reading this documentation today, it is
likely that she/he is only interested in email and probably is not even
familiar with the concept of newsgroups at all (Usenet news are
unfortunately mostly dead nowadays), so "this only work if you use long
file names" must be very confusing!

Clearly this part of documentation should be improved so that anyone
with reasonable knowledge should understand it. The documentation should
clearly state how should the directory layout look like, and it would be
useful to describe how exactly one is supposed to import old email
archives here, maybe with some examples. In general I feel that Gnus
documentation should have more focus on the email use case in modern
times.

Pablo


In GNU Emacs 27.1 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.24, cairo 
version 1.16.0)
 of 2021-03-28, modified by Debian built on x86-conova-01
Windowing system distributor 'The X.Org Foundation', version 11.0.12011000
System Description: Debian GNU/Linux 11 (bullseye)


Configured using:
 'configure --build x86_64-linux-gnu --prefix=/usr
 --sharedstatedir=/var/lib --libexecdir=/usr/lib
 --localstatedir=/var/lib --infodir=/usr/share/info
 --mandir=/usr/share/man --enable-libsystemd --with-pop=yes
 
--enable-locallisppath=/etc/emacs:/usr/local/share/emacs/27.1/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/27.1/site-lisp:/usr/share/emacs/site-lisp
 --with-sound=alsa --without-gconf --with-mailutils --build
 x86_64-linux-gnu --prefix=/usr --sharedstatedir=/var/lib
 --libexecdir=/usr/lib --localstatedir=/var/lib
 --infodir=/usr/share/info --mandir=/usr/share/man --enable-libsystemd
 --with-pop=yes
 
--enable-locallisppath=/etc/emacs:/usr/local/share/emacs/27.1/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/27.1/site-lisp:/usr/share/emacs/site-lisp
 --with-sound=alsa --without-gconf --with-mailutils --with-cairo
 --with-x=yes --with-x-toolkit=gtk3 --with-toolkit-scroll-bars
 'CFLAGS=-g -O2
 -ffile-prefix-map=/build/emacs-LlFm6W/emacs-27.1+1=. -fstack-protector-strong
 -Wformat -Werror=format-security -Wall' 'CPPFLAGS=-Wdate-time
 -D_FORTIFY_SOURCE=2' LDFLAGS=-Wl,-z,relro'

Configured features:
XPM JPEG TIFF GIF PNG RSVG CAIRO SOUND GPM DBUS GSETTINGS GLIB NOTIFY
INOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE HARFBUZZ M17N_FLT LIBOTF
ZLIB TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM MODULES THREADS LIBSYSTEMD
JSON PDUMPER LCMS2 GMP

Important settings:
  value of $LC_MESSAGES: C
  value of $LC_NUMERIC: C
  value of $LC_TIME: en_DK.utf8
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Messages

Minor modes in effect:
  semantic-minor-modes-format: ((:eval (if (or semantic-highlight-edits-mode 
semantic-show-unmatched-syntax-mode semantic-idle-scheduler-mode)  S)))
  hexl-follow-ascii: t
  global-git-commit-mode: t
  magit-auto-revert-mode: t
  shell-dirtrack-mode: t
  ido-everywhere: t
  show-paren-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
/usr/share/emacs/site-lisp/elpa/apache-mode-2.2.0/apache-mode hides 
/usr/share/emacs/site-lisp/elpa-src/apache-mode-2.2.0/apache-mode
/usr/share/emacs/site-lisp/elpa/apache-mode-2.2.0/apache-mode-pkg hides 
/usr/share/emacs/site-lisp/elpa-src/apache-mode-2.2.0/apache-mode-pkg
/usr/share/emacs/site-lisp/elpa/apache-mode-2.2.0/apache-mode-autoloads hides 
/usr/share/emacs/site-lisp/elpa-src/apache-mode-2.2.0/apache-mode-autoloads

Features:
(shadow emacsbug pcase edebug mhtml-mode tramp-ftp finder lisp-mnt
apropos debug backtrace gnus-eform deuglify gnus-cus gnus-demon
gnus-diary nndiary gnus-draft gnus-html gnus-kill gnus-logic gnus-mh
mh-comp mh-scan mh-gnus mh-e mh-compat mh-buffers mh-loaddefs
gnus-registry registry gnus-salt gnus-topic gnus-uu yenc gnus-vm nndraft
nnmh utf-7 nnfolder nnnil reveal ibuf-ext ibuffer ibuffer-loaddefs
magit-bundle magit-gitignore magit-patch magit-subtree mailalias
smtpmail sendmail gnus-fun flow-fill smiley gnus-cite mail-extr qp
gnus-async gnus-bcklg gnus-agent gnus-srvr gnus-score score-mode
nnvirtual nntp gnus-ml gnus-msg disp-table nndoc gnus-cache gnus-dup
mm-archive debbugs-gnu debbugs-compat debbugs semantic/bovine/grammar
semantic/bovine semantic/grammar semantic/idle semantic/analyze
semantic/sort semantic/scope semantic/analyze/fcn semantic/grammar-wy
semantic/format semantic/tag-ls semantic/ctxt semantic/wisent
semantic/wisent/wisent yaml-mode generic tramp-archive holidays
hol-loaddefs man locate bib-mode battery w32-vars ebnf2ps ps-print
ps-print-loaddefs ps-def lpr dos-vars url-tramp url-nfs url-file
url-dired url-irc webjump tramp-smb tramp-rclone tramp-gvfs zeroconf
dbus tramp-adb ntlm calc calc-loaddefs calc-macs hmac-md5 hex-util md4
net-utils ldap kermit gravatar goto-addr eudc-vars ange-ftp vc
vc-dispatcher srecode soap-client rng-xsd xsd-regexp pcvs-defs grep
glasses gdb-mi bindat gud flymake-proc flymake warnings etags fileloop
xref project emerge elide-head ediff ediff-merg ediff-mult ediff-wind
ediff-diff ediff-help ediff-init ediff-util ede/project-am
ede/autoconf-edit autoconf autoconf-mode semantic/find ede/makefile-edit
make-mode ede/linux semantic/db semantic/util-modes semantic/util
semantic semantic/tag semantic/lex semantic/fw mode-local ede/make
ede/speedbar ede/files ede ede/detect ede/base ede/auto ede/source
eieio-base eieio-speedbar eieio-custom cedet ebrowse ebuff-menu
copyright compare-w check-declare calculator vhdl-mode hippie-exp
verilog-mode compile vera-mode tcl sql view simula sieve sieve-mode
sieve-manage sasl sasl-anonymous sasl-login sasl-plain sh-script
executable ruby-mode python ps-mode prolog align perl-mode pascal
opascal octave rng-nxml rng-valid rng-loc rng-uri rng-parse nxml-parse
rng-match rng-dt rng-util rng-pttrn nxml-ns nxml-mode nxml-outln
nxml-rap nxml-util nxml-enc xmltok modula2 meta-mode m4-mode ld-script
js info-look idlwave idlwave-help idlw-help icon hideshow fortran f90
dcl-mode css-mode smie sgml-mode color cperl-mode cfengine cfengine3
bat-mode asm-mode antlr-mode cc-mode cc-fonts cc-guess cc-menus cc-cmds
cc-styles cc-align cc-engine cc-vars cc-defs timeclock time-stamp
tar-mode sort snmp-mode tempo saveplace remember hexl generic-x forms
forms-mode dns-mode conf-mode arc-mode archive-mode gnus-art mm-uu
mml2015 mm-view mml-smime smime dig image-converter iimage thumbs
image-file mpc image-dired cl-print doctor novice eieio-opt speedbar
sb-image ezimage dframe tramp-cmds tramp-cache tramp-sh tramp
tramp-loaddefs trampver tramp-integration files-x tramp-compat ls-lisp
help-fns radix-tree gnutls network-stream url-http url-gw nsm url-cache
url-auth eww mm-url url-queue dired-aux tabify cal-iso mule-util
cal-move scroll-lock misearch multi-isearch ledger-mode ledger-check
ledger-texi ledger-test ledger-sort ledger-report ledger-reconcile
ledger-occur ledger-fonts ledger-fontify ledger-state ledger-complete
ledger-schedule ledger-init ledger-xact ledger-post ledger-exec
ledger-navigate eshell esh-cmd esh-ext esh-opt esh-proc esh-io
esh-module esh-groups ledger-context ledger-commodities esh-arg esh-util
ledger-regex org-indent org-element avl-tree generator ol-eww ol-rmail
ol-mhe ol-irc ol-info ol-gnus nnir gnus-sum url url-proxy url-privacy
url-expand url-methods url-history mailcap shr url-cookie url-domsuf
url-util svg xml dom gnus-group gnus-undo gnus-start gnus-cloud nnimap
nnmail mail-source utf7 netrc nnoo parse-time iso8601 gnus-spec gnus-int
gnus-range gnus-win gnus nnheader ol-docview ol-bibtex bibtex ol-bbdb
ol-w3m org ob ob-tangle ob-ref ob-lob ob-table ob-exp org-macro
org-footnote org-src ob-comint org-pcomplete org-list org-faces
org-entities noutline outline org-version ob-emacs-lisp ob-core ob-eval
org-table ol org-keys org-compat advice org-macs org-loaddefs find-func
suomalainen-kalenteri cal-menu calendar cal-loaddefs bug-reference
thingatpt vc-git doc-view jka-compr image-mode exif magit-extras
face-remap magit-submodule magit-obsolete magit-blame magit-stash
magit-reflog magit-bisect magit-push magit-pull magit-fetch magit-clone
magit-remote magit-commit magit-sequence magit-notes magit-worktree
magit-tag magit-merge magit-branch magit-reset magit-files magit-refs
magit-status magit magit-repos magit-apply magit-wip magit-log
which-func imenu magit-diff smerge-mode diff diff-mode git-commit
log-edit easy-mmode message rmc puny rfc822 mml mml-sec epa derived epg
epg-config gnus-util rmail rmail-loaddefs text-property-search time-date
mm-decode mm-bodies mm-encode mail-parse rfc2231 rfc2047 rfc2045 mm-util
ietf-drums mail-prsvr mailabbrev mail-utils gmm-utils mailheader
pcvs-util add-log magit-core magit-autorevert autorevert filenotify
magit-margin magit-transient magit-process with-editor shell pcomplete
comint ring ansi-color magit-mode transient cl-extra help-mode
format-spec magit-git magit-section magit-utils crm dash pp cus-edit
cus-start cus-load wid-edit server finder-inf ido dired-x dired
dired-loaddefs tango-dark-theme edmacro kmacro avoid paren rx info
package easymenu browse-url url-handlers url-parse auth-source cl-seq
eieio eieio-core cl-macs eieio-loaddefs password-cache json subr-x map
url-vars seq byte-opt gv bytecomp byte-compile cconv cl-loaddefs cl-lib
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode elisp-mode
lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow isearch
timer select scroll-bar mouse jit-lock font-lock syntax facemenu
font-core term/tty-colors frame minibuffer cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote threads dbusbind
inotify lcms2 dynamic-setting system-font-setting font-render-setting
cairo move-toolbar gtk x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 1657405 1422245)
 (symbols 48 65645 28)
 (strings 32 374934 308616)
 (string-bytes 1 11612472)
 (vectors 16 120812)
 (vector-slots 8 2294622 470364)
 (floats 8 10334 2822)
 (intervals 56 97887 32148)
 (buffers 1000 375))