bug#67331: 30.0.50; New Todo mode feature: changing item date style

[Stephen Berman]

This post is a followup to bug#66395, where I posted (and subsequently
installed) a patch to support the ISO date style in Todo mode item
headers.  As I pointed out in that bug thread, despite supporting
different date styles, Todo mode does not support interactively and
automatically changing between different date styles.  I have attempted
to add this functionality to Todo mode, and the attached patch contains
my work-in-progress implementation.  I'm still testing these changes, so
even if they are deemed acceptable I'm not yet ready to install them,
but they raise some questions and how I proceed will be influenced and
perhaps decided by the answers to these questions.

The rest of this post is directed mainly to the Emacs maintainers and I
apologize to them for its length, but unless they want to leave it
entirely up to me how to proceed with these changes, I think they need
to know in at least some detail what the issues are.  As an aid to
contextualizing these issues without having to carefully examine the
code changes, here is a ChangeLog-style summary of the changes:

   New Todo mode feature: changing item date format

   This can be done both interactively by invoking a function to set
   the new date style, and also automatically when switching from a
   todo-mode buffer that uses a given date style to another one that
   uses a different one.

   To facilitate this functionality the internal todo-mode metadata
   sexp has been augmented to include the current setting of the date
   style in addition to the existing list of todo categories and
   their items counts.  When a todo file containing the previous
   categories-only sexp is visited, the sexp is automatically changed
   to the new file metadata format.

   * lisp/calendar/calendar.el
   (calendar-after-set-date-style-function): New variable.
   (calendar-set-date-style): Call it when non-nil.

   * lisp/calendar/todo-mode.el (todo--file-metadata): New variable.
   (todo--date-pattern-groups): This is now a function instead of a
   defconst.  Use it...
   (todo--make-date-pattern): ...in this new function.  Use it to get
   the value of...
   (todo-date-pattern): ...this variable, which is now a defvar
   instead of a defconst.
   (todo--set-file-metadata, todo--get-file-metadata)
   (todo--update-file-metadata): New functions to set, access and
   update todo file metadata.
   (todo-show): Use `todo--get-file-metadata' to test for a new todo file.
   (todo-add-file): Improve doc string.  Insert file metadata sexp.
   (todo-add-category, todo-rename-category): Update file metadata by
   calling `todo--update-file-metadata'.
   (todo-move-category): Insert file metadata sexp if moved to file
   is new.  Simplify updating todo categories metadata in moved to file.
   (todo-edit-item--header): Use function `todo--date-pattern-groups'.
   Improve a comment.
   (todo-set-categories, todo-update-categories-sexp)
   (todo-check-format): Adjust to new file metadata format.
   (todo--maybe-update-date-style, todo--change-date-header-form):
   New functions to determine whether to alter the format of todo
   item date headers, and if so, to automatically do it.  The latter
   function is added as the overriding value of the new variable
   `calendar-after-set-date-style-function' to enable interactively
   changing the date header format.
   (todo-category-completions): Adjust to new file metadata format
   and remove test for new todo file, since that is now done...
   (todo-read-category): ...here, using `todo--get-file-metadata'.
   (todo-key-bindings-t+a+f): Add binding for `calendar-set-date-style'
   to facilitate interactively changing the date header format.
   (todo-modes-set-3): Make `todo--file-metadata' buffer-local and
   evaluate it.  Call `todo--maybe-update-date-style' both directly
   and as a hook on `window-buffer-change-functions', to adjust item
   date headers if necessary.  The former is used when switching to
   another todo-mode buffer as a result of invoking a relevant
   todo-mode operation, the latter when switching from outside of
   todo-mode (e.g., when burying a todo-mode buffer makes another
   todo-mode buffer current).

Here are the main issues these changes raise that I think should be
resolved before proceeding:

- Although the use and effect of this feature is confined to
  todo-mode.el, it depends on a making a change to calendar.el, as you
  can see in the above summary and in the patch.  The reason, as I
  already noted in bug#66395, is that Todo mode date styles are
  specified by `calendar-date-display-form', which you can change by
  customizing `calendar-date-style' or by executing
  `calendar-set-date-style'.  To keep this connection I added a hook to
  calendar.el callable from the latter function, that can be used to
  trigger the date header changes in Todo mode (but it's not a hook in
  the usual sense of being intended for user customization; rather, it's
  intended for use by packages).  There is precedence for this in commit
  a8f4bb8361, where I similarly added a hook to diary-lib.el for the
  benefit of todo-mode.el.  In addition, since the hook has the value
  nil by default, there is no change to the default behavior of the
  Emacs Calendar (or Diary).  So, is this change to calendar.el
  acceptable?  (In fact, diary-lib.el could probably also use this
  change in calendar.el to implement date-style switching in the Emacs
  Diary.)

- My implementation of this feature crucially involves a change in the
  internal form of todo files.  Till now the first line of each todo
  file has contained a sexp listing the categories in the file (this
  line is hidden in todo-mode), but to allow switching date styles I
  have augmented this sexp to also contain a specification of the
  current date style used by the item headers in the file.  A number of
  existing todo-mode functions make use of this sexp, so they have also
  had to be adapted.  All these changes are strictly internal, so the
  Todo mode UI remains unchanged.  The implementation also includes
  automatically converting existing todo files to the augmented metadata
  format, so that users of Todo mode can continue using existing todo
  files, as well as the new functionality, without needing to be aware
  of the internal changes.  But once converted, the todo files will not
  be compatible with previous versions of todo-mode.el.  This would be a
  problem if someone wanted to use the same todo files on different
  systems, not all of which have an Emacs where todo-mode.el has these
  changes.

  Is this problem considered unacceptable?  If so, the easiest solution
  I can think of is to add a function that would detect if the feature
  is available and if not, automatically convert todo files with the new
  metadata format to the previous format, i.e., undoing the automatic
  change the current implementation makes.  I haven't implemented such
  an undoing function yet, because it wouldn't be needed if the problem
  is not considered unacceptable.  (This kind of issue came up when I
  announced my rewrite of the original version of todo-mode.el years ago
  on emacs-devel, and at that time it was decided against keeping the
  older format available; but that rewrite involved significant
  incompatible UI changes, whereas the present extension of UI
  functionality makes only internal incompatible changes.)

  Another possible solution, if the problem is not acceptable, is to
  make it possible to install both the new and a previous version of
  todo-mode.el in parallel.  AFAICS this would require making
  todo-mode.el an ELPA package.  This may be a worthwhile alternative
  regardless of the question of backwards compatibility, since
  todo-mode.el is by no means an indispensable part of Emacs.  (In fact,
  I thought about this alternative before my rewrite of todo-mode.el was
  added to Emacs, but for me keeping todo-mode.el in Emacs proper has
  been the path of least resistance.)

- A related issue concerns the scope of the date-style switching
  operation.  As implemented in the attached patch, date headers are
  changed only in the currently visited todo file, either interactively
  by invoking a command that changes the date style and then each item
  date header, or automatically when switching to a todo file whose
  metadata specifies a different date style from the current one.  No
  changes are made to non-visited todo files.  I did it this way because
  it struck me as wasteful to change files even if the change is unseen
  (and the date style and item headers could be repeatedly changed in a
  given file or selection of files, without visiting the majority of
  todo files), and in my testing, both interactive and automatic
  switching seemed practically instantaneous, even with my largest todo
  file with well over a thousand items.

  On the other hand, in the current implementation the automatic change
  of the metadata sexp to the new format is also only on demand, i.e.,
  on visiting a todo file.  But once that change is made, it is
  permanent, so that on revisiting the file it then already has the new
  format.  Because of this it might seem more efficient to convert all
  todo files once and for all.  I haven't done that for two reasons: one
  is that the change it practically imperceptible, so doing it on demand
  should cause no annoyance.  And secondly, if it is decided to enable
  reverting to the old format for use with older versions of
  todo-mode.el, it will of course then be necessary to change back to
  the new format again on revisiting a file whose format has been
  reverted with the new version of todo-mode.el, so if all files have
  been changed, that might make a perceptible and possibly annoying
  difference (though I haven't made any measurements).

  So is it acceptable to change the metadata format and the item header
  for each todo file on demand instead of for all todo files at once?

- Another thorny issue is how to deal with unit tests.  Currently, there
  are a number of tests for todo-mode.el using ERT.  Since they all
  require a todo file as input data and many make use of the file
  metadata (currently in the old format), at least these tests will
  require adjusting.  So should there be parallel tests for todo-mode
  with the new and with the previous metadata format?  Or should the
  older format be declared obsolete?  (However, AFAIK there is no formal
  obsoletion mechanism for data formats as opposed to functions and
  variables.)  I think the decision about this is contingent on the
  decision about allowing reverting the format for backwards
  compatibility.

- The Tode mode manual will of course need to be updated to document the
  new feature.  Currently, the manual does not document the internal
  structure of todo files, including the metadata sexp (it is only
  mentioned in passing in a couple of places).  If reverting to the
  older metadata format is made an option, this would seem to invite
  giving at least some details of the incompatible metadata formats, but
  I'm not sure it's necessary.

Finally, I want to point out an issue that I think is mainly orthogonal
to the new feature and the preceding considerations, but it helps
clarify the extent of the feature.  Currently, Todo mode supports item
date headers with either the American, European or ISO date style, but
all items can only be displayed with the same style, and that
restriction will persist if the changes discussed here are installed:
then you can switch date styles, but doing so changes all item headers
in a visited todo file.  In other words, there can be no mixing of date
styles within a todo file, or when switching between todo files.  This
restriction is justified by avoiding ambiguity if dates are displayed
simultaneously in the European and American styles with numbers for
months.  Admittedly, `calendar-date-display-form' uses month names by
default in these styles, which avoids ambiguity (and there's no
ambiguity between the ISO style and the other styles, unless years are
displayed with just the two least significant digits, which is common in
the American and European styles but AFAIK the ISO standard requires
four digits for years).  Hence, at least for such display forms, mixing
date display styles should be possible (though whether it's desirable is
questionable).  I haven't given serious thought to how to implement
that, nor to how it would interact with the proposed feature of
switching date styles without mixing, but it would likely require
changes to the implementation in the attached patch.

Steve Berman

txtBoDJ6wls9v.txt
Description: Todo mode feature patch