[O] Citations, continued

[Richard Lawrence]

Hi all,

I wanted to continue the discussion that began in this thread about
adding citation support to Org:
http://thread.gmane.org/gmane.emacs.orgmode/94352/focus=94412

Here are some thoughts I have after reviewing that discussion:

1) Lots of people seem to need/want better support for citations in Org
documents.  Other projects (Pandoc among them) already have support for
citations which is good enough that at least some people are using them
to fill this perceived gap in Org's support.

2) There are at least several different backend reference database
formats (BibTeX, Zotero, etc.) used by Orgsters.  Not all such databases
use human-readable keys.  Org also has a nice internal format for
storing reference information: org-bibtex.

3) There are also several different frontend solutions for making
citations in Org documents (org-ref, various `home-brewed' solutions,
raw LaTeX \cite commands, etc.).  The variety here is at least in part
due to our different requirements for the format of exported documents:
e.g., some people only care about exporting to LaTeX, others need
something that will work for HTML or ODT; some people need to specify
pre- and post-text for citations; some people need to specify citation
types.

4) Because individual Orgsters have widely varying needs, there is some
disagreement about what `proper' citation support should look like.  (Do
we need new syntax, or can existing syntax be used?  Which features need
to be supported by Org, and which can be provided by external tools?
etc.)

It seems there are three distinct but related problems:
  - representing citations in Org documents
  - exporting citations in an Org document to a backend document format
    like LaTeX, ODT, or HTML
  - searching for and manipulating keys in a reference database from
    within Org, and otherwise fostering good communication between
    such a database and citations in Org documents

Here's my personal opinion about how we might solve them.

As for the first problem, I think a good case can be made for adding new
syntax to Org to represent citations, instead of repurposing/extending
existing syntax (most notably, the link syntax).  

Here's why.  Citations are complicated, and it's clear that some sort of
new syntax is needed to represent them.  Even link-based solutions
introduce new syntax `inside' the link syntax, such as using `::' to
separate pre-text and post-text in a link description, as org-ref does.
Thus, the issue is not *whether* there should be additional syntax, but
just *how constrained* it should be.  In particular, the question is
whether we want to make citation syntax a subset of link syntax, or
whether citations and links should be distinct types of syntactic
elements.

It seems to me that the needs of citation users are wide enough and
complicated enough that it is worth shedding the constraints imposed by
the link syntax.  Eventually, packing all the representations we need
for a general solution (citation type, pre- and post-text, suppressing
author name, etc. etc.) into the link syntax will tend to make citations
unreadable.

Moreover, citations are not really links, even though it is often useful
to treat them (or parts of them) as links.  For example, a link can only
point to one target; yet a single citation often points to multiple
sources.  And what if you need different pre- or post-text for different
sources within the same citation, like ``See @Doe1999 for an overview; a
more extensive discussion is in @Foobar2000''?

Finally, it seems to me that Org strives to be a document format which
is usable independently of any particular export format.  This counts in
favor of having citation syntax as a first-class citizen in Org; at
present, Org documents can only contain citations through a combination
of extensions and external tools.

For these reasons, I would support a separate citation syntax, but one
that can behave like a link when useful.  For example, suppose we
borrowed the Pandoc [ ... @key1 ...; ... @key2 ...] syntax.  When point
is on address@hidden', C-c C-o could be bound to find the key in the reference
database, or another useful action, depending on the reference database
format.

As I mentioned in the earlier thread, I think the Pandoc syntax is a
good place to start, and I think it would be valuable to have the two
syntaxes be compatible.  But even Pandoc's citation syntax might not be
general enough to satisfy everyone's needs, so the first step for
introducing citation syntax to Org should be compiling a list of all the
things such a syntax should represent.

The other problems, I think, must wait until a stable citation syntax
emerges -- export support in particular.  (Using an existing syntax from
another project could help ease the transition here: if people can
export citations using an existing tool, they'll be able to switch to
that syntax immediately, and use the external tool in the meantime while
Org-internal support for it catches up.)

I hope this is a useful starting point for further discussion!

Best,
Richard