Re: Large source block causes org-mode to be unusable

[Tim Cross]

Tom Gillespie <tgbugs@gmail.com> writes:

>> That said, I think keeping 2000 lines of source code inside an
>> org src block is neither a standard use case nor a reasonable idea.
>
> I would say that it certainly is a standard use case for people who
> want to keep everything in a single file (e.g. to simplify
> reproducibility and avoid the mess of trying to distribute multiple
> files to non-technical users). #+INCLUDE is not a substitute if you
> are going to be tangling files, breaks many workflows, and as a result
> should rarely be recommended as a solution when src blocks are
> involved. Org should definitely be able to handle this case because
> there is no reason why performance should be any worse than having a
> 2000 line file in another buffer.
>

While I agree with the sentiment, I disagree with the conclusion "there
is no reason why performance should be any worse than having a 2000 line
file in another buffer.". Org mode is pushing right up against the
limits of Emacs' architecture with what it is doing. The way modes and
font-locking work in Emacs was never designed to support multiple modes
and multiple font-locking schemes within a single buffer. The fact org
is able to do this is a testament to the power of Emacs, but this comes
at a cost and that cost is primarily with respect to performance. You
can experience similar performance degradation when you use other
'modes' which try to support multiple modes in a single buffer (like
mmm-mode). This is not specific to org mode, but shows up more due to
the larger user base for org.

> Org babel has many basic interactivity performance pitfalls that need
> to be investigated. I personally have many workarounds for bad emacs
> performance degradations related to code executing in the event loop
> because I need to get on with the task at hand, but they need to be
> fixed, not dismissed.

I think the big challenge here is that the 'fixes' needed by org mode
really need to be at a deeper level inside Emac core architecture. It
isn't something which can be effectively addressed at the org level.
This makes it a much bigger and more complex task which is further
hampered by the need to maintain compatibility with all the existing
modes and libraries.

>From what I've seen in the org-devel list, there is on-going work which
is focused on improvements to font-lock efficiency and I think support
for multiple modes is also being considered in some of this work. Other
developments, such as native compilation and pure gtk implementations
may also help in this area. Unfortunately, it may take some time before
we see the benefits of this work as it is complex and non-trivial work. 

In the meantime, we need to consider all possible work-arounds and
tweaks which can help and in some cases, accept some compromise. Having
source blocks which are 2000 lines long is, at this time, a challenge
for org to handle given existing facilities. There are no quick and easy
fixes, but there are some tweaks, such as turning off native
fontification or using include files which can make the system usable.
While not ideal, we have ot work within the limits of the current
architecture while we wait for improvements, which can be expected to be
incremental and slow.  

-- 
Tim Cross