Re: emacs build command for org-files

[John Kitchin]

briangpowell . <address@hidden> writes:

> "Actually, if it was possible to get M-x compile to run an elisp function
> instead of a make file, it might be all I need, but it looks like it runs
> shell commands."
>
> You probably know this but just in case:
>
> "You can call simulate an EmacsLisp-based script by putting this in a batch
> script:
>
> emacs -batch -l ~/.emacs.editor --eval="(require 'foo)" \
> --eval="(require 'bar)" \
> --eval="(some-function $*)"
> Starting with the Emacs 22, you can write Emacs scripts just as if you were
> writing Bash or Perl scripts, when you include this at the top of your
> script file:
>
> #!/usr/bin/emacs --script'
>
> --I mean, Elisp functions can be called as batch files from shell commands
> called by make makefiles

I have tried this before, but my emacs startup time is pretty slow, so
this takes longer than running a src block, and there is no shared state
between commands. With src blocks, there can be shared state, e.g. one
block computes something another uses.

If anyone is interested in how to use emacs in scripts, see 
http://kitchingroup.cheme.cmu.edu/blog/2014/08/11/Using-org-mode-outside-of-Emacs-sort-of/

It just doesn't make sense to me to have to write an emacs shell script/command
like that, when there is already a src block with just the elisp
commands in them that does just that. In that post, I show how you could
call that block from a shell command, but I don't think that is really
what I want.

What I am doing is more aligned with the org-publish capability,
where you effectively can have targets with specific directions on how
to build each component. You can publish one component, or one file, or
the whole project. It has at least some primitive checking of
whether an org-file is newer than the output file expected. I have to
confess I have struggled to get a project defined that does what I need
here though.

For example, in my proposal project, I have to submit a pdf of the body
with no references in it, and a pdf of just the references. That is done
by two separate exports. One is straightforward, it just uses
\nobibliography to get just the body with all the correct citations, but
no references at the end. That step generates a .bbl file that is used
in the second export, where I remove the body in a copy of the generated
tex file, insert the contents of the bbl file, and then build the pdf
from that. I don't know of a more straightforward way to do this.

>
> But I believe you're looking for something better, Python is great, but, it
> shouldn't be applied to every problem

I am just looking for another way to use src blocks in org-mode to build
documents that require complicated manipulations, powered by elisp. I am
not advocating for or suggesting Python be used for every build problem.

It is a preference to not add more files, e.g. by tangling the src
blocks just so I can use make. Even when I have done this, it is always
triggered by running a src block in an org file that first tangles
everything, and then runs a shell command. That is usually for literate
programming projects though, where the desired output is compiled code.

>
> "Python will be the Emacs Lisp of the 1990's"--Guido Van Rossum, in one of
> his 1st meetings on his new language Python--I was there, I wrote down this
> quote in an Emacs Lisp book
>
> Python is a masterpiece developed at the Stichting Mathematisch Centrum in
> Amsterdam, Holland--during a Christmas break
>
> But Python is, strictly speaking, very similar to Lisp & ELisp

It isn't close in the sense that in a Python shell, you don't
create buffers (or anything like them), and then use Python functions to
transform them to other forms.

Emacs is practically a lisp machine, where you can do just that, for
better or worse.

>
> There are reasons why functional languages like Lisp, Python, Erlang,
> Haskell, Clojure, etc. Pure Functional Languages, are still not fully
> dominating--though Python is very popular
>
> For such purposes, time reveals what's best--make makefiles & shell
> programming will continue to be best for many things

I agree this might be best for compiling code via command line programs
that were built with a pipe philosophy, and where shared state is either
not needed, or is communicated via files.

I am not yet convinced it is necessary to think of building org-files
that way just because it works so well for code. You can do this all
within the elisp environment where you are editing the org file. Once
that works, why abstract it out to a shell command? Why not have some
easy way to just run the blocks that build what you want?

make/shell programs fall flat on their face in the following way: In an
elisp script we have the finest tooling around to see the docstrings of
any function, definitions of variables, to navigate to function
definitions, and to debug and step through the code. This makes this
approach so superior to anything else in my opinion, it is hard to argue
any other feature of make/shell is worth giving it up for. I will
concede this is just a better tooling argument though, but these tools
do not exist as far as I know in a shell. You might get tab-completion,
but not easy access to documentation and code. If there was a lispy
version of make that made this possible, I would use it.

Anyway, I am not arguing make is not good for what it does. It is fine
for that. It isn't what I want to use for this though.

>
> Have you looked into emacs-lisp shell--i.e. the "eshell"?
>
> --and I believe you can call the "eshell" in batch mode from a shell
> command--and the "eshell" can execute emacs lisp functions of course
>
> --and you can even do all that from the Python interactive shell if you
> like--and call remote shells from it and/or use "IPython"
> https://en.wikipedia.org/wiki/IPython
>
>
>
>
>
> On Sun, Jan 26, 2020 at 2:38 PM briangpowell . <address@hidden>
> wrote:
>
>> "don't want it to necessarily use Makefiles"
>>
>> Why the hey not Dr. Kitchin!?
>>
>> Make is an extremely powerful language
>>
>> Problem is people make makefiles that are often hard for others to
>> read--suggest you try hard to keep it simple & avoid the fancy ways of
>> doing things
>>
>> Make has flow control & is a very rich language
>>
>> I use makefiles for every project Then Ccc to compile and/or run
>> everything--in Emacs--can't get easier and faster than that--as long as you
>> make the makefiles so easy that anyone can read & understand exactly what
>> it does
>>
>> This is my default, which I may edit on-the-fly and just put in another
>> project name:
>>
>> make --ignore-errors --jobs=555 -w --keep-going --warn-undefined-variables
>> --environment-overrides -f ~/n/n/etc/1cv2tex2doc-project_mak.makefile
>> 1cv2tex2doc-project-exe
>>
>> All slightly interesting ways to improve make have failed slightly, in one
>> way or another--they catch on for a few years until they're mostly
>> abandoned--for very good reasons
>>
>> Look at it this way: Is there anything wrong with make?  Where does it
>> fail?
>>
>> Everyone should learn & use make
>>
>> --Brian G. Powell, M.S.--that guy that still loves Makefiles and still
>> loves using asterisks ("***") as bullets in OrgMode
>>
>> On Sun, Jan 26, 2020 at 1:11 PM John Kitchin <address@hidden>
>> wrote:
>>
>>> Hi everyone,
>>>
>>> This is only semi-on-topic. I am looking for something like M-x compile
>>> for my org-files, but I don't want it to necessarily use Makefiles. I am
>>> looking for suggestions of existing solutions to this, or thoughts on how
>>> to implement this.
>>>
>>> Actually, if it was possible to get M-x compile to run an elisp function
>>> instead of a make file, it might be all I need, but it looks like it runs
>>> shell commands.
>>>
>>> For most of my files, simple exporting is totally adequate. But, some
>>> files are more complicated, and what I usually do in these cases is write
>>> an elisp code block in a section that is tagged :noexport: and then I run
>>> that block to build the result. A recent example was a proposal where I
>>> needed a pdf of the body, and separate pdf of the references.
>>>
>>> I have separate elisp functions that generate these, and then I added
>>> some custom cleanup code in the block to delete some intermediate
>>> directories. I don't want to put these in a makefile because they are
>>> specific to this document. While this works, in a large document I find it
>>> a little inconvenient to make a small change say at the top, and then to
>>> jump to the bottom to run the build block to see how it changed. What I
>>> would prefer is to just run a command like M-x org-compile that would know
>>> about this build block and run it. That block could of course be a shell
>>> block that runs a makefile, but it would most often be an elisp block. I
>>> could even imagine that there is a makefile block that is tangled before
>>> running a shell block that runs a make command.
>>>
>>> What I do in a function now is something like this in a save-excursion:
>>>
>>> (when (not (stringp (org-babel-goto-named-src-block "build")))
>>>       (org-babel-execute-src-block))
>>>
>>> I don't use this in these projects, but they highlight some of the
>>> complexities I am trying to simplify. These are book like projects with
>>> special formatting needs, and multiple outputs.
>>>
>>> In this project (
>>> https://github.com/jkitchin/pycse/blob/master/pycse.org#L15096), I have
>>> multiple output targets that I would run.
>>>
>>> In this project (
>>> https://github.com/jkitchin/dft-book/blob/master/dft.org#build) I use
>>> some temporary filters to save the src blocks to files, and to embed them
>>> in the pdf so they can be opened.
>>>
>>> Anyway, it feels like I am reinventing something here, and that there
>>> might be some better approach already out there. Maybe some elisp
>>> equivalent of a makefile or something?
>>>
>>> Thoughts?
>>>
>>> John
>>>
>>> -----------------------------------
>>> Professor John Kitchin
>>> Doherty Hall A207F
>>> Department of Chemical Engineering
>>> Carnegie Mellon University
>>> Pittsburgh, PA 15213
>>> 412-268-7803
>>> @johnkitchin
>>> http://kitchingroup.cheme.cmu.edu
>>>
>>>


--
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu