Re: [O] babel and long-running computations

[John Kitchin]

You are describing what I see as a need to run processes asynchronously.

We use the Torque queue system for this. It takes care of scheduling and running jobs on a cluster.

On Thu, Apr 17, 2014 at 6:52 AM, Christoph Groth <address@hidden> wrote:

Hi Julian,

> Generally, org-babel is suited for long running computations.  Its
> caching can help you avoid rerunning code chunks.  And long runtime does
> not conflict with the idea of reproducible research, it just may be not
> very comfortable for the user.

I agree of course that it’s generally a good idea to structure the
analysis in small logical steps and to save intermediate results.  Let
me narrow down my question then: does org-babel support working with
tasks that take several minutes to execute?  By this I mean (most
important first):

- Not freezing the editor during the execution of a task

You code-block should exit if a job is submitted, and record the job-id so that you can check it later. Alternatively, you could do this in ipython notebooks if you have python functions that return jobids.

 

- Being able to execute multiple independent tasks in parallel

No problem. this is what queue systems were designed for.

 

- Being able to interrupt a running task

you do this with the queue commands, e.g. qdel jobid

 

- Being able to inspect the incomplete output of a running task

This is just checking the output files in the running jobs directories.

 

I’d love to hear about any frameworks or workflows that fulfill these
requirements.

Cheers

Christoph


PS.

I’m using ipython notebooks but I’m not happy with them because of the
freezing problem and the complete lack of isolation of tasks within a
single notebook (they live in a common mutable namespace).  I think a
useful framework must be “functional” at the highest level for caching
and dependencies to be useful.