[Orgmode] Re: [babel] How to kill two birds with one stone?

[Sébastien Vauban]

Hi Dan,

Myself quickly reacting on this...

Dan Davison wrote:
> Cool post. I hope someone has some good ideas in this thread.

Thanks. Once solved, this one (and many more experiments I'm jotting down)
could become "case studies" on Worg, or so.

> Some quick responses / questions below.
>
>> #+TITLE:     Document a shell script as separate blocks
>> #+DATE:      2011-02-04
>> #+LANGUAGE:  en_US
>>
>> * Abstract
>>
>> When writing shell scripts, I'd like to kill *two* birds with one Babel 
>> stone:
>>
>> - Be able to *execute the script in situ*, so that I get a copy of the 
>> results
>>   stored in (and "versioned" with) my documentation.
>>
>> - Be able to *chain the code blocks*, so that I tangle a "natural" script 
>> into
>>   a file for later execution.
>>
>> I guess I currently miss some points in order to reach that goal in a clean
>> way. Could you give me advice on how to get a step further down the route?
>>
>> * Sample code
>>
>> For the sake of clarity, let's take a simple problem: I'd like to generate a
>> DOT graph of links between all files from a directory tree.
>
> So if file a.org contains
>
> -----
> jhasg
> [[file:b.org]]
> [[file:c.org]]
> hbjgv
> -----
>
> then you want an arrow from node a to node b and from node a to node c, right?

Exactly. Though, this is not limited to Org files, but any file (and any
filename) in fact...

>> The procedure:
>>
>> 1. (Recursively) list all files inside the directory.
>>
>> 2. For each file, (recursively) search for its name referenced in all the
>>    files.
>>
>> 3. Generate a DOT representation of the link between files.
>>
>> ** List all files under current directory
>>
>> #+srcname: dw-file-tree
>> #+begin_src sh :results output
>> cd ~/Some-Project-Dir
>> find . -type f -print | grep -v .svn | head -n 5
>> #+end_src
>>
>> #+results: dw-file-tree
>> #+begin_example
>> ./.cvsignore
>> ./charge_dim
>> ./charge_fct
>> ./compte
>> ./controle_config
>> #+end_example
>>
>> Here, I voluntary limit the number of results to the first 5 files, for the
>> compactness of this example. This sample does not include any file from
>> subdirectories, but it doesn't matter.
>>
>> ** Search recursively for anything about a file
>>
>> Search through all files (ignoring =.svn= directories) for any reference to
>> filename given as parameter.
>>
>> #+srcname: search-files-pointing-to-this-file
>> #+begin_src sh :results output :var f="charge_dim"
>> cd ~/Some-Project-Dir
>> find . -not \( -name .svn -prune \) -type f -print0 |\
>> xargs -0 grep -i --files-with-matches "$f"
>> #+end_src
>>
>> #+results: search-files-pointing-to-this-file
>> #+begin_example
>> ./.cvsignore
>> ./charge_dim
>> ./compte
>> ./IFP/Chrg_dim
>> ./IFP/Chrg_dim.avant_simple_recovery_mode_2008_03_12
>> ./principal.env
>> ./29Aalst/Chrg_dim
>> ./29Aalst/Chrg_dim_interactif
>> ./29Aalst/Publ_dim_interactif
>> #+end_example
>>
>> HERE, I'M GIVING A FILENAME AS DEFAULT VALUE OF =f= IN ORDER TO SEE A SAMPLE
>> RESULTS.
>>
>> ** Convert to a DOT representation
>>
>> For every file pointing to the file given in parameter, generate an "arrow"
>> (edge) in DOT representation.
>>
>> #+srcname: dot-arrow-from-files-pointing-to-this-file
>> #+begin_src sh :results output :var f="charge_dim" :var 
>> data=search-files-pointing-to-this-file
>> for i in $(echo "$data"); do echo "    $(basename $i) -> $f"; done
>> #+end_src
>>
>> #+results: dot-arrow-from-files-pointing-to-this-file
>> #+begin_example
>>     .cvsignore -> charge_dim
>>     charge_dim -> charge_dim
>>     compte -> charge_dim
>>     Chrg_dim -> charge_dim
>>     Chrg_dim.avant_simple_recovery_mode_2008_03_12 -> charge_dim
>>     principal.env -> charge_dim
>>     Chrg_dim -> charge_dim
>>     Chrg_dim_interactif -> charge_dim
>>     Publ_dim_interactif -> charge_dim
>> #+end_example
>>
>> HERE, I'M WORKING GIVING ONCE AGAIN THE SAME DEFAULT VALUE FOR TESTING (AND
>> DOCUMENTATION) PURPOSE.
>>
>> * Problem
>>
>> All of the above nicely answer my first goal ("Be able to execute the script
>> in situ, so that I get a copy of the results stored in my documentation").
>>
>> It does not allow for the second one: *how to chain the calls together*?
>>
>> For example, just for chaining steps 2 and 3 (the easier to chain, I think),
>> I'd like to be able to write something like this:
>>
>> #+srcname: search-links-and-generate-dot-arrow
>> #+begin_src sh :results output :var f="charge_dim" :var 
>> data=search-files-pointing-to-this-file :noweb yes
>> for i in (
>>     <<search-files-pointing-to-this-file>>);
>> do echo "    $(basename $i) -> $f"; done
>> #+end_src
>
> Don't forget about C-c C-v C-v `org-babel-expand-src-block' when
> debugging this problem. I've started using that a lot.
>
> My shell scripting may be too weak to help here. (Which is one point,
> I'd be tempted to bail out to another language on this one)

That's another point. Once written in Bash scripts, I'd be interested (and I
think it is very interesting from a Babel point of view) to see the same
functionality implemented in other languages.

> What exactly are you doing there? Executing the code from the other block in
> a subshell and using the stdout as the set to loop over?

I'm not especially wanting to try executing the code block in a subshell. Yes
for the "in situ execution" point of view, no from the "code to be tangled"
point of view...

> You're sure you don't want parentheses
> <<search-files-pointing-to-this-file>>() ? Or $( ... ) / backticks ?

Parenthesis would run the code... But I need the code to be written into this
block of code (in another valid form) so that the tangled file will be runable
in any directory, with any structure below it, at any point in time.

> How are you going to pass arguments to the search-files-pointing src block?

Good question. Unanswered at this point in time...

>> #+results: search-links-and-generate-dot-arrow
>>
>> But it yields an error:
>>
>> #+begin_src text :eval no
>> sh: line 14: syntax error near unexpected token `('
>> sh: line 14: `for i in ('
>> #+end_src
>>
>> Note, in the latter code block, that I did not even tried to really chain
>> steps 2 and 3: I'm rewriting step 3, including step 2 inside it.
>>
>> *I certainly miss a smarter way* to achieve the above.
>
> I think a relevant point here is that Org doesn't yet have the ability
> to pass data on standard input to a code block. I.e. a :stdin header
> arg. I don't think it's that hard, someone would just need to rework
> `org-babel-eval' so that it puts the code into a temporary file, freeing
> up stdin to be used for data (and therefore we would no longer be able
> to use shell-command-on-region but some other command (call-process I
> think?).)
>
> Then you'd be able to do something like
>
> #+srcname: search-links-and-generate-dot-arrow
> #+header: :stdin search-files-pointing-to-this-file
> #+begin_src sh :results output :var f="charge_dim"
> while read f; do
>     echo "    $(basename $i) -> $f";
> done
> #+end_src

I must admit it's too late now for me to be sure to grasp this latter
paragraphs. I'll read them tomorrow when I'll be able to think about it.

> I'll be interested to see the solution to all this.

So am I, with or without any extension to the current Babel. I'm not sure yet
that it is, or that it is not, doable right now... Maybe a mind swap is just
necessary to view the problem under new light...

Thanks for your help.

Best regards,
  Seb

-- 
Sébastien Vauban