[Orgmode] Re: TaskJuggler3, revisited

[Anthony Lander]

Hi everyone,

Christian Egli and I have been having an off-list exchange about  
making a TaskJuggler3 exporter. In fact, Christian would like to  
change the current exporter to export to either TaskJuggler2 or  
TaskJuggler3, depending on a defcustom! We have quite a number of  
ideas in the pipeline now, and thought it would be best to move the  
discussion, and access to the source code, back onto the list.

Christian has set up a github repository with the changes he is making  
here: http://github.com/egli/org-mode

And the code I sent to Christian, which makes the exporter work with  
TJ3, and also introduces some bug fixes and new features is here: http://github.com/alander/org-taskjuggler3 
. Please note that this is different from Christian's code, and that  
he is the maintainer of the exporter that is part of org-mode. I  
needed something to work, so I spent an afternoon hacking until it did  
what I needed.

If you are interested in the details, our e-mail exchange is  
reproduced below.

  -Anthony


------------------------8<-------------------------

        From:   address@hidden
        Subject:        Re: [Orgmode] Re: TaskJuggler 3, revisited
        Date:   October 19, 2010 5:29:42 AM GMT-04:00
        To:     address@hidden
        Cc:     address@hidden

Hi Anthony

Anthony Lander <address@hidden> writes:

> Please find attached a somewhat improved version of your TJ2 exporter,
> a drop-in replacement that exports to TJ3, and also a sample .org
> file.

This is all exciting stuff. It's a little hard to digest (many changes
rolled into one, white space formatting changes that make it hard to  
find
the real change, common lisp idioms which I'm not familiar with). Let me
get back to you with some questions and then we can discuss how to most
easily merge the stuff.

- Why did you compute the leafiness? I seem to have experienced that tj3
 has a problem with zero effort tasks that aren't marked as milestones.
 Is that the reason?

- Why do you add a duration of 1d if the task has neither a duration, an
 end, a period nor an effort? Shouldn't that be a milestone instead?

- I see that there is a need to annotate a project with stuff such as
 scenarios, etc and I was missing a way to have file specific reports
 or other globals. Your additional tags solve that problem. However I'm
 a bit reluctant to add more magic tags that mark the trees in some
 way. I was hoping to find a more general way for this problem. So far
 I haven't found a good solution though.

- I like the idea of the TJ drawer, but in the end you just seem to use
 it for the project node and the globals node. So in essence they are
 taskjuggler source code blocks (in disguise) which are tied to a
 particular node. In fact they are not even really tied to a particular
 node, especially in the case of the globals. This goes back to the
 previous question about a good way to add file specific globals. Maybe
 some kind taskjuggler specific export option is really what we are
 looking for.

As an aside, I think it's better to post the source code to the list.
There might be other people interested in it and pitching in with
opinions and improvements.

Thanks
--
Christian Egli
Swiss Library for the Blind, Visually Impaired and Print Disabled
Grubenstrasse 12, CH-8045 Zürich, Switzerland

------------------------8<-------------------------

        From:   address@hidden
        Subject:        Re: [Orgmode] Re: TaskJuggler 3, revisited
        Date:   October 19, 2010 8:55:38 AM GMT-04:00
        To:     address@hidden

Hi Christian,


On 10-Oct-19, at 5:29 AM, Christian Egli wrote:

> This is all exciting stuff. It's a little hard to digest (many changes
> rolled into one, white space formatting changes that make it hard to  
> find
> the real change, common lisp idioms which I'm not familiar with).  
> Let me
> get back to you with some questions and then we can discuss how to  
> most
> easily merge the stuff.

Apologies for the big blob of changes - I was initially just trying to  
get stuff to work, and, well, just kept going. As for the spacing/ 
formatting, I ran M-x indent-region RET on the whole file to get the  
formatting consistent. I know it makes diffing hard, though :(

Is the unfamiliar idiom the backtick list with the ,variables in it,  
by chance?

> - Why did you compute the leafiness? I seem to have experienced that  
> tj3
> has a problem with zero effort tasks that aren't marked as milestones.
> Is that the reason?
> - Why do you add a duration of 1d if the task has neither a  
> duration, an
> end, a period nor an effort? Shouldn't that be a milestone instead?

Ah, sorry. This I should have documented. The problem is that TJ3  
fails to compile the file if there is a leaf node with no computable  
end date. TJ2 happily ignored the situation, but TJ3 throws an error.   
So the problem I'm trying to solve is that as you are working on your  
project plan, it won't compile unless you put in a bunch of dummy  
estimates for the leaf nodes.

I found it easier to let the exporter put them in, so that I don't  
always think to myself, "did I put this in as a real estimate or just  
to make the compiler happy?" Perhaps it would be better to change 1  
day to 1 minute or 1 second to make it very obviously a dummy duration  
that is put in only to satisfy the compiler. And maybe even a comment  
in the exported file, too.

Also, I hadn't thought of milestones. I think the code needs a bit of  
a fix because it needs to know that a milestone has zero effort, but  
won't cause the compiler to fail.

> - I see that there is a need to annotate a project with stuff such as
> scenarios, etc and I was missing a way to have file specific reports
> or other globals. Your additional tags solve that problem. However I'm
> a bit reluctant to add more magic tags that mark the trees in some
> way. I was hoping to find a more general way for this problem. So far
> I haven't found a good solution though.

I agree with you about the magic tags. I hate the feeling of  
reimplementing the entire TJ3 file format in org. That's asking to  
always be playing catch-up, and to always be missing features. I like  
very much the idea of "borrowing" features from org that make sense,  
and doing something useful on export. Examples are your % doneness of  
a task, and your "ordered tasks in a hierarchy" ideas. And in fact I  
added using the todo states and tags as flags so you can easily build  
tj reports that reference them (see the "active" gantt chart, for  
example).

In the end, I felt the project needed a special marker because there  
are specific TJ3 features that go with it. And also, the globals need  
to go somewhere. I am also not thrilled with making all these tags,  
but I didn't have a better solution either.

In any event, the TJ drawer is my attempt at a catch-all solution. It  
at least allows a user to use missing features without waiting for a  
development change. So for example, I personally will never use the  
accounting stuff, and would not be very motivated to add it to the  
exporter, but if someone needed it, they could put the required code  
in the TJ drawer, and they could still use the TJ exporter for their  
project instead of having to abandon org-mode entirely for one missing  
feature.

> - I like the idea of the TJ drawer, but in the end you just seem to  
> use
> it for the project node and the globals node. So in essence they are
> taskjuggler source code blocks (in disguise) which are tied to a
> particular node. In fact they are not even really tied to a particular
> node, especially in the case of the globals. This goes back to the
> previous question about a good way to add file specific globals. Maybe
> some kind taskjuggler specific export option is really what we are
> looking for.

Interesting thought. I very much like the idea that you can do other  
things with your org-file than export it to taskjuggler. For example,  
keep actual notes in the file, and export it to other formats like  
html or LaTeX, for purposes other than project planning. Being able to  
say "no, this is not a task" in the task tree would be very useful. I  
don't know how to do it nicely either (yet).

> As an aside, I think it's better to post the source code to the list.
> There might be other people interested in it and pitching in with
> opinions and improvements.

That's a good idea. How about if we set up a git repository on github  
with the files, and then post the link on the mailing list? I worry  
that posting code and patches will badly confuse the patch manager  
software than grabs code from the mailing list. Also, if we're going  
to do that, I think we should put your email, and this reply to the  
list as well. Your thoughts?

All the best,

 -Anthony

------------------------8<-------------------------

        From:   address@hidden
        Subject:        Re: [Orgmode] Re: TaskJuggler 3, revisited
        Date:   October 28, 2010 3:28:13 AM GMT-04:00
        To:     address@hidden
        Cc:     address@hidden

Hi Anthony

Anthony Lander <address@hidden> writes:

> Is the unfamiliar idiom the backtick list with the ,variables in it,
> by chance?

No it was more stuff like `return' and `return-from'.

> Ah, sorry. This I should have documented. The problem is that TJ3
> fails to compile the file if there is a leaf node with no computable
> end date. TJ2 happily ignored the situation, but TJ3 throws an error.

OK, I understand. I managed to slowly integrate (some of) your changes.
It should now generate milestones for nodes that cannot be scheduled and
are leaf nodes. So basically it should export valid code for tj3 if you
set org-export-taskjuggler-target-version to 3.0. Of course the report
definitions in the defcustom org-export-taskjuggler-default-reports are
not compatible with tj3 so you will need to change these.

I have (or I guess rather had) some fondness for recursive functions, so
I re-implemented your leafiness function recursively. However I ran into
problems with max-lisp-eval-depth and it appears that recursion is not
encouraged in Emacs lisp
(http://www.gnu.org/s/emacs/manual/html_node/elisp/Compilation-Tips.html#Compilation-Tips 
).
So I guess it's back to iterative style.

> In any event, the TJ drawer is my attempt at a catch-all solution. It
> at least allows a user to use missing features without waiting for a
> development change. So for example, I personally will never use the
> accounting stuff, and would not be very motivated to add it to the
> exporter, but if someone needed it, they could put the required code
> in the TJ drawer, and they could still use the TJ exporter for their
> project instead of having to abandon org-mode entirely for one missing
> feature.

I like the drawer stuff and will integrate it.

> Interesting thought. I very much like the idea that you can do other
> things with your org-file than export it to taskjuggler. For example,
> keep actual notes in the file, and export it to other formats like
> html or LaTeX, for purposes other than project planning.

Absolutely, I agree.

> Being able to say "no, this is not a task" in the task tree would be
> very useful. I don't know how to do it nicely either (yet).

Hm, no this is not possible right now. Why would you want to do this?
Maybe we could just mark it as a comment and make the exporter honor
comments.

> How about if we set up a git repository on github
> with the files, and then post the link on the mailing list?

I set up a repo and pushed my changes to the code there
(http://github.com/egli/org-mode).

> I think we should put your email, and this reply to the
> list as well.

Yes, I almost forgot. Would you mind doing that?

Thanks
Christian

--
Christian Egli
Swiss Library for the Blind, Visually Impaired and Print Disabled
Grubenstrasse 12, CH-8045 Zürich, Switzerland

------------------------8<-------------------------

        From:   address@hidden
        Subject:        Re: [Orgmode] Re: TaskJuggler 3, revisited
        Date:   October 31, 2010 11:45:34 AM GMT-04:00
        To:     address@hidden


On 10-Oct-28, at 3:28 AM, Christian Egli wrote:

> Anthony Lander <address@hidden> writes:
>
> > Is the unfamiliar idiom the backtick list with the ,variables in it,
> > by chance?
>
> No it was more stuff like `return' and `return-from'.

I see. Well, once you were explicitly requiring 'cl.... :) And as you  
discovered, without a tail-recursion optimizing compiler, recursive  
functions are prohibitively expensive in elisp. Once you are  
iterating, you need a way to exit iteration scopes before the end of  
the iteration.


> So basically it should export valid code for tj3 if you
> set org-export-taskjuggler-target-version to 3.0. Of course the report
> definitions in the defcustom org-export-taskjuggler-default-reports  
> are
> not compatible with tj3 so you will need to change these.

Right. Or perhaps better, have different default reports for TJ2 or  
TJ3, which the exporter selects as appropriate.

> > Interesting thought. I very much like the idea that you can do other
> > things with your org-file than export it to taskjuggler. For example,
> > keep actual notes in the file, and export it to other formats like
> > html or LaTeX, for purposes other than project planning.
>
> Absolutely, I agree.
>
> > Being able to say "no, this is not a task" in the task tree would be
> > very useful. I don't know how to do it nicely either (yet).
>
> Hm, no this is not possible right now. Why would you want to do this?
> Maybe we could just mark it as a comment and make the exporter honor
> comments.

Yes, marking it as a comment is not a bad idea. The reason I want to  
do that goes with the idea of using the org file for additional  
purposes than just a task tree. So for example, I would like to be  
able to have nodes with meeting notes, or links to related files etc  
etc under a task. That way I am only maintaining one project document,  
not separate ones for notes, scheduling, etc. I think that is very  
much in the spirit of org-mode, which so heavily emphasizes sparse  
trees, agenda views etc as ways to tease out relevant information from  
an intermingled file.

> > How about if we set up a git repository on github
> > with the files, and then post the link on the mailing list?
>
> I set up a repo and pushed my changes to the code there
> (http://github.com/egli/org-mode).
>
> > I think we should put your email, and this reply to the
> > list as well.
>
> Yes, I almost forgot. Would you mind doing that?

I'll do that, yes.

Best regards,

 -Anthony