Re: Managing data files in workflows

[Konrad Hinsen]

Hi Ricardo,

> We can fix the problem with symlinks by restoring the target of the link
> instead of the link itself, but I feel that we need to take a step back
> and consider what this cache is really to be used for.

Indeed, and I have to admit that this isn't clear to me yet. What is it
supposed to protect against? Modification of files by other processes of
the workflow? Modification of files outside of the workflow? Both?

For the second situation (modification outside of the workflow), I think
it would be sufficient to store a checksum, and terminate the workflow
with an error if it detects such tampering.

The first situation is more difficult. There are actually two cases:
 1. The workflow intentionally updates files as it proceeds.
 2. The workflow modifies a file by mistake.

Only the workflow author can make the distinction, so this needs some
specific input syntax. Case 2 could then again be handled by a simple
checksum test for signalling an error.

This leaves case 1, for which the only good solution is to make a copy
of the file at the end of each process, and restore it in later runs.

Cheers,
  Konrad.