Re: GWL as a build-automation

[zimoun]

Hi,

I am late to the party. :-)

On Mon, 06 Jun 2022 at 10:50, Ricardo Wurmus <rekado@elephly.net> wrote:

> It is not entirely surprising to me that the GWL can express this,
> because it has really simple abstractions: that of a process and that of
> a workflow consisting of processes.

[...]

> Perhaps there is space for a different tool that takes lessons from the
> GWL and Scsh alike, with a focus on command composition and shell
> abstractions.  Perhaps that tool already exists and is called Metabash:
>
>   https://github.com/artyom-poptsov/metabash

>From my understanding, metabash allows to remotely run processes, i.e.,
distribute the pipeline.  Somehow, it could be see as an extension of
Scsh.

However, a pipeline is a linear sequence of processes.  When a workflow
is a DAG of processes.  Therefore, it would appear difficult to me to be
able to express a build-system using only pipelines.

Last, it appears to me expected that GWL could be considered as a
build-system.  A scientific workflow system [1] (as GWL) is just a
specialized implementation to deal with a graph of dependencies.
Software folks speak about the venerable Make as build automation
workflow, while bioinfo folks speak about a specific Python
implementation SnakeMake as data analysis workflow. Just the same
concepts but viewed by different communities. :-)

If I might, an interesting analysis of different strategies for dealing
with the graph of dependencies is done in the paper «Build systems à la
carte» [2].  It presents the various abstractions using Haskell
notations.

1: <https://en.wikipedia.org/wiki/Scientific_workflow_system>
2: <https://doi.org/10.1017/S0956796820000088>


Cheers,
simon