Re: Environment of a shepherd service

[Edouard Klein]

Thank you Maxime for your answer :)

Maxime Devos writes:

> edk@beaver-labs.com schreef op zo 11-04-2021 om 21:31 [+0200]:
>> Dear fellow Guixers,
>
>> [...]
>> But, when I try to run it with shepherd, it fails because it can't find
>> flask (a dependency of the software, which I've put as a
>> propagated-input, and is indeed installed in the container).
>
> Propagated inputs can be inconvenient at times.  I would advise
> looking where requisomatic is referring to flask, and replacing
> flask --> (string-append (assoc-ref inputs "flask") "/bin/flask")
> using substitute*.

OK, I understand this part :) The executable is gunicorn, so I'll just
use its full path in the script that launches the service.

My previous solution was to source the profile beforehand:

https://gitlab.com/edouardklein/requisomatic/-/blob/baf3fe51ad8bbabbcbc467dff92ec02c43e6daf1/guix.scm#L200

I do agree that a rich profile feels dirty and I too don't like
propagated inputs too much, but with Python I don't understand what the
best way to do it is ?

For example, in order to package a Python application in a way that does
not break the host system, I had to put all the search-paths in a
wrapper script and change all propagated inputs as just inputs. I find
the resulting code quite ugly, and I don't think I'm doing things right:
https://gitlab.com/edouardklein/gendscraper/-/blob/980f2597cc36bc79a9be7af5814e0fcf2313b677/gendscraper.scm#L1176

I wouldn't like to have to this for every service. If propagated inputs
are inconvenient, and if services run in an environment where the
search-paths of the installed packages are not available, how can we
easily tell the software where to find its dynamically-loaded parts ?


>
>> But, when I try to run it with shepherd, it fails because it can't find
>> flask (a dependency of the software, which I've put as a
>> propagated-input, and is indeed installed in the container).
>> [...]
>
> Some advice (warning: I'm not familiar with gunicorn or requisomatic
> at all).

I used flask and gunicorn in this project, but I'm not familiar with
them either ;)

>
>> 
>> -----extract from my operating-system declaration file-------
>> (define requisomatic-shepherd-service
>>   ([...] (shepherd-service
>>             [...]
>             (documentation "Run the requisomatic server")
>>             (start #~((make-forkexec-constructor
>>                     ;;   (append
>>                     ;;    (if db-file
>>                     ;;      `("env"
>>                     ;;        ,(string-append "REQUISOMATIC_DB_FILE=" 
>> db-file))
>>                     ;;      '())
>>                                         '("gunicorn" "requisomatic:app")
>
> Normally, services refer by absolute path to the binary to run, and not rely
> on the PATH (the latter would require polluting the system profile).
> Idiomatically, one would write
>
>   '(#$(file-append gunicorn "/bin/gunicorn")
>     #$(file-append requisomatic) "/wherever/the/binary/is")
>
I did not know about file-append even though it is in the manual. Thanks :)


>>                        #:directory (string-append #$requisomatic 
>> "/bin/requisomatic/")
>
> Why are you changing the working directory to
> (string-append #$requisomatic "/bin/requisomatic/"),

because . is by default where flask will look for the code it needs.

>and why is "/bin/requisomatic" a
> directory and not an executable?

Because flask/jinja/etc. are normative about where they want stuff to be
and I put everything in a single dir. As most of the assets is code
(some are html templates) I put it in bin/... but I could put them in
/lib of wherever. This is not the cleanest, but I really wanted to get a
guix operating system declaration to work before I did things cleanly.

> Is that a gunicorn thing?

More like flask, but yeah, it's a framework thing.

>
>> Why is the PYTHONPATH (and the other env vars, for that matter) not
>> propagated from the package to the shepherd service by default ?
>
> How is the shepherd service supposed to automagically know which packages
> to include in the environment variables?
>

I realised afterwards that by extending the profile-service-type,
services can say what packages they need (I currently add them to the
operating-system definition). I would expect the shepherd
command to run in an environment where the search-paths of those package
is set. I have no idea if this is complicated to do, but as a service
writer I would enjoy it very much.

>> And how can I make it so ?
>
> Use the #:environment-variables option, see e.g.
> bitlbee-shepherd-service

I see the concept, but the list is hardcoded, and for even moderately
complex Python application (or any other dynamic language) the list of
env variables to set will become huge.

> Or create a wrapper.  See e.g. wrapped-dbus-service.

This looks like a clean version of what I did for gendscraper, but the
problem remains of which variable to set. This information exists in
each individual package. Some will need additions to e.g. PYTHONPATH,
others to LD_LIBRARY_PATH, etc. We need to walk the entire dependency
DAG somehow.
>
>> Follow up question, can shepherd services be specified to run in a
>> specific profile ?
>
> IIUC, currently shepherd services aren't run in *any* profile at all.
> It would be useful to have a function manifest->environment-gexp
> though.

Or package->search-paths ? 

>
>> So that I can have two services with incompatible
>> dependencies running at the same time in the same operating-system ?
> Yes, it with "dependencies" you mean packages, and not other services.
>

I meant packages :)


Thank you very much for your help, it seems you know services inside
out. I can now pinpoint way better what it is that I don't understand. I
hope I was more clear in expressing it.

Cheers !

Edouard.

> Greetings,
> Maxime.