I was browsing the message archives this morning and noticed a few threads in regards to using Stow as a package manager. Suggestions of features, extensions, etc. I was not a subscriber of the list so I would've not have seen these threads when they were created, but I've been using Stow as my package manager for over 5 yrs. I thought I'd share some on what I did:
I was working on this project where I needed to build linux devices from source,manage software versions, and keep track of every file that was on the device. I did this in the past, but without any package management it was hard to track where files came from and if they should even be there.
I found Stow and what I liked about Stow is that it made it easy for me to audit this system. Every file in /bin was a symlink into /usr/pkg/$(uname -m)/pkg-name/bin/..... I could create scripts that would search / look for files that were NOT symlinks.
To prepare for building these systems from scratch I worked through LFE (Linux From Scratch) and BLFS (Beyond ..). After /tools was built, per the doc, I added a few perl modules and GNU Stow to /tools/ I then wrote shell scripts for each page, did DESTDIR builds of everything, and used Stow for EVERY PAGE. When I was done the whole system booted into X and was all created using Stow. Every file in /bin, /sbin, /lib, /usr/lib were symlinks to their owners,. The project was as if you went to Navy Seals school and asked "show me the hardest way to succeed here." If you look at the BLFS page on ntp you'll notice that the instructions require you to add a group. Instead of doing that during the build, I add all that to post-install scripts and the group is added after install if it does not exist.
GNU Stow is not a package manager. It needs a wrapper to do package management. I wrote one. We'll call it sstow.
1. Every pkg directory has a hidden json file with details about the pkg.. Pkg name, version, files in the pkg. The names of other pkgs that must be installed before (dependencies), lists of libraries that the ELF files were linked to use, etc. To see this info you execute: query package: 'sstow --qp pkg'.
2. I created strict rules where no package could edit any files in another. This means that /etc/httpd/httpd.conf that would normally be httpd.conf -> ../../usr/pkg/apache-2.4.26/etc/httpd/httpd.conf had to be converted from a symlink to a regular file. Stow can't do any of that so I had to add stow-pre/post-install and stow-pre/post-remove operations. Now, sstow will execute those as required. stow-post-install converts /etc/httpd/httpd.conf from a link to a real file and you can not edit it without breaking those rules. The rules allow you to tar the /usr/pkg/apache-2.4.26 direcroy, transfer to another system and install via 'sstow --tarball apache-2,4,26.tar.xz;'
One an audit, all files in /bin would be symlinks and most files in /etc would be real files that were converted from symlinks.
3. Some programs can't handle symlinks and expect the realpath to be the same as the symlink. Python is an example of this. For perl and python modules I create packages for each module.I have to explicitly tell Python where to find things. In these cases I'll copy what normally goes into /usr/bin to /lib/libexec. I install a wrapper as /usr/bin/python and that wrapper does the right thing.
Something like this:
PP="/opt/name"
LD_PRELOAD="libkeepalive.so"
PYTHONPATH="${PYTHONPATH}:${PP}/usr/lib/python3.7/site-packages"
PYTHONDONTWRITEBYTECODE=1
export PYTHONPATH PYTHONDONTWRITEBYTECODE
if [[ ! -e "${PP}/opt/extra/lib/libssl.so.1.1" ]]; then
echo "ERROR: PKG openssl1 not installed."
exit 1
fi
exec "${PP}/lib/libexec/python3.7" "$@"
Also, some programs have strict rules that don't allow symlinks. Apache's cgi-bin is an example. You can either patch apache, or convert /srv/www/cgi-bin to a real dire
copy and convert the cgi files from your own package into real files.
GCC is another example of a program that can break with symlinks. When you work the GCC page with LFS, after /tools, you'll run into some issues that will require some changes to the instructions building GCC.
4. My goal is to start with a simple directory tree at /, run a for loop against a list of tarballs, run sstow on those files, and be running. I also want to run the same loop, do a sstow --purge, and have a directory tree as pristine as when I started. stow-post-remove scripts do remove files like /etc/httpd/httpd.conf. To accomplish this I support folding, but I start out with a Linux FHS structure where each directory contains a hidden file '.nofold'. That is enough to keep Stow from folding /bin into the first package it installs. Works great for /var/log, /var/run/ /tmp, /etc, etc.
5. Files that are added to for each perl module can be a problem.
perllocal.pd and .packlist that are included with each perl module can cause issue for Stow. I don't need these files so I simply delete them when I build those packages. I build over 500 perl packages and install via Stow. Originally, I hadded those files to stow-local-ignore and then the post-install routine would add info to perllocal.pd and .packlist found in /
6. Packages are created as standard, deve, doc, extra, etc. A full install for a device is over 1G and 1,899 packages. DOC, DEV, and pkgs ina prune list are purged. Now it is 503 packages and 250M. That directory becomes a squashfs image that runs on the device. sstow-get will download a package from a repo via https ,install on a portion of SSD that is non-volatile, and at boot is installed via sstow and GNU Stow.
I liked the strict symlink farm that GNU Stow provided and it reminded me of SCO OpenServer 5 Software Storage Objects. That was also a symlink farm. Creating a system that uses GNU Stow as just the symlink farm I've basically created a system that is not much different than apt-get, yum, etc. I have created most of their features.
In summary, I'm saying to those that have inquired about using Stow as a package manager that it is possible, but you have to put a lot of work into it. Proficiency in Linux/UNIX is also a requirement since you have to build software and install them into DESTDIR, create a package, and a system to install it correctly. When I needed guidance on how to do a DESTDIR install of a piece of OSS I would look at LFS, BLFS, or Arch Linux. The Arch Linux PKGBUILD files were VERY useful because it installs everything into a DESTDIR before it installs anything into /. I would guess that I've spent a few hundred hours creating the wrapper around Stow. Even so, it was a very good investment. We now know where every file comes from. We know every version of what we have. We know where the source came from. We can recreate packages. We can install packages on devices that were needed in the "firmware".
sstow --installed
....
100 : apache-2.4.46-1
101 : tomcat-connectors-1.2.48-1
102 : libjson-maybexs-perl-1.003008-1
103 : libcrypt-openssl-guess-perl-0.11-1
104 : libcrypt-openssl-random-perl-0.15-1
105 : libcrypt-openssl-rsa-perl-0.31-1
106 : libconvert-asn1-perl-0.27-1
107 : libcrypt-openssl-bignum-perl-0.09-1
108 : libcrypt-le-perl-0.35-1
109 : liblog-log4perl-perl-1.47
110 : libperl-tidy-perl-20201001-1
111 : libtest-output-perl-1.031-1
112 : libpackage-alias-perl-0.13
113 : libnet-netmask-perl-1.9022-1
114 : libcgi-perl-4.51-1
....
sstow --qp libmysql-connector-java
PKG : libmysql-connector-java-8.0.22-1
NAME : libmysql-connector-java
VER : 8.0.22-1
GROUP : PROD
ARCH : x86_64
BUILD : 2020-11-04 13:35:33
SIZE : 2.28M
FILES :
./l/java/mysql-connector-java-8.0.22.jar
./l/java/mysql-connector-java.jar -> mysql-connector-java-8.0.22.jar
sstow --qp --qp-libs busybox
PKG : busybox-1.29.2-1
NAME : busybox
VER : 1.29.2-1
GROUP : PROD
ARCH : x86_64
BUILD : 2020-11-04 11:42:27
SIZE : 646.41K
FILES :
./b/busybox
/lib64/ld-linux => /lib64/ld-linux-x86-64.so.2
libc.so.6 => /lib64/libc.so.6
libm.so.6 => /lib64/libm.so.6
linux-vdso.so.1 => linux-vdso.so.1
./sb/ifconfig -> ../bin/busybox
./sb/telnet -> ../bin/busybox
sstow-deps rrdtool
PKG : rrdtool-1.4.8-1
NAME : rrdtool
VER : 1.4.8-1
Dependencies:
I: bzip2 1.0.6-1 PROD
I: expat 2.2.0-1 PROD
I: fontconfig 2.12.1-1 PROD
I: freetype 2.7.1-1 PROD
I: glib 2.30.0-1 PROD
I: libcairo 1.6.4-1 PROD
I: libdbi 0.9.0-1 PROD
I: libffi 3.2.1-1 PROD
I: libpango 1.21.1-1 PROD
I: libpixman 0.10.0-1 PROD
I: libpng 1.6.28-1 PROD
I: libxml2 2.6.26-1 PROD
I: pcre 8.35-1 PROD
I: system 0.0.0-1 PROD
I: zlib 1.2.8-1 PROD
Total : 15
Packages dependent on rrdtool:
I: ops-cs-singularity 1.0.0 PROD
INSTALL : 1
Total : 1
sstow-get --not-installed
URL :
http://10.0.20.57:9090/REPO: pkg/cs/dev/i586
Downloading: md5.txt
######################################################################## 100.0%
Downloading: index.json.xz
######################################################################## 100.0%
Packages not installed.
1 : 3proxy-0.8.12-1 0.8.12-1
2 : 3proxy-doc-0.8.12-1 0.8.12-1
3 : SoapyRTLSDR-1542310902-1 1542310902-1
4 : SoapyRemote-1542309964-1 1542309964-1
It never started out as complex as above, it never does. When building the firmware I need to make sure all libs required are installed so I had to add --qp-libs. This required the program that generates the info file to search the pkg directory, look for ELF, run ldd, and catalog the output. This will not work on .so files hidden in java jars. I needed a way to install packages that I needed rarely, while keeping firmware image down to 60M. I had to add a repo feature. The repo feature required a way to install deps too. I have to add dependency tracking. sstow also support a .acl file created by getfacl or the build. It will set those perms after install, but before running any post-install.
Chris