Monochromatic

z3bra, the stripes apart

Pack it up!

— 16 September, 2016

Today was a big day for me. It was the day all my software could play together to reach a single goal:

Maintaining and hosting software packages!

Here are the tools:

Tied together with a fistful of scripts, and some easy preparation, they help me deploy and manage my own utilities accross multiple operating systems without efforts.

This process includes three main tasks: packaging, deploying and installing. This whole process is still at and early stage, and will get stronger and sharper with time (and shell script will most likely turn into actual programs).
Here is an explanation of how it works:

  1. pack an application and sign the tarball
  2. upload the tarball to an online repository
  3. get the latest tarball for a utility
  4. install/update it locally

The process is straigh-forward, and multiple parts can be automated.

Packaging

In order to build my softwares, I need either make or mk (it's tending to be mk only, but for now I still need both). Then process is then fairly simple:

$ cd $utility
$ mk
# mk install

I wrote a quick script to "pack" these utilities for me. It will build the software, install it to a temporary location, create a tarball out of it, and sign this tarball with my private key:

#!/bin/sh

# user specific variables
SICKKEY=$HOME/.sick.d/$USER.key
REPO=/var/www/htdocs/dl.z3bra.org/pack

# guess the pack name from current directory, or use given name
DIR=$(basename `pwd`)
PKG=${1:-$DIR}

# set version from latest git tag, if applicable
TAG=$(git tag | sed -n '$p' | tr -d a-z)
VER=${TAG:-0.0}

# this part should die...
test -f mkfile && MK=mk || MK=make

# build pack and install to ./rootfs/usr
$MK
$MK DESTDIR=$(pwd)/rootfs PREFIX=/usr MANDIR=/usr/share/man install
(
    cd rootfs
    mkdir -p $REPO
    # pack and sign the installed utility
    tar cvj * | sick -s -f ${SICKKEY} > $REPO/${PKG}#${VER}.tar.bz2
)
rm -rf rootfs

# simply ensure that the file has been created correctly
echo
ls $REPO/${PKG}#${VER}.tar.bz2

At this point, to pack one of my utilities, all I need is:

$ git clone git://z3bra.org/skroll
$ cd skroll
$ pack

And I'm done :)

Deploying

This part require a bit of setup. My current repository is at http://dl.z3bra.org/pack, which is, locally in /var/www/htdocs/dl.z3bra.org/pack. My tool synk can get a file synchronized between two peers, but they will have the same path, which is why I also created this directory on my local machine. I also need to upload my public key (for sick checks) and a list of what's currently in the repo.

First, here is the repogen script, which will list the content of the local repo, and write the pack names and version available to a file:

#!/bin/sh

REPO=/var/www/htdocs/dl.z3bra.org/pack
for tarball in $(find $REPO -name '*.tar.bz2'); do
        pkg=$(basename $tarball | cut -d# -f1)
        ver=$(basename $tarball | cut -d# -f2 | sed 's/.tar.bz2//')
        printf '%s\t%s\n' "$pkg" "$ver"
done | sort | tee $REPO/.list

I also copied the public key as .key in the directory.

Now everything is ready for synkronisation (over a VPN, in this case):

find /var/www/htdocs/dl.z3bra.org/pack -type f | synk -h apophis.2f30

It can be automated using wendy, so that everytime the .list file is modified by repogen, everything is replicated on the remote repo:

wendy -m 8 -f $REPO/.list sh -c "find $REPO -type f | synk -h apophis.2f30"

Installing

Now that we can create packs and upload them quickly to the repository, it's time to install them!

Using the .list file, we can check what's available. With the .key file, we can ensure that no-one tampered with our pack during the retrieval process. Using pm, we can install and update our packs for daily use.

All we need now, is a utility to fetch packs from the repo. I named this script "repo"!

#!/bin/sh

url="http://dl.z3bra.org/pack"
cache="$HOME/.cache/repo"
keyring="$HOME/.sick.d"

usage() {
    echo "usage: $(basename $0) [-s] [-t reponame] [PKG..]" >&2
    exit 1
}

reposync() {
    mkdir -p ${cache}
    curl -Ls ${url}/.list | tee ${cache}/.list
}

repocheck() {
    sick -f $HOME/.sick.d/egull.pub
}

repotrust() {
    curl -sL ${url}/.key > ${keyring}/${name}.pub
}

repoget() {
    pkg="$1"
    ver=$(grep -E "^${pkg}  " ${cache}/.list | tac | sed 1q | cut -f2)

    file="${pkg}#${ver}.tar.bz2"
    html="${pkg}%23${ver}.tar.bz2"

    curl -sL ${url}/${html} | repocheck | ifne sponge ${cache}/${file}
    test -f ${cache}/${file} \
        && readlink -f ${cache}/${file} \
        || echo "$pkg: signature check failed" >&2
}

repolist() {
    pg -e ${cache}/.list
}

test $# -eq 0 && { repolist && exit 0; }

case $1 in
    -s) reposync; exit 0 ;;
    -l) repolist; exit 0 ;;
    -t) test -z "$2" && usage || { repotrust $2; exit 0; } ;;
esac

for n in $@; do
    repoget $n
done

exit 0

First, we retrieve the ed25519 public key from the repo:

$ repo -t z3bra

Then, the package list:

$ repo -s
libwm   1.1
skroll  0.6

And finally, install it!

$ ROOT=$HOME/.local
$ export ROOT
$ pm -a $(repo skroll)
$ pm -i
skroll  0.6
$ echo amazing! | skroll

Conclusion

Ok, so this whole post was rather long, and not especially good at describing the actual workflow. So as usual. Here is a quick video to show off the whole process!

Packaging, deploying and installing skroll on my system