Monochromatic

z3bra, the stripes apart

Make your own distro

17 January, 2016

Happy new year everyone! For the new year I've decided to focus more on projects, and less on "ricing" which bores me more and more...
So here is a fun project I'm working on: Making my own linux distro!

I consider building a linux distribution from scratch to be one of the coolest things one could do with their computers. And I finally got into it!

I've been writing my "tales" in my journal. Be sure to read it for a more "day-to-day" like approach. This post is only meant to detail the tools used and the workflow I have when I work on it.

First off all, I have never built an LFS system, and will probably never do (their documentation is great though).

Scope

So here I am, sitting in front of my computer, deciding what this "new distro" should be. I've always liked the idea of sta.li, an similar distributions ala rlsd2. Minimalist systems trying to bring fresh new ideas on the table of linux distributions. So I decided to list the things I wanted in my distro:

I chose each point because I though it would be and interresting challenge, allowing me to learn more about linux distributions in the process. I'm not trying to argue that X is better than Y, or anyhing amongst those lines.

Toolchains

There are, from my experience, three important toolchains when building a distro:

Each one is as important as the two others, because they will define how efficient you'll be for the whole project.

Compilation toolchain

I am using pcc as my complier of choice, and musl as my libc.

To this day, I couldn't find a suitable replacement for GNU's binutils, but am still looking forward to elfutils and elftoolchain.

To make it easier to work with this toolchain, I decided to build a cross-compiler featuring both. I also had to create one using gcc, until I can figure out a patch for softwares that can't compile with pcc. The script used to generate the cross-compiler can be found here.

In order to use it, add the path to the toolchain in your path. Calling x86_64-linux-musl-gcc will automatically call libs/include from the cross prefix.

Using the cross-compiler to build a software statically against musl libc.

Package management toolchain

There are a lot of them. But the most challenging one will be the one I'll write myself. This is why I wrote pm, my pack manager. It's still a work in progress, but it can install, delete and inspect packs in a given directory, which is perfect for my needs.

For the packaging of softwares, I decided to go with an mk(1) based port-tree ala BSD. Inspired by the morpheus port-tree, it follows the way the *BSD handle their own port-tree, while using mk(1) from plan9, which is, in my opinion, more predictable. For more information on mk, check the link maintaining files on plan9 with mk.

The ports are available here. To build a port, you need to configure the config.mk file to match your environment, and run mk <port>. This will build all the deps required to build you port, and install them under your cross-compilation root directory. To clean the tree from this deps, run mk clean. The tarball will be packed in the port directory of your tool, eg pm/pm#1.0.tar.bz. Theses file are suitable for installation via pm.

Show casing the creation of a port within the mk port-tree and building/installing it under a specific directory

Testing toolchain

To actually test the distro, you need a way to boot from it. I chose to use linux containers (LXC) over traditionnal VMs (virtualbox, qemu) as I'm not playing with different CPU architectures (yet!). This way I can treat my distro as a simple chroot and modify it from outside in real time. It also let me play with those containerization technologies I'm not familiar with.

The first thing to do is to install the packs under your chroot. For this purpose, I chose /ns/pm/rootfs. After installing a few packs, you can fire up a container using this directory as its root to check wether the distro boots, and if the tools you installed work as expected.

As of today, spawning gettys doesn't work. I'll need to figure out why, but otherwise, it's been a pretty successful (and fun!) experience.

Demonstrating the whole testing process, from creating the containers and saying "hello world!" from within this container

In the process, I learnt how to spawn containers "the hard way". Here is a quick peak at it:

ip link add veth0 type veth peer name eth0
ip netns add foo
ip link set eth0 netns foo
ip netns exec foo ip addr add 192.168.0.100/24 dev eth0 broadcast +
ip netns exec foo ip link set eth0 up
ip netns exec foo ip route add default via 192.168.0.254
brctl addif br0 veth0
ip netns exec unshare -fpium --mount-proc env -i chroot /ns/pm/rootfs /sbin/init

This is far from a final-run-everywhere solution of course, because you'll miss /dev mounts, ttys and such, and won't be able to log within this container, or inject processes in to it.

But.

That's neat.