Make your own distro
17 January, 2016
Happy new year everyone! For the new year I've decided to focus more on
projects, and less on "ricing" which bores me more and more...
So here is a fun project I'm working on: Making my own linux distro!
I consider building a linux distribution from scratch to be one of the coolest things one could do with their computers. And I finally got into it!
I've been writing my "tales" in my journal. Be sure to read it for a more "day-to-day" like approach. This post is only meant to detail the tools used and the workflow I have when I work on it.
First off all, I have never built an LFS system, and will probably never do (their documentation is great though).
So here I am, sitting in front of my computer, deciding what this "new distro" should be. I've always liked the idea of sta.li, an similar distributions ala rlsd2. Minimalist systems trying to bring fresh new ideas on the table of linux distributions. So I decided to list the things I wanted in my distro:
- statically compiled
- stripped down userland
- simplified file system
- port-tree featuring plan9's mk(1)
- featuring new/obscure softwares when possible
I chose each point because I though it would be and interresting challenge, allowing me to learn more about linux distributions in the process. I'm not trying to argue that X is better than Y, or anyhing amongst those lines.
There are, from my experience, three important toolchains when building a distro:
- Compilation toolchain
- Package management toolchain
- Testing toolchain
Each one is as important as the two others, because they will define how efficient you'll be for the whole project.
I am using pcc as my complier of choice, and musl as my libc.
To this day, I couldn't find a suitable replacement for GNU's binutils, but am still looking forward to elfutils and elftoolchain.
To make it easier to work with this toolchain, I decided to build a cross-compiler featuring both. I also had to create one using gcc, until I can figure out a patch for softwares that can't compile with pcc. The script used to generate the cross-compiler can be found here.
In order to use it, add the path to the toolchain in your path. Calling
x86_64-linux-musl-gcc will automatically call libs/include from the cross
Using the cross-compiler to build a software statically against musl libc.
Package management toolchain
There are a lot of them. But the most challenging one will be the one I'll write myself. This is why I wrote pm, my pack manager. It's still a work in progress, but it can install, delete and inspect packs in a given directory, which is perfect for my needs.
For the packaging of softwares, I decided to go with an mk(1) based port-tree ala BSD. Inspired by the morpheus port-tree, it follows the way the *BSD handle their own port-tree, while using mk(1) from plan9, which is, in my opinion, more predictable. For more information on mk, check the link maintaining files on plan9 with mk.
The ports are available here. To build
a port, you need to configure the
config.mk file to match your environment, and
mk <port>. This will build all the deps required to build you port, and
install them under your cross-compilation root directory.
To clean the tree from this deps, run
mk clean. The tarball will be packed in the
port directory of your tool, eg
pm/pm#1.0.tar.bz. Theses file are suitable for
installation via pm.
Show casing the creation of a port within the mk port-tree and building/installing it under a specific directory
To actually test the distro, you need a way to boot from it. I chose to use linux containers (LXC) over traditionnal VMs (virtualbox, qemu) as I'm not playing with different CPU architectures (yet!). This way I can treat my distro as a simple chroot and modify it from outside in real time. It also let me play with those containerization technologies I'm not familiar with.
The first thing to do is to install the packs under your chroot. For this purpose, I
/ns/pm/rootfs. After installing a few packs, you can fire up a container
using this directory as its root to check wether the distro boots, and if the tools
you installed work as expected.
As of today, spawning gettys doesn't work. I'll need to figure out why, but otherwise, it's been a pretty successful (and fun!) experience.
Demonstrating the whole testing process, from creating the containers and saying "hello world!" from within this container
In the process, I learnt how to spawn containers "the hard way". Here is a quick peak at it:
ip link add veth0 type veth peer name eth0 ip netns add foo ip link set eth0 netns foo ip netns exec foo ip addr add 192.168.0.100/24 dev eth0 broadcast + ip netns exec foo ip link set eth0 up ip netns exec foo ip route add default via 192.168.0.254 brctl addif br0 veth0 ip netns exec unshare -fpium --mount-proc env -i chroot /ns/pm/rootfs /sbin/init
This is far from a final-run-everywhere solution of course, because you'll miss /dev mounts, ttys and such, and won't be able to log within this container, or inject processes in to it.