Setting up a board farm for postmarketOS

I've recently been working on putting together a CI system for postmarketOS that will allow us to do proper automated integration testing. That is to say - when someone opens a merge request that modifies our initramfs (for example), we should be able to click a button and some minutes later know that this change doesn't break any of our important usecases.

QEMU absolutely can (and will) get us most of the way there, but at some point we need to just run the same software that we're running on end user devices. Furthermore, QEMU can't tell us anything about changes in the kernel that might affect our devices, and manually testing during kernel upgrades, frankly, sucks.

So we need a fancy board farm, this is one of those things where folks with the right technical background could build something over the course of a week. But for someone like me it's full of trial and error and hidden complexity... It's easy enough to do this with one device - just hack something together, but to be successful we need something reliable and adaptable, that we can adjust to fit our needs in the future, and the wide range of devices we support.

Now this is an article you won't come across very often, as the number of people setting up something like this who can actually talk openly about it - someone doing this for a closed company probably can't - is probably quite small. A great read.