NixOS on ARM

This is a WIP to cross compile NixOS to run on ARM targets.

Building
Burning to an SD Card
Burning to the eMMC
- Adding Burner Support to Your Board
NixOps Deployments
Debugging
Images Overview
- Image Templates
- Demos
Contributing
Current State

Building

Clone the latest release:

git clone -b 0.7.0 --recursive --shallow-submodules https://github.com/illegalprime/nixos-on-arm.git
cd nixos-on-arm

This repository was reorganized to be able to build different boards if/when different ones are written. To build use:

nix build -f . \
  -I nixpkgs=nixpkgs \
  -I machine=machines/BOARD_TYPE \
  -I image=images/NIX_CONFIGURATION

Using Cachix

This repository uses Travis to keep a fresh cachix cache, which you can use to speed up your builds:

# install cachix if you haven't already
nix-env -iA cachix -f https://cachix.org/api/v1/install
# use this cache when building
cachix use cross-armed

A Note on Image Size

Many things affect image size and recently a lot of work has been done to minimize it:

splitting gcc libs into different output (NixOS/nixpkgs#58606)
strip in cross builds (NixOS/nixpkgs#59787) (NixOS/nixpkgs#21667 (comment)) (NixOS/nixpkgs#15339)

A lot of things still have to be done to remove x86 remnants from accidentally getting into the image (like updating patchShebangs NixOS/nixpkgs#33956), and contaminants can be checked by running ./check-contamination.sh result.

See the Images Overview for a breakdown of image sizes.

BeagleBone Green

nix build -f . \
  -I nixpkgs=nixpkgs \
  -I machine=machines/beaglebone \
  -I image=images/ap-puns

Currently images/ap-puns provides a service which will send out AP beacons of WiFi puns. This is a demo showing how one can build their own OS configured to do something out-of-the-box. (NOTE you need a USB WiFi dongle, I included kernel modules for the Ralink chipset)

I think it's neat, much better than installing a generic Linux and configuring services yourself on the target.

UniFi Controller

You can build an image which starts a UniFi controller so you don't have to buy one! This is useful if you have a UniFi router or AP, which uses this controller for extra memory and processing power. Currently tested with the beaglebone:

nix build -f . \
  -I nixpkgs=nixpkgs \
  -I machine=machines/beaglebone \
  -I image=images/unifi

Since the beaglebone is slow, it could take a while to boot.

Raspberry Pi Zero (W)

Both raspberry pi zeros are supported now! They come with cool OTG features:

nix build -f . \
  -I nixpkgs=nixpkgs \
  -I machine=machines/raspberrypi-zerow \
  -I image=images/rpi0-otg-serial

This will let you power and access the Raspberry Pi via serial through it's USB port. Be sure to plug your micro USB cable in the data port, not the power port.

The first boot takes longer since it resizes the SD card to fill its entire space, so the serial device (usually /dev/ttyACM0) might take longer to show up.

You can also build an image with turns the USB port into an Ethernet adapter, letting you SSH into the raspberry pi by plugging it into your computer:

nix build -f . \
  -I nixpkgs=nixpkgs \
  -I machine=machines/raspberrypi-zerow \
  -I image=images/rpi0-otg-ether

copy it to an SD card ('Installing' section), plug it in, wait for it to boot and to show up as an Ethernet device, then just:

ssh root@10.0.3.1

Raspberry Pi 2

The Raspberry Pi 2 is a 32-bit quad core @ 900MHz ARMv7 board with 1GB of RAM. You can build for the Raspberry Pi 2, and it works with NixOps!

nix build -f . \
  -I nixpkgs=nixpkgs \
  -I machine=machines/raspberrypi-2 \
  -I image=images/nixops

Raspberry Pi 3

The Raspberry Pi 3 is a 64-bit quad core @ 1.2GHz ARMv8 board with 1GB of RAM. It also has built-in 802.11n WiFi, and 10/100 Ethernet port. Bluetooth doesn't work on this platform yet. You can build for the Raspberry Pi 3, and it works with NixOps!

nix build -f . \
  -I nixpkgs=nixpkgs \
  -I machine=machines/raspberrypi-3 \
  -I image=images/nixops

Odroid C2

This was a really interesting board to work on and a lot of help was taken from jumpnow/meta-odroid-c2. It's a good example of how to build u-boot, sign it, and couple it with vendor-specific boot loader code. This is a pretty good reference implementation for secure-boot and 64-bit arm boards. Build it with:

nix build -f . \
  -I nixpkgs=nixpkgs \
  -I machine=machines/odroid-c2 \
  -I image=images/ssh

I haven't implemented building an SD burner for this board yet, but it should be straightforward to do and it will be implemented once I buy an eMMC.

Toradex Apalis IMX6 (Community)

Board configurations for this just landed thanks to @deadloko! I do not own this board so I cannot test it on every release, but it should be similar to the BeagleBone. Build it with:

nix build -f . \
  -I nixpkgs=nixpkgs \
  -I machine=machines/toradex_apalis_imx6 \
  -I image=images/mini

Burning to an SD Card

bmap is really handy here (nix-shell -p bmap-tools).

sudo bmaptool copy --nobmap result/sd-image/*.img /dev/sdX

Burning to the eMMC

When your image is all ironed out you might want to store it in a more permanent place on your board: the eMMC. This type of storage is great because it can't be easily dislodged like an SD card, but it's harder to access.

If you have an SD card port and an eMMC you're in luck, this repository defines an output (a directory in outputs) that will build an SD card image that will boot and burn another image onto the eMMC. You specify the image you want burned with the usual -I image= option.

We now use bmaptool to burn the image over which is faster, has integrity checks, and makes sure to sync the file system. We also compress the image so that it can be safely copied over to the burner image.

All you have to do is use the burner output:

nix build \
  -f outputs/burner \
  -I nixpkgs=nixpkgs \
  -I machine=machines/beaglebone \
  -I image=images/nixops

Burn the result to an SD card (see Burning to an SD Card) and boot into it. If LEDs for this board were configured, you should see one of the following patterns:

Each LED lighting up one by one in sequence means the eMMC is being written to.
All LEDs on means the write has finished successfully, the board will soon be powered off.
All LEDs blinking slowly means an error has occurred during the writing process.

(note that on the BeagleBone you must hold the USER button down, plug in power, then let go to boot into your SD card if there's already a boot loader on the eMMC)

Adding Burner Support to Your Board

If you're writing the definition for a board you might want to enable support for this feature, to do so just implement the options in the crosspkgs/modules/hardware/burner module, which at the time of writing consists of only a couple of options:

hardware.burner.disk: the path to dd the image into (the path of the eMMC device)
hardware.burner.preBurnScript: a script to run before dd is called

You may also define LEDs in the crosspkgs/modules/hardware/leds module, which the burner script will use to display its status. LEDs are just names of directories in the /sys/class/leds/ directory.

Take a look at the beaglebone image definition if you want a concrete example.

NixOps Deployments

NixOps support is here! That means you can manage all your devices without re-flashing!

The Setup

To start burn a base image onto an SD card (this image just contains SSH and a larger boot partition to store new configurations in):

nix build -f . \
  -I nixpkgs=nixpkgs \
  -I machine=machines/beaglebone \
  -I image=images/nixops

Then boot it and add your SSH key because NixOps only supports SSH keys:

# copy over the key
ssh-copy-id root@IP_ADDRESS

# load your SSH keys into your environment
eval $(ssh-agent -s)
ssh-add ~/.ssh/id_rsa

Finally, create a NixOps deployment for your devices (iot can be changed to anything):

nixops create ./nixops.nix -d iot

Deploying New Configurations

Now you're ready to update your board with some new configurations! Just use nixops deploy with the IP environment variable set to the your device's (here we deploy the ap-puns image over our original nixops image):

IP=YOUR_DEVICES_IP \
  nixops deploy \
  -I nixpkgs=nixpkgs \
  -I machine=machines/beaglebone \
  -I image=images/ap-puns \
  -d iot

If you have access to a serial connection, you can pick your generation on boot:

------------------------------------------------------------
1:      NixOS - Default
2:      NixOS - Configuration 2 (2019-09-16 14:23 - 19.09pre-git)
3:      NixOS - Configuration 1 (1970-01-01 01:32 - 19.09pre-git)
Enter choice:

Otherwise you can take out the SD card and manually edit the /extlinux/extlinux.conf file in the boot partition. Replace DEFAULT nixos-default with DEFAULT nixos-42 in that file to temporarily boot from the 42nd generation.

Debugging

Sometimes when thing go wrong you need to test specific parts of the build, this repository is organized so it's easy to do that.

Let's say dhcp was broken, you can build just that package with:

nix build -f . \
  -I nixpkgs=nixpkgs \
  -I machine=machines/beaglebone \
  -I image=images/mini \
  pkgs.dhcp

Similarly you can drop into a shell to inspect the build process for dhcp like:

nix-shell --pure . \
  -I nixpkgs=nixpkgs \
  -I machine=machines/beaglebone \
  -I image=images/mini \
  -A pkgs.dhcp

then you can just call genericBuild in the nix-shell and simulate building that package.

If you wanted to inspect the final configuration values and other stuff, you can drop into a repl:

nix repl . \
  -I nixpkgs=nixpkgs \
  -I machine=machines/beaglebone \
  -I image=images/mini \

Then the variable config contains the system configuration.

Images Overview

Some images are full-fledged demos with a use case, and others are just templates for you to build your own images with.

Image Templates

(the size is based off BeagleBone builds)

Name	Size	Description
base	> 2GB	the smallest changes to the nix configuration needed to cross-build
mini	584MB	smaller than base, with most non-critical services turned off, like `polkit`, `udisks`, `containers`, etc.
micro	564MB	smaller than mini, meant to be flashed once and not updated directly (but updated by flashing another image)
ssh	584MB	based off mini but with SSH access

The micro image isn't very micro right now, but hopefully it will be soon. It's meant to not have any nix utilities or the daemon, a smaller kernel, and generally the bare minimum needed to run on the board. Currently, it's not very different from the mini image.

Demos

ap-puns use aircrack-ng to send out fake AP Beacons with pun names
rpi0-otg-ether run a Raspberry Pi 0 as an Ethernet Adapter with SSH in OTG mode
rpi0-otg-serial run a Raspberry Pi 0 as a serial adapter
unifi boot into a UniFi controller that manages UniFi APs

Contributing

For inspiration either look at the currently open issues or What Needs to Be Done. Otherwise just try it out and put in fixes as you find them, ultimately all fixes that end up here will be sent upstream so all of nixpkgs can benefit.

Alternatively, send it directly upstream and link the commit in an issue, it will possibly be cherry-picked here.

Current State

What Works

BeagleBone Green (and now the Raspberry Pi Zero & Zero W!)
Networking & SSH
the BeagleBone's UART (Raspberry Pi Zero's serial port)
a bunch of standalone packages (vim, nmap, git, gcc, python, etc.)
all the nix utilities!
the USB port!

What Doesn't Work

nix channels are also not packaged with the image for some reason do nix-channel --update
there's still a good amount of x86 stuff that gets in there accidentally
bluetooth on the raspberry pi zeros (and likely on all the other platforms)

rajivr/nixos-on-arm