Category: linux

  • Managing secrets in Nix using Agenix

    If tomorrow my laptop were to go in flames, I would lose two things. First I would lose my data, and this is what backups are for. The second thing I would lose is my configuration (software installed, network setup, …). Nix offers a way to preserve this (assuming the configuration is replicated externally like GitHub).

    But storing one’s configuration is Nix comes with challenges. One of those challenges is how to handle secrets. Some parts of the configuration is tied to secrets : passwords, keys, API tokens, … . The question is how to manage this. Usually it requires some form of cryptography.

    How I store my nix configuration

    Before I explain Agenix, I think it is worth to explain how I manage my Nix configuration on the various machines I have installed Nix. This is because Agenix integrates to my nix configuration, and Agenix makes more sense in context.

    There is the file layout of my Nix configuration over at (GitHub)[https://github.com/mparusinski/nix-config/]

    .
    ├── build.sh
    ├── flake.lock
    ├── flake.nix
    ├── modules
    │   ├── nixos
    │   │   ├── users.nix 
    │   │   ├── ...
    │   │   └── gc.nix
    │   └── home-manager
    │       ├── git.nix
    │       ├── ...
    │       └── zsh.nix
    └── hosts
        ├── dell-precision-7530
        │   ├── configuration.nix
        │   ├── hardware-configuration.nix
        │   └── home.nix
        └── wsl1
            ├── configuration.nix
            └── home.nix
    
    

    There are two parts each divided into two sub-parts : * The build system build on top of Nix and Flakes * flake.nix and flake.lock describe how to build a nix configuration on a given system. * build.sh which launches the appropriate Nix build command for the underlying system. * Nix modules which defines the system configuration * modules defines reusable configuration code across various systems * hosts defines configuration for given systems.

    Naively secrets (passwords, keys, API tokens, …) would be stored in plain text somewhere in the files mentioned here.

    Agenix explained

    To include secrets safely in a Git repository storing configuration there are multiple options. A common choice to externalize secrets. Essentially storing them not in Git. That could be simple files one has to place manually, key vaults, etc. But in that case you can’t simply replicate configuration unless you maintain those secrets. Another choice is to encrypt the secrets before hand, and, it has to be said, making sure the encryption key does not end up in Git.

    Agenix offers a way to include those secrets encrypted in Git. Furthermore Agenix is aware of actors involved. By actors I mean the repository maintainer, i.e. you, the systems that will have to decrypt the secrets, …

    Agenix is build on top of age which is an encryption tool. Its particularity is Age encrypted secrets can have multiple recipients and can use SSH asymmetric encryption (a key for encryption, a key for decryption). Agenix uses age to make the secret decryptable by authorized actors : you and the systems (which have their own SSH keys) that need to access the secrets and no one else.

    To be used Agenix needs to be installed to encrypt secrets but you also need to specify the nix configuration to setup agenix (so it can manage secrets) which amount of having it installed, the secrets specified and when to use them.

    Managing secrets

    The first step is to install Nix. In my case I am using flakes so I followed the steps from install guide for flakes.

    So more details about using flakes for your NixOS configuration see this link If you want to pass agenix to your configuration module file (usually named configuration.nix) you need to specify the specialArgs = { inherit inputs; } option when calling nixpkgs.lib.nixosSystem. To check if agenix is correctly installed run agenix -h.

    Now that agenix is installed the next step is to setup secrets. Inside your nix configuration folder do the following

    mkdir secrets
    cd secrets
    touch secrets.nix

    This means that the secrets.nix file will be version controlled. And start editing the secrets.nix file.

    let
      # SSH public keys of the users who will be adding secrets through agenix
      john = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJCizmOo5KevfHd6pqwxVjgvVYWv4Az5TbAclvuhF2AC";
      users = [ john ];
    
      # SSH public keys of the systems who will need to access the secrets
      webserver = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDA1+TbC/tXsVAKUjSzipoC0ibOgSWuNvzVdb8Xxwi0T";
      systems = [ webserver ];
    in
      {
        # Specify for each secret who has access to it
        "apikey.age".publicKeys = users ++ systems;
      }

    Once this is done you can create the secret files. In the above example that would agenix -e apikey.age. This file contains the encrypted secret to be used.

    Once you have done you will need to specify the secrets twice in the configuration nix files: once to declare the secret with the location of the age encrypted file, the second time to link the secret to some configuration item.

    {
      # configuration.nix
      # ...
      age.secrets.apikey.file = relativepathto/secrets/apikey.age;
      # ...
      # some service requiring the above secret
      services.special.config.adminFile = config.age.secrets.apikey.path;
    }

    Limitations

    From my experience sometimes Agenix does not always work smoothly with Nix. For instance in one case the nix build system would invalidate a prometheus configuration because the Agenix path would not exist yet. This had to with the prometheus Nix module validating the generate configuration yaml (see here).

  • Switching systemd-boot to GRUB on Archlinux

    I intend to use BTRFS snapshots, snapper and pac-snap to be able to rollback archlinux if an update goes wrong. However current it uses systemd-boot and the tools I intend to use required GRUB.

    Requirements

    This tutorial is only meant for UEFI systems. It requires a EFI partition mounted under /boot with enough disk space (in my case 512 MB should be enough).

    Install efibootmgr to validate changes

    Swapping to GRUB with BTRFS support

    Install grub and run

    grub-install --target=x86_64-efi --efi-directory=/boot --bootloader-id=GRUB

    Edit the file /etc/default/grub and copy over the relevant fields from the systemd boot loader usually found under /boot/loader/entries/arch.conf

    For instance if your systemd-boot config looks like this

    title   Arch Linux
    linux   /vmlinuz-linux
    initrd  /intel-ucode.img
    initrd  /initramfs-linux.img
    options cryptdevice=UUID=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:crypt root=/dev/mapper/crypt rootflags=subvol=arch_root splash rw

    You need to copy options field to GRUB_CMDLINE_LINUX but do not copy root

    Then generate the GRUB configuration grub-mkconfig -o /boot/grub/grub.cfg and check the content of the file.

  • Switching between High DPI and Low DPI in XMonad

    Context

    I enjoying using a tiling window manager on Linux. However it is hard to get them to work with my multiple monitor requirements.

    My situation can be described as follows: 1. I have multiple monitors, not all of them have the same DPI. 2. Not all of them are connected at the same time as I use a laptop. Sometimes I no monitors, sometimes three. And I like to plug them while the session is open. 3. Whenever possible I would like to have the highest DPI setting possible.

    DPI means dots per inch and loosely speaking refers to resolution. High DPI monitors have such a high resolution they require bigger fonts and upscaling.

    In general on Linux, Wayland based window managers and desktop environments, like Gnome or Hyprland handle the best high DPI settings. They support fractional scaling (for when monitors are halfway between low and high DPI). They work nicely with mixed setups like described above.

    However I am not yet satisfied with either Gnome (which isn’t a dynamic tiling window manager), or Hyprland (which seems to have a static approach to multiple monitors). And I haven’t tried Sway yet. In the future I planned to revisit Sway and Hyprland but in the meantime I wanted to get XMonad to work well because I like how XMonad handle multiple monitors.

    Finding out current DPI

    I wrote a simple script using xdpyinfo to fetch information about the current DPI:

    #!/usr/bin/env bash
    xdpyinfo | grep dots | tr -s ' ' | cut -d ' ' -f 3 | cut -d 'x' -f 1

    This will return a number giving the current DPI setting. The purpose of this script to allow XMonad to tell the DPI setting and adapt accordingly. In my case I use it to tell which XMobar config to load (either a low DPI setting or a high DPI setting).

    Creating configs for each DPI setting

    The main trick I use for multiple DPI setting is having multiple config files for each DPI config. The files I differentiate are: * .xsession files for loading Xft settings * xmobarrc files for the XMobar status bar (with different dpi settings)

    So for instance here is the two x session files

    FileLow DPIHigh DPI
    X Session fileXft.dpi: 96 Xft.autohint: 0 Xft.lcdfilter: lcddefault Xft.hintstyle: hintfull Xft.hinting: 1 Xft.antialias: 1 Xft.rgba: rgbXft.dpi: 192 Xft.autohint: 0 Xft.lcdfilter: lcddefault Xft.hintstyle: hintfull Xft.hinting: 1 Xft.antialias: 1 Xft.rgba: rgb
    XMobar configConfig { overrideRedirect = False , font = "Fira Code 12" , additionalFonts = [ "Font Awesome 6 Free Solid 12" , "Font Awesome 6 Brands 12" ] , bgColor = "#2E383C" , fgColor = "#D3C6AA" , dpi = 96 , position = TopH 30 , commands = [ Run Cpu [ "-L", "5" , "-H", "50" , "--high", "red" , "--low", "green" ] 10 , Run Battery [ "--template", ": % - " , "--" , "-O", "AC" , "-o", "Bat" , "--high", "green" , "--low", "red" ] 10 , Run Memory ["--template", "Mem: %"] 10 , Run Date "%a %Y-%m-%d %H:%M" "date" 10 , Run XMonadLog ] , sepChar = "%" , alignSep = "}{" , template = "%XMonadLog% }{ %battery% | %cpu% | %memory% | %date% " }Config { overrideRedirect = False , font = "Fira Code 12" , additionalFonts = [ "Font Awesome 6 Free Solid 12" , "Font Awesome 6 Brands 12" ] , bgColor = "#2E383C" , fgColor = "#D3C6AA" , dpi = 192 , position = TopH 60 , commands = [ Run Cpu [ "-L", "5" , "-H", "50" , "--high", "red" , "--low", "green" ] 10 , Run Battery [ "--template", ": % - " , "--" , "-O", "AC" , "-o", "Bat" , "--high", "green" , "--low", "red" ] 10 , Run Memory ["--template", "Mem: %"] 10 , Run Date "%a %Y-%m-%d %H:%M" "date" 10 , Run XMonadLog ] , sepChar = "%" , alignSep = "}{" , template = "%XMonadLog% }{ %battery% | %cpu% | %memory% | %date% " }

    The main notable thing is that the font dpi changes in both case. The rest is not important in this context.

    Toggling between DPI settings

    To switch between the two DPI a second script is created :

    #!/usr/bin/env bash
    
    DPI=$(~/.local/bin/what-dpi.sh)
    if [ "$DPI" -eq "192" ]; then
        echo "Switching to low DPI (96)"
        xrdb -merge ~/.config/xsession/xsession.lowdpi
        xrandr -s 1920x1080
    else
        echo "Switching to high DPI (192)"
        xrdb -merge ~/.config/xsession/xsession.hidpi
        xrandr -s 3840x2160
    fi
    
    

    This script picks up the current DPI setting and applies the new setting matching the opposite DPI setting. It does three things : 1. Change the Xft settings for fonts 2. Change the screen resolution

    Incorporating in XMonad

    Now we incorporate the logic in XMonad. First I define a data type (to follow good programming practices)

    data DPI = HiDPI | LowDPI

    I create a status bar hook that takes a DPI as an input (and monitor number) and creates the appropriate instances of xmobar:

    import XMonad.Hooks.StatusBar
    import XMonad.Hooks.StatusBar.PP
    
    mySB num dpi = statusBarProp ( "xmobar -x " ++ show num ++ dpiConf dpi) (pure myXmobarPP)
      where dpiConf LowDPI = " ~/.config/xmobar/xmobarrc.lowdpi"
            dpiConf HiDPI  = " ~/.config/xmobar/xmobarrc.hidpi"
    
    myXmobarPP :: PP
    myXmobarPP = def 
      { ppSep             = " | "
      , ppCurrent         = xmobarColor "#7FBBB3" "" . wrap "[" "]" 
      , ppVisible         = xmobarColor "#A7C080" "" . wrap "[" "]" 
      , ppHidden          = xmobarColor "#E67E80" "" . wrap "(" ")" 
      , ppHiddenNoWindows = xmobarColor "#D3C6AA" "" . wrap " " " " 
      , ppTitle           = xmobarColor "#D3C6AA" "" . shorten 60
      , ppTitleSanitize   = xmobarStrip
      }
    
    

    I then hook this into the main script

    fetchDPI :: IO DPI 
    fetchDPI = do
      (_, maybeOutH, _, _) <- createProcess shellCmd { std_out = CreatePipe }
      maybe (return LowDPI) processOut maybeOutH
      where shellCmd = shell "~/.local/bin/what-dpi.sh"
            processOut :: Handle -> IO DPI 
            processOut h = do
              output <- hGetLine h
              let currDpi = readMaybe output :: Maybe Int 
              return $ maybe HiDPI intToDPIEnum currDpi
              where intToDPIEnum x = if x == 192 then HiDPI else LowDPI
    
    main :: IO ()
    main = do
      dpi <- fetchDPI
      xmonadMain dpi 
      where xmonadMain dpi = xmonad
                           . ewmhFullscreen
                           . ewmh
                           . withEasySB (mySB 0 dpi) toggleStrutsKey
                           . withEasySB (mySB 1 dpi) toggleStrutsKey
                           $ myConfig dpi 
                           where toggleStrutsKey :: XConfig Layout -> (KeyMask, KeySym)
                                 toggleStrutsKey XConfig { modMask = myModMask } = (myModMask, xK_b)

    So let’s explain what happens : 1. When XMonad launches it fetches the current DPI from the what-dpi.sh script. This part is done in fetchDPI and it is basically a lot of boilerplate code to start a process and return the DPI setting with lots of error off ramps to default to a low DPI setting (in exotic cases it is safer to assume low DPI) 2. Launch the status bar using withEasySB multiple times as we may have multiple monitors with the appropriate DPI setting.

    Now this allows XMonad upon start to adapt but not when monitors are plugged in or plugged out. For this I create a keyboard shortcut as follows

    myKeys     =
      [ ("M-v",                     toggleSmartSpacing)
      , ("<XF86MonBrightnessUp>",   spawn "brightnessctl s +10%")
      , ("<XF86MonBrightnessDown>", spawn "brightnessctl s 10-%")
      , ("<XF86AudioRaiseVolume>",  raiseVolume 3 >> return ())
      , ("<XF86AudioLowerVolume>",  lowerVolume 3 >> return ())
      , ("<XF86AudioMute>",         toggleMute >> return ())
      , ("C-<Print>",               spawn "scrot -s")
      , ("C-'",                     spawn "~/.local/bin/switch-dpi.sh; killall xmobar; xmonad --restart")
      ]

    Here I add a keyboard shortcut Control-’ to request a DPI switch, kill xmobar and restart xmonad.

  • Managing databases users using nix

    Recently I had the following challenge: manage database (robotic) users using nix. More specifically I wanted to declare a list of MariaDB/MySQL users, alongside passwords, and permissions using nix. And while nix has some options for this (e.g. services.mysql.ensureUsers.*.name). Some aspects can’t be declared with nix options like the credentials or its permissions for given databases.

    There is a way to solve this using nix using three steps: 1. Declaring the user configuration in an object 2. Creating nix packaged scripts that will apply a configuration to the database 3. Creating a systemd unit that runs the previous scripts when appropriate

    In this article I will go into details on how this is done. Note: although I focus on MySQL/MariaDB, the underlying principles can be applied for PostgreSQL, Redis, …

    Declaring the user configuration

    The first step is to encapsulate the information required for configuring a user in a nix object (key-value dictionary). In my case it looks like this

    dbUsecaseConfig = {
      database = "usecase_db";
      passwordFile = config.age.secrets.usecasePass.path;
      username = "usecase_username";
      hostname = "usecase_hostname";
    };

    Should be self explanatory. Note passwordFile is a path to a file, here I use agenix to manage secrets, which decrypts the password file with adequate permissions to prevent leaks (see article about agenix).

    Set up script to apply configuration

    The second step is to create a script that applies the configuration. This is done using pkgs.writeShellScript which generates a shell script and stores it as nix component (under /nix/store/).

    The syntax of the function

    writeShellScript "file-name"
      ''
      Content of the file
      '';

    The file name here is not very important. The content of the script, however, is. To make this modular, I use a function that takes a configuration as input and produces a corresponding shell script:

    applyMySQLConf = 
      {
        database,
        passwordFile,
        username,
        hostname
      }:
      pkgs.writeShellScript "applyMySQLConf"
        ''
          PASSWORD=$(cat ${passwordFile})
          ${pkgs.mariadb}/bin/mysql -u root -e "GRANT ALL PRIVILEGES ON ${database}.* TO '${username}'@'${hostname}' IDENTIFIED BY '$PASSWORD';"
        '';

    The details will depends on the use case. For instance for a PostgreSQL database the actual commands used in the script will change.

    To generate a script for a given configuration will look this:

    dbUsecaseConfig = {
      database = "usecase_db";
      passwordFile = config.age.secrets.usecasePass.path;
      username = "usecase_username";
      hostname = "usecase_hostname";
    };
    usecaseConfScript = applyMySQLConf dbUsecaseConfig;

    Automatically apply the configuration using systemd

    Now, we have a script which applies a configuration, but nothing actually runs the script. This is solved by using systemd services :

    systemd.services.apply-mysql-conf = {
      description = "Apply MySQL configurations";
      wants = [ "mysql.service" ];
      wantedBy = [ "multi-user.target" ];
      serviceConfig = {
        PremissionsStartOnly = true;
        RemainAfterExit = true;
        ExecStart = ''
          ${usecaseConfScript}
          ...
        '';
      };
    };

    I personally use one systemd service to apply all configurations for the database. You may prefer using multiple systemd unit files.

    Creating a Nix module

    It is good practice to create NixOS modules as it allows the code to be reusable and better organised.

    A NixOS module is created by adding path/to/module.nix file.

    {
      lib,
      config,
      pkgs,
      ...
    }:
    {
      # Content of the module
    }

    More specifically the module will define a NixOS option called mysqlInitialConfiguration so it can be used in the configuration.nix:

    {
      lib,
      config,
      pkgs,
      ...
    }:
    {
      options.mysqlInitialConfiguration = {
        enable = lib.mkEnableOption "Activate initial MySQL configuration";
        configurations = lib.mkOption = {
          default = [ ];
          description = ''
            List of initial MySQL configurations
            For instance :
            [
              {
                database = "usecase_db";
                passwordFile = config.age.secrets.usecasePass.path;
                username = "usecase_username";
                hostname = "usecase_hostname";
              }
            ]
          '';
        };
      };
    }

    Now this defines the NixOS option but not what happens when it is enabled and configured. To do so we need to add the config part of the option which applies a configuration based on option.

    {
      lib,
      config,
      pkgs,
      ...
    }:
    {
      options.mysqlInitialConfiguration = {
        enable = lib.mkEnableOption "Activate initial MySQL configuration";
        configurations = lib.mkOption = {
          default = [ ];
          description = ''
            List of initial MySQL configurations
            For instance :
            [
              {
                database = "usecase_db";
                passwordFile = config.age.secrets.usecasePass.path;
                username = "usecase_username";
                hostname = "usecase_hostname";
              }
            ]
          '';
        };
      };
    
      config = lib.mkIf config.mysqlInitialConfiguration {
        # Actual configuration
      };
    }

    Now we add the actual configuration (systemd files, scripts, …) and for practical reasons use the let ... in ... nix clause.

    {
      lib,
      config,
      pkgs,
      ...
    }:
    let
      applyMySQLConf =
        {
          database,
          passwordFile,
          username,
          hostname,
        }:
        pkgs.writeShellScript "applyMySQLConf" ''
          PASSWORD=$(cat ${passwordFile})
          ${pkgs.mariadb}/bin/mysql -u root -e "GRANT ALL PRIVILEGES ON ${database}.* TO '${username}'@'${hostname}' IDENTIFIED BY '$PASSWORD';"
        '';
    
      # Function taking a list as input and returns a string
      genSystemdScript = list: lib.strings.concatStrings (map (conf: applyMySQLConf conf + "\n") list);
      systemdScript = pkgs.writeShellScript "mysqlInitialConfiguration" ''
        ${genSystemdScript cfg.configurations}
      '';
    
      cfg = config.mysqlInitialConfiguration;
    in
    {
      options.mysqlInitialConfiguration = {
        # ...
      };
    
      config = lib.mkIf config.mysqlInitialConfiguration {
        systemd.services.mysqlInitialConfiguration = {
          description = "Initial MySQL configuration";
          wants = [ "mysql.service" ];
          wantedBy = [ "multi-user.target" ];
          serviceConfig = {
            PermissionsStartOnly = true;
            RemainAfterExit = true;
            ExecStart = "${systemdScript}";
          };
        };
      };
    }

    Note that config.mysqlInitialConfiguration is replaced with cfg in the config part.

    Now the new option can be used in the configuration.nix file as so:

    {
      # ...
    }:
    {
      # ...
      imports = [
        # ...
        ./path/to/mysql/module.nix
        # ...
      ];
      # ...
      mysqlInitialConfiguration.enable = true;
      mysqlInitialConfiguration.configurations = [
        {
          database = "usecase_db";
          passwordFile = config.age.secrets.usecasePass.path;
          username = "usecase_username";
          hostname = "usecase_hostname";
        }
      ];
    }

    Summary

    In this article I show to apply MySQL user configuration automatically using Nix and put all of this in a module.

  • Publishing a pelican website using nix

    This website is statically generated using pelican and development on the website is done via nix flakes. Here is how :

    Overview

    I use pelican to publish this website. Pelican is a python based command line tool to generate a static website. Pelican requires writing the articles in reStructuredText or Markdown and placing these files in a folder which I call content. To generate the website I simply run a command

    pelican -r content -o output -s pelicanconf.py

    I will not cover the details on how to install Pelican. In the above command I specify the source folder content, the output destination folder and finally a configuration file called pelicanconf.py that contain various website information like the name of website, the locale or the theming.

    While pelican is simple, it still requires a lot of small steps every time you want to publish an article. Which is why I use nix to automate as much as possible to leave as little friction.

    I will explain how I use Nix flakes to manage dependencies, build the website and run it locally. I will leave publishing for possible future article.

    Creating a development environment (nix-shell)

    The purpose of a nix shell is to have a shell environment where all the tooling is installed and one can develop various helper scripts (e.g. script to run the website locally).

    First I create the flake.nix file:

    {
        description = "Simple blog project";
        inputs = {
            nixpkgs.url = "github:NixOS/nixpkgs/nixos-24.11";
        }
        outputs = { self, nixpkgs }: let
            system = "x86_64-linux";
            pkgs = import nixpkgs { inherit system; }
            deps = with pkgs; [
                (python3.withPackages (ps: [ ps.pelican ps.markdown ]))
            ];
        in {
            devShells.${system} = pkgs.mkShell {
                buildInputs = deps;
            };
        };
    }

    Basically this tells how to make a shell environment with pelican installed and markdown support. It can be invoked like this:

    nix develop
    pelican -r ./content -o ./dev-output -s pelicanconf.py

    However, this may not work for everyone because I use an external theme.

    When it is not a flake project (i.e the project root does not have a flake.nix) we turn the external dependency into a derivation from scratch. This can be done using fetchFromGitHub:

    {
        description = "Simple blog project";
        inputs = {
            nixpkgs.url = "github:NixOS/nixpkgs/nixos-24.11";
        }
        outputs = { self, nixpkgs }: let
            system = "x86_64-linux";
            pkgs = import nixpkgs { inherit system; }
            pelican-flex-theme = pkgs.fetchFromGitHub {
                owner = "alexandrevicenzi";
                repo = "Flex";
                rev = "master";
                sha256 = "sha256-I/WEqFMZbnJvX4NMeqB+o9LzD4im1BwOwgaqqDqbL2s=";
            };
            deps = with pkgs; [
                pelican-flex-theme
                (python3.withPackages (ps: [ ps.pelican ps.markdown ]))
            ];
        in {
            devShells.${system} = pkgs.mkShell {
                buildInputs = deps;
                shellHook = ''
                  export PELICAN_THEME=${pelican-flex-theme}
                '';
            };
        };
    }
    
    

    Now after running nix develop I can access the path to the theme via the environment variable PELICAN_THEME (which happens to be what I use in pelicanconf.py).

    Building the website

    Next step is to implement the build step. This is done by specifying the packages.${system}.default output derivation. Now
    by building the website, I mean generating the static files, not running a local instance. Running the website locally will be explained later.

    {
        description = "Simple blog project";
        inputs = {
            nixpkgs.url = "github:NixOS/nixpkgs/nixos-24.11";
        }
        outputs = { self, nixpkgs }: let
            system = "x86_64-linux";
            pkgs = import nixpkgs { inherit system; }
            pelican-flex-theme = pkgs.fetchFromGitHub {
                owner = "alexandrevicenzi";
                repo = "Flex";
                rev = "master";
                sha256 = "sha256-I/WEqFMZbnJvX4NMeqB+o9LzD4im1BwOwgaqqDqbL2s=";
            };
            deps = with pkgs; [
                pelican-flex-theme
                (python3.withPackages (ps: [ ps.pelican ps.markdown ]))
            ];
        in {
            devShells.${system} = pkgs.mkShell {
                buildInputs = deps;
                shellHook = ''
                  export PELICAN_THEME=${pelican-flex-theme}
                '';
            };
            packages.${system}.default = pkgs.stdenv.mkDerivation {
                name = "my-website";
                src = ./.;
                nativeBuildInputs = deps;
                buildPhase = ''
                    export PELICAN_THEME=${pelican-flex-theme}
                    pelican content -o output -s pelicanconf.py
                '';
                installPhase = ''
                    mkdir -p $out
                    cp -r output/* $out
                '';
            };
        };
    }
    
    

    Just like a Makefile we specify how to build the project using the buildPhase and how to install using the installPhase; all of this within stdenv.mkDerivation.

    There is however one problem with this derivation. It fetches all the files instead of only the files it needs. To fix this I use gitignore.nix which filters the source folder using the .gitignore file.

    {
        description = "Simple blog project";
        inputs = {
            nixpkgs.url = "github:NixOS/nixpkgs/nixos-24.11";
            gitignore = {
                url = "github:hercules-ci/gitignore.nix";
                inputs.nixpkgs.follows = "nixpkgs";
            };
        }
        outputs = { self, nixpkgs, gitignore }: let
            # ... previous code omitted ...
            deps = with pkgs; [
                pelican-flex-theme
                (python3.withPackages (ps: [ ps.pelican ps.markdown ]))
            ];
            inherit (gitignore.lib) gitignoreSource;
        in {
            # ... previous code omitted ...
            packages.${system}.default = pkgs.stdenv.mkDerivation {
                name = "my-website";
                src = gitignoreSource ./.;
                nativeBuildInputs = deps;
                buildPhase = ''
                    export PELICAN_THEME=${pelican-flex-theme}
                    pelican content -o output -s pelicanconf.py
                '';
                installPhase = ''
                    mkdir -p $out
                    cp -r output/* $out
                '';
            };
        };
    }
    
    

    Now you can build the website by running the command

    nix build

    You can access the website in the result folder (which is a link to the created derivation in the nix store).

    Launching the website locally

    To run the locally we define an application in the Nix flake file. To run the website locally will be done in three steps : 1. Create a shell script to run the website (using nix of course) 2. Define an application in flake.nix 3. Make the shell script and the package a dependency of the application

    Here is how it is done:

    {
        # ... inputs ...
        outputs = { self, nixpkgs, gitignore }: let
            # ... previous code omitted ...
            deps = with pkgs; [
                pelican-flex-theme
                (python3.withPackages (ps: [ ps.pelican ps.markdown ]))
            ];
            inherit (gitignore.lib) gitignoreSource;
            static-website = pkgs.stdenv.mkDerivation {
                name = "my-website";
                src = gitignoreSource ./.;
                nativeBuildInputs = deps;
                buildPhase = ''
                    export PELICAN_THEME=${pelican-flex-theme}
                    pelican content -o output -s pelicanconf.py
                '';
                installPhase = ''
                    mkdir -p $out
                    cp -r output/* $out
                '';
            };
        in {
            # ... previous code omitted ...
            packages.${system}.default = static-website;
            apps.${system}.default = let
                serv = pkgs.writeShellApplication {
                    name = "serve";
                    runtimeInputs = [ static-website ] ++ deps;
                    text = ''
                        pelican --relative-urls -p 8080 -l ${static-website}
                    '';
                };
            in {
                type = "app";
                program =" ${serv}/bin/serve";
            }
        };
    }

    In the above code we refactor the code by moving the package definition into the static-website variable in the let ... in ... clause. We also create a sub-component as a shell application derivation called serv which simply run the pelican local server.

    To run the website locally simply do

    nix run

    Updating the website

    Now if you make changes to your website (e.g. add a new article markdown file) you may not see the changes after running nix run or nix build, the most likely reason for this is that flakes track the latest state of the git repository; meaning you may need to git add files for changes to be reflected.

    Summary

    In this article I show how to use Nix flakes to manage the development life-cycle of a statically generated website using pelican.

    Nix flakes are used to manage the development environment (like virtual environments in Python), build the website (like in make build) and run the website locally (like a docker run).

  • Converting docker containers into AppImages

    In Linux, packages are the conventional way to distribute software. These packages are usually made of executable files, scripts and various other files. Examples include RPMs and Deb archives. While very useful, these packaging formats have a downside: they are distribution specific. Even if two distributions use the same packaging format, for instance Ubuntu and Debian, they are not compatible. Even further, packages may not be compatible between two different releases of the same Linux distribution, for instance Debian stable and Debian unstable. Consequently software, on Linux, needs to be built for every combinations of Linux distribution / releases out there.

    For simple open source software, like terminal applications, this isn’t usually a big issue, as volunteers can usually take the code and release it for the distribution of their choice making small adjustments to the code (at least in principle). For more complicated software and proprietary software this is trickier. However there is a solution to this problem: open source developers have created various universal packaging formats. The main ones are Flatpak, Snaps and AppImage. But there are many others, technically Docker is also a mean to distributed software. One can count also Steam distributables, if you think about it. And even Nix packages are in some way distribution agnostic as you can install Nix on many Linux distributions.

    I personally think AppImage lacks a little bit of love. It does not offer as many features as Flatpak or Snaps, which handle sand-boxing, software distribution, and sharing frameworks, but it is very simple and easy to grasp. AppImages are already fairly popular amongst QT applications, electron applications, and even some games like OpenRA. However there are few tools to create AppImages. So I decided to create another tool, which creates AppImages from Docker containers. Furthermore, there is plenty documentation on creating Docker images and the collection of publicly available Docker images is very large. The tool’s name docker2appimage available on Github at https://github.com/mparusinski/docker2appimage.

    Using docker2appimage

    If you have a container called relaxed_beaver, to create an AppImage from this containers, triggering default command, you run.

    docker2appimage relaxed_beaver

    You can perform more complex tasks like request docker2appimage to create an AppImage named alpine-ls from the alpine Docker image that calls ls

    docker2appimage -x ls --name alpine-ls -i alpine

    In order to work the tool requires the appimagetool AppImage as well imagemagick to be installed on the system. The entire tool, in order to be portable, is a simple shell script.

    Here is an even more complex example where we create an universal binary for ripgrep a popular alternative for grep:

    docker run -dit --name=ripgrep ubuntu bash
    docker exec -i ripgrep bash <<'EOF'
      apt-get update
      apt-get install -y ripgrep
    EOF
    docker2appimage -x rg ripgrep

    Most Docker images are server side software (e.g. nginx), OS containers (e.g. alpine) and programming frameworks. Naturally docker2appimage will fit well to create CLI-only AppImage although it is possible to create GUI based AppImages as well.

    Shortcomings and next steps

    There are some limitations of course. Many Docker images default commands are shells like bash, sh, … one might even include Python as a type of shell. AppImage works by using environment variables and Fuse to create a pseudo virtual environment; this virtual environment is then used to launched commands. As a result, handling shell commands is tricky and right now is not supported.

    The project is still ongoing development to support many applications and Linux distributions. If you have some time and are curious do not hesitate to test the tool and report any bugs you see.

  • Popularity of Linux universal packaging formats

    Counting the number of applications distributed as Snaps, Flatpaks or AppImages as a measure of the popularity of Linux universal packaging formats, let’s see how much existing Linux software has moved to Snap, Flatpak or AppImage. It should give us an idea of how successful these new packaging formats are with old open source projects.

    Disclaimer: Now I must be honest: answering this question is difficult and easily subjective. I tried being impartial and accurate but you should take everything I state with a grain of salt. Try researching on your own.

    Counting package format adoption

    Because counting on my own whether each piece of software on Linux is available as a Snaps, as a Flatpak or as an AppImage is time-consuming, I decided to take a subset of popular Linux applications. To avoid putting personal bias in the list, I chose the set of software listed on FossMint. Because Google Drive and Evernote are not officially packaged for Linux, I have excluded them. This should give us a rough idea of the percentage of applications adopting universal packaging, and thus help evaluating the popularity of Linux universal packaging formats.

    Using this list, I searched for the following:

    • Are the projects officially adopting AppImage, Flatpak or Snaps as a means of distribution? In each case, I checked the project’s official website.
    • Are AppImages, Flatpaks or Snaps available for the software in question, regardless whether the project official supports any of the format.
    • Are AppImages, Flatpak or Snaps shipping the latest version?

    What do the numbers say

    About 14% of projects official distribute their software as AppImage, Flatpak or Snaps. Meaning that adoption is slow (the youngest of the three formats is 4 years old). Of the 14% (~10 applications) which moved to universal packaging, 40% (~4) have adopted Flatpak, 40% (~4) have adopted AppImage and 20% (~2) have adopted Snaps. However, because the sample of applications officially distributed as AppImages, Flatpaks and Snaps is so small, one should not conclude yet that Flatpak or AppImage is being more successful.

    Concerning whether software is being made available as Snaps, Flatpaks or AppImages, AppImage appears to be last with about 37%. Snaps and Flatpak are ahead with respectively about 46% and 47% adoption rates respectively. AppImage project’s insistence on upstream packaging explains this discrepancy. Overall, this means Snaps and Flatpaks are being more successful, but only because volunteers (not officially associated with open source software projects) package popular Linux software as Snaps and Flatpaks.

    Concerning whether we have the latest version packaged in each format, we see a big lead for Snaps. About 60% of Snaps provided ship the latest version, 53% of Flatpaks provided ship the latest version and only 37.5% of AppImages ship the latest version. Because probonopd, one of the main developers behind AppImage, created most AppImage bundles and has not been updating them for more than a year, the number of up-to-date AppImage bundles is fairly low. The numbers are healthier for Snaps and Flatpaks but it is still far from 100 percent. Many volunteers who originally packaged software as Snaps, Flatpaks or AppImages have stopped updating them, meaning a lot of the software is out of date.

    So which one is ahead ?

    Based on this list I don’t think one can claim Snaps is way ahead; if anything, it appears to be strongly in competition with Flatpak. One should not conclude AppImage is losing steam either, AppImage is doing well, with 634 applications distributed as AppImage according to AppImageHub and many developers choosing AppImage as the official format for Linux. A lot of software is distributed today as Snaps, as Flatpaks and as AppImages by enthusiasts, because projects are not officially adopting either formats. This means will still need to rely on maintainers to distribute and maintain software as Snaps, Flatpak or AppImages.

    What’s holding back either Snaps, FlatPak or AppImage?

    What stop developers from using Snaps, Flatpaks or AppImages? I searched various forums and GitHub for answers. Here are some of the main reasons:

    • Lack of time, resources or motivation: This is one of the main reasons preventing a package format adoption. See for instance the following links: 1, 2, 3, 4. Hobbyists, who have limited time to work on their projects, are reluctant to learn or build for a new packaging format.
    • Not even packaged: Many software projects are not even doing the packaging on Linux themselves. They assume maintainers will package for them.
    • Software no longer developed: Lots of software are no longer maintained. For instance, Frozen Bubble was last updated in 2010 whilst Banshee was last updated in 2014. It is hard to expect projects with no developers to magically create a Snaps, Flatpaks or AppImages.
    • Lack of agreement over which format to use: This is not frequent but fairly telling. How come there are 3 competing universal packaging formats? RStudio could not agree over which format to use.

    Lots of people in the Linux community criticise Snaps, Flatpak and AppImage on technical grounds and disagree with the philosophy of universal packaging, but ironically this didn’t show up much as a reason not to support Snaps, Flatpak or AppImage. I think the people complaining about universal packaging are usually not software developers; the criticism comes more from system administrators, Linux distribution developers/maintainers and users who choose Linux for its security.

    Conclusion

    Overall, traditional Linux software appears reluctant to adopt Snap/Flatpak/AppImage. So, in terms of the popularity of Linux universal packaging formats, I’d say it is still fair game.