Navigating unwillful misinformation

A good part of a sysadmin journey is really just navigating thru a bunch of other people's experiences, and my latest sysadmin journey (upgrading an entire vdev in a ZFS pool) has brought me to some... less than accurate information. What I'm hoping writing this down will do (and praying that SEO picks it up) is to bust a myth that I keep seeing on Google - that the Dell H310 Mini HBA does not support 4Kn drives.

The actual answer is that the H310 Mini actually does indeed support 4Kn drives. As proof:

[~] lspci
...
02:00.0 RAID bus controller: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
...
[~] lstopo
...
      PCIBridge
        PCI 02:00.0 (RAID)
          Block(Disk) "sdf"
          Block(Disk) "sdd"
          Block(Disk) "sdm"
          Block(Disk) "sdb"
          Block(Disk) "sdk"
          Block(Disk) "sdi"
          Block(Disk) "sdg"
          Block(Disk) "sde"
          Block(Disk) "sdc"
          Block(Disk) "sda"
          Block(Disk) "sdj"
          Block(Disk) "sdh"
...
[~] fdisk -l /dev/sdg
Disk /dev/sdg: 10.91 TiB, 12000138625024 bytes, 2929721344 sectors
Disk model: HGST HUH721212AL
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: DEADBEEF-CAFE-CAFE-CAFE-DEADBEEFCAFE

Device     Start        End    Sectors  Size Type
/dev/sdg1    256 2929721087 2929720832 10.9T FreeBSD ZFS

The Twist

Of course, there is no free lunch. The key here is the lspci call which shows the H310 Mini in my R720XD as a LSI SAS2008 device. This is a telltale sign that the H310 is in IT Mode.

There are valid reasons to avoid IT Mode - namely that practically every feature of the H310 is disabled, whic means no RAID from the H310. The H310 will act as a passthrough for the drives. For my purposes, which is using ZFS, this is actually advantageous, but if you need hardware RAID, then you might be out of luck.

I'm not going to cover the process of flashing the H310 Mini into IT mode - there's a great tutorial out there that I can point you to, and once you're in IT mode, the drives will show up just fine.

One final word of caution - your Linux drive letters will change, and if you're using drive letters (which is a bad, bad, bad practice) in /etc/fstab, you'll need to change that.

Upgrading your ZFS storage

My ZFS array started out with 4 2TB drives in RAID-Z2. After a while, I added 4 6TB drives in RAID-Z2 into the pool. My R720xd thus has 8 of it's 12 drive bays filled, but I would really rather not fill all 12 bays right away. (I skimp out on drive costs-wise, so RAID-Z2 is a must on drives I cannot trust wholeheartedly). Fortunately, ZFS allows us to upgrade an entire vdev - once the entire vdev grows in size, the pool gains extra storage.

Firstly you'll need to set autoexpand=on like so:

# zpool set autoexpand=on pool

Then use replace to swap over to the new drives:

# zpool replace /dev/disks/by-id/{old-disk-id} /dev/disks/by-id/{new-disk-id}

You can replace them all at once, but just to be cautious, I'd replace them either one-at-a-time or # of parity drives at a time (which in my case is 2). As of time of writing my vdev is still resilvering, but at the end of this I should have 36TB of capacity on my my server - not too bad!

My wallet will recover from this bad purchase... eventually...

FFmpeg is weird...

...and it has to be! As the undisputed multimedia tool that interfaces with tons of other dependencies, it needs to work alongside, and with, conflicting licenses, patents, etc.

The result is that to access the best of what the multimedia world has to offer, you'd need to compile FFmpeg yourself - a tedious process that has spawned tons and tons of scripts that purport to "help" users who need specific FFmpeg dependencies.

I personally find these schemes to be terrible, and working with FFmpeg to be terrible as a whole (not the CLI - I actually think the FFmpeg CLI is pretty intuitive if you have a good understanding of multimedia fundamentals) as such. It took me working on FFmpeg itself to understand how to actually compile FFmpeg with specific dependencies.

Enter: Nix

Pulling out the "Purely Functional Software Deployment Model" paper is now kind of a meme at this point - but deterministic environments like the ones afforded to us by Nix actually solves the FFmpeg issue!

Wait, but isn't this just a variation of the "build scripts" that you purport to hate?

It's roughly in the same idea-space, but since we're piggybacking outselves onto the Nix package manager rather than some script that is undocumented and easy to break

Wait but the Nix package manager is pretty inscrutible itself

Yeah. I hope it gets better too. I don't think Nix's issues with being absolutely difficult to use is quite the same as my gripe with the typical build scripts, though.

Example 1: Dev flake

This example only requries the presense of the Nix package manager, and is how I invoke FFmpeg on my server. We define a flake our usual way:

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    flake-utils.url = "github:numtide/flake-utils";
  };
  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem
      (system:
        let
          pkgs = import nixpkgs {
            inherit system;
          };
        in
        with pkgs;
        {
          devShells.default = mkShell {
            buildInputs = [
                # We add our packages here
            ];
          };
        }
      );
}

then add an overlay to the output

    ...
    flake-utils.lib.eachDefaultSystem
      (system:
        let
          pkgs = import nixpkgs {
            inherit system;
            overlays = [
                (self: super: {ffmpeg-c = super.ffmpeg.override {
                    ## We define the build options here
                    withUnfree = true; 
                    withDav1d = true;
                };})
            ];
          };
        in
    ...

and then add this newly defined package to our devShell

        ...
        with pkgs;
        {
          devShells.default = mkShell {
            buildInputs = [
                ffmpeg-c
            ];
          };
        }
        ...

Hey, why are we defining the package to a new name? I thought the point with overlays is to override packages?

I'm glad you asked! In this example, because only the devShell is depenedent on our new ffmpeg-c package, we don't see big issues by naming it ffmpeg, but of course, this just forshadowing for the next part, where...

Example 2: Overriding in NixOS

...we're no longer just defining just a devShell, but a NixOS system as a whole.

In a "system" flake for NixOS or nix-darwin, instead of devShells.default as the output of our flake, we have nixOSConfigurations and darwinConfigurations. Also remember - FFmpeg is everywhere. Most fresh Linux systems have a non-encumbered version of FFmpeg somewhere to handle video - the first step in making a system usable is making it play cat videos.

So let's try defining an overlay module in overlays/default.nix that overrides ffmpeg!

# ./overlays/default.nix
{ config, pkgs, lib, ... }:

{
  nixpkgs.overlays = [
    (self: super: {
        ffmpeg = super.ffmpeg.override {
            withUnfree = true;
            withDav1d = true;
        };
    })
  ];
}

and let our system use it,

    ...
    mySystem = nixpkgs.lib.nixosSystem {
        specialArgs = { inherit inputs; }; # Pass flake inputs to our config
        modules = [
            ...
            ./nixos/atlantis/configuration.nix
            (import ./overlays/default.nix)
            ...
        ];
    };
    ...

then try running nixos-rebuild switch...

Hey, it's been three hours and nixos is trying to rebuild all of KDE from source. What the hell is going on?

We're running into the issue that the ffmpeg package is no longer what nixpkgs says it is - we've overridden it, and NixOS needs to evaluate if the system even compiles with the new config. Obviously, we want to avoid that. We want to use binaries that other people spent their precious machine hours on.

The solution is what I alluded to earlier in Example 1:

# ./overlays/default.nix
{ config, pkgs, lib, ... }:

{
  nixpkgs.overlays = [
    (self: super: {
        ffmpeg-c = super.ffmpeg.override {
            withUnfree = true;
            withDav1d = true;
        };
    })
  ];
}

Introducing a new package that other packages don't depend on. This does mean that you'll have two FFmpeg binaries floating around in your system, but that's a small price to pay.

Despite all that, this system of managing FFmpeg's dependencies is still the cleanest and most robust solution to the FFmpeg dependency problem. Not only that, but you also learn a little bit about how to manage your own devShells for niche use cases - something that will almost certainly help.

Have fun with your multimedia!

-r/c/s