Sign in Subscribe

Virtualised MooseFS - Chunkserver on Proxmox with Physical Disks

Benjamin Arntzen

• 6 min read

This'll be a quick one.

I've recently bought a new machine from a friend - an Athlon 3000G, Fractal Design 302-based NAS. It has slots for 6x 3.5" hard drives, of which 3 came filled with 14TB drives.

A white storage server, sitting next to a larger PC, with a Kobol Helios64 NAS stacked on top.

For background, I run a MooseFS distributed storage system at home. It's a fancy way of arranging and working with hard drives, SSDs and other storage devices which allows you to see them as one big filesystem - a reasonably-big-for-home-use 113TiB of storage, in my case. I planned to turn the new machine into a storage node for MooseFS, also called a "chunkserver".

After getting it home, I stuffed a Radeon RX6800 into it, with maybe 0.2mm of clearance (for virtualised GPU mining), enabled virtualisation in the BIOS, installed Proxmox and joined it to the WingsLab cluster.

Then I created a new VM - inky to go along with my Pacman-themed chunkserver naming scheme - and installed Ubuntu 22.04. Once it was installed, I shut it back down. Following the appropriate guide, I added all 3 disks to the new VM;

wings:~/ $ ssh root@10.1.1.155                                       [18:16:18]
Linux charmeleon 5.13.19-2-pve #1 SMP PVE 5.13.19-4 (Mon, 29 Nov 2021 12:10:09 +0100) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sat Apr 30 18:00:20 2022 from 10.1.1.22
root@charmeleon:~# lsblk |awk 'NR==1{print $0" DEVICE-ID(S)"}NR>1{dev=$1;printf $0" ";system("find /dev/disk/by-id -lname \"*"dev"\" -printf \" %p\"");print "";}'|grep -v -E 'part|lvm'
NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT DEVICE-ID(S)
sda                            8:0    0  12.7T  0 disk   /dev/disk/by-id/wwn-0x5000c500c82e3726 /dev/disk/by-id/ata-ST14000NE0008-2JK101_ZHZ76T44
sdb                            8:16   0  12.7T  0 disk   /dev/disk/by-id/wwn-0x5000c500c40d9d0c /dev/disk/by-id/ata-ST14000NE0008-2JK101_ZHZ5HJ26
sdc                            8:32   0  12.7T  0 disk   /dev/disk/by-id/wwn-0x5000c500c8194b3e /dev/disk/by-id/ata-ST14000NE0008-2JK101_ZHZ7AX11
nvme0n1                      259:0    0 238.5G  0 disk   /dev/disk/by-id/nvme-eui.6479a73812561323 /dev/disk/by-id/nvme-Sabrent_27FF0706169702270434
root@charmeleon:~# qm set 111 -scsi1 /dev/disk/by-id/wwn-0x5000c500c82e3726
update VM 111: -scsi1 /dev/disk/by-id/wwn-0x5000c500c82e3726
root@charmeleon:~# qm set 111 -scsi2 /dev/disk/by-id/wwn-0x5000c500c40d9d0c
update VM 111: -scsi2 /dev/disk/by-id/wwn-0x5000c500c40d9d0c
root@charmeleon:~# qm set 111 -scsi3 /dev/disk/by-id/wwn-0x5000c500c8194b3e
update VM 111: -scsi3 /dev/disk/by-id/wwn-0x5000c500c8194b3e

Then I started inky back up, connected and run updates, and opened a root shell:

wings:~/ $ ssh 10.1.1.203
Welcome to Ubuntu 22.04 LTS (GNU/Linux 5.15.0-27-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Sat Apr 30 10:25:19 AM UTC 2022

  System load:  0.0                Processes:              126
  Usage of /:   21.8% of 29.36GB   Users logged in:        0
  Memory usage: 5%                 IPv4 address for ens18: 10.1.1.50
  Swap usage:   0%


4 updates can be applied immediately.
To see these additional updates run: apt list --upgradable


Last login: Sat Apr 30 10:25:19 2022
wings@inky:~$ sudo apt update && sudo apt upgrade -y && sudo -i

Installed MooseFS Pro (a chunkserver and a metalogger):

root@inky:~# wget -O - https://ppa.moosefs.com/moosefs.key | apt-key add <snip>
OK
root@inky:~# echo "deb [arch=amd64] http://ppa.moosefs.com/moosefs-4/apt/ubuntu/focal focal main" > /etc/apt/sources.list.d/moosefs.list
root@inky:~# apt update
<snip>
All packages are up to date.
W: http://ppa.moosefs.com/moosefs-4/apt/ubuntu/focal/dists/focal/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
root@inky:~# apt install -y moosefs-pro-chunkserver moosefs-pro-metalogger
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  moosefs-pro-libpro
The following NEW packages will be installed:
  moosefs-pro-chunkserver moosefs-pro-libpro moosefs-pro-metalogger
0 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
Need to get 1,484 kB of archives.
After this operation, 4,617 kB of additional disk space will be used.
Get:1 http://ppa.moosefs.com/moosefs-4/apt/ubuntu/focal focal/main amd64 moosefs-pro-libpro amd64 4.37.5-1 [535 kB]
Get:2 http://ppa.moosefs.com/moosefs-4/apt/ubuntu/focal focal/main amd64 moosefs-pro-chunkserver amd64 4.37.5-1 [797 kB]
Get:3 http://ppa.moosefs.com/moosefs-4/apt/ubuntu/focal focal/main amd64 moosefs-pro-metalogger amd64 4.37.5-1 [152 kB]
Fetched 1,484 kB in 3s (426 kB/s)
Selecting previously unselected package moosefs-pro-libpro.
(Reading database ... 73192 files and directories currently installed.)
Preparing to unpack .../moosefs-pro-libpro_4.37.5-1_amd64.deb ...
Unpacking moosefs-pro-libpro (4.37.5-1) ...
Selecting previously unselected package moosefs-pro-chunkserver.
Preparing to unpack .../moosefs-pro-chunkserver_4.37.5-1_amd64.deb ...
Unpacking moosefs-pro-chunkserver (4.37.5-1) ...
Selecting previously unselected package moosefs-pro-metalogger.
Preparing to unpack .../moosefs-pro-metalogger_4.37.5-1_amd64.deb ...
Unpacking moosefs-pro-metalogger (4.37.5-1) ...
Setting up moosefs-pro-metalogger (4.37.5-1) ...
Setting up moosefs-pro-libpro (4.37.5-1) ...
Setting up moosefs-pro-chunkserver (4.37.5-1) ...
Processing triggers for man-db (2.10.2-1) ...
Scanning processes...
<snip>
root@inky:~#

Created some filesystems on the 3 disks:

wings@inky:~$ for i in b c d ; do sudo mkfs.btrfs -f /dev/sd$i ; done
btrfs-progs v5.16.2
See http://btrfs.wiki.kernel.org for more information.

Performing full device TRIM /dev/sdb (12.73TiB) ...
NOTE: several default settings have changed in version 5.15, please make sure
      this does not affect your deployments:
      - DUP for metadata (-m dup)
      - enabled no-holes (-O no-holes)
      - enabled free-space-tree (-R free-space-tree)

Label:              (null)
UUID:               f198296b-5db4-44de-b6ee-079764d0e85f
Node size:          16384
Sector size:        4096
Filesystem size:    12.73TiB
Block group profiles:
  Data:             single            8.00MiB
  Metadata:         DUP               1.00GiB
  System:           DUP               8.00MiB
SSD detected:       no
Zoned device:       no
Incompat features:  extref, skinny-metadata, no-holes
Runtime features:   free-space-tree
Checksum:           crc32c
Number of devices:  1
Devices:
   ID        SIZE  PATH
    1    12.73TiB  /dev/sdb
<snip>
Devices:
   ID        SIZE  PATH
    1    12.73TiB  /dev/sdc
<snip>
Devices:
   ID        SIZE  PATH
    1    12.73TiB  /dev/sdd

Created some mountpoints:

wings@inky:~$ sudo mkdir -p /mnt/mfsbrick.1 /mnt/mfsbrick.2 /mnt/mfsbrick.3

Created entries in /etc/fstab for the new disks:

root@inky:~# echo "UUID=f198296b-5db4-44de-b6ee-079764d0e85f /mnt/mfsbrick.1 btrfs defaults,noatime,nofail 0 0" >> /etc/fstab
root@inky:~# echo "UUID=eeb064fc-f87f-4bda-afb2-f61fe949d176 /mnt/mfsbrick.2 btrfs defaults,noatime,nofail 0 0" >> /etc/fstab
root@inky:~# echo "UUID=5ca48008-cd9a-4e72-8d7c-ac8479ccb8e5 /mnt/mfsbrick.3 btrfs defaults,noatime,nofail 0 0" >> /etc/fstab

Mounted the new disks:

sudo mount -a

Created some subfolders in the disks (I rarely, if ever, put a disk into MooseFS without a subfolder):

wings@inky:~$ sudo mkdir -p /mnt/mfsbrick.1/brick /mnt/mfsbrick.2/brick /mnt/mfsbrick.3/brick && sudo chown -R mfs:mfs /mnt/mfsbrick.*/brick

Defined the new disks for MooseFS:

wings@inky:~$ echo "/mnt/mfsbrick.1/brick" | sudo tee -a /etc/mfs/mfshdd.cfg
/etc/mfsbrick.1
wings@inky:~$ echo "/mnt/mfsbrick.2/brick" | sudo tee -a /etc/mfs/mfshdd.cfg
/etc/mfsbrick.2
wings@inky:~$ echo "/mnt/mfsbrick.3/brick" | sudo tee -a /etc/mfs/mfshdd.cfg
/etc/mfsbrick.3

And finally, enable and start the MooseFS chunkserver and metalogger

wings@inky:~$ sudo systemctl start moosefs-pro-chunkserver && sudo systemctl enable moosefs-pro-chunkserver && sudo systemctl start moosefs-pro-metalogger && sudo systemctl enable moosefs-pro-metalogger
Created symlink /etc/systemd/system/multi-user.target.wants/moosefs-pro-chunkserver.service → /lib/systemd/system/moosefs-pro-chunkserver.service.
Created symlink /etc/systemd/system/multi-user.target.wants/moosefs-pro-metalogger.service → /lib/systemd/system/moosefs-pro-metalogger.service.
Success!

Everything looks good! I now have 151TiB of MooseFS storage.

fin

PS: See that dashed line? The new chunkserver will automatically "rebalance" with the rest of the MooseFS storage hosts, and take on data until it's roughly equal with them. (Caveat: Some of my data is "pinned" to particular hosts, to make Chia more viable on MooseFS).

Sign up for more like this.

Enter your email
Subscribe