Homelab Proxmox + VyOS + Debian server configuration - WIP

04 Sep, 2022

Here I document my home server config. I’m trying to integrate the router in it by using a second USB ethernet as WAN port. Running proxmox I can run a Debian installation for my usual stuff and an OpenWRT router image for routing.

Goal

I’m looking for a small and energy efficient server with some storage capability.

Small form factor (<5 liter)
Low power consumption home server + router configuration
Nice to have: virtualized router and home server (to satisfy above)
Server requirements
- A few TB storage for home use (backup, pictures, etc.)
- Stable and secure Linux OS preferred
- Run media server
- Run home assistant
Router requirements (from home networking setup)
- Should support 100 MBit WAN (NAT/firewalling requirement)
- Separate networks for internal, guest, and buggy IoT devices (VLAN-aware ethernet and WiFi)
- Ability to prevent buffer bloat (need decent QoS)
- Gigabit LAN
- Low-power (ideally <10W for full setup)
- LAN-wide adblocking (DNS-based pi-hole or related)
- Home VPN server (IKEv2 or Wireguard, >50Mbps)

Hardware

I’m looking for a small and energy efficient server with some storage capability. I’ve settled for using a NUC with an extra 2.5" bay, which suits my needs.

NUC8i3BEH
2TB Samsung 970 EVO Plus M.2 80mm PCIE
4TB Samsung 860 2.5" SATA
16GB DDR4 RAM
Conbee Zigbee USB dongle
USB Ethernet dongle for split WAN/LAN network
USB to smartmeter cable (FTD)
USB to heatmeter cable (FTD)
Optional: USB port to power ESP8266 water meter board
Optional: Bluetooth USB dongle (for more range)

Target services & architecture

I considered the following virtualization software:

Proxmox or XCP-NG + XO
- Proxmox is easier, uses less power. Only need to solve LVM on multiple disks

For routing, I considered the following platforms:

OpenWRT or VyOS or pfSense or OPNsense
- OpenWRT has difficulties upgrading?
- VyOS seems nice, is linux, have experience, but rolling release either means upgrading a lot or having random stability
- Don’t know OPNsense/pfSense

In the end I settled for Proxmox + VyOS

Proxmox (8GB storage + 2GB RAM)
- SMB server serving bulk data partition to guest OS (keeping guest OS small)
VyOS VM (8GB storage + 2GB RAM)
- dns adblock list
- wireguard
- regular router config
Debian Stable LXC (256GB thinvol + 12GB RAM)
- Nginx (for website & reverse proxy)
- Letsencrypt (for SSL)
- Docker & Portainer (for managing servers)
  - Nextcloud –> upgrade from snap package (for file sharing, could be nginx webdav?)
  - bpatrik/pigallery2 (for personal photo sharing)
  - Home Assistant –> upgrade to Docker under Supervised mode instead of current python virtual env (for monitoring)
  - lscr.io/linuxserver/unifi-controller (for Unifi AP management)
- Collectd (for data generation/collection)
- Home automation worker scripts (for data generation/collection)
  - many
- Influxdb (for data storage)
- Grafana (for data visualization)
- Transmission (downloading torrents)
- Mosquitto (glueing home automation)
- Plex/Jellyfin (HTPC)
- smbd (for Time Machine backups)

Software distribution

I went with Ubuntu Server 20.04-LTS before, but this had frequent updates (feels like daily, also non-security). For my next server I’m thinking of Debian Stable instead.

Installation & configuration

Proxmox

Installation

Install Proxmox as described on their wiki, reserving a small part for the OS and most disk space for VMs:

hdsize full
swapsize 4GB
maxroot 8GB
minfree 0.5GB

Post-install fixes, sourced from Proxmox Helper Scripts. Never run scripts from the internet.

# From https://raw.githubusercontent.com/tteck/Proxmox/main/misc/post-pve-install.sh
# Disable Enterprise Repository
sed -i "s/^deb/#deb/g" /etc/apt/sources.list.d/pve-enterprise.list
# Enable No-Subscription Repository
cat <<EOF >>/etc/apt/sources.list
deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription
EOF
# Disable Subscription Nag
echo "DPkg::Post-Invoke { \"dpkg -V proxmox-widget-toolkit | grep -q '/proxmoxlib\.js$'; if [ \$? -eq 1 ]; then { echo 'Removing subscription nag from UI...'; sed -i '/data.status/{s/\!//;s/active/NoMoreNagging/}' /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js; }; fi\"; };" >/etc/apt/apt.conf.d/no-nag-script
apt --reinstall install proxmox-widget-toolkit &>/dev/null
# Upgrade proxmox now
apt-get update
apt-get dist-upgrade

Enable colors in shell & vi:

sed -i "s/^# alias /alias /" ~/.bashrc

cat <<EOF >>~/.bashrc
alias grep='grep --color=auto'
alias fgrep='fgrep --color=auto'
alias egrep='egrep --color=auto'
EOF

apt install vim

Set ondemand cpu governer for power saving. Add intel_pstate=disable to boot parameters (2019):

sed -i 's/^GRUB_CMDLINE_LINUX_DEFAULT="quiet/GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_pstate=disable/' /etc/default/grub
# vi /etc/default/grub
# GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_pstate=disable"
update-grub && reboot

Set scaling governer to ondemand

apt install cpufrequtils
echo ondemand > /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

Alternatively try enabling speedstep in BIOS (2018) (not needed for my setup).

Measure power consumption:

only network (no HDMI, keyboard, USB) : ~2.5W+-0.3W, spikes from 1.6-3.5W (ondemand governor, chi-by-eye PM231E)
only network + ASUS USB NIC (w/o cable): ~2.8W+-0.3W (ondemand governor, chi-by-eye PM231E)
only network + Achate ASIX USB3.0-C NIC (w/o cable): ~6.0W+-0.3W (ondemand governor, chi-by-eye PM231E)

Networking

Setup VLAN-aware networking on management interface. Resulting /etc/network/interfaces config:

auto lo
iface lo inet loopback

iface eno1 inet manual

auto enx7c10c9194780
iface enx7c10c9194780 inet manual

auto vmbr0
iface vmbr0 inet manual
	bridge-ports eno1
	bridge-stp off
	bridge-fd 0
	bridge-vlan-aware yes
	bridge-vids 10-30
#LAN port

auto vmbr1
iface vmbr1 inet manual
	bridge-ports enx000ec6955446
	#bridge_maxwait 40	
	bridge-stp off
	bridge-fd 0
#WAN

auto vmbr0.10
iface vmbr0.10 inet static
	address 172.17.10.4/24
	gateway 172.17.10.1
#Mgmt interface

Storage

Expand Proxmox LVM over two disks. Sources:

lsblk
# Create full-disk LVM partition
cfdisk /dev/sda
# Wipe disk of previous LVM config
pvremove -y -ff /dev/sda*

# Create new PV
pvcreate /dev/disk/by-id/ata-Samsung_SSD_860_EVO_4TB_S45JNB0M500432F-part1
vgextend pve /dev/disk/by-id/ata-Samsung_SSD_860_EVO_4TB_S45JNB0M500432F-part1

# Remove 'local-lvm' storage via GUI or pvesm
pvesm remove local-lvm

# Remove data LVM pool, recreate new one (could also extend probably)
lvremove pve/data

# Optional, to extend logical volume over full volume group
# lvextend --poolmetadatasize +30G pve/data` # too big
# lvextend -l +100%FREE pve/data

Set up storage via LVM thin pools / thin volumes

Data requirements/architecture: I have two disks of different speed. I want the VMs to run on the fast disk and bulk data to run on the slow disk. To ensure this I create two thinpools, the first one for VMs which resides on the fast disk, the second spans the remainder of the fast disk + the full slow disk. This wastes a bit of space between the two thinpools, but ensures the VMs run on the fast disk.
- LVM (0.012TB)
  - Proxmox: 8 GB (root) + 4 GB (swap)
- LVM thin ’thinpool_vms’ (0.5TB)
  - VyOS: 0.008TB
  - Debian: 0.25TB
  - Other/future/debian growth
- LVM thin ’thinpool_data’ (remainder = 5.5TB for 5.5TB data)
  - Bulk data: 3TB
  - Backups VMS: 0.25TB
  - Backups MBP: 1TB
  - Backups MBA: 0.25TB
  - Backups remote: 1.25TB

Create data setup in LVM

# Create thinpool on fast disk
lvcreate --thin -L 0.5TB pve/thinpool_vms
# Create thinpool on remainder of fast disk + slow disk
lvcreate --thin -l 100%FREE pve/thinpool_data

# Ensure data split across disks was succesful
pvdisplay -m /dev/sda1
pvdisplay -m /dev/nvme0n1p3
lvdisplay -m /dev/pve/thinpool_vms
lvdisplay -m /dev/pve/thinpool_data

# Create LVs for future use on thinpool_data
lvcreate --thinpool pve/thinpool_data --name lv_bulk --virtualsize 3.0T
lvcreate --thinpool pve/thinpool_data --name lv_backup_vms --virtualsize 0.25T
lvcreate --thinpool pve/thinpool_data --name lv_backup_mbp --virtualsize 1T
lvcreate --thinpool pve/thinpool_data --name lv_backup_mba --virtualsize 0.25T
lvcreate --thinpool pve/thinpool_data --name lv_backup_tex --virtualsize 1.25T

Create filesystems

mkfs.ext4 /dev/mapper/pve-lv_bulk
mkfs.ext4 /dev/mapper/pve-lv_backup_vms
mkfs.ext4 /dev/mapper/pve-lv_backup_mbp
mkfs.ext4 /dev/mapper/pve-lv_backup_mba
mkfs.ext4 /dev/mapper/pve-lv_backup_tex

Mount in Proxmox

mkdir /mnt/bulk
mkdir -p /mnt/backup/{vms,mba,mbp,tex}
mount /dev/mapper/pve-lv_bulk /mnt/bulk/
mount /dev/mapper/pve-lv_backup_vms /mnt/backup/vms
mount /dev/mapper/pve-lv_backup_mbp /mnt/backup/mbp
mount /dev/mapper/pve-lv_backup_mba /mnt/backup/mba
mount /dev/mapper/pve-lv_backup_tex /mnt/backup/tex

chmod og-rx /mnt/backup/{vms,mba,mbp,tex}
chmod og-rx /mnt/bulk/

Automount

cat <<EOF >>/etc/fstab
/dev/mapper/pve-lv_bulk       /mnt/bulk        ext4  defaults  0  2
/dev/mapper/pve-lv_backup_vms /mnt/backup/vms  ext4  defaults  0  2
/dev/mapper/pve-lv_backup_mbp /mnt/backup/mbp  ext4  defaults  0  2
/dev/mapper/pve-lv_backup_mba /mnt/backup/mba  ext4  defaults  0  2
/dev/mapper/pve-lv_backup_tex /mnt/backup/tex  ext4  defaults  0  2
EOF

- Add directory to PVE storage manager
```sh
pvesm add dir backup --path /mnt/backup/vms --content vztmpl,iso,backup
```
- Add thin pool to PVE storage manager
```sh
pvesm scan lvmthin pve
pvesm add lvmthin thinpool_vms --vgname pve --thinpool thinpool_vms
```
- Push back backups from elsewhere & optionally [resize disks/partitions](https://serverfault.com/questions/784890/how-to-shrink-the-disk-of-a-lxc-container-on-proxmox-4)
```sh
e2fsck -fy /dev/pve/vm-200-disk-0
resize2fs /dev/pve/vm-200-disk-0 300G
lvreduce -L 300G /dev/pve/vm-200-disk-0

# Edit LXC config in /etc/pve/lxc
#rootfs: thinpool_vms:vm-200-disk-0,size=300G
```

Set up SMB on Proxmox for file sharing.

Install, disable unnecessary netbios deamon, and stop samba itself during configuration.

apt install samba
systemctl stop nmbd.service
systemctl disable nmbd.service
systemctl stop smbd.service

Configure

[global]
   server string = pve.vanwerkhoven.org
   server role = standalone server
   interfaces = lo vmbr0.10
   bind interfaces only = yes
   disable netbios = yes
   smb ports = 445
   log file = /var/log/samba/smb.log
   max log size = 10000
   # log level = 3 passdb:5 auth:5

Add users

adduser --home /mnt/backup/mbp --no-create-home --shell /usr/sbin/nologin --disabled-password --ingroup sambashare backupmbp
adduser --home /mnt/backup/mba --no-create-home --shell /usr/sbin/nologin --disabled-password --ingroup sambashare backupmba
adduser --home /mnt/bulk --no-create-home --shell /usr/sbin/nologin --disabled-password --ingroup sambashare sambarw

chown :sambashare /mnt/backup/{mba,mbp}
chown :sambashare /mnt/bulk

chmod 2770 /mnt/backup/{mba,mbp}
chmod 2770 /mnt/bulk

openssl rand -base64 20
smbpasswd -a backupmbp
smbpasswd -a backupmba
smbpasswd -a sambarw

smbpasswd -e backupmba
smbpasswd -e backupmbp
smbpasswd -e sambarw

Set up shares

[bulk]
    path = /mnt/bulk
    browseable = yes
    read only = no
    writable = yes
    force create mode = 0660
    force directory mode = 2770
    valid users = sambarw
[backupmbp]
    comment = Time Machine mbp
    path = /mnt/backup/mbp
    browseable = yes
    writeable = yes
    create mask = 0600
    directory mask = 0700
    spotlight = yes
    vfs objects = catia fruit streams_xattr
    fruit:aapl = yes
    fruit:time machine = yes
    valid users = backupmbp
[backupmba]
    comment = Time Machine MBA
    path = /mnt/backup/mba
    browseable = yes
    writeable = yes
    create mask = 0600
    directory mask = 0700
    spotlight = yes
    vfs objects = catia fruit streams_xattr
    fruit:aapl = yes
    fruit:time machine = yes
    valid users = backupmba

Restart Samba

systemctl restart smbd.service

Optional tweaks & bugfixes

Optional: assign USB dongle interface a nice name. N.B. this breaks proxmox recognizing the adapter as network interface in the GUI, disabling some configuration options.

echo 'SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="7c:10:c9:19:47:80", NAME="usb0"' | tee -a /etc/udev/rules.d/70-persistent-net.rules

echo "auto usb0
iface usb0 inet dhcp" | tee -a /etc/network/interfaces

udevadm control --reload-rules && udevadm trigger

Optional: Fix boot delay after bluetooth driver error. Looks like it’s caused by DHCP timeout on unconnected ethernet port, leave as is. Add intel-ibt-17* bluetooth driver for NUC –> does not work, conflicts with proxmox kernel. Wait until adopted in main PVE kernel.

Add non-free debian packages in /etc/apt/sources.list or related
Install firmware apt install firmware-iwlwifi

Bugfix: bridge brought up before physical port not up: “error: vmbr1: bridge port enx000ec6955446 does not exist”

Increase bridge_maxwait to 40s
Alternative: Increase bridge_waitport ?
Try something else

Install VyOS & configure as VM

Get image, install and add serial socket for using xterm.js support (copy-pasting). Also start on boot.

ls /var/lib/vz/template/iso/
qm create 200 --name vyos --memory 2048 --net0 virtio,bridge=vmbr0 --net1 virtio,bridge=vmbr1 --ide2 media=cdrom,file=local:iso/vyos-1.4-rolling-202211010829-amd64.iso --virtio0 data:8
qm set 200 --net1 virtio,bridge=vmbr1
qm set 200 -serial0 socket
qm set 200 --onboot 1

Open terminal via Spice/xterm.js, install image, remove image, and reboot

qm start 200
# in guest: `install image`
qm set 200 --ide2 none
qm reboot 200

Enable QEMU guest agent in Proxmox (VyOS has this since 2018). Source

qm set 200 --agent 1
qm agent 200 ping

Configure VyOS

Set global settings

set set system host-name vyos
set system domain-name lan.vanwerkhoven.org

Configure eth1 (=vmbr1=WAN) as DHCP client

#TODO replace with dhcp query @ right VLAN id later
set interfaces ethernet eth1 vif 1
set interfaces ethernet eth1 vif 1 description WAN
set interfaces ethernet eth1 vif 1 address dhcp

Setup VLAN on LAN network, see here

set interfaces ethernet eth0 description LAN
set interfaces bridge br100 enable-vlan
# set interfaces bridge br100 member interface eth0 allowed-vlan 2-4092
set interfaces bridge br100 member interface eth0 allowed-vlan 10
set interfaces bridge br100 member interface eth0 allowed-vlan 20
set interfaces bridge br100 member interface eth0 allowed-vlan 30
set interfaces bridge br100 member interface eth0 allowed-vlan 40
set interfaces bridge br100 vif 10 address 172.17.10.1/24
set interfaces bridge br100 vif 10 description 'VLAN10-Mgmt'
set interfaces bridge br100 vif 20 address 172.17.20.1/24
set interfaces bridge br100 vif 20 description 'VLAN20-Trusted'
set interfaces bridge br100 vif 30 address 172.17.30.1/24
set interfaces bridge br100 vif 30 description 'VLAN30-Guest'
set interfaces bridge br100 vif 40 address 172.17.40.1/24
set interfaces bridge br100 vif 40 description 'VLAN40-IoT'
set interfaces bridge br100 stp

Enable SSH on only management interface without password auth.

set service ssh port '22'
set service ssh listen-address 172.17.10.1
set service ssh disable-password-authentication
set system login user vyos authentication public-keys tim@neptune type ssh-rsa
set system login user vyos authentication public-keys tim@neptune key AAAAB3NzaC1yc2EAAAABIwAAAgEAu5cydWmE/eaug3WhqRZXDD4k+El7ooQAp5gogVUdox8CVFLtlL/XTzEwFcNm5jhOSHG2Tb0Spl7H0YFSMbgcK3icgTg3hRuyFdLW2eUmMOmrNaYHeHhOiM/bOHCD8z7FXWZdlIDaoOv7UQjBGa7B6TWpqxjnYZkN05u8+iBzdInlgbrXZ6E1iJIS5iPeRaOLmUgWwxql4P1BrGXp2X93fkjjhVIXED0A0LE+JAwjg4Le8+xqWune7F9OvnLg34yZJpfQxRrMKsxA0xliow737ogUYWvHmnhgNs5kQpfPcO1JkeU5PHuSr3XLVSm0w2r1+yv3KZw0GfUqOGoFoOAAnGvSb6xFe77RxcGnL4FaBwEcAxNrP8UKpdrKs0NI0Ao6za61cdwuMoU1+uUbglCt0jE7apr1dQWRnnuciKQBUxSKgFi7k4dBLRRdp3EeqsMyyDKJq15GwPjUyxE0dGjqne8nNNSB4RZ6FJT+bomtfrqKcm6o7oVQnX4aGW2xoW7KvXHB+EAP0obamDpmm74iwsI5bz7EWvL7PXU/3CIMcxoQ5hBOfanE0B1jEmmHg/S2DZQ8lfgnxjcJc16DCXloPm8WOIlNhB9C0zPt1VxCgxNIwlq8VHtmiCU7WX6M6/4ezDR2Z5X3KWyVrLLUJ/Wnb9igYCABoN4FlDWGJagBsHs=

Harden SSH, only allow strong ciphers, don’t use md5/sha1, don’t use contested nistp256:

set service ssh ciphers aes128-cbc
set service ssh ciphers aes128-ctr           
set service ssh ciphers aes128-gcm@openssh.com
set service ssh ciphers aes192-cbc           
set service ssh ciphers aes192-ctr           
set service ssh ciphers aes256-cbc           
set service ssh ciphers aes256-ctr           
set service ssh ciphers aes256-gcm@openssh.com
set service ssh ciphers chacha20-poly1305@openssh.com
set service ssh mac hmac-sha2-256        
set service ssh mac hmac-sha2-256-etm@openssh.com
set service ssh mac hmac-sha2-512        
set service ssh mac hmac-sha2-512-etm@openssh.com
set service ssh key-exchange curve25519-sha256    
set service ssh key-exchange curve25519-sha256@libssh.org
set service ssh key-exchange diffie-hellman-group-exchange-sha256
set service ssh key-exchange diffie-hellman-group14-sha256
set service ssh key-exchange diffie-hellman-group16-sha512
set service ssh key-exchange diffie-hellman-group18-sha512

Set timezone & ntp server from global pool:

set system time-zone Europe/Amsterdam
delete system ntp
set system ntp server 0.nl.pool.ntp.org
set system ntp server 1.nl.pool.ntp.org

Enable DNS from DHCP, also for local machine. Set cache to 100k, 10x up from dnsmasq default. More local caching should give more speed and more privay (less public querying).

# TODO: fix when going live to WAN interface
# Specifically use name servers received for the interface that is using DHCP client to get an IP
set service dns forwarding dhcp eth1.1
set service dns forwarding allow-from 172.17.0.0/16
set service dns forwarding domain lan.vanwerkhoven.org server 172.17.10.1
set service dns forwarding listen-address 172.17.10.1
set service dns forwarding listen-address 172.17.20.1
set service dns forwarding listen-address 172.17.30.1
set service dns forwarding listen-address 172.17.40.1
set service dns forwarding cache-size 100000
# set system name-server 172.17.10.1 # use static
# set system name-servers eth1.1 # use from dhcp -- not working?

Configure DHCP server ranges for VLAN. 100-254 is dynamic, 1-100 is for static hosts.

delete service dhcp-server shared-network-name vlan10
set service dhcp-server shared-network-name vlan10 subnet 172.17.10.0/24 range vlan10range start 172.17.10.100
set service dhcp-server shared-network-name vlan10 subnet 172.17.10.0/24 range vlan10range stop 172.17.10.254
set service dhcp-server shared-network-name vlan10 subnet 172.17.10.0/24 default-router 172.17.10.1
set service dhcp-server shared-network-name vlan10 subnet 172.17.10.0/24 domain-name lan.vanwerkhoven.org
set service dhcp-server shared-network-name vlan10 subnet 172.17.10.0/24 name-server 172.17.10.1

delete service dhcp-server shared-network-name vlan20
set service dhcp-server shared-network-name vlan20 authoritative
set service dhcp-server shared-network-name vlan20 subnet 172.17.20.0/24 range vlan20range start 172.17.20.100
set service dhcp-server shared-network-name vlan20 subnet 172.17.20.0/24 range vlan20range stop 172.17.20.254
set service dhcp-server shared-network-name vlan20 subnet 172.17.20.0/24 default-router 172.17.20.1
set service dhcp-server shared-network-name vlan20 subnet 172.17.20.0/24 domain-name lan.vanwerkhoven.org
set service dhcp-server shared-network-name vlan20 subnet 172.17.20.0/24 name-server 172.17.20.1

delete service dhcp-server shared-network-name vlan30
set service dhcp-server shared-network-name vlan30 subnet 172.17.30.0/24 range vlan30range start 172.17.30.100
set service dhcp-server shared-network-name vlan30 subnet 172.17.30.0/24 range vlan30range stop 172.17.30.254
set service dhcp-server shared-network-name vlan30 subnet 172.17.30.0/24 default-router 172.17.30.1
set service dhcp-server shared-network-name vlan30 subnet 172.17.30.0/24 domain-name lan.vanwerkhoven.org
set service dhcp-server shared-network-name vlan30 subnet 172.17.30.0/24 name-server 172.17.30.1

delete service dhcp-server shared-network-name vlan40
set service dhcp-server shared-network-name vlan40 subnet 172.17.40.0/24 range vlan40range start 172.17.40.100
set service dhcp-server shared-network-name vlan40 subnet 172.17.40.0/24 range vlan40range stop 172.17.40.254
set service dhcp-server shared-network-name vlan40 subnet 172.17.40.0/24 default-router 172.17.40.1
set service dhcp-server shared-network-name vlan40 subnet 172.17.40.0/24 domain-name lan.vanwerkhoven.org
set service dhcp-server shared-network-name vlan40 subnet 172.17.40.0/24 name-server 172.17.40.1

Set up masquerading for outbound traffic

#TODO: fix WAN VLAN
set nat source rule 5010 outbound-interface 'eth1.1'
set nat source rule 5010 source address '172.17.0.0/16'
set nat source rule 5010 translation address masquerade
set nat source rule 5010 protocol all
set nat source rule 5010 description 'Masquerade for WAN'

Set up static ips/host names

set system static-host-mapping host-name vyos.lan.vanwerkhoven.org inet 172.17.10.1 # not sure if this works, already set to 127.0.0.1
set system static-host-mapping host-name pve.lan.vanwerkhoven.org inet 172.17.10.4
set system static-host-mapping host-name proteus.lan.vanwerkhoven.org inet 172.17.10.2

Set up port forwarding

!!! @TODO

443 to 172.17.99.2:443
80 to 172.17.99.2:80
10022 to 172.17.99.2:22
1883 to 172.17.99.2:8883

Set up SSH forwarding with hairpint nat:

# Port forward for SSH
set nat destination rule 100 description 'Port Forward: SSH to 172.17.10.2'
set nat destination rule 100 destination port '22'
set nat destination rule 100 inbound-interface 'eth1.1'
set nat destination rule 100 protocol 'tcp'
set nat destination rule 100 translation address '172.17.10.2'

!!! @TODO solve hairpin NAT, not working, connection refused?

# Hairpin NAT for SSH port forwarding - not working
set nat destination rule 110 description 'Port Forward: SSH to 172.17.10.2 (NAT Reflection: INSIDE)'
set nat destination rule 110 destination port '22'
set nat destination rule 110 inbound-interface 'br100.10'
# set nat destination rule 110 inbound-interface 'br100'
# set nat destination rule 110 inbound-interface 'eth0'
set nat destination rule 110 protocol 'tcp'
set nat destination rule 110 translation address '172.17.10.2'
delete nat destination rule 110

set nat source rule 110 description 'Port Forward: SSH to 172.17.10.2 (NAT Reflection: INSIDE)'
set nat source rule 110 destination address '172.17.10.0/24'
set nat source rule 110 outbound-interface 'br100.10'
# set nat source rule 110 outbound-interface 'br100'
# set nat source rule 110 outbound-interface 'eth0'
set nat source rule 110 protocol 'tcp'
set nat source rule 110 source address '172.17.10.0/24'
set nat source rule 110 translation address 'masquerade'
delete nat source rule 110

Set up zone-based firewall according to this rule table

!!! @TODO

	to Local	to Mgmt	to Trusted	to Guest	to IoT
Local		FW_ACCEPT	FW_ACCEPT
Mgmt	FW_ACCEPT
Trust.	FW_DROP	FW_DROP
Guest	FW_DROP	FW_DROP
IoT	FW_DROP	FW_DROP
WAN	FW_WAN2LOCAL	FW_WAN2OTHERS	FW_WAN2OTHERS	FW_WAN2OTHER	FW_DROP

Set up QoS. Source: https://gist.github.com/jbrodriguez/cc0b1d9f72f66e555ad7

set traffic-policy shaper WAN_QUEUE bandwidth '50Mbit'
# Default traffic
set traffic-policy shaper WAN_QUEUE default bandwidth '95%'
set traffic-policy shaper WAN_QUEUE default priority '3'
set traffic-policy shaper WAN_QUEUE default queue-type 'fq-codel'
set traffic-policy shaper WAN_QUEUE description "WAN QoS shaper"
# megasuper priority dns and ssh and icmp
set traffic-policy shaper EGRESS_QOS class 10 bandwidth '10%'
set traffic-policy shaper EGRESS_QOS class 10 priority '5'
set traffic-policy shaper EGRESS_QOS class 10 queue-type 'fq-codel'
set traffic-policy shaper EGRESS_QOS class 10 match icmp ip protocol icmp
set traffic-policy shaper EGRESS_QOS class 10 match dns ip source port 53
# TODO: fix WAN VLAN when deploying
set interfaces ethernet eth1.1 traffic-policy out WAN_QUEUE

Set up Wireguard VPN, see the official VyOS docs and this tutorial including firewalling.

config
run generate pki wireguard key-pair install interface wg0
# Public key: uGc4JMJ4IJc0aoIY/ITOrFGWjmn+RxnqRQMecOS4uB8=

set interfaces wireguard wg0 address 172.17.40.1/24
set interfaces wireguard wg0 description Roadwarrior
set interfaces wireguard wg0 port 51820

# Add first peer
run generate pki wireguard preshared-key install interface wg0 peer tim
set interfaces wireguard wg0 peer tim persistent-keepalive 15
set interfaces wireguard wg0 peer tim allowed-ips 172.17.40.100/32

run generate pki wireguard key-pair
set interfaces wireguard wg0 peer tim public-key IBiGrXgZRWdDoUmWwSgUCbH4mTcfNUDtJdl461ACySE=

commit; save; exit

Now generate client configs and install these on your clients

generate wireguard client-config tim interface wg0 server home.vanwerkhoven.org address 172.17.40.10/24

Install & configure Debian server as LXC

Options:

LXC: light container, lower overhead, possibility for hardware acceleration – current proposal, easier
VM: fully separated, higher overhead, more secure

Could check Reddit post “Proxmox 7 LXC vs VM performance”

Get images using Proxmox’ Proxmox VE Appliance Manager:

pveam update
pveam available
pveam download local debian-11-standard_11.6-1_amd64.tar.zst
pveam list local

Check storage to use

pvesm status

Create and configure LXC container based on downloaded image, use all remaining RAM and storage (trial and error for disk size):

pct create 300 local:vztmpl/debian-11-standard_11.6-1_amd64.tar.zst --description "Debian LXC server" --hostname proteus --rootfs thinpool1:1000 --ssh-public-keys ~/.ssh/id_rsa-tim.pub
pct set 300 --cores 4 --memory 12288

Now configure networking, on Proxmox’ vmbr0 with VLAN ID 10. This means the guest can only

# this does not work, cannot create network device on vmbr0.10
# pct set 300 --net0 name=eth0,bridge=vmbr0.10,firewall=0,gw=172.19.10.1,ip=172.19.10.2/24
# does not work
# pct set 300 --net0 name=eth0,bridge=vmbr0,firewall=0,gw=172.17.10.1,ip=172.17.10.2/24,trunks=10
# works
pct set 300 --net0 name=eth0,bridge=vmbr0,firewall=0,gw=172.17.10.1,ip=172.17.10.2/24,tag=10
pct set 300 --onboot 1
pct set 300 --searchdomain lan.vanwerkhoven.org --nameserver 172.17.10.1

Start & log in, set root password, configure some basics

pct start 300
pct enter 300

passwd
apt install sudo
dpkg-reconfigure locales

Add regular user, add to groups, and set ssh key

adduser tim
usermod -aG sudo,adm,dialout,cdrom,dip,plugdev,render tim
mkdir -p ~tim/.ssh/
touch ~tim/.ssh/authorized_keys
chown -R tim:tim ~tim/.ssh
chmod og-rwx ~tim/.ssh/authorized_keys
cat <<EOF >>~tim/.ssh/authorized_keys
ssh-rsa AAAA[...]Hs= tim@neptune
EOF

# Allow non-root to use ping
setcap cap_net_raw+p $(which ping)

If SSH into guest fails or takes a long time, this can be due to LXC / Apparmor security features which prevent mount from executing. To solve, ensure nesting is allowed:

pct set 300 --features nesting=1

Mount Samba share automatically from pve host:

sudo apt install smbclient cifs-utils

cat <<EOF >>/root/.smbcredentials
user=sambarw
password=redacted
EOF

smbclient '//pve.lan.vanwerkhoven.org/bulk' -U sambarw

Automount

sudo mkdir /mnt/bulk
sudo chown root:users /mnt/bulk/
sudo chmod g+rw /mnt/bulk/
sudo cat <<EOF >>/etc/fstab
//pve.lan.vanwerkhoven.org/bulk /mnt/bulk   cifs    credentials=/root/.smbcredentials,rw,uid=tim,gid=users,noauto,x-systemd.automount,_netdev      0       0
EOF

Install docker and portainer, start adding containers:

# Now enter guest via SSH as regular user
sudo apt install docker.io
sudo docker volume create portainer_data
sudo docker run -d -p 8000:8000 -p 9443:9443 --name portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ce:latest
sudo docker ps

Proxmox hardening

Tips from Samuel’s Website and pveproxy(8) man page

Limit server access to specific IPs:

cat <<EOF >>/etc/default/pveproxy
# TvW 20230114 added for security reasons
DENY_FROM="all"
ALLOW_FROM="172.17.10.0/24"
POLICY="allow"

# For PVE-Manager >= 6.4 only.
LISTEN_IP="172.17.10.4"
EOF

Disable NFS:

cat <<EOF >>/etc/default/nfs-common
# TvW 20230114 disabled for security reasons
NEED_STATD=no
EOF

Migrate services

Install dependencies, prefer python packages via apt for system-wide install and potentially some security because we don’t install from public pip repository

apt install jq curl python3-netcdf4

InfluxDB + data (port X) - via apt 1.6 or special repo 1.8
- Port configuration from old proteus
- Set up new influxdb with accounts (generic write user and read user and admin)
Worker scripts Worker name: <source>2<target>, e.g. knmi2influxdb 2. co2signal * normalize, separate secrets, add influxdb login: OK * tested: OK 6. knmi * normalize, separate secrets, add influxdb login * tested: OK
1. SBF capture
  - Check which scripts are being used, archive old ones
  - Read secrets from external file
2. epexspot
3. evohome
  - migrate to HA: yes?
4. hue
5. mkwebdata
6. mqtt2influxdb
7. multical
8. smeter
9. water_meter_reader
Collectd (for data generation/collection) – on proxmox
- install on proxmox
- migrate configuration
Grafana (port Y) - via special apt
Nginx + letsencrypt (port 80/443)
Docker
- portainer (port 9000)
- Nextcloud (remap ports to nginx reverse proxy)
- bpatrik/pigallery2 (for personal photo sharing) (remap ports to nginx reverse proxy)
1. Home Assistant (port X) - as docker image
- lscr.io/linuxserver/unifi-controller (for Unifi AP management)

Transmission (downloading torrents)
Mosquitto (glueing home automation) *
Plex/Jellyfin (HTPC) – needs hw accel, required running in privileged container
smbd (for Time Machine backups) – at proxmox now OK

migrate all services
Get HW accel in guest/container: https://www.reddit.com/r/jellyfin/comments/s417qw/hardware_acceleration_inside_proxmox_lxc_not/

InfluxDB

Use (old) native Debian package for stability & least apt repositories

apt install influxdb-client influxdb
influxd restore -portable 20230124/

Add users in Influxdb

CREATE USER influxadmin WITH PASSWORD 'pwd' WITH ALL PRIVILEGES

CREATE USER influxwrite WITH PASSWORD 'pwd'
GRANT WRITE ON collectd TO influxwrite
GRANT WRITE ON smarthomev3 TO influxwrite
CREATE USER influxread WITH PASSWORD 'pwd'
GRANT READ ON collectd TO influxread
GRANT READ ON smarthomev3 TO influxread

Test account wiht curl

chmod o-r ~/.profile
cat <<EOF >>~/.profile
export INFLUX_USERNAME=influxadmin
export INFLUX_PASSWORD=pwd
EOF

curl -G http://localhost:8086/query -u influxwrite:pwd   --data-urlencode "q=SHOW DATABASES"

Restore retention policies (https://web.archive.org/web/20230104021722/https://atomstar.tweakblogs.net/blog/17748/influxdb-retention-policy-and-data-downsampling)

SHOW RETENTION POLICIES ON collectd
CREATE RETENTION POLICY "always" ON "collectd" DURATION INF REPLICATION 1
CREATE RETENTION POLICY "five_days" ON "collectd" DURATION 5d REPLICATION 1 DEFAULT

# For Grafana viewing - see https://github.com/grafana/grafana/issues/4262#issuecomment-475570324
INSERT INTO always rp_config,idx=1 rp="five_days",start=0i,end=432000000i -9223372036854775806
INSERT INTO always rp_config,idx=2 rp="always",start=432000000i,end=3110400000000i -9223372036854775806

# Restore continuous queries
cq_60m_cpu       CREATE CONTINUOUS QUERY cq_60m_cpu ON collectd BEGIN SELECT mean(value) AS value INTO collectd.always.cpu FROM collectd.five_days.cpu GROUP BY time(1h), * END
cq_60m_cpufreq   CREATE CONTINUOUS QUERY cq_60m_cpufreq ON collectd BEGIN SELECT mean(value) AS value INTO collectd.always.cpufreq FROM collectd.five_days.cpufreq GROUP BY time(1h), * END
cq_60m_df        CREATE CONTINUOUS QUERY cq_60m_df ON collectd BEGIN SELECT mean(value) AS value INTO collectd.always.df FROM collectd.five_days.df GROUP BY time(1h), * END
cq_60m_interface CREATE CONTINUOUS QUERY cq_60m_interface ON collectd BEGIN SELECT mean(rx) AS rx, mean(tx) AS tx INTO collectd.always.interface FROM collectd.five_days.interface GROUP BY time(1h), * END
cq_60m_iwinfo    CREATE CONTINUOUS QUERY cq_60m_iwinfo ON collectd BEGIN SELECT mean(value) AS value INTO collectd.always.iwinfo FROM collectd.five_days.iwinfo GROUP BY time(1h), * END
cq_60m_load      CREATE CONTINUOUS QUERY cq_60m_load ON collectd BEGIN SELECT mean(longterm) AS longterm, mean(midterm) AS midterm, mean(shortterm) AS shortterm INTO collectd.always.load FROM collectd.five_days.load GROUP BY time(1h), * END
cq_60m_memory    CREATE CONTINUOUS QUERY cq_60m_memory ON collectd BEGIN SELECT mean(value) AS value INTO collectd.always.memory FROM collectd.five_days.memory GROUP BY time(1h), * END
cq_60m_ping      CREATE CONTINUOUS QUERY cq_60m_ping ON collectd BEGIN SELECT mean(value) AS value INTO collectd.always.ping FROM collectd.five_days.ping GROUP BY time(1h), * END

Home assistant

Set up home assistant

# Create backup on old config (HA Core)
sudo tar cvf ~/homeassistant.tar.gz ~homeassistant/.homeassistant

# Move to new machine & right place
scp oldserver:homeassistant.tar.gz newserver:/var/lib/
cd /var/lib/ && sudo tar cvf ./homeassistant.tar.gz
sudo mv .homeassistant homeassistant
sudo chown root:root homeassistant
sudo chmod og-rwx homeassistant/

# Run new docker, do not use --privileged for safety and easier running in LXC
# https://community.home-assistant.io/t/why-does-the-documentation-say-we-need-priviledged-mode-for-a-docker-install-now/336556/2
docker run -d \
  --name homeassistant \
  --restart=unless-stopped \
  -e TZ=Europe/Brussels \
  -v /var/lib/homeassistant:/config \
  --network=host \
  ghcr.io/home-assistant/home-assistant:stable

Migrate HA to MariaDB

Optimize configuration: add mariadb, influxdb, tweak recorder to only store relevant stuff https://smarthomescene.com/guides/optimize-your-home-assistant-database/ https://community.home-assistant.io/t/migrating-home-assistant-database-from-sqlite-to-mariadb/96895

# Remove 
sudo apt remove mysql-server-8.0 mysql-server mysql-client-8.0 mysql-client-core-8.0 mysql-common
sudo apt-get remove --purge
sudo apt install mariadb-server

# Fix apparmor because of old mysql installation
# https://askubuntu.com/questions/1185710/mariadb-fails-despite-apparmor-profile
# https://stackoverflow.com/questions/40997257/mysql-service-fails-to-start-hangs-up-timeout-ubuntu-mariadb
echo "# TvW 20230127 fix apparmor issue mariadb" | sudo tee -a /etc/apparmor.d/usr.sbin.mysqld
echo "/usr/sbin/mysqld { }" | sudo tee -a /etc/apparmor.d/usr.sbin.mysqld
sudo apparmor_parser -v -R /etc/apparmor.d/usr.sbin.mysqld
sudo systemctl restart mariadb
sudo reboot
# Did not help? Try reboot
#sudo /etc/init.d/apparmor reload

sudo mysql_secure_installation

## create database
mysql -e 'CREATE SCHEMA IF NOT EXISTS `hass_db` DEFAULT CHARACTER SET utf8mb4'
## create user (use a safe password please)
mysql -e "CREATE USER 'hass_user'@'localhost' IDENTIFIED BY 'pwd'"
mysql -e "GRANT ALL PRIVILEGES ON hass_db.* TO 'hass_user'@'localhost'"
mysql -e "GRANT usage ON *.* TO 'hass_user'@'localhost'"

Migrate: method 1, use only SQL

pip install sqlite3-to-mysql
sqlite3mysql -f ./home-assistant_v2.db -d hass_db -u hass_user -p

Migrate: method 2, todo

sqlite3 ~homeassistant/.homeassistant/home-assistant_v2.db .dump > hadump.sql
git clone https://github.com/athlite/sqlite3-to-mysql


recorder:
  auto_purge: true
  purge_keep_days: 21
  auto_repack: true
  db_url: mysql://user:password@localhost/homeassistant?unix_socket=/var/run/mysqld/mysqld.sock&charset=utf8mb4

Grafana

@TODO

Mosquitto

@TODO

Worker scripts

@TODO

Collectd

@TODO

Nginx

@TODO

Transmission

@TODO

Jellyfin

@TODO

Various

Pass through USB devices

https://forum.proxmox.com/threads/lxc-usb-passthrough-zwave-stick.30058/ https://medium.com/@konpat/usb-passthrough-to-an-lxc-proxmox-15482674f11d https://forum.proxmox.com/threads/lxc-usb-webcam-passthrough.107309/

Destroy VGs

vgdisplay vgreduce –removemissing –force pve vgremove pve pvdisplay pvremove /dev/sda1

XCP-NG

Install XCP-NG – Has Linux xcp-ng 4.19.0+1
Install Xen Orchestra
Fix scaling governor – xenpm get-cpufreq-para – xenpm set-scaling-governor ondemand – xenpm set-scaling-governor powersave

Partitioning scheme LVM + LUKS + UEFI

I’m using LVM to merge two SSDs into one volume. Additionally I’m encrypting the volumes (maybe). If you want to boot with UEFI, you need a partition layout like:

disk0p1: UEFI (~100MB)
disk0p2: /boot (~1900MB)
LVM of disk0 and disk1 (1+5TB) – swap (1-2x RAM, i.e. 8GB) — Optional: encrypted swap – / (remainder, ~5TB) — Optional: encrypted /

Encrypting volumes requires entering the passphrase upon boot. This makes sense but is highly inconvenient. Maybe I should use user-space encryption instead.

Packages / configuration

Migrate Deconz to Home Assistant

Stop deconz

sudo service deconz stop

Start Zigbee Home Assistant

Enable pairing mode

Write new poll script to influxdb

–> push from HA via mqtt OR use HA API to query?

automation on change, publish

sensor.innr_sp_120_smartenergysummation -- dryer
sensor.innr_sp_120_smartenergysummation_3 -- fridge
sensor.innr_sp_120_smartenergysummation_2 -- laptop
sensor.innr_sp_120_425efe01_smartenergy_metering_summation_delivered -- metercupboard

sensor.lumi_lumi_weather_humidity -- metercupboard
sensor.lumi_lumi_weather_pressure -- metercupboard
sensor.lumi_lumi_weather_temperature -- metercupboard

Uninstall deconz

sudo apt remove deconz-dev -y

Home Assistant recorder

influxdb:
host: 192.168.0.xxx port: 8086 database: homeassistant username: homeassistant password: !secret influxdb_pass max_retries: 3 default_measurement: state

recorder: purge_keep_days: 5 db_url: sqlite:////home/user/.homeassistant/test exclude: domains: - automation - updater entity_globs: - sensor.weather_* entities: - sun.sun # Don’t record sun data - sensor.last_boot # Comes from ‘systemmonitor’ sensor platform - sensor.date event_types: - call_service # Don’t record service calls

include: domains: - sensor - switch

recorder: include: domains: - alarm_control_panel - light entity_globs: - binary_sensor.*_occupancy exclude: entities: - light.kitchen_light

Known issues

Check if we get issues in the ASIX AX88179 USB dongle https://github.com/FreddyXin/ax88179_178a/issues/6

Using USB network dongle under linux

Malfunctioning USB-C dongle (RTL8152B)

Model as identified under Windows: ‘Realtek USB FE Family’
Label on dongle ‘USB 2.0 to fast ethernet adatper Model NO: JCX-010-LAN 100’
Chipset Realtek RTL8152B

Base situation:

Not recognized in dmesg
Not recognized under lsusb

sudo apt install firmware-realtek

Does not help

Trying https://github.com/awesometic/realtek-r8152-dkms

Build DKMS module ourselves

sudo apt install linux-headers-5.10.0-16-amd64

Also did not work.

Trying drivers from github/wget, with proxmox pve headers and build tools:

wget https://github.com/wget/realtek-r8152-linux/archive/refs/tags/v2.16.3.20221209.tar.gz
tar xvf v2.16.3.20221209.tar.gz
cd realtek-r8152-linux-2.16.3.20221209/

apt install pve-headers-$(uname -r)
apt install build-essential

make
make install

depmod -a
update-initramfs -u

Also does not work

Configuring ASIX AX88179 USB 3.0/C ethernet adapter

Chip specs: https://www.asix.com.tw/en/product/USBEthernet/Super-Speed_USB_Ethernet/AX88179

Should work in proxmox: https://forum.proxmox.com/threads/solved-the-problem-problem-with-2-usb-network-cards-asix-ax88179.101732/

Netgear thread: https://forum.netgate.com/topic/105696/intel-nuc-with-startech-usb-gigabit-nic-chipset-asix-ax88179/2

Review of USB ethernet dongles: https://www.virten.net/2020/09/tips-for-using-usb-network-adapters-with-vmware-esxi/

Enable Thunderbolt port

Power usage

Right after clean Debian install, no monitor, wired internet, no WLAN, SSH enabled: 3.9-4.0W

Reduce power consumption

Software

#networking #nextcloud #nginx #security #server #smarthome #debian #vyos #proxmox #unix

Homelab Proxmox + VyOS + Debian server configuration - WIP

Contents

Goal

Hardware

Target services & architecture

Software distribution

Installation & configuration

Proxmox

Installation

Networking

Storage

Optional tweaks & bugfixes

Install VyOS & configure as VM

Configure VyOS

Install & configure Debian server as LXC

Proxmox hardening

Migrate services

InfluxDB

Home assistant

Migrate HA to MariaDB

Grafana

Mosquitto

Worker scripts

Collectd

Nginx

Transmission

Jellyfin

Various

Pass through USB devices

Destroy VGs

XCP-NG

Partitioning scheme LVM + LUKS + UEFI

Packages / configuration

Migrate Deconz to Home Assistant

Home Assistant recorder

Known issues

Using USB network dongle under linux

Malfunctioning USB-C dongle (RTL8152B)

Configuring ASIX AX88179 USB 3.0/C ethernet adapter

Power usage

Software