Differences

This shows you the differences between two versions of the page.

--- notes:proxmox [2024/04/02 06:54] – Nikita Kipriyanov
+++ notes:proxmox [2024/06/26 06:33] (current) – [ECC error notifications] Nikita Kipriyanov
@@ Line 1: / Line 1: @@
 ====== Proxmox virtualization cluster ======
+  * LXC images collection: http://download.proxmox.com/images/system/
+===== Storage =====
+===== Network structure =====
+===== System configuration =====
+==== Bootloader ====
+=== Console redirection, VSP, IPMI SoL ===
+At least when installation was done using serial console, the Proxmox installer configures system in a very useful manner, so that bootloader and the kernel appear both on the serial and the VGA console:
+<file - /etc/default/grub.d/installer.cfg>
+GRUB_TERMINAL_INPUT="console serial"
+GRUB_TERMINAL_OUTPUT="gfxterm serial"
+GRUB_SERIAL_COMMAND="serial --unit=0 --speed=115200"
+GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUX    console=ttyS0,115200"
+</file>
+This allows later convenient access to the system via IPMI Serial-over-LAN. The inconvenient part of this that it diverts kernel boot messages (''kmsg'') away from VGA console. It's possible to show them both on VGA and TTY, for which we create yet another file:
+<file - /etc/default/grub.d/vgaconsole.cfg>
+GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUX console=tty0"
+</file>
+I also don't like the default "quiet" option for boot messsages, so I override it:
+<file /etc/default/grub.d/verbose.cfg>
+GRUB_CMDLINE_LINUX_DEFAULT=""
+</file>
+In case of installation over Debian, where Proxmox installer did not run, all this setup needs to be replicated:
+<file - /etc/default/grub.d/console.cfg>
+GRUB_CMDLINE_LINUX_DEFAULT=""
+GRUB_CMDLINE_LINUX="console=tty0 console=ttyS1,57600n8"
+GRUB_TERMINAL="serial console"
+GRUB_SERIAL_COMMAND="serial --speed=57600 --unit=1 --word=8 --parity=no --stop=1"
+</file>
+Other COM port is used with another speed here, just for illustration. Notice also, instead of separate ''GRUB_TERMINAL_INPUT'' and ''GRUB_TERMINAL_OUTPUT'' I use a single setting for both.
+==== Fast reboots with kexec ====
+FIXME this needs more work
+  * https://forum.proxmox.com/threads/tip-fast-reboots-with-kexec.35624/
+  * https://forum.proxmox.com/threads/proxmox-7-fast-reboot-with-kexec.93422/
+==== Time sync ====
+Proxmox recommends using ''chrony'' for the NTP synchronization, and **advises against** ''systemd-timesync''. To configure NTP servers, it's necessary to create an additional file in a drop directory:
+<code bash>
+echo 'server 10.226.130.130 iburst' > /etc/chrony/sources.d/local-ntp-server.sources
+echo 'server 10.226.130.131 iburst' >> /etc/chrony/sources.d/local-ntp-server.sources
+chronyc reload sources
+</code>
+==== ECC error notifications ====
+Install `rasdaemon` utility to receive reports from hardware via EDAC interface and get them into logs.
+See [[https://www.setphaserstostun.org/posts/monitoring-ecc-memory-on-linux-with-rasdaemon/]]
+==== Backup ====
+Useful to set up using ''zstd'' by default, since it's both **completes faster** and **has better compression** at the same time than ''gzip''. Also we want it to use multiple cores. For that, new systems should have in ''/etc/vzdump.conf'' at least the following:
+<code>
+compress: zstd
+pigz: 0
+zstd: 0
+</code>
+''0'' means "use half of available cores". If you know the number of the cores in the target system, you can use other, more tailored settings here. We set ''pigz'' too, just for the case somebody changes to gzip.
+**Note:** it's possible to [[https://forum.proxmox.com/threads/reading-blob-files-qemu-server-conf-blob.95551/|read blob files]] directly on the server, either with `proxmox-backup-debug` or "by hand":
+<code>
+proxmox-backup-debug inspect file /path/to/blob --decode -
+dd if=/path/to/blob bs=1 skip=12 | zstdcat
+</code>
+=== Node backup ===
+Hosts may need backup, too. Theoretically we need "thin" backups as barely needed for the recovery, as explained here: https://pve.proxmox.com/wiki/Proxmox_VE_4.x_Cluster#Re-installing_a_cluster_node . Yet, I still find it useful to do just full host backup. It will not be very large (in my experience, around 5 GB), it will be strongly deduplicated (between backups of same node and between nodes — they're similar), so why bother?
+It is useful to create a simple shell script and run it, say, mounthly:
+<file bash /etc/cron.mounthly/backup-node.sh>
+#!/usr/bin/bash
+export PBS_FINGERPRINT=<whatever PBS shows>
+export PBS_REPOSITORY=<token name>@<server>:<datastore>
+export PBS_PASSWORD=<token secret>
+NS=<namespace>
+NOTES=$(hostname -f)
+TMP=$(mktemp -d -p /dev/shm)
+if mountpoint -q /boot/efi
+then
+    # for modern UEFI boot
+    proxmox-backup-client backup --ns ${NS} root.pxar:/ pve.pxar:/etc/pve exp.pxar:/boot/efi 2>&1 | tee ${TMP}/client.log
+else
+    # for legacy BIOS boot
+    proxmox-backup-client backup --ns ${NS} root.pxar:/ pve.pxar:/etc/pve sda1.img:/dev/sda1 2>&1 | tee ${TMP}/client.log
+fi
+SNAPSHOT=$(grep "Starting backup:" ${TMP}/client.log | cut -d':' -f 3-)
+proxmox-backup-client snapshot upload-log --ns ${NS} ${SNAPSHOT} ${TMP}/client.log
+rm -rf ${TMP}
+proxmox-backup-client snapshot notes update --ns ${NS} ${SNAPSHOT} ${NOTES}
+</file>
+This mathes the disk structure the Proxmox's installer creates:
+  * the paritition table is GPT
+  * ''sda1'' is a ''bios_grub'' partition of 1 MiB minus 34 sectors; it's used only on legacy systems
+  * ''sda2'' is ESP of 1GiB mounted as ''/boot/efi''; it's used only on UEFI systems
+  * ''sda3'' has everything else: it's either LVM or ZFS or BTRFS
+  * ''/etc/pve'' is a Proxmox's configuration file system mounted with FUSE, so it neeeds a dedicated clause for it contents to be backed up as files. It's already backed up into ''root.pxar'' because it is actually contained inside a SQLite file ''/var/lib/pve-cluster/config.db'', but to use that backup we need to mount it, which could be tricky in the event of disaster, so for convenience we back up it's files too
+There is no need to backup other copies of ESP or ''bios_grub'' partitions (e.g. ''/dev/sdb1'' and ''/dev/sdb2'' in case of "software RAID"), one copy is enough.
+If the node installation was performed by converting Debian bookworm system, you need to adjust the backup command accordingly.
+If you find this too wasteful, read this thread and invent your own backup script: https://forum.proxmox.com/threads/backup-and-restore-node.115161/
 ===== Debian repositories =====
@@ Line 56: / Line 178: @@
 On the HPE servers it's recommended to add their SDR MCP repository to access the useful tools to administer and monitor the hardware: https://downloads.linux.hpe.com/SDR/project/mcp/
-**Don't use** the [[https://downloads.linux.hpe.com/SDR/keys.html|procedure described by HP]] to add the keys into the keyring with `apt-key`. It's deprecated in Debian. Modern scheme is to put keys as separate files into ''/etc/apt/trusted.gpg.d'', either as PEM-encoded ''.asc'' files or GnuPG-encoded ''.gpg'' bundles. It's enough to **only download the last HPE key file** into the mentioned directory and rename it to have correct suffix:
+**Don't use** the [[https://downloads.linux.hpe.com/SDR/keys.html|procedure described by HP]] to add the keys into the keyring with ''apt-key''. It's deprecated in Debian. Modern scheme is to put keys as separate files into ''/etc/apt/trusted.gpg.d'', either as PEM-encoded ''.asc'' files or GnuPG-encoded ''.gpg'' bundles. It's enough to **only download the last HPE key file** into the mentioned directory and rename it to have correct suffix:
 <code bash>