Differences

This shows you the differences between two versions of the page.

--- notes:proxmox [2024/04/02 12:05] – Nikita Kipriyanov
+++ notes:proxmox [2024/06/26 06:33] (current) – [ECC error notifications] Nikita Kipriyanov
@@ Line 1: / Line 1: @@
 ====== Proxmox virtualization cluster ======
+  * LXC images collection: http://download.proxmox.com/images/system/
 ===== Storage =====
 ===== Network structure =====
 ===== System configuration =====
+==== Bootloader ====
+=== Console redirection, VSP, IPMI SoL ===
+At least when installation was done using serial console, the Proxmox installer configures system in a very useful manner, so that bootloader and the kernel appear both on the serial and the VGA console:
+<file - /etc/default/grub.d/installer.cfg>
+GRUB_TERMINAL_INPUT="console serial"
+GRUB_TERMINAL_OUTPUT="gfxterm serial"
+GRUB_SERIAL_COMMAND="serial --unit=0 --speed=115200"
+GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUX    console=ttyS0,115200"
+</file>
+This allows later convenient access to the system via IPMI Serial-over-LAN. The inconvenient part of this that it diverts kernel boot messages (''kmsg'') away from VGA console. It's possible to show them both on VGA and TTY, for which we create yet another file:
+<file - /etc/default/grub.d/vgaconsole.cfg>
+GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUX console=tty0"
+</file>
+I also don't like the default "quiet" option for boot messsages, so I override it:
+<file /etc/default/grub.d/verbose.cfg>
+GRUB_CMDLINE_LINUX_DEFAULT=""
+</file>
+In case of installation over Debian, where Proxmox installer did not run, all this setup needs to be replicated:
+<file - /etc/default/grub.d/console.cfg>
+GRUB_CMDLINE_LINUX_DEFAULT=""
+GRUB_CMDLINE_LINUX="console=tty0 console=ttyS1,57600n8"
+GRUB_TERMINAL="serial console"
+GRUB_SERIAL_COMMAND="serial --speed=57600 --unit=1 --word=8 --parity=no --stop=1"
+</file>
+Other COM port is used with another speed here, just for illustration. Notice also, instead of separate ''GRUB_TERMINAL_INPUT'' and ''GRUB_TERMINAL_OUTPUT'' I use a single setting for both.
+==== Fast reboots with kexec ====
+FIXME this needs more work
+  * https://forum.proxmox.com/threads/tip-fast-reboots-with-kexec.35624/
+  * https://forum.proxmox.com/threads/proxmox-7-fast-reboot-with-kexec.93422/
 ==== Time sync ====
+Proxmox recommends using ''chrony'' for the NTP synchronization, and **advises against** ''systemd-timesync''. To configure NTP servers, it's necessary to create an additional file in a drop directory:
+<code bash>
+echo 'server 10.226.130.130 iburst' > /etc/chrony/sources.d/local-ntp-server.sources
+echo 'server 10.226.130.131 iburst' >> /etc/chrony/sources.d/local-ntp-server.sources
+chronyc reload sources
+</code>
+==== ECC error notifications ====
+Install `rasdaemon` utility to receive reports from hardware via EDAC interface and get them into logs.
+See [[https://www.setphaserstostun.org/posts/monitoring-ecc-memory-on-linux-with-rasdaemon/]]
 ==== Backup ====
@@ Line 17: / Line 72: @@
 ''0'' means "use half of available cores". If you know the number of the cores in the target system, you can use other, more tailored settings here. We set ''pigz'' too, just for the case somebody changes to gzip.
+**Note:** it's possible to [[https://forum.proxmox.com/threads/reading-blob-files-qemu-server-conf-blob.95551/|read blob files]] directly on the server, either with `proxmox-backup-debug` or "by hand":
+<code>
+proxmox-backup-debug inspect file /path/to/blob --decode -
+dd if=/path/to/blob bs=1 skip=12 | zstdcat
+</code>
 === Node backup ===
-Hosts may need backup, too. It is useful to create a simple shell script and run it, say, mounthly:
+Hosts may need backup, too. Theoretically we need "thin" backups as barely needed for the recovery, as explained here: https://pve.proxmox.com/wiki/Proxmox_VE_4.x_Cluster#Re-installing_a_cluster_node . Yet, I still find it useful to do just full host backup. It will not be very large (in my experience, around 5 GB), it will be strongly deduplicated (between backups of same node and between nodes — they're similar), so why bother?
+It is useful to create a simple shell script and run it, say, mounthly:
 <file bash /etc/cron.mounthly/backup-node.sh>
@@ Line 26: / Line 89: @@
 export PBS_REPOSITORY=<token name>@<server>:<datastore>
 export PBS_PASSWORD=<token secret>
+NS=<namespace>
+NOTES=$(hostname -f)
+TMP=$(mktemp -d -p /dev/shm)
 if mountpoint -q /boot/efi
 then
     # for modern UEFI boot
-    proxmox-backup-client backup --ns <namespace> root.pxar:/ pve.pxar:/etc/pve esp.pxar:/boot/efi
+    proxmox-backup-client backup --ns ${NS} root.pxar:/ pve.pxar:/etc/pve exp.pxar:/boot/efi 2>&1 | tee ${TMP}/client.log
 else
     # for legacy BIOS boot
-    proxmox-backup-client backup --ns <namespace> root.pxar:/ pve.pxar:/etc/pve sda1.img:/dev/sda1
+    proxmox-backup-client backup --ns ${NS} root.pxar:/ pve.pxar:/etc/pve sda1.img:/dev/sda1 2>&1 | tee ${TMP}/client.log
 fi
+SNAPSHOT=$(grep "Starting backup:" ${TMP}/client.log | cut -d':' -f 3-)
+proxmox-backup-client snapshot upload-log --ns ${NS} ${SNAPSHOT} ${TMP}/client.log
+rm -rf ${TMP}
+proxmox-backup-client snapshot notes update --ns ${NS} ${SNAPSHOT} ${NOTES}
 </file>
@@ Line 47: / Line 120: @@
 If the node installation was performed by converting Debian bookworm system, you need to adjust the backup command accordingly.
+If you find this too wasteful, read this thread and invent your own backup script: https://forum.proxmox.com/threads/backup-and-restore-node.115161/
 ===== Debian repositories =====