Linux Automatic Network Install

Purpose: to install Linux automatically unattended, without any manual user input during the process.

This can be used in further automation using tool like Ansible. It should create a VM and obtain its MAC address from the virtualization environment, or reach the BMC and obtain a MAC address of the physical server. Then, knowing this info, it can generate all the boot configuration and installation scripts and wait for the OS to be installed. At last, it can customize it further in the next play. This facilitates a complete server provision with a single tool using a single configuration file in a single run.

General outline of the technology:

PXE boot server with support for “legacy” BIOS and EFI boot; most VMs use “legacy” and most recent equipment uses EFI
HTTP server for EL (it doesn't support downloading a kickstart file from TFTP server, what a shame)
A set of boot images and minimal autoinstall files for Debian (at least, Bookworm) and EL (at least, Oracle Linux 8)

Possible optional additions:

A proxy server to perform installation and updates in restricted environments
Set of repositories to store packages needed by installers, beneficial for mass installs. The set of “popular” packages could be even retrieved from the proxy server logs

Boot server configuration

We can use ISC DHCPD + BIND + tftpd-hpa, or dnsmasq on the server side, and GRUB or PXELinux/SysLinux as downloadable Linux boot loader. GRUB is beneficial because it is a single piece of software that supports both “legacy” and EFI boot with a single set of configuration files; however, the prior development of this topic was focused on primarily PXELinux. The latter doesn’t support EFI boot, which can be supported by SysLinux (a sibling program), but there is a caveat: it doesn’t have a convenient way to make per-system dedicated boot configuration. Dnsmasq’s feature can be used to fill this gap. Overall, GRUB looks more promising.

PXE bootloader

All (more or less) dynamic data is to be stored in /srv/tftp. The structure of the directory:

GRUB

GRUB manual is available here: https://www.gnu.org/software/grub/manual/grub/

srv/
┠─ dhcp-hostsfile.cfg
┗━ tftp/
   ┠─ grub.0 → grub/i386-pc/core.0
   ┠─ grub.efi → grub/x86_64-efi/core.efi
   ┣━ grub/
   ┃  ┣━ fonts/
   ┃  ┃  ┗━ ...
   ┃  ┣━ locale/
   ┃  ┃  ┗━ ...
   ┃  ┣━ i386-pc/
   ┃  ┃  ┠─ core.0
   ┃  ┃  ┗━ ...
   ┃  ┣━ x86_64-efi
   ┃  ┃  ┠─ core.efi
   ┃  ┃  ┗━ ...
   ┃  ┠─ localboot
   ┃  ┠─ menu
   ┃  ┠─ debian-bookworm
   ┃  ┠─ ol8u9
   ┃  ┠─ grub.cfg → menu
   ┃  ┠─ docker-vm → localboot
   ┃  ┠─ test-pve → debian-bookworm
   ┃  ┠─ test-lvm → ol8u9
   ┃  ┠─ grub.cfg-01-bc-24-11-51-cf-fc → docker-vm
   ┃  ┠─ grub.cfg-01-bc-24-11-4b-ae-dd → test-pve
   ┃  ┖─ grub.cfg-01-bc-24-11-28-81-76 → test-lvm
   ┣━ debian-bookworm/
   ┃  ┠─ initrd.gz
   ┃  ┠─ linux
   ┃  ┖─ preseed-noswap.cfg
   ┣━ ol8u9/
   ┃  ┣━ images/
   ┃  ┃  ┖─ install.img
   ┃  ┠─ initrd.img
   ┃  ┠─ ks.cfg
   ┃  ┖─ vmlinuz
   ┗━ gentoo/
      ┠─ gentoo
      ┖─ initramfs.igz

The most logic lies within /srv/tftp/grub directory. There are several boot configurations:

localboot — exit a bootloader, causing firmware to try next configured boot source
menu — present a human-friendly menu, with a timeout to do same thing as localboot
debian-bookworm — start an unattended installation of a Debian 12 release
ol8u9 — start an unattended installation of a Oracle Linux 8 release

The rest items are symlinks to these files:

grub.cfg — will be used by GRUB if no other configuration file was matched; here it symlinks to the menu, but can be diverted anywhere else
test-lvm, test-pve and docker-vm are VMs or HW machines which are provisioned using this system, and in particular, test-lvm is being provisioned with OL8, test-pve is being provisioned with Debian and the docker-vm is already installed, so it is instructed to boot locally
grub.cfg-01-xx-xx-xx-xx-xx-xx are symlinks named after the respective system's MAC address. If there are several NICs, we create several files, for each MAC. GRUB will first try such a file and load it if it exists, and if it doesn't, it will load grub.cfg. This way, we precisely target each system to divert its boot path independently of each other.

This approach is new to me, but it can support both “legacy” and EFI clients, and what's even more appelaing, with a single set of configuration files.

To generate the contents of that directory (including GRUB images themselves) a Gentoo system was used, where GRUB was emerged with GRUB_PLATFORMS="i386 x86_64-efi", with the following commands:

grub-mknetdir --net-directory=/srv/tftp --subdir=/grub -d /usr/lib/grub/i386-pc
grub-mknetdir --net-directory=/srv/tftp --subdir=/grub -d /usr/lib/grub/x86_64-efi

Then the /srv/tftp/grub directory was simply archived and transferred to the boot server.

PXELinux

srv/
┠─ dhcp-hostsfile.cfg
┗━ tftp/
   ┠─ pxelinux.0
   ┠─ ldlinux.c32
   ┣━ pxelinux.cfg/
   ┃  ┠─ harddisk
   ┃  ┠─ menu
   ┃  ┠─ debian-bookworm
   ┃  ┠─ ol8u9
   ┃  ┠─ default → menu
   ┃  ┠─ docker-vm → harddisk
   ┃  ┠─ test-pve → debian-bookworm
   ┃  ┠─ test-lvm → ol8u9
   ┃  ┠─ 01-bc-24-11-51-cf-fc → docker-vm
   ┃  ┠─ 01-bc-24-11-4b-ae-dd → test-pve
   ┃  ┖─ 01-bc-24-11-28-81-76 → test-lvm
   ┣━ debian-bookworm/
   ┃  ┠─ initrd.gz
   ┃  ┠─ linux
   ┃  ┖─ preseed-noswap.cfg
   ┣━ ol8u9/
   ┃  ┣━ images/
   ┃  ┃  ┖─ install.img
   ┃  ┠─ initrd.img
   ┃  ┠─ ks.cfg
   ┃  ┖─ vmlinuz
   ┗━ gentoo/
      ┠─ gentoo
      ┖─ initramfs.igz

The most logic is within /srv/tftp/pxelinux.cfg directory. There are several boot configurations:

harddisk — boot from local hard disk
menu — present a human-friendly menu, with a timeout to do the same thing as harddisk
debian-bookworm — start an unattended installation of a Debian 12 release
ol8u9 — start an unattended installation of a Oracle Linux 8 release

The rest items are symlinks to these files:

default — will be used by PXELinux if no other configuration file was matched; here it symlinks to the menu, but can be diverted anywhere else
test-lvm, test-pve and docker-vm are VMs or HW machines which are provisioned using this system, and in particular, test-lvm is being provisioned with OL8, test-pve is being provisioned with Debian and the docker-vm is already installed, so it is instructed to boot locally
01-xx-xx-xx-xx-xx-xx are symlinks named after the respective system's MAC address. If there are several NICs, we create several files, for each MAC. PXELinux will first try such a file and load it if it exists, and if it doesn't, it will load default. This way, we precisely target each system to divert its boot path independently of each other.

This is well tested setup. The main downside is that PXELinux doesn't support EFI boot, and we need to use some another bootloader which is capable of targeting computers precisely and manage its configuration separately.

Boot menu can contain other entries if needed, for example, it can boot Gentoo LiveCD converted for PXE boot. It defaults to booting from hard disk after timeout.

We need some more work in preseed/kickstart/whatever autoinstall partitioning schemes, to support both “legacy” and EFI partitioning and boot settings, or develop another generation of autoinstall files for EFI and select which one to present in a bootloader.

DNS, DHCP and TFTP servers

dnsmasq

The default Debian's dnsmasq configuration can use the base configuration and the drop directory. We opt to use the latter.

The most configuration is /etc/dnsmasq.d/base.conf:

/etc/dnsmasq.d/base.conf

no-dhcp-interface = tun0

# upstream DNS
server = 10.226.130.130
server = 10.226.130.131
no-resolv

domain = auto.example.org, 172.31.1.0/24,   local
domain = auto.example.org, 172.31.255.0/24, local
domain-needed
expand-hosts

# no dynamic allocation for DHCP service on these subnets; static allocation via dhcp-host and dhcp-hostsfile will still be possible
dhcp-range = 192.168.205.224, static
dhcp-range = 172.31.255.0,    static

# allocate dynamically from this subnet
dhcp-range = 172.31.1.34, 172.31.1.62, 1h

dhcp-authoritative
dhcp-option = option:ntp-server, 10.226.130.130, 10.226.130.131

dhcp-hostsfile = /srv/dhcp-hostsfile.cfg

enable-tftp
tftp-root = /srv/tftp

dhcp-match = set:bios, option:client-arch,  0
dhcp-match = set:uefi, option:client-arch,  7
dhcp-match = set:uefi, option:client-arch,  9
dhcp-match = set:http, option:client-arch, 16

dhcp-boot = tag:bios, grub.0
dhcp-boot = tag:uefi, grub.efi

log-queries
log-dhcp

It references the /srv/dhcp-hostsfile.cfg where we can match certain MACs to IP addresses and names. This is convenient for networks defined as “static DHCP ranges” above, since this way only hosts listed in the hostsfile will be provided a service. Also we can put other hosts there, for example, VMs, to pin IP addresses to somewhere outside the dynamic allocation range. Example contents is:

/srv/dhcp-hostsfile.cfg

BC:24:11:51:CF:FC, 172.31.1.5,  docker-vm
BC:24:11:70:C7:44, 172.31.1.6,  docker-ct
BC:24:11:4B:AE:DD, 172.31.1.4,  test-pve
BC:24:11:D4:97:9C, 172.31.1.27, hpprobook-image
BC:24:11:C9:B6:69, 172.31.1.7,  gns3
BC:24:11:28:81:76, 172.31.1.28, test-lvm

Also the static hosts may be defined in a sidecar file living in /etc/dnsmasq.d/:

/etc/dnsmasq.d/dhcp-static-hosts.conf

dhcp-host = 38:ea:a7:33:b5:98, 38:ea:a7:33:b5:99, 172.31.255.5,  vh-e1-b5
dhcp-host = 38:ea:a7:32:40:7c, 38:ea:a7:32:40:7d, 172.31.255.6,  vh-e1-b6
dhcp-host = 38:ea:a7:91:77:a8, 38:ea:a7:91:77:a9, 172.31.255.13, vh-e1-b13
dhcp-host = 38:ea:a7:91:76:44, 38:ea:a7:91:76:45, 172.31.255.14, vh-e1-b14

Auxiliary DNS records are put into yet another separate file:

/etc/dnsmasq.d/dns-auxiliary.conf

cname = boot.test.ucom.am, vmgw.test.ucom.am

ISC DHCPD, ISC BIND, tftpd-hpa

Configuring PXE servers for UEFI-based clients in a Linux environment

HTTP server

Needed for EL variants to be able to run installer from the network. I went as low as using mini_httpd, the simplest I found. Only need to set data_dir=/srv/tftp in /etc/mini-httpd.conf.

This directory is already served via TFTP; we can use a separate directory, but this setup feels simpler. If it will be decided to serve private local repositories, which is infeasible to share over TFTP, it might be worth splitting the HTTP-only and TFTP-only parts.

Debian: preseed

Network installer files

Debian conveniently maintains all the required bits for the netboot as an archive distributed through its mirrors, available, for example, here. From it, we need only two files (which can be also downloaded separately):

File	Target	Description
`debian-installer/amd64/initrd.gz`	`/srv/tftp/debian-bookworm/initrd.gz`	initramfs
`debian-installer/amd64/linux`	`/srv/tftp/debian-bookworm/linux`	kernel

Also need to create a bootloader configuration as /srv/tftp/pxelinux.cfg/debian-bookworm and a preseed file as /srv/tftp/debian-bookworm/preseed-noswap.cfg.

Bootloader configuration

GRUB

/srv/tftp/grub/debian-bookworm

set timeout=1

menuentry 'Debian Bookworm PXE installer' {
    linux debian-bookworm/linux priority=critical debian-installer/language=en debian-installer/country=AM debian-installer/locale=ru_RU.UTF-8 keyboard-configuration/xkb-keymap=ru netcfg/get_hostname=install keyboard-configuration/optionscode=grp:caps_lock_toggle,grp_led:scroll preseed/url=tftp://boot.test.ucom.am/debian-bookworm/preseed-noswap.cfg
    initrd debian-bookworm/initrd.gz
}

PXELinux

/srv/tftp/pxelinux.cfg/debian-bookworm

default debian-bookworm-auto
timeout 0

label debian-bookworm-auto
    kernel debian-bookworm/linux
    append priority=critical initrd=debian-bookworm/initrd.gz debian-installer/language=en debian-installer/country=AM debian-installer/locale=ru_RU.UTF-8 keyboard-configuration/xkb-keymap=ru netcfg/get_hostname=install keyboard-configuration/optionscode=grp:caps_lock_toggle,grp_led:scroll preseed/url=tftp://boot.test.ucom.am/debian-bookworm/preseed-noswap.cfg

Example preseed file

Works for Bullseye too (it was actually originally developed for Bullseye, but worked with Bookworm like a charm)!

BIOS

/srv/tftp/debian-bookworm/preseed-noswap.cfg

#_preseed_V1

# Preseeding only locale sets language, country and locale.

# For network booting, this comes too late; one needs to put those into command line,
# and the language there should be "en" or it will ask about it on the terminal
d-i debian-installer/language string ru
d-i debian-installer/country string AM
d-i debian-installer/locale string ru_RU.UTF-8
d-i localechooser/supported-locales multiselect ru_RU.UTF-8

# Keyboard selection.
d-i console-setup/ask_detect boolean false
d-i keyboard-configuration/xkb-keymap select ru
d-i keyboard-configuration/optionscode string grp:caps_toggle,lv3:ralt_switch,compose:rctrl,grp_led:scroll
d-i keyboard-configuration/layoutcode string us,ru
d-i keyboard-configuration/variantcode string ,
d-i keyboard-configuration/toggle select Caps Lock
d-i keyboard-configuration/altgr select Right Alt (AltGr)
d-i keyboard-configuration/compose select Right Control

### Clock and time zone setup
d-i clock-setup/utc boolean true
d-i time/zone string Asia/Yerevan

### Mirror settings
# If you select ftp, the mirror/country string does not need to be set.
d-i mirror/country string AM
d-i mirror/http/directory string /debian/
d-i mirror/http/hostname string deb.debian.org
d-i mirror/https/hostname string deb.debian.org
d-i mirror/http/mirror select deb.debian.org
d-i mirror/https/mirror select deb.debian.org
#d-i mirror/http/proxy string http://gw:8118/
#d-i mirror/https/proxy string http://gw:8118/

### Partitioning
d-i partman-auto/method string regular
d-i partman-auto/disk string /dev/sda
d-i partman-auto/expert_recipe string system-disk ::          \
              2047 0 -1 ext4                                  \
                      $primary{ }                             \
                      method{ format } format{ }              \
                      use_filesystem{ } filesystem{ ext4 }    \
                      mountpoint{ / }                         \
              .                                               \

# empty line above is REQUIRED due to continuation

# This makes partman automatically partition without confirmation.
d-i partman-partitioning/confirm_write_new_label boolean true
d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i partman/confirm_nooverwrite boolean true

# Skip question about not having swap partition
partman-basicfilesystems partman-basicfilesystems/no_swap boolean false

### Account setup
# Set root password encrypted using a crypt(3) hash
#d-i passwd/root-password-crypted password $6$Ljketui4$hSnALWF8hKnbaWZpkj6xtx7PzWdMdgTJL7cdvfrj5byHq1BVp0KBxQbkKm5O9onjdwyKPmz3W22F.Q

# Set root with no password; the login will be possible with SSH key
d-i passwd/root-password-crypted password *

# Don't disable a root login
d-i passwd/root-login boolean true

# Skip a user creation
d-i passwd/make-user boolean false

# Or make user with known passord
#d-i passwd/user-uid string 1000
#d-i passwd/user-fullname string ${var.ssh_fullname}
#d-i passwd/username string ${var.ssh_username}
#d-i passwd/user-password password ${var.ssh_password}
#d-i passwd/user-password-again password ${var.ssh_password}
#d-i user-setup/allow-password-weak boolean true
#d-i user-setup/encrypt-home boolean false

### Package selection
d-i hw-detect/load_media boolean false
apt-cdrom-setup apt-setup/cdrom/set-first boolean false

#tasksel tasksel/first standard
tasksel tasksel/first multiselect standard, ssh-server
#d-i pkgsel/include string openssh-server qemu-guest-agent
d-i pkgsel/include string openssh-server sudo mc
d-i pkgsel/install-language-support boolean false

# disable automatic package updates
d-i pkgsel/update-policy select none
d-i pkgsel/upgrade select full-upgrade

# disable popularity contest
d-i popularity-contest/participate boolean false

# choose cloud kernel and minimum initramfs
bootstrap-base base-installer/initramfs-tools/driver-policy select  dep
bootstrap-base base-installer/kernel/linux/initramfs-tools/driver-policy string dep
bootstrap-base base-installer/kernel/image select  linux-image-cloud-amd64

### Bootloader
# This is fairly safe to set, it makes grub install automatically to the MBR
# if no other operating system is detected on the machine.
d-i grub-installer/only_debian boolean true

# This one makes grub-installer install to the MBR if it also finds some other
# OS, which is less safe as it might not be able to boot that other OS.
d-i grub-installer/with_other_os boolean true

# Avoid a question!
grub-installer grub-installer/bootdev string /dev/sda

# Avoid that last message about the install being complete, just reboot when finished
d-i finish-install/reboot_in_progress note

### Add SSH key for root
d-i preseed/late_command string \
  mkdir -p -m 700 /target/root/.ssh; \
  echo "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAKQCaZovi1a1UKy6Az3JSFP/3P4tP3ReknM//5eoOvk merlin@uc-s4m75657" > /target/root/.ssh/authorized_keys; \
  in-target chown --recursive root:root /root/.ssh; \
  in-target chmod 0600 /root/.ssh/authorized_keys;

UEFI

Additional documentation

Enterprise Linux: kickstart

Oddly enough, OL8 installs with root enabled SSH with password by default. I guess we should not set any password for root, disable the password authentication (change to PermitRootLogin: without-password) and set a key in a %post section.

Need to consider which else packages may be omitted.

Network installer files

For Oracle Linux, the best is to obtain a “Boot UEK” ISO image, approx. 1G of size (obviously, the more “enterpriseish” is the system, the less efficient it is).

We extract the following files:

File	Target	Description
`images/install.img`	`/srv/tftp/ol8u9/images/install.img`	SquashFS with the installer
`images/pxeboot/initrd.img`	`/srv/tftp/ol8u9/initrd.img`	initramfs
`images/pxeboot/vmlinuz`	`/srv/tftp/ol8u9/vmlinuz`	kernel

Also need to create a bootloader configuration as /srv/tftp/pxelinux.cfg/ol8u9 and a kickstart file as /srv/tftp/ol8u9/ks.cfg (see below for examples). Since installer can't obtain components from TFTP server, we need a HTTP server to serve a kickstart file and a large squashfs image; technically, they may be placed elsewhere, but for simplicity I keep a single tree and make it available both via TFTP and HTTP.

Bootloader configuration

GRUB

/srv/tftp/grub/ol8u9

set timeout=1

menuentry 'Oracle Linux 8 PXE installer' {
    linux ol8u9/vmlinuz inst.repo=http://vmgw/ol8u9 inst.ks.sendmac inst.ks=http://vmgw/ol8u9/ks.cfg
    initrd ol8u9/initrd.img
}

PXELinux

/srv/tftp/pxelinux.cfg/ol8u9

default ol8u9

label ol8u9
kernel ol8u9/vmlinuz
append initrd=ol8u9/initrd.img inst.repo=http://vmgw/ol8u9 inst.ks.sendmac inst.ks=http://vmgw/ol8u9/ks.cfg
# it will append .../images/install.img or .../LiveOS/squashfs.img

Example kickstart file

BIOS

/srv/tftp/ol8u9/ks.cfg

#platform=x86, AMD64, or Intel EM64T
#version=OL8

# Firewall configuration
firewall --enabled --service=ssh

# Use Oracle Linux yum server repositories as installation source
repo --name="ol8_AppStream" --baseurl="https://yum.oracle.com/repo/OracleLinux/OL8/appstream/x86_64/"
repo --name="ol8_UEKR7" --baseurl="https://yum.oracle.com/repo/OracleLinux/OL8/UEKR7/x86_64/"
url --url="https://yum.oracle.com/repo/OracleLinux/OL8/baseos/latest/x86_64"

# Root password
#rootpw --iscrypted SHA512_password_hash
rootpw ...

# Use text only install
skipx
text
firstboot --disable
reboot --eject

# Keyboard layouts
keyboard --vckeymap=us --xlayouts='us'

# System language
lang en_US.UTF-8
# Note: problems with console font if using anything else

## SELinux configuration
#selinux --enforcing

## Installation logging level
#logging --level=info

# System timezone
timezone Asia/Yerevan

# Network information
network  --bootproto=dhcp --onboot=yes --hostname=test-ol8

# System bootloader configuration
bootloader --location=mbr --boot-drive=sda

## Non-administrative user
#user --name=user --homedir=/home/user --password=SHA512_password_hash --iscrypted

# Partition information
clearpart --all --initlabel --drives=sda
# NO SANE SERVER SYSTEM COULD EVER NEED MORE THAN 2G OF SWAP
part  swap --size 2047 --ondisk sda
part / --fstype ext4 --size 1 --grow --ondisk sda

# Disable kdump by default, frees up some memory
%addon com_redhat_kdump --disable
%end

services --disabled="kdump"

# Minimal install with a few convenience packages
%packages
@core
qemu-guest-agent
tmux
mc

-iwl7260-firmware
-iwl6000-firmware
-iwl2030-firmware
-iwl1000-firmware
-iwlax2xx-firmware
-iwl6000g2a-firmware
-iwl5150-firmware
-iwl3160-firmware
-iwl2000-firmware
-iwl105-firmware
-iwl100-firmware
-iwl6050-firmware
-iwl5000-firmware
-iwl135-firmware

-plymouth

-rhnsd
-rhn-setup
-python3-dnf-plugin-spacewalk
-dnf-plugin-spacewalk
-python3-rhn-client-tools
-rhn-client-tools
-rhnlib

-sssd-client
-sssd-common
-sssd-kcm

-btrfs-progs
-bcache-tools
%end

%post
mkdir /root/.ssh
echo "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXg9044dPzASa21RzPZAfN06/vZGAQOHGTmQcPc0fxEd6XFEJKcNwYJy0WRAY6tB1b3UcJeE7+lUjLJORrv4rs0L75qKoucJ+s1b9IegABs4qCW/A6sbanPHI3w7Lw6DYLXVCA1nXqpCnlhjlS3O6KBfbLo1jRAK10v9F0HTGWLm1lN05PT6ItL9WnQ88nrsZ/ON2bC5JyDL/CUxeV9qXIWIelFYkNoGjUM+baoMOb2N7ytuZA17qTkjZTRTQrCgJq19nu2el8/OYdUoIYDDm1ZJVQ/ahebtuobEFyTVMO2SGoL5YgKWHo8P/yCKuN3iWDcu6atwj+JjBjhrqSOxIN merlin@uc-s4m75657" > /root/.ssh/authorized_keys
chmod 0700 /root/.ssh
chmod 0640 /root/.ssh/authorized_keys
%end

Table of Contents

Linux Automatic Network Install

Boot server configuration

PXE bootloader

GRUB

PXELinux

DNS, DHCP and TFTP servers

dnsmasq

ISC DHCPD, ISC BIND, tftpd-hpa

HTTP server

Debian: preseed

Network installer files

Bootloader configuration

GRUB

PXELinux

Example preseed file

BIOS

UEFI

Additional documentation

Enterprise Linux: kickstart

Network installer files

Bootloader configuration

GRUB

PXELinux

Example kickstart file

BIOS

UEFI

Additional documentation