Purpose: to install Linux automatically unattended, without any manual user input during the process.

This can be used in further automation using tool like Ansible. It should create a VM and obtain its MAC address from the virtualization environment, or reach the BMC and obtain a MAC address of the physical server. Then, knowing this info, it can generate all the boot configuration and installation scripts and wait for the OS to be installed. At last, it can customize it further in the next play. This facilitates a complete server provision with a single tool using a single configuration file in a single run.

General outline of the technology:

• PXE boot server with support for “legacy” BIOS and EFI boot; most VMs use “legacy” and most recent equipment uses EFI
• HTTP server for EL (it doesn't support downloading a kickstart file from TFTP server, what a shame)
• A set of boot images and minimal autoinstall files for Debian (at least, Bookworm) and EL (at least, Oracle Linux 8)

Possible optional additions:

• A proxy server to perform installation and updates in restricted environments
• Set of repositories to store packages needed by installers, beneficial for mass installs. The set of “popular” packages could be even retrieved from the proxy server logs

We can use ISC DHCPD + BIND + tftpd-hpa, or dnsmasq.

All (more or less) dynamic data is t be stored in /srv/tftp, which will be served via TFTP and HTTP. The latter is needed for EL:

srv/
┠─ dhcp-hostsfile.cfg
┗━ tftp/
   ┠─ bootx64.efi
   ┠─ grubx64.efi
   ┠─ pxelinux.0
   ┠─ ldlinux.c32
   ┣━ pxelinux.cfg/
   ┃  ┠─ harddisk
   ┃  ┠─ debian-bookworm
   ┃  ┠─ ol8u9
   ┃  ┠─ menu
   ┃  ┠─ default → menu
   ┃  ┠─ docker-vm → harddisk
   ┃  ┠─ test-pve → debian-bookworm
   ┃  ┠─ test-lvm → ol8u9
   ┃  ┠─ 01-bc-24-11-51-cf-fc → docker-vm
   ┃  ┠─ 01-bc-24-11-4b-ae-dd → test-pve
   ┃  ┖─ 01-bc-24-11-28-81-76 → test-lvm
   ┣━ grub
   ┃  ┖─ grub.cfg
   ┣━ debian-bookworm/
   ┃  ┠─ initrd.gz
   ┃  ┠─ linux
   ┃  ┖─ preseed-noswap.cfg
   ┣━ debian-bookworm/
   ┃  ┣━ images/
   ┃  ┃  ┖─ install.img
   ┃  ┠─ initrd.img
   ┃  ┠─ ks.cfg
   ┃  ┖─ vmlinuz
   ┗━ gentoo/
      ┠─ gentoo
      ┖─ initramfs.igz

The most logic is within /srv/tftp/pxelinux.cfg directory. There are several autoboot configurations:

• harddisk — boot from local hard disk
• debian-bookworm — start an unattended installation of a Debian 12 release
• ol8u9 — start an unattended installation of a Oracle Linux 8 release
• menu — present a human-friendly menu

The rest items are symlinks to these files:

• default — will be used by PXELinux if no other configuration file was matched; here it symlinks to the menu, but can be diverted anywhere else
• test-lvm, test-pve and docker-vm are VMs or HW machines which are provisioned using this system, and in particular, test-lvm is being provisioned with OL8, test-pve is being provisioned with Debian and the docker-vm is already installed, so it is instructed to boot locally
• 01-xx-xx-xx-xx-xx-xx are symlinks named after the respective system's MAC address. If there are several NICs, we create several files, for each MAC. PXELinux will first try such a file and load it if it exists, and if it doesn't, it will load default. This way, we precisely target each system to divert its boot path independently of each other.

Boot menu can contain more entries if needed, for example, it can boot Gentoo LiveCD converted for PXE boot. It defaults to booting from hard disk after timeout. GRUB is using a single file currently and it's unknown how to divert its boot path based on MAC address or other system identifier, so EFI boot is only possible into a menu-style currently.

The default Debian's dnsmasq configuration can use the base configuration and the drop directory. We opt to use the latter.

The most configuration is /etc/dnsmasq.d/base.conf:

/etc/dnsmasq.d/base.conf

no-dhcp-interface = tun0

server = 10.226.130.130
server = 10.226.130.131
no-resolv

domain = test.ucom.am, 172.31.1.0/24,   local
domain = test.ucom.am, 172.31.255.0/24, local
domain-needed
expand-hosts

# no dynamic allocation for DHCP service on this subnet; static allocation via dhcp-host will still be possible
dhcp-range = 192.168.205.224, static
dhcp-range = 172.31.255.0,    static

# allocate dynamically from this subnet
dhcp-range = 172.31.1.34, 172.31.1.62, 1h

dhcp-authoritative
dhcp-option = option:ntp-server, 10.226.130.130, 10.226.130.131

dhcp-hostsfile = /srv/dhcp-hostsfile.cfg

enable-tftp
tftp-root = /srv/tftp

dhcp-match = set:bios, option:client-arch,  0
dhcp-match = set:uefi, option:client-arch,  7
dhcp-match = set:uefi, option:client-arch,  9
dhcp-match = set:http, option:client-arch, 16

dhcp-boot = tag:bios, pxelinux.0
dhcp-boot = tag:uefi, bootx64.efi

#pxe-prompt = "Select a system to boot"
#pxe-prompt = "Press F8 for menu.", 60
#pxe-service = x86PC, "Boot from local disk"
#pxe-service = x86PC, "Run pxelinux", pxelinux

log-queries
log-dhcp

It references the /srv/dhcp-hostsfile.cfg where we can match certain MACs to IP addresses and names. This is mostly convenient for networks defined as “static DHCP ranges” above, since this way only hosts listed in the hostsfile will be provided a service. Also we can put other hosts there, for example, VMs, to fix IP addresses. Example contents is:

/srv/dhcp-hostsfile.cfg

BC:24:11:51:CF:FC, 172.31.1.5,  docker-vm
BC:24:11:70:C7:44, 172.31.1.6,  docker-ct
BC:24:11:4B:AE:DD, 172.31.1.4,  test-pve
BC:24:11:D4:97:9C, 172.31.1.27, hpprobook-image
BC:24:11:C9:B6:69, 172.31.1.7,  gns3
BC:24:11:28:81:76, 172.31.1.28, test-lvm

Also the static hosts may be defined in a sidecar file living in /etc/dnsmasq.d/:

/etc/dnsmasq.d/dhcp-static-hosts.conf

dhcp-host = 38:ea:a7:33:b5:98, 38:ea:a7:33:b5:99, 172.31.255.5,  vh-e1-b5
dhcp-host = 38:ea:a7:32:40:7c, 38:ea:a7:32:40:7d, 172.31.255.6,  vh-e1-b6
dhcp-host = 38:ea:a7:91:77:a8, 38:ea:a7:91:77:a9, 172.31.255.13, vh-e1-b13
dhcp-host = 38:ea:a7:91:76:44, 38:ea:a7:91:76:45, 172.31.255.14, vh-e1-b14

I put auxiliary DNS records into a separate file too:

-

cname = boot.test.ucom.am, vmgw.test.ucom.am

Needed for EL variants to be able to run installer from the network. We only set data_dir=/srv/tftp in /etc/mini-httpd.conf; this directory is already served by TFTP; we can use a separate directory, but this setup feels simpler.

Debian conveniently maintains all the required bits for the netboot as an archive distributed through its mirrors, available, for example, here. From it, we need only two files (which can be also downloaded separately):

Also need to create a bootloader configuration as /srv/tftp/pxelinux.cfg/debian-bookworm and a preseed file as /srv/tftp/debian-bookworm/preseed-noswap.cfg.

/srv/tftp/pxelinux.cfg/debian-bookworm

default debian-bookworm-auto
timeout 0

label debian-bookworm-auto
    kernel debian-bookworm/linux
    append priority=critical initrd=debian-bookworm/initrd.gz debian-installer/language=en debian-installer/country=AM debian-installer/locale=ru_RU.UTF-8 keyboard-configuration/xkb-keymap=ru netcfg/get_hostname=install keyboard-configuration/optionscode=grp:caps_lock_toggle,grp_led:scroll preseed/url=tftp://boot.test.ucom.am/debian-bookworm/preseed-noswap.cfg

Works for Bullseye too (it was actually originally developed for Bullseye, but worked with Bookworm like a charm)!

/srv/tftp/debian-bookworm/preseed-noswap.cfg

#_preseed_V1

# Preseeding only locale sets language, country and locale.

# For network booting, this comes too late; one needs to put those into command line,
# and the language there should be "en" or it will ask about it on the terminal
d-i debian-installer/language string ru
d-i debian-installer/country string AM
d-i debian-installer/locale string ru_RU.UTF-8
d-i localechooser/supported-locales multiselect ru_RU.UTF-8

# Keyboard selection.
d-i console-setup/ask_detect boolean false
d-i keyboard-configuration/xkb-keymap select ru
d-i keyboard-configuration/optionscode string grp:caps_toggle,lv3:ralt_switch,compose:rctrl,grp_led:scroll
d-i keyboard-configuration/layoutcode string us,ru
d-i keyboard-configuration/variantcode string ,
d-i keyboard-configuration/toggle select Caps Lock
d-i keyboard-configuration/altgr select Right Alt (AltGr)
d-i keyboard-configuration/compose select Right Control

### Clock and time zone setup
d-i clock-setup/utc boolean true
d-i time/zone string Asia/Yerevan

### Mirror settings
# If you select ftp, the mirror/country string does not need to be set.
d-i mirror/country string AM
d-i mirror/http/directory string /debian/
d-i mirror/http/hostname string deb.debian.org
d-i mirror/https/hostname string deb.debian.org
d-i mirror/http/mirror select deb.debian.org
d-i mirror/https/mirror select deb.debian.org
#d-i mirror/http/proxy string http://gw:8118/
#d-i mirror/https/proxy string http://gw:8118/

### Partitioning
d-i partman-auto/method string regular
d-i partman-auto/disk string /dev/sda
d-i partman-auto/expert_recipe string system-disk ::          \
              2047 0 -1 ext4                                  \
                      $primary{ }                             \
                      method{ format } format{ }              \
                      use_filesystem{ } filesystem{ ext4 }    \
                      mountpoint{ / }                         \
              .                                               \

# empty line above is REQUIRED due to continuation

# This makes partman automatically partition without confirmation.
d-i partman-partitioning/confirm_write_new_label boolean true
d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i partman/confirm_nooverwrite boolean true

# Skip question about not having swap partition
partman-basicfilesystems partman-basicfilesystems/no_swap boolean false

### Account setup
# Set root password encrypted using a crypt(3) hash
#d-i passwd/root-password-crypted password $6$Ljketui4$hSnALWF8hKnbaWZpkj6xtx7PzWdMdgTJL7cdvfrj5byHq1BVp0KBxQbkKm5O9onjdwyKPmz3W22F.Q

# Set root with no password; the login will be possible with SSH key
d-i passwd/root-password-crypted password *

# Don't disable a root login
d-i passwd/root-login boolean true

# Skip a user creation
d-i passwd/make-user boolean false

# Or make user with known passord
#d-i passwd/user-uid string 1000
#d-i passwd/user-fullname string ${var.ssh_fullname}
#d-i passwd/username string ${var.ssh_username}
#d-i passwd/user-password password ${var.ssh_password}
#d-i passwd/user-password-again password ${var.ssh_password}
#d-i user-setup/allow-password-weak boolean true
#d-i user-setup/encrypt-home boolean false

### Package selection
d-i hw-detect/load_media boolean false
apt-cdrom-setup apt-setup/cdrom/set-first boolean false

#tasksel tasksel/first standard
tasksel tasksel/first multiselect standard, ssh-server
#d-i pkgsel/include string openssh-server qemu-guest-agent
d-i pkgsel/include string openssh-server sudo mc
d-i pkgsel/install-language-support boolean false

# disable automatic package updates
d-i pkgsel/update-policy select none
d-i pkgsel/upgrade select full-upgrade

# disable popularity contest
d-i popularity-contest/participate boolean false

# choose cloud kernel and minimum initramfs
bootstrap-base base-installer/initramfs-tools/driver-policy select  dep
bootstrap-base base-installer/kernel/linux/initramfs-tools/driver-policy string dep
bootstrap-base base-installer/kernel/image select  linux-image-cloud-amd64

### Bootloader
# This is fairly safe to set, it makes grub install automatically to the MBR
# if no other operating system is detected on the machine.
d-i grub-installer/only_debian boolean true

# This one makes grub-installer install to the MBR if it also finds some other
# OS, which is less safe as it might not be able to boot that other OS.
d-i grub-installer/with_other_os boolean true

# Avoid a question!
grub-installer grub-installer/bootdev string /dev/sda

# Avoid that last message about the install being complete, just reboot when finished
d-i finish-install/reboot_in_progress note

### Add SSH key for root
d-i preseed/late_command string \
  mkdir -p -m 700 /target/root/.ssh; \
  echo "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAKQCaZovi1a1UKy6Az3JSFP/3P4tP3ReknM//5eoOvk merlin@uc-s4m75657" > /target/root/.ssh/authorized_keys; \
  in-target chown --recursive root:root /root/.ssh; \
  in-target chmod 0600 /root/.ssh/authorized_keys;

• https://wiki.debian.org/DebianInstaller/Preseed#Preseed_syntax
• https://www.debian.org/releases/stable/amd64/apbs03.en.html
• https://www.debian.org/releases/bullseye/example-preseed.txt
• https://hands.com/d-i/

Oddly enough, OL8 installs with root enabled SSH with password by default. I guess we should not set any password for root, disable the password authentication (change to PermitRootLogin: without-password) and set a key in a %post section.

Need to consider which else packages may be omitted.

For Oracle Linux, the best is to obtain a “Boot UEK” ISO image, approx. 1G of size (obviously, the more “enterpriseish” is the system, the less efficient it is).

We extract the following files:

Also need to create a bootloader configuration as /srv/tftp/pxelinux.cfg/ol8u9 and a kickstart file as /srv/tftp/ol8u9/ks.cfg (see below for examples). Since installer can't obtain components from TFTP server, we need a HTTP server to serve a kickstart file and a large squashfs image; technically, they may be placed elsewhere, but for simplicity I keep a single tree and make it available both via TFTP and HTTP.

/srv/tftp/pxelinux.cfg/ol8u9

default ol8u9

label ol8u9
kernel ol8u9/vmlinuz
append initrd=ol8u9/initrd.img inst.repo=http://vmgw/ol8u9 inst.ks.sendmac inst.ks=http://vmgw/ol8u9/ks.cfg
# it will append .../images/install.img or .../LiveOS/squashfs.img

/srv/tftp/ol8u9/ks.cfg

#platform=x86, AMD64, or Intel EM64T
#version=OL8

# Firewall configuration
firewall --enabled --service=ssh

# Use Oracle Linux yum server repositories as installation source
repo --name="ol8_AppStream" --baseurl="https://yum.oracle.com/repo/OracleLinux/OL8/appstream/x86_64/"
repo --name="ol8_UEKR7" --baseurl="https://yum.oracle.com/repo/OracleLinux/OL8/UEKR7/x86_64/"
url --url="https://yum.oracle.com/repo/OracleLinux/OL8/baseos/latest/x86_64"

# Root password
#rootpw --iscrypted SHA512_password_hash
rootpw ...

# Use text only install
skipx
text
firstboot --disable
reboot --eject

# Keyboard layouts
keyboard --vckeymap=us --xlayouts='us'

# System language
lang en_US.UTF-8
# Note: problems with console font if using anything else

## SELinux configuration
#selinux --enforcing

## Installation logging level
#logging --level=info

# System timezone
timezone Asia/Yerevan

# Network information
network  --bootproto=dhcp --onboot=yes --hostname=test-ol8

# System bootloader configuration
bootloader --location=mbr --boot-drive=sda

## Non-administrative user
#user --name=user --homedir=/home/user --password=SHA512_password_hash --iscrypted

# Partition information
clearpart --all --initlabel --drives=sda
# NO SANE SERVER SYSTEM COULD EVER NEED MORE THAN 2G OF SWAP
part  swap --size 2047 --ondisk sda
part / --fstype ext4 --size 1 --grow --ondisk sda

# Disable kdump by default, frees up some memory
%addon com_redhat_kdump --disable
%end

services --disabled="kdump"

# Minimal install with a few convenience packages
%packages
@core
qemu-guest-agent
tmux
mc

-iwl7260-firmware
-iwl6000-firmware
-iwl2030-firmware
-iwl1000-firmware
-iwlax2xx-firmware
-iwl6000g2a-firmware
-iwl5150-firmware
-iwl3160-firmware
-iwl2000-firmware
-iwl105-firmware
-iwl100-firmware
-iwl6050-firmware
-iwl5000-firmware
-iwl135-firmware

-plymouth

-rhnsd
-rhn-setup
-python3-dnf-plugin-spacewalk
-dnf-plugin-spacewalk
-python3-rhn-client-tools
-rhn-client-tools
-rhnlib

-sssd-client
-sssd-common
-sssd-kcm

-btrfs-progs
-bcache-tools
%end

%post
mkdir /root/.ssh
echo "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXg9044dPzASa21RzPZAfN06/vZGAQOHGTmQcPc0fxEd6XFEJKcNwYJy0WRAY6tB1b3UcJeE7+lUjLJORrv4rs0L75qKoucJ+s1b9IegABs4qCW/A6sbanPHI3w7Lw6DYLXVCA1nXqpCnlhjlS3O6KBfbLo1jRAK10v9F0HTGWLm1lN05PT6ItL9WnQ88nrsZ/ON2bC5JyDL/CUxeV9qXIWIelFYkNoGjUM+baoMOb2N7ytuZA17qTkjZTRTQrCgJq19nu2el8/OYdUoIYDDm1ZJVQ/ahebtuobEFyTVMO2SGoL5YgKWHo8P/yCKuN3iWDcu6atwj+JjBjhrqSOxIN merlin@uc-s4m75657" > /root/.ssh/authorized_keys
chmod 0700 /root/.ssh
chmod 0640 /root/.ssh/authorized_keys
%end

• https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html-single/boot_options_for_rhel_installer/index
• https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/performing_an_advanced_rhel_8_installation/kickstart-commands-and-options-reference_installing-rhel-as-an-experienced-user
• https://access.redhat.com/documentation/ru-ru/red_hat_enterprise_linux/8/html/performing_an_advanced_rhel_8_installation/making-kickstart-files-available-to-the-installation-program_installing-rhel-as-an-experienced-user
• https://docs.oracle.com/en/operating-systems/oracle-linux/8/install/install-AutomatinganOracleLinuxInstallationbyUsingKickstart.html#kickstart-install-options
• https://docs.oracle.com/en/operating-systems/oracle-linux/8/install/install-CreatingaNetworkInstallationSetup.html#bios-clients
• https://docs.oracle.com/cd/E20815_01/html/E20840/sflog.gisti.html#scrolltoc
• https://unix.stackexchange.com/questions/678345/pxe-boot-kickstart-how-to-include-installation-tree
• https://www.dbi-services.com/blog/automating-oracle-linux-installation-with-kickstart/
• https://github.com/Linuxfabrik/kickstart — very nice example of working minimal kickstart
• https://docs.centos.org/en-US/8-docs/advanced-install/assembly_kickstart-script-file-format-reference/
• https://forums.centos.org/viewtopic.php?t=47262
• https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/security_hardening/assembly_securing-rhel-during-installation-security-hardening#Minimal_install_securing-rhel-during-installation
• https://serverfault.com/questions/29071/red-hat-server-minimal-install