Raspberry Pi 4 -Diskless boot with Rasbian Buster (Debian 10) - From PXE Server (Debian 10)

This is the most tricky work I have done to boot Raspberry Pi via Network. Main reasons behind this goal were

• SdCard can't afford r/w operations and get corrupted often. Making again bootable sd card was time taking and money wasting. I found it useless in my operational case. 
• I was building a HPC cluster for my small work. so making Raspberry Pi diskless fulfill all the requirement for me. 
• Making raspberry pi diskless was the only option that makes me headache free for most of my work. 

Requirements

• Raspberry Pi (I used Raspberry Pi 3 Model B)
• Linux Server works as PXE-Server (I have debian 10 ready server)
• SD Card(I promise just for few initial task you need it after that you can throw it out)
• Home or LAN Network(Ethernet Connection is required for Raspberry Pi)
• Diskless boot with wifi is not going to work so you need an Ethernet Cable for RP.

Additional Information

PXE-Server is having 192.168.10.222/24 Static IP and connected to a switch where we will also connect our raspberry Pi with Ethernet cable. Make sure no other DHCP server is running on the same network. Otherwise it will create headache for you.

PXE-Server configuration.

I have Debian 10 buster installed with static IP 192.168.10.222.

Install following packages.

#apt install isc-dhcp-server tftpd-hpa nfs-kernel-server -y


TFTP Server Configuration

When installed it will create /srv/tftp directory. In raspberry pi case it will be boot drive which will present all required bootable files pointed thourgh DHCP.

Make sure /srv/tftp directory is created otherwise create it.

My setting for tftpd-hpa (/etc/default/tftpd-hpa)

# /etc/default/tftpd-hpa

TFTP_USERNAME="tftp"
TFTP_DIRECTORY="/srv/tftp"
TFTP_ADDRESS="0.0.0.0:69"
TFTP_OPTIONS="--secure -c -vv"

#systemctl enable tftpd-hpa
#systemctl start tftpd-hpa

NFS Server Configurations.

When installed it will create /etc/export file for directory configurations.

#mkdir /srv/nfs1

My /etc/exports

/srv/tftp 192.168.10.0(rw,sync,no_subtree_check,no_root_squash)
/srv/nfs1 192.168.10.0/24(rw,async,no_subtree_check,no_root_squash)

#systemctl enable nfs-kernel-server
#systemctl start nfs-kernel-server

DHCP Server configurations

This is the most tricky part. Make sure my below /etc/dhcp/dhcpd.conf configurations matches.


default-lease-time 86400;
max-lease-time 86400;

authoritative;

log-facility local7;

subnet 192.168.10.0 netmask 255.255.255.0 {
  range 192.168.10.41 192.168.10.45;

  #option domain-name "home.ridgway.io";
  option domain-name-servers 192.168.10.1, 192.168.10.250;
  option routers 192.168.10.250;
  next-server 192.168.10.222;
  option tftp-server-name "192.168.10.222";
}

host hpcnode1 {
  hardware ethernet b8:27:xx:xx:xx:31;
  fixed-address 192.168.10.41;
  option host-name "hpcnode1";
#  next-server 192.168.10.222;
#  option tftp-server-name "192.168.10.222";
}

host hpcnode2 {
  hardware ethernet b8:27:xx:xx:xx:32;
  fixed-address 192.168.10.42;
  option host-name "hpcnode2";
#  next-server 192.168.10.222;
#  option tftp-server-name "192.168.10.222";
}


What these settings does it will assign IP to Raspberry pi at the boot time. I have assigned fixed IP to raspberry pi with its MAC address. It will also assign unique hostname to Raspberry Pi when it will be booted up. Make sure your /etc/hostname file in Rasbian is empty. Assign no hostname to raspbian Linux when you will boot it from sd card. 

Make sure you have added next-server and option tftp-server-name to dhcp settings according to your network.

OK Now be ready for Raspberry Pi Section. Really tricky things here you have to work with.

 Make your Raspberry Pi Network Bootable

This section only applies to the original Raspberry Pi 3B; if you are using the 3B+, then ignore this section and skip to the server section below then.

Before a Raspberry Pi will network boot, it needs to be booted from an SD card with a config option to enable USB boot mode. This will set a bit in the OTP (One Time Programmable) memory in the Raspberry Pi SoC that enables network booting. Once this is done, the SD card is no longer required.

Install Raspbian Lite (or Raspbian with Raspberry Pi Desktop) on the SD card in the usual way.

Afterwards, set up USB boot mode by preparing the /boot directory with the latest boot files:

#sudo apt update && sudo apt full-upgrade

Now, enable USB boot mode with the following command:

#echo program_usb_boot_mode=1 | sudo tee -a /boot/config.txt

This adds program_usb_boot_mode=1 to the end of /boot/config.txt. Reboot the Raspberry Pi with sudo reboot. Once the client Raspberry Pi has rebooted, check that the OTP has been programmed with:

$ vcgencmd otp_dump | grep 17:
17:3020000a

Ensure the output 0x3020000a is correct.

The client configuration is almost done.

The final thing to do is to remove the program_usb_boot_mode line from config.txt (make sure there is no blank line at the end). You can do this with sudo nano /boot/config.txt, for example. Finally, shut the client Raspberry Pi down with sudo poweroff.

Disable Swap
#sudo dphys-swapfile swapoff
#sudo dphys-swapfile uninstall
#sudo update-rc.d dphys-swapfile remove

Enable SSH
Run sudo raspi-config, selection Option 5 (Interfacing Options), P2 (SSH) and select Yes to Enable SSH.

Update the Pi (firmware)

Certain files from the next branch of the rpi-firmware repository are required for this process to work.
This step is probably optional if bootcode.bin is used from GitHub in the TFTP steps below.
Edit: I've also observed using the 'stock' raspbian kernel img files can cause nasty boot errors.

Run
#sudo BRANCH=next rpi-update.
or
Simply  Run

#sudo rpi-update

It is likely to complete by saying:

 *** A reboot is needed to activate the new firmware

So, reboot now with sudo reboot.


Clone Raspbian for diskless boot

I'm going to skip the process of downloading Raspbian and making sd card bootable. You can follow RP offical website for this process.

Right now I have bootable sd card and I booted Raspberry Pi with it.

there are two way you can clone the whole filesystem including files permission. You must preserve permissions otherwise this is not going to work.

First method is to make compressed tar and scp it to /srv/nfs1 of the server.

create a tar archive of the nfs folder (you could gzip it but as the Pi isn't all that powerful its probably not worth it vs transfer time, but this is just an assumption).

The -p flag should preserve permissions etc.



#sudo mkdir /opt/nfs

#sudo tar -cpf  /    /opt/nfs/nfs.tar

I ended up with a 1.3Gb /opt/nfs/nfs.tar:


now SCP nfs.tar file to server /srv/nfs1

#scp /opt/nfs/nfs.tar root@192.168.10.222:/srv/nfs1/

when transfer is done.

on server run these commands

# cd /srv/nfs1
# tar -xvf nfs.tar 

when this is done, delete nfs1.tar we do not need it anymore.

# rm nfs.tar



Second Method 

You can also directory copy to server with rsync like below

# rsync -vax /* root@192.168.10.222:/srv/nfs1

In my case second method was messed up with preserve permission because of pi user on Rasbian as it was not present on Debian PXE-Server.


Now on PXE-Server

#cp -vax /serv/nfs1/boot/* /srv/tftp/
#chmod 777 /srv/tftp/*

Now edit /srv/tftp/cmdline.txt
root= onward remove everything and add root=/dev/nfs nfsroot=192.168.10.222:/srv/nfs1,vers=3 rw ip=dhcp rootwait elevator=deadline to it. I have mentioned nfs shares as nfsroot= with vers=3


My /srv/tftp/cmdline.txt

dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/nfs nfsroot=192.168.10.222:/srv/nfs1,vers=3 rw ip=dhcp rootwait elevator=deadline

Now edit /srv/nfs1/etc/fstab 

My fstab file is as below 

proc            /proc           proc    defaults          0       0

192.168.10.222:/srv/boot /boot nfs defaults,vers=3 0 0 

NOW CONNECT YOUR RASPBERRY PI WITH ETHERNET CABLE AND POWER IT UP. 
NOW WAIT FOR SOME TIME YOU WILL GET ETHERNET LIGHTS BLINKING. ON SCREEN YOU WILL SEE IT BOOTING UP. CONGRATULATIONS.... YOU HAVE DISKLESS RAPBERRY PI. 

To see logs on pxe-server check syslogs
#tail -f /var/log/syslog. 
Before powering up Raspberry pi tail the syslog file so you will get real time logs. If you get any error from DHCP or TFTP Server you can see it here and work on it in case you made any mistake.


Problem Faced.......

Permissions and Sudo
 
If not using rsync as above or tar with the specific permissions flags described earlier then after creating the NFS file system the permissions may be inherited from the extraction path or similar if the specific tar instructions above were not followed:



To fix this give root ownership to all files except home, note the use of the UID and GID as they will not match the server entity names:

# sudo chown root:root . -Rf
# chown 1000:1000 home/pi -Rf
If trying to sudo on a network booted Pi and getting the following error:

sudo: /usr/bin/sudo must be owned by uid 0 and have the setuid bit set
Then ensure the setuid bit is set:
 
# chmod u+s usr/bin/sudo
 
Permissions can be verified against the local Ubuntu install:


This is still not perfect so it is strongly advised to use rsync or tar as described previously.

Backups
 
The NFS root can easily be backed-up:

# cp /srv/nfs1 /backup -Rf
It would be wise to back it up before making any material configuration or package changes, just in-case.

Boot Failures
 
If the Pi is not requesting anything from TFTP it could be a DHCP configuration problem, or a TFTP access problem.

If you see the Pi get little or no further than requesting bootcode.bin then it is possible the wrong version of the files are in the tftp folder, ensure the next branch versions are used:

Mar  2 23:11:31 nas tftpd[5385]: tftpd: trying to get file: bootcode.bin
 
If it gets further, but then hangs as shown further above it could be a NFS/filesystem access problem.

Beware TFTP File Caching
 
Not proven, however when debugging the misreported RAM issue I replaced my custom boot folder (with new symlinks) but the touchscreen Pi still appeared to pick up the old files as it was reporting lower RAM.

After an tftpd-hpa restart all was fine...

Performance
 
Installing and configuring packages etc is naturally going to be slower as the Pi only has 100Mbit Ethernet and the overheads of using a network-based file system.

For speed all config could be done on SD then copied over using the steps above.

You can ask freely you faced anyother problem. You're welcomed here.