Category Archives: freebsd

FreeBSD BHYVE why use it?A Bhyve suspend/resume howto + disaster recovery with UFS and ZFS

INTRO:

FreeBSD Bhyve virtualization why? Sure there are different virtualization packages out there, but you have FreeBSD! Why not use ESXI? From experience any closed source software generally has a max lifetime of 10 years(abandonware), and opensource will generally over throw them by then. Also anything you don’t have full control over on root prompt to read the source code will be a nightmare to administer.

What about Linux and KVM? Yes, before Bhyve they were the best, the virt-manager and virsh/virt-viewer commands were unbeatable. Here is the thing with Redhat however, they over the course of just 2 years now have made most of sys admins angry on the internet. First they removed the megaraid drivers from the kernel(yes people still run Dell r710’s etc as fileservers), which caused a lot of people grief, then on upgrade to Redhat 9, they killed off a lot of Dell servers as well, people who upgraded had to reinstall with Rocky Linux 8 as they used a glibc that did not work on a lot of people’s systems. And to top it off they killed off Centos by moving it to Centos-stream, angering rest of internet. So everyone moved to Rocky Linux(original creator of Centos), to keep their hosts “production”.

The worst thing about Linux to begin with as a virtualization host, is every 2 years when a new major release comes out, most people have to take time out of their day to reinstall the OS. A core OS! With FreeBSD you can update to new major releases by way of simple freebsd-update command or rebuilding source code from scratch with “make buildworld” as well. No downtime reinstalling, no host to replace, a sys admin dream.

I’ve had fights with programmers over the years over Linux and FreeBSD, especially when it came to libevent and libev, where people swore by epoll over kqueue, from experience put FreeBSD under load and then a Linux server, your loads will be better served on FreeBSD with less CPU utilization, the network stack is better, more efficient, even Netflix uses it.

And now I know what everyone is going to say, been using Linux/KVM virt-manager for so many years, FreeBSD doesn’t even have suspend/resume stable. I’m not loosing my 20 SSH connections to my guests just because I had to reboot for a kernel update. I can’t disagree with this, I’d hate to be working 4 hours on a script on one of them, reboot for a kernel update, and lost all my work because I forgot to save on a host and host didn’t resume guests properly after reboot.

So why use Linux at all? I think Linux is fine as a guest, just not in the frontline of battle for reasons I mentioned above. There are still lots of things that will only run on Linux, if your a flutter programmer, FreeBSD still has not ported Dart language, so your apps will still need a Linux guest to test builds. If I’m considering working on an app, I would most likely install an ubuntu guest for that, do all your dart/c/java there along with windows 11, and your python Fastapi’s interfacing with Mysql on Linux and/or FreeBSD.

Honestly for your server part of your app, I would use a FreeBSD guest to begin with hosting Fastapi, I will show you how you could pass a second disk through to the guest with a proper 16k blocksize for max Mysql performance for your app. The biggest reason to use FreeBSD upfront, you can be stuck 6 months to a year working on an app, you really need a week of downtime because some new Linux release came out and you have to reinstall? I didn’t think so…let FreeBSD “Shine bright like a diamond” as Rihanna would sing.

So today I am going to take what is the most important and exciting #1 feature release for FreeBSD 14, suspend and resume. I will be installing FreeBSD current to test it. I will also show how FreeBSD is a force to be reckoned with as the core OS with virtualization.

Test case scenario:

Dell r720, Intel Enterprise 480GB SSD as core OS slapped into an icy dock for core OS, along with 2 12TB rust spinning drives for separate ZFS backup pool all on a megaraid JBOB controller. FreeBSD current, with production flags, with experimental suspend and resume features enabled in kernel and userland.

For main OS we will use default ZFS install, for guest tests we will install 2 FreeBSD guests, one with ZFS, and one with UFS. We will suspend them both, then resume them and see if we loose our SSH connections 🙂 I will most likely do a part 2 on installing Linux and Windows guests as I see this article will get pretty lengthy just with these 2 guests…

For third-party packages to interface with suspend and resume, we can’t use any. We will have to do everything manually, I will keep it simple with just bash scripts. Maybe I’ll code an advanced interface with python and asyncio in future. One third party package I did run across I kind of liked was vm-bhyve. To be honest only thing I liked about it was his directory layout, so that’s only thing I will try to keep consistent for our scenario.

Bhyve still needs a good interface to it, I think what would be best for it is python fastapi or C/Rust with kqueue as server end. Front end hands down flutter, will build on Linux, apache/nginx, android, IOS, windows and even our TVs in livingroom for a front end interface. Even on FreeBSD natively once Dart is ported to it. I wouldn’t even bother with Java or Kotlin for these reasons alone. I think if that was built for a year would turn FreeBSD into everyone’s favorite virtualization OS.

Back to sys admin stuff, let’s set this up once really well and we should be good for next 5-10 years, just clone the drive if moving to PCIE 4 or 5 machine this is not Linux 🙂

Setup Bhyve:

Let’s start with something simple, setup our directory structure like vm-bhyve, mod what we need and uninstall his package. I’m going to assume at this point you just have a FreeBSD release like FreeBSD 13.1 installed, so let’s begin….

https://github.com/churchers/vm-bhyve

pkg install vm-bhyve bhyve-firmware
zfs create -o mountpoint=/vm zroot/vm
sysrc vm_enable="YES"
sysrc vm_dir="zfs:pool/vm"
vm init
cp /usr/local/share/examples/vm-bhyve/* /vm/.templates/

OK great now we have a directory structure to work with, he likes to keep all guest related things in /vm/<guestname>, so let’s keep with his directory structure idea and delete the package now.

pkg delete vm-bhyve
nano /etc/rc.conf
(now edit /etc/rc.conf and remove/comment out his sysrc lines)
(while we in here let's add support for 4 guest tap interfaces)

#support for 4 test guests - substitute "ix0" for your own interface
cloned_interfaces="bridge0 tap0 tap1 tap2 tap3"
ifconfig_bridge0_name="br0"
ifconfig_br0="addm ix0 addm tap0 addm tap1 addm tap2 addm tap3"
(exit /etc/rc.conf)
#now let's just create what that does manually on command line for now
ifconfig bridge create
ifconfig tap0 create
ifconfig tap1 create
ifconfig tap2 create
ifconfig tap3 create
ifconfig bridge0 addm ix0 addm tap0 addm tap1 addm tap2 addm tap3
ifconfig bridge0 name br0
ifconfig br0 up
#damn starting to like /etc/rc.conf better already :)
(alright now let's edit /etc/sysctl.conf)
nano /etc/sysctl.conf (add following:)
#BHYVE
net.link.tap.up_on_open=1
#BYHVE + PF nat
net.inet.ip.forwarding=1
net.inet6.ip6.forwarding=1
(exit /etc/sysctl.conf)
nano /boot/loader.conf
(have it look like this:)
autoboot_delay="5"
kernels="kernel kernel.old"
boot_serial="YES"
#stuff added by install
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
cryptodev_load="YES"
zfs_load="YES"
#bhyve
vmm_load="YES"
nmdm_load="YES"
if_bridge_load="YES"
if_tap_load="YES"
(exit /boot/loader.conf)

At this point we should be able to do an “ifconfig” and see our bridge setup properly for 4 possible guests, just add more tap interfaces for additional guests. Now let’s reboot and make sure our bridge etc is still there:

shutdown -r now
ifconfig -a
#let's install a few packages to help us
pkg install screen tightvnc git bash

Moving to FreeBSD current with experimental suspend/resume support(MOVIE TIME!):

Ok the time consuming part, hope you have a lot of cores/fast CPUs 🙂

Alright even if you have current installed you still will not have support enabled so we are going to recompile kernel and userland for support. Honestly this is way to upgrade FreeBSD from source anytime, only difference on stable/release is I would checkout a different branch with git and be using GENERIC instead of GENERIC-NODEBUG to copy from. You may be thinking I’ll checkout a release branch and just compile in support, tried that, it won’t work properly, this is your only way to test it before 14 release where it actually works for most part.

The make buildworld, make buildkernel, make installworld commands will take a long time, I recommend going to play your favorite video game or watch a movie after executing those commands, return in a few hours to check in on them time to time. Honestly once you have the buildworld out of the way its not that bad after. I’d recommend running everything in a screen session as well, that way if something happens you can log back in and do a “screen -r”. After your done, return tomorrow and we will continue on with testing suspend and resume 🙂

screen
git clone https://git.FreeBSD.org/src.git /usr/src
cd /usr/src/sys/amd64/conf
(edit MYKERNEL) cp GENERIC-NODEBUG MYKERNEL (add: options         BHYVE_SNAPSHOT)
cd /usr/src
#(find amount of CPUs and adjust -j below - "dmesg|grep SMP")
make -j12 buildworld -DWITH_BHYVE_SNAPSHOT -DWITH_MALLOC_PRODUCTION
make -j12 buildkernel KERNCONF=MYKERNEL
make installkernel KERNCONF=MYKERNEL
shutdown -r now
cd /usr/src; make installworld
shutdown -r now
etcupdate -B
pkg bootstrap -f #if new freebsd version
pkg upgrade -f   #if new freebsd version
shutdown -r now

Preliminary Thoughts on Creating Guests:

We are back and ready to roll! So creating guests is probably something every sys admin sits there and thinks hours upon hours about. What performance improvements could I make, will I be able to mount /etc and /boot directory of an offline guest if anything goes wrong or you screw anything up on guest. How will you do backups for them, how will you recover from disaster if it strikes. These guests once setup can run 5-10 years, no room for error, everything has to be accounted for.

I will give you my take on this from decades of experience. When you are starting out fresh, your main concern is the main virtualization host is doing nothing but virtualization and routing. You’d don’t want apache, email servers or any type of server processes running on it that are better suited for guests that can crash if need be and not affect the main host and other guests running on it. You want the main host rock stable at all times. You can play with the guests and give them the memory and CPUs they require to run what they need.

On the main FreeBSD host – virtualization, DNS, isc-DHCP/isc-DHCPD6, RADVD, PF FIREWALL, and at most backups/ZFS snapshots. Over the years before FreeBSD had Bhyve, I offloaded all backups to ZFS guest for backups there as well, using rsync or zfs send/recv. The choice is yours, but my recommendation is you are running as little as possible on main host as far as internet services.

If you did it all right, you’ll find your barely doing anything on main host and always logged into guests instead, then you know you did everything right. You might edit PF firewall to block something from all guests time to time, or do updates, about it. You have to remember your main host will save you with disaster recovery on guests, create new guests, basically be the blood and soul of your system, and this is where FreeBSD shines.

If you have ability at all to run FreeBSD as main host, you’ll save yourself years of headaches, where every Linux sys admin is reinstalling a new release on main host every 2-4 years, you did a freebsd-update or rebuilt the world and went and watched a movie while other sys admins were pulling their hair out all week wishing they documented their configs better 🙂

What about mission critical? In this situation your going to learn a lot about ZFS and send/recv to clone guests on the fly. For every other situation, a simple rsync once a night of the /etc, /usr/local/etc, /boot, /root, /home directories is all you need, why waste space? I’m not going to clone a 100GB guest byte for byte, if something happens to that guest, I have all its configs, I’m good to go. Install a simple rsyncd.conf on each guest to backup its configs each night. Every host is in charge of all their guests backups to a directory. Then its your decision to do offline backups with rsync or zfs send/recv from that host.

So I know all the worries, been there done that. As I’m moving through this guide, I’m going to show you disaster mitigation techniques on the host as well, so your well prepared if disaster ever strikes on a guest.

MY RESEARCH:

To run FreeBSD effectively as a main host in production a lot had to be accounted for, and I will list them here:

  1. All guests can be resumed quickly after a main host reboots for kernel update after suspending all guests. No ssh connections lost to guests, no accidental forgetting to save work bites you on resumes of guests.
  2. If a guest fails to boot at anytime, mandatory mounting of /etc , /boot etc directories of guest to fix problems on host.
  3. If a guest runs out of space, ability to resize guest on host, and grow the FS on the guest afterwards with as little downtime as possible.
  4. Ability to suspend a guest quickly to fix any problems. With this typically you want to reboot guest properly and fix issues from host after that.
  5. FreeBSD only: ability to ZFS snapshot guests and roll them back to any previous snapshot if needed.

Bugs I found and possible remedies:

In my preliminary research I found what bhyve actually does is run itself if a while loop, if it exists with error code 0, then bhyve is run again, effectively a “shutdown -r now” working properly. If it exits with any other error code then loop is broken, everything can be cleaned up after guest. The problem I found with suspend/resume so far on this issue is once suspended it exits with error code 0 as well. There definitely should be a different error code attached. There is a progress bar written to screen on STDOUT before it does shut down, only way currently to differentiate between the two is to capture that output. Something that would be better left to python asyncio/fastapi server process keeping all guests in a loop and determining difference between a suspend and a clean exit, then you could have a separate command line utility to access API’s on server process would probably be the best solution right now. A coding exercise in python that shouldn’t take more than a week to code. A 6-12 month front end to that with GUI and flutter would probably be best overall supporting the most platforms from one code base.

On further research there are mainly 3 types of ways to pass disks to guests, virtio-blk, virtio-scsi and nvme. On my tests with suspend/resume virtio-blk is stable each time I ran it. On virtio-scsi I had issues, on resume I could do an “ls -al”, see the filesystem properly but after doing anything else like a “df -k” or logging into system, it would hang then eventually crash with error code 139. I reported this is freebsd-current mailing list and hopefully someone gets around to looking at it.

Further research on virtio-blk has shown that it is slowly being phased out for virtio-scsi. Apparently this is the new version of passing disks that will be more common in the future. The belief behind it stems from being to hard to rework the virtio-blk code as well as virtio-scsi has more features and ability to pass way more disks from a host to guests. As far as nvme, I did not have any to test with, regardless of SSD type I believe the move will attempt to include all SSD/nvme to one virtio-scsi configuration in the future so users should embrace virtio-scsi once it is stable with suspend/resume on FreeBSD.

Upon further testing of attempting to mount a ZFS guest to the main host, it was unstable with no plans to fix it. Upon contacting freebsd-current mailing list about this issue, I was informed it causes deadlocks due to lock recursion and is why the sysctl vfs.zfs.vol.recursive was turned off by default. Suggestion was to use scsi instead, which in my testing did mount the ZFS guest without issues, a further suggestion that virtio-scsi is the future.

Upon examining the C source code to the virtio-scsi driver, it creates /dev/cam/* devices that can be used for the guests to passthrough targets with setup luns on the host.

router:/root # ls -al /dev/cam/ctl*
crw------- 1 root operator 0, 164 Nov 10 02:29 /dev/cam/ctl
crw------- 1 root operator 0, 167 Nov 10 08:14 /dev/cam/ctl1.0
crw------- 1 root operator 0, 168 Nov 10 08:14 /dev/cam/ctl2.0
router:/root #

What happens here is a port is created that can be used in the virtio-scsi line of a guests config to pass devices. Upon attaching to targets, the luns create /dev/da* devices.

router:/root # ls -al /dev/da*
crw-r----- 1 root operator 0, 134 Nov 10 02:29 /dev/da0
crw-r----- 1 root operator 0, 135 Nov 10 02:29 /dev/da0s1
crw-r----- 1 root operator 0, 136 Nov 10 02:29 /dev/da0s2
crw-r----- 1 root operator 0, 138 Nov 10 02:29 /dev/da0s2a
crw-r----- 1 root operator 0, 181 Nov 10 08:14 /dev/da1
crw-r----- 1 root operator 0, 183 Nov 10 08:14 /dev/da1p1
crw-r----- 1 root operator 0, 184 Nov 10 08:14 /dev/da1p2
crw-r----- 1 root operator 0, 185 Nov 10 08:14 /dev/da1p3
crw-r----- 1 root operator 0, 182 Nov 10 08:14 /dev/da2
crw-r----- 1 root operator 0, 186 Nov 10 08:14 /dev/da2p1
crw-r----- 1 root operator 0, 187 Nov 10 08:14 /dev/da2p2
crw-r----- 1 root operator 0, 188 Nov 10 08:14 /dev/da2p3
crw-r----- 1 root operator 0, 206 Nov 10 08:14 /dev/da3
crw-r----- 1 root operator 0, 207 Nov 10 08:14 /dev/da3p1
crw-r----- 1 root operator 0, 208 Nov 10 08:14 /dev/da3p2
router:/root #

The first one /dev/da0 is reserved for scsi itself, every other lun created in a target creates a new /dev/da* device, first one beginning at /dev/da1 and so forth.

For my purposes I had the ZFS guest as first lun in my test and was able to manipulate /dev/da1 successfully to mount the ZFS guest. I could also pass a target to a guest with as many disks/cdroms/ISOs(luns) as I wanted just by passing the /dev/cam/ctl1.0 target.

Let’s do a quick illustration of passing zvols in FreeBSD to scsi instead, you could also pass disk images if you wanted:

nano /etc/ctl.conf:
(add following:)
portal-group pg0 {
        discovery-auth-group no-authentication
        listen 127.0.0.1:3260
}
target iqn.2005-02.com.sunsaturn:target0 {
        auth-group no-authentication
        portal-group pg0
        #bhyve virti-iscsi disk - /dev/cam/ctl1.0
        port ioctl/1
        lun 0 {
                path /dev/zvol/zroot/asterisk
                #blocksize 128
                serial 000c2937247001
                device-id "iSCSI Disk 000c2937247001"
                option vendor "FreeBSD"
                option product "iSCSI Disk"
                option revision "0123"
                option insecure_tpc on
        }
}
target iqn.2005-02.com.sunsaturn:target1 {
        auth-group no-authentication
        portal-group pg0
        #bhyve virti-iscsi disk - /dev/cam/ctl2.0
        port ioctl/2

        lun 0 {
                path /dev/zvol/zroot/asterisk2
                #blocksize 128
                serial 000c2937247002
                device-id "iSCSI Disk 000c2937247002"
                option vendor "FreeBSD"
                option product "iSCSI Disk"
                option revision "0123"
                option insecure_tpc on
        }
        lun 1 {
                path /vm/.iso/FreeBSD-14.0-CURRENT-amd64-20221103-5cc5c9254da-259005-disc1.iso
                #byhve seems to just hang when I set it to an actual CDROM so let it default to type 0
                #device-type 5
                serial 000c2937247003
                device-id "iSCSI CDROM ISO 000c2937247003"
                option vendor "FreeBSD CDROM"
                option product "iSCSI CDROM"
                option revision "0123"
                option insecure_tpc on
        }
}

(close and exit)
nano /etc/iscsi.conf
(add following:)
t0 {
        TargetAddress   = 127.0.0.1:3260
        TargetName      = iqn.2005-02.com.sunsaturn:target0
}
t1 {
        TargetAddress   = 127.0.0.1:3260
        TargetName      = iqn.2005-02.com.sunsaturn:target1
}
(close and exit)
nano /etc/rc.conf
(add following:)
#ISCSI - service ctld start && service iscsid start
#server
ctld_enable="YES"          #load /etc/ctl.conf
iscsid_enable="YES"        #start iscsid process to connect to ctld
#client - service iscsictl start
iscsictl_enable="YES"      #connect to all targets in /etc/iscsi.conf
iscsictl_flags="-Aa"

(close and exit)
(now let's create some zvols to install guests on)
zfs create -V30G -o volmode=dev zroot/asterisk
zfs create -V30G -o volmode=dev zroot/asterisk2
cd /vm/.iso
wget https://download.freebsd.org/snapshots/amd64/amd64/ISO-IMAGES/14.0/FreeBSD-14.0-CURRENT-amd64-20221103-5cc5c9254da-259005-disc1.iso
(let's start scsi manually)
service ctld start && service iscsid start && service iscsictl start

Now we can see all those /dev/da* devices for us to manipulate on host for mounting shut down guests, as well as those /dev/cam/* devices for passing to guests.

Next let’s actually install 2 test guests, asterisk and asterisk2. For asterisk guest I will install UFS FreeBSD, and for asterisk2 I will install ZFS guest as well as pass the FreeBSD ISO to that guest.

To make this easier, let’s use a bash script I quickly coded to test suspend/resume capabilities of each guest, let’s call them asterisk.sh and asterisk2.sh:

cd /root
nano -w asterisk.sh
(add following)
#!/bin/bash
#
# General script to test bhyve suspend/resume features
#
# Requirements: FreeBSD current
# screen
# git clone https://git.FreeBSD.org/src.git /usr/src
# cd /usr/src/sys/amd64/conf (edit MYKERNEL) cp GENERIC MYKERNEL-NODEBUG; (add: options         BHYVE_SNAPSHOT)
# cd /usr/src
# (find amount of CPUs and adjust -j below - "dmesg|grep SMP")
# make -j12 buildworld -DWITH_BHYVE_SNAPSHOT -DWITH_MALLOC_PRODUCTION
# make -j12 buildkernel KERNCONF=MYKERNEL
# make installkernel KERNCONF=MYKERNEL
# shutdown -r now
# cd /usr/src; make installworld
# shutdown -r now
# etcupdate -B
# pkg bootstrap -f #if new freebsd version
# pkg upgrade -f   #if new freebsd version 
#
# Report anomolies to dan@sunsaturn.com

##############EDIT ME#####################


HOST="127.0.0.1"                        # vncviewer 127.0.0.1:5900 - pkg install tightvnc
PORT="5900"
WIDTH="800"
HEIGHT="600"
VMNAME="asterisk"
ISO="/vm/.iso/FreeBSD-14.0-CURRENT-amd64-20221103-5cc5c9254da-259005-disc1.iso"
DIR="/vm/asterisk"                      # Used to hold files when guest suspended
SERIAL="/dev/nmdm_asteriskA"           # For "screen /dev/nmdm_asteriskB" - pkg install screen
TAP="tap0"
CPU="8"
RAM="8G"

#For testing virtio-scsi
STORAGE="/dev/cam/ctl1.0"               # port from /etc/ctl.conf(port ioctl/1) - core dumping on resume
DEVICE="virtio-scsi"

#for testing virtio-blk                 # Comment out above 2 lines if using these
#DEVICE="virtio-blk"                    
#STORAGE="/dev/zvol/zroot/asterisk"     # Standard zvol
#STORAGE="/dev/da1"                     # Block device created from iscsictl

#########################################

usage() {
   echo "Usage: $1 start    (Start the guest: $VMNAME)"; 
   echo "Usage: $1 stop     (Stop the guest: $VMNAME)"; 
   echo "Usage: $1 resume   (Resume the guest from last suspend: $VMNAME)"; 
   echo "Usage: $1 suspend  (Suspend the guest: $VMNAME)"; 
   echo "Usage: $1 install  (Install new guest: $VMNAME)"; 
   exit
}

if [ ! -d "$DIR" ]; then 
   mkdir -p $DIR
fi

#if [ -z "$2" ]; then
#   usage
#else
#   VMNAME=$2
#fi


if [ "$1" == "install" ]; then
   #Kill it before starting it
   echo "Execute: screen $SERIAL"
   bhyvectl --destroy --vm=$VMNAME
   bhyve -c $CPU -m $RAM -w -H -A \
      -s 0:0,hostbridge \
      -s 3:0,ahci-cd,$ISO \
      -s 4:0,$DEVICE,$STORAGE  \
      -s 5:0,virtio-net,$TAP \
      -s 29,fbuf,tcp=$HOST:$PORT,w=$WIDTH,h=$HEIGHT \
      -s 30,xhci,tablet \
      -s 31,lpc -l com1,stdio \
      -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
      $VMNAME
   #kill it after 
   bhyvectl --destroy --vm=$VMNAME
elif [ "$1" == "start" ]; then 
   while true
   do
      echo "Starting $VMNAME -s 29,fbuf,tcp=$HOST:$PORT,w=$WIDTH,h=$HEIGHT"
      #Kill it before starting it
      bhyvectl --destroy --vm=$VMNAME > /dev/null 2>&1
      bhyve -c $CPU -m $RAM -w -H -A \
         -s 0:0,hostbridge \
         -s 4:0,$DEVICE,$STORAGE  \
         -s 5:0,virtio-net,$TAP \
         -s 29,fbuf,tcp=$HOST:$PORT,w=$WIDTH,h=$HEIGHT \
         -s 30,xhci,tablet \
         -s 31,lpc -l com1,$SERIAL \
         -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
         $VMNAME
      #DISABLING REBOOT LOOP AS SUSPEND RETURNS ERROR CODE 0 AS WELL
      #if [ "$?" != 0 ];
      #then
      #   echo "The exit code was not reboot code 0!: $?"
      #   exit
      #fi
      echo "The exit code was : $?"
      exit
   done
elif [ "$1" == "resume" ]; then 
   while true
   do
      echo "Starting $VMNAME -s 29,fbuf,tcp=$HOST:$PORT,w=$WIDTH,h=$HEIGHT"
      #Kill it before starting it
      bhyvectl --destroy --vm=$VMNAME > /dev/null 2>&1
      if [ -f "$DIR/default.ckp" ]; then
         bhyve -c $CPU -m $RAM -w -H -A \
            -s 0:0,hostbridge \
            -s 4:0,$DEVICE,$STORAGE  \
            -s 5:0,virtio-net,$TAP \
            -s 29,fbuf,tcp=$HOST:$PORT,w=$WIDTH,h=$HEIGHT \
            -s 30,xhci,tablet \
            -s 31,lpc -l com1,$SERIAL \
            -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
            -r $DIR/default.ckp \
            $VMNAME
      else
         echo "Guest was never suspended"
         exit
      fi
      #DISABLING REBOOT LOOP AS SUSPEND RETURNS ERROR CODE 0 AS WELL
      #if [ "$?" != 0 ];
      #then
      #   echo "The exit code was not reboot code 0!: $?"
      #   exit
      #fi
      echo "The exit code was : $?"
      exit
   done
elif [ "$1" == "suspend" ];
then 
   bhyvectl --suspend $DIR/default.ckp --vm=$VMNAME

elif [ "$1" == "stop" ]; then 
   bhyvectl --destroy --vm=$VMNAME 
else 
   usage
fi

Let’s also create asterisk2.sh:

#!/bin/bash
#
# General script to test bhyve suspend/resume features
#
# Requirements: FreeBSD current
# screen
# git clone https://git.FreeBSD.org/src.git /usr/src
# cd /usr/src/sys/amd64/conf (edit MYKERNEL) cp GENERIC MYKERNEL-NODEBUG; (add: options         BHYVE_SNAPSHOT)
# cd /usr/src
# (find amount of CPUs and adjust -j below - "dmesg|grep SMP")
# make -j12 buildworld -DWITH_BHYVE_SNAPSHOT -DWITH_MALLOC_PRODUCTION
# make -j12 buildkernel KERNCONF=MYKERNEL
# make installkernel KERNCONF=MYKERNEL
# shutdown -r now
# cd /usr/src; make installworld
# shutdown -r now
# etcupdate -B
# pkg bootstrap -f #if new freebsd version
# pkg upgrade -f   #if new freebsd version 
#
# Report anomolies to dan@sunsaturn.com

##############EDIT ME#####################


HOST="127.0.0.1"                        # vncviewer 127.0.0.1:5900 - pkg install tightvnc
PORT="5901"
WIDTH="800"
HEIGHT="600"
VMNAME="asterisk2"
ISO="/vm/.iso/FreeBSD-14.0-CURRENT-amd64-20221103-5cc5c9254da-259005-disc1.iso"
DIR="/vm/asterisk2"                      # Used to hold files when guest suspended
SERIAL="/dev/nmdm_asterisk2A"           # For "screen /dev/nmdm_asterisk2B" - pkg install screen
TAP="tap1"
CPU="8"
RAM="8G"

#For testing virtio-scsi
STORAGE="/dev/cam/ctl2.0"               # port from /etc/ctl.conf(port ioctl/1) - core dumping on resume
DEVICE="virtio-scsi"

#for testing virtio-blk                 # Comment out above 2 lines if using these
#DEVICE="virtio-blk"                    
#STORAGE="/dev/zvol/zroot/asterisk2"     # Standard zvol
#STORAGE="/dev/da2"                     # Block device created from iscsictl

#########################################

usage() {
   echo "Usage: $1 start    (Start the guest: $VMNAME)"; 
   echo "Usage: $1 stop     (Stop the guest: $VMNAME)"; 
   echo "Usage: $1 resume   (Resume the guest from last suspend: $VMNAME)"; 
   echo "Usage: $1 suspend  (Suspend the guest: $VMNAME)"; 
   echo "Usage: $1 install  (Install new guest: $VMNAME)"; 
   exit
}

if [ ! -d "$DIR" ]; then 
   mkdir -p $DIR
fi

#if [ -z "$2" ]; then
#   usage
#else
#   VMNAME=$2
#fi


if [ "$1" == "install" ]; then
   #Kill it before starting it
   echo "Execute: screen $SERIAL"
   bhyvectl --destroy --vm=$VMNAME
   bhyve -c $CPU -m $RAM -w -H -A \
      -s 0:0,hostbridge \
      -s 3:0,ahci-cd,$ISO \
      -s 4:0,$DEVICE,$STORAGE  \
      -s 5:0,virtio-net,$TAP \
      -s 29,fbuf,tcp=$HOST:$PORT,w=$WIDTH,h=$HEIGHT \
      -s 30,xhci,tablet \
      -s 31,lpc -l com1,stdio \
      -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
      $VMNAME
   #kill it after 
   bhyvectl --destroy --vm=$VMNAME
elif [ "$1" == "start" ]; then 
   while true
   do
      echo "Starting $VMNAME -s 29,fbuf,tcp=$HOST:$PORT,w=$WIDTH,h=$HEIGHT"
      #Kill it before starting it
      bhyvectl --destroy --vm=$VMNAME > /dev/null 2>&1
      bhyve -c $CPU -m $RAM -w -H -A \
         -s 0:0,hostbridge \
         -s 4:0,$DEVICE,$STORAGE  \
         -s 5:0,virtio-net,$TAP \
         -s 29,fbuf,tcp=$HOST:$PORT,w=$WIDTH,h=$HEIGHT \
         -s 30,xhci,tablet \
         -s 31,lpc -l com1,$SERIAL \
         -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
         $VMNAME
      #DISABLING REBOOT LOOP AS SUSPEND RETURNS ERROR CODE 0 AS WELL
      #if [ "$?" != 0 ];
      #then
      #   echo "The exit code was not reboot code 0!: $?"
      #   exit
      #fi
      echo "The exit code was : $?"
      exit
   done
elif [ "$1" == "resume" ]; then 
   while true
   do
      echo "Starting $VMNAME -s 29,fbuf,tcp=$HOST:$PORT,w=$WIDTH,h=$HEIGHT"
      #Kill it before starting it
      bhyvectl --destroy --vm=$VMNAME > /dev/null 2>&1
      if [ -f "$DIR/default.ckp" ]; then
         bhyve -c $CPU -m $RAM -w -H -A \
            -s 0:0,hostbridge \
            -s 4:0,$DEVICE,$STORAGE  \
            -s 5:0,virtio-net,$TAP \
            -s 29,fbuf,tcp=$HOST:$PORT,w=$WIDTH,h=$HEIGHT \
            -s 30,xhci,tablet \
            -s 31,lpc -l com1,$SERIAL \
            -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
            -r $DIR/default.ckp \
            $VMNAME
      else
         echo "Guest was never suspended"
         exit
      fi
      #DISABLING REBOOT LOOP AS SUSPEND RETURNS ERROR CODE 0 AS WELL
      #if [ "$?" != 0 ];
      #then
      #   echo "The exit code was not reboot code 0!: $?"
      #   exit
      #fi
      echo "The exit code was : $?"
      exit
   done
elif [ "$1" == "suspend" ];
then 
   bhyvectl --suspend $DIR/default.ckp --vm=$VMNAME

elif [ "$1" == "stop" ]; then 
   bhyvectl --destroy --vm=$VMNAME 
else 
   usage
fi

Great now let’s install the guests:

chmod 755 *.sh
./asterisk.sh install
(at this point just install FreeBSD with default UFS options, I actually chose to remove swap partition at the end, and the root partition then add them back as swap 2nd and root partition last to avoid headaches having to grow the disk later on, I suggest you do the same)
./asterisk2.sh install (install with GPT+UEFI)
(install FreeBSD again with default ZFS install, there is no headache here partitions are perfect here as defaults on zroot)

(At this point I may edit /etc/fstab to change whatever hardcoded device names
are in there to GPT names from /dev/gpt/* so when we switch between virtio-scsi and virtio-blk devices it won't matter. Ie for swap switch it to:
/dev/gpt/swap0 instead) If you are ever wanting to show what the GPT label names are you can either do "gdisk -l <device>" or "gpart show -l <device>" to figure out what to put into /etc/fstab. If they have no label , give them one. )

#now let's start them both after the install
./asterisk.sh start
#another terminal
./asterisk2.sh start
#another terminal
screen /dev/nmdm_asteriskB
#another terminal
screen /dev/nmdm_asterisk2B
#another terminal
./asterisk.sh suspend
./asterisk.sh resume
#another terminal
./asterisk2.sh suspend
./asterisk2.sh resume

Now you will notice on your screen session, after resuming guest “ls” etc works but as soon as we do anything else, “df -h” it will hang and after about a minute it will core dump.

router:/root # ./asterisk2.sh resume
Starting asterisk2 -s 29,fbuf,tcp=127.0.0.1:5901,w=800,h=600
fbuf frame buffer base: 0x229792600000 [sz 16777216]
Pausing pci devs...
pci_pause: no such name: virtio-blk
pci_pause: no such name: ahci
pci_pause: no such name: ahci-hd
pci_pause: no such name: ahci-cd
Restoring vm mem...
[8192.000MiB / 8192.000MiB] |################################################################################################################################################|
Restoring pci devs...
vm_restore_user_dev: Device size is 0. Assuming virtio-blk is not used
vm_restore_user_dev: Device size is 0. Assuming virtio-rnd is not used
vm_restore_user_dev: Device size is 0. Assuming e1000 is not used
vm_restore_user_dev: Device size is 0. Assuming ahci is not used
vm_restore_user_dev: Device size is 0. Assuming ahci-hd is not used
vm_restore_user_dev: Device size is 0. Assuming ahci-cd is not used
Restoring kernel structs...
Resuming pci devs...
pci_resume: no such name: virtio-blk
pci_resume: no such name: ahci
pci_resume: no such name: ahci-hd
pci_resume: no such name: ahci-cd
./asterisk2.sh: line 145: 10883 Segmentation fault      (core dumped) bhyve -c $CPU -m $RAM -w -H -A -s 0:0,hostbridge -s 4:0,$DEVICE,$STORAGE -s 5:0,virtio-net,$TAP -s 29,fbuf,tcp=$HOST:$PORT,w=$WIDTH,h=$HEIGHT -s 30,xhci,tablet -s 31,lpc -l com1,$SERIAL -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -r $DIR/default.ckp $VMNAME
The exit code was : 139
router:/root # 

Great so scsi is unstable with suspend/resume. Now run same tests but let’s modify asterisk2.sh script to use virtio-blk instead with /dev/da2 and run same tests:

#For testing virtio-scsi
#STORAGE="/dev/cam/ctl2.0"              # port from /etc/ctl.conf(port ioctl/1) - core dumping on resume
#DEVICE="virtio-scsi"

#for testing virtio-blk                 # Comment out above 2 lines if using these
DEVICE="virtio-blk"
#STORAGE="/dev/zvol/zroot/asterisk2"    # Standard zvol
STORAGE="/dev/da2"                      # Block device created from iscsictl

You may be wondering how I know which /dev/da* asterisk2 is using:
I simply run:
iscsictl 
#This will give you a list of who is on what, it can be completely random
#always check this list before mounting a guest so you don't mount wrong one
#typically everything on lun0 will be numbered first, then lun1 but this is not #always the case so make sure to run that
#on a non-testing host it may look something like this:
router:/root # iscsictl 
Target name                          Target portal    State
iqn.com.sunsaturn.asterisk:target1   127.0.0.1:3260   Connected: da1 da5 
iqn.com.sunsaturn.rocky:target2      127.0.0.1:3260   Connected: da2 da6 
iqn.com.sunsaturn.ubuntu:target3     127.0.0.1:3260   Connected: da3 da8 
iqn.com.sunsaturn.windows:target4    127.0.0.1:3260   Connected: da4 da7 
router:/root # 

Here is something good to add to your .bash_profile then you don't have to 
think about it ever again:

if [ "$HOSTNAME" == "test.test.com" ]; then
   echo "Checking which devices guests connected to:"
   echo "#######################################################"
   iscsictl
   echo "#######################################################"
fi

I personally have a 2nd root account I set to bash so I don't touch default root
shell, can use toor if you like. I suggest everyone do that, there will come a day where you are like damn I can't remember my root password anymore because been using ssh keys so long. Yes you have to su to root everytime, but I even automate that these days.

Now let’s run it:

./asterisk2.sh start
#another terminal (make sure watching screen terminal running these)
./asterisk2.sh suspend
./asterisk2.sh resume

WORKS PERFECTLY:
Now go uncomment STORAGE line with 
STORAGE="/dev/zvol/zroot/asterisk2"
and comment out:
#STORAGE="/dev/da2"
WORKS PERFECTLY TO

So what can we see from suspend/resume stability, it works on virtio-blk perfectly. No matter if I use the /dev/da* devices from scsi or the zvols directly.

What about importing ZFS pool from asterisk2? Sure let’s do it on host, shut it down first:

router:/root # gpart show /dev/da2
=>      40  62914480  da2  GPT  (30G)
        40    532480    1  efi  (260M)
    532520      2008       - free -  (1.0M)
    534528  16777216    2  freebsd-swap  (8.0G)
  17311744  45600768    3  freebsd-zfs  (22G)
  62912512      2008       - free -  (1.0M)

router:/root # gdisk -l /dev/da2
GPT fdisk (gdisk) version 1.0.9

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.
Disk /dev/da2: 62914560 sectors, 30.0 GiB
Sector size (logical): 512 bytes
Disk identifier (GUID): 8ACD2112-5EBF-11ED-8F56-00A098E3C14E
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 40, last usable sector is 62914519
Partitions will be aligned on 8-sector boundaries
Total free space is 4016 sectors (2.0 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1              40          532519   260.0 MiB   EF00  efiboot0
   2          534528        17311743   8.0 GiB     A502  swap0
   3        17311744        62912511   21.7 GiB    A504  zfs0
router:/root # 

So we can see this is definitely our ZFS guest, and we already know we cannot manipulate /dev/zvol/zroot/asterisk2 directly because of recursion issues, but we can manipulate /dev/da2 through scsi just fine:

#Import ZFS guest under /mnt (shut down the guest)
#First import it under /mnt under a different temporary name that won't save the name when we export it after with "-t" option
zpool import (we should see asterisk2's zroot ready for import)
zpool import -fR /mnt -t zroot testing
zfs mount -o mountpoint=/mnt testing/ROOT/default
(go ahead fix /mnt/etc /mnt/boot problems)
zpool export testing
rm -rf /mnt/* (cleanup left over directories)

#hell let's mount asterisk guest with UFS as well on /mnt
#I did crash it a few times so probably need recovery
router:/root # gpart show /dev/da1
=>       40  104857520  da1  GPT  (50G)
         40         24       - free -  (12K)
         64     532480    1  efi  (260M)
     532544   16777216    2  freebsd-swap  (8.0G)
   17309760   87547776    3  freebsd-ufs  (42G)
  104857536         24       - free -  (12K)

router:/root # mount -t ufs /dev/da1p3 /mnt
mount: /dev/da1p3: R/W mount of / denied. Filesystem is not clean - run fsck. Forced mount will invalidate journal contents: Operation not permitted
router:/root # fsck /dev/da1p3 
** /dev/da1p3
** SU+J Recovering /dev/da1p3

USE JOURNAL? [yn] y

** Reading 182419456 byte journal from inode 4.

RECOVER? [yn] y

** Building recovery table.
** Resolving unreferenced inode list.
** Processing journal entries.

WRITE CHANGES? [yn] y


***** FILE SYSTEM IS CLEAN *****
** 96 journal records in 6656 bytes for 46.15% utilization
** Freed 27 inodes (4 dirs) 0 blocks, and 20 frags.

***** FILE SYSTEM MARKED CLEAN *****
router:/root # mount -t ufs /dev/da1p3 /mnt
router:/root # ls /mnt
bin  boot  COPYRIGHT  dev  entropy  etc  home  lib  libexec  media  mnt  net  proc  rescue  root  sbin  sys  tmp  usr  var
router:/root # 

Summary:

Suspend/resume does work, just not passing in virtio-scsi with those /dev/cam/* devices. Seems we can do “ls” just fine, so something else is going on.

For now what you can do is just use the virtio-blk devices, until virtio-scsi works. In fact I would set it up this way since we already know virtio-blk is on its way out, plus it was so much nicer to pass in a CDROM on asterisk2 in just 1 line to guest, or as many disks as we want. We learned we can use /dev/da* devices for direct ZFS importing of guests pools for disaster recovery, for anything that isn’t ZFS we could also use the /dev/zvol/zroot/* devices directly if we wanted, but it is a cool work around from FreeBSD team, and sets you up using virtio-scsi now.

If you do not care about suspend/resume right now you could just continue using virtio-scsi to future proof yourself for virtio-blk phase out, or you could just pass virtio-blk devices temporarily till it is fixed and use suspend/resume all you want. My personal preference till this gets sorted out is leave iscsi processes running till it gets resolved, and use virtio-blk directly against /dev/zvol/root/<guestname> for now, can switch them later, this way you can use suspend/resume all you want as well as mount any ZFS guests through the /dev/da* devices anytime you need. This way your at least future proofed. If you don’t want to use the /dev/da* devices :

zfs set volmode=full zroot/asterisk
(if you want to be able to mount non-ZFS guests directly at:
/dev/zvol/zroot/asterisk for instance, what this will do is separate asterisk into asteriskp1 asteriskp2 asteriskp3 etc for your mounting purposes)
#Just get used to scsi already :) I know old habits die hard and no one wants to #give up NIS either and learn LDAP, hell neither do I, whatever works :)

Hope you enjoyed this article on suspend/resume experimental support for FreeBSD, perhaps in another article I will show you Linux and Windows guests, and how we can disaster recovery them as well like we did for these 2 UFS and ZFS guests today, till then….

Shine bright like a diamond,

Dan.

FreeBSD certbot wildcard automatic renewals with bind

As many have experienced, wildcard automatic renewals are not working. Where we used to have “certbot renew” to just take care of everything, that no longer works.

My goal then is to have it work again without touching our current cronjobs so let’s get started.

lets encrypt wildcard instructions

pkg install py37-certbot-dns-rfc2136
tsig-keygen -a HMAC-SHA512 acme-update

add contents to named.conf from above command

EXAMPLE: named.conf

key “acme-update” {
algorithm hmac-sha512;
secret “my long ass secret with double quotes”;
};

//test.com
zone “test.com” {
type master;
file “master/test.com”;
update-policy {
grant “acme-update” name _acme-challenge.test.com TXT;
};
};

pico /usr/local/etc/letsencrypt/rfc2136.ini (add the following for certbot)

dns_rfc2136_server = 5.5.5.5 (PUT YOUR IP ADDRESS)
dns_rfc2136_name = acme-update
dns_rfc2136_secret = mylongasssecret
dns_rfc2136_algorithm = HMAC-SHA512

chmod 600 /usr/local/etc/letsencrypt/rfc2136.ini

certbot certonly --dns-rfc2136 --dns-rfc2136-credentials /usr/local/etc/letsencrypt/rfc2136.ini --server https://acme-v02.api.letsencrypt.org/directory --email admin@test.com --agree-tos --no-eff-email --domain 'test.com' --domain '*.test.com'

Congratulations, for now on your normal “certbot renew” command in your cronjob will work like it did before.

FreeBSD 12.1 + Alpine with GPG

Intro:
I decided to install GPG on FreeBSD with alpine. What does this do? It’s the old days, using pgp to encrypt your email before sending. This is a howto so everyone can start encrypting their emails. Why do it? Back in the 90s when I was sitting in computer science class it was common courtesy and etiquette to always provide people with your PGP key when sending emails. So by not providing people with your PGP key, it’s considered disrespectful among the computer professionals. This is a tribute to my old classmates Isaac Eaglestone and Jason Barlow, I wish I could find them again. Especially Isaac who would bitch me out every other day for not using it 🙂 To be fair it was a headache to get anything working back then with an email client, considering all we had to work with was Slackware Linux back then, so I decided let’s go through FreeBSD, pull our hair out fixing any errors that pop up and let’s get a reliable Alpine + GPG setup going!

Can this stop quantum computers?

By now we all know Shor’s algorithm is set to break all asymmetric encryption. So what we will do is use best encryption we can with GPG using symmetric encryption, GPG supports AES256, so we will use that along with using RSA for compatibility. For all said purposes we will use the strongest that makes sense and stay compatible with other people’s keys as well.

Why use alpine?

If you ssh into systems on a regular basis, it makes no sense to download your email to an insecure device at home. If your using openvpn to download over VPN to a client such as Kmail to your Google Pixel Phone, it should be ok. What makes a phone insecure is trusting to many app developers. FaceBook for instance has been known to go behind people’s backs and upload your contacts to their servers. FaceBook also owns whatsapp. If you want to keep your phone secure, don’t put these on your phone.

FreeBSD Prerequisites: PART 1

Firstly I prefer using alpine with Postfix and Maildir support, since the Maildir patch is not available with standard pkg system. Off to the ports we go:

Let’s install alpine from ports, lock it from package manager updating it, install alpine gpg addon from pkg system, and see what directories it used for installing it.

cd /usr/ports/mail/alpine
make config #(Select Maildir patch)
make
make install
pkg lock alpine
pkg install ez-pine-gpg 
pkg list ez-pine-gpg

(Assuming your using bash for your shell and nano for your editor)
Next we want to get rid of any “pinentry” errors that may come up, the first problem I ran into, the following will solve it next login:

alias pico='nano -w'
pico ~/.bash_profile #(add the following next line,save and exit)
export GPG_TTY=$(tty)

At this point at least run alpine once to get your .pinerc created if its not already, then let’s open .pinerc and REPLACE display-filters and sending-filters with the following:

# This variable takes a list of programs that message text is piped into
# after MIME decoding, prior to display.
display-filters=_BEGINNING("-----BEGIN PGP")_ /usr/local/bin/ez-pine-gpg-incoming

# This defines a program that message text is piped into before MIME
# encoding, prior to sending
sending-filters=/usr/local/bin/ez-pine-gpg-sign-and-encrypt _INCLUDEALLHDRS_ _RECIPIENTS_,
        /usr/local/bin/ez-pine-gpg-encrypt _RECIPIENTS_,
        /usr/local/bin/ez-pine-gpg-symmetric _RECIPIENTS_,
        /usr/local/bin/ez-pine-gpg-sign _INCLUDEALLHDRS_

Alright we are getting closer, now we want to actually create our gpg key if you don’t have one already, now we run into the ssh X11 forwarding headache if you have it enabled when you su to another user, so to make sure we have no issues ssh to localhost as that user without X manually so we don’t get any end of file errors creating our brand new key. This generally happens because when you su to another user, the tty is still owned by user who logged in on tty device and permissions are generally 600 on it. You can get around it by chowning the tty device or using tmux, but honestly why go through the trouble, just ssh as user you want to create the key with:

ssh -x localhost #(disable X forwarding for this user)
gpg --full-generate-key

Enable encrypted swap space if you have newer hardware and your CPU supports AES-NI, here is a quick test:

swapoff -a
kldload -n aesni
swapon -a
dmesg #this should show if CPU supports it or not

If above is supported, put aesni_load=”YES” in /boot/loader.conf and append the “.eli” suffix to all swap devices. Enjoy your encrypted swap space.

Ok now let’s create our gpg.conf file, so we can remove unsecure memory errors if you don’t have secure memory space and set some defaults for encryption. You remembered to run gpg at least once so ~/.gnupg directory got created right? Just type in random crap and hit CTRL-D.

pico ~/.gnupg/gpg.conf #add following to the file, save and exit:
no-secmem-warning
cipher-algo AES256
personal-cipher-preferences AES256 AES192 AES CAST5
personal-digest-preferences SHA512 SHA384 SHA256 SHA224
default-preference-list SHA512 SHA384 SHA256 SHA224 AES256 AES192 AES CAST5 ZLIB BZIP2 ZIP Uncompressed
cert-digest-algo SHA512
s2k-digest-algo SHA512
s2k-cipher-algo AES256
keyid-format 0xlong
with-fingerprint
use-agent
charset utf-8

Now let’s edit our keyserver daemon’s config file, this always goes through tor, so unless you want to be waiting 30 seconds for it to timeout all the time, just install tor! Yes, you could just set option for it not to use tor, but only thing that is going to do is make it start faster, after process forks it will try tor anyways, so trust me save yourself the hours of headache and just install tor so your not always waiting on dirmngr to timeout using tor, unless of course you enjoy sitting there waiting 30+ seconds for a simple command like gpg –search-key <KEYID>. This dirmngr.conf file and daemon is only used when dealing with public internet keyservers.

There is nothing wrong with using Tor, just as long as you aren’t an exit node, if you can run one awesome! Is Tor secure? Absolutely not, NSA pulled 2 people out of DefCon conference when 2 researchers from university found a way to exploit TLS in there, then they have been in talks with source code developers of Tor as well. If that is not enough they are known to run honeypots all over the system. If you want secure, your best to run a VPN, then run that through Tor. Personally I think of having Tor on my system just like enabling IPV6 on it, just another network I can talk to. For our purposes we are going to use it just so dirmngr doesn’t piss us off 🙂 Besides we can just do “service tor stop” anytime we not talking to a keyserver if we like.

pico ~/.gnupg/dirmngr.conf #add following to the file, save and exit:
keyserver hkps://keys.openpgp.org

#now setup Tor if you haven't already
pkg install tor
pico /etc/rc.conf  #add the following, save and exit
tor_enable="YES"

#I am not going through configuring tor in this article, at least go into
#/usr/local/etc/tor and configure torrc and torsocks.conf so port 9050 is #working, just make sure your not running an exit node!

service start tor  #start tor

Now you may be asking why I am using keys.openpgp.org instead of sks key servers. The reason is simple, there is a DOS attack that has been known for decades that still works on all internet key servers. The DOS is quite simple, sign someone’s key 150k+ times and upload it to keyserver effectively destroying their gpg installation once they refresh their keys. The key server I have picked for us today, is only one on internet as of this date that at least attempts to mitigate this attack. So do yourself a favor and use it! As of recent versions of gpg all keyserver lines go in this file now,  not gpg.conf anymore.  As of this date I am using version 2.2.21.

Great now let’s get on with the real stuff, playing with gpg itself. First we want to generate our key. I suggest leaving things at defaults, but you can set RSA key to 4096, there really not much security between 2048, maybe 40 bits, according to gpg website, trade off in performance not really worth it.

I am saying use defaults just to be compatible with other people, in future what we really want to do is just replace RSA with elliptical curve bitcoin uses by selecting secp256k1 in expert mode down the road. (ie: selecting 10 and 9 with next command) Let’s not do that now. We still using AES256, even with quantum computers with 4096 qubits would only knock that down to AES128, by time they have that 1 million qubit computer hopefully the quantum algorithm is out. Pick a good password! With today’s technology they can brute force 2.8 billion tries a day, that is enough to try every lower case character a-z of a 10 character password in one day! Mix it up with upper case, numbers, special characters and use 4 words if you can!

Back in 90s, I used sentences with pgp, then I always forgot what it was cause I wasn’t using it that frequently like a password you would use to login with email or ssh, so keep it to something you can remember!

gpg --full-generate-key --expert #generate our key!

You will notice it put a revocation certificate in: ~/.gnupg/openpgp-revocs.d/
You will need this to revoke your key down the road if you loose it.

Ok at this point go test it, go into alpine, send an email to yourself, hit CTRL-x like usual to send, but before typing “Y” to send, hit CTRL-p instead to scroll through sending filters, select something like sign and encrypt, then hit “Y” to send.

If all goes well, you should get prompted for your password, gpg-agent will then store this password in shared memory for a set amount of time, which you can actually specify for how long in config file, or until server is rebooted or gpg-agent is killed and restarted. You’ll notice in “ps aux” every time you deal with gpg in anyway, that gpg-agent is running.

This is how gpg works. Try thinking gpg as a key-ring management tool. Everytime we use gpg we are mostly just a client talking to that gpg-agent server process running. We can list keys in there, tell it to remember our password in shared memory for X amount of time, sign things, encrypt and de-crypt files and much more. So now just sending email to our self is not very useful.

At this point I want you to add another user to your system, I want you to repeat all these steps and do it for this second user. Send email to itself, and when you got that working and ready to send to each other, let’s continue… Remember when using su – $USER you cannot create a key if he does not have his own tty, make sure to ssh -x $USER@localhost so he get his own tty so you have no issues!

PART 2 – Actually working with GPG (our key manager)

OK if you made it this far, congratulations! The hard part is done! You completed the setup! Give yourself a pat on the back. Only things we are going to do now is play with gpg command itself, that’s about it and learn what cool things we can do with it.

Let’s start off by continuing where we left off, I told you to create another user on system and setup his .gnupg directory. So at this point let’s send our practice emails to that user back and forth.

First Step: Let’s export our public keys for each of these accounts:

cd /tmp
gpg --armor --output user1@example.com.public-key.gpg --export user1@example.com
#now for other user:
cd /tmp
gpg --armor --output user2@example.com.public-key.gpg --export user2@example.com
#I like to keep copies of these in my .gnupg directory so let's do that
#for each user do following:
cd ~/.gnupg
cp /tmp/user1@example.com.public-key.gpg .
cp /tmp/user2@example.com.public-key.gpg .
#now for user1:
gpg --import user2@example.com.public-key.gpg
gpg --sign-key user2@example.com
#now for user2:
gpg --import user1@example.com.public-key.gpg
gpg --sign-key user1@example.com

Ok great, what we did was import the key, then signed the keys for each user because we trust them. Great now jump in alpine on each user and send emails back and forth with the CTRL-p filters. Play with it for a bit, you will notice gpg-agent daemon starts asking you for your password. Pinentry program runs here to ask for it, which is

sunsaturn:~/.gnupg # ls -al /usr/local/bin/pinentry
lrwxr-xr-x 1 root wheel 12 Oct 11  2019 /usr/local/bin/pinentry -&gt; pinentry-tty
sunsaturn:~/.gnupg # 

gpg-agent keeps your password in shared memory approx 2 hours, unless you change that in config file or you restart gpg-agent. You can kill gpg-agent and dirmngr daemons anytime you want with “gpgconf –kill all”. Or the old school reliable way of “ps aux” “kill -9 <pid1> <pid2>”

Wonderful you have done it! But wait we probably want to submit our key for at least ourselves to the internet keyservers! We don’t have to but it would be nice if we could link our email address to a PGP key on internet so people could find us easily.

Ok what we will do is submit our key to SKS keyservers as well as our default openpgp key server. Then we will add our key to our .signature file in alpine so whole world now knows we have a PGP key. We will even put a copy of our public PGP in our .signature file so people can grab it anytime through a website, sound cool? Great let’s do it…

 

gpg --list-keys #let's start by listing the keys and find our <KEYID>

pub   rsa2048/0xFF6F49977311C386 2020-07-17 [SC]
      Key fingerprint = A1A7 6E84 FB0B 8994 C3B5  A1BA FF6F 4997 7311 C386
uid                   [ultimate] Dan The Man (Dan @ SunSaturn)

For example above here is my key, my <KEYID> is the numbers/letters after the pub rsa2048/0x string. So here my <KEYID> is FF6F49977311C386. The reason we have 0x in front of it is because in our gpg.conf file we have “keyid-format 0xlong”. It’s just to prevent problems really, had I done just “keyid-format long” then it would not have the “0x” in front of it. Also you can see the fingerprint of my public key. So since we are using 0xlong I can use 0xFF6F49977311C386 as my <KEYID> here.

Alright let’s submit our key to keys.openpgp.org, since that is in our dirmngr.conf file as the keyserver that is what we will default to.

gpg --send-key 0xFF6F49977311C386     #use your KEYID!
gpg --search-key 0xFF6F49977311C386   #use your KEYID!

Great if all went well we submitted our key to keys.opengpg.org and then searched it and got it back. Now wouldn’t it be cool to search by our email address instead? Go in your browser now to : https://keys.openpgp.org follow instructions in your email and this site to verify your email address so people can search for your key by your email address. Once you are done that awesome let’s see if it worked:

gpg --search-key user@domain.com #use your email now!
gpg: data source: https://keys.openpgp.org:443
(1) Dan The Man (Dan @ SunSaturn) 
2048 bit RSA key 0xFF6F49977311C386, created: 2020-07-17

Good job, now let’s submit our key to SKS servers as well:

gpg --send-key --keyserver pool.sks-keyservers.net 0xFF6F49977311C386
gpg --search-key 0xFF6F49977311C386 #use your KEYID for both!
gpg --search-key user@domain.com    #use email to if you like

Now you have to realize pool.sks-keyservers.net is a pool of addresses, it may take time for them all to sync, if you ran command “host -t A pool.sks-keyservers.net”, you can see IP address is going to rotate each time, but if you ran those 2 commands above quickly you may have gotten same IP address twice and it successfully searched the key. Don’t worry about this, check back in in 24 hours. One good thing is we don’t have to do any email verification checks to list our key on SKS servers, so we are done. For a list of pools visit : https://sks-keyservers.net/overview-of-pools.php

Almost there, last thing we want to do is tell the world in our .signature file on alpine we are able to use PGP/GPG if people wish to add our public key to their keyring to encrypt emails/files to us. For that we want our public key on our website somewhere, and we want our fingerprint for the file so we can include that for people so they know it was not tampered with.

gpg --armor --output /path/to/website/root/pgp.txt --export user@example.com
gpg --list-keys

In first command above we exported our public key to directory of our website, or just copy pgp.txt to your website on another server if needed. In the second command we looking for “Key fingerprint” line so in my case:

gpg --list-keys
pub rsa2048/0xFF6F49977311C386 2020-07-17 [SC]
Key fingerprint = A1A7 6E84 FB0B 8994 C3B5 A1BA FF6F 4997 7311 C386

My fingerprint is “A1A7 6E84 FB0B 8994 C3B5 A1BA FF6F 4997 7311 C386”. By putting a link to pgp.txt file and giving them this fingerprint in .signature file this gives people 3 ways now they can find us. Through openpgp keyserver, through SKS keyservers, and also our emails. So let’s edit our .signature:

pico ~/.signature #add something as follows:
PGP Key: https://SunSaturn.com/pgp.txt
A1A7 6E84 FB0B 8994 C3B5 A1BA FF6F 4997 7311 C386

For reference here is my .signature with my email address/phone number removed for this blog, obviously put your pgp.txt and fingerprint ID in it’s place 🙂


Dan The Man
CEO & Founder
Websites, Domains and Everything else
PGP Key: https://SunSaturn.com/pgp.txt
A1A7 6E84 FB0B 8994 C3B5 A1BA FF6F 4997 7311 C386

That’s it we are done! Congratulations for doing the entire setup! For now on all you will ever have to do is remember to hit CTRL-p to send with PGP/GPG and your password. I hope to god you can remember your password. Place a file somewhere giving you hints what it is if needed.

Closing Thoughts:

Keep your private key secure. We all know by now intelligence agencies store encrypted data to save at a later date when technology gets better to decrypt. That being said your emails should be fine until they have quantum computers with millions of qubits. If your really paranoid, create an advanced key with elliptic curves from this setup, it just won’t be compatible with most people at this point for anyone running older versions of gpg. For any important files you need to encrypt, always use the best you can. When quantum algorithms come out, unencrypt your files, then encrypt them again with newest standard. If your key ever becomes compromised revoke the keys on both keyservers we submitted to and go through creating new key again. You can also do a shared password between the both of you using one of the sending filters, cool right?

If you have really sensitive information to send someone, both of you agree to access the file over a vpn/ssh connection to download the PGP file. Gives you a double later of protection. Even better yet, ssh over a VPN connection for a third layer 🙂 Someone storing encrypted data would have to break your VPN key, your SSH key and then your PGP key, you’ll most likely be dead by then, should be good 🙂 For advanced users: create a cronjob that switches your vpn key and ssh secret/public keys at regular intervals, ultimate protection. Store your encrypted files in an encrypted filesystem preferably on an SSD with many layers of 4 or more like QLC with trim support, more layers there are, harder it is for forensics teams to grab deleted data, they will give up. Personally I don’t have any sensitive data, but if I did those are avenues I would use. For me I use VPN’s for what they were made for, accessing internal machines on remote servers like I was there.

Encrypt files: (create files with .gpg ending)

#have a friends public key imported? 
#this method you cannot decrypt .gpg file after
#only his secret key can
gpg --encrypt --recipient myfriend@gmail.com myfile.txt
ls -al myfile.txt* #decide what to do with myfile.txt
#even better way, encrypt with yours and your friends secret keys
#this way you can both decrypt myfile.txt.gpg
gpg --encrypt --recipient myemail@domain.com --recipient myfriend@gmail.com myfile.txt
rm -f myfile.txt #send him myfile.txt.gpg
#or share same password between you both
gpg --symmetric myfile.txt
rm -f myfile.txt #send him myfile.txt.gpg

Decrypt file:

gpg -d myfile.txt.gpg > myfile.txt

Enjoy your new setup!

Dan.

FreeBSD 10 Ports and pkg2ng, how to deal with binaries and ports collection at same time

Intro:
FreeBSD has implemented a new packaging system called pkg2ng. While that is great for a number of reasons, like allowing commercial companies to now have their own remote repositories or even better faster update times, we still have 1 issue. We would like to also keep the most powerful feature of FreeBSD, the custom port build options we like on our custom ports build.

Ideally what we would like to do then, since FreeBSD now stuffs binary installs and ports collection installs in same SQLite database for pkg2ng, is take advantage of updating with binaries first, then update our custom port builds as well. A big danger we face, is if we just do a normal “pkg upgrade”, we could effectively loose all our custom options in certain ports collection builds. So let’s talk about a way that we can have best of both worlds.

Pre-requisites:
I assume you are running at least FreeBSD 10, and using pkg2ng for package management, and portmaster for port upgrades. Also let us take a custom realistic scenario, let say we want to run alpine with custom options for maildir patch support, also we want to install apache22-itk-mpm, with custom modules built into apache, as well we want to use the new mariadb instead of MySQL for database.

So for this scenario we are not going to want to let binaries touch anything building MySQL, as it will install real MySQL. We also do not want binaries upgrading our apache modules as it will try to install regular apache, so let’s begin.

pkg install apache22-itk-mpm mariadb55-client mariadb55-server
cd /usr/ports/mail/alpine; make install; pkg lock alpine
cd /usr/ports/www/mod_auth_mysql_another; make install; pkg lock ap22-mod_auth_mysql_another
cd /usr/ports/www/mod_geoip2; make install; pkg lock ap22-mod_geoip2
cd /usr/ports/www/mod_perl2; make install; pkg lock ap22-mod_perl2
cd /usr/ports/www/mod_rpaf2; make install; pkg lock ap22-mod_rpaf2
(enable apache module in make config below)
cd /usr/ports/lang/php55; make config;make install; pkg lock php55
pkg install php55-extensions
pkg install php55-mysql (this is fine as binary since it uses custom MySQL build)

OK Great, now let’s see how we can upgrade for now on.

pkg version -v (see what ports out of date)
pkg upgrade (upgrade binary ports first)
locked (alias locked='pkg info -ak|grep yes')( check locked ports to unlock for portmaster and update list to unlock/lock below )
(ADD ANY PACKAGES MISSING HERE FROM "locked" COMMAND - unlock ports)
pkg unlock alpine; pkg unlock p5-DBD-mysql; pkg unlock ap22-mod_auth_mysql_another; pkg unlock ap22-mod_geoip2; pkg unlock ap22-mod_perl2; pkg unlock ap22-mod_rpaf2; pkg unlock php55
portmaster -adG (upgrade ports)
(ADD ANY PACKAGES MISSING HERE FROM "locked" COMMAND - lock ports back)
pkg lock alpine; pkg lock p5-DBD-mysql; pkg lock ap22-mod_auth_mysql_another; pkg lock ap22-mod_geoip2; pkg lock ap22-mod_perl2; pkg lock ap22-mod_rpaf2; pkg lock php55
pkg clean

There we have it, now we can get fastest build times as possible, first upgrading binaries, and then our custom ports. Locking packages is what prevents “pkg upgrade” from doing bad things to our custom ports. Of course we need to unlock them after to run portmaster on them. The sad part in this is we have to keep track of what packages to lock and unlock all the time for upgrade purposes. I have opened this issue: https://github.com/freebsd/pkg/issues/744

So if developers get around to adding this new feature to pkg2ng, it could make our life simpler, for now any time you lock a package, just update your upgrade notes, I normally toss it into /etc/motd for easy access.

Till next time, happy FreeBSD 10!

Dan.

KVM – Adding Space To FreeBSD 10 zfs on root guest on a Centos 6.5 LVM host

Intro:
I decided to write this after I could not find any documentation on internet how to easily add space to a FreeBSD 10 guest that had zfs on root install. This should show you how to do it easily and quickly. It is important as we may need to add more space from our LVM to our FreeBSD guest at some point, and we need to know exactly how to do that.

Pre-requisites:
I assume you have a Centos host running KVM guests, as well as gdisk installed.

First thing we want to do is extend size of our guest

lvextend -L +10G /dev/vps/sunsaturn (add 10G to sunsaturn)
gdisk /dev/vps/sunsaturn (we let gdisk fix partition table or we will not be able to add new space)

Command (? for help): w
Warning! Secondary header is placed too early on the disk! Do you want to
correct this problem? (Y/N): Y
Have moved second header and partition table to correct location.”

Just save and exit to fix our partition tables.
Now let us run gdisk again to add the new space, since we have following:

gdisk -l /dev/vps/sunsaturn
Number  Start (sector)    End (sector)  Size       Code  Name
   1              34            1057   512.0 KiB   A501  gptboot0
   2            1058         8389665   4.0 GiB     A502  swap0
   3         8389666       335544286   156.0 GiB   A504  zfs0
virt-filesystems --long --parts --blkdevs -h -a /dev/vps/sunsaturn
Name       Type       MBR  Size  Parent
/dev/sda1  partition  -    512K  /dev/sda
/dev/sda2  partition  -    4.0G  /dev/sda
/dev/sda3  partition  -    156G  /dev/sda
/dev/sda   device     -    160G  -

Our last partition can easily be expanded just deleting last partition and add it back with the new space.

gdisk /dev/vps/sunsaturn (now delete last partition(3) and add it back, set code and name back to A504 and zfs0)
partprobe /dev/vps/sunsaturn
(if for any reason you cannot see the space just run:)
gdisk /dev/vps/sunsaturn (then hit "w")
partprobe /dev/vps/sunsaturn
(now you should be able to do above)

Alright we have expanded our LVM, now we need to restart the guest and enable new space within the guest.

virsh shutdown sunsaturn (actually wait till it is shutdown, till "virsh list" shows it gone)
virsh start sunsaturn

IN THE GUEST:

zpool status (find out device and put it below)
zpool online -e zroot vtbd0p3 (zfs will now grab the additional space)

That’s it, now you know how to add space on the fly to any FreeBSD guest with zfs on root, on the fly.

Dan.

Update: a few months after writing this I came across this article which does a good representation.