Easily create machines with FreeBSD and vm-bhyve. If you come from a world of using virsh with Linux, going this route will save you some massive headaches. You won’t have to reinstall your OS every new release as FreeBSD provides freebsd-update utility for that, as well as you won’t have to deal with anymore laggy virt-manager remote sessions, and easily be able to connect to console just as with virsh. Yeah let’s leave the Linux redhat/centos/rocky blah war behind us, and use something that will be solid and powerful on bare metal for a decade to come….
For this setup, what I have is 2 10gigabit ports, so I’ve already gone and created my LACP interface in /etc/rc.conf, then I created a bridge on top of that for the VMs. In Linux world that would be your bond0 interface and your br0 interface on top of that.
In FreeBSD its as simple as:
cloned_interfaces="lagg0 bridge0" ifconfig_br0="addm lagg0 up description services" ifconfig_bridge0_name="br0" ifconfig_br0_alias0="inet 192.168.0.1 netmask 255.255.255.0 mtu 9000" ifconfig_br0_ipv6="inet6 fc00:192:168:1::1/64 accept_rtadv auto_linklocal" ifconfig_lagg0="laggproto lacp laggport ix0 laggport ix1" ifconfig_ix0="up" ifconfig_ix1="up"
What if you have a 100 gigabit home lab? I’ll just take a second to drool, then basically say that would be same setup as above but you’d most likely be setting your MTU to 16k instead of 9k in your FreeBSD rc.conf and on your switch ports configured for LACP.
Or without 10 gigabit LACP following should work, use your own device name for ix0:
cloned_interfaces="bridge0" ifconfig_br0="addm ix0 up description services" ifconfig_bridge0_name="br0" ifconfig_br0_alias0="inet 192.168.0.1 netmask 255.255.255.0" ifconfig_br0_ipv6="inet6 fc00:192:168:1::1/64 accept_rtadv auto_linklocal" ifconfig_ix0="up"
Now we have a simple br0 interface for our VMs. If you not using LACP just remove the lagg0 stuff and addm your real interface instead to ifconfig_br0 line, the main point is we have our br0 device ready to go, don’t use MTU 9000 if you haven’t upgraded to 10 gigabit in your home lab 🙂 Using samba with this setup I get full 1 GB/s each way off NVME’s, and using SMB server multi channel I’ve gotten to around 1.3 GB/s each way to my Windows 11 desktop. Can’t complain it’s an older Dell r720 using clover to boot 3 NVMEs, if I had a newer server I’d definitely look into RDMA.
Next let’s make sure we have sysctl variables setup to make sure we can work with bhyve, so let’s open /etc/sysctl.conf and add the following:
#BHYVE net.link.tap.up_on_open=1 #BYHVE + PF nat net.inet.ip.forwarding=1 net.inet6.ip6.forwarding=1
Now enter: “service sysctl restart” on command line to apply the changes.
Now assuming your using your NVME drives for VMs, personally I used 3 1TB NVME drives in a stripe on zroot. Even though it is a raid0 it doesn’t matter, I have another server I bring online occasionally to back this one up, think of it as a raid 10 just across 2 machines 🙂
Alright let’s install what we need:
pkg install grub2-bhyve bhyve-firmware tmux vm-bhyve zfs create -o mountpoint=/vm zroot/vm
Now we will want to add VMs to startup automatically when we reboot so let’s add vm-bhyve to /etc/rc.conf:
vm_enable="YES" vm_dir="zfs:zroot/vm" #vm_list="vm1 vm2" #vm_delay="5"
We will leave last 2 lines commented out so we can decide what to start after install 🙂 Now let’s get vm-bhyve setup:
vm init cp /usr/local/share/examples/vm-bhyve/* /vm/.templates/ vm switch create -t manual -b br0 services vm set console=tmux
Next thing we want to do is edit our default template for creating new VMs. Personally i use pico which is an alias to “nano -w”. So “cd /vm/.templates; pico default.conf”
loader="uefi" graphics="yes" graphics_listen="192.168.0.1" graphics_port="5900" #If you want it to wait connecting with VNC uncomment next line #graphics_wait="yes" #Valid Options: 1920x1200,1920x1080,1600x1200,1600x900,1280x1024,1280x720,1024x768,800x600,640x480 #graphics_res="1920x1080" xhci_mouse="yes" #OPENBSD - uncomment hostbridge line and use .img not .iso from download site #https://wiki.freebsd.org/bhyve/OpenBSD - vm install openbsd install74.img #hostbridge="amd" #conservative 1 cpu socket for windows, they charge apparently for multiple sockets cpu=4 cpu_sockets=1 cpu_cores=4 memory=4G #Use e1000 for NetBSD or you can't set mtu to 9000 #network0_type="e1000" network0_type="virtio-net" #assign tap devices to manual switch we created (vm switch create -t manual -b br0 services) network0_switch="services" disk0_type="nvme" disk0_name="disk0.img" #windows virtio driver to enable virtio-net network drivers #https://github.com/virtio-win/virtio-win-pkg-scripts/blob/master/README.md #vm iso https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-virtio/virtio-win.iso #NetKVM->2k22->amd64 (for virtio ethernet driver, use device manager to update driver to this directory) #uncomment for windows #disk1_type="ahci-cd" #disk1_dev="custom" #disk1_name="/vm/.iso/virtio-win.iso" #windows expects the host to expose localtime by default, not UTC #uncomment for windows #utctime="no"
This gives me a generic template for creating VM’s with windows, Linux, FreeBSD, OpenBSD etc on my NVME drives.
So when we now create VM’s it will install this template to /vm/$VM_NAME directory along with 2 lines adding unique mac address and UUID. That’s important because we don’t want crazy things happening in our ARP tables 🙂 Mainly what is nice about that is for DHCPD. If you want to control what IP address your VM gets you can add that mac address to your /usr/local/etc/dhcpd.conf file and know what IP it will get if not hardcoding it. UUID is for dhcpd6.conf where you can try to hardcode IPV6 address for host using that. Since all OS’s don’t really respect it, I normally leave it alone.
May be wondering why I use raw images instead of zvols? Because raw images have better performance read here: https://klarasystems.com/articles/virtualization-showdown-freebsd-bhyve-linux-kvm/
No worries, I’ve Crystal Mark tested a windows VM, and it would definitely blow away Linux KVM on disk speed.
So at this point we are done our setup, now on to creating VM’s.
So let’s take an example, I want to create a VM called asterisk, another FreeBSD 14 machine here we go:
vm iso http://blah.iso vm create -s 50G asterisk #add this new VM IP address to /etc/hosts pico /etc/hosts cd /vm/asterisk cat asterisk.conf #look for mac address and add to DHCPD pico /usr/local/etc/dhcpd.conf service isc-dhcpd restart pico asterisk.conf
So what we doing here is telling vm-bhyve to fetch iso for us and put it in /vm/.iso. Then we tell it to create a 50G image called disk0.img in /vm/asterisk directory, an asterisk.conf from our /vm/.templates/default.conf file with 2 unique lines added for mac address and UUID. Pretty sweet. Don’t over think it here with disk0.img, all its doing is a simple “truncate -s +50G disk0.img” to create a blank image.
I then go on to add IP address I picked out to /etc/hosts and /usr/local/etc/dhcpd.conf but you can skip this if you like, but it’s a useful habit to get into for installing things like say home assistant VMs.
Now final step is edit asterisk.conf to your liking. Normally only thing I change per VM is graphics port and I may use virtio CDROM for windows. I’ve successfully installed FreeBSD, Ubuntu, Rocky Linux, OpenBSD and windows server 2022 using this template so it’s a pretty good one 🙂 Also since we using NVME makes windows install less painful as we won’t need any windows drivers for hard drive, think you might for network card still, why I generally leave cdrom attached in windows. For graphics_port I normally set that to last digits of VMs IP address, so like say I set asterisk to use 192.168.0.3, then my graphics_port will be 5903. I then add that host to my SecureCRT list, and add the VNC to my mobaxterm list 🙂
Honestly it’s more steps, but at end of day when VM’s are all running, now its one simple click on ssh session list or VNC session list to get to VM, so nice habit to get into.
Now let’s start the install. Normally when you first start trying to install an OS you want CDROM or USB stick in to install it, then remove it as host reboots. vm-bhyve does something amazing here, does it for you 🙂
vm install asterisk FreeBSD-14.0-RELEASE-amd64-bootonly.iso
That’s it! It will take ISO we downloaded to /vm/.iso/blah.ISO , add it to VM for install. At this point what I normally do is either “vm list; vm console asterisk” or just use mobaxterm to connect to graphics port. Tmux is nice addition to serial port of host 🙂 But normally to be able to use that you have to configure each host to use the serial port. Like adding a line at end of grub on Linux or just editing FreeBSD /boot/loader.conf.
If you want to wipe a VM out and start over you have 2 options:
vm destroy asterisk OR zfs destroy -r zroot/vm/asterisk #-r because we'd have to killoff snapshots to if they exist
Your all done. Congratulations.
Now what I normally like to do with a new VM is make sure it shuts down from host when I want it to, so I may issue a “vm stop asterisk” and keep watching VM and “vm list” to see it has ended. When I’m happy, I’ll “vm start asterisk”, then add asterisk to /etc/rc.conf as something to start on boot for example.
What if you screwed up install and say installed asterisk here with 10G less that you wanted? Easy enough, on the host just do the following:
vm stop asterisk cd /vm/asterisk truncate -s +10G disk0.img mdconfig disk0.img gpart recover md0 #recover disk after adding space gpart show md0 #find partition number with freebsd-zfs in it, in my case 3 gpart resize -i3 md0 #expand partition freebsd-zfs partition 3, could also be 4 if you didn't install with GPT(EFI) mdconfig -du md0 #unmount md0
But how do you get the guest to see the new space? Do following:
vm start asterisk #login to guest and: zpool status (look for device) zpool online -e zroot /dev/nda0p3 #use your device from command above
That’s it. But what if you screwed up something on guest in /boot/loader.conf or /etc/rc.conf that prevents you from booting? Simple let’s mount it and fix it on host:
vm stop asterisk #make sure guest is stopped or risk data corruption! cd /vm/asterisk mdconfig disk0.img mkdir mnt zpool import #you should see md0 zroot disk to import zpool import -fR /vm/asterisk/mnt -t zroot testing zfs mount -o mountpoint=/vm/asterisk/mnt testing/ROOT/default #cd /mnt/etc or /mnt/boot etc and fix stuff.... zpool export testing rm -rf mnt mdconfig -du md0 #unmount guest
That’s it, then just “vm start asterisk” and problem solved 🙂 Even simpler for Linux obviously, just look for partition and mount it, but same would apply for a debian zfs root install.
Now I bet someone is wondering what if that was an ubuntu install, and I screwed up grub, how would I recover that? Well simple, “vm poweroff ubuntu”. “vm install ubuntu ubuntu-22.04.3-live-server-amd64.iso” to load the install ISO just so we can use it to drop to a shell. Connect with VNC viewer, go to help -> shell. Now assuming you installed with all default options with LVM and EFI then here is how to fix grub:
fdisk -l lvscan mount /dev/ubuntu-vg/ubuntu-lv /mnt mount /dev/nvme0n1p2 /mnt/boot mount /dev/nvme0n1p1 /mnt/boot/efi mount --bind /dev /mnt/dev mount --bind /proc /mnt/proc mount --bind /sys /mnt/sys chroot /mnt nano /etc/default/grub (fix it) grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=GR update-grub umount /mnt/dev umount /mnt/proc umount /mnt/sys umount /mnt/boot/efi umount /mnt/boot umount /mnt sync reboot
Again all things you can do right from FreeBSD host. And again someone will probably ask I tried to install Kali Linux and after install it won’t boot how do i fix it? Well I thought this guy did a great job of explaining it:
https://record99.blogspot.com/2021/12/bdsdex-failed-to-load-boot0001-uefi-bhyve-sata-disk.html
So let’s just modify that for Kali Linux:
1. Enter exit get out of the UEFI Shell (you will come to BHYVE panel) 2. Choose Boot Maintenance Manager 3. (In the Boot Maintenace Manager ) Choose Boot From File 4. (In File Explorer) NO VOLUME LABEL, [PciRoot(0x0)/ Pci()] press Enter 5. (Still in the File Explorer) Choose <EFI> 6. (Still in the File Explorer) Choose debian 7. (Still in the File Explorer) Choose grubx64.efi 8. Then you will get into the system How can you fix it: 1) Get access inside your Kali Guest VM. 2) as root: cd /boot/efi/EFI 3) mkdir BOOT 4) cp kali/grubx64.efi BOOT/bootx64.efi 5) reboot Basically you need to have this file in this path: /boot/efi/EFI/BOOT/bootx64.efi
And now someone is going to ask how did I get windows server 2022 to work. I downloaded the windows_server_2022.iso and virtio-win.iso to my /vm/.iso directory and edited my template as follows:
loader="uefi" graphics="yes" graphics_listen="192.168.0.1" graphics_port="5907" #enable this briefly on windows installs, install won't do anything till you connect with VNC #graphics_wait="yes" #Valid Options: 1920x1200,1920x1080,1600x1200,1600x900,1280x1024,1280x720,1024x768,800x600,640x480 graphics_res="1920x1080" xhci_mouse="yes" #conservative 1 cpu socket for windows, they charge apparently for multiple sockets cpu=4 cpu_sockets=1 cpu_cores=4 memory=4G network0_type="virtio-net" #assign tap devices to manual switch we created (vm switch create -t manual -b br0 services) network0_switch="services" disk0_type="nvme" disk0_name="disk0.img" #windows virtio driver to enable virtio-net network drivers #uncomment for windows disk1_type="ahci-cd" disk1_dev="custom" disk1_name="/vm/.iso/virtio-win.iso" # windows expects the host to expose localtime by default, not UTC #uncomment for windows utctime="no" uuid="6efefb12-8da5-11ee-906d-e4434bf65f00" network0_mac="58:9c:fc:0f:b8:a6"
Then I did a simple “vm install windows windows_server_2022.iso” and installed it. When it was installed I went to device manager, selected the ethernet adapter that obviously wouldn’t work, then pointed it to look on the virtio-win.iso drive at “NetKVM->2k22->amd64”. About it. Thought I’d throw this in there as I’m sure there is somebody that likes to setup active directory to authenticate everything. But honestly look at passkeys, it’s the new thing apparently, might as well learn it now, search for webauthn.
Ok we are now experts, so let’s install Home Assistant VM then to make sure 🙂 So home assistant only provides an image and not a ISO, so that’s pretty simple to, all we really need is a raw image, so I see they provide a qcow2 image on their site, so we can just convert that easily after downloading it, and for good measure let’s break root on it to after install 🙂
vm create -s 10G ha vm iso https://github.com/home-assistant/operating-system/releases/download/11.1/haos_ova-11.1.qcow2.xz cd /vm/.iso unxz haos_ova-11.1.qcow2.xz pkg install qemu-tools gdisk qemu-img convert -f qcow2 -O raw haos_ova-11.1.qcow2 haos_ova-11.1.img rm haos_ova-11.1.qcow2 gdisk -l ./haos_ova-11.1.img cp /vm/.iso/haos_ova-11.1.img /vm/ha/disk0.img
Ok what I’ve done here is download the image, go to ISO/image directory and uncompress it. I install some tools to convert image to raw image type we need for BHYVE. Delete old format to save disk space, then run gdisk on image to see all its partitions to know we have a working OS on this image. Could have used gpart I suppose as well but nice to have a command that works on FreeBSD and Linux.
I then just copy image over the one vm-bhyve added, so I could have even done 1G instead of 10G creation or whatever, it didn’t matter.
Alright, remember I said have good habits with VMs, at this point cd /vm/ha and look at ha.conf for any edits, and set an IP for this home assistant VM in /etc/hosts and toss it’s IP and mac address in /usr/local/etc/dhcpd.conf. “service isc-dhcpd restart”. I assigned it 192.168.0.15, so I’m pretty sure when it boots its going to call on a DHCPD server for an IP.
vm start ha vm console ha
Well I typed in “root” and hit enter and I have root access, so much for trying to mount all the partitions and break it 🙂 Also went to VNC port and at ha> prompt just typed in “login” and got root to. Doing a cat /etc/shadow it seems root has no password to login. Running a “ps aux” I see dropbear running instead of sshd, like an android system, yet this is a x86_64 image, unreal.
Alright well this is still inconvenient, rather add it to my ssh list to with root access, since they just let you blindly login with root on console like this, they probably have devel docs somewhere how to do this.
YEP, check it out here: https://developers.home-assistant.io/docs/operating-system/debugging/
Just says add your pub ssh key and your good to go with ssh root@homeassistant.local -p 22222. Fair enough let’s go back to our tmux console and add it then:
cd /root/.ssh vi authorized_keys chmod 600 authorized_keys
Well they didn’t have nano installed, but most OS’s include vi, so was as easy as just opening a file, making sure its permissions was ok and good to go, can add it to your ssh session list now 🙂
I must admit I now played with it for a bit, and quite like this, added all my smart devices to it, downloaded the app to my phone, guess all I need is a microphone/speaker so can control all my lights etc if internet goes out 🙂
Anyways, this really is how simple using vm-bhyve is, vm list, vm iso/create/stop/start etc. I honestly like it, and will continue to use it, reminds me of using virsh with libvirtools on Linux. Tmux console is great, CTRL-a-s to switch between VM’s quite handy. I was only used to using screen before tmux, so I remapped CTRL-a-d to work on tmux to:
pico /usr/local/etc/tmux.conf #and added: # remap prefix to Control + a set -g prefix C-a # bind 'C-a C-a' to type 'C-a' bind C-a send-prefix unbind C-b
Great works like screen, now I won’t forget 🙂 tmux is pretty good, liking it better these days than screen, but I still keep screen and minicom around because tmux can’t connect ports with baud rates, parity etc.
Anyways I hope this has been a great introduction to working with VMs under FreeBSD BHYVE with vm-bhyve, vm-bhyve adds some powerful features and makes it easy for us. FreeBSD gives us the power of its ports collection, most stable ZFS OS, and powerful networking features like /etc/pf.conf and working with setfibs(its like rt_tables on linux but more customizable) 🙂
Now go setup your zfs snapshots for your VMs in crontab: “crontab -e”
0 3 * * * (zfs destroy -r zroot/vm@`/bin/date +\%A`; zfs snapshot -r zroot/vm@`/bin/date +\%A`) > /dev/null 2>&1 0 3 1 * * (zfs destroy -r zroot/vm@`/bin/date +\%B`; zfs snapshot -r zroot/vm@`/bin/date +\%B`) > /dev/null 2>&1
What this does is first line takes a snapshot daily, Mon-Sunday, so during a 7 day period you can rollback, and it will delete old ones to save disk space. The “-r” just says do it for each VM. I really only had to escape the “%” sign for cron about it. Second line does a monthly snapshot, so you’ll get 12 over course of a year, and they’ll delete each other to save disk space and keep things organized, simple and sweet…
If you need to rollback a VM I suggest you setup an alias to see your snapshots in your .bash_profile or whatever shell you use:
alias snapshot='zfs list -t snapshot'
Then you can just rollback to whatever you see in list:
snapshot zfs rollback zroot/vm/asterisk@Monday
That’s the power of ZFS on a virtual machine 🙂 Now go add your new VM to your vm list in /etc/rc.conf if you want it to startup when host reboots. Good to go, now can rollback a VM if you screw it up 🙂
And now someone is going to ask how to I get vm console serial access to all OS’s because you’d don’t like using VNC viewer. Well first of all for windows obviously your going to have to use VNC viewer, for the rest, let’s start with FreeBSD, add to your /boot/loader.conf
autoboot_delay="5" comconsole_speed="115200" boot_multicons="YES" boot_serial="YES" console="efi"
Then just reboot, and use vm console to watch it. If you have resizing issues with tmux just type “resize” and you’ll be fine, if you don’t have the command install xterm on guest. Should be installing xterm and xauth on every guest. Now what about ubuntu/kali? Edit /etc/default/grub as such:
-GRUB_TIMEOUT_STYLE=hidden +#GRUB_TIMEOUT_STYLE=hidden # Optional kernel options that you very likely want. Don't affect GRUB itself. # Remove quiet to show the boot logs on terminal. Otherwise, you see just init onwards. # Add console=ttyS0, or else kernel does not output anything to terminal. -GRUB_CMDLINE_LINUX_DEFAULT="quiet splash" -GRUB_CMDLINE_LINUX="" +GRUB_CMDLINE_LINUX_DEFAULT="" +GRUB_CMDLINE_LINUX="console=tty1 console=ttyS0,115200" # Show grub on both serial and on display. -#GRUB_TERMINAL=console +GRUB_TERMINAL="console serial"
Then just type “update-grub” and your good to go.
What about centos/redhat/rocky linux, you installed the newest 9.3 as of today? Well since all that is pretty much redhat look at following url:
https://access.redhat.com/articles/3166931#config9
So basically just do:
grubby --update-kernel=ALL --args="console=tty0 console=ttyS0,115200" grubby --info=ALL|grep -i args
What about OpenBSD?
openbsd:~ # cat /etc/boot.conf stty com0 115200 set tty com0 openbsd:~ #
What about NetBSD?
echo "consdev=com0,115200" >> /boot.cfg
Other helpful hints? Just remember the “resize” command when working with vm console VMNAME. Tmux is great at screwing up your terminal. Other than that, I have this added to my .bash_profile to remind me to make sure to setup X11 forwarding and have that resize command work:
if ! command -v xterm >/dev/null 2>&1
then
echo "WARNING: xterm not installed, resize command may not exist for tmux!"
fi
if ! command -v xauth >/dev/null 2>&1
then
echo "WARNING: xauth not installed, X11 forwarding will not work, enable it in sshd.conf as well"
fi
Where to go from here? If its just a home lab maybe you should authenticate a user to login to every VM in one spot, maybe active directory, maybe kerberos, or just keep it simple and use NFS and NIS. I’ve tested NFS and NIS across all OS’s except windows obviously, only issue is with Redhat/Rocky/Centos having removed all NIS stuff calling it insecure.
Well that’s true and not true, if someone was sitting on my network with tcpdump or wireshark, for sure it’s possible to eventually sniff out encrypted hash of password. Personally I blame the old /etc/shadow /etc/master.passwd format. If they would just upgrade that to using public and private keys they could sniff all they wanted and take 10 years trying to crack it then. It is what it is, but for a home lab, NIS is just fine, at least you can use yppasswd to update your password across all VMs and your good to go…
Other fun things to try with FreeBSD as host? Get yourself a Starlink dish on sale, install it, and using /etc/pf.conf and setfibs make certain IPs in your home lab go out Starlink and others your default internet connection. After free month trial cancel it. Then if your internet ever goes down you have a layer of redundancy to just enable Starlink on your phone 🙂
Or play with wireguard, then using /etc/pf.conf and setfibs, make certain IPs in your home lab go out VPN, and others your default connection. Maybe start a VPN service afterwards.
Play with samba and NFS, export them to your VLC app on your google TV chromecast in living room. Configure utorrent to download to the samba directory that you have mapped as network drive on windows.
Play with snapshots, take snapshots of everything so you can rollback. You have power of ZFS on main virtualization host, take advantage of it 🙂
Install a rsyncd.conf on every VM, using crontab from host backup important configuration files on each VM once a night.
Install a windows server 2022 VM that you configure to start first in /etc/rc.conf. Watch some youtube videos on active directory, configure some VMs to authenticate off that for fun. Maybe play with passkeys instead.
Write some HOWTO’s on internet how to increase VM performance even more, I’d love to read it!
Setup a real dedicated server online, use same setup here. Install VMs for a webserver, DNS server, mail server etc. Try to run as few services on main host as you can other than DNS , wireguard and ssh. Use one central /etc/pf.conf to firewall for host and every VM(be very careful unless your provider can give you console access to machine through a KVM etc, take it from experience 🙂 ) If not place firewall rules and revert them in 60 seconds something like: “pfctl -F all; pfctl -f /etc/pf.conf; sleep 60; pfctl -F all; pfctl -f /etc/pf.conf.old”. I’d really recommend testing on your home lab first 🙂 Try to install all services you can on a FreeBSD vm, for anything else can use an ubuntu VM for development. Setup a wireguard tunnel to back this server up once a night using rsync or zfs send/receive. For advanced users maybe rotate wireguard keys once in awhile automated. The habit you want to get into is you are rarely logging into main host for anything other than security updates, your mostly using the VM’s to do everything. That way if you screw anything up you can simply do a “vm console VM” on host. Let the worker bees be children, and the parent parenting 🙂
Maybe you’d like some custom FreeBSD kernel tuning, you can recompile the whole world and add features as you like. I’ve done this on numerous occasions to do custom things. I should note do not do this on production servers online unless you know what you are doing, cause it can take you that much longer to apply a security update, I almost feel sorry for NetFlix in that regard as they run all custom FreeBSD current custom servers(but not that sorry, the hardware they run on is very powerful so would take them almost no time 🙂 ) If you decide to do this compile everything on main host, nfs mount /usr/src and /usr/obj on all FreeBSD VMs and do a quick install on those to.
Maybe play with crypto, maybe learn GPU passthrough on vm-bhyve, possibilities are endless, you are now in complete control….
Happy BHYVE’ing…..
SunSaturn