All posts by SunSaturn

Using Home Assistant Kasa Matter Plugs and a Victron MPPT VEDirect cable to deplete your batteries for next day — Updated Version

Batteries, kasa, LifePo4, plug, VEDirect, victron

In my last version I used home assistant to connect to KASA matter plugs, I have found nothing but problems with matter, especially kp125M plugs, that use matter. One day I was turning on and off grid breakers and then I had to delete devices from google and home assistant because they could not connect using matter anymore! In this version we will use non-matter KASA smart plugs and skip home assistant. You can review what we did before here: https://blog.sunsaturn.com/freebsd/using-home-assistant-kasa-matter-plugs-and-a-victron-mppt-vedirect-cable-to-deplete-your-batteries-for-next-day/

In this version since we used 2 different scripts accessing a lot of same functions, makes no sense to keep it that way, so let’s stick our functions for KASA plugs and Victron’s VEDIRECT in a separate class file, and remake our 2 scripts for our cronjob.

Let’s start by creating a file called victronclass.py to hold our 2 functions, and add following to it:

import vedirect
from kasa import SmartPlug

class VictronApp:
    def __init__(self):
        pass

    async def victron(self, serial_port, serial_baud):
        try:
            device = vedirect.VEDirect(serial_port, serial_baud)
        except Exception as e:
            print(f"VICTRON [FAIL] : Read bad data: {e}")
            return False
        try:
            float(device.battery_volts)
        except Exception as e:
            print(f"VICTRON [FAIL] : Return type was not a float: {e}")
            return False
        if not device.battery_volts > 0:
            print(f"VICTRON [FAIL] : Voltage not greater than 0!")
            return False
        print(f"VICTRON [SUCCESS] : Voltage is {device.battery_volts}")
        return device.battery_volts


    # Usage: status = await kasa_plug(IPADDRESS, "on/off/check")
    # Returns True if plug turned on or off or if "check" is used returns True if plug is on
    async def kasa_plug(self, ip, cmd):
        p = SmartPlug(ip)
        try:
            await p.update()
            if cmd == "check":
                if p.is_on:
                    print("KASA [CHECK] : Plug is on")
                    return True
                else:
                    print("KASA [CHECK] : Plug is off")
                    return False
        except Exception as e:
            print(f"KASA [FAIL] : Exception received connecting: {e}")
            return False
        if cmd == "on":
            if p.is_on:
                print("KASA [SUCCESS] : Plug already on doing nothing")
                return True
            else:
                try:
                    await p.turn_on()
                    await p.update()
                    if p.is_on:
                        print("KASA [SUCCESS] : Turned Plug on")
                    else:
                        print("KASA [FAIL] : Plug still off after turning it on")
                    return p.is_on
                except Exception as e:
                    print(f"KASA [FAIL] : Exception Received Turning on Plug: {e}")
                    return False
        elif cmd == "off":
            if p.is_off:
                print("KASA [SUCCESS] : Plug already off doing nothing")
                return True
            else:
                try:
                    await p.turn_off()
                    await p.update()
                    if p.is_off:
                        print("KASA [SUCCESS] : Turned Plug off")
                    else:
                        print("KASA [FAIL] : Plug still on after turning it off")
                    return p.is_off
                except Exception as e:
                    print(f"KASA [FAIL] : Exception Received Turning off Plug: {e}")
                    return False
        else:
            print("KASA [FAIL] : INVALID ARGUMENT: use [on] [off] or [check]")
            return False

We created 1 class to hold both our KASA and victron vedirect functions, that way we can reuse these in our 2 main files. Let’s create our first main file for our cronjob and call it victron.py:

import asyncio
from victronclass import VictronApp

async def main():
    # Config START
    # VICTRON
    # trying 25.9(35% spring getting 4KWh per day atm)
    voltage_max = 25.9          # we don't want voltage higher than this for morning solar charging--leaving it at 50% each night
    # https://batteryfinds.com/lifepo4-voltage-chart-3-2v-12v-24v-48v/
    serial_port = '/dev/cuaU0'  # serial_port for VEDIRECT, on linux try "setserial -g /dev/ttyUSB[01]" to find your serial port
    serial_baud = '19200'       # serial baud rate, 19200 seems recommended setting from Victron
    # KASA PLUG
    # kasa_ip = '192.168.0.x'    # Ip address of your Kasa plug
    kasa_ip = 'grid_plug'       # Ip address or /etc/hosts name of your Kasa plug
    # Config END

    v = VictronApp()
    # check voltage of battery
    volts = await v.victron(serial_port, serial_baud)  
    if not volts:
        print(f"ERROR: no volts, exiting program....")
        exit(1)
    kasa = await v.kasa_plug(ip=kasa_ip, cmd="check")

    if kasa:  # if plug is on
        # plug is on, check it we need to turn it off
        if volts > voltage_max:
            print(f"Voltage is greater than {voltage_max} Turning off Grid Plug")
            await v.kasa_plug(ip=kasa_ip, cmd="off")
        else:
            print(f"Grid Plug is on and voltage less than {voltage_max} DOING NOTHING")
    else:
        # plug is off, check it we need to turn it on
        if volts < voltage_max:
            print(f"Voltage is less than {voltage_max} Turning on Grid Plug")
            await v.kasa_plug(ip=kasa_ip, cmd="on")
        else:
            print(f"Grid Plug is off and voltage greater than {voltage_max} DOING NOTHING")

asyncio.run(main())

Now you can add this to cron as follows:

*/10 23 * * * (/root/cronjobs/victron.py) > /dev/null 2>&1
*/10 0-6 * * * (/root/cronjobs/victron.py) > /dev/null 2>&1

This should allow you to depleted your batteries between 11pm and 6:50am each day.

Now let’s create our program to turn on the grid KASA plug no matter what at 7am, let’s call it grid_on.py:

import asyncio
from victronclass import VictronApp

async def main():
    # Config START
    # kasa_ip = '192.168.0.x'    # Ip address of your Kasa plug
    kasa_ip = 'grid_plug'       # Ip address or /etc/hosts name of your Kasa plug

    # Config END

    v = VictronApp()
    kasa = await v.kasa_plug(ip=kasa_ip, cmd="on")

asyncio.run(main())

and add that to your crontab as follows:

0 7 * * * (/root/cronjobs/grid_on.py) > /dev/null 2>&1

Great your all set with new improved version not using home assistant for KASA anymore. After reading up on matter protocol I am not to fond of it. Not only does it use IPV6, it takes a IPV4 address on your home network if it becomes a border router. Most devices try to become a border router, in fact different companies fight over fact they should become it, then you got all these devices that are border routers.

So to make more of a headache for you , now you have to track those down and deal with them, whether you VLAN them off or just block them completely on your firewall.

I think what is safer is just have 1 hub communicating with a bunch of zigbee devices, at least zigbee devices don’t suddenly attempt to become border routers on your network and do as they please. Like phillip hues lights for instance, you just have one hub to deal with if it’s misbehaving in anyway, not have to deal with every single light trying to become a border router lol. For smart plugs you definitely want full control at all times with a definite IP address. Things like light bulbs should never be matter devices attempting to become border routers. If someone is going to hack your home network , they shouldn’t be able to plant a backdoor on your light bulb that became a border router, they should need to be in range, not in China!

Ultimate security is keep everything zigbee etc, and use your own server to control all the zigbee devices on home assistant, then you don’t even have to worry about a phillips hue hub etc doing bad things on your network. Or if your crazy security conscious I guess look for a smart plug that is zigbee/bluetooth etc controllable.

So probably your best bet with KASA is their kp125 without the “M” on the end. This way you get energy stats as well turn plugs on and off as you please. Other versions like their ep25 etc stay far away from, if you loose internet at your house, they require authentication to cloud to use them. WHY ON EARTH? No idea, great right, internet goes out and now you lost control over your devices! Unreal.

I mean this is whole reason people use Home Assistant etc in first place. If internet goes out, telling google to turn on a simple light is replied with, “Sorry I don’t care about you anymore your internet is down blah blah”….. let’s hope it’s not someone’s pacemaker one day!

As for more revisions to this program, if I get my hands on a Cerbo GX one day, I’ll probably update for that using TCP instead of serial for vedirect. Or I may update it to be just one long running program at some point that accommodates for winter/spring/summer/fall, as well as maybe weather conditions. I mean you can get like 4KW a day on 4 panels or more when sun in shining but in winter and cloudy days you get almost nothing, so after I get a year’s history of my solar I’ll have a better idea how to proceed.

Keep in mind you could always use a hybrid inverter for switching, although I don’t think it would support depleting your batteries to a certain voltage at night lol. Also honestly they all suck, their switching times are horrendous, like 10-20ms, that would knock out my old servers, and any old electronics instantly. Better is a straight up inverter and use a real PDU like a Tripplite on ebay. You can get them fairly cheap these days, and you get sub 2ms switching times, then your golden…

Till next time…

SunSaturn

Using WSL as regular Linux host on same network with 10 gigabit and NFS

linux, NFS, wsl

What if you could use WSL as a regular Linux host on same network? For developers this would be big right? What about app developers that test apps on Windows, Linux, Android, Web etc. You could do it all from one place, and have convenience of mounting your NAS on your WSL machine with NFS to backup your WSL files you been working on, whether you working with flutter, flet, or whatever your preference is. Or maybe just you like to ssh into your WSL machine to run Unix commands on your windows desktop from your laptop etc…

Alright let’s assume we are a developer and not a hacker, we will want latest Ubuntu LTS release(as developers always test with this distro more than anything else), but feel free to install Kali instead if that’s your thing…

Let’s do it, what we will accomplish here is install Ubuntu-22.04 WLS2, we will start this instance when we login to windows in background so we can ssh into it, we will also mount NFS directory from our NAS so you can just copy anything you want to backup to it easily. So let’s say I do a lot of work in my /home/dan directory, I will mount NAS to /data directory and anytime I want to backup anything, I’ll copy it there. We could go more advanced and just mount our /home/dan directory from a FreeBSD zfs pool and revert snapshots to anytime we want 🙂 , but we will keep it simple…

On security end of things, you should only NFS mount anything on WSL that you normally map with samba from your NAS. If ever get malicious code on your windows computer, you don’t want to open your whole NAS to it. Worst case scenario you loose 2 days of your life reinstalling windows again with backups from private NAS backups and rolling back snapshots on your ZFS system.

Notes on backups

I personally have a 2nd server I leave offline and only bring it online occasionally to backup first NAS, blame SSD corporate greed, spinning drives cheaper but can only spin 3-6 years before they fail, so this ensures I don’t have 2 drives fail at same time on each if they don’t rust first lol. NVME’s are still only mainly used as OS drives, quite sad really. When I used to work for fortune 500 companies they used to backup regularly to tape drives, and every few months send that backup offsite as well. If their colocation caught fire, they didn’t loose everything this way. So while it’s a good idea to do an onsite solution like I have, it’s also a good idea to rent a VPS/dedicated/colocated server and backup stuff there occasionally as well.

In the 90s I use used to program in C etc, I kicked my PC to many times and the drive failed, lost a whole year worth of C programming libraries I made, never touched it again, went into perl/php/python after that. Another example, one backup server one time I forgot to create ZFS snapshots on it, so one day when first server had drive fail, I replaced it with a blank slate, the backup server then rsync’d an empty drive one night before I got to it and lost almost everything. So moral of story is, a) snapshots b) 2nd NAS to backup first c) copy things you want to keep offsite in case of a fire. You decide, but I know how long apps take to develop when your new at it, I’d hate to see you give up on a project because you didn’t use github or a VPS.

Notes on FreeBSD

I can’t stress enough how your servers should be running FreeBSD as main host. By utilizing BHYVE virtualization your virtual hosts will be rock solid, and the ports collection is enormous, it is essentially a sys admin’s dream. See my previous posts on running FreeBSD with vm-byve. Not just the fact you have more packages available to you than any other OS on the planet, the main reason is ZFS. Linus Torvalds already stated he won’t allow ZFS in Linux kernel, and always people hacking it in there on their Linux distros is a real waste of time, for backing up data, there is no better OS. At home my FreeBSD servers do everything, wireguard, DNS, DHCPD, backups, snapshots, virtual hosts, 10 gigabit fiber, you name it, it’s something you can’t do with prebuilt solutions like FreeNAS, proxmox, etc taking the control away from you. Take control back and you won’t regret it. I even demonstrated how to run home assistant with it and not needing to buy a raspberry pie with inferior hardware 🙂

Getting started with WSL prerequisites

First thing we need is a bridge, this will allow us to be on same network as our regular LAN for dhcpd etc… If you don’t have 10 gigabit skip #2, replace your username dan with whatever your C:\Users\$username is…

1) create a bridge:
search for hyper-v, should see something to enable you to add that addons to windows, add then restart PC.
Now open Hyper-V manager, select <your PC name>, and select "virtual switch manager"
Now create a new virtual switch that is "external" and call it "WSL Bridge"
Check it has our external network 10GB card etc.
2) enable jumbo packet on new virtual switch
control panel -> Network and Sharing Center -> "vEthernet (WSL Bridge)" -> properties -> configure -> advanced
set Jumbo Packet to "9014"

Now we need to edit the main wsl file to use this bridge, again replace dan with your username:

3) In WSL:
nano /mnt/c/Users/dan/.wslconfig OR just do it from your C:\home\Users\dan\.wslconfig windows file
ADD:
[wsl2]
vmIdleTimeout=-1
networkingMode = bridged
vmSwitch = "WSL Bridge"
ipv6=true

PLEASE NOTE: could not get X11 forwarding working without ipv6 enabled
the reason is you would have to change in /etc/ssh/sshd_config "AddressFamily any" to "AddressFamily inet"

Let’s actually install WSL

wsl --list --online
wsl --install -d Ubuntu-22.04

Now let’s create a .bat file that runs when we login to start WSL automatically, in your C:\Users\$USER directory create a file called “WSL_start.bat” and let’s add to it:

@ECHO OFF
:: Start WSL in backgound
TITLE Starting WSL

:: Section 1: Starting WSL in background
set OS=Ubuntu-22.04
ECHO ==========================
ECHO Starting WSL %OS% in background, please wait...
ECHO ============================

wsl -d %OS% --exec dbus-launch true

Close the file remembering to replace OS= with whatever distribution you want to start. You may say why not just use that one wsl line to do it instead, well when you reboot your PC and login for first time, what you would have is just a blank terminal screen executing something and you would have no idea what it is, this way it is informative, and you always know what it is, especially if you don’t have a superfast computer PCIE 5 PC, I’m on PCIE 4, and it’s around seconds.

Now let’s add that .bat file to “Task Scheduler”. Search for it and open it.

Go to “create task”. In “General” make sure “Run only when user is logged on” is checked. In “Triggers” select “Specific User” should be yourself. And on “Begin the task” select “At log on”. In “Actions” click “browse” and point to .bat file we created.

And that’s it we have a task created to start our WSL anytime we login, perfect.

Let’s start WSL

Open powershell and just type WSL and hit <tab> should autocomplete for you and hit enter:

PS C:\Users\dan> .\WSL_start.bat
==========================
Starting WSL Ubuntu-22.04 in background, please wait...
============================
PS C:\Users\dan>

Great now just login with “wsl” or “bash”. You can now check with “ifconfig -a” or “ip a” that your on same subnet as your regular LAN. 

sudo su
apt install openssh-server
apt install  nfs-common
systemctl enable --now ssh

You should now be able to login with ssh, good to go. Go ahead and restart WSL and make sure everything working alright:
wsl --shutdown
.\WSL_start.bat
wsl
nano /etc/wsl.conf
#ADD following:
[network]
hostname = wsl
generateHosts = false

What we are doing here is setting our hostname and saying don’t overwrite /etc/hosts, that way you can add your IP to it etc without it getting overwritten. There is another option you can add to not overwrite /etc/resolv.conf, personally I feed it info with my DHCPD server so I like leaving if but it is “generateResolvConf = false”. If you set this option you need to restart WSL and it will completely nuke /etc/resolv.conf first time, so best to make a copy before rebooting then copy it back. Doing this however is only reliable way then to use hostnames in your /etc/fstab instead of IP address. Personally I feed the host it’s static IP address, MTU, DNS etc with my DHCPD server using it’s mac address so it’s good for me, if you want an example from my dhcpd.conf on FreeBSD I have following with isc-dhcpd currently for WSL:

subnet 192.168.0.0 netmask 255.255.255.0 {
       default-lease-time 259200;
       max-lease-time 432000;
       option broadcast-address 192.168.0.255;
       option domain-name "sunsaturn.com";
       option routers 192.168.0.1;
       range 192.168.0.150 192.168.0.200;
       option domain-name-servers 192.168.0.1, 8.8.8.8;
}

host wsl {
  hardware ethernet 5e:bb:f6:9e:ee:fa;
  fixed-address 192.168.0.19;
  option interface-mtu 9000;
}

This is a more convenient way to do it, that way you control WSL and everything on your network from one file. Obviously remove MTU line if you don't have 10 gigabit :) If you want to get more fancy, also edit your /etc/hosts , and also your reverse DNS on bind and add it as well, but for most part this will suffice. I'm generally lazy editing bind reverse file, dhcpd.conf and /etc/hosts on main host is fine.

This is what my /etc/hosts file looks like on wsl:

::1             localhost localhost.sunsaturn.com
127.0.0.1       localhost localhost.sunsaturn.com
192.168.0.19    wsl

Now let’s setup NFS, please note with NFS use IP address and not a hostname, only reliable way to use hostnames with NFS in /etc/fstab is if “generateResolvConf = false”, if that is something you really want then this is how I would do it:

cp /etc/resolv.conf /etc/resolv.conf.old
wsl --shutdown (windows)
wsl -d Ubuntu-22.04 --exec dbus-launch true (windows)
wsl
cp /etc/resolv.conf.old /etc/resolv.conf; pico /etc/resolv.conf

But I don’t, just here for reference.

NFS setup:

Add to /etc/fstab your mount, on your NAS make sure to allow IP etc in /etc/exports and restart mountd so wsl can access it then:

mkdir /data
pico /etc/fstab
#ADD following:
192.168.0.1:/data/yourfolder /data nfs vers=4,auto,noatime,nolock,tcp 0 0

Use your NAS IP and folder, but that it, now try it out:

mount /data

If everything went alright and you can see your files “ls -al /data”, your good to reboot and check NFS mount mounts on startup:

wls --shutdown
.\WSL_start.bat
wsl

And that’s it, you have NFS.

Final Thoughts

Microsoft should be adding ways to start WSL headless for us, also a way to access serial console. This however should get you going, until next time…..

SunSaturn

Using Home Assistant Kasa Matter Plugs and a Victron MPPT VEDirect cable to deplete your batteries for next day

Batteries, freebsd, home assistant, LifePo4, VEDirect, victron, ,

Wouldn’t it be nice to automate depleting your batteries at night so that you have unwasted electrons coming in next day from your solar panels? My setup is a Kasa matter plug for grid and a Kasa matter plug for solar going into a PDU as an automatic transfer switch. So at any point on my phone I could turn off the grid plug and let the batteries drain for next day, but that is manual labor so we should automate it with python.

I am going to do this on FreeBSD, but would work for Linux etc. as well by changing your serial port for VEDirect, so we’ll just make that a configuration option. Another reason I’m going to use home assistant for turning plug on or off, is they are matter plugs, the kp125m to be exact, so just using python-kasa or whatever is not going to work for matter plugs(not currently anyways). However we can use Home Assistant API calls to turn them off or on.

And I know there is going to be someone saying why not just use a Victron Cerbo GX instead and use a TCP port on network. Fair enough, just don’t have one yet, so I’ll just run the VEDirect cable to my servers USB port and do it that way for now.

Alright let’s get started, figuring out voltage max…

First thing we have to do is figure out a battery voltage max that we want to have for next day before sun comes out. So I’m going to do this by example so you can figure out yours. In my setup I have 280AH lifepov4 batteries @24V. Now I know right now it’s winter time and most I’m going to get a day is 2Kw a day off my 920watts of solar panels from my Victron history graphs on the app.

so math: 1Kw = 1000w / 24V = 41.6 amps (24V might actually be 26.blahV for absorption voltage, but whatever) So 41.6 x 2 = 83.33 amps for 2Kw. So I need to drain batteries 280ah – 83.33ah = 197 AH. Now that I know I need it at 197AH each day to not have wasted electrons, I need to look at a chart:

https://batteryfinds.com/lifepo4-voltage-chart-3-2v-12v-24v-48v/

Chart is in percentages, so now I need to convert that 197AH to a percentage so I can match up voltages with it. So let’s do that, 197AH/280AH*100=70.35%. Great now I have a percentage so I can cross reference that chart at 70% to see what my max voltage should be. According to chart, 70% is 26.4V. Perfect we are golden, but remember to adjust this for summertime when your getting 6-7 Kw a day instead!

Installing pre-requisite packages

For VEDirect we’ll use the vedirect module. For Home Assistant all we’ll need is pythons requests and json libraries to keep it simple. For FreeBSD:

pkg install py39-pip py39-requests
pip install vedirect

Figuring out what we need to code

So what I’m thinking is we need 2 python scripts, one to turn on or off the grid plug depending on the voltage, and a 2nd to turn it on no matter what before sun comes out. We will toss the 2 scripts in a cronjob , so first one runs every 5 minutes between 11pm and 6:50am, and 2nd one to run at 7am to turn on grid plug no matter what.

Now we will need 2 things from Home Assistant, so let’s login there. First we need an API long lived token for accessing the API, you can find this by clicking under developer tools and settings on your nav bar. Just click on your name at very bottom. Now scroll all way to bottom and click “create token”. Copy it and save it for later.

Next we need to know what Home Assistant actually calls our plug, not the name we gave it, also called the “entity”. For that just go to your home screen, click on your plug and go to settings icon. In there you should see something called “entity ID”. Copy that as well, that’s how we talk to home assistant.

Let’s code the 2nd script first as it should be shorter….

First Python script to turn on the plug

Let’s call this one ha_grid_on.py : 

#!/usr/local/bin/python3.9

#pkg install py39-pip py39-requests

import requests
import json
import time

def plug(address,token,entity,state):
   #returns on/off with "state=check" else 1
   if state == "on":
      url = "http://" + address + "/api/services/switch/turn_on"
      req = "requests.post(url, headers=headers, json=data)"
   elif state == "off":
      url = "http://" + address + "/api/services/switch/turn_off"
      req = "requests.post(url, headers=headers, json=data)"
   elif state == "check":
      url = "http://" + address + "/api/states/" + entity
      req = "requests.get(url, headers=headers)"
   else:
      print(f"SunSaturn# [FAIL] : Only off/on/check should be called")
      exit(1) #exit program no point doing anything
   #print(f"url is {url}")
   #print(f"req is {req}")
   data = {"entity_id": entity}
   #print(data['entity_id'])
   headers = {
      "Authorization": "Bearer "+ token,
      "content-type": "application/json",
   }
   try:
      response = eval(req)
   except Exception as e:
      print(f"SunSaturn# [FAIL] : Failed connect to home assistant: {e}")
      exit(1) #exit program no point doing anything   
   if not response.ok:
      print(f"SunSaturn# [FAIL] : Failed response code not 200: {response.text}")
      exit(1) #exit program no point doing anything
   check = json.loads(response.text)
   if state == "check":
      check = json.loads(response.text)
      return check['state']
   return 1


#CONFIG - EDIT ME
#######################
#HOME ASSISTANT
#HA assistant token
address='ha:8123'                     #Home Assistant API [IP:PORT]
                                      #HA token, create one in HA
token='eyJhbGciOiJIUzI1NiIsInR5dCI6IkpXVCJ9.eyJpc3MiOiIyYmQ1MDM0YTg2NjY0NDIzOWZlZjg2NmZiNGY4N2E7MyIsImlyuCI6MTcwMTk5MTU3OCwiZXhwIjoyMDE3MzUxNTc4fQ.swUTIoUFWbXEul3JuWRRiKpE-Ene-kKeM1ch8uNQF5o'
entity='switch.kasa_smart_wi_fi_plug' #grab entity of Grid Plug from settings for device on homepage of HA
#######################

plug(address,token,entity,"on") #last arguement can be 1 of on/off/check
print(f"Turned on Plug, checking if it's really on...")
time.sleep(1) #we need a sleep of 1 to give HA time to update
check = plug(address,token,entity,"check")
if check == "on":
   print(f"YEP")
else:
   print(f"NOPE")

Ok I stuffed most of logic in a function, also since all we care about is the script just failing anywhere it fails, we just stuff a exit(1) anywhere to stop execution instead of returning from function. Next we setup configuration variables to edit like the token and entity we got from HA already. Only other config we need is the IP:port. Adjust this for your own HA, I have just ha in /etc/hosts pointing to its real IP.

Next we turn the plug on, then do a check to see if it’s actually on as well. Pretty simple script, first one done.

2nd python script

Let’s called this one victron.py:

#!/usr/local/bin/python3.9

#pkg install py39-pip py39-requests
#pip install vedirect 




import vedirect
import requests
import json

def victron(serial_port, serial_baud):
   try:
      device = vedirect.VEDirect(serial_port,serial_baud)
   except Exception as e:
      print(f"SunSaturn# [FAIL] : Victron read bad data: {e}")
      exit(1) #exit program no point doing anything
   try:
      float(device.battery_volts)
   except Exception as e:
      print(f"SunSaturn# [FAIL] : Return type was not a float: {e}")
      exit(1) #exit program no point doing anything
   if not device.battery_volts > 0:
      print(f"SunSaturn# [FAIL] : Voltage not greater than 0!")
      exit(1) #exit program no point doing anything
   #print(dir(device))
   #print(type(device.battery_volts))
   return device.battery_volts


#http://ha:8123/api/states/switch.kasa_smart_wi_fi_plug
#http://ha:8123/api/services/switch/turn_on
#http://ha:8123/api/services/switch/turn_off
def plug(address,token,entity,state):
   #returns on/off with "state=check" else 1
   if state == "on":
      url = "http://" + address + "/api/services/switch/turn_on"
      req = "requests.post(url, headers=headers, json=data)"
   elif state == "off":
      url = "http://" + address + "/api/services/switch/turn_off"
      req = "requests.post(url, headers=headers, json=data)"
   elif state == "check":
      url = "http://" + address + "/api/states/" + entity
      req = "requests.get(url, headers=headers)"
   else:
      print(f"SunSaturn# [FAIL] : Only off/on/check should be called")
      exit(1) #exit program no point doing anything
   #print(f"url is {url}")
   #print(f"req is {req}")
   data = {"entity_id": entity}
   #print(data['entity_id'])
   headers = {
      "Authorization": "Bearer "+ token,
      "content-type": "application/json",
   }
   try:
      response = eval(req)
   except Exception as e:
      print(f"SunSaturn# [FAIL] : Failed connect to home assistant: {e}")
      exit(1) #exit program no point doing anything   
   if not response.ok:
      print(f"SunSaturn# [FAIL] : Failed response code not 200: {response.text}")
      exit(1) #exit program no point doing anything
   check = json.loads(response.text)
   if state == "check":
      check = json.loads(response.text)
      return check['state']
   return 1


#CONFIG - EDIT ME
#######################
#HOME ASSISTANT
#HA assistant token
address='ha:8123'                     #Home Assistant API [IP:PORT]
                                      #HA token, create one in HA
token='eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiIyYmQ1MDM0YTg2NjY0NDIzOWZlZjg2NmZiNGY4N2E1MyIsImlhdCI6MTcwMTk5MTU3OCwiZXhwIjoyMDE3MzUxNTc4fQ.swUTIoUFWbXEul3JuWRRiKpE-Ene-kKeM1ch8uNQF5o'
entity='switch.kasa_smart_wi_fi_plug' #grab entity of Grid Plug from settings for device on homepage of HA
#VICTRON
voltage_max=26.1                      #we don't want voltage higher than this for morning solar charging
serial_port='/dev/cuaU0'              #serial_port for VEDIRECT, on linux try "setserial -g /dev/ttyUSB[01]" to find your serial port
serial_baud='19200'                   #serial baud rate, 19200 seems recommended setting from Victron
#######################

#check voltage of battery
volts=victron(serial_port,serial_baud) #will exit program if any failures with exit(1)
#print(f"Volts is: {volts} ",type(volts))

#check if plug is on or off
check = plug(address,token,entity,"check") #last arguement can be 1 of on/off/check
print(f"Battery voltage is {volts}, checking what to do...")

if check == "on":
   #plug is on, check it we need to turn it off
   if volts > voltage_max:
      plug(address,token,entity,"off")
      print(f"Voltage is greater than {voltage_max} Turned off Grid Plug")
   else:
      print(f"Grid Plug is on and voltage less than {voltage_max} DOING NOTHING")
else:
   #plug is off, check it we need to turn it on
   if volts < voltage_max:
      plug(address,token,entity,"on")
      print(f"Voltage is less than {voltage_max} Turned on Grid Plug")
   else:
      print(f"Grid Plug is off and voltage greater than {voltage_max} DOING NOTHING")

Again we reuse the Home Assistant function, but add an extra function for Victron to pull battery voltage off the solar charge controller. New config options are max_voltage we figured out earlier, and serial_port and serial_baud rate. Adjust these to work with your setup. So what we are doing is checking if plug is on or off first, then depending on that we check voltage against voltage_max to decide to turn the plug on or off. Pretty simple 🙂

Now we are almost done….Personally I do everything as root(you may need to so you can access serial port), so I created a /root/cronjobs directory and stuck victron.py and ha_grid_on.py in there. Last thing to do is toss them in cron: “crontab -e”

# run every 15 minutes between 11pm and 6:59 am, suppress logging
*/15 23 * * * (/root/cronjobs/victron.py) > /dev/null 2>&1
*/15 0-6 * * * (/root/cronjobs/victron.py) > /dev/null 2>&1
#make sure grid plug on no matter what by 7am
0 7 * * * (/root/cronjobs/ha_grid_on.py) > /dev/null 2>&1

And don’t forget to “chmod 700 *.py” 🙂

Keep in mind you have to accommodate for “voltage rebound”, this happens when no load is present, battery voltage will jump up .3-.4, so you will want to lower your voltage for day 🙂 Unfortunately this means every 5 min its on solar, every 5 min on grid because of the voltage rebound. To compensate I use to run cron every 5 minutes, now I run every 15 minutes, long as not running a really high load should be fine.

And that’s it, all automated, never worry about it again 🙂 Of course adjust $max_voltage come summertime, but other than that, your golden 🙂 I’ll leave adjusting python script to actually use different max_voltages depending on the month as an exercise to the reader….

Till next time ….

SunSaturn

FreeBSD BHYVE using vm-bhyve to create FreeBSD, Linux, OpenBSD, NetBSD, Windows Server 2022 and Home Assistant VMs on NVME drives with UEFI

bhyve, freebsd, Virtual Machines

Easily create machines with FreeBSD and vm-bhyve. If you come from a world of using virsh with Linux, going this route will save you some massive headaches. You won’t have to reinstall your OS every new release as FreeBSD provides freebsd-update utility for that, as well as you won’t have to deal with anymore laggy virt-manager remote sessions, and easily be able to connect to console just as with virsh. Yeah let’s leave the Linux redhat/centos/rocky blah war behind us, and use something that will be solid and powerful on bare metal for a decade to come….

For this setup, what I have is 2 10gigabit ports, so I’ve already gone and created my LACP interface in /etc/rc.conf, then I created a bridge on top of that for the VMs. In Linux world that would be your bond0 interface and your br0 interface on top of that.

In FreeBSD its as simple as:

cloned_interfaces="lagg0 bridge0"
ifconfig_br0="addm lagg0 up description services"
ifconfig_bridge0_name="br0"
ifconfig_br0_alias0="inet 192.168.0.1 netmask 255.255.255.0 mtu 9000"
ifconfig_br0_ipv6="inet6 fc00:192:168:1::1/64 accept_rtadv auto_linklocal"
ifconfig_lagg0="laggproto lacp laggport ix0 laggport ix1"

ifconfig_ix0="up"
ifconfig_ix1="up"

What if you have a 100 gigabit home lab? I’ll just take a second to drool, then basically say that would be same setup as above but you’d most likely be setting your MTU to 16k instead of 9k in your FreeBSD rc.conf and on your switch ports configured for LACP.

Or without 10 gigabit LACP following should work, use your own device name for ix0:

cloned_interfaces="bridge0"
ifconfig_br0="addm ix0 up description services"
ifconfig_bridge0_name="br0"
ifconfig_br0_alias0="inet 192.168.0.1 netmask 255.255.255.0"
ifconfig_br0_ipv6="inet6 fc00:192:168:1::1/64 accept_rtadv auto_linklocal"
ifconfig_ix0="up"

Now we have a simple br0 interface for our VMs. If you not using LACP just remove the lagg0 stuff and addm your real interface instead to ifconfig_br0 line, the main point is we have our br0 device ready to go, don’t use MTU 9000 if you haven’t upgraded to 10 gigabit in your home lab 🙂 Using samba with this setup I get full 1 GB/s each way off NVME’s, and using SMB server multi channel I’ve gotten to around 1.3 GB/s each way to my Windows 11 desktop. Can’t complain it’s an older Dell r720 using clover to boot 3 NVMEs, if I had a newer server I’d definitely look into RDMA.

Next let’s make sure we have sysctl variables setup to make sure we can work with bhyve, so let’s open /etc/sysctl.conf and add the following:

#BHYVE
net.link.tap.up_on_open=1
#BYHVE + PF nat
net.inet.ip.forwarding=1
net.inet6.ip6.forwarding=1

Now enter: “service sysctl restart” on command line to apply the changes.

Now assuming your using your NVME drives for VMs, personally I used 3 1TB NVME drives in a stripe on zroot. Even though it is a raid0 it doesn’t matter, I have another server I bring online occasionally to back this one up, think of it as a raid 10 just across 2 machines 🙂

Alright let’s install what we need:

pkg install grub2-bhyve bhyve-firmware tmux vm-bhyve
zfs create -o mountpoint=/vm zroot/vm

Now we will want to add VMs to startup automatically when we reboot so let’s add vm-bhyve to /etc/rc.conf:

vm_enable="YES"
vm_dir="zfs:zroot/vm"

#vm_list="vm1 vm2"
#vm_delay="5"

We will leave last 2 lines commented out so we can decide what to start after install 🙂 Now let’s get vm-bhyve setup:

vm init
cp /usr/local/share/examples/vm-bhyve/* /vm/.templates/
vm switch create -t manual -b br0 services
vm set console=tmux

Next thing we want to do is edit our default template for creating new VMs. Personally i use pico which is an alias to “nano -w”. So “cd /vm/.templates; pico default.conf”

loader="uefi"
graphics="yes"
graphics_listen="192.168.0.1"
graphics_port="5900"
#If you want it to wait connecting with VNC uncomment next line
#graphics_wait="yes"
#Valid Options: 1920x1200,1920x1080,1600x1200,1600x900,1280x1024,1280x720,1024x768,800x600,640x480
#graphics_res="1920x1080"
xhci_mouse="yes"

#OPENBSD - uncomment hostbridge line and use .img not .iso from download site
#https://wiki.freebsd.org/bhyve/OpenBSD - vm install openbsd install74.img
#hostbridge="amd"

#conservative 1 cpu socket for windows, they charge apparently for multiple sockets
cpu=4
cpu_sockets=1 
cpu_cores=4 
memory=4G

#Use e1000 for NetBSD or you can't set mtu to 9000
#network0_type="e1000"
network0_type="virtio-net"

#assign tap devices to manual switch we created (vm switch create -t manual -b br0 services)
network0_switch="services"
disk0_type="nvme"
disk0_name="disk0.img"

#windows virtio driver to enable virtio-net network drivers
#https://github.com/virtio-win/virtio-win-pkg-scripts/blob/master/README.md
#vm iso https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-virtio/virtio-win.iso
#NetKVM->2k22->amd64 (for virtio ethernet driver, use device manager to update driver to this directory)
#uncomment for windows
#disk1_type="ahci-cd"
#disk1_dev="custom"
#disk1_name="/vm/.iso/virtio-win.iso"

#windows expects the host to expose localtime by default, not UTC
#uncomment for windows
#utctime="no"

This gives me a generic template for creating VM’s with windows, Linux, FreeBSD, OpenBSD etc on my NVME drives.

So when we now create VM’s it will install this template to /vm/$VM_NAME directory along with 2 lines adding unique mac address and UUID. That’s important because we don’t want crazy things happening in our ARP tables 🙂 Mainly what is nice about that is for DHCPD. If you want to control what IP address your VM gets you can add that mac address to your /usr/local/etc/dhcpd.conf file and know what IP it will get if not hardcoding it. UUID is for dhcpd6.conf where you can try to hardcode IPV6 address for host using that. Since all OS’s don’t really respect it, I normally leave it alone.

May be wondering why I use raw images instead of zvols? Because raw images have better performance read here: https://klarasystems.com/articles/virtualization-showdown-freebsd-bhyve-linux-kvm/

No worries, I’ve Crystal Mark tested a windows VM, and it would definitely blow away Linux KVM on disk speed.

So at this point we are done our setup, now on to creating VM’s.

So let’s take an example, I want to create a VM called asterisk, another FreeBSD 14 machine here we go:

vm iso http://blah.iso
vm create -s 50G asterisk
#add this new VM IP address to /etc/hosts
pico /etc/hosts 
cd /vm/asterisk
cat asterisk.conf
#look for mac address and add to DHCPD
pico /usr/local/etc/dhcpd.conf
service isc-dhcpd restart
pico asterisk.conf

So what we doing here is telling vm-bhyve to fetch iso for us and put it in /vm/.iso. Then we tell it to create a 50G image called disk0.img in /vm/asterisk directory, an asterisk.conf from our /vm/.templates/default.conf file with 2 unique lines added for mac address and UUID. Pretty sweet. Don’t over think it here with disk0.img, all its doing is a simple “truncate -s +50G disk0.img” to create a blank image.

I then go on to add IP address I picked out to /etc/hosts and /usr/local/etc/dhcpd.conf but you can skip this if you like, but it’s a useful habit to get into for installing things like say home assistant VMs.

Now final step is edit asterisk.conf to your liking. Normally only thing I change per VM is graphics port and I may use virtio CDROM for windows. I’ve successfully installed FreeBSD, Ubuntu, Rocky Linux, OpenBSD and windows server 2022 using this template so it’s a pretty good one 🙂 Also since we using NVME makes windows install less painful as we won’t need any windows drivers for hard drive, think you might for network card still, why I generally leave cdrom attached in windows. For graphics_port I normally set that to last digits of VMs IP address, so like say I set asterisk to use 192.168.0.3, then my graphics_port will be 5903. I then add that host to my SecureCRT list, and add the VNC to my mobaxterm list 🙂

Honestly it’s more steps, but at end of day when VM’s are all running, now its one simple click on ssh session list or VNC session list to get to VM, so nice habit to get into.

Now let’s start the install. Normally when you first start trying to install an OS you want CDROM or USB stick in to install it, then remove it as host reboots. vm-bhyve does something amazing here, does it for you 🙂

vm install asterisk FreeBSD-14.0-RELEASE-amd64-bootonly.iso

That’s it! It will take ISO we downloaded to /vm/.iso/blah.ISO , add it to VM for install. At this point what I normally do is either “vm list; vm console asterisk” or just use mobaxterm to connect to graphics port. Tmux is nice addition to serial port of host 🙂 But normally to be able to use that you have to configure each host to use the serial port. Like adding a line at end of grub on Linux or just editing FreeBSD /boot/loader.conf.

If you want to wipe a VM out and start over you have 2 options:

vm destroy asterisk
OR
zfs destroy -r zroot/vm/asterisk
#-r because we'd have to killoff snapshots to if they exist

Your all done. Congratulations.

Now what I normally like to do with a new VM is make sure it shuts down from host when I want it to, so I may issue a “vm stop asterisk” and keep watching VM and “vm list” to see it has ended. When I’m happy, I’ll “vm start asterisk”, then add asterisk to /etc/rc.conf as something to start on boot for example.

What if you screwed up install and say installed asterisk here with 10G less that you wanted? Easy enough, on the host just do the following:

vm stop asterisk
cd /vm/asterisk
truncate -s +10G disk0.img
mdconfig disk0.img
gpart recover md0      #recover disk after adding space
gpart show md0         #find partition number with freebsd-zfs in it, in my case 3 
gpart resize -i3 md0   #expand partition freebsd-zfs partition 3, could also be 4 if you didn't install with GPT(EFI)       
mdconfig -du md0       #unmount md0

But how do you get the guest to see the new space? Do following:

vm start asterisk
#login to guest and:
zpool status (look for device)
zpool online -e zroot /dev/nda0p3 #use your device from command above

That’s it. But what if you screwed up something on guest in /boot/loader.conf or /etc/rc.conf that prevents you from booting? Simple let’s mount it and fix it on host:

vm stop asterisk #make sure guest is stopped or risk data corruption!
cd /vm/asterisk
mdconfig disk0.img
mkdir mnt
zpool import      #you should see md0 zroot disk to import
zpool import -fR /vm/asterisk/mnt -t zroot testing
zfs mount -o mountpoint=/vm/asterisk/mnt testing/ROOT/default
#cd /mnt/etc or /mnt/boot etc and fix stuff....
zpool export testing
rm -rf mnt
mdconfig -du md0  #unmount guest

That’s it, then just “vm start asterisk” and problem solved 🙂 Even simpler for Linux obviously, just look for partition and mount it, but same would apply for a debian zfs root install.

Now I bet someone is wondering what if that was an ubuntu install, and I screwed up grub, how would I recover that? Well simple, “vm poweroff ubuntu”. “vm install ubuntu ubuntu-22.04.3-live-server-amd64.iso” to load the install ISO just so we can use it to drop to a shell. Connect with VNC viewer, go to help -> shell. Now assuming you installed with all default options with LVM and EFI then here is how to fix grub:

fdisk -l
lvscan
mount /dev/ubuntu-vg/ubuntu-lv /mnt
mount /dev/nvme0n1p2 /mnt/boot
mount /dev/nvme0n1p1 /mnt/boot/efi

mount --bind /dev /mnt/dev
mount --bind /proc /mnt/proc
mount --bind /sys /mnt/sys

chroot /mnt
nano /etc/default/grub  (fix it)
grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=GR
update-grub

umount /mnt/dev
umount /mnt/proc
umount /mnt/sys
umount /mnt/boot/efi
umount /mnt/boot    
umount /mnt
sync

reboot

Again all things you can do right from FreeBSD host. And again someone will probably ask I tried to install Kali Linux and after install it won’t boot how do i fix it? Well I thought this guy did a great job of explaining it:

https://record99.blogspot.com/2021/12/bdsdex-failed-to-load-boot0001-uefi-bhyve-sata-disk.html

So let’s just modify that for Kali Linux:

1. Enter exit get out of the UEFI Shell (you will come to BHYVE panel)
2. Choose Boot Maintenance Manager
3. (In the Boot Maintenace Manager ) Choose Boot From File
4. (In File Explorer) NO VOLUME LABEL, [PciRoot(0x0)/ Pci()]  press Enter
5. (Still in the File Explorer) Choose <EFI>
6. (Still in the File Explorer) Choose debian 
7. (Still in the File Explorer) Choose grubx64.efi
8. Then you will get into the system
How can you fix it:
1) Get access inside your Kali Guest VM.
2) as root: cd /boot/efi/EFI
3) mkdir BOOT
4) cp kali/grubx64.efi BOOT/bootx64.efi
5) reboot
Basically you need to have this file in this path:
/boot/efi/EFI/BOOT/bootx64.efi

And now someone is going to ask how did I get windows server 2022 to work. I downloaded the windows_server_2022.iso and virtio-win.iso to my /vm/.iso directory and edited my template as follows:

loader="uefi"
graphics="yes"
graphics_listen="192.168.0.1"
graphics_port="5907"
#enable this briefly on windows installs, install won't do anything till you connect with VNC
#graphics_wait="yes"
#Valid Options: 1920x1200,1920x1080,1600x1200,1600x900,1280x1024,1280x720,1024x768,800x600,640x480
graphics_res="1920x1080"
xhci_mouse="yes"
#conservative 1 cpu socket for windows, they charge apparently for multiple sockets
cpu=4
cpu_sockets=1 
cpu_cores=4 
memory=4G
network0_type="virtio-net"
#assign tap devices to manual switch we created (vm switch create -t manual -b br0 services)
network0_switch="services"
disk0_type="nvme"
disk0_name="disk0.img"

#windows virtio driver to enable virtio-net network drivers
#uncomment for windows
disk1_type="ahci-cd"
disk1_dev="custom"
disk1_name="/vm/.iso/virtio-win.iso"

# windows expects the host to expose localtime by default, not UTC
#uncomment for windows
utctime="no"
uuid="6efefb12-8da5-11ee-906d-e4434bf65f00"
network0_mac="58:9c:fc:0f:b8:a6"

Then I did a simple “vm install windows windows_server_2022.iso” and installed it. When it was installed I went to device manager, selected the ethernet adapter that obviously wouldn’t work, then pointed it to look on the virtio-win.iso drive at “NetKVM->2k22->amd64”. About it. Thought I’d throw this in there as I’m sure there is somebody that likes to setup active directory to authenticate everything. But honestly look at passkeys, it’s the new thing apparently, might as well learn it now, search for webauthn.

Ok we are now experts, so let’s install Home Assistant VM then to make sure 🙂 So home assistant only provides an image and not a ISO, so that’s pretty simple to, all we really need is a raw image, so I see they provide a qcow2 image on their site, so we can just convert that easily after downloading it, and for good measure let’s break root on it to after install 🙂

vm create -s 10G ha
vm iso https://github.com/home-assistant/operating-system/releases/download/11.1/haos_ova-11.1.qcow2.xz
cd /vm/.iso
unxz haos_ova-11.1.qcow2.xz
pkg install qemu-tools gdisk
qemu-img convert -f qcow2 -O raw haos_ova-11.1.qcow2 haos_ova-11.1.img
rm haos_ova-11.1.qcow2
gdisk -l ./haos_ova-11.1.img
cp /vm/.iso/haos_ova-11.1.img /vm/ha/disk0.img

Ok what I’ve done here is download the image, go to ISO/image directory and uncompress it. I install some tools to convert image to raw image type we need for BHYVE. Delete old format to save disk space, then run gdisk on image to see all its partitions to know we have a working OS on this image. Could have used gpart I suppose as well but nice to have a command that works on FreeBSD and Linux.

I then just copy image over the one vm-bhyve added, so I could have even done 1G instead of 10G creation or whatever, it didn’t matter.

Alright, remember I said have good habits with VMs, at this point cd /vm/ha and look at ha.conf for any edits, and set an IP for this home assistant VM in /etc/hosts and toss it’s IP and mac address in /usr/local/etc/dhcpd.conf. “service isc-dhcpd restart”. I assigned it 192.168.0.15, so I’m pretty sure when it boots its going to call on a DHCPD server for an IP.

vm start ha
vm console ha

Well I typed in “root” and hit enter and I have root access, so much for trying to mount all the partitions and break it 🙂 Also went to VNC port and at ha> prompt just typed in “login” and got root to. Doing a cat /etc/shadow it seems root has no password to login. Running a “ps aux” I see dropbear running instead of sshd, like an android system, yet this is a x86_64 image, unreal.

Alright well this is still inconvenient, rather add it to my ssh list to with root access, since they just let you blindly login with root on console like this, they probably have devel docs somewhere how to do this.

YEP, check it out here: https://developers.home-assistant.io/docs/operating-system/debugging/

Just says add your pub ssh key and your good to go with ssh root@homeassistant.local -p 22222. Fair enough let’s go back to our tmux console and add it then:

cd /root/.ssh
vi authorized_keys
chmod 600 authorized_keys

Well they didn’t have nano installed, but most OS’s include vi, so was as easy as just opening a file, making sure its permissions was ok and good to go, can add it to your ssh session list now 🙂

I must admit I now played with it for a bit, and quite like this, added all my smart devices to it, downloaded the app to my phone, guess all I need is a microphone/speaker so can control all my lights etc if internet goes out 🙂

Anyways, this really is how simple using vm-bhyve is, vm list, vm iso/create/stop/start etc. I honestly like it, and will continue to use it, reminds me of using virsh with libvirtools on Linux. Tmux console is great, CTRL-a-s to switch between VM’s quite handy. I was only used to using screen before tmux, so I remapped CTRL-a-d to work on tmux to:

pico /usr/local/etc/tmux.conf
#and added:
# remap prefix to Control + a
set -g prefix C-a
# bind 'C-a C-a' to type 'C-a'
bind C-a send-prefix
unbind C-b

Great works like screen, now I won’t forget 🙂 tmux is pretty good, liking it better these days than screen, but I still keep screen and minicom around because tmux can’t connect ports with baud rates, parity etc.

Anyways I hope this has been a great introduction to working with VMs under FreeBSD BHYVE with vm-bhyve, vm-bhyve adds some powerful features and makes it easy for us. FreeBSD gives us the power of its ports collection, most stable ZFS OS, and powerful networking features like /etc/pf.conf and working with setfibs(its like rt_tables on linux but more customizable) 🙂

Now go setup your zfs snapshots for your VMs in crontab: “crontab -e”

0 3 * * * (zfs destroy -r zroot/vm@`/bin/date +\%A`; zfs snapshot -r zroot/vm@`/bin/date +\%A`) > /dev/null 2>&1
0 3 1 * * (zfs destroy -r zroot/vm@`/bin/date +\%B`; zfs snapshot -r zroot/vm@`/bin/date +\%B`) > /dev/null 2>&1

What this does is first line takes a snapshot daily, Mon-Sunday, so during a 7 day period you can rollback, and it will delete old ones to save disk space. The “-r” just says do it for each VM. I really only had to escape the “%” sign for cron about it. Second line does a monthly snapshot, so you’ll get 12 over course of a year, and they’ll delete each other to save disk space and keep things organized, simple and sweet…

If you need to rollback a VM I suggest you setup an alias to see your snapshots in your .bash_profile or whatever shell you use:

alias snapshot='zfs list -t snapshot'

Then you can just rollback to whatever you see in list:

snapshot
zfs rollback zroot/vm/asterisk@Monday

That’s the power of ZFS on a virtual machine 🙂 Now go add your new VM to your vm list in /etc/rc.conf if you want it to startup when host reboots. Good to go, now can rollback a VM if you screw it up 🙂

And now someone is going to ask how to I get vm console serial access to all OS’s because you’d don’t like using VNC viewer. Well first of all for windows obviously your going to have to use VNC viewer, for the rest, let’s start with FreeBSD, add to your /boot/loader.conf

autoboot_delay="5"
comconsole_speed="115200"
boot_multicons="YES"
boot_serial="YES"
console="efi"

Then just reboot, and use vm console to watch it. If you have resizing issues with tmux just type “resize” and you’ll be fine, if you don’t have the command install xterm on guest. Should be installing xterm and xauth on every guest. Now what about ubuntu/kali? Edit /etc/default/grub as such:

-GRUB_TIMEOUT_STYLE=hidden
+#GRUB_TIMEOUT_STYLE=hidden

# Optional kernel options that you very likely want. Don't affect GRUB itself.
# Remove quiet to show the boot logs on terminal. Otherwise, you see just init onwards.
# Add console=ttyS0, or else kernel does not output anything to terminal.
-GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
-GRUB_CMDLINE_LINUX=""
+GRUB_CMDLINE_LINUX_DEFAULT=""
+GRUB_CMDLINE_LINUX="console=tty1 console=ttyS0,115200"

# Show grub on both serial and on display.
-#GRUB_TERMINAL=console
+GRUB_TERMINAL="console serial"

Then just type “update-grub” and your good to go.

What about centos/redhat/rocky linux, you installed the newest 9.3 as of today? Well since all that is pretty much redhat look at following url:

https://access.redhat.com/articles/3166931#config9

So basically just do:

grubby --update-kernel=ALL --args="console=tty0 console=ttyS0,115200"
grubby --info=ALL|grep -i args

What about OpenBSD?

openbsd:~ # cat /etc/boot.conf 
stty com0 115200
set tty com0
openbsd:~ #

What about NetBSD? 

echo "consdev=com0,115200" >> /boot.cfg

Other helpful hints? Just remember the “resize” command when working with vm console VMNAME. Tmux is great at screwing up your terminal. Other than that, I have this added to my .bash_profile to remind me to make sure to setup X11 forwarding and have that resize command work:

if ! command -v xterm >/dev/null 2>&1
then
    echo "WARNING: xterm not installed, resize command may not exist for tmux!"
fi
if ! command -v xauth >/dev/null 2>&1
then
    echo "WARNING: xauth not installed, X11 forwarding will not work, enable it in sshd.conf as well"
fi

Where to go from here? If its just a home lab maybe you should authenticate a user to login to every VM in one spot, maybe active directory, maybe kerberos, or just keep it simple and use NFS and NIS. I’ve tested NFS and NIS across all OS’s except windows obviously, only issue is with Redhat/Rocky/Centos having removed all NIS stuff calling it insecure.

Well that’s true and not true, if someone was sitting on my network with tcpdump or wireshark, for sure it’s possible to eventually sniff out encrypted hash of password. Personally I blame the old /etc/shadow /etc/master.passwd format. If they would just upgrade that to using public and private keys they could sniff all they wanted and take 10 years trying to crack it then. It is what it is, but for a home lab, NIS is just fine, at least you can use yppasswd to update your password across all VMs and your good to go…

Other fun things to try with FreeBSD as host? Get yourself a Starlink dish on sale, install it, and using /etc/pf.conf and setfibs make certain IPs in your home lab go out Starlink and others your default internet connection. After free month trial cancel it. Then if your internet ever goes down you have a layer of redundancy to just enable Starlink on your phone 🙂

Or play with wireguard, then using /etc/pf.conf and setfibs, make certain IPs in your home lab go out VPN, and others your default connection. Maybe start a VPN service afterwards.

Play with samba and NFS, export them to your VLC app on your google TV chromecast in living room. Configure utorrent to download to the samba directory that you have mapped as network drive on windows.

Play with snapshots, take snapshots of everything so you can rollback. You have power of ZFS on main virtualization host, take advantage of it 🙂

Install a rsyncd.conf on every VM, using crontab from host backup important configuration files on each VM once a night.

Install a windows server 2022 VM that you configure to start first in /etc/rc.conf. Watch some youtube videos on active directory, configure some VMs to authenticate off that for fun. Maybe play with passkeys instead.

Write some HOWTO’s on internet how to increase VM performance even more, I’d love to read it!

Setup a real dedicated server online, use same setup here. Install VMs for a webserver, DNS server, mail server etc. Try to run as few services on main host as you can other than DNS , wireguard and ssh. Use one central /etc/pf.conf to firewall for host and every VM(be very careful unless your provider can give you console access to machine through a KVM etc, take it from experience 🙂 ) If not place firewall rules and revert them in 60 seconds something like: “pfctl -F all; pfctl -f /etc/pf.conf; sleep 60; pfctl -F all; pfctl -f /etc/pf.conf.old”. I’d really recommend testing on your home lab first 🙂 Try to install all services you can on a FreeBSD vm, for anything else can use an ubuntu VM for development. Setup a wireguard tunnel to back this server up once a night using rsync or zfs send/receive. For advanced users maybe rotate wireguard keys once in awhile automated. The habit you want to get into is you are rarely logging into main host for anything other than security updates, your mostly using the VM’s to do everything. That way if you screw anything up you can simply do a “vm console VM” on host. Let the worker bees be children, and the parent parenting 🙂

Maybe you’d like some custom FreeBSD kernel tuning, you can recompile the whole world and add features as you like. I’ve done this on numerous occasions to do custom things. I should note do not do this on production servers online unless you know what you are doing, cause it can take you that much longer to apply a security update, I almost feel sorry for NetFlix in that regard as they run all custom FreeBSD current custom servers(but not that sorry, the hardware they run on is very powerful so would take them almost no time 🙂 ) If you decide to do this compile everything on main host, nfs mount /usr/src and /usr/obj on all FreeBSD VMs and do a quick install on those to.

Maybe play with crypto, maybe learn GPU passthrough on vm-bhyve, possibilities are endless, you are now in complete control….

Happy BHYVE’ing…..

SunSaturn

Long Live the King

king charles

What a wonderful coronation today for King Charles. All the energy in the crowd, the bishop placing the crown, the son kneeling and kissing his father, the gold carriage, and last but not least the jets with the beautiful red, white and blue display in the skies while the King and Queen stood over the balcony. A definite moment for the history books, an almost replica to Queen Elizabeth’s proceedings in the 50s.

The energy in the crowd was amazing, millions of people walking from Westminster Abbey to Buckingham Palace, Canadians there wearing Canada on their shirts, caps or waving flags. RCMP on horses there showing the pride and heritage of Canada. All the Americans there standing in awe at the beautiful ceremony, and of course all the Britain’s there showing their allegiance to the King.

I am always one that looks at the future, where I am going, not where I or others have been. However when you experience such a beautiful display, I am reminded of what Tim McGraw says in one of his songs, “When you get to where you’re going, help the next one in line”. So even though we may be coming to a time of interplanetary exploration, it’s always important to look back and reflect and respect. Let’s hope our corrupt leaders of this world do not start WW3 and set us back decades, let’s stay bright and hope our real leaders will come and stand up when they are most needed.

Dear King:

With the world in chaos, government communications broken down, USA weaponizing the dollar, Russia attacking out its borders in response to USA building NATO on its border and even biological weapons in Ukraine on its borders, USA preparing for WW3 once China attacks Taiwan, Russia not taking Ukraine in response to preparing to battle the USA, and a possible WW3 coming with nuclear power and AI drones, we pray for your wisdom.

With the world approaching being a space faring species, WW3 would set humanity back decades and is not fair to the younger generations that will suffer as a result, for this we pray for your wisdom.

With the world leaving a mess for the younger generations with CBDC currencies leaving no anonymity or privacy for them, facial recognition and AI leaving them again with no privacy, again we pray for your wisdom.

With the world on the brink of nuclear warfare, that will destroy land for generations to come, we pray for your wisdom.

With the world with a large gap between older generations and younger generations on what idealism and the future should be, we pray for your wisdom.

To your beautiful coronation ceremony today I dedicate this song to you:

King Charles when I was young, maybe about 9 years old, one of the most beautiful displays on humanity I ever saw and was proud to be born into this world was a day in the 80s when many singers came together to sing, “We are the world”, by USA For Africa. I think it’s a really heartfelt song that all generations should never forget:

I pray we can again can show the younger generations our hearts….

Long live the King,

SunSaturn(may we leave the world to our kids as we would want it left to us)

Booting FreeBSD, USB to NVME boot with clover on older machines

freebsd

I ran into an interesting situation, why not take an old Dell r720 etc, install a PCI card with an NVME on it and boot from that instead for our core OS’s and virtual machines. As we know older machines do not support direct booting from NVME.

What I thought would be no more than an hour to do, turned into a whole day ordeal fighting with clover to boot it! The idea was simple: put Clover on a USB stick, configure my Dell r720 to only boot off that, and have USB stick boot the NVME with FreeBSD on it. I managed to do it finally, but it was not fun! Was it worth it? Damn rights, 3500MB/s on an NVME, our virtual machines should fly, especially on FreeBSD “make installworld” on virtual machines! Also we know bhyve is faster than even Linux KVM when using NVME as boot option, so I will show you how to do it and hopefully this only takes you an hour out of your day 🙂

Problems I ran into:

Just getting clover on a usb stick on windows 11 then modifying files proved troublesome, if I had to do it over again, I’ll show you 2 better ways to do it. Fighting with windows 11 just for access to the USB stick EFI partition, vs just mounting it with FreeBSD and modifying it from there turned into a real headache. Also originally I installed FreeBSD on the NVME and then clover no matter what I did would not see it, but in the end actually installing FreeBSD booting from Clover first did the trick.

Hardware list(home server):

a) Dell r720 – if I had to pick an old Dell machine to do this again I would have instead picked a Dell r730 up. Reason is r730s up have bifurcation support you can enable in BIOS. This would allow you to get a PCI card and put multiple NVMEs on it. Without that bifurcation support your limited to just 1 NVME on the card. As far as Dell r710s down, avoid them like the plague, they still have PCIE version 2 at the back, which would limit you to 1500MB/s instead of getting full 3500MB/s on PCIe version 3 with Dell r720s and up. If price is a factor a single NVME on a Dell r720 is a good option as well. If you cannot afford a PCIE version 4 or 5 build, a Dell r730 stacked with 3 NVME on a PCIE card would be best bang for your buck for a commercial use or a single NVME with a Dell r720 for home use.

b) Samsung 970 EVO Plus – These are cheaper these days for 1 and 2TB options, I chose the 1TB option as I still don’t like fact I cannot get a 20TB NVME for 300 bucks at end of 2022 still. It should be enough space to run a lot of VMs. Look at Ebay, Kijiji, Craiglist for used ones first, if you can’t find one amazon is not bad for them these days. If going commercial pick something off Ebay like Netflix has used in past like 4 WD SN720s and put them all on a PCI controller like the “ASUS Hyper M.2 X16 PCIe 3.0 X4 Expansion Card V2 Supports 4 NVMe M.2″( https://www.amazon.ca/gp/product/B07NQBQB6Z ) With 4 of those in a ZFS stripe on a FreeBSD install you should rock more IO speed than a single PCIE 4 system with a single NVME at 7000MB/s read/write and have more longevity.

c) PCIE card – Keeping it cheap and simple I chose M.2 NVME to PCIe 3.0 x4 Adapter with Aluminum Heatsink Solution: https://www.amazon.ca/dp/B07JJTVGZM

So for this upgrade, the Samsung NVME and the PCIE card cost me maybe 100 dollars or less, the real challenge is now to make it bootable 🙂 Before this I was just using a 480GB 2.5 inch SSD and stuck it in an icy dock and tossed it along with other drives in the r720 raid I use for SSD storage on my 3.5 inch backplane. Two other spinning rust 16TB drives in there with a BYOD controller I use for ZFS storage. The core OS and VMs will get the new shiny 1 TB NVME. For backups I have a similar system I bring online only for backups. So everything in this system is in raid0, but technically its raid1 using another machine for secondary backups whenever I decide to bring it online through the Drac controller.

So if we put this is perspective, I should go from say 350MB/s on that 2.5 inch SSD to 3500MB/s on this new NVME. What a difference that is going to make running windows server in a VM or “make installworld” in a FreeBSD VM etc all for under 100 dollars. This is real reason to do it because obviously running samba etc we will be running it from the spinning rust drives, which at best will saturate only half of a 10 gigabit network card at maybe 250-500MB/s for movies etc. Also for speed, remember even sockets are just at the end of day files, so we should get an increase in socket performance to.

Clover USB stick:

The simplest way to install latest Clover is using a Mac utility someone wrote called BDUtility but it also works on a windows 11, literally download the program, stick in a USB stick and your golden from: https://cvad-mac.narod.ru/index/bootdiskutility_exe/0-5

The second way with windows 11 is just to grab latest ISO file and burn it with Rufus, you can find latest clover ISO at: https://github.com/CloverHackyColor/CloverBootloader/releases (should just be able to uncompress it with 7zip program and your good to go)

If your using first way I didn’t have an issue accessing files on windows 11, if you did it rufus way, we are going to need a program called “Explorer++” from explorerplus.com. So if you need it download it and make sure to run program as administrator or it won’t work.

Now if for any reason you go to “This PC” and the USB stick is not there after installing clover on it, you’ll have to mount it manually. For that just use “cmd” as administrator, type “mountvol” to see the volumes with no letter, then just manually mount it with “mountvol M: <long ass GUID volume>”. May have to mount a few of them with different letters to find that clover partition.

Now that we have access to clover USB stick with explorer++ or regular access with BDUtility route, we need to go into the EFI -> CLOVER -> drivers directory. We will have a directory called something like “off”. We need to copy the file NvmExpressDxe.efi to the other folders in drivers directory, sometimes they called BIOS and UEFI, just copy them into there. Another blog for a FreeBSD user he said he copied following as well to make it work with FreeBSD so copy them all if you wish…

  • AudioDxe.efi
  • CsmVideoDxe.efi
  • DataHubDxe.efi
  • EmuVariableUefi.efi
  • FSInject.efi
  • Fat.efi
  • NvmExpressDxe.efi
  • PartitionDxe.efi
  • SMCHelper.efi
  • UsbKbDxe.efi
  • UsbMouseDxe.efi

Now we should be able to boot off the USB stick to test it, then we can hit “F3” to see any hidden EFI partitions we can boot off as well we should be able to hit “F2” to create a misc directory on USB stick that will contain our GUID we need to customize the CLOVER->config.plist file for auto booting afterwards.

And so the fight began autobooting that NVME that cost me a day. So here is way I got it to work, create another USB stick with FreeBSD installer on it for that NVME. We are going to place the Clover and FreeBSD USB sticks into the server at same time. When we boot up machine we select say “F11” for Dell servers to go into boot menu. We will select the Clover USB. Once in Clover we hit “F3” to find the FreeBSD install USB, boot it, then just install FreeBSD regularly on NVME drive. Once we are done we can remove the FreeBSD USB stick, configure BIOS to only boot that Clover USB stick at this point and once we boot it, and hit F3, we should have option now to boot our new FreeBSD install on the NVME drive!

So at this point we are golden, but obviously we do not want to reboot machine and have to always go into clover to boot our NVME drive, be nice if we could reboot machine, and just automatically boot NVME drive. At this point since we have FreeBSD installed on NVME, let’s just boot it and mount the Clover USB stick there instead for modifications instead. Make sure to hit “F2” beforehand so we create a log file in misc directory on USB stick we will need next. Then hit F3, select the NVME and boot it.

Automating Clover Booting FreeBSD:

Normally when we install FreeBSD with ZFS, FreeBSD mounts its own EFI partition automatically for us at : /boot/efi. What we will want to do is modify /etc/fstab to also have access to Clover EFI at /boot/efi2 on demand. First do:

mkdir /boot/efi2

then nano /etc/fstab (and add the following, assuming da0 is your clover USB stick, gpart show da0):

/dev/da0s1 /boot/efi2 msdosfs rw 2 2

Now we should just be able to do a simple “mount /boot/efi2” and on reboots we will always have it.

cd /boot/efi2/EFI/CLOVER/; cat misc/preboot.log
mv config.plist config.plist.old; nano config.plist

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Boot</key>
  <dict>
    <key>Timeout</key>
    <integer>0</integer>
    <key>DefaultVolume</key>
    <string>D8189770-86A8-11ED-B706-E4434BF65F00</string>
  </dict>
  <key>GUI</key>
  <dict>
    <key>TextOnly</key>
    <true/>
    <key>Custom</key>
    <dict>
      <key>Entries</key>
      <array>
        <dict>
          <key>Hidden</key>
          <false/>
          <key>Volume</key>
          <string>D8189770-86A8-11ED-B706-E4434BF65F00</string>
          <key>Disabled</key>
          <false/>
          <key>Type</key>
          <string>Linux</string>
          <key>Title</key>
          <string>DELL R720 NVMe boot</string>
        </dict>
      </array>
    </dict>
  </dict>
</dict>
</plist>

Now you will notice 2 lines where I have “D8189770-86A8-11ED-B706-E4434BF65F00”. Replace this with GUID of your own NVME drive you got from “cat misc/preboot.log”. Save file and we are almost done!

If you have problems locating it, you should look for something like in my case:

38:018  0:000  - [07]: Volume: PciRoot(0x2)\Pci(0x2,0x0)\Pci(0x0,0x0)\NVMe(0x1,0D-B9-9E-01-5B-38-25-00)\HD(2,GPT,D81D9A46-86A8-11ED-B706-E4434BF65F00,0x82800,0x1000000)
38:019  0:000          Result of bootcode detection: bootable unknown (legacy)

From here I can see that D8189770-86A8-11ED-B706-E4434BF65F00 is the GUID I need to use as that is one with name NVME in it. Think of it like all those Linux /dev/disk/by-whatever GUID disk names you’d use for /etc/fstab or passing through disks on KVM on Linux.

Alright you’d think we are done and everything would work properly right? Nope for me it went to FreeBSD bootloader, and stalled there this time so to fix this I added following on FreeBSD to bypass the FreeBSD bootloader:

nano /boot/loader.conf (and add)

autoboot_delay="-1"

Save and exit, this setting will not allow Clover to interrupt our FreeBSD bootloader. I’m sure I could spend another day playing with all the Clover settings to find something that works, but I’m not going to, this is good enough, change it to 5 if you need the boot screen for whatever reason down the road.

Now we can successfully reboot machine any time with our NVME drive!

RECAP:

While this was a real pain in the butt to get working, the pros outweigh the cons. Now we can continue life as normal rebooting machine at will for updates. We are using an actual NVME! We are getting all the performance benefits. Now go install all your VMs to use “nvme” instead of “virtio-blk” on all your vm-bhyve ZFS datasets and enjoy using bhyve as it was intended 🙂 You are now faster than Linux KVM now your not using virtio-blk anymore 🙂

FreeBSD BHYVE why use it?A Bhyve suspend/resume howto + disaster recovery with UFS and ZFS

bhyve, freebsd, resume, suspend

INTRO:

FreeBSD Bhyve virtualization why? Sure there are different virtualization packages out there, but you have FreeBSD! Why not use ESXI? From experience any closed source software generally has a max lifetime of 10 years(abandonware), and opensource will generally over throw them by then. Also anything you don’t have full control over on root prompt to read the source code will be a nightmare to administer.

What about Linux and KVM? Yes, before Bhyve they were the best, the virt-manager and virsh/virt-viewer commands were unbeatable. Here is the thing with Redhat however, they over the course of just 2 years now have made most of sys admins angry on the internet. First they removed the megaraid drivers from the kernel(yes people still run Dell r710’s etc as fileservers), which caused a lot of people grief, then on upgrade to Redhat 9, they killed off a lot of Dell servers as well, people who upgraded had to reinstall with Rocky Linux 8 as they used a glibc that did not work on a lot of people’s systems. And to top it off they killed off Centos by moving it to Centos-stream, angering rest of internet. So everyone moved to Rocky Linux(original creator of Centos), to keep their hosts “production”.

The worst thing about Linux to begin with as a virtualization host, is every 2 years when a new major release comes out, most people have to take time out of their day to reinstall the OS. A core OS! With FreeBSD you can update to new major releases by way of simple freebsd-update command or rebuilding source code from scratch with “make buildworld” as well. No downtime reinstalling, no host to replace, a sys admin dream.

I’ve had fights with programmers over the years over Linux and FreeBSD, especially when it came to libevent and libev, where people swore by epoll over kqueue, from experience put FreeBSD under load and then a Linux server, your loads will be better served on FreeBSD with less CPU utilization, the network stack is better, more efficient, even Netflix uses it.

And now I know what everyone is going to say, been using Linux/KVM virt-manager for so many years, FreeBSD doesn’t even have suspend/resume stable. I’m not loosing my 20 SSH connections to my guests just because I had to reboot for a kernel update. I can’t disagree with this, I’d hate to be working 4 hours on a script on one of them, reboot for a kernel update, and lost all my work because I forgot to save on a host and host didn’t resume guests properly after reboot.

So why use Linux at all? I think Linux is fine as a guest, just not in the frontline of battle for reasons I mentioned above. There are still lots of things that will only run on Linux, if your a flutter programmer, FreeBSD still has not ported Dart language, so your apps will still need a Linux guest to test builds. If I’m considering working on an app, I would most likely install an ubuntu guest for that, do all your dart/c/java there along with windows 11, and your python Fastapi’s interfacing with Mysql on Linux and/or FreeBSD.

Honestly for your server part of your app, I would use a FreeBSD guest to begin with hosting Fastapi, I will show you how you could pass a second disk through to the guest with a proper 16k blocksize for max Mysql performance for your app. The biggest reason to use FreeBSD upfront, you can be stuck 6 months to a year working on an app, you really need a week of downtime because some new Linux release came out and you have to reinstall? I didn’t think so…let FreeBSD “Shine bright like a diamond” as Rihanna would sing.

So today I am going to take what is the most important and exciting #1 feature release for FreeBSD 14, suspend and resume. I will be installing FreeBSD current to test it. I will also show how FreeBSD is a force to be reckoned with as the core OS with virtualization.

Test case scenario:

Dell r720, Intel Enterprise 480GB SSD as core OS slapped into an icy dock for core OS, along with 2 12TB rust spinning drives for separate ZFS backup pool all on a megaraid JBOB controller. FreeBSD current, with production flags, with experimental suspend and resume features enabled in kernel and userland.

For main OS we will use default ZFS install, for guest tests we will install 2 FreeBSD guests, one with ZFS, and one with UFS. We will suspend them both, then resume them and see if we loose our SSH connections 🙂 I will most likely do a part 2 on installing Linux and Windows guests as I see this article will get pretty lengthy just with these 2 guests…

For third-party packages to interface with suspend and resume, we can’t use any. We will have to do everything manually, I will keep it simple with just bash scripts. Maybe I’ll code an advanced interface with python and asyncio in future. One third party package I did run across I kind of liked was vm-bhyve. To be honest only thing I liked about it was his directory layout, so that’s only thing I will try to keep consistent for our scenario.

Bhyve still needs a good interface to it, I think what would be best for it is python fastapi or C/Rust with kqueue as server end. Front end hands down flutter, will build on Linux, apache/nginx, android, IOS, windows and even our TVs in livingroom for a front end interface. Even on FreeBSD natively once Dart is ported to it. I wouldn’t even bother with Java or Kotlin for these reasons alone. I think if that was built for a year would turn FreeBSD into everyone’s favorite virtualization OS.

Back to sys admin stuff, let’s set this up once really well and we should be good for next 5-10 years, just clone the drive if moving to PCIE 4 or 5 machine this is not Linux 🙂

Setup Bhyve:

Let’s start with something simple, setup our directory structure like vm-bhyve, mod what we need and uninstall his package. I’m going to assume at this point you just have a FreeBSD release like FreeBSD 13.1 installed, so let’s begin….

https://github.com/churchers/vm-bhyve

pkg install vm-bhyve bhyve-firmware
zfs create -o mountpoint=/vm zroot/vm
sysrc vm_enable="YES"
sysrc vm_dir="zfs:pool/vm"
vm init
cp /usr/local/share/examples/vm-bhyve/* /vm/.templates/

OK great now we have a directory structure to work with, he likes to keep all guest related things in /vm/<guestname>, so let’s keep with his directory structure idea and delete the package now.

pkg delete vm-bhyve
nano /etc/rc.conf
(now edit /etc/rc.conf and remove/comment out his sysrc lines)
(while we in here let's add support for 4 guest tap interfaces)

#support for 4 test guests - substitute "ix0" for your own interface
cloned_interfaces="bridge0 tap0 tap1 tap2 tap3"
ifconfig_bridge0_name="br0"
ifconfig_br0="addm ix0 addm tap0 addm tap1 addm tap2 addm tap3"
(exit /etc/rc.conf)
#now let's just create what that does manually on command line for now
ifconfig bridge create
ifconfig tap0 create
ifconfig tap1 create
ifconfig tap2 create
ifconfig tap3 create
ifconfig bridge0 addm ix0 addm tap0 addm tap1 addm tap2 addm tap3
ifconfig bridge0 name br0
ifconfig br0 up
#damn starting to like /etc/rc.conf better already :)
(alright now let's edit /etc/sysctl.conf)
nano /etc/sysctl.conf (add following:)
#BHYVE
net.link.tap.up_on_open=1
#BYHVE + PF nat
net.inet.ip.forwarding=1
net.inet6.ip6.forwarding=1
(exit /etc/sysctl.conf)
nano /boot/loader.conf
(have it look like this:)
autoboot_delay="5"
kernels="kernel kernel.old"
boot_serial="YES"
#stuff added by install
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
cryptodev_load="YES"
zfs_load="YES"
#bhyve
vmm_load="YES"
nmdm_load="YES"
if_bridge_load="YES"
if_tap_load="YES"
(exit /boot/loader.conf)

At this point we should be able to do an “ifconfig” and see our bridge setup properly for 4 possible guests, just add more tap interfaces for additional guests. Now let’s reboot and make sure our bridge etc is still there:

shutdown -r now
ifconfig -a
#let's install a few packages to help us
pkg install screen tightvnc git bash

Moving to FreeBSD current with experimental suspend/resume support(MOVIE TIME!):

Ok the time consuming part, hope you have a lot of cores/fast CPUs 🙂

Alright even if you have current installed you still will not have support enabled so we are going to recompile kernel and userland for support. Honestly this is way to upgrade FreeBSD from source anytime, only difference on stable/release is I would checkout a different branch with git and be using GENERIC instead of GENERIC-NODEBUG to copy from. You may be thinking I’ll checkout a release branch and just compile in support, tried that, it won’t work properly, this is your only way to test it before 14 release where it actually works for most part.

The make buildworld, make buildkernel, make installworld commands will take a long time, I recommend going to play your favorite video game or watch a movie after executing those commands, return in a few hours to check in on them time to time. Honestly once you have the buildworld out of the way its not that bad after. I’d recommend running everything in a screen session as well, that way if something happens you can log back in and do a “screen -r”. After your done, return tomorrow and we will continue on with testing suspend and resume 🙂

screen
git clone https://git.FreeBSD.org/src.git /usr/src
cd /usr/src/sys/amd64/conf
(edit MYKERNEL) cp GENERIC-NODEBUG MYKERNEL (add: options         BHYVE_SNAPSHOT)
cd /usr/src
#(find amount of CPUs and adjust -j below - "dmesg|grep SMP")
make -j12 buildworld -DWITH_BHYVE_SNAPSHOT -DWITH_MALLOC_PRODUCTION
make -j12 buildkernel KERNCONF=MYKERNEL
make installkernel KERNCONF=MYKERNEL
shutdown -r now
cd /usr/src; make installworld
shutdown -r now
etcupdate -B
pkg bootstrap -f #if new freebsd version
pkg upgrade -f   #if new freebsd version
shutdown -r now

Preliminary Thoughts on Creating Guests:

We are back and ready to roll! So creating guests is probably something every sys admin sits there and thinks hours upon hours about. What performance improvements could I make, will I be able to mount /etc and /boot directory of an offline guest if anything goes wrong or you screw anything up on guest. How will you do backups for them, how will you recover from disaster if it strikes. These guests once setup can run 5-10 years, no room for error, everything has to be accounted for.

I will give you my take on this from decades of experience. When you are starting out fresh, your main concern is the main virtualization host is doing nothing but virtualization and routing. You’d don’t want apache, email servers or any type of server processes running on it that are better suited for guests that can crash if need be and not affect the main host and other guests running on it. You want the main host rock stable at all times. You can play with the guests and give them the memory and CPUs they require to run what they need.

On the main FreeBSD host – virtualization, DNS, isc-DHCP/isc-DHCPD6, RADVD, PF FIREWALL, and at most backups/ZFS snapshots. Over the years before FreeBSD had Bhyve, I offloaded all backups to ZFS guest for backups there as well, using rsync or zfs send/recv. The choice is yours, but my recommendation is you are running as little as possible on main host as far as internet services.

If you did it all right, you’ll find your barely doing anything on main host and always logged into guests instead, then you know you did everything right. You might edit PF firewall to block something from all guests time to time, or do updates, about it. You have to remember your main host will save you with disaster recovery on guests, create new guests, basically be the blood and soul of your system, and this is where FreeBSD shines.

If you have ability at all to run FreeBSD as main host, you’ll save yourself years of headaches, where every Linux sys admin is reinstalling a new release on main host every 2-4 years, you did a freebsd-update or rebuilt the world and went and watched a movie while other sys admins were pulling their hair out all week wishing they documented their configs better 🙂

What about mission critical? In this situation your going to learn a lot about ZFS and send/recv to clone guests on the fly. For every other situation, a simple rsync once a night of the /etc, /usr/local/etc, /boot, /root, /home directories is all you need, why waste space? I’m not going to clone a 100GB guest byte for byte, if something happens to that guest, I have all its configs, I’m good to go. Install a simple rsyncd.conf on each guest to backup its configs each night. Every host is in charge of all their guests backups to a directory. Then its your decision to do offline backups with rsync or zfs send/recv from that host.

So I know all the worries, been there done that. As I’m moving through this guide, I’m going to show you disaster mitigation techniques on the host as well, so your well prepared if disaster ever strikes on a guest.

MY RESEARCH:

To run FreeBSD effectively as a main host in production a lot had to be accounted for, and I will list them here:

  1. All guests can be resumed quickly after a main host reboots for kernel update after suspending all guests. No ssh connections lost to guests, no accidental forgetting to save work bites you on resumes of guests.
  2. If a guest fails to boot at anytime, mandatory mounting of /etc , /boot etc directories of guest to fix problems on host.
  3. If a guest runs out of space, ability to resize guest on host, and grow the FS on the guest afterwards with as little downtime as possible.
  4. Ability to suspend a guest quickly to fix any problems. With this typically you want to reboot guest properly and fix issues from host after that.
  5. FreeBSD only: ability to ZFS snapshot guests and roll them back to any previous snapshot if needed.

Bugs I found and possible remedies:

In my preliminary research I found what bhyve actually does is run itself if a while loop, if it exists with error code 0, then bhyve is run again, effectively a “shutdown -r now” working properly. If it exits with any other error code then loop is broken, everything can be cleaned up after guest. The problem I found with suspend/resume so far on this issue is once suspended it exits with error code 0 as well. There definitely should be a different error code attached. There is a progress bar written to screen on STDOUT before it does shut down, only way currently to differentiate between the two is to capture that output. Something that would be better left to python asyncio/fastapi server process keeping all guests in a loop and determining difference between a suspend and a clean exit, then you could have a separate command line utility to access API’s on server process would probably be the best solution right now. A coding exercise in python that shouldn’t take more than a week to code. A 6-12 month front end to that with GUI and flutter would probably be best overall supporting the most platforms from one code base.

On further research there are mainly 3 types of ways to pass disks to guests, virtio-blk, virtio-scsi and nvme. On my tests with suspend/resume virtio-blk is stable each time I ran it. On virtio-scsi I had issues, on resume I could do an “ls -al”, see the filesystem properly but after doing anything else like a “df -k” or logging into system, it would hang then eventually crash with error code 139. I reported this is freebsd-current mailing list and hopefully someone gets around to looking at it.

Further research on virtio-blk has shown that it is slowly being phased out for virtio-scsi. Apparently this is the new version of passing disks that will be more common in the future. The belief behind it stems from being to hard to rework the virtio-blk code as well as virtio-scsi has more features and ability to pass way more disks from a host to guests. As far as nvme, I did not have any to test with, regardless of SSD type I believe the move will attempt to include all SSD/nvme to one virtio-scsi configuration in the future so users should embrace virtio-scsi once it is stable with suspend/resume on FreeBSD.

Upon further testing of attempting to mount a ZFS guest to the main host, it was unstable with no plans to fix it. Upon contacting freebsd-current mailing list about this issue, I was informed it causes deadlocks due to lock recursion and is why the sysctl vfs.zfs.vol.recursive was turned off by default. Suggestion was to use scsi instead, which in my testing did mount the ZFS guest without issues, a further suggestion that virtio-scsi is the future.

Upon examining the C source code to the virtio-scsi driver, it creates /dev/cam/* devices that can be used for the guests to passthrough targets with setup luns on the host. 

router:/root # ls -al /dev/cam/ctl*
crw------- 1 root operator 0, 164 Nov 10 02:29 /dev/cam/ctl
crw------- 1 root operator 0, 167 Nov 10 08:14 /dev/cam/ctl1.0
crw------- 1 root operator 0, 168 Nov 10 08:14 /dev/cam/ctl2.0
router:/root #

What happens here is a port is created that can be used in the virtio-scsi line of a guests config to pass devices. Upon attaching to targets, the luns create /dev/da* devices. 

router:/root # ls -al /dev/da*
crw-r----- 1 root operator 0, 134 Nov 10 02:29 /dev/da0
crw-r----- 1 root operator 0, 135 Nov 10 02:29 /dev/da0s1
crw-r----- 1 root operator 0, 136 Nov 10 02:29 /dev/da0s2
crw-r----- 1 root operator 0, 138 Nov 10 02:29 /dev/da0s2a
crw-r----- 1 root operator 0, 181 Nov 10 08:14 /dev/da1
crw-r----- 1 root operator 0, 183 Nov 10 08:14 /dev/da1p1
crw-r----- 1 root operator 0, 184 Nov 10 08:14 /dev/da1p2
crw-r----- 1 root operator 0, 185 Nov 10 08:14 /dev/da1p3
crw-r----- 1 root operator 0, 182 Nov 10 08:14 /dev/da2
crw-r----- 1 root operator 0, 186 Nov 10 08:14 /dev/da2p1
crw-r----- 1 root operator 0, 187 Nov 10 08:14 /dev/da2p2
crw-r----- 1 root operator 0, 188 Nov 10 08:14 /dev/da2p3
crw-r----- 1 root operator 0, 206 Nov 10 08:14 /dev/da3
crw-r----- 1 root operator 0, 207 Nov 10 08:14 /dev/da3p1
crw-r----- 1 root operator 0, 208 Nov 10 08:14 /dev/da3p2
router:/root #

The first one /dev/da0 is reserved for scsi itself, every other lun created in a target creates a new /dev/da* device, first one beginning at /dev/da1 and so forth.

For my purposes I had the ZFS guest as first lun in my test and was able to manipulate /dev/da1 successfully to mount the ZFS guest. I could also pass a target to a guest with as many disks/cdroms/ISOs(luns) as I wanted just by passing the /dev/cam/ctl1.0 target.

Let’s do a quick illustration of passing zvols in FreeBSD to scsi instead, you could also pass disk images if you wanted:

nano /etc/ctl.conf:
(add following:)
portal-group pg0 {
        discovery-auth-group no-authentication
        listen 127.0.0.1:3260
}
target iqn.2005-02.com.sunsaturn:target0 {
        auth-group no-authentication
        portal-group pg0
        #bhyve virti-iscsi disk - /dev/cam/ctl1.0
        port ioctl/1
        lun 0 {
                path /dev/zvol/zroot/asterisk
                #blocksize 128
                serial 000c2937247001
                device-id "iSCSI Disk 000c2937247001"
                option vendor "FreeBSD"
                option product "iSCSI Disk"
                option revision "0123"
                option insecure_tpc on
        }
}
target iqn.2005-02.com.sunsaturn:target1 {
        auth-group no-authentication
        portal-group pg0
        #bhyve virti-iscsi disk - /dev/cam/ctl2.0
        port ioctl/2

        lun 0 {
                path /dev/zvol/zroot/asterisk2
                #blocksize 128
                serial 000c2937247002
                device-id "iSCSI Disk 000c2937247002"
                option vendor "FreeBSD"
                option product "iSCSI Disk"
                option revision "0123"
                option insecure_tpc on
        }
        lun 1 {
                path /vm/.iso/FreeBSD-14.0-CURRENT-amd64-20221103-5cc5c9254da-259005-disc1.iso
                #byhve seems to just hang when I set it to an actual CDROM so let it default to type 0
                #device-type 5
                serial 000c2937247003
                device-id "iSCSI CDROM ISO 000c2937247003"
                option vendor "FreeBSD CDROM"
                option product "iSCSI CDROM"
                option revision "0123"
                option insecure_tpc on
        }
}

(close and exit)
nano /etc/iscsi.conf
(add following:)
t0 {
        TargetAddress   = 127.0.0.1:3260
        TargetName      = iqn.2005-02.com.sunsaturn:target0
}
t1 {
        TargetAddress   = 127.0.0.1:3260
        TargetName      = iqn.2005-02.com.sunsaturn:target1
}
(close and exit)
nano /etc/rc.conf
(add following:)
#ISCSI - service ctld start && service iscsid start
#server
ctld_enable="YES"          #load /etc/ctl.conf
iscsid_enable="YES"        #start iscsid process to connect to ctld
#client - service iscsictl start
iscsictl_enable="YES"      #connect to all targets in /etc/iscsi.conf
iscsictl_flags="-Aa"

(close and exit)
(now let's create some zvols to install guests on)
zfs create -V30G -o volmode=dev zroot/asterisk
zfs create -V30G -o volmode=dev zroot/asterisk2
cd /vm/.iso
wget https://download.freebsd.org/snapshots/amd64/amd64/ISO-IMAGES/14.0/FreeBSD-14.0-CURRENT-amd64-20221103-5cc5c9254da-259005-disc1.iso
(let's start scsi manually)
service ctld start && service iscsid start && service iscsictl start

Now we can see all those /dev/da* devices for us to manipulate on host for mounting shut down guests, as well as those /dev/cam/* devices for passing to guests.

Next let’s actually install 2 test guests, asterisk and asterisk2. For asterisk guest I will install UFS FreeBSD, and for asterisk2 I will install ZFS guest as well as pass the FreeBSD ISO to that guest.

To make this easier, let’s use a bash script I quickly coded to test suspend/resume capabilities of each guest, let’s call them asterisk.sh and asterisk2.sh:

cd /root
nano -w asterisk.sh
(add following)
#!/bin/bash
#
# General script to test bhyve suspend/resume features
#
# Requirements: FreeBSD current
# screen
# git clone https://git.FreeBSD.org/src.git /usr/src
# cd /usr/src/sys/amd64/conf (edit MYKERNEL) cp GENERIC MYKERNEL-NODEBUG; (add: options         BHYVE_SNAPSHOT)
# cd /usr/src
# (find amount of CPUs and adjust -j below - "dmesg|grep SMP")
# make -j12 buildworld -DWITH_BHYVE_SNAPSHOT -DWITH_MALLOC_PRODUCTION
# make -j12 buildkernel KERNCONF=MYKERNEL
# make installkernel KERNCONF=MYKERNEL
# shutdown -r now
# cd /usr/src; make installworld
# shutdown -r now
# etcupdate -B
# pkg bootstrap -f #if new freebsd version
# pkg upgrade -f   #if new freebsd version 
#
# Report anomolies to dan@sunsaturn.com

##############EDIT ME#####################


HOST="127.0.0.1"                        # vncviewer 127.0.0.1:5900 - pkg install tightvnc
PORT="5900"
WIDTH="800"
HEIGHT="600"
VMNAME="asterisk"
ISO="/vm/.iso/FreeBSD-14.0-CURRENT-amd64-20221103-5cc5c9254da-259005-disc1.iso"
DIR="/vm/asterisk"                      # Used to hold files when guest suspended
SERIAL="/dev/nmdm_asteriskA"           # For "screen /dev/nmdm_asteriskB" - pkg install screen
TAP="tap0"
CPU="8"
RAM="8G"

#For testing virtio-scsi
STORAGE="/dev/cam/ctl1.0"               # port from /etc/ctl.conf(port ioctl/1) - core dumping on resume
DEVICE="virtio-scsi"

#for testing virtio-blk                 # Comment out above 2 lines if using these
#DEVICE="virtio-blk"                    
#STORAGE="/dev/zvol/zroot/asterisk"     # Standard zvol
#STORAGE="/dev/da1"                     # Block device created from iscsictl

#########################################

usage() {
   echo "Usage: $1 start    (Start the guest: $VMNAME)"; 
   echo "Usage: $1 stop     (Stop the guest: $VMNAME)"; 
   echo "Usage: $1 resume   (Resume the guest from last suspend: $VMNAME)"; 
   echo "Usage: $1 suspend  (Suspend the guest: $VMNAME)"; 
   echo "Usage: $1 install  (Install new guest: $VMNAME)"; 
   exit
}

if [ ! -d "$DIR" ]; then 
   mkdir -p $DIR
fi

#if [ -z "$2" ]; then
#   usage
#else
#   VMNAME=$2
#fi


if [ "$1" == "install" ]; then
   #Kill it before starting it
   echo "Execute: screen $SERIAL"
   bhyvectl --destroy --vm=$VMNAME
   bhyve -c $CPU -m $RAM -w -H -A \
      -s 0:0,hostbridge \
      -s 3:0,ahci-cd,$ISO \
      -s 4:0,$DEVICE,$STORAGE  \
      -s 5:0,virtio-net,$TAP \
      -s 29,fbuf,tcp=$HOST:$PORT,w=$WIDTH,h=$HEIGHT \
      -s 30,xhci,tablet \
      -s 31,lpc -l com1,stdio \
      -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
      $VMNAME
   #kill it after 
   bhyvectl --destroy --vm=$VMNAME
elif [ "$1" == "start" ]; then 
   while true
   do
      echo "Starting $VMNAME -s 29,fbuf,tcp=$HOST:$PORT,w=$WIDTH,h=$HEIGHT"
      #Kill it before starting it
      bhyvectl --destroy --vm=$VMNAME > /dev/null 2>&1
      bhyve -c $CPU -m $RAM -w -H -A \
         -s 0:0,hostbridge \
         -s 4:0,$DEVICE,$STORAGE  \
         -s 5:0,virtio-net,$TAP \
         -s 29,fbuf,tcp=$HOST:$PORT,w=$WIDTH,h=$HEIGHT \
         -s 30,xhci,tablet \
         -s 31,lpc -l com1,$SERIAL \
         -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
         $VMNAME
      #DISABLING REBOOT LOOP AS SUSPEND RETURNS ERROR CODE 0 AS WELL
      #if [ "$?" != 0 ];
      #then
      #   echo "The exit code was not reboot code 0!: $?"
      #   exit
      #fi
      echo "The exit code was : $?"
      exit
   done
elif [ "$1" == "resume" ]; then 
   while true
   do
      echo "Starting $VMNAME -s 29,fbuf,tcp=$HOST:$PORT,w=$WIDTH,h=$HEIGHT"
      #Kill it before starting it
      bhyvectl --destroy --vm=$VMNAME > /dev/null 2>&1
      if [ -f "$DIR/default.ckp" ]; then
         bhyve -c $CPU -m $RAM -w -H -A \
            -s 0:0,hostbridge \
            -s 4:0,$DEVICE,$STORAGE  \
            -s 5:0,virtio-net,$TAP \
            -s 29,fbuf,tcp=$HOST:$PORT,w=$WIDTH,h=$HEIGHT \
            -s 30,xhci,tablet \
            -s 31,lpc -l com1,$SERIAL \
            -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
            -r $DIR/default.ckp \
            $VMNAME
      else
         echo "Guest was never suspended"
         exit
      fi
      #DISABLING REBOOT LOOP AS SUSPEND RETURNS ERROR CODE 0 AS WELL
      #if [ "$?" != 0 ];
      #then
      #   echo "The exit code was not reboot code 0!: $?"
      #   exit
      #fi
      echo "The exit code was : $?"
      exit
   done
elif [ "$1" == "suspend" ];
then 
   bhyvectl --suspend $DIR/default.ckp --vm=$VMNAME

elif [ "$1" == "stop" ]; then 
   bhyvectl --destroy --vm=$VMNAME 
else 
   usage
fi

Let’s also create asterisk2.sh:

#!/bin/bash
#
# General script to test bhyve suspend/resume features
#
# Requirements: FreeBSD current
# screen
# git clone https://git.FreeBSD.org/src.git /usr/src
# cd /usr/src/sys/amd64/conf (edit MYKERNEL) cp GENERIC MYKERNEL-NODEBUG; (add: options         BHYVE_SNAPSHOT)
# cd /usr/src
# (find amount of CPUs and adjust -j below - "dmesg|grep SMP")
# make -j12 buildworld -DWITH_BHYVE_SNAPSHOT -DWITH_MALLOC_PRODUCTION
# make -j12 buildkernel KERNCONF=MYKERNEL
# make installkernel KERNCONF=MYKERNEL
# shutdown -r now
# cd /usr/src; make installworld
# shutdown -r now
# etcupdate -B
# pkg bootstrap -f #if new freebsd version
# pkg upgrade -f   #if new freebsd version 
#
# Report anomolies to dan@sunsaturn.com

##############EDIT ME#####################


HOST="127.0.0.1"                        # vncviewer 127.0.0.1:5900 - pkg install tightvnc
PORT="5901"
WIDTH="800"
HEIGHT="600"
VMNAME="asterisk2"
ISO="/vm/.iso/FreeBSD-14.0-CURRENT-amd64-20221103-5cc5c9254da-259005-disc1.iso"
DIR="/vm/asterisk2"                      # Used to hold files when guest suspended
SERIAL="/dev/nmdm_asterisk2A"           # For "screen /dev/nmdm_asterisk2B" - pkg install screen
TAP="tap1"
CPU="8"
RAM="8G"

#For testing virtio-scsi
STORAGE="/dev/cam/ctl2.0"               # port from /etc/ctl.conf(port ioctl/1) - core dumping on resume
DEVICE="virtio-scsi"

#for testing virtio-blk                 # Comment out above 2 lines if using these
#DEVICE="virtio-blk"                    
#STORAGE="/dev/zvol/zroot/asterisk2"     # Standard zvol
#STORAGE="/dev/da2"                     # Block device created from iscsictl

#########################################

usage() {
   echo "Usage: $1 start    (Start the guest: $VMNAME)"; 
   echo "Usage: $1 stop     (Stop the guest: $VMNAME)"; 
   echo "Usage: $1 resume   (Resume the guest from last suspend: $VMNAME)"; 
   echo "Usage: $1 suspend  (Suspend the guest: $VMNAME)"; 
   echo "Usage: $1 install  (Install new guest: $VMNAME)"; 
   exit
}

if [ ! -d "$DIR" ]; then 
   mkdir -p $DIR
fi

#if [ -z "$2" ]; then
#   usage
#else
#   VMNAME=$2
#fi


if [ "$1" == "install" ]; then
   #Kill it before starting it
   echo "Execute: screen $SERIAL"
   bhyvectl --destroy --vm=$VMNAME
   bhyve -c $CPU -m $RAM -w -H -A \
      -s 0:0,hostbridge \
      -s 3:0,ahci-cd,$ISO \
      -s 4:0,$DEVICE,$STORAGE  \
      -s 5:0,virtio-net,$TAP \
      -s 29,fbuf,tcp=$HOST:$PORT,w=$WIDTH,h=$HEIGHT \
      -s 30,xhci,tablet \
      -s 31,lpc -l com1,stdio \
      -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
      $VMNAME
   #kill it after 
   bhyvectl --destroy --vm=$VMNAME
elif [ "$1" == "start" ]; then 
   while true
   do
      echo "Starting $VMNAME -s 29,fbuf,tcp=$HOST:$PORT,w=$WIDTH,h=$HEIGHT"
      #Kill it before starting it
      bhyvectl --destroy --vm=$VMNAME > /dev/null 2>&1
      bhyve -c $CPU -m $RAM -w -H -A \
         -s 0:0,hostbridge \
         -s 4:0,$DEVICE,$STORAGE  \
         -s 5:0,virtio-net,$TAP \
         -s 29,fbuf,tcp=$HOST:$PORT,w=$WIDTH,h=$HEIGHT \
         -s 30,xhci,tablet \
         -s 31,lpc -l com1,$SERIAL \
         -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
         $VMNAME
      #DISABLING REBOOT LOOP AS SUSPEND RETURNS ERROR CODE 0 AS WELL
      #if [ "$?" != 0 ];
      #then
      #   echo "The exit code was not reboot code 0!: $?"
      #   exit
      #fi
      echo "The exit code was : $?"
      exit
   done
elif [ "$1" == "resume" ]; then 
   while true
   do
      echo "Starting $VMNAME -s 29,fbuf,tcp=$HOST:$PORT,w=$WIDTH,h=$HEIGHT"
      #Kill it before starting it
      bhyvectl --destroy --vm=$VMNAME > /dev/null 2>&1
      if [ -f "$DIR/default.ckp" ]; then
         bhyve -c $CPU -m $RAM -w -H -A \
            -s 0:0,hostbridge \
            -s 4:0,$DEVICE,$STORAGE  \
            -s 5:0,virtio-net,$TAP \
            -s 29,fbuf,tcp=$HOST:$PORT,w=$WIDTH,h=$HEIGHT \
            -s 30,xhci,tablet \
            -s 31,lpc -l com1,$SERIAL \
            -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
            -r $DIR/default.ckp \
            $VMNAME
      else
         echo "Guest was never suspended"
         exit
      fi
      #DISABLING REBOOT LOOP AS SUSPEND RETURNS ERROR CODE 0 AS WELL
      #if [ "$?" != 0 ];
      #then
      #   echo "The exit code was not reboot code 0!: $?"
      #   exit
      #fi
      echo "The exit code was : $?"
      exit
   done
elif [ "$1" == "suspend" ];
then 
   bhyvectl --suspend $DIR/default.ckp --vm=$VMNAME

elif [ "$1" == "stop" ]; then 
   bhyvectl --destroy --vm=$VMNAME 
else 
   usage
fi

Great now let’s install the guests:

chmod 755 *.sh
./asterisk.sh install
(at this point just install FreeBSD with default UFS options, I actually chose to remove swap partition at the end, and the root partition then add them back as swap 2nd and root partition last to avoid headaches having to grow the disk later on, I suggest you do the same)
./asterisk2.sh install (install with GPT+UEFI)
(install FreeBSD again with default ZFS install, there is no headache here partitions are perfect here as defaults on zroot)

(At this point I may edit /etc/fstab to change whatever hardcoded device names
are in there to GPT names from /dev/gpt/* so when we switch between virtio-scsi and virtio-blk devices it won't matter. Ie for swap switch it to:
/dev/gpt/swap0 instead) If you are ever wanting to show what the GPT label names are you can either do "gdisk -l <device>" or "gpart show -l <device>" to figure out what to put into /etc/fstab. If they have no label , give them one. )

#now let's start them both after the install
./asterisk.sh start
#another terminal
./asterisk2.sh start
#another terminal
screen /dev/nmdm_asteriskB
#another terminal
screen /dev/nmdm_asterisk2B
#another terminal
./asterisk.sh suspend
./asterisk.sh resume
#another terminal
./asterisk2.sh suspend
./asterisk2.sh resume

Now you will notice on your screen session, after resuming guest “ls” etc works but as soon as we do anything else, “df -h” it will hang and after about a minute it will core dump.

router:/root # ./asterisk2.sh resume
Starting asterisk2 -s 29,fbuf,tcp=127.0.0.1:5901,w=800,h=600
fbuf frame buffer base: 0x229792600000 [sz 16777216]
Pausing pci devs...
pci_pause: no such name: virtio-blk
pci_pause: no such name: ahci
pci_pause: no such name: ahci-hd
pci_pause: no such name: ahci-cd
Restoring vm mem...
[8192.000MiB / 8192.000MiB] |################################################################################################################################################|
Restoring pci devs...
vm_restore_user_dev: Device size is 0. Assuming virtio-blk is not used
vm_restore_user_dev: Device size is 0. Assuming virtio-rnd is not used
vm_restore_user_dev: Device size is 0. Assuming e1000 is not used
vm_restore_user_dev: Device size is 0. Assuming ahci is not used
vm_restore_user_dev: Device size is 0. Assuming ahci-hd is not used
vm_restore_user_dev: Device size is 0. Assuming ahci-cd is not used
Restoring kernel structs...
Resuming pci devs...
pci_resume: no such name: virtio-blk
pci_resume: no such name: ahci
pci_resume: no such name: ahci-hd
pci_resume: no such name: ahci-cd
./asterisk2.sh: line 145: 10883 Segmentation fault      (core dumped) bhyve -c $CPU -m $RAM -w -H -A -s 0:0,hostbridge -s 4:0,$DEVICE,$STORAGE -s 5:0,virtio-net,$TAP -s 29,fbuf,tcp=$HOST:$PORT,w=$WIDTH,h=$HEIGHT -s 30,xhci,tablet -s 31,lpc -l com1,$SERIAL -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -r $DIR/default.ckp $VMNAME
The exit code was : 139
router:/root #

Great so scsi is unstable with suspend/resume. Now run same tests but let’s modify asterisk2.sh script to use virtio-blk instead with /dev/da2 and run same tests:

#For testing virtio-scsi
#STORAGE="/dev/cam/ctl2.0"              # port from /etc/ctl.conf(port ioctl/1) - core dumping on resume
#DEVICE="virtio-scsi"

#for testing virtio-blk                 # Comment out above 2 lines if using these
DEVICE="virtio-blk"
#STORAGE="/dev/zvol/zroot/asterisk2"    # Standard zvol
STORAGE="/dev/da2"                      # Block device created from iscsictl

You may be wondering how I know which /dev/da* asterisk2 is using:
I simply run:
iscsictl 
#This will give you a list of who is on what, it can be completely random
#always check this list before mounting a guest so you don't mount wrong one
#typically everything on lun0 will be numbered first, then lun1 but this is not #always the case so make sure to run that
#on a non-testing host it may look something like this:
router:/root # iscsictl 
Target name                          Target portal    State
iqn.com.sunsaturn.asterisk:target1   127.0.0.1:3260   Connected: da1 da5 
iqn.com.sunsaturn.rocky:target2      127.0.0.1:3260   Connected: da2 da6 
iqn.com.sunsaturn.ubuntu:target3     127.0.0.1:3260   Connected: da3 da8 
iqn.com.sunsaturn.windows:target4    127.0.0.1:3260   Connected: da4 da7 
router:/root # 

Here is something good to add to your .bash_profile then you don't have to 
think about it ever again:

if [ "$HOSTNAME" == "test.test.com" ]; then
   echo "Checking which devices guests connected to:"
   echo "#######################################################"
   iscsictl
   echo "#######################################################"
fi

I personally have a 2nd root account I set to bash so I don't touch default root
shell, can use toor if you like. I suggest everyone do that, there will come a day where you are like damn I can't remember my root password anymore because been using ssh keys so long. Yes you have to su to root everytime, but I even automate that these days.

Now let’s run it:

./asterisk2.sh start
#another terminal (make sure watching screen terminal running these)
./asterisk2.sh suspend
./asterisk2.sh resume

WORKS PERFECTLY:
Now go uncomment STORAGE line with 
STORAGE="/dev/zvol/zroot/asterisk2"
and comment out:
#STORAGE="/dev/da2"
WORKS PERFECTLY TO

So what can we see from suspend/resume stability, it works on virtio-blk perfectly. No matter if I use the /dev/da* devices from scsi or the zvols directly.

What about importing ZFS pool from asterisk2? Sure let’s do it on host, shut it down first:

router:/root # gpart show /dev/da2
=>      40  62914480  da2  GPT  (30G)
        40    532480    1  efi  (260M)
    532520      2008       - free -  (1.0M)
    534528  16777216    2  freebsd-swap  (8.0G)
  17311744  45600768    3  freebsd-zfs  (22G)
  62912512      2008       - free -  (1.0M)

router:/root # gdisk -l /dev/da2
GPT fdisk (gdisk) version 1.0.9

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.
Disk /dev/da2: 62914560 sectors, 30.0 GiB
Sector size (logical): 512 bytes
Disk identifier (GUID): 8ACD2112-5EBF-11ED-8F56-00A098E3C14E
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 40, last usable sector is 62914519
Partitions will be aligned on 8-sector boundaries
Total free space is 4016 sectors (2.0 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1              40          532519   260.0 MiB   EF00  efiboot0
   2          534528        17311743   8.0 GiB     A502  swap0
   3        17311744        62912511   21.7 GiB    A504  zfs0
router:/root #

So we can see this is definitely our ZFS guest, and we already know we cannot manipulate /dev/zvol/zroot/asterisk2 directly because of recursion issues, but we can manipulate /dev/da2 through scsi just fine:

#Import ZFS guest under /mnt (shut down the guest)
#First import it under /mnt under a different temporary name that won't save the name when we export it after with "-t" option
zpool import (we should see asterisk2's zroot ready for import)
zpool import -fR /mnt -t zroot testing
zfs mount -o mountpoint=/mnt testing/ROOT/default
(go ahead fix /mnt/etc /mnt/boot problems)
zpool export testing
rm -rf /mnt/* (cleanup left over directories)

#hell let's mount asterisk guest with UFS as well on /mnt
#I did crash it a few times so probably need recovery
router:/root # gpart show /dev/da1
=>       40  104857520  da1  GPT  (50G)
         40         24       - free -  (12K)
         64     532480    1  efi  (260M)
     532544   16777216    2  freebsd-swap  (8.0G)
   17309760   87547776    3  freebsd-ufs  (42G)
  104857536         24       - free -  (12K)

router:/root # mount -t ufs /dev/da1p3 /mnt
mount: /dev/da1p3: R/W mount of / denied. Filesystem is not clean - run fsck. Forced mount will invalidate journal contents: Operation not permitted
router:/root # fsck /dev/da1p3 
** /dev/da1p3
** SU+J Recovering /dev/da1p3

USE JOURNAL? [yn] y

** Reading 182419456 byte journal from inode 4.

RECOVER? [yn] y

** Building recovery table.
** Resolving unreferenced inode list.
** Processing journal entries.

WRITE CHANGES? [yn] y


***** FILE SYSTEM IS CLEAN *****
** 96 journal records in 6656 bytes for 46.15% utilization
** Freed 27 inodes (4 dirs) 0 blocks, and 20 frags.

***** FILE SYSTEM MARKED CLEAN *****
router:/root # mount -t ufs /dev/da1p3 /mnt
router:/root # ls /mnt
bin  boot  COPYRIGHT  dev  entropy  etc  home  lib  libexec  media  mnt  net  proc  rescue  root  sbin  sys  tmp  usr  var
router:/root #

Summary:

Suspend/resume does work, just not passing in virtio-scsi with those /dev/cam/* devices. Seems we can do “ls” just fine, so something else is going on.

For now what you can do is just use the virtio-blk devices, until virtio-scsi works. In fact I would set it up this way since we already know virtio-blk is on its way out, plus it was so much nicer to pass in a CDROM on asterisk2 in just 1 line to guest, or as many disks as we want. We learned we can use /dev/da* devices for direct ZFS importing of guests pools for disaster recovery, for anything that isn’t ZFS we could also use the /dev/zvol/zroot/* devices directly if we wanted, but it is a cool work around from FreeBSD team, and sets you up using virtio-scsi now.

If you do not care about suspend/resume right now you could just continue using virtio-scsi to future proof yourself for virtio-blk phase out, or you could just pass virtio-blk devices temporarily till it is fixed and use suspend/resume all you want. My personal preference till this gets sorted out is leave iscsi processes running till it gets resolved, and use virtio-blk directly against /dev/zvol/root/<guestname> for now, can switch them later, this way you can use suspend/resume all you want as well as mount any ZFS guests through the /dev/da* devices anytime you need. This way your at least future proofed. If you don’t want to use the /dev/da* devices :

zfs set volmode=full zroot/asterisk
(if you want to be able to mount non-ZFS guests directly at:
/dev/zvol/zroot/asterisk for instance, what this will do is separate asterisk into asteriskp1 asteriskp2 asteriskp3 etc for your mounting purposes)
#Just get used to scsi already :) I know old habits die hard and no one wants to #give up NIS either and learn LDAP, hell neither do I, whatever works :)

Hope you enjoyed this article on suspend/resume experimental support for FreeBSD, perhaps in another article I will show you Linux and Windows guests, and how we can disaster recovery them as well like we did for these 2 UFS and ZFS guests today, till then….

Shine bright like a diamond,

Dan.

PCIE 5.0 is it worth it?

PCIE, PCIE 5

If you just built a new PCIE 4.0 system is PCIE 5.0 worth it? At this time with a recession looming, food prices high, gas prices high, I would say definitely not. Your money would be better spent on solar panels, lithium batteries and inverters, something you could eventually get your money back on.

The CONS:

So just how good is PCIE 5.0. It doubles the lane speeds once again, but here is the thing, no one has even put out a graphics card that can saturate a PCIE 3 x 16 lane, your PC won’t boot any faster on PCIE 5.0. Here is the other thing, the 2nm fabs are being set to go into production at end of 2024. Intel says it will have 1.5nm at that time, we will see. I think a better upgrade date would be end of 2025 if you already have a PCIE 4 system, giving enough time for motherboards and chips to come out for the 2nm fabrication process.

Also the good motherboards are not out yet for PCIE 5.0(at least the good ones). I expect they will start showing up around end of 2023.

The PROS:

With an extra 1gz of CPU speed that may help people’s compile times with things like app development and flutter. For a desktop, the extra SSD speed won’t even be noticeable compared to your PCIE 4.0 system. Where it would make sense is companies that depend on databases to serve their customers, the added speedup of searching millions of rows in a database would be a welcomed improvement. For gamers, graphics cards can’t even saturate PCIE 4 yet, not even worth it other than for bragging rights.

So basically for a server always, but a desktop it just isn’t there yet. Video card manufacturers cared more about crypto miners and their bottom line during pandemic, now they have some catching up to do and price drops to do for their gaming clients to make up for it.

RECAP:

All in all, if you have money to blow, it would definitely be a fun upgrade for the desktop, but that’s all it would be. I’d wait it out till the 2nm fabrication process comes out, Tailand expected to get one of the ASML machines sometime in 2024. Also I expect in next year or 2 we might finally have a quantum PCIE card to try.

The quantum world isn’t even going to start till they can get a quantum card in everyone’s computer so all the developers around the world can start programming for them, so engineers are going to have to step it up a notch in next couple years and get a product to market, as the technology will be useless for a year or 2 till developers around the world can make it useful.

I also think how artificial intelligence is done now, a really lame quantum physics algorithm on using neural nets to do probabilities will be a thing of the past as well, and that will be redone with quantum for real AI. It really is such a joke that even physicists put funny notes out there doors saying, “You are probably here”. I admit does crack me up a bit, if you understand how physicists think with observables and probabilities 🙂 I just can’t see in 2030 people still doing dumb things like still multiplying matrixes together to do AI, or it would be a sad world.

Think one of my saddest and happiest moments this year was talking with NASA, and telling them how behind they are, they have not even come up with a theory for faster than light speed travel yet. Granted a month after I did that now a theory exists, but that’s all it will remain for awhile now. Honestly its difficult what they do, I’d need to smoke a lot of weed to to come up with a practical way to solve that one, but at least they are somewhat on right track now.

Unfortunately NASA is now pre-occupied with certain asteroids that could end humanity, understandable, I would be to! Here is the thing though, without faster than light speed travel, is humanity really ever going to make it to a new exoplanet millions of light years away, there will always be a new asteroid out there coming to wipe us out like the dinosaurs. I think perhaps our best solution as a human race is yes to take out asteroids that may wipe us out in next century, but at same time develop the technology to travel faster than light speed otherwise we will just always be fighting the inevitable.

Also another thing we have to consider in 2022 with the new James Webb Telescope, is that the big bang theory is now disproven. Yes you heard that right, it is. So back to drawing board for astrophysicists.

2020-2030 is probably the best decade to be alive right now, we will see the most innovations in this time period than humanity has ever seen, as long as we don’t have to keep fighting with INTEL keeping Moore’s law going 🙂 At this point I think we are all tired of companies like ASML taking years to build a machine to keep that law going, hopefully quantum hits soon. I mean what are they going to do after the 2nm fabrication process, probably go to pinta units, it will never end, bring quantum out!

Speaking of quantum, I do see a downside already. The benefit will be enormous for humanity as a whole, but we need to regulate those pharmaceutical companies. I do not want to see humanity suffering because a bunch of biochemists played with quantum computers and CRISPR, then charge a fortune for their cures. That is like stealing from the open source world and not giving back. They never even would have had those quantum computers if it wasn’t for everyone that built it for them in first place. I think world needs to start an opensource project ASAP to combat this. Governments should pay them a one time fee for formula/algorithm how to produce it in their own labs, and governments should hand that over to open source project.

I really see a world in the future where everyone is trained in physics, engineering, programming and biochemistry with CRISPR. These should be the school curriculum for kids these days. When this does happen, and you get sick one day, even your friend next door could formulate the drugs for you to keep you alive or give you a CRISPR needle to modify your DNA so you are cured.

Make it fun for them, teach them how to setup a solar panel, teach them to program an app they would love world to have or make a robot do something. Teach them about DNA, RNA and CRISPR in a fun way. One day a generation is going to have to be so smart they can go underground with geothermal energy when Sun is ready to engulf the earth or asteroid hits first(most likely). Best thing we can do is give them the knowledge we should all have right now, but most don’t. A professor at a university once told me, “Never spoon feed anyone, only give them the tools and let them figure it out on their own”. Sound advice for the future, YouTube, free library books on their tablets, online shopping, the world is their oyster. I’ve done that as a university teacher myself in the past, given back to them. They loved it! Need any help rewriting their curriculums, give me a shout.

Shine bright like a diamond,

SunSaturn.

Rocky Linux Install 2022 with KVM support

kvm, linux

INTRO:

Originally I had tried a software update on a Dell 2950 III server from Rocky Linux 8 to 9, only to end up with rocky linux , “glibc error: cpu does not support x86-64-v2”. Basically I fried my entire system as this CPU cannot support these new CPU calls.

Today I am going to walk you through a complete Rocky Linux 2022 install, which now replaces Centos from original Centos creator. Today I put in three different USB sticks containing Oracle Linux, Rocky Linux and Centos Stream Linux to see if I could install over my existing partitions as to not wipe out my FreeBSD KVM on the LVM. Well today I am sad to say, the installer does not see the LVM partitions, so I am forced to reinstall Rocky Linux as well as my guests all over again.

I am going to walk you through a safer way to install Centos(Rocky Linux), to future proof yourself in case of reinstalls or problems.

Rocky Linux 8 vs 9:

Run the following shell script:

pico glibc_check.sh (add following)

#!/usr/bin/awk -f

BEGIN {
while (!/flags/) if (getline < "/proc/cpuinfo" != 1) exit 1
if (/lm/&&/cmov/&&/cx8/&&/fpu/&&/fxsr/&&/mmx/&&/syscall/&&/sse2/) level = 1 
if (level == 1 && /cx16/&&/lahf/&&/popcnt/&&/sse4_1/&&/sse4_2/&&/ssse3/) level = 2 
if (level == 2&&/avx/&&/avx2/&&/bmi1/&&/bmi2/&&/f16c/&&/fma/&&/abm/&&/movbe/&&/xsave/) level = 3 
if (level == 3 && /avx512f/&&/avx512bw/&&/avx512cd/&&/avx512dq/&&/avx512vl/) level = 4 
if (level > 0) { print "CPU supports x86-64-v" level; exit level + 1 }
exit 1
}

Save it then:

chmod +x glibc_check.sh; ./glibc_check.sh

host:~ # ./glibc_check.sh
CPU supports x86-64-v1
host:~ #

Now if you get like the above with only version 1, you can only install Rocky Linux 8, otherwise if you have version 2 or higher you can install Rocky Linux 9. I believe reason RHEL made this change was to make glibc calls faster.

INSTALL Rocky Linux:

Download latest Rocky Linux ISO and install it with rufus to a USB stick. I am not going to cover a simple graphics installer, but I want to cover partitioning your drive. Let’s go to “Installation Destination” on installer. Now to prevent any future wipeouts of LVMs with installers, we are going to use standard partitions and only install our LVM afterwards manually. This way if we every run into situation again where installer cannot see into our LVM install, it will definitely see our standard partitions and we won’t have to wipe out our LVMs ever again.

Here are my recommendations for /boot, / and swap. For /boot from experience 1 GB is not enough like they say, after you start installing enough kernels, or start adding custom kernels for supporting things like ZFS or Ksmbd, it adds up quickly. So my recommendation for /boot is 3 GB. For swap it is 20% of your memory, I have 32 GB on this server, but I am going to go 8GB, as I rarely like going under that these days. For / partition we will want at least 80-100 GB. At some point you are going to run out of space un-tarring enough kernel sources, or doing 4k file tests, or space for your ISOs, you need some breathing room on your main OS!

(all ext4 standard partitions setup as follows)

/boot – sda1 – 3GB
swap – sda2 – 8GB
/ – sda3 – 80GB

Save with this setup and finish your install finishing up your install packages, network, password and so on. What we are going to do is setup sda4 manually after the install for our LVM. Double check everything, make sure they are all ext4 partitions, and there is no LVM anywhere!!!

Post Install Tasks — setting up for LVM, KVM, wireguard:

Let us start by upgrading the system, setting up for KVM, and upgrading the kernel from stock default. Remember we cannot run a lot of thing with less than a 5.15.x kernel, and if you start getting into things like ZFS we would need to be exactly on 5.15.x kernel currently. For our purposes we will just use kernel-ml and can downgrade to 5.15.x for ZFS later if we choose to manually compile our own kernels

dnf update
shutdown -r now (reboot with updated system)
cat /proc/cpuinfo | egrep "vmx|svm"  (check we have virtualization enabled)
dnf install @virt virt-top libguestfs-tools virt-install virt-manager xauth virt-viewer

systemctl enable --now libvirtd (everything working? ifconfig -a)
#get rid of virbr0 in ifconfig
virsh net-destroy default
virsh net-undefine default
service libvirtd restart
(check https://www.elrepo.org/ for below install command for rocky 8 or 9)
#we will not be able to install wireguard with less than 5.15 kernel
yum install https://www.elrepo.org/elrepo-release-8.el8.elrepo.noarch.rpm
rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
dnf makecache
dnf --enablerepo="elrepo-kernel" install -y kernel-ml
#let's setup our br0 bridge before we reboot
cd /etc/sysconfig/network-scripts
nano ifcfg-enp10s0f0 (your <device name>
#add "BRIDGE=br0" at end of this file
nano ifcfg-br0
#add something like this:
#should match a lot of your file above
STP=no
TYPE=Bridge
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
IPADDR=192.168.0.2
GATEWAY=192.168.0.1
PREFIX=24
DNS1=192.168.0.3
DNS2=8.8.8.8
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
NAME=br0
UUID=36e9e2ce-47h1-4e02-ab76-34772d136a21
DEVICE=br0
ONBOOT=yes
AUTOCONNECT_SLAVES=yes
IPV6INIT=no
#change UID to something different above and put your own IPs in
shutdown -r now (reboot with new kernel)
ifconfig -a (check br0 is there, then all good)

Ok now we have new kernel and br0 is setup for KVM guests, let’s move on to LVM.

Creating our LVM for our KVM guests:

#make sure we have right device
fdisk -l /dev/sda 
fdisk /dev/sda
p
<enter> (should have partition 4 and adding rest of disk space to it)
t (toggle partition 4 - change from linux to LVM)
8e (change to LVM)
w (write changes to partition tables)
partprobe /dev/sda (inform OS of partition changes)
pvcreate /dev/sda4 (now we have it as a LVM-can check with "pvs")
vgcreate vps /dev/sda4 (creating our volume group - "vgdisplay")
#now we are all setup, we can create as many KVM guests as we want
#for example give 70G to one guest and give remaining space to a devel guest
lvcreate -n cappy -L 70G vps (create a 70G guest - "lvdisplay")
lvcreate -n devel -l 100%FREE vps(give remaining space to this guest)
#can always delete it later
pvdisplay (check we used up all the space on vps)
#let's make sure guests can suspend and resume on host reboots:
pico /etc/sysconfig/libvirt-guests 
"ON_SHUTDOWN=suspend"
systemctl start libvirt-guests
systemctl enable libvirt-guests

Congratulations on your new install.

Dan.

Batteries Lifepo4 – Lithium Iron Phosphate upgrade and why

Batteries, LifePo4, Lithion Iron Phosphate

The past setup:

Well my two marine batteries died I had used as a backup for servers. What I had done for my servers at home was modified a UPS and hooked up two 100ah marine batteries in parallel for 200ah of capacity. While this was a great cheap solution at the time, 2 marine batteries were 100 dollars each and a modified UPS to run off them was a great idea at the time.

New tech in batteries – LifePo4(Lithium Iron Phosphate)

I was using alkaline 12V batteries that long ago, but now with the new contender on the market Lifepo4, short for Lithium Iron Phosphate we really need to look at why they are so much better for our homes than the old alkaline batteries. I had to watch countless YouTube videos on electrical engineering to get to bottom of this. To basically sum it up, let’s say we have one 12 volt car battery at 100ah(amp hours), and we have one 12 volt 100ah Lifepo4 battery. Alkaline batteries need to be charged once they reach 50% capacity, Lifepo4 batteries can be completely discharged and still run loads needed.

The recommendation is to charge Lifepov4 batteries by time it hits 20% so you don’t drain it completely. Also alkaline batteries die after about 500 cycles, where the new Lifepov4 standard can last as long as 6000 cycles! So I have found Lifepov4 batteries will not only last you longer, but you can use them longer before they are completely discharged, and for an added bonus, they are safe in your house because they won’t cause fires like other batteries will!

This basically all means then a 200ah alkaline battery would equal the performance of capacity of a 100ah Lifepov4 battery, and will last longer, not only 500 cycles! Definitely then worth twice the cost of an alkaline battery.

Current pricing of Lifepov4:

Alright that is all nice and all, but let’s look at real world prices of replacing my alkaline 200ah setup. The current best priced 100ah Lifepov4 battery on amazon is:

This one

WOW $570 + tax CDN@!!! So what $650 CDN just for a 100ah battery! And this gets worse as we go to 200ah batteries at 1k or more, and 300ah batteries at nearly 2k it seems! Ok this is completely unacceptable, we are being robbed in the USA and Canada on these China made cells in these, what if we build our own?

What are current stats and prices from China?:

Checkout following link from reputable supplier in China on Alibaba:

https://szluyuan.en.alibaba.com/productgrouplist-916502401/Non_grade_A_lifepo4_battery.html?spm=a2700.shop_co.88.17

The forum thread for her where people had good experiences is here:

https://diysolarforum.com/threads/where-to-buy-from-these-days.43265/

China sells them by the cell at 3.4 volts and a certain amount of AH(amp hours) for each cell. So a quick lesson in electrical engineering is in order to understand some of this. There are two ways to hookup batteries to each other, in series or in parallel. So if we had 4 of these 3.4 volt China cells in series, you add the voltage together and the amp hours stay the same. So in this case if we took 4 of these and connected them all together we would have the 12 volt battery we need. If we instead hooked them up in parallel, the volts stay the same but the amp hours increase. So in parallel we would only have 3.4 volts, not what we want. We want to start at a 12 volt battery and get the most AH for our money.

So what we need is 4 of these cells to build a 12 volt battery, then in future we can build another one and hook those two up in parallel to increase capacity if we want. For series connection all we do is hook each battery up negative to positive with cables, for parallel all we do is hookup terminals on each battery negative to negative and positive to positive with cables or bus bars, whatever you prefer, easy enough right? China supplies bus bars generally so we good there, we don’t have to go off and buy 2 gauge cables off Amazon or anything.

Another quick lesson, the total Power of a battery would always be the formula:

Watts = Volts x Amps , so to get any of those values all we need is 2 of them: ie: to get amps knowing watts and volts, we would just do watts/volts=amps and so on.

Grade A vs Grade B cells from China:

People all over the internet will argue over Grade A vs Grade B EVE cells, basically Grade A someone put in the effort to fully test them, discharge and charge and will charge you double for that. With Grade B they do same tests, but don’t go through effort of fully discharging and charging them, but it is from same production line, so for cost effectiveness we want to go with Grade B obviously and take our chances. They should all be same volts when we get them, only thing they didn’t test what capacity with fully charging and discharging them, so that means we could get a better battery to lol. So if it is for home use, use Grade B for price alone, if for a business go for Grade A.

Let’s pick a cell from China:

If you look at that Alibaba webpage from above, we know Amy is reputable, may take us two months to get our batteries, but cost wise it will be better. Let’s take a quote from the forum link above about someone’s experience:

 “I placed my order of 4 x EVE LF280K with Amy on June 7th. Shipment was two boxes (2 batteries per box) and box#1 arrived July 28th and box#2 followed the next day on July 29th (52 days delivery time to Toronto Canada).

The cells were described by Amy as “LF280K, brand new. Grade B —the voltage and internal resistance are matched, the capacity is not. The actual capacity is 275AH-284AH. QR code has B stamp , $111/pcs”.

Shipping was $140 and final delivery made via UPS. Tracking number was provided when order was placed, but package could not be tracked until it arrived in Canada (by sea).

Transaction was super smooth and went without a hitch. Thanks to members of this forum for recommending Amy as a reliable and trustworthy source. Will definitely order from her again.

I’m a degenerate gambler, so I’m looking forward to running capacity tests on each cell to see if I got a good batch or not “

Wow he paid a mere $650 CDN total for a 280AH battery! That would cost us 2k on amazon or anywhere else!

Is that all or do we need more?:

Technically you could just do that and be fine but there are actually two more things needed for a Lifepov4 battery: a BMS and a charger.

A BMS(Battery management system) is a card you buy online that has little wires hooked up to all the battery terminals, think of it as something that protects the battery from overcharging, over discharging, temperature cutoffs and so forth, it is basically a good thing to put on the battery. They run cheap crappy ones, and hundred or 2 for the good ones that even come with a Bluetooth app for your phone giving you all detailed stats about your new Lifepov4 battery you put together. It is a good idea and would recommend it. Last thing we need is a charger meant for Lifepov4, and we are set.

In all honesty, I think a good quality BMS, charger, and perhaps a box to put the battery in when your finished is a good idea. If BMS dies, charger dies, or battery cell dies, it’s as easy as just pulling it out and replacing it, unlike a normal battery where they put so much glue and stuff on it to prevent you from opening it that you’d have to toss the whole thing out.

Putting it all together:

Ok this is definitely not the faint of heart, in order to now change our setup from modified UPS using 2 alkaline batteries we have to rethink our whole setup for Lifepo4. So the UPS will need to be removed, we will need the following in CDN dollars:(take off 25% if USD)

  • a) Inverter(1500 watts or more) $400
  • b) Automatic Transfer Switch $200
  • c) 280 AH Lifepov4 battery $650-800 + $2-300 for BMS card and charger

So is it even worth it? Our upfront costs are already $1500-2k to replace our simple setup, why not just go to Costco and pay $400 for 2 more batteries that will die on us again. Also why not just get a UPS that supports Lifepov4 instead of going inverter/transfer switch route.

Here is why, a UPS that supports lifepov4 is very expensive, I believe Tripp Lite makes one, I think their top model supports around 750 watts and your going to pay 1k for something that can’t even handle a microwave oven or many servers running at same time? What if the power grid goes out, I would definitely need to be able to run around 1000 watts safely for servers, routers, internet, TV etc in livingroom. Of course you could get a top of the line one that supports more, but then your paying thousands for a rackmount option, not worth it.

How about we keep our inverter separate from rest of system that way we can upgrade it anytime we want, also add more lifepov4 batteries in parallel in the future if we want more capacity. This one 12V battery at 280 AH is like 6 alkaline batteries in parallel and is lighter to boot! $200 a piece from Costco = $1200 in alkaline batteries that would just die on you again!

Keep in mind we will stay 12V with our 1500 watt inverter or even a 2000 watt inverter, it is only when you start scaling up to 3000 watt inverters we would have to start scaling our batteries to 24V or even 48V if you wanted to run a whole house.

I left the best thing for last:

This new setup would allow you to do so much more, we could toss in a solar charge controller for a couple hundred dollars and some used solar panels after to the mix, and get free energy now if we wanted. We could switch our our automatic transfer switch to use the batteries when they are full from solar energy, and switch back when our batteries hit 20% discharge! I bet that sounds like fun, paying less in electricity bills just because you did a proper setup. Also you can check battery status from your phone at all times vs buying one off amazon that includes nothing!

Let’s get to some links for our parts list already!:

a) Inverter:

https://www.voltworks.cc/collections/used-products

Let’s start with 1500 watt inverter, this will be cheapest we will find for good quality, even if we go new still be cheaper than amazon. There are cheaper 1200 watt inverters on amazon, but I’d feel more comfortable with 1500 watts. If you have a lot of money go with an expertpower expensive inverter, they have transfer switch and charger built into them already, but I don’t think it’s worth it, what if your inverter dies, that is an expensive replacement then, but if money is no issue get a 3000 watt inverter from them. Why did I pick this one? Because you find a 1500 watt inverter with a pure sine wave not modified sine wave(the former protects your electronics) for a better price and quality parts 🙂

b) Automatic Transfer Switch:

This will allow us to use the grid and switch to inverter batteries if grid goes down on us so our servers/PC’s stay on and whatever else you want for up to 1500 watts of the inverter we picked. Keep in mind on this you have to change a jumper setting for an instant switch as it’s set for a generator with a time delay when you get it. Also they have a pre-wired version so you don’t have to hack up extension cords if you choose on US amazon site. I believe this one will support up to a 2000 watt inverter, they have a 50 amp version as well.

This is just a smart choice, if our inverter ever dies or we want to upgrade to 2000 watts, it is a quick swap out, and we don’t have to pay for an expensive inverter with a transfer switch already built it saving us money again. For reference $300CDN vs $7-800 is a big deal especially it it dies one day, I’d rather put that money towards an inverter that has more watts.

c) Lifepov4 280 AH battery cells:

https://www.alibaba.com/product-detail/Luyuan-4pcs-280AH-LiFePO4-LFP-3_62583909993.html?spm=a2700.shop_plgr.41413.33.3b044a44HTxwRK

Talk to Amy on Alibaba about some Grade B cells, also explore her links maybe get a JBD BMS(I hear the overkill BMS is just that one anyways) and a battery box to put everything in. Lot’s of YouTube videos showing how to hookup BMS to these cells, pick one you like, preferably with a phone app, treat yourself here.

What if you want to expand the setup to solar?

Swap out the Automatic Transfer Switch above with this instead:

Then get yourself a solar controller charger(stick to one with same watts as your inverter) and some used solar panels your good to go! This thing will feed off your batteries when solar has charged them up, then switch back to grid when your batteries are low. Free energy, why not put the battery you built to use 🙂

I’ll add more links and information to this setup once I get all these parts, but this is how I would go about it for best setup for your home. Start small with 1500 watt inverter, scale up as needed, maybe one day you want to run your fridge, deepfreeze and air conditioner as well, that would be a good time to scale up to say a 3000 watt inverter and solar charge controller.

I would recommend you build your own battery with BMS and try a setup like this, not only will you be happier in the end, not just the cost savings, just the pure knowledge you will gain from learning how to set it all up from YouTube videos will make you a much better person going green but at same time give you the confidence to go completely off grid one day if you choose, that knowledge is invaluable.

Until next time….

Dan.