Monthly Archives: March 2014

Rsync Restart

How to continue rsync on disconnects?

Rsync is a great software, I have been using it for decades to transfer files from different servers. However one issue with it, is if you were to get any network disconnects, it would be nice to have a way to restart rsync automatically on any errors.

To accomplish this, we can write a shell script to catch any rsync errors in a loop, and restart it only on errors. Perfect!

To start let’s do the following:

alias pico='nano -w'

Now we have our file open let us copy and paste this following script inside the file. I have 2 examples here, first one commented out I use with no bandwidth limits, second one I use to place a bandwidth limit of 30 Megabytes so I do not overload poor gigabit connection. Replace the uncommented rsync line with your rsync line, do not forget the ending semi colon to finish.


#keep rsync going! 
#Run in screen session - "screen -S rsync"
#Ctrl-a-d to disconnect, screen -r rsync(to re-attach)

export Result=1;
while [ $Result -ne 0 ]; do
   echo "STARTING ($Result) @" `date`;
   echo "Started On `date`" >> rsync.log
   #PUT YOUR SYNC COMMAND HERE:               
   #rsync -Wal4vv --progress --partial --timeout=10 --bwlimit=0 --delete --force /data;
   rsync -Wal4vv --progress --partial --timeout=10 --bwlimit=30M --delete --force /data;
   sleep 1;

Once you pasted that in, exit with Ctrl-x and save the file.
If you are running –delete in your rsync command, do not run this program in that directory or you will erase this program and our rsync.log tracker file!

Let’s take a look at what this script is doing(for those interested in what program is actually doing). First thing it starts a bash script with our she-bang line. We then remove any rsync.log we might have had, so we know how many times rsync restarted for only this session. Next we export a starting variable “Result” and assign that to 1. Now we start a while loop that will loop infinitely with our rsync command until such time as Result is not equal to 0. So it first goes into the while loop because Result is not 0, it is 1. Next we display to screen(STDOUT) we are starting along with date, and put everytime rsync starts in a file called rsync.log so we can check how many time rsync restarted time to time. Now we execute our rsync command. After rsync command exits, bash stores exit code of a program in a special variable called “$?”, which is always an integer, normally 0 or 1, and assign that to variable Result. If rsync fails in anyway, Result would be set to “0”. In programming 0 means false, and 1 means true. If rsync completes successfully, Result will equal to 1 and break the while loop because Result will no longer not equal 0. Our next line “sleep 1”, is simply good practice to not let cpu run at 100% in an infinite loop if something ever went wrong by letting it pause for a moment while in an infinite loop. So if rsync completes successfully it will exit while loop and program in this case, otherwise it will start over with first echo lines in the while loop till it is successful.

Alright next thing we want to do is run the script:

chmod 755
screen -S rsync
ctrl-a-d (disconnect from screen session)

Great now we are off to the races! Hit “Ctrl-a-d” to disconnect screen and if you have terrabytes of data to transfer, just check in occasionally on transfer.

tail -100 rsync.log
screen -r rsync

Until next time, happy transferring.


FreeBSD 10 Ports and pkg2ng, how to deal with binaries and ports collection at same time

FreeBSD has implemented a new packaging system called pkg2ng. While that is great for a number of reasons, like allowing commercial companies to now have their own remote repositories or even better faster update times, we still have 1 issue. We would like to also keep the most powerful feature of FreeBSD, the custom port build options we like on our custom ports build.

Ideally what we would like to do then, since FreeBSD now stuffs binary installs and ports collection installs in same SQLite database for pkg2ng, is take advantage of updating with binaries first, then update our custom port builds as well. A big danger we face, is if we just do a normal “pkg upgrade”, we could effectively loose all our custom options in certain ports collection builds. So let’s talk about a way that we can have best of both worlds.

I assume you are running at least FreeBSD 10, and using pkg2ng for package management, and portmaster for port upgrades. Also let us take a custom realistic scenario, let say we want to run alpine with custom options for maildir patch support, also we want to install apache22-itk-mpm, with custom modules built into apache, as well we want to use the new mariadb instead of MySQL for database.

So for this scenario we are not going to want to let binaries touch anything building MySQL, as it will install real MySQL. We also do not want binaries upgrading our apache modules as it will try to install regular apache, so let’s begin.

pkg install apache22-itk-mpm mariadb55-client mariadb55-server
cd /usr/ports/mail/alpine; make install; pkg lock alpine
cd /usr/ports/www/mod_auth_mysql_another; make install; pkg lock ap22-mod_auth_mysql_another
cd /usr/ports/www/mod_geoip2; make install; pkg lock ap22-mod_geoip2
cd /usr/ports/www/mod_perl2; make install; pkg lock ap22-mod_perl2
cd /usr/ports/www/mod_rpaf2; make install; pkg lock ap22-mod_rpaf2
(enable apache module in make config below)
cd /usr/ports/lang/php55; make config;make install; pkg lock php55
pkg install php55-extensions
pkg install php55-mysql (this is fine as binary since it uses custom MySQL build)

OK Great, now let’s see how we can upgrade for now on.

pkg version -v (see what ports out of date)
pkg upgrade (upgrade binary ports first)
locked (alias locked='pkg info -ak|grep yes')( check locked ports to unlock for portmaster and update list to unlock/lock below )
pkg unlock alpine; pkg unlock p5-DBD-mysql; pkg unlock ap22-mod_auth_mysql_another; pkg unlock ap22-mod_geoip2; pkg unlock ap22-mod_perl2; pkg unlock ap22-mod_rpaf2; pkg unlock php55
portmaster -adG (upgrade ports)
pkg lock alpine; pkg lock p5-DBD-mysql; pkg lock ap22-mod_auth_mysql_another; pkg lock ap22-mod_geoip2; pkg lock ap22-mod_perl2; pkg lock ap22-mod_rpaf2; pkg lock php55
pkg clean

There we have it, now we can get fastest build times as possible, first upgrading binaries, and then our custom ports. Locking packages is what prevents “pkg upgrade” from doing bad things to our custom ports. Of course we need to unlock them after to run portmaster on them. The sad part in this is we have to keep track of what packages to lock and unlock all the time for upgrade purposes. I have opened this issue:

So if developers get around to adding this new feature to pkg2ng, it could make our life simpler, for now any time you lock a package, just update your upgrade notes, I normally toss it into /etc/motd for easy access.

Till next time, happy FreeBSD 10!


KVM – Adding Space To FreeBSD 10 zfs on root guest on a Centos 6.5 LVM host

I decided to write this after I could not find any documentation on internet how to easily add space to a FreeBSD 10 guest that had zfs on root install. This should show you how to do it easily and quickly. It is important as we may need to add more space from our LVM to our FreeBSD guest at some point, and we need to know exactly how to do that.

I assume you have a Centos host running KVM guests, as well as gdisk installed.

First thing we want to do is extend size of our guest

lvextend -L +10G /dev/vps/sunsaturn (add 10G to sunsaturn)
gdisk /dev/vps/sunsaturn (we let gdisk fix partition table or we will not be able to add new space)

Command (? for help): w
Warning! Secondary header is placed too early on the disk! Do you want to
correct this problem? (Y/N): Y
Have moved second header and partition table to correct location.”

Just save and exit to fix our partition tables.
Now let us run gdisk again to add the new space, since we have following:

gdisk -l /dev/vps/sunsaturn
Number  Start (sector)    End (sector)  Size       Code  Name
   1              34            1057   512.0 KiB   A501  gptboot0
   2            1058         8389665   4.0 GiB     A502  swap0
   3         8389666       335544286   156.0 GiB   A504  zfs0
virt-filesystems --long --parts --blkdevs -h -a /dev/vps/sunsaturn
Name       Type       MBR  Size  Parent
/dev/sda1  partition  -    512K  /dev/sda
/dev/sda2  partition  -    4.0G  /dev/sda
/dev/sda3  partition  -    156G  /dev/sda
/dev/sda   device     -    160G  -

Our last partition can easily be expanded just deleting last partition and add it back with the new space.

gdisk /dev/vps/sunsaturn (now delete last partition(3) and add it back, set code and name back to A504 and zfs0)
partprobe /dev/vps/sunsaturn
(if for any reason you cannot see the space just run:)
gdisk /dev/vps/sunsaturn (then hit "w")
partprobe /dev/vps/sunsaturn
(now you should be able to do above)

Alright we have expanded our LVM, now we need to restart the guest and enable new space within the guest.

virsh shutdown sunsaturn (actually wait till it is shutdown, till "virsh list" shows it gone)
virsh start sunsaturn


zpool status (find out device and put it below)
zpool online -e zroot vtbd0p3 (zfs will now grab the additional space)

That’s it, now you know how to add space on the fly to any FreeBSD guest with zfs on root, on the fly.


Update: a few months after writing this I came across this article which does a good representation.

Server Coming Along

All is coming along for new server, I look forward to bringing hardware upgrades to all users. I expect in next month we will have some exceptional speed for all databases, websites and anything else. I am pleased to bring you a worthy upgrade.

Tech mumbo jumbo:

Hardware for server all in play now. Working on virtualizing host over next week or 2, and getting any important configurations done. Things are coming along, looks like I will be moving main back to FreeBSD since I only converted it to Linux since last server did not support KVM, and rest, CPANEL etc will stay Linux. So focus will largely be on moving from openvz to KVM, and operating system change for itself.

A lot of people may say why not move to VMware. Well for a lot of reasons. First of all, I like control of main host to do any advanced configurations, firewalls, openvpn, NFS or whatever I choose. I do not like control of host taken away from me 🙂 As well, host may serve as a SolusVM host down road if I so choose, so reasons not to give control away. Secondly, KVM virtio drivers have been said to outperform VMware ESXI, since we do not want to loose any I/O speed, the choice seems obvious. In reality I only see good coming from open source companies, take a look at MySQL and Mariadb, I have a feeling we will see same thing with Linux KVM/ovirt and VMware. How VMware was even allowed to submit code into Linux kernel, is beyond me. I think in end, open source will prevail, like it always does, when there is no more money in hypervisor market one day, VMware will just be abandon wear like all companies that have come before them. For now, they seem to be needed in enterprise world for companies with many servers to control, and fill a gap, time will tell for how long 🙂