All posts by SunSaturn

Rsync Restart

rsync, software

How to continue rsync on disconnects?

Rsync is a great software, I have been using it for decades to transfer files from different servers. However one issue with it, is if you were to get any network disconnects, it would be nice to have a way to restart rsync automatically on any errors.

To accomplish this, we can write a shell script to catch any rsync errors in a loop, and restart it only on errors. Perfect!

To start let’s do the following:

alias pico='nano -w'
cd
pico rsync_restart.sh

Now we have our file open let us copy and paste this following script inside the file. I have 2 examples here, first one commented out I use with no bandwidth limits, second one I use to place a bandwidth limit of 30 Megabytes so I do not overload poor gigabit connection. Replace the uncommented rsync line with your rsync line, do not forget the ending semi colon to finish. 

#!/bin/bash

#keep rsync going! 
#Run in screen session - "screen -S rsync"
#Ctrl-a-d to disconnect, screen -r rsync(to re-attach)

export Result=1;
while [ $Result -ne 0 ]; do
   echo "STARTING ($Result) @" `date`;
   echo "Started On `date`" >> rsync.log
   #PUT YOUR SYNC COMMAND HERE:               
   #rsync -Wal4vv --progress --partial --timeout=10 --bwlimit=0 --delete --force 192.168.0.3::data /data;
   rsync -Wal4vv --progress --partial --timeout=10 --bwlimit=30M --delete --force 192.168.0.3::data /data;
   Result=$?;
   sleep 1;
done

Once you pasted that in, exit with Ctrl-x and save the file.
If you are running –delete in your rsync command, do not run this program in that directory or you will erase this program and our rsync.log tracker file!

Let’s take a look at what this script is doing(for those interested in what program is actually doing). First thing it starts a bash script with our she-bang line. We then remove any rsync.log we might have had, so we know how many times rsync restarted for only this session. Next we export a starting variable “Result” and assign that to 1. Now we start a while loop that will loop infinitely with our rsync command until such time as Result is not equal to 0. So it first goes into the while loop because Result is not 0, it is 1. Next we display to screen(STDOUT) we are starting along with date, and put everytime rsync starts in a file called rsync.log so we can check how many time rsync restarted time to time. Now we execute our rsync command. After rsync command exits, bash stores exit code of a program in a special variable called “$?”, which is always an integer, normally 0 or 1, and assign that to variable Result. If rsync fails in anyway, Result would be set to “0”. In programming 0 means false, and 1 means true. If rsync completes successfully, Result will equal to 1 and break the while loop because Result will no longer not equal 0. Our next line “sleep 1”, is simply good practice to not let cpu run at 100% in an infinite loop if something ever went wrong by letting it pause for a moment while in an infinite loop. So if rsync completes successfully it will exit while loop and program in this case, otherwise it will start over with first echo lines in the while loop till it is successful.

Alright next thing we want to do is run the script:

chmod 755 rsync_restart.sh
screen -S rsync
./rsync_restart.sh
ctrl-a-d (disconnect from screen session)

Great now we are off to the races! Hit “Ctrl-a-d” to disconnect screen and if you have terrabytes of data to transfer, just check in occasionally on transfer.

cd
tail -100 rsync.log
screen -r rsync

Until next time, happy transferring.

Dan.

FreeBSD 10 Ports and pkg2ng, how to deal with binaries and ports collection at same time

freebsd, ports

Intro:
FreeBSD has implemented a new packaging system called pkg2ng. While that is great for a number of reasons, like allowing commercial companies to now have their own remote repositories or even better faster update times, we still have 1 issue. We would like to also keep the most powerful feature of FreeBSD, the custom port build options we like on our custom ports build.

Ideally what we would like to do then, since FreeBSD now stuffs binary installs and ports collection installs in same SQLite database for pkg2ng, is take advantage of updating with binaries first, then update our custom port builds as well. A big danger we face, is if we just do a normal “pkg upgrade”, we could effectively loose all our custom options in certain ports collection builds. So let’s talk about a way that we can have best of both worlds.

Pre-requisites:
I assume you are running at least FreeBSD 10, and using pkg2ng for package management, and portmaster for port upgrades. Also let us take a custom realistic scenario, let say we want to run alpine with custom options for maildir patch support, also we want to install apache22-itk-mpm, with custom modules built into apache, as well we want to use the new mariadb instead of MySQL for database.

So for this scenario we are not going to want to let binaries touch anything building MySQL, as it will install real MySQL. We also do not want binaries upgrading our apache modules as it will try to install regular apache, so let’s begin.

pkg install apache22-itk-mpm mariadb55-client mariadb55-server
cd /usr/ports/mail/alpine; make install; pkg lock alpine
cd /usr/ports/www/mod_auth_mysql_another; make install; pkg lock ap22-mod_auth_mysql_another
cd /usr/ports/www/mod_geoip2; make install; pkg lock ap22-mod_geoip2
cd /usr/ports/www/mod_perl2; make install; pkg lock ap22-mod_perl2
cd /usr/ports/www/mod_rpaf2; make install; pkg lock ap22-mod_rpaf2
(enable apache module in make config below)
cd /usr/ports/lang/php55; make config;make install; pkg lock php55
pkg install php55-extensions
pkg install php55-mysql (this is fine as binary since it uses custom MySQL build)

OK Great, now let’s see how we can upgrade for now on.

pkg version -v (see what ports out of date)
pkg upgrade (upgrade binary ports first)
locked (alias locked='pkg info -ak|grep yes')( check locked ports to unlock for portmaster and update list to unlock/lock below )
(ADD ANY PACKAGES MISSING HERE FROM "locked" COMMAND - unlock ports)
pkg unlock alpine; pkg unlock p5-DBD-mysql; pkg unlock ap22-mod_auth_mysql_another; pkg unlock ap22-mod_geoip2; pkg unlock ap22-mod_perl2; pkg unlock ap22-mod_rpaf2; pkg unlock php55
portmaster -adG (upgrade ports)
(ADD ANY PACKAGES MISSING HERE FROM "locked" COMMAND - lock ports back)
pkg lock alpine; pkg lock p5-DBD-mysql; pkg lock ap22-mod_auth_mysql_another; pkg lock ap22-mod_geoip2; pkg lock ap22-mod_perl2; pkg lock ap22-mod_rpaf2; pkg lock php55
pkg clean

There we have it, now we can get fastest build times as possible, first upgrading binaries, and then our custom ports. Locking packages is what prevents “pkg upgrade” from doing bad things to our custom ports. Of course we need to unlock them after to run portmaster on them. The sad part in this is we have to keep track of what packages to lock and unlock all the time for upgrade purposes. I have opened this issue: https://github.com/freebsd/pkg/issues/744

So if developers get around to adding this new feature to pkg2ng, it could make our life simpler, for now any time you lock a package, just update your upgrade notes, I normally toss it into /etc/motd for easy access.

Till next time, happy FreeBSD 10!

Dan.

KVM – Adding Space To FreeBSD 10 zfs on root guest on a Centos 6.5 LVM host

freebsd, kvm, linux, zfs

Intro:
I decided to write this after I could not find any documentation on internet how to easily add space to a FreeBSD 10 guest that had zfs on root install. This should show you how to do it easily and quickly. It is important as we may need to add more space from our LVM to our FreeBSD guest at some point, and we need to know exactly how to do that.

Pre-requisites:
I assume you have a Centos host running KVM guests, as well as gdisk installed.

First thing we want to do is extend size of our guest

lvextend -L +10G /dev/vps/sunsaturn (add 10G to sunsaturn)
gdisk /dev/vps/sunsaturn (we let gdisk fix partition table or we will not be able to add new space)

Command (? for help): w
Warning! Secondary header is placed too early on the disk! Do you want to
correct this problem? (Y/N): Y
Have moved second header and partition table to correct location.”

Just save and exit to fix our partition tables.
Now let us run gdisk again to add the new space, since we have following:

gdisk -l /dev/vps/sunsaturn
Number  Start (sector)    End (sector)  Size       Code  Name
   1              34            1057   512.0 KiB   A501  gptboot0
   2            1058         8389665   4.0 GiB     A502  swap0
   3         8389666       335544286   156.0 GiB   A504  zfs0
virt-filesystems --long --parts --blkdevs -h -a /dev/vps/sunsaturn
Name       Type       MBR  Size  Parent
/dev/sda1  partition  -    512K  /dev/sda
/dev/sda2  partition  -    4.0G  /dev/sda
/dev/sda3  partition  -    156G  /dev/sda
/dev/sda   device     -    160G  -

Our last partition can easily be expanded just deleting last partition and add it back with the new space.

gdisk /dev/vps/sunsaturn (now delete last partition(3) and add it back, set code and name back to A504 and zfs0)

Alright we have expanded our LVM, now we need to restart the guest and enable new space within the guest.

virsh shutdown sunsaturn (actually wait till it is shutdown, till "virsh list" shows it gone)
virsh start sunsaturn

IN THE GUEST:

zpool status (find out device and put it below)
zpool online -e zroot vtbd0p3 (zfs will now grab the additional space)

That’s it, now you know how to add space on the fly to any FreeBSD guest with zfs on root, on the fly.

Dan.

Update: a few months after writing this I came across this article which does a good representation.

Server Coming Along

updates

All is coming along for new server, I look forward to bringing hardware upgrades to all users. I expect in next month we will have some exceptional speed for all databases, websites and anything else. I am pleased to bring you a worthy upgrade.

Tech mumbo jumbo:

Hardware for server all in play now. Working on virtualizing host over next week or 2, and getting any important configurations done. Things are coming along, looks like I will be moving main SunSaturn.com back to FreeBSD since I only converted it to Linux since last server did not support KVM, and rest, CPANEL etc will stay Linux. So focus will largely be on moving from openvz to KVM, and operating system change for SunSaturn.com itself.

A lot of people may say why not move to VMware. Well for a lot of reasons. First of all, I like control of main host to do any advanced configurations, firewalls, openvpn, NFS or whatever I choose. I do not like control of host taken away from me 🙂 As well, host may serve as a SolusVM host down road if I so choose, so reasons not to give control away. Secondly, KVM virtio drivers have been said to outperform VMware ESXI, since we do not want to loose any I/O speed, the choice seems obvious. In reality I only see good coming from open source companies, take a look at MySQL and Mariadb, I have a feeling we will see same thing with Linux KVM/ovirt and VMware. How VMware was even allowed to submit code into Linux kernel, is beyond me. I think in end, open source will prevail, like it always does, when there is no more money in hypervisor market one day, VMware will just be abandon wear like all companies that have come before them. For now, they seem to be needed in enterprise world for companies with many servers to control, and fill a gap, time will tell for how long 🙂

Dan.

New Server

updates

Time to build a new server!

Seems we have outgrown our current server, and an upgrade is needed. Currently we are having issues with not enough memory, and not enough disk space. Other issues include older cpu chipsets that do not do full hardware virtualization, which is a known problem on the Dell 2850 servers. What I propose is we order a barebones new server and upgrade components inside it to bring us exceptional speed, full hardware virtualization, lots of memory and diskspace, and everyone is happy with how fast their websites load and databases perform.

We use enterprise SSD’s for speed/reliability and load it with tons of memory to ensure we never have a memory shortage again. I thank everyone for their patience on this upgrade, I did comment to many people I would commence on this after the new year, and I have, I am hoping to have this completed by worse case scenario, end of March/14. For anyone interested in technical builds, read on below, for the rest, I look forward to bringing you a great hosting platform for March /14.

Completed:
a) Ordered a barebones Dell 2950 generation 3(will provide full virtualization support on 2 quad cpu chipset(8 cpus total @2.83ghz))
b) Ordered a memory upgrade to bring it up to 32gb(arrived and installed today – system detects full 32gb)

In Progress:
a) Memory checks
b) Order bay trays
c) Order Icy Dock Units
d) Order Sata Power Splitter + SSD cabling for checking onboard sata performance vs raid controller.
d) Order Intel s3500 SSD’s
e) Test SSD’s against perc 6i controller for single disks, raid 0, as well as onboard sata performance in software raid configurations.
f) Configure linux operating system using deadline I/O scheduler, VM’s with NOOP I/O scheduler, virtualize with KVM, convert all openvz hosts over to new KVM virtualization.
g) Ship unit to colocation and do final setups over a week period before conversion.

Dan.
SunSaturn Wishes You A Happy New Year

Moving to windows 8 from windows 7

windows

So it got in my mind with Microsoft moving from windows 7 to windows 8, and the much needed upgrade to make their operating system work on smart phones, that this was the start of a needed upgrade to the future as they will eventually drop support for anything previous to windows 8 to stay in competition with the likes of iphones, androids and blackberries.

Reasons to upgrade from top of my head:
1) Developers must upgrade in order to learn how to stay up to date with technology, i.e.: writing apps for windows phone, or playing with latest features to see what apps they can write, or just in general to help others with their OS from experience with it.
2) It boots faster, and if done up right, can serve just as windows 7 did, just faster and better(as developers stop coding for windows 7 and backwards)
3) You will most likely want the latest and greatest software, and this will most likely only run on windows 8 on in the future.
4) Windows 8 incorporates all your Facebook, LinkedIn, personal email, Gmail, Hotmail, Skype, and even China’s Facebook equivalent: “Sina Weibo”, so you have one app called “People”, where you can talk to everyone at once which is very handy.
5) You will be able to use latest internet explorer browsers. If your a developer you should be using internet explorer, firefox and chrome for testing your website builds to begin with, so another good reason to upgrade.
6) The list goes on, but what it comes down to regardless, is every new laptop sold will have windows 8, so the move is inevitable, embrace the change, and move on.

Reasons not to upgrade:
1) If you are running on really old hardware, and still on something like Windows XP, then you might not meet specs to even install it, and even if you did, it would run to slow because you most likely do not have the computer memory to handle it.
2) You are anti-Microsoft. Well even though there are a lot of those, wouldn’t it make more sense to install their latest and greatest so you aren’t just giving intelligent people un-useful lip service and just get flamed in forums from your lack of knowledge of the new product?
3) It is an unnecessary expense when windows 7 works fine. Well then you cannot play with all the new toys and how much fun is that? Surely if you do not want to pay for it, you can just download a copy off a torrent site and use MS toolkit to run it for free, so really comes down to time vs money in this situation, so get your tech family member to do it.
4) You run Linux or Apple as a client. Well biggest problem here is people write new apps and then you cannot use them unless they lend support to these 2 OS’s. In either case you are still better off installing a virtualization technology like “VMware” and running an instance of Windows 8 inside there so you have the best of all worlds. Personally I run Linux only on servers, and use Windows as a client to access them so I don’t have complaints from my better half wanting to use my PC or I cannot run this and that because I am limited. There are free virtualization technologies, I don’t suggest going out and spending money on one if you have tech ability and can get a free one. Things like KVM under Linux or open source projects on sourceforge.net for windows ones are just fine. I ran Windows 8 beta when it first came out under KVM in Linux and it was fine, but I would not do that on my main desktop as I have triple monitor setup and want to utilize all my monitors. Graphical power users I know usually have MAC laptops and run a Windows 8 instance inside it as well.

I would also like to state: I have 2 HP printers, one of them I always had to go to website and manually install driver in Windows 7, in Windows 8 I did not have to do anything. As you can see support is starting to drop for Windows 7, and it will make your life harder not better.

My experience of dislikes with installing Windows 8 and how to fix the issues:
Now remember I am a power user, so moving quickly around to do things is very important, so I will go over the issues and how to deal/cope with issues.

1) There is no start menu. This makes it difficult to work with desktop, where if your like me, where you will spend most of your time being productive instead of in the app screens. Solution: Install one. Go here: http://sourceforge.net/projects/classicshell/ , now you can install whatever version you like of it, XP, Windows 7 etc. and it is a time saver especially when first installing to quickly right click apps off it to pin to your taskbar.
2) You cannot go directly to desktop after logging in. The great folks who wrote classic shell above solved this issue to, now waking my computer up I go right to the desktop or even rebooting.
3) Too many clicks to find the shutdown or restart button. Again classic shell solves this above putting it back in place it should be.
4) You have to click through a start screen in order to login all the time. Who wants to do this? This is annoying and an extra click to get into your PC. Here is a quick way to disable that lock screen:
http://blog.laptopmag.com/how-to-eliminate-the-win8-lock-screenz

1.Hit the Start key, type gpedit.msc, and press Enter. This will open the Local Group Policy Editor.
2.Navigate to Computer Configuration > Administrative Templates > Control Panel > Personalization
3.Double click “Do not display the lock screen,” and select Enabled from the dialog that pops up. Click OK.

5) Power users really dislike UAC being enabled and Windows always asking to confirm this and that all the time. Unfortunately our old way of sliding slider to bottom to disable UAC in Windows 7 does not actually disable it. I was trying to patch a program in Programs Directory or modify system files and it would not let me! That’s great security against hackers, but just more annoying for people cracking programs from torrent downloads. Other issue completely disabling it is Microsoft chose to make sure apps on app screens do not run till you re-enable it again. So with this knowledge in hand this is what we do since we are to lazy to go into registry every time to disable it. Create this file called , “disable_UAC.reg” on your desktop and put this code into it:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System]
"ConsentPromptBehaviorAdmin"=dword:00000000
"EnableLUA"=dword:00000000

Now every time you need to modify system or program files directly, double click on that on desktop, reboot, do what you need to, then go click on any app on app screen after that will give you a link to enable it again, then reboot, and your good to go.
6) Another annoying issue is when we store things on network drives and install programs from there. Microsoft seems to detect if file is not owned by you and denies the install! Good security from someone trying to elevate their privileges as a hacker, but bad for us. Solution: copy program from network drive to your desktop, files will then be owned by you, then you can install as usual.
7) Triple monitor setup and I want screen saver to run on all 3, different wall papers on each monitor, and use a different monitor as my primary monitor , my center screen, instead of one on left. Solution:
install “Ultramon” and this solves all the problems.
8) Microsoft’s way of moving mouse to top of right screen to go to apps menu has a key flaw, in a triple monitor setup, instead of just throwing my mouse over there, I have to be exact so my mouse does not go over to next monitor, this is really annoying and more of a pain than anything. Unfortunately something we seem to have to live with, hoping to find a solution to this issue, but overall I do not go in apps screens enough to care, all 3 usually in desktop mode just like Windows 7 was.
9) Pain to close apps and move around to different ones. Only 2 solutions I could find was drag app from top to bottom to close it or use the old Alt-Tab trick to move around between different things quickly. The other way of going over to top left of screen and right clicking to close, takes way to long.

Overall if your a Windows 7 user, you can see how I solved most issues to function just as Windows 7 did, plus having next operating system at my disposal now. It is a bit of a pain to get use to, but spend an hour or 2 with interface and you’ll be an expert in no time.

Till we meet again,

Dan.

SunSaturn

University students

updates

For students I have taught or any educational institutions, I have opened a special offer. $19.99 for the year which includes a domain of your choosing and free hosting for the year on any of the TLD’s from main site. Ie: .com, .net etc.

This is to show SunSaturn’s strong support for the next generation of talented designers and programmers as a way for you to show case your skills to future employers with examples of work for them, or to develop your skills further on live productions systems you will encounter with all the latest technology installed for your use.

To apply this offer, email support@sunsaturn.com the domain you would like to use for your website or me personally after you have filled out: https://secure.sunsaturn.com/register.php so I have you on file, then I will send you an email back, how to pay the invoice, ie credit card, or paypal and you should be good to go!

Dan.

SunSaturn Washington DC, at presidents inaugeration

updates

Will be at canadian embassy on Inaugeration Day, Jan 21/2013, I look forward to getting good pictures of event, should be an exciting day. Do give me a call if you are in the area! I am going to spend some time here, I plan on returning to canada on February 20th, helping a friend drive their car back for 5 days from February 15th-20th, should be a nice road trip! In past many weeks I have spent some time at Myrtle Beach, and virginia beach having a nice holiday, I am enjoying my vacation, I also celebrated a birthday yesterday, it has been wonderful so far…

Dan.

Washington D.C.

updates

Happy new year everyone, I will be in Washington D.C. in January, I will have my laptop with me, and 800 number should reach my cell phone if any issues. I expect to be back in Canada end of January, but definitely no later than end of February, again happy new year, and do call me up if anyone in Washington, we’ll have dinner!

Happy Holidays,

Dan.