There are two kinds of people, those who backup regularly, and those that never had a hard drive fail

As you can tell the above is my favorite quote. It is so true and I believe everyone should evaluate how much their data (emails, documents, files) is worth to them and, based on that value, create a backup strategy that suits them. I know for sure that if I ever lost the pictures and videos of my family I would be devastated since those are irreplaceable.

So the question is how can I have an inexpensive backup solution? All my documents and emails are stored in Google, since my domain is on Google Apps. What happens to the live/development servers though that host all my work? I program on a daily basis and the code has to be backed up regularly so as to avoid any hard drive failures and thus result in loss of time and money.

So here is my solution. I have an old computer (IBM Thincentre) which I decided to beef up a bit. I bought 4Gb of RAM from eBay for less than $100 for it. Although this is was not necessary since my solution would be based on Linux (Gentoo in particular), I wanted to have faster compilation times for packages.

I bought two external drives (750Gb and 500Gb respectively) and one 750Gb internal drive. I already have a 120Gb hard drive in the computer. The two external ones are connected to the computer using USB while the internal ones are connected using SATA.

The external drives are formatted using NTFS while the whole computer is built using ReiserFS.

Here is the approach:

• I have installed and have a working Gentoo installation on the machine
• I have an active Internet connection
• I have installed LVM on the machine and set up the core system on the 120Mb drive while the 500Mb is on LVM
• I have 300Mb active on the LVM (from the available 500Mb)
• I have generated a public SSH key (I will need this to exchange it with the target servers
• I have mounted the internal 500Mb drive to the /storage folder
• I have mounted the external USB 750Mb drive to the /backup_hourly folder
• I have mounted the external USB 500Mb drive to the /backup_daily folder

Here is how my backup works:

Every hour a script runs. The script uses rsync to syncrhonize files and folders from a remote server locally. Those files and folders are kept in relevant server named subfolders in the /storage folder (remember this is my LVM). So for instance my subfolders will be /storage/beryllium.niden.net, /storage/nitrogen.niden.net, /storage/argon.niden.net etc.

Once the rsync completes, the script continues by compressing the relevant ‘server’ folder and creates the compressed file with a date-time stamp on its name.

When all compressions are completed, if the time that the script has executed is midnight, the backups are moved from the /storage folder to the /backup_daily folder (which has the external USB 500Gb mounted). If it is any other time, the files are moved in the /backup_hourly folder (which has the external USB 750Gb mounted).

This way I ensure that I keep a lot of backups (daily and hourly ones). The backups are being recycled, so older ones get deleted. The amount of data that you need to archive as well as the storage space you have available dictate how far back you can go in your hourly and daily cycles.

So let’s get down to business. The script itself:

#!/bin/bash
DATE=`date +%Y-%m-%d-%H-%M`
DATE2=`date +%Y-%m-%d`
DATEBACK_HOUR=`date --date='6 days ago' +%Y-%m-%d`
DATEBACK_DAY=`date --date='60 days ago' +%Y-%m-%d`
FLAGS="--archive --verbose --numeric-ids --delete --rsh='ssh'"
BACKUP_DRIVE="/storage"
DAY_USB_DRIVE="/backup_daily"
HOUR_USB_DRIVE="/backup_hourly"

These are some variables that I need for the script to work. The DATE and DATE2 are used to date/time stamp the backups, while the DATEBACK_* are used to clear previous backups. In this case it is set to 6 days ago (for my system). It can be set to whatever you want provided that you do not run out of space.

The FLAGS variable keeps the rsync command options while the BACKUP_DRIVE, DAY_USB_DRIVE and HOUR_USB_DRIVE hold the locations of the rsync folders, daily backup and hourly backup sorage areas.

The script works with arrays. I have 4 arrays to do the work and the 3 of them must have exactly the same elements.

# RSync Information
rsync_info[1]="beryllium.niden.net html rsync"
rsync_info[2]="beryllium.niden.net db rsync"
rsync_info[3]="nitrogen.niden.net html rsync"
rsync_info[4]="nitrogen.niden.net html db"
rsync_info[5]="nitrogen.niden.net html svn"
rsync_info[6]="argon.niden.net html rsync"

This is the first array which holds descriptions to what needs to be done as far as source is concerned. This gets appended to the log and helps me identify what step I am in.

# RSync Source Folders
rsync_source[1]="beryllium.niden.net:/var/www/localhost/htdocs/"
rsync_source[2]="beryllium.niden.net:/niden_backup/db/"
rsync_source[3]="nitrogen.niden.net:/var/www/localhost/htdocs/"
rsync_source[4]="nitrogen.niden.net:/niden_backup/db"
rsync_source[5]="nitrogen.niden.net:/niden_backup/svn"
rsync_source[6]="argon.niden.net:/var/www/localhost/htdocs/"

This array holds the source host and folder. Remember that I have already exchanged SSH keys with each server, therefore when the script runs there is a direct connection to the source server. If you need to keep things a bit more secure for you, then you will need to alter the contents of the rsync_source array so that it reflects the user that you log in with as well as the password.

# RSync Target Folders
rsync_target[1]="beryllium.niden.net/html/"
rsync_target[2]="beryllium.niden.net/db/"
rsync_target[3]="nitrogen.niden.net/html/"
rsync_target[4]="nitrogen.niden.net/db/"
rsync_target[5]="nitrogen.niden.net/svn/"
rsync_target[6]="argon.niden.net/html/"

This array holds the target locations for the rsync. These folders exist in my case under the /storage subfolder.

# GZip target files
servers[1]="beryllium.niden.net"
servers[2]="nitrogen.niden.net"
servers[3]="argon.niden.net"

This array holds the names of the folders to be archived. These are the folders directly under the /storage folder and I am also using this array for the prefix of the compressed files. The suffix of the compressed files is a date/time stamp.

Here is how the script evolves:

echo "BACKUP START"  >> $BACKUP_DRIVE/logs/$DATE.log
date >> $BACKUP_DRIVE/logs/$DATE.log

echo "BACKUP START"  >> $BACKUP_DRIVE/logs/$DATE.log
date  >> $BACKUP_DRIVE/logs/$DATE.log

# Loop through the RSync process
element_count=${#rsync_info[@]}
let "element_count = $element_count + 1"
index=1
while [ "$index" -lt "$element_count" ]
do
    echo ${rsync_info[$index]} >> $BACKUP_DRIVE/logs/$DATE.log
    rsync $FLAGS ${rsync_source[$index]} $BACKUP_DRIVE/${rsync_target[$index]} >> $BACKUP_DRIVE/logs/$DATE.log
    let "index = $index + 1"
done

The snippet above loops through the rsync_info array and prints out the information in the log file. Right after that it uses the rsync_source and rsync_target arrays (as well as the FLAGS variable) to rsync the contents of the source server with the local folder. Remember that all three arrays have to be identical in size (rsync_info, rsync_source, rsync_target).

The next thing to do is zip the data (I loop through the servers array)

# Looping to GZip data
element_count=${#servers[@]}
let "element_count = $element_count + 1"
index=1
while [ "$index" -lt "$element_count" ]
do
    echo "GZip ${servers[$index]}" >> $BACKUP_DRIVE/logs/$DATE.log
    tar cvfz $BACKUP_DRIVE/${servers[$index]}-$DATE.tgz $BACKUP_DRIVE/${servers[$index]} >> $BACKUP_DRIVE/logs/$DATE.log
    let "index = $index + 1"
done

The compression method I use is tar/gzip. I found it to be fast with a good compression ratio. You can choose anything you like.

Now I need to delete old files from the drives and copy the files on those drives. I use the servers array again.

# Looping to copy the produced files (if applicable) to the daily drive
element_count=${#servers[@]}
let "element_count = $element_count + 1"
index=1

while [ "$index" -lt "$element_count" ]
do
    # Copy the midnight files
    echo "Removing old daily midnight files" >> $BACKUP_DRIVE/logs/$DATE.log
    rm -f $DAY_USB_DRIVE/${servers[$index]}/${servers[$index]}-$DATEBACK_DAY*.* >> $BACKUP_DRIVE/logs/$DATE.log
    echo "Copying daily midnight files" >> $BACKUP_DRIVE/logs/$DATE.log
    cp -v $BACKUP_DRIVE/${servers[$index]}-$DATE2-00-*.tgz $DAY_USB_DRIVE/${servers[$index]}  >> $BACKUP_DRIVE/logs/$DATE.log
    rm -f $BACKUP_DRIVE/${servers[$index]}-$DATE2-00-*.tgz >> $BACKUP_DRIVE/logs/$DATE.log

    # Now copy the files in the hourly
    echo "Removing old hourly files" >> $BACKUP_DRIVE/logs/$DATE.log
    rm -f $HOUR_USB_DRIVE/${servers[$index]}/${servers[$index]}-$DATEBACK_HOUR*.* >> $BACKUP_DRIVE/logs/$DATE.log
    echo "Copying daily midnight files" >> $BACKUP_DRIVE/logs/$DATE.log
    cp -v $BACKUP_DRIVE/${servers[$index]}-$DATE.tgz $HOUR_USB_DRIVE/${servers[$index]} >> $BACKUP_DRIVE/logs/$DATE.log
    rm -f $HOUR_USB_DRIVE/${servers[$index]}/${servers[$index]}-$DATEBACK*.* >> $BACKUP_DRIVE/logs/$DATE.log
    let "index = $index + 1"
done

echo "BACKUP END"  >> $BACKUP_DRIVE/logs/$DATE.log

The last part of the script loops through the servers array and:

1. Deletes the old files (recycling of space) from the daily backup drive (/storage/backup_daily) according to the DATEBACK_DAY variable. If the files are not found a warning will appear in the log.
2. Copies the daily midnight file to the daily drive (if the file does not exist it will simply echo a warning in the log – I do not worry about warnings of this kind in the log file and was too lazy to use an IF EXISTS condition)
3. Removes the daily midnight file from the /storage drive.

The reason I am using copy and then remove instead of the move (mv) command is that I have found this method to be faster.

Finally the same thing happens with the hourly files

1. Old files are removed (DATEBACK_HOUR variable)
2. Hourly file gets copied to the /backup_hourly drive
3. Hourly file gets deleted from the /storage drive

All I need now is to add the script in my crontab and let it run every hour.

NOTE: The first time you will run the script you will need to do it manually (not in a cron job). The reason behind it is that the first time rsync will need to download all the contents of the source servers/folders in the /storage drive so as to create an exact mirror. Once that lengthy step is done, the script can be added in the crontab. Subsequent runs of the script will download only the changed/deleted files.

This method can be very effective while not using a ton of bandwidth every hour. I have used this method for the best part of a year now and it has saved me a couple of times.

The last thing I need to present you is the backup script that I have for my databases. As you notice above the source folder of beryllium.niden.net as far as databases are concerned is beryllium.niden.net/db/. What I do is I dump and zip the databases every hour on my servers. Although this is not a very efficient way of doing things and it adds to the bandwidth consumption every hour (since the dump will create a new file every hour) I have the following script running on my database servers every hour at the 45th minute:

#!/bin/bash

DBUSER=mydbuser
DBPASS='dbpassword'
DBHOST=localhost
BACKUPFOLDER="/niden_backup"
DBNAMES="`mysql --user=$DBUSER --password=$DBPASS --host=$DBHOST --batch --skip-column-names -e "show databases"| sed 's/ /%/g'`"
OPTIONS="--quote-names --opt --compress "

# Clear the backu folder
rm -fR $BACKUPFOLDER/db/*.*

for i in $DBNAMES; do
    echo Dumping Database: $i
    mysqldump --user=$DBUSER --password=$DBPASS --host=$DBHOST $OPTIONS $i > $BACKUPFOLDER/db/$i.sql
    tar cvfz $BACKUPFOLDER/db/$i.tqz $BACKUPFOLDER/db/$i.sql
    rm -f $BACKUPFOLDER/db/$i.sql
done

That’s it.

The backup script can be downloaded here.

Update: The metric units for the drives were GB not MB. Thanks to Jani Hartikainen for pointing it out.

Related Posts

• December 10, 2009 -- Faster rsync and emerge in Gentoo [How-To] (0)
• November 16, 2009 -- Gentoo Stage 1 Installation [How-To] (1)
• August 1, 2010 -- Subversion Backup [How-To] (5)
• January 10, 2010 -- Create a SSL Certificate (Linux) [How-To] (4)
• November 24, 2009 -- Chromium OS Part 3 [How-To] (1)

Anyone that has a Google Apps account and wants to access other services like Google Reader, Google Voice etc. knows that the username and password of the Google Apps account does not work for these services, since those are not available for Google Apps accounts. To get around this limitation, what you could do (and what I have done in the past) is to sign up for a new Google Account with the same email address as your Google Apps account.  This of course creates confusion at times, duplication of data and disassociation of services. The most obvious example of this is on an Android device. To use my phone, I need to sign in with my Google Apps account. However, to use my Google Voice number, I  have to use sign in again for that service but now using my Google Account (which uses the same email address). This works but I still have to keep two sets of contacts – one for the Google Apps and one in Google Voice.

According to Google, 9 of the top 20 requests from Google Apps customers are for their accounts to work with more services from Google. To facilitate this, the Google Apps account had to be merged with the Google Account. This page in Google Apps Help that describes the transition.

Unfortunately this was not a simple flip of a switch for Google. Significant infrastructure changes had to be in place prior to the merging of the accounts. We were promised that this change would be in place by fall and we have not been disappointed. Google is rolling out the update slowly but if you want to ‘speed up’ the process, you can sign up for an early round of testing of the new infrastructure. I have signed up for several of my Google Apps domains a couple of weeks ago.

I am happy to announce that one of my domains has already gone through the merge process. The domain is BeautyAndTheGeek.IT which is a Google Apps Premier edition. My other domains that are hosted in Google Apps have not transitioned yet and I suspect that Google is first merging Premium accounts, then Educational, Government and it will finish up with the Standard edition of Google Apps, which kind of makes sense (paid customers first!).

Email Invitation

Invitation Email to the Admin

The day before yesterday I received an email inviting me to participate in the new infrastructure change and merge the Google Apps accounts with the Google Accounts of my users.

The merge process works as follows:

- If your Google Apps account email address has never been used to access Google products such as Google Reader, Google Voice, Google Code etc. then your Google Apps account will be converted to a Google Account.

- If your Google Apps account email address has been used to access Google products such as Google Reader, Google Voice, Google Code etc. then your Google Apps account is conflicting with your Google Account. This page in Google Apps Administrator Help provides an overview of conflicting accounts. In short both of your accounts will merge and the data will be associated with your Google Apps account, which will now be the same as your Google Account.

There are certain limitations to this merge (as one would expect ). These are as follows:

1. If the offline feature is enabled in GMail, the user has to synchronize before the transition. If they don’t, they will lose their offline messages.
2. Offline Google Calendar doesn’t work for Google Apps accounts that have transitioned.
3. The following Google products don’t work with Google Apps accounts that have transitioned:
   • Android Market for Developers
   • Google Extra Storage (bummer)
   • Health (bummer again)
   • PowerMeter
   • Profiles
   • Web History
   • YouTube
4. The purchase of additional Google Storage is currently unavailable for Google Apps accounts. No storage can be purchased for a Google Apps account, although it will be available later this year. If a user account has already purchased extra storage (like I have) in an existing Google Account, the storage will remain in that account and will not transfer to the respective Google Apps account after the transition.
5. Delegating email only works with the same account type (i.e. Google Apps)
6. Any Picasa Web Album, Profile, or Wave usernames cannot be moved from an existing account to your Google Apps account.

Resources: Transition readiness checklist, Conflicting accounts, Early adopters, Additional storage for Google Apps, Google Apps Administrator Help Center

The process

Google Apps – Dashboard Warning

Dashboard Warning

Once I logged in my Google Apps Administration area, the Dashboard presented a new notice at the top. In short the notice states that new services will be available to the Google Apps accounts. This will be achieved with the transition of the Google Apps Account to a Google Account. The update is free and Google will automatically roll it out on September 30, 2010.

Resource: Google Apps core suite

Google Apps – Understand Transition

Understanding Transition

Clicking the “Get Started” button at the bottom of the notice will start a wizard that helps with the transition of the domain’s accounts to Google accounts of your users (some or all). The first screen contains information that aids in understanding the transition and what is involved.   Moving to the new infrastructure will update your control panel and give you control over which Google services your users can access with their accounts.

Resource: Transition readiness checklist, Google Apps Administrator Help Center

Google Apps – New Services

Review New Services

Based on the services that are available for your domain user accounts, the screen above might be slightly different. It does however give you (the administrator) the ability to enable or disable services for your users. Note that turning off a service will disallow your users to sign up for that service using their Google Apps account for your domain. It will however not stop them from using a totally different Google Account (personal for instance) to access that service.

Google Apps – Notify Conflicting Accounts

Notify Conflicting Accounts

Confirming the new services, the wizard will display a new screen, that will allow you to notify those users that have conflicting accounts. This screen provides information on how many users have conflicting accounts (in my case only 1), what happens to those accounts (the merge process) and what should you do as the administrator. Google will not provide information on which of your users have conflicting accounts. What they will however do, is offer you a temporary email address and an email template, that you can use to email your users. The email template is shown in the “Google Apps – Email to User“ section below.

Google Apps – Select Users

Select Users

Clicking “Continue“, brings up the User Selection screen. Here the administrator can select whose account will be transitioned for now. You can choose to pilot test this transition with a small set of users or everyone. The difference is that if you choose a small set of users to pilot test this transition, you can revert their accounts back to what they were prior to this step. If you choose “Everyone” from this screen, the change will be across the organization and cannot be undone. Reminder here that the transition will happen either way by the end of September, 2010. Finally you have the option to inform the users when their account transition is complete. This generates an email (in English) to the user with relevant information. You can see the email in section “Google Apps – Email to User” below.

Resource: Early adopters

Google Apps – Confirmation

Confirmation

Clicking “Continue“again will bring up the final screen which is the confirmation page. Google is very thorough and I like the confirmation screen. The information in this screen is a summary of what the wizard collected in previous steps. You (the administrator) will have to confirm in several places that you have understood the process, read the agreement and then accept the whole process. Clicking “I accept. Start the transition.” will make things happen

Resources: Transition readiness checklist

Google Apps – Transition in progress

Transition in Progress

Navigating back to the dashboard, you will see a message notifying you that there is a transition in place for the accounts of your domain and can take up to 24 hours to be completed. This serves as a reminder on what has just happened. The notice will disappear once the transition is completed.

Google Apps – Email to User

Email to User

If you (the administrator) chose to notify your users of this transition (see step Select Users above), they will receive an email like the one shown above. If you chose to notify your users with a different message then that message should have been sent to the temporary email address that Google has provided (see step Notifying Conflicting Accounts above)

Resources: Data ownership, Transition readiness checklist, Google Apps Administrator Help Center

Google Apps - Email to User after Transition

Email to User after Transition

Finally another email will be sent to the user summarizing what has happened with their account. This email contains links to additional resources.

Resources: List of products, New sign in option,  Data ownership, What is a browser?, Using multiple accounts, Help Center for Admins, Google Apps Administrator Help Center

Google Apps - User Login message for Account Merge

Once the transition operation is completed, the next time the user logs in their account, they will see the screen above. In short this screen informs the user of what has happened and provides links to online resources for help. The user must click “I accept. Continue to my account.” for them to have access to their account. This is  a one off process.

Resources: Google Apps Administrator Help Center, Google Security and Privacy, Using multiple accounts, Terms of Service

Conclusion

Without a doubt this is a huge step forward for Google and for us as admins or users. The ability to have a single sign on to use their products not only helps them but us by simplifying everything. That also means that if a hacker guesses that my password is “password1” they have access to all my Google related services… but oh well

Joking apart though, I am really excited that this change has finally occurred. I am guessing that the next steps would be to give users or admins the ability to purchase additional storage for a single account is something that in my view will make quite a lot of money for Google. A lot of people will want to keep all their emails and are nearing or have reached their email quota. Administrators would be happy to pay for certain users that have depleted their email storage allocation without converting their whole Google Apps edition to the Premier one. After all, if your organization has 100 users and 5 of them are near capacity, even if you pay $20 per user it will cost $100 per year. If however the whole account is changed to a Premier account (with 25Gb per mailbox) the cost will go up to $5,000.

Some questions come to mind:

1. Will the user be able to purchase storage for all services or just for email?
2. Will the 1Gb limit from Docs be lifted and will it be combined with storage for mail?
3. $20 a year gives you 80Gb in Picasa. Will Google follow that model or change it? According to the Additional storage for Google Apps page it appears that it will be more expensive now to have more space.
4. If storage is so cheap (Picasa) will it be extended to Google Documents? It appears that 7.5Gb is enough for the vast majority of users as far as email is concerned, but 1Gb for Docs is not enough. For a society moving towards a full electronic document storage this could help a lot.

Still a long way to go until everything clears up. I am not sure that Google has all the answers yet but they are moving forward and this is the most encouraging news of all!

Related Posts

• November 24, 2009 -- Chromium OS Part 3 [How-To] (1)
• November 24, 2009 -- Chromium OS Part 2 [How-To] (2)
• November 24, 2009 -- Chromium OS Part 1 [How-To] (3)
• November 16, 2009 -- Google Apps [Review] (0)
• August 21, 2010 -- Create an inexpensive hourly remote backup [How-To] (3)

I will start this post once again with the words of a wise man:

There are two kinds of people, those who backup regularly, and those that never had a hard drive fail

So the moral of the story here is backup often. If something is to happen, the impact on your operations will be minimal if your backup strategy is in place and operational.

There are a lot of backup scenarios and strategies. Most of them suggest a backup once a day, usually at the early hours of the day. This however might not work very well with a fast paced environment where data changes several times per hour. This kind of environment is usually a software development one.

If you have chosen Subversion to be your software version control software then you will need a backup strategy for your repositories. Since the code changes very often, this strategy cannot rely on the daily backup schedule. The reason being is that, in software, a day’s worth of work usually costs a lot more than the actual daily rate of the programmers.

Below are some of the scripts I have used over the years for my incremental backups, that I hope will help you too. You are more than welcome to copy and paste the scripts and use them  or modify them to suit your needs. Please note though that the scripts are provided as is and that you must check your backup strategy with a full backup/restore cycle. I cannot assume responsibility of something that might happen in your system.

Now that the ‘legal’ stuff are out of the way, here are the different strategies that you can adopt.

svn-hot-backup

This is a script that is provided with Subversion. It copies (and compresses if requested) the whole repository to a specified location. This technique allows for a full copy of the repository to be moved to a different location. The target location can be a resource on the local machine or a network resource. You can also backup on the local drive and then as a next step transfer the target files to an offsite location with FTP, SCP, RSync or any other mechanism you prefer.

#!/bin/bash

# Grab listing of repositories and copy each
# repository accordingly

SVNFLD="/var/svn"
BACKUPFLD="/backup"

# First clean up the backup folder
rm -f $BACKUPFLD/*.*

for i in $(ls -1v $SVNFLD); do
    if [ $i != 'conf' ]; then
        /usr/bin/svn-hot-backup --archive-type=bz2 $SVNFLD/$i $BACKUPFLD
    fi
done

This script will create a copy of each of your repositories and compress it as a bz2 file in the target location. Note that I am filtering for ‘conf’. The reason being is that I have a conf file with some configuration scripts in the same SVN folder. You can adapt the script to your needs to include/exclude repositories/folders as needed.

This technique gives the ability to immediately restore a repository (or more than one) by changing the configuration file of SVN to point to the backup location. If you run the script every hour or so then your downtime and loss will be minimal, should something happens.

There are some configuration options that you can tweak by editing the actual svn-hot-backup script. In Gentoo it is located under /usr/bin/. The default number of backups (num_backups) that the script will keep is 64. You can choose 0 to ‘keep them all’ but you can adjust it according to your storage or your backup strategy.

One last thing to note is that you can change the compression mechanism by changing the parameter of the –archive-type option. The compression types supported are gz (.tar.gz), bz2 (.tar.bz2) and zip (.zip)

Full backup using dump

This method is similar to the svn-hot-backup. It works by ‘dumping’ the repository in a portable file format and compressing it.

#!/bin/bash

# Grab listing of folders and dump each
# repository accordingly

SVNFLD="/var/svn"
BACKUPFLD="/backup"

# First clean up the backup folder
rm -f $BACKUPFLD/svn/*.*

for i in $(ls -1v $SVNFLD); do
    if [ $i != 'conf' ]; then
        svnadmin dump $SVNFLD/$i/ > $BACKUPFLD/$i.svn.dump
        tar cvfz $BACKUPFLD/svn/$i.tgz $BACKUPFLD/$i.svn.dump
        rm -f $BACKUPFLD/$i.svn.dump
    fi
done

As you can see, this version does the same thing as the svn-hot-backup. It does however give you a bit more control over the whole backup process and allows for a different compression mechanism – since the compression happens on a separate line in the script.

NOTE: If you use the hotcopy parameter in svnadmin (svnadmin hotcopy …..) you will effectively be duplicating the behavior of svn-hot-backup.

Incremental backup using dump based on revision

This last method is what I use at work. We have our repositories backed up externally and we rely on the backup script to have everything backed up and transferred to the external location within an hour, since our backup strategy is an hourly backup. We have discovered that sometimes the size of a repository can cause problems with the transfer, since the Internet line will not be able to transfer the files across in the allocated time. This happened once in the past with a repository that ended up being 500Mb (don’t ask ).

So in order to minimize the upload time, I have altered the script to dump each repository’s revision in a separate file. Here is how it works:

We backup using rsync. This way the ‘old’ files are not being transferred.

Every hour the script loops through each repository name and does the following:

1. Checks if the *.latest file exists in the svn-latest folder. If no, then it sets the LASTDUMP variable to 0.
2. If the file exists, it reads it and obtains the number stored in that file. It then stores that number incremented by 1 in the LASTDUMP variable.
3. Checks the number of the latest revision and stores it in the LASTVERSION variable
4. It loops through the repository, dumps each revision (LASTDUMP to LASTVERSION) and compresses it

This method creates new files every hour so long as new code has been added in each repository via the checkin process. The rsync command will then pick only the new files and nothing else, therefore the data transferred is reduced to a bare minimum allowing easily for hourly external backups. With this method we can also restore a single revision in a repository if we need to.

The script that achieves that is as follows:

#!/bin/bash

# Grab listing of folders and dump each
# repository accordingly

SVNFLD="/var/svn"
BACKUPFLD="/backup"
CHECKFLD=$BACKUPFLD/svn-latest

for i in $(ls -1v $SVNFLD); do
    if [ $i != 'conf' ]; then
        # Find out what our 'start' will be
        if [ -f $CHECKFLD/$i.latest ]
        then
            LATEST=$(cat $CHECKFLD/$i.latest)
            LASTDUMP=$LATEST+1
        else
            LASTDUMP=0
        fi

        # This is the 'end' for the loop
        LASTREVISION=$(svnlook youngest $SVNFLD/$i/)

        for ((r=$LASTDUMP; r< =$LASTREVISION; r++ )); do
            svnadmin dump $SVNFLD/$i/ --revision $r > $BACKUPFLD/$i-$r.svn.dump
            tar cvfz $BACKUPFLD/svn/$i-$r.tgz $BACKUPFLD/$i-$r.svn.dump
            rm -f $BACKUPFLD/$i-$r.svn.dump
            echo $r > $CHECKFLD/$i.latest
        done
    fi
done

Conclusion

You must always backup your data. The frequency is dictated by the rate that your data updates and how critical your data is. I hope that the methods presented in this blog post will complement your programming and source control should you choose to adopt them.

Related Posts

• August 21, 2010 -- Create an inexpensive hourly remote backup [How-To] (3)
• January 10, 2010 -- Create a SSL Certificate (Linux) [How-To] (4)
• December 10, 2009 -- Faster rsync and emerge in Gentoo [How-To] (0)
• November 16, 2009 -- Gentoo Stage 1 Installation [How-To] (1)
• November 24, 2009 -- Chromium OS Part 3 [How-To] (1)

A note about these series. It appears that Giorgio Sironi and I had the same idea regarding Design Patterns and blogging about them. He covers the Factory design pattern thoroughly in his blog post, which is recommended reading.

The Problem

I started off my IT career as a network administrator. This was back in the good old Novell 3.11 days. After that it was Novell 4.0, Microsoft Servers etc. Following that I got more and more involved with Visual Basic and when Microsoft decided to move everyone to .NET I chose not to follow and ended up coding in PHP.

Since my programming knowledge came from within (studying, reading articles, trial and error), the problems that I was facing on a daily basis are the same as almost every developer faces. One particularly challenging problem that I had in the VB days as well as the PHP days was repetition of code and how to eliminate it.

When I started programming for the Ferrari Fans Fun Forecast site I was running the site using my apartment’s ADSL line. In the beginning there were only 20 users or so, therefore that setup was fine. The scripts were VBScript against a local instance of Microsoft SQL Server. Later on though, I switched to PHP while keeping Microsoft SQL Server.

Initial implementation

I knew that I would have to change my scripts later on to work against MySQL since I was to change the hosting of the site. Through laziness or poor design (you can pick either or both ) I chose to create a class for Microsoft SQL and later on I would just change it to the MySQL. It seemed the easiest thing to do at the time.

So my class was something like:

class DbMSSQL
{
    private $_conn = null;

    public function connect()
    {
         // Connect
        // This is where we have the connection parameters
        include_once 'connection.inc.php';

        $this->_conn = mssql_connect($host, $user, $password);
        if (!$this->_conn) {
            throw new Exception('Cannot connect to the database.');
        }
    }

    public function disconnect()
    {
         // Disconnect
        mssql_close($this->_conn);
    }

    public function selectdb()
    {
         // Select the db
        $db = mssql_select_db($database, $this->_conn);
        if (!$db) {
            throw new Exception('Cannot select the database');
        }
    }

    public function query($sql)
    {
	    // Query the db
        $_result = mssql_query($sql);

        if (!$_result) {
            throw new Exception('Error in query : ' . $sql);
        }

        $data = array();

        while ($row = mssql_fetch_assoc($_result)) {
            $data[$row['id']] = $row;
        }

        mssql_free_result($_result);

        return $data;
    }
}
?>

Everything worked fine so I did not worry about a thing. A few months later though I was forced to move the database (as I expected) to MySQL. I could not move everything in one go so I had to move some of the tables initially and a week later everything else.

To tackle this requirement I created a second class to handle operations against MySQL. The class that I ended up with was:

class DbMySQL
{
    private $_conn = null;

    public function connect()
    {
         // Connect
        // This is where we have the connection parameters
        include_once 'connection.inc.php';

        $this->_conn = mysql_connect($host, $user, $password);
        if (!$this->_conn) {
            throw new Exception('Cannot connect to the database :' . mysql_error());
        }
    }

    public function disconnect()
    {
         // Disconnect
        mysql_close($this->_conn);
    }

    public function selectdb()
    {
         // Select the db
        $db = mysql_select_db($database, $this->_conn);
        if (!$db) {
            throw new Exception('Cannot select the database : ' . mysql_error());
        }
    }

    public function query($sql)
    {
	    // Query the db
        $_result = mysql_query($sql);

        if (!$_result) {
            throw new Exception('Error in query : ' . mysql_error() . "\n" . $sql);
        }

        $data = array();

        while ($row = mysql_fetch_assoc($_result)) {
            $data[$row['id']] = $row;
        }

        mysql_free_result($_result);

        return $data;
    }
}
?>

You can easily see the problem here. There is a lot of repetition in the code, not so much as the actual method properties but the methods themselves. Both classes have connect(), disconnect(), selectdb() and query() as methods. In reality the code changes only slightly since the call for an operation against Microsoft SQL Server is mssql_* while for MySQL is mysql_*. During the transition week I was in programming hell. At some point I mixed the class names, I was trying to read and update the wrong server etc. (see Jani Hartikainen‘s post about 6 programming project mistakes you should avoid – I did all that!).

That week though taught me that I need to pay more attention in designing rather than going full speed ahead with programming and later on paying the consequences.

After thorough research, I discovered a library that would support both platforms. The library that I found was ADOdb which is a perfect example of the Factory Pattern. I used that library later on for a different project, but just looking at the code and understanding the flow of operations as well as the implementation of the pattern itself was invaluable to me.

Interfaces

First of all I need to explain what an interface is and why we need them. According to PHP.net:

Object interfaces allow you to create code which specifies which methods a class must implement, without having to define how these methods are handled.

So imagine an interface something like a graft, a blueprint on what I need to construct. The Interface will have the common methods and properties that I need to implement.

When dealing with database connections to two different database servers (Microsoft SQL Server and MySQL), I can clearly define a few methods that will follow CRUD (Create, Read, Update, Delete). Those are:

1. Connect to the database server
2. Select database
3. Insert record
4. Delete record
5. Update record
6. Select record(s)
7. Close connection to the database server

My interface would therefore be:

interface iDatabase
{
    public function connect();
    public function disconnect();
    public function selectdb();
    public function query($sql);
}
?>

Design Patterns – Factory

A class implementing the Factory Pattern is like a car manufacturing plant producing three different cars on the same assembly line. All cars have common characteristics like 4 wheels, 4 doors (well most of them), a steering wheel, a dashboard etc. and all of them perform certain operations i.e. drive, reverse etc.

In my problem earlier I could have used the Factory Pattern to create one class that would have implemented my blueprint, the interface which defines the CRUD operations that I need. So based on the above, the implementation will result in three classes.

MSSQL class – stored in the file Db_mssql.php

class Db_mssql implements iDatabase
{
    private $_conn = null;

    public function connect()
    {
         // Connect
        // This is where we have the connection parameters
        include_once 'connection.inc.php';

        $this->_conn = mssql_connect($host, $user, $password);
        if (!$this->_conn) {
            throw new Exception('Cannot connect to the database.');
        }
    }

    public function disconnect()
    {
         // Disconnect
        mssql_close($this->_conn);
    }

    public function selectdb()
    {
         // Select the db
        $db = mssql_select_db($database, $this->_conn);
        if (!$db) {
            throw new Exception('Cannot select the database');
        }
    }

    public function query($sql)
    {
	    // Query the db
        $_result = mssql_query($sql);

        if (!$_result) {
            throw new Exception('Error in query : ' . $sql);
        }

        $data = array();

        while ($row = mssql_fetch_assoc($_result)) {
            $data[$row['id']] = $row;
        }

        mssql_free_result($_result);

        return $data;
    }
}
?>

MySQL class – stored in the file Db_mysql.php

class Db_mysql implements iDatabase
{
    private $_conn = null;

    public function connect()
    {
         // Connect
        // This is where we have the connection parameters
        include_once 'connection.inc.php';

        $this->_conn = mysql_connect($host, $user, $password);
        if (!$this->_conn) {
            throw new Exception('Cannot connect to the database :' . mysql_error());
        }
    }

    public function disconnect()
    {
         // Disconnect
        mysql_close($this->_conn);
    }

    public function selectdb()
    {
         // Select the db
        $db = mysql_select_db($database, $this->_conn);
        if (!$db) {
            throw new Exception('Cannot select the database : ' . mysql_error());
        }
    }

    public function query($sql)
    {
	    // Query the db
        $_result = mysql_query($sql);

        if (!$_result) {
            throw new Exception('Error in query : ' . mysql_error() . "\n" . $sql);
        }

        $data = array();

        while ($row = mysql_fetch_assoc($_result)) {
            $data[$row['id']] = $row;
        }

        mysql_free_result($_result);

        return $data;
    }
}
?>

Notice that both these classes are almost identical to the initial implementation shown earlier in this post. The only difference is that they are both implementing the iDatabase interface.

So what is different now? The class that implements the Factory Pattern.

class Db
{
    public static function factory($type)
    {
        $fileName = 'Db_' . strtolower($type) . '.php';
        if (!file_exists($fileName)) {
            throw new Exception('File not found : ' . $fileName);
        }

        $className = 'Db_' . strtolower($type);

        return new $className;
    }
}

What this class does now is it allows me to load the relevant database connection class on the fly. If I want a Microsoft SQL connection I would call:

    $mssql = Db::factory('mssql');

while for MySQL the command becomes:

    $mysql = Db::factory('mysql');

Again since both underlying classes implement the iDatabase interface, I know exactly what to expect as far as methods and functionality is concerned from each class.

Conclusion

The Factory Design Pattern is one of the most powerful design patterns. It provides ‘decoupling’ i.e. breaks the inherited dependency of a class and its subclasses. It also allows for great flexibility while keeping the same interface for your clients.

Zend Framework uses the Factory Pattern in Zend_Db. Specifically the example on the site shows:

// We don't need the following statement because the
// Zend_Db_Adapter_Pdo_Mysql file will be loaded for us by
// the Zend_Db factory method.

// require_once 'Zend/Db/Adapter/Pdo/Mysql.php';

// Automatically load class Zend_Db_Adapter_Pdo_Mysql
// and create an instance of it.
$db = Zend_Db::factory('Pdo_Mysql', array(
          'host'     => '127.0.0.1',
          'username' => 'webuser',
          'password' => 'xxxxxxxx',
          'dbname'   => 'test'
      ));

The Zend_Db factory accepts the name of the adapter used for the database connection as the first parameter while the second parameter is an array with connection specific information. With the use of the Factory Pattern, Zend_Db exposes a common interface which allows programmers to connect to a number of databases using the same methods. Should in the future the application needs to access a different database, the impact to the developer is minimal – in most cases a change to the adapter name (first parameter of the factory class) is all it takes.

Related Posts

• January 22, 2010 -- Design Patterns – Singleton [Series][How-To] (12)
• January 6, 2010 -- Design Patterns – Series [Series][How-To] (0)
• December 15, 2009 -- URL Beautification [How-To] (0)
• December 11, 2009 -- Variables in php.ini [How-To] (1)
• November 16, 2009 -- Flexible Storage in MySQL [How-To] (0)

A note about these series. It appears that Giorgio Sironi and I had the same idea regarding Design Patterns and blogging about them. He covers the Singleton design pattern thoroughly in his blog post, which is recommended reading.

The Problem

When I started programming in PHP I was faced with creating a simple database driven web page for a Ferrari Fans Fun club. The page had 5 different sections that each accessed the database to retrieve data. Each section was included in more than one page and only the menu, header and footer were common in all pages.

My first design and implementation was horrible. I still have the files and for the purposes of this blog post I went back and checked on them and I can safely say I am ashamed of that code. But then again we all start from somewhere so that was my start and more importantly I do not program like that any more. The picture below shows how the page was constructed:

Each of the sections was a different php script (header.php, menu.php, content.php, footer.php, info.php) and in order to retrieve information from the database for each section I had the following snippet at the top of each script:

$dbName = 'FFFF';
$dbUser = 'ffff_user';
$dbPass = 'mypassword';
$dbHost = 'localhost';

$conn = mysql_connect($dbHost, $dbUser, $dbPass);
if (!$conn) {
    die(('Cannot connect to the database :' . mysql_error());
}

$db = mysql_select_db($dbName, $conn);
if (!$db) {
    die ('Cannot select the database : ' . mysql_error());
}

Some might comment on my error handling or the naming of the variables. That is not the problem. The problem is that the snippet of code above was used in every script file (all 5 of them). As a result every page load was hitting the database 5 times. Although the intended user base was no more than 100 people, due to this design flaw I had the equivalent of 500 users.

The first step – Primitive refactoring

You might argue that the two files (header, menu) can easily be combined into one (and the same with additional information and footer) and that will save me 3 connections. The layout does not change but now each of the shaded areas represent one script:

Although this is a good start it is not the solution to the problem. I have effectively reduced the number of hits to 3 per visitor (300 vs 500 before). The goal is to have one connection per visitor.

One step further – A global variable

I need to create the database connection, store it in a global variable, and then let the rest of the scripts access that variable – and subsequently the database connection – when needed. The pseudo code is as follows:

1. Load script header.php
2. Get the database credentials
3. Create a database connection
4. Select the database
5. Display the data
6. Load script content.php
7. Get the database connection
8. Display the data
9. Load script footer.php
10. Get the database connection
11. Display the data

I need to ensure that my database connection is initiated at the beginning of every page. The script header.php is the most obvious place:

$dbName = 'FFFF';
$dbUser = 'ffff_user';
$dbPass = 'mypassword';
$dbHost = 'localhost';

$DBconn = mysql_connect($dbHost, $dbUser, $dbPass);
if (!$DBconn) {
    die(('Cannot connect to the database :' . mysql_error());
}

$db = mysql_select_db($dbName, $DBconn);
if (!$db) {
    die ('Cannot select the database : ' . mysql_error());
}

In every script thereafter I need to reference the global variable and I can then use it in that script:

global $DBconn;

Although this is an “acceptable” way of programming, maintaining all the global variables can easily be a nightmare for

• maintenance
• testing
• quality control
• any part of the code in any script that references this global variable can effectively change that variable
• that the code will look “ugly” (hey I am proud of the code that I write )

Design Patterns – Singleton

A better approach to solve this problem is to use a design pattern. In this case I will use the Singleton Pattern.

The Singleton pattern is applied to a class which when called will create a database connection if the connection does not exist or pass the connection back to the caller if it has already been instantiated. This way I really do not care where the database credentials will be added and when the connection will be instantiated. The first time that I am calling the class that implements the Singleton pattern will connect to the database and have the connection stored ready to be used. The pseudo code is as follows:

1. Load script header.php
2. Get the database credentials
3. Create a database connection
4. Select the database
5. Display the data
6. Load script content.php
7. Get the database connection
8. Display the data
9. Load script footer.php
10. Get the database connection
11. Display the data

The class that I created is as follows:

<?php
class Db
{
    private static $_db   = null;
    private static $_conn = null;

    private function __construct()
    {
        // This is where we have the connection parameters
        include_once 'connection.inc.php';

        $this->_conn = mysql_connect($host, $user, $password);
        if (!$this->_conn) {
            throw new Exception('Cannot connect to the database :' . mysql_error());
        }

        $db = mysql_select_db($database, $this->_conn);
        if (!$db) {
            throw new Exception('Cannot select the database : ' . mysql_error());
        }
    }

    private function __destruct()
    {
        mysql_close($this->_conn);
    }

    // The singleton method
    public static function getInstance()
    {
        if (null === self::$db) {
            self::$_db = new Db($options);
        }

        return self::$_db;
    }

    public function query($sql)
    {
        $_result = mysql_query($sql);

        if (!$_result) {
            throw new Exception('Error in query : ' . mysql_error() . "\n" . $sql);
        }

        $data = array();

        while ($row = mysql_fetch_assoc($_result)) {
            $data[$row['id']] = $row;
        }

        mysql_free_result($_result);

        return $data;
    }
}
?>

With this class available I really do not care if my database connection code is at the beginning of my scripts or not. Using this class allows me to create the database connection (if it is not established) and persist/reuse it further down the script execution.

The code for my header and menu (see graphic above) becomes:

    $sql = 'SELECT menu_id, menu_name FROM tbl_menu';
    $menu = Db::getInstance()->query($sql);

while the one for the rest of the site is exactly identical sans the query to be executed. The problem is solved (I now have one connection per visitor) and the code seems a lot tidier.

Note that the connection parameters are in a separate file which is accessed during the __construct() method of the class. You can use anything you want to supply these parameters in your class.

Conclusion

The Singleton Design Pattern is a blessing in disguise. If the ground work has not been done (i.e. create tests for your code and thoroughly document it) then it is difficult for a new developer coming into a project to understand what is going on, especially when the new developer needs to make alterations and run newly created tests.

A word of caution: If you choose to use this pattern in your application, make sure that everything you do is thoroughly documented and tested. This will make your life a lot easier in the long run and will aid in maintenance.

Update: Thanks to Jani Hartikainen for pointing out an error in the code.

Related Posts

• February 2, 2010 -- Design Patterns – Factory [Series][How-To] (0)
• January 6, 2010 -- Design Patterns – Series [Series][How-To] (0)
• December 15, 2009 -- URL Beautification [How-To] (0)
• December 11, 2009 -- Variables in php.ini [How-To] (1)
• November 16, 2009 -- Flexible Storage in MySQL [How-To] (0)

There are times that I want to set up a secure communication with the server I am working on. This might be because I want to run phpMyAdmin over SSL (I do not like unencrypted communications over the Internet), install a certificate for an eShop for a client or just for my personal use.

The first time I did this, I had to research on the Internet and after a bit of a trial and error I managed to get everything working. However if you do not do something on a regular basis you will forget. I am no exception to this rule hence this post to allow me to remember what I did and hopefully help you too.

Prerequisites:

This how-to assumes that you are running Gentoo, however these instructions can easily be applied to any other Linux distribution.

I need to check if openssl is installed:

vanadium ~ # emerge --pretend dev-libs/openssl

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild   R   ] dev-libs/openssl-0.9.8l-r2

If you do not see the [R] next to the package (and you see a N for instance) that means that you need to install the package. Issuing:

vanadium  ~ # emerge --verbose dev-libs/openssl

will do the trick.

Generate the Private Key

I like to generate keys with a lot of bits. All of my certificates have 4096 bits. This is a personal preference and it does not hurt to keep that value. Your host or Signing Authority (like GoDaddy, VeriSign, Thawte etc.) might ask you in their instructions to generate one with 2048 bits so don’t be alarmed there.

Creating the RSA private key with 4096 bits using Triple-DES:

vanadium ~ # openssl genrsa -des3 -out /root/vanadium.niden.net.locked.key 4096
Generating RSA private key, 4096 bit long modulus
.............................................................++
...........++
e is 65537 (0x10001)
Enter pass phrase for /root/vanadium.niden.net.locked.key:
Verifying - Enter pass phrase for /root/vanadium.niden.net.locked.key:

Remove the passphrase from the Private Key

The key that was created earlier has a passphrase. Although this is good, it does have a side effect that any web server administrator does not like – the passphrase itself. Once the certificate is installed using the key (with the passphrase), every time that Apache is restarted, it will prompt the operator for the passphrase. This can be very inconvenient if your web server reboots in the middle of the night. Since Apache will be waiting for the passphrase, your site will be inaccessible.

To avoid this inconvenience, I am removing the passphrase from the key. If you noticed the key that I have created above has the ‘locked’ phrase in its name. The reason is that I know that that particular key has the passphrase on it. I first need to copy the key and then remove the passphrase:

vanadium ~ # cp -v vanadium.niden.net.locked.key vanadium.niden.net.key
`vanadium.niden.net.locked.key' -> `vanadium.niden.net.key'
vanadium ~ # openssl rsa -in vanadium.niden.net.locked.key -out vanadium.niden.net.key
Enter pass phrase for vanadium.niden.net.locked.key:
writing RSA key

Generate the Certificate Signing Request (CSR)

The purpose of the CSR is to be sent to one of the Certificate Authorities (GoDaddy, VeriSign, Thawte etc.) for verification. Alternatively I can self-sign the CSR (see below).

Upon generation of this CSR I am asked about particular pieces of information to be incorporated in the CSR. The most important piece of information that I need to ensure that is correct is the “Common Name“. The answer to that question has to be the name of my web server – vanadium.niden.net in my case.

NOTE: I am using the key without the passphrase.

The command to generate the CSR is as follows:

vanadium ~ # openssl req -new -key vanadium.niden.net.key -out vanadium.niden.net.csr
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:Virginia
Locality Name (eg, city) []:Arlington
Organization Name (eg, company) [Internet Widgits Pty Ltd]:niden.net
Organizational Unit Name (eg, section) []:IT
Common Name (eg, YOUR name) []:vanadium.niden.net
Email Address []:domains@niden.net

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

Once this step is completed, I open the CSR file with a text editor, copy the contents and paste them in the relevant field of the Certification Authority (in my case GoDaddy), so that they can verify the CSR and issue the certificate.

If however this is a development box or you do not want your certificate signed by a Certification Authority, you can check the section below on how to generate a self-signed certificate.

Generating a Self-Signed Certificate

Generating a certificate valid for 365 days, I need to issue the following command:

At this point you will need to generate a self-signed certificate because you either don’t plan on having your certificate signed by a CA, or you wish to test your new SSL implementation while the CA is signing your certificate. This temporary certificate will generate an error in the client browser to the effect that the signing certificate authority is unknown and not trusted.
To generate a temporary certificate which is good for 365 days, issue the following command:

vanadium ~ # openssl x509 -req -days 365 -in vanadium.niden.net.csr -signkey vanadium.niden.net.key -out vanadium.niden.net.crt
Signature ok
subject=/C=US/ST=Virginia/L=Arlington/O=niden.net/OU=IT/CN=vanadium.niden.net/emailAddress=domains@niden.net
Getting Private key

Installation

For my system, the certificates are kept under /etc/apache2/ssl/ so I am going to copy them there (your need to adjust the instructions below to suit your system/installation):

vanadium ~ # cp -v vanadium.niden.net.key /etc/apache2/ssl/
vanadium ~ # cp -v vanadium.niden.net.crt /etc/apache2/ssl/

I also need to open the relevant file to enable the certificate

vanadium ~ # nano -w /etc/apache2/vhosts.d/00_default_ssl_vhost.conf

In that file I need to change the following directives:

SSLCertificateFile /etc/ssl/apache2/vanadium.niden.net.crt
SSLCertificateKeyFile /etc/ssl/apache2/vanadium.niden.net.key

If my certificate was issued by a Certificate Authority, the files that I have received have the CA certificate file. I can enable it in the following line:

SSLCACertificateFile /etc/ssl/apache2/vanadium.niden.net.ca-bundle.crt

Restarting Apache

vanadium ~ # /etc/init.d/apache2 restart
vanadium ~ # /etc/init.d/apache2 restart
 * Stopping apache2 ...                              [ ok ]
 * Starting apache2 ...                              [ ok ]

Navigating to https://vanadium.niden.net should tell me if what I did was successful or not. If your browser (Google Chrome in my case) gives you a bright red screen with all sorts of warnings, that means that

• either you self signed the certificate – in which case it complains about the certificate not being signed by a Certificate Authority or
• you made a mistake and the Common Name in the certificate is not the same as the host name.

Both these errors are easy to fix.

Related Posts

• August 21, 2010 -- Create an inexpensive hourly remote backup [How-To] (3)
• December 10, 2009 -- Faster rsync and emerge in Gentoo [How-To] (0)
• November 16, 2009 -- Gentoo Stage 1 Installation [How-To] (1)
• August 1, 2010 -- Subversion Backup [How-To] (5)
• November 24, 2009 -- Chromium OS Part 3 [How-To] (1)

A lot of developers – including myself – have at some point in their programming careers found themselves repeating a task that they have completed for a previous project or for a previous part of the same project.

This problem of repeating code is solved by design patterns. A design pattern is the way that the code must be structured so that the problem at hand can be solved. The characteristics of design patterns are:

• Name – it lets other programmers know what this pattern does.
• Problem – where the pattern can be applied
• Solution – how this pattern is implemented

In the course of the next few weeks, I will publish a series of blog posts, where  I will try to present you with some common design patterns and how I have understood their use. Most books and online materials concentrate on just presenting the issue at hand and do not focus on how we ended up there. In these blog post series I will try to show you what I found along the way and how I solved each problem in turn.

As usual comments are more than welcome.

The design patterns I will discuss in posts of these series are as:

1. Singleton Pattern
2. Factory Pattern
3. Registry Pattern
4. Decorator Pattern
5. Adapter Pattern
6. Iterator Pattern
7. Strategy Pattern
8. Active Record Pattern
9. Value Object Pattern
10. Model View Controller Pattern
11. MockObject Pattern
12. Observer Pattern
13. Proxy Pattern
14. Table Data Gateway Pattern
15. Data Mapper Pattern

You can check this blog every week, subscribe to the RSS feed, follow me on Twitter or follow my Google Reader (or all of the above).

Related Posts

• February 2, 2010 -- Design Patterns – Factory [Series][How-To] (0)
• January 22, 2010 -- Design Patterns – Singleton [Series][How-To] (12)
• December 15, 2009 -- URL Beautification [How-To] (0)
• December 11, 2009 -- Variables in php.ini [How-To] (1)
• December 2, 2009 -- The world with <angular/> Part 3 [Review][How-To] (1)

The last few years I have been using Quicken to manage the home bank accounts and spending. My wife never had any problems with me doing so and it helps us manage our cash flow better. Of course like every couple we buy things that we need and always provision a small amount of money for personal expenses.

Some time ago when downloading the transactions from our bank, I noticed a rather odd name coming in for a very small charge (less than 20 dollars). So I asked my wife what that shop is. Her response was “Beautification“. My reply was “What do you mean? You don’t need anything – you are gorgeous as is…“. Although this did earn me brownie points it also infuriated my wife since she was thinking I was auditing her. My sole purpose of that question was to assess the category of the expense and move on. I was not interested the least about the particulars of the expense. I got my reply – a detailed one – as to what the item was for (makeup really) and once my wife finished I explained to her that all this explanation was unecessary and all I needed was “Personal Care“…. The night ended in laughter as you can imagine and since then we always refer as “Beautification” when we pass the makeup area in the grocery store.

The purpose of this post is not to try to make you go buy makeup. It is however a post that can show you how you can make your application’s behavior as far as URLs are concerned “prettier”.

“Beautification” in URLs is also known as Pretty URLs in some cases. It is a method where mod_rewrite is used to make a URL look easier to remember. Although this process started in an attempt to increase SEO and make a site’s URLs easier to index/follow, it has now become a must for every site that wants to create awareness and make people remember as much as possible so that they can revisit. WordPress (this blog’s blogging software) as well as other blogging software use this methodology to ensure that the posts are easily indexed and remembered (see the URL of this post and the title of this post).

First Step

Instead of rewriting my whole application to have beautiful and easy to remember URLs, I started with changing the way I processed parameters. Imagine the scenario where I have a site with various pages and one script that processes everything. My URL could be something like:

..../site.php?page=contact-us
..../site.php?page=about

Following discussions with other developers and reading the web the URL was beautified as:

..../site.php?page/contact-us
..../site.php?page/about

This effectively is one parameter, so in my bootstrap I have the following snippet:

$getData  = array();
$params = explode('/',$_SERVER['QUERY_STRING']);
$params[] = '';
$paramsLength = sizeof($params);
for ($counter = 0; $counter < $paramsLength; $counter++) {
    $getData[$params[$counter]] = (isset($params[$counter + 1])) ? $params[$counter + 1] : '';
}

unset($_GET);

The $getData array contains the parameters that have been passed and allows all search engines to index the URL a lot easier since this is considered to be one parameter.

Please note that I am not going to expand on security here. In my production code there are multiple checks on the variables passed so as to ensure that there are no SQL injection vulnerabilities.

A better approach with .htaccess

In order to make the URL look a lot more “beautified”, I need to remove the script name and the questionmark of the query string. So my URL can easily be like:

..../page/contact-us
..../page/about

To achieve this, I need (in Apache) mod_rewrite enabled and a couple of directives in httpd.conf

Options +FollowSymLinks
RewriteEngine On

If your configuration is on a virtual host, you can add these directives in your vhosts file.

The .htaccess file that I have in the root folder of my site has the following directives:

Options +FollowSymLinks
RewriteEngine On
RewriteCond %{SCRIPT_FILENAME} !-d
RewriteCond %{SCRIPT_FILENAME} !-f
RewriteRule .* site.php [L]

What the above file does is pass everything in the site.php script. The processing now falls in the site.php script. At the top of the script I have:

<?php

// This is where we get all the parameters
$params = $_SERVER['REQUEST_URI'];

if (substr('/', $params) > 0) {
    switch ($params) {
        case '/page/contact-us':
            include 'contact_us.php';
            break;
        case '/page/about':
            include 'about.php';
            break;
        default:
            include 'filenotfound.php';
            break;
    }

// No parameters passed - display something default
} else {
   include 'filenotfound.php';
}

From here on the possibilities are endless. I can add more logic to the rewrite rules so that everything ends in .html for instance, making search engines think that they are visiting unique html pages. Discussion forum software uses this technique to make their content easily searchable. For instance a URL like:

/showpost.php?p=557799

can be shown as:

/show/post/post/557799

or

/show/post/post-557799.html

Something like the above can be achieved with the following rule in .htaccess:

Options +FollowSymLinks
RewriteEngine On
RewriteRule ^post-([0-9]+)+\.html$ /showpost.php?p=$1 [NC,L]

Zend Framework

If I want to push the envelope further I will need to use Zend Framework to do all this effortlessly. Zend Framework’s front controller provides amazing flexibility in terms of how my application URLs can be displayed on the web.

Zend_Controller_Router_Route, Zend_Controller_Router_Route_Static and Zend_Controller_Router_Rewrite are some of the types of routes available in the front controller. Using a configuration file or issuing relevant directives in the bootstrap file are options that I can use to manipulate my application’s URLs.

By default the front controller uses the controller/action scheme but I can easily change this using a router and adding that router to my controller.

$router = $frontController->getRouter();

$router->addRoute(
    'post',
    new Zend_Controller_Router_Route(
        'post/:post',
        array('controller' => 'post',
              'action' => 'show'))
    );

This is a small example on how a simple controller/action sequence can still work as expected and the resulting URL is “beautified”.

Final thoughts

Having easy to remember URLs in an application is a must. It will not only help search engines crawl your site easier – thus making your site more easily discoverable – but it will also help your users remember key areas of your site.

Zend Framework is by far one of the best solutions available, since it introduces a small learning curve in terms of routing and mod_rewrite while providing pretty URLs. However, for more a complicated rewriting scheme your application might need a very sophisticated .htaccess file.

Related Posts

• February 2, 2010 -- Design Patterns – Factory [Series][How-To] (0)
• January 22, 2010 -- Design Patterns – Singleton [Series][How-To] (12)
• December 11, 2009 -- Variables in php.ini [How-To] (1)
• January 6, 2010 -- Design Patterns – Series [Series][How-To] (0)
• December 2, 2009 -- The world with <angular/> Part 3 [Review][How-To] (1)

In my workplace we have been using Zend Framework for more than a year primarily as a glue framework. We have managed to integrate a lot of components from the framework to suit our needs and slowly we are moving towards the full MVC pattern.

In the meantime our own framework, or collection of code if you prefer, is slowly evolving with every new project. New classes are added while others are enhanced more to provide more flexible input/output. Add to that the continuous evolution of Zend Framework and we are faced with at least one version of our own codebase per project.

Lately we have been following a more Test Driven Development approach, where we create tests for every component we create or alter. After all this is the purpose of TDD – to make sure that your code has 100% coverage and that it works!

The same thing applies when a new version of Zend Framework is released. We download the latest version on the development boxes and run our tests against it. If something has changed and/or is broken we just fix it and our framework is ready for the next project.

Following this approach, the bulk of our projects are running the latest code (Zend Framework and our own framework). There are however a number of projects that we need to run against an older version of Zend Framework. The reasons for not are “not enough money for updating“, “not enough time“, “it works so don’t tinker with it” to name a few.

So how do we deal with the issue of having different versions of Zend Framework on our servers and including the correct version to the relevant project? The solution that we use is variables in php.ini.

For starters we have a network share mounted on all of our development servers (the ones that are in house). This share we call it /resources. In there we have all the needed Zend Framework versions i.e.:

gemini ~ # ls -la /resources/ZendFramework/
total 0
drwxrwxr-x 8 webuser webdev 192 Nov  5 05:48 .
drwxr-xr-x 7 webuser webdev 168 Jul 17 10:55 ..
drwxrwxr-x 3 webuser webdev  72 Jun 23 13:48 1.0.3
drwxrwxr-x 5 webuser webdev 216 Apr 21  2009 1.5.2
drwxrwxr-x 7 webuser webdev 280 Apr 21  2009 1.6.2
drwxrwxr-x 8 webuser webdev 304 Apr 21  2009 1.7.8
drwxrwxr-x 8 webuser webdev 296 Jun  8  2009 1.8.3
drwxrwxr-x 8 webuser webdev 296 Oct 26 12:52 1.9.5

Since this is a network share, all we do is update one location and all servers have access to the same pool of files.

We also modify the php.ini file of each server, adding the following directives at the end of the file:

[ZendFramework]
ZF_LATEST="/resources/ZendFramework/1.9.5/library"
ZF_1_9_5="/resources/ZendFramework/1.9.5/library"
ZF_1_8_3="/resources/ZendFramework/1.8.3/library"
ZF_1_7_8="/resources/ZendFramework/1.7.8/library"
ZF_1_6_2="/resources/ZendFramework/1.6.2/library"
ZF_1_5_2="/resources/ZendFramework/1.5.2/library"
ZF_1_0_3="/resources/ZendFramework/1.0.3/library"

Restarting apache gives us access to those variables using the get_cfg_var PHP function. So in our bootstrap file (called init.php) we have the following:

$zfPath = get_cfg_var('ZF_1_9_5');

$root = dirname(__FILE__);

set_include_path($zfPath         . PATH_SEPARATOR .
                       $root    . '/lib' . PATH_SEPARATOR .
                       $root            . PATH_SEPARATOR .
                       get_include_path());

Changing to a different version of the Zend Framework can be as easy as changing the path of the ZF_* directive in the php.ini file (and restarting Apache) or by changing the variable we need to use in the init.php file of each project.

This approach allows us to keep our include_path free from any Zend Framework paths since the server has projects that use different versions of the Zend Framework as well as ours. It also helps us in testing, since we can easily check whether a particular project works as expected in a newer or older version of each framework.

We opted out from using constants inside .htaccess files, due to the fact that we have many projects and this would have meant that we had to change every .htaccess file, should we wanted to change the version of Zend Framework used.

php.ini has a lot of directives that can prove useful for the PHP developer. Extending those directives to each project or collection of projects can save you time and headaches.

Related Posts

• February 2, 2010 -- Design Patterns – Factory [Series][How-To] (0)
• January 22, 2010 -- Design Patterns – Singleton [Series][How-To] (12)
• December 15, 2009 -- URL Beautification [How-To] (0)
• January 6, 2010 -- Design Patterns – Series [Series][How-To] (0)
• December 2, 2009 -- The world with <angular/> Part 3 [Review][How-To] (1)

Recently I have started setting up a cluster of 7 Gentoo boxes for a project I am working on. The problem with boxes coming right out of the setup process of a hosting company is that they do not contain the packages that you need. Therefore you need to setup your USE flags and emerge the packages you require as per the role of every box.

I have implemented the following procedure many times in my local networks (since I have more than one Gentoo boxes) and have also implemented the same process at work (we run 3 Gentoo boxes).

The way to speed up rsync and emerge is to run a local rsync mirror and to use http-replicator. This will not make the packages compile faster but what it will do is reduce the resource usage (downloads in particular) of your network since each package will be downloaded only one time and reduce the time you have to wait for each package to be downloaded. The same applies with the rsync.

My network has as I said 7 boxes. 5 of them are going to be used as web servers so effectively they have the same USE flags and 2 as database servers. For the purposes of this tutorial I will name the web servers ws1, ws2, ws3, ws4, ws5 and the database servers db1, db2. The ws1 box will be used as the local rsync mirror and will run http-replicator.

I am going to set up the /etc/hosts file on each machine so that the local network is resolved in each box and no hits to the DNS are required. So for my network I have:

10.13.18.101  ws1
10.13.18.102  ws2
10.13.18.103  ws3
10.13.18.104  ws4
10.13.18.105  ws5
10.13.18.201  db1
10.13.18.202  db2

Modify the above to your specific setup needs.

Setting up a local rsync

Server setup (ws1)

There is a really good tutorial can be found in the Gentoo Documentation but here is the short version:

The ws1 box already has the rsync package in there. All I need to do is start the daemon. Some configuration is necessary before I start the service:

nano -w /etc/rsyncd.conf

and what I should have in there is:

# Restrict the number of connections
max connections = 5
# Important!! Always use chroot
use chroot = yes
# Just in case you are allowed only read only access
read only = yes
# The user has no privileges
uid = nobody
gid = nobody
# Recommended: Restrict via IP (subnets or just IP addresses)
hosts allow = 10.13.18.0/24
# Everyone else denied
hosts deny  = *

# The local portage
[niden-gentoo-portage]
path = /usr/portage
comment = niden.net Gentoo Portage tree
exclude = /distfiles /packages

That’s it. Now I add the service to the default runlevel and start the service

rc-update add rsyncd default
/etc/init.d/rsyncd start

NOTE: If you have a firewall using iptables, you will need to add the following rule:

# RSYNC
-A INPUT --protocol tcp --source 10.13.18.0/24 --match state --state NEW --destination-port 873 --jump ACCEPT

Client setup

In my clients I need to edit the /etc/make.conf file and change the SYNC directive to:

SYNC="rsync://ws1/niden-gentoo-portage"

or I can use the IP address:

SYNC="rsync://10.13.18.101/niden-gentoo-portage"

Note that the path used in the SYNC command is what I have specified as a section in the rsyncd.conf file (niden-gentoo-portage in my setup). This path can be anything you like.

Testing

I have already run

emerge --sync

in the ws1 box, so all I need to do now is run it on my clients. Once I run it I can see the following (at the top of the listing):

emerge --sync
>>> Starting rsync with rsync://10.13.18.101/niden-gentoo-portage...
receiving incremental file list
......

So everything works as I expect it.

Setting up http-replicator

http-replicator is a proxy server. When a machine (the local or a remote) requests a package, http-replicator checks its cache and if the file is there, it passes it to the requesting machine. If the file doesn’t exist though, http-replicator downloads it from a mirror and then passes it to the requesting machine. The file is then kept in http-replicator’s cache for future requests. This way I save on resources by downloading once and serving many times locally.

Although this might not seem as a ‘pure speedup’ it will make your installations and updates faster since the download factor will be reduced to a bare minimum. Waiting for packages like mysql, Gnome or others to be downloaded does take a long time. Multiply that time with the number of machines you have on your network and you can see the benefits of having a setup like this.

Server setup (ws1)

First of all I need to emerge the package

emerge http-replicator

Once everything is done I need to change the configuration file to suit my needs:

nano -w /etc/conf.d/http-replicator

and the file should have:

GENERAL_OPTS="--dir /var/cache/http-replicator"
GENERAL_OPTS="$GENERAL_OPTS --user portage"
DAEMON_OPTS="$GENERAL_OPTS"
DAEMON_OPTS="$DAEMON_OPTS --alias /usr/portage/packages/All:All"
DAEMON_OPTS="$DAEMON_OPTS --log /var/log/http-replicator.log"
DAEMON_OPTS="$DAEMON_OPTS --ip 10.13.18.*"
## The proxy port on which the server listens for http requests:
DAEMON_OPTS="$DAEMON_OPTS --port 8080"

The last line with the –port parameter specifies the port that the http-replicator will listen to. You can change it to whatever you want. Also the –ip parameter restricts who is allowed to connect to this proxy server. I have allowed my whole internal network; change it to suit your needs. Lastly the –dir option is where the cached data is stored. You can change it to whatever you like. I have left it to what it is. Therefore I need to create that folder:

mkdir /var/cache/http-replicator

Since I have specified that the user that this proxy will run as is portage (see –user directive above) I need to change the owner of my cache folder:

chown portage:portage /var/cache/http-replicator

Prior to starting the service, there is a small change to be made to the init script. This is a bug which I do not know when it will be fixed but this workaround keeps everything running nice and smooth. I need to edit /etc/init.d/http-replicator and change:

    ebegin "Starting Http-Replicator"
    start-stop-daemon --start --pidfile /var/run/http-replicator...

to

    ebegin "Starting Http-Replicator"
    start-stop-daemon --start --oknodo --pidfile /var/run/http-replicator...

- The paramenter –oknodo has been added.

I saved the file and now I add the service to the default runlevel and start the service

rc-update add http-replicator default
/etc/init.d/http-replicator start

NOTE: If you have a firewall using iptables, you will need to add the following rule:

# HTTP-REPLICATOR
-A INPUT --protocol tcp --source 10.13.18.0/24 --match state --state NEW --destination-port 8080 --jump ACCEPT

You will need also to regularly run

repcacheman

and

rm -rf /usr/portage/distfiles/*

to clear the distfiles folder. I have added those in a bash script and I run it every night using my cron.

Client setup

In my clients I need to edit the /etc/make.conf and change the SYNC directive to:

http_proxy="http://ws1:8080"
RESUMECOMMAND=" /usr/bin/wget -t 5 --passive-ftp  \${URI} -O \${DISTDIR}/\${FILE}"

I have commented any previous RESUMECOMMAND statements.

Testing

The testing begins in one of the clients (you can choose any package):

emerge logrotate

and see in the output that everything works fine

ws2 ~ # emerge logrotate
Calculating dependencies... done!

>>> Verifying ebuild manifests

>>> Emerging (1 of 1) app-admin/logrotate-3.7.8
>>> Downloading 'http://distfiles.gentoo.org/distfiles/logrotate-3.7.8.tar.gz'
--2009-12-10 06:46:47--  http://distfiles.gentoo.org/distfiles/logrotate-3.7.8.tar.gz
Resolving ws1... 10.13.18.101
Connecting to ws1|10.13.18.101|:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 43246 (42K)
Saving to: `/usr/portage/distfiles/logrotate-3.7.8.tar.gz'

100%[=============================>] 43,246      --.-K/s   in 0s

2009-12-10 06:46:47 (89.6 MB/s) - `/usr/portage/distfiles/logrotate-3.7.8.tar.gz' saved [43246/43246]
.....

Final thoughts

Setting up local proxies allows your network to be as efficient as possible. It does not only reduce the download time for your updates but it is also courteous to the Gentoo community. Since mirrors are run by volunteers or non-profit organizations, it is only fair to not abuse the resources by downloading an update more than once for your network.

I hope this quick guide will help you and your network

Related Posts

• August 21, 2010 -- Create an inexpensive hourly remote backup [How-To] (3)
• November 16, 2009 -- Gentoo Stage 1 Installation [How-To] (1)
• January 10, 2010 -- Create a SSL Certificate (Linux) [How-To] (4)
• November 24, 2009 -- Chromium OS Part 3 [How-To] (1)
• November 24, 2009 -- Chromium OS Part 2 [How-To] (2)