Data backup with rsync

     Rsync is a file synchronization and transfer utility.  It can be used to transfer files between two local or remote locations.  It is fast, supports compression and encryption ( through ssh ).  Its usage is simple and can be used as a backup utility.

Syntax:

rsync <options> <source> <destination>

     Both the source and destination can be local directories, or the source can be a remote location ( pull ) or the destination can be remote ( push ).  For remote file transfer rsync can make use of either ssh/rsh shell as a transport or use rysnc as a daemon.  Here we will look at the local copy and ssh based remote copy.

     You can run rsync without any options, but using the following options will be good when you are backing up data.

-a : Archive - preserves symbolic links,permissions,timestamp,user/group
-z : Compression - compress data while transferring - good for remote date transfer

     If you are backing up a directory tree, then you should be using "-r" option to recursively go through all sub directories and files.  But if you are using "-a" option this is not required as -a does recursive archiving.

     Let us see how we can do a local file transfer ( This is the way I backup my home directory to external usb hard disk )

rsync -az /home/safeer  /media/USB_HD_01/Backup_02_2011

     After the first run, I periodically run the same command  to keep the backup in sync with my home directory.
In the subsequent runs, rsync only copies the files that has changed in my home directory since the last run.  This saves a lot of time.

     Any time you want to do a restore (say after an OS reinstall ), all you need to do is reverse the source and destination and run the rsync command.

rsync -az /media/USB_HD_01/Backup_02_2011 /home/safeer

Note: When you provide the source directory name, if it terminates with "/" only the contents under the directory will be transferred.  But if you omit the ending "/" the same directory will be created at the destination and the contents will be copied over.

     Now backing up into a remote server

rsync -az -e ssh /home/safeer safeer@backup.safeer.in:/home/safeer/Backup_02_2011

"-e ssh" specifies the shell to be used for remote transport.

     Now let us backup remote content locally.

rsync -az -e ssh safeer@web.safeer.in:/var/www/html /media/USB_HD_01/Web_02_2011

     The syntax for a remote location is always USER@REMOTE_SERVER:REMOTE_DIRECTORY.  If USER is omitted the local username will be used, if REMOTE_DIRECTORY is omitted users home directory will be used.

     We will check a few useful options

To monitor the transfer:

-v : Verbose - provides a summary of what is happening
--progress : Displays the progress of data transfer.
--stats : Displays statistics abut the file transfer

To select and deselect files/folders:

--include PATTERN : Include files that match the given pattern
--exclude PATTERN : Exclude files that match the given pattern
--include-from FILE and --exclude-from FILE - to include and exclude files based on patterns in the file
--files-from FILE : Transfer only files listed in FILE
--filter RULE : Filter files based on RULE

     All these options are allowed multiple times.

Deletion/Update

-u : Do not transfter/overwrite if destination file is newer than source.
--delete : Delete file that exist in destination but not in source
--delete-before/--delete-during/--delete-delay/--delete-after/--delete-excluded are some options worth checking

For more detailed options and explanations, checkout the rsync man page.

Test your IMAP server with Telnet

     Internet Message Access Protocol version 4 (IMAP 4 ) is an email retrieval protocol which is a lot more robust than the Post Office Protocol version 3 ( POP3).   It provides features like name spaces,folders,sub folders,flags etc.. A discussion of all the features is beyond the scope of this article.  Here we will see how we can test the basic functionalities of an IMAP server using telnet.

     First, establish a connection to the IMAP server using telnet on port 143.

safeer@penguinpower:~$ telnet imap.safeer.in 143
Trying 10.25.0.3...
Connected to imap.safeer.in
Escape character is '^]'.
* OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE ACL ACL2=UNION] Courier-IMAP ready. Copyright 1998-2011 Double Precision, Inc.  See COPYING for distribution information.

     The last line of the output is a greeting from the IMAP server.  Further conversation with the IMAP server will be a series of requests and responses.   Requests are also known as commands and should begin with an arbitrary tag (an alpha numeric string ) followed by a command and its optional arguments, all separated by spaces.  The response, if positive will be the request tag name followed by OK and further by a server message.  This is for a single line response.  If the response is multi line, all except the last line with begin with a "*" and the last line will be request tag followed by OK and then server message.  A negative response will begin with request tag followed by a "NO" and a server message.

     To test the IMAP server at a minimum we should login,select INBOX, retrieve a message and logout.  Let us start by providing a tag followed by the command "login" followed by the IMAP username and password, all separated by spaces.

A01 LOGIN safeer fake_Password
A01 OK LOGIN Ok.

     With IMAP, you can have multiple mailboxes.  "INBOX" is the default and only one mailbox if you have not configured additional mailboxes.  Let us see the mailboxes and its sub folders using list command.  It takes two arguments, first being the folder to list and the second being a search parameter.  Since we are looking for mail boxes, we should be searching the root of the IMAP indicated by a blank "".  The search parameter is a wildcard ( "*" ) here since we want to see all folders under the root.

A02 list "" "*"
* LIST (\Marked \HasNoChildren) "." "INBOX"
1 OK LIST completed

     This is a multi line response from the server and it says there is only one mailbox "INBOX" with no sub folders.  Now let us see the  status of the "INBOX" and find out the number of messages in it.

A03 STATUS INBOX messages
* STATUS "INBOX" (MESSAGES 1)
A03 OK STATUS Completed.

     There is only one message in the INBOX. In order to view messages in the mailbox INBOX ( or any IMAP folder for that matter ) we should first select the folder.

A04 SELECT INBOX
* FLAGS (\Draft \Answered \Flagged \Deleted \Seen \Recent)
* OK [PERMANENTFLAGS (\* \Draft \Answered \Flagged \Deleted \Seen)] Limited
* 1 EXISTS
* 1 RECENT
* OK [UIDVALIDITY 1358664910] Ok
* OK [MYRIGHTS "acdilrsw"] ACL
A04 OK [READ-WRITE] Ok

     In the response, 1 EXISTS means there is only one message in INBOX and 1 RECENT means that 1 message is unread.  We can list/read the emails by using the fetch command.    Let us read the message ( since there is only one message in the INBOX, the id of the message will be 1 and that is passed as an argument to fetch command).  Let us first view the email headers.

A05 FETCH 1 body[header]
* 1 FETCH (BODY[HEADER] {501}
Return-Path: <info@safeer.in>
X-Original-To: safeer@safeer.in
Delivered-To: safeer@smtp.safeer.in
Received: from penguinpower (penguinpower [128.1.0.5])
    by smtp.safeer.in (Postfix) with ESMTP id AD10226089C
    for <safeer@safeer.in>; Thu, 10 Feb 2011 00:42:29 +0530 (IST)
Subject: POP3 Check
Message-Id: <20110210191235.AD10226089C@smtp.safeer.in>
Date: Thu, 10 Feb 2011 00:42:29 +0530 (IST)
From: info@safeer.in

)
* 1 FETCH (FLAGS (\Seen \Recent))
A05 OK FETCH completed.

     You can see the message headers.  Now let us read the message text.

A06 FETCH 1 body[text]
* 1 FETCH (BODY[TEXT] {14}
Testing POP3
)
A06 OK FETCH completed.

     We are done retrieving the message.  Note that fetch command can operate on a single message id or a range or ids in the format START:END.

    Now logout of the session:

A07  LOGOUT
* BYE Courier-IMAP server shutting down
A07 OK LOGOUT completed
Connection closed by foreign host.

    To know more about IMAP checkout the IMAP RFC at ITEF