Configure mutt as IMAP client

     Mutt is a lightweight yet powerful command line email client.  We will look at a minimal configuration for an IMAP account here.

     Mutt has a global configuration file in either /etc/Muttrc or /usr/local/share/mutt/Muttrc.   This is useful for configuring system wide settings.  But to configure your own account you have to configure your personalized muttrc file in your home directory at ~/.muttrc.

     Here is a sample muttrc file that can get yo start  off with mutt.  Explanations are given as comments.

## Set your imap server name and port here.
set folder=imaps://my.imap.server.name:imapPort/
## Where to look for the mails
## the "+" simbol means, append this to the value of "folder" setting
set spoolfile=+Inbox
## Your imap username
set imap_user=my_imap_user
## Set your imap password - Not preferred for security reasons
## If not provided mutt will prompt you for a password
set imap_pass=my_imap_pass
## Setting sent folder ( note the "+" sign )
set record=+Sent
## Setting draft folder ( note the "+" sign )
set postponed=+Drafts
## To cache downloaded header
## Good for performance
## directory should be manually created
set header_cache=~/.mutt/cache/headers
set message_cachedir=~/.mutt/cache/bodies
## Your preferred editor for composing
set editor=vim
## How to sort your mail
set sort='last-date-received'

A quick primer to GNU Screen - The terminal multiplexer

  Screen is a terminal multiplexer that allows you to open up multiple virtual terminal sessions inside a single terminal.  This enables the user to run multiple programs from a single terminal independently and in parallel.  It also allows switching between them and scrolling through the history as if you were using real independent terminal. 

     Now this may not be an attraction in today's Linux desktop environment where you can open up multiple tabs of the terminal program you want to use ( like gnu-terminal, kde konsole etc).  But imagine the case where your local machine don’t have GUI installed or you are connected to a remote host's terminal via ssh connection.  In the case of local machines terminal, there is a limit to the number of terminals a system can have.  In the case of ssh connection, you can open multiple connections but it takes up network and to manage them you need to open up an equal number of local terminals.  Screen solves this limitation by allowing to spawn virtual shell within a single shell and easily navigate between them.

     This is once advantage of screen but the best part is yet to come.  Any program that you run from a shell is usually spawned as a child of the shell/terminal program.  Hence terminating the shell/terminal session will kill any child process running in it.  Screen has the ability to detach itself from the parent shell process and hook itself as a child of the init process.  This makes screen resilient to logout ( for local terminals ) and network disconnects ( for remote terminals ).  This is a great tool for a power user.

Now let us see how we can use screen.

     First install the screen program using your favourite package manager or download from the screen website.

     Simplest invocation of screen is by just typing the command "screen" on your terminal (local/remote).  You might be shown an optional welcome message, press "Enter" to get rid of the message and drop tot he terminal.  This will start the first virtual terminal session and will look like a regular terminal.  You can run any commands/programs you want from this terminal.  You can finish your work on the terminal and type exit or press Control+D as you do on a regular terminal to exit out of screen.  But this do not add any value to our terminal usage. 

     To make the most out of screen, one should know how to spawn additional virtual terminals, detach the screen session allowing the running program to continue independently and browse through the numerous virtual terminals you open.

     To do all these task you have to sent special commands otherwise known as "key bindings" to the screen application, and all of them start with an escape sequence, followed by one or more characters.  The default escape sequence is "Control+a" ( press control key followed by character a ).  By default screen will pass any input to the current virtual terminal, but sending an escape sequence will make screen aware that it is a special command.  Let us see some useful screen commands/key bindings.

   Control+a c ( Press "Control", then "a" followed by "c" ) - Open up a new virtual terminal ( hereafter called "VT")  in screen. 

     You can create any number of VTs in this fashion, each screen will have a number starting from zero, zero being the number of the VT opened when screen is started.

Control+a " - List all VTs. 

    An example screen session with 13 VTs ( 0-13) is given below.  There will be a selection bar on the VT you are currently on.  You can use up/down arrow keys to select a different terminal or press the terminal number to select it ( this work only for terminals from 0-9 ).  Pressing "Enter" will activate the selected VT.

Num Name

 0  bash
 1  bash
 2  bash
 3  bash
 4  bash
 5  bash
 6  bash
 7  bash
 8  bash
 9  bash
10  bash
11  bash
12  bash

Navigation commands

Control+a p - Move to previous VT ( "Control+a Backspace" can also be used )
Control+a n - Move to next VT ( "Control+a Space" can also be used )
Control+a 0-9 - Move to the numbered VT ( works only for terminals 0-9 )
Control+a Control+a - Toggle between current and previous window

Screen Command Prompt

Screen has a command prompt from which you can input commands for screen.

Control+a : - Activate the command prompt for the current VT

Providing "title TITLE_NAME" from the screen command prompt will rename the current VT (same as control+a A )

Typing help at the prompt will show you a full list of helpful commands.

Usability commands

Control+a C - Clear screen
Control+a Esc OR Control+a [ - Activate copy mode ( in copy mode you can scroll up the output , select and copy text).
Enter to copy, Control+a ] to paste
Control+a ? - Show help/key bindings
Control+a w - Show a list of virtual terminals ( displayed at the bottom of current VT, no option to select VTs )
Control+a M - Monitor for activity
control+a _ - Monitor for silence more than 30 seconds

Control+a A - Rename the current VT. 
    
     As you can see, each of these VTs have a number as well as a name.  While the number increases for each VT, the name by default is the name of the shell/command spawned.  While the numbers are convenient, you might lose track of which terminal is doing what.  Giving it a meaningful name would help you identify the VT.  When this command is given, you will be given a prompt with the default name already in it, erase it with backspace, type in your new name and press enter.

A sample screen session with three renamed VTs

Num Name

 0  Mutt Email Client
 1  Perl Scripting
 2  Screen 3


Detach / Attach

   Now that you know how to operate within a screen session, let us see how to detach/attach to en existing screen session.

Detach : You are running programs under multiple VTs inside a screen session.  Now you want to disconnect from this screen for some reason ( logging out, terminating remote ssh connection etc.. ) and still let the programs under the VTs to continue.  This is where you should use the detach option, which is activated by the key binding "Catrl+a d".  When the key combination is pressed, screen drops you off to the shell from where you initiated the screen session.  Now you can logout / disconnect and the programs will keep running.

This is the sample output you get when you detach from a screen session.
[detached from 9108.maverick]

Now to connect back, you should know the screen session's name ( do not confuse this with the VT names inside the screen session ).  To list all current screen sessions, issue the command "screen -ls"

safeer@maverick:~$ screen -ls
There are screens on:
        9108.pts-0.maverick        (Sunday 20 March 2011 03:49:35  IST)    (Detached)
1 Socket in /var/run/screen/S-safeer.

There is only one screen session open right now and the name is 9108.pts-0.maverick.  The format of the screen name is [PID of screen].[TTY Numebr].[hostname].  This is the same name you saw when you where detatching the screen session.

To re attach to the disconnected session, use "screen -r <SCREEN NAME>", ie screen -r 9108.pts-0.maverick.

When re attaching the PID part is optional and can be reattached this way : "screen -r pts-0.maverick"

With a couple of screen sessions, it is easier to track which screen is what, but not always.  So we can custom name the screen sessions, for that start the screen session with "screen -R <screen name>".  This name replaces the [TTY].[hostname] part, but the pid part will remain.  Since PID is optional when reattaching the screen, you can provide just this screen name while reattaching.  See an

safeer@maverick:/var/run/screen$ screen -ls
There are screens on:  
        21454.SAFEER_TEST       (Sunday 20 March 2011 04:03:52  IST)    (Attached)
        9108.pts-0.maverick       (Sunday 20 March 2011 03:49:35  IST)    (Detached)
2 Sockets in /var/run/screen/S-safeer.


     You can re-attach to an attached session active in one terminal from another terminal after detaching it from the current terminal.

Either do it in two steps as

screen -d <SCREEN NAME>
screen -r <SCREEN NAME>

or together

screen -dr <SCREEN NAME>

Multi user mode

     Multi user mode is one speciality of screen where the same or different users in a system can share the same screen session.  You can control the access using ACLs, which allows restrictions based on usernames, VTs, commands and read/write/execute modes.

     To enable multi user mode, we will have to use the screen command prompt invoked by "Control+a :" which will give you a prompt.  Enter "multiuser on" in this prompt and multi user access will be enabled for this screen session.  Once enabled, this screen session can be accessed by other users from different terminals using the command "screen -x <screen name>"

     A complete discussion of all multi user mode is out of the scope of this article, so I will just list the commands that can be used inside screen command prompt.

multiuser on - Enable multi user mode.
display - Show currently connected users
acladd - add screen access for one or more users
acldel - revoke screen access for one or more users
aclchg - Add/modify permissions ( for commands,VTS,read/write/exec ) to one or more users

     To know more about screen, checkout the man page and the official website of screen.

Block device encryption and management in Linux

     The standard for Linux block level encryption is LUKS ( Linux Unified Key Setup ).  DM-Crypt is the Linux kernel's device mapper module for transparently encrypting and decrypting block devices.  To make use of these features, install cryptsetup package.

safeer@lin01:~$yum install cryptsetup-luks

     First, format the LUKS encryption layer (on the partition).  This is for standardizing the partition header and the format of the bulk data.

safeer@lin01:~$sudo /sbin/cryptsetup luksFormat /dev/sdb1
WARNING!
========
This will overwrite data on /dev/sdb1 irrevocably.

Are you sure? (Type uppercase yes): yes
Enter LUKS passphrase:
Verify passphrase:
Command successful.

     The command will ask for a passphrase, make sure you remember the passphrase you provide, as this will be later used to mount the partition.  Optionally you can directly provide a filename containing the passphrase as argument of above command with "--key-file <file-name>".

Open LUKS encryption layer ( device map the encrypted disk )

safeer@lin01:~$sudo /sbin/cryptsetup luksOpen /dev/sdb1 secretDisk
Enter passphrase for /dev/sdb1:

    Provide the password you created during the first setup.  Once authenticated, /dev/sdb1 will be devise mapped to the name you provided in the last command ( secretDisk ).  To verify, run:

safeer@lin01:~$ls -l /dev/mapper/secretDisk
lrwxrwxrwx 1 root root 7 Apr 14 11:10 /dev/mapper/secretDisk -> ../dm-0

Now format the device mapper with appropriate files system, here am chosing ext3.

safeer@lin01:~$sudo /sbin/mkfs.ext3 /dev/mapper/secretDisk

I want to mount this device to /home/safeer/secretDisk.  To mount it permanently, add following line to /etc/fstab

/dev/mapper/secretDisk    /home/safeer/secretDisk        ext3 defaults 0 0

safeer@lin01:~$sudo mount -a

df -h /home/safeer/secretDisk

     During boot up, the devise mapping of encrypted disk should happen prior to mounting, this is achieved by registering the device in the file /etc/crypttab

     For each crypt device, there should be one line in this file.  Aminimum of two fileds is necessary here, first is the mapper name without "/dev/mapper" prefix and the second is the actual encrypted block device.  The third field is passphrase file and if left empy or the value "none" is given, user will be prompted for the passphrase prior to mounting.  If this has to be automated, create a file with the passphrase ( non newline character at the end) and provide the file name as the third field.  The fourth field is options, which can be used to speficy encryption algorithms etc... This can  be left blank in most cases.

safeer@lin01:~$grep secret /etc/crypttab
secretDisk    /home/safeer/secretDisk none

To ensure things are working properly, reboot the host once.

A couple of things to know if you want more advansed setup.

     LUKS allows having 8 passphrases for a partition, this means 8 users can mount the filesystem with passphrases of their own choice and eliminates the need to share a common passphrase.  Check out the cryptsetup man page for options  luksAddKey,luksRemoveKey and luksDump.

Data backup with rsync

     Rsync is a file synchronization and transfer utility.  It can be used to transfer files between two local or remote locations.  It is fast, supports compression and encryption ( through ssh ).  Its usage is simple and can be used as a backup utility.

Syntax:

rsync <options> <source> <destination>

     Both the source and destination can be local directories, or the source can be a remote location ( pull ) or the destination can be remote ( push ).  For remote file transfer rsync can make use of either ssh/rsh shell as a transport or use rysnc as a daemon.  Here we will look at the local copy and ssh based remote copy.

     You can run rsync without any options, but using the following options will be good when you are backing up data.

-a : Archive - preserves symbolic links,permissions,timestamp,user/group
-z : Compression - compress data while transferring - good for remote date transfer

     If you are backing up a directory tree, then you should be using "-r" option to recursively go through all sub directories and files.  But if you are using "-a" option this is not required as -a does recursive archiving.

     Let us see how we can do a local file transfer ( This is the way I backup my home directory to external usb hard disk )

rsync -az /home/safeer  /media/USB_HD_01/Backup_02_2011

     After the first run, I periodically run the same command  to keep the backup in sync with my home directory.
In the subsequent runs, rsync only copies the files that has changed in my home directory since the last run.  This saves a lot of time.

     Any time you want to do a restore (say after an OS reinstall ), all you need to do is reverse the source and destination and run the rsync command.

rsync -az /media/USB_HD_01/Backup_02_2011 /home/safeer

Note: When you provide the source directory name, if it terminates with "/" only the contents under the directory will be transferred.  But if you omit the ending "/" the same directory will be created at the destination and the contents will be copied over.

     Now backing up into a remote server

rsync -az -e ssh /home/safeer safeer@backup.safeer.in:/home/safeer/Backup_02_2011

"-e ssh" specifies the shell to be used for remote transport.

     Now let us backup remote content locally.

rsync -az -e ssh safeer@web.safeer.in:/var/www/html /media/USB_HD_01/Web_02_2011

     The syntax for a remote location is always USER@REMOTE_SERVER:REMOTE_DIRECTORY.  If USER is omitted the local username will be used, if REMOTE_DIRECTORY is omitted users home directory will be used.

     We will check a few useful options

To monitor the transfer:

-v : Verbose - provides a summary of what is happening
--progress : Displays the progress of data transfer.
--stats : Displays statistics abut the file transfer

To select and deselect files/folders:

--include PATTERN : Include files that match the given pattern
--exclude PATTERN : Exclude files that match the given pattern
--include-from FILE and --exclude-from FILE - to include and exclude files based on patterns in the file
--files-from FILE : Transfer only files listed in FILE
--filter RULE : Filter files based on RULE

     All these options are allowed multiple times.

Deletion/Update

-u : Do not transfter/overwrite if destination file is newer than source.
--delete : Delete file that exist in destination but not in source
--delete-before/--delete-during/--delete-delay/--delete-after/--delete-excluded are some options worth checking

For more detailed options and explanations, checkout the rsync man page.

Test your IMAP server with Telnet

     Internet Message Access Protocol version 4 (IMAP 4 ) is an email retrieval protocol which is a lot more robust than the Post Office Protocol version 3 ( POP3).   It provides features like name spaces,folders,sub folders,flags etc.. A discussion of all the features is beyond the scope of this article.  Here we will see how we can test the basic functionalities of an IMAP server using telnet.

     First, establish a connection to the IMAP server using telnet on port 143.

safeer@penguinpower:~$ telnet imap.safeer.in 143
Trying 10.25.0.3...
Connected to imap.safeer.in
Escape character is '^]'.
* OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE ACL ACL2=UNION] Courier-IMAP ready. Copyright 1998-2011 Double Precision, Inc.  See COPYING for distribution information.

     The last line of the output is a greeting from the IMAP server.  Further conversation with the IMAP server will be a series of requests and responses.   Requests are also known as commands and should begin with an arbitrary tag (an alpha numeric string ) followed by a command and its optional arguments, all separated by spaces.  The response, if positive will be the request tag name followed by OK and further by a server message.  This is for a single line response.  If the response is multi line, all except the last line with begin with a "*" and the last line will be request tag followed by OK and then server message.  A negative response will begin with request tag followed by a "NO" and a server message.

     To test the IMAP server at a minimum we should login,select INBOX, retrieve a message and logout.  Let us start by providing a tag followed by the command "login" followed by the IMAP username and password, all separated by spaces.

A01 LOGIN safeer fake_Password
A01 OK LOGIN Ok.

     With IMAP, you can have multiple mailboxes.  "INBOX" is the default and only one mailbox if you have not configured additional mailboxes.  Let us see the mailboxes and its sub folders using list command.  It takes two arguments, first being the folder to list and the second being a search parameter.  Since we are looking for mail boxes, we should be searching the root of the IMAP indicated by a blank "".  The search parameter is a wildcard ( "*" ) here since we want to see all folders under the root.

A02 list "" "*"
* LIST (\Marked \HasNoChildren) "." "INBOX"
1 OK LIST completed

     This is a multi line response from the server and it says there is only one mailbox "INBOX" with no sub folders.  Now let us see the  status of the "INBOX" and find out the number of messages in it.

A03 STATUS INBOX messages
* STATUS "INBOX" (MESSAGES 1)
A03 OK STATUS Completed.

     There is only one message in the INBOX. In order to view messages in the mailbox INBOX ( or any IMAP folder for that matter ) we should first select the folder.

A04 SELECT INBOX
* FLAGS (\Draft \Answered \Flagged \Deleted \Seen \Recent)
* OK [PERMANENTFLAGS (\* \Draft \Answered \Flagged \Deleted \Seen)] Limited
* 1 EXISTS
* 1 RECENT
* OK [UIDVALIDITY 1358664910] Ok
* OK [MYRIGHTS "acdilrsw"] ACL
A04 OK [READ-WRITE] Ok

     In the response, 1 EXISTS means there is only one message in INBOX and 1 RECENT means that 1 message is unread.  We can list/read the emails by using the fetch command.    Let us read the message ( since there is only one message in the INBOX, the id of the message will be 1 and that is passed as an argument to fetch command).  Let us first view the email headers.

A05 FETCH 1 body[header]
* 1 FETCH (BODY[HEADER] {501}
Return-Path: <info@safeer.in>
X-Original-To: safeer@safeer.in
Delivered-To: safeer@smtp.safeer.in
Received: from penguinpower (penguinpower [128.1.0.5])
    by smtp.safeer.in (Postfix) with ESMTP id AD10226089C
    for <safeer@safeer.in>; Thu, 10 Feb 2011 00:42:29 +0530 (IST)
Subject: POP3 Check
Message-Id: <20110210191235.AD10226089C@smtp.safeer.in>
Date: Thu, 10 Feb 2011 00:42:29 +0530 (IST)
From: info@safeer.in

)
* 1 FETCH (FLAGS (\Seen \Recent))
A05 OK FETCH completed.

     You can see the message headers.  Now let us read the message text.

A06 FETCH 1 body[text]
* 1 FETCH (BODY[TEXT] {14}
Testing POP3
)
A06 OK FETCH completed.

     We are done retrieving the message.  Note that fetch command can operate on a single message id or a range or ids in the format START:END.

    Now logout of the session:

A07  LOGOUT
* BYE Courier-IMAP server shutting down
A07 OK LOGOUT completed
Connection closed by foreign host.

    To know more about IMAP checkout the IMAP RFC at ITEF