A quick primer to GNU Screen - The terminal multiplexer

  Screen is a terminal multiplexer that allows you to open up multiple virtual terminal sessions inside a single terminal.  This enables the user to run multiple programs from a single terminal independently and in parallel.  It also allows switching between them and scrolling through the history as if you were using real independent terminal. 

     Now this may not be an attraction in today's Linux desktop environment where you can open up multiple tabs of the terminal program you want to use ( like gnu-terminal, kde konsole etc).  But imagine the case where your local machine don’t have GUI installed or you are connected to a remote host's terminal via ssh connection.  In the case of local machines terminal, there is a limit to the number of terminals a system can have.  In the case of ssh connection, you can open multiple connections but it takes up network and to manage them you need to open up an equal number of local terminals.  Screen solves this limitation by allowing to spawn virtual shell within a single shell and easily navigate between them.

     This is once advantage of screen but the best part is yet to come.  Any program that you run from a shell is usually spawned as a child of the shell/terminal program.  Hence terminating the shell/terminal session will kill any child process running in it.  Screen has the ability to detach itself from the parent shell process and hook itself as a child of the init process.  This makes screen resilient to logout ( for local terminals ) and network disconnects ( for remote terminals ).  This is a great tool for a power user.

Now let us see how we can use screen.

     First install the screen program using your favourite package manager or download from the screen website.

     Simplest invocation of screen is by just typing the command "screen" on your terminal (local/remote).  You might be shown an optional welcome message, press "Enter" to get rid of the message and drop tot he terminal.  This will start the first virtual terminal session and will look like a regular terminal.  You can run any commands/programs you want from this terminal.  You can finish your work on the terminal and type exit or press Control+D as you do on a regular terminal to exit out of screen.  But this do not add any value to our terminal usage. 

     To make the most out of screen, one should know how to spawn additional virtual terminals, detach the screen session allowing the running program to continue independently and browse through the numerous virtual terminals you open.

     To do all these task you have to sent special commands otherwise known as "key bindings" to the screen application, and all of them start with an escape sequence, followed by one or more characters.  The default escape sequence is "Control+a" ( press control key followed by character a ).  By default screen will pass any input to the current virtual terminal, but sending an escape sequence will make screen aware that it is a special command.  Let us see some useful screen commands/key bindings.

   Control+a c ( Press "Control", then "a" followed by "c" ) - Open up a new virtual terminal ( hereafter called "VT")  in screen. 

     You can create any number of VTs in this fashion, each screen will have a number starting from zero, zero being the number of the VT opened when screen is started.

Control+a " - List all VTs. 

    An example screen session with 13 VTs ( 0-13) is given below.  There will be a selection bar on the VT you are currently on.  You can use up/down arrow keys to select a different terminal or press the terminal number to select it ( this work only for terminals from 0-9 ).  Pressing "Enter" will activate the selected VT.

Num Name

 0  bash
 1  bash
 2  bash
 3  bash
 4  bash
 5  bash
 6  bash
 7  bash
 8  bash
 9  bash
10  bash
11  bash
12  bash

Navigation commands

Control+a p - Move to previous VT ( "Control+a Backspace" can also be used )
Control+a n - Move to next VT ( "Control+a Space" can also be used )
Control+a 0-9 - Move to the numbered VT ( works only for terminals 0-9 )
Control+a Control+a - Toggle between current and previous window

Screen Command Prompt

Screen has a command prompt from which you can input commands for screen.

Control+a : - Activate the command prompt for the current VT

Providing "title TITLE_NAME" from the screen command prompt will rename the current VT (same as control+a A )

Typing help at the prompt will show you a full list of helpful commands.

Usability commands

Control+a C - Clear screen
Control+a Esc OR Control+a [ - Activate copy mode ( in copy mode you can scroll up the output , select and copy text).
Enter to copy, Control+a ] to paste
Control+a ? - Show help/key bindings
Control+a w - Show a list of virtual terminals ( displayed at the bottom of current VT, no option to select VTs )
Control+a M - Monitor for activity
control+a _ - Monitor for silence more than 30 seconds

Control+a A - Rename the current VT. 
    
     As you can see, each of these VTs have a number as well as a name.  While the number increases for each VT, the name by default is the name of the shell/command spawned.  While the numbers are convenient, you might lose track of which terminal is doing what.  Giving it a meaningful name would help you identify the VT.  When this command is given, you will be given a prompt with the default name already in it, erase it with backspace, type in your new name and press enter.

A sample screen session with three renamed VTs

Num Name

 0  Mutt Email Client
 1  Perl Scripting
 2  Screen 3


Detach / Attach

   Now that you know how to operate within a screen session, let us see how to detach/attach to en existing screen session.

Detach : You are running programs under multiple VTs inside a screen session.  Now you want to disconnect from this screen for some reason ( logging out, terminating remote ssh connection etc.. ) and still let the programs under the VTs to continue.  This is where you should use the detach option, which is activated by the key binding "Catrl+a d".  When the key combination is pressed, screen drops you off to the shell from where you initiated the screen session.  Now you can logout / disconnect and the programs will keep running.

This is the sample output you get when you detach from a screen session.
[detached from 9108.maverick]

Now to connect back, you should know the screen session's name ( do not confuse this with the VT names inside the screen session ).  To list all current screen sessions, issue the command "screen -ls"

safeer@maverick:~$ screen -ls
There are screens on:
        9108.pts-0.maverick        (Sunday 20 March 2011 03:49:35  IST)    (Detached)
1 Socket in /var/run/screen/S-safeer.

There is only one screen session open right now and the name is 9108.pts-0.maverick.  The format of the screen name is [PID of screen].[TTY Numebr].[hostname].  This is the same name you saw when you where detatching the screen session.

To re attach to the disconnected session, use "screen -r <SCREEN NAME>", ie screen -r 9108.pts-0.maverick.

When re attaching the PID part is optional and can be reattached this way : "screen -r pts-0.maverick"

With a couple of screen sessions, it is easier to track which screen is what, but not always.  So we can custom name the screen sessions, for that start the screen session with "screen -R <screen name>".  This name replaces the [TTY].[hostname] part, but the pid part will remain.  Since PID is optional when reattaching the screen, you can provide just this screen name while reattaching.  See an

safeer@maverick:/var/run/screen$ screen -ls
There are screens on:  
        21454.SAFEER_TEST       (Sunday 20 March 2011 04:03:52  IST)    (Attached)
        9108.pts-0.maverick       (Sunday 20 March 2011 03:49:35  IST)    (Detached)
2 Sockets in /var/run/screen/S-safeer.


     You can re-attach to an attached session active in one terminal from another terminal after detaching it from the current terminal.

Either do it in two steps as

screen -d <SCREEN NAME>
screen -r <SCREEN NAME>

or together

screen -dr <SCREEN NAME>

Multi user mode

     Multi user mode is one speciality of screen where the same or different users in a system can share the same screen session.  You can control the access using ACLs, which allows restrictions based on usernames, VTs, commands and read/write/execute modes.

     To enable multi user mode, we will have to use the screen command prompt invoked by "Control+a :" which will give you a prompt.  Enter "multiuser on" in this prompt and multi user access will be enabled for this screen session.  Once enabled, this screen session can be accessed by other users from different terminals using the command "screen -x <screen name>"

     A complete discussion of all multi user mode is out of the scope of this article, so I will just list the commands that can be used inside screen command prompt.

multiuser on - Enable multi user mode.
display - Show currently connected users
acladd - add screen access for one or more users
acldel - revoke screen access for one or more users
aclchg - Add/modify permissions ( for commands,VTS,read/write/exec ) to one or more users

     To know more about screen, checkout the man page and the official website of screen.

Block device encryption and management in Linux

     The standard for Linux block level encryption is LUKS ( Linux Unified Key Setup ).  DM-Crypt is the Linux kernel's device mapper module for transparently encrypting and decrypting block devices.  To make use of these features, install cryptsetup package.

safeer@lin01:~$yum install cryptsetup-luks

     First, format the LUKS encryption layer (on the partition).  This is for standardizing the partition header and the format of the bulk data.

safeer@lin01:~$sudo /sbin/cryptsetup luksFormat /dev/sdb1
WARNING!
========
This will overwrite data on /dev/sdb1 irrevocably.

Are you sure? (Type uppercase yes): yes
Enter LUKS passphrase:
Verify passphrase:
Command successful.

     The command will ask for a passphrase, make sure you remember the passphrase you provide, as this will be later used to mount the partition.  Optionally you can directly provide a filename containing the passphrase as argument of above command with "--key-file <file-name>".

Open LUKS encryption layer ( device map the encrypted disk )

safeer@lin01:~$sudo /sbin/cryptsetup luksOpen /dev/sdb1 secretDisk
Enter passphrase for /dev/sdb1:

    Provide the password you created during the first setup.  Once authenticated, /dev/sdb1 will be devise mapped to the name you provided in the last command ( secretDisk ).  To verify, run:

safeer@lin01:~$ls -l /dev/mapper/secretDisk
lrwxrwxrwx 1 root root 7 Apr 14 11:10 /dev/mapper/secretDisk -> ../dm-0

Now format the device mapper with appropriate files system, here am chosing ext3.

safeer@lin01:~$sudo /sbin/mkfs.ext3 /dev/mapper/secretDisk

I want to mount this device to /home/safeer/secretDisk.  To mount it permanently, add following line to /etc/fstab

/dev/mapper/secretDisk    /home/safeer/secretDisk        ext3 defaults 0 0

safeer@lin01:~$sudo mount -a

df -h /home/safeer/secretDisk

     During boot up, the devise mapping of encrypted disk should happen prior to mounting, this is achieved by registering the device in the file /etc/crypttab

     For each crypt device, there should be one line in this file.  Aminimum of two fileds is necessary here, first is the mapper name without "/dev/mapper" prefix and the second is the actual encrypted block device.  The third field is passphrase file and if left empy or the value "none" is given, user will be prompted for the passphrase prior to mounting.  If this has to be automated, create a file with the passphrase ( non newline character at the end) and provide the file name as the third field.  The fourth field is options, which can be used to speficy encryption algorithms etc... This can  be left blank in most cases.

safeer@lin01:~$grep secret /etc/crypttab
secretDisk    /home/safeer/secretDisk none

To ensure things are working properly, reboot the host once.

A couple of things to know if you want more advansed setup.

     LUKS allows having 8 passphrases for a partition, this means 8 users can mount the filesystem with passphrases of their own choice and eliminates the need to share a common passphrase.  Check out the cryptsetup man page for options  luksAddKey,luksRemoveKey and luksDump.

Data backup with rsync

     Rsync is a file synchronization and transfer utility.  It can be used to transfer files between two local or remote locations.  It is fast, supports compression and encryption ( through ssh ).  Its usage is simple and can be used as a backup utility.

Syntax:

rsync <options> <source> <destination>

     Both the source and destination can be local directories, or the source can be a remote location ( pull ) or the destination can be remote ( push ).  For remote file transfer rsync can make use of either ssh/rsh shell as a transport or use rysnc as a daemon.  Here we will look at the local copy and ssh based remote copy.

     You can run rsync without any options, but using the following options will be good when you are backing up data.

-a : Archive - preserves symbolic links,permissions,timestamp,user/group
-z : Compression - compress data while transferring - good for remote date transfer

     If you are backing up a directory tree, then you should be using "-r" option to recursively go through all sub directories and files.  But if you are using "-a" option this is not required as -a does recursive archiving.

     Let us see how we can do a local file transfer ( This is the way I backup my home directory to external usb hard disk )

rsync -az /home/safeer  /media/USB_HD_01/Backup_02_2011

     After the first run, I periodically run the same command  to keep the backup in sync with my home directory.
In the subsequent runs, rsync only copies the files that has changed in my home directory since the last run.  This saves a lot of time.

     Any time you want to do a restore (say after an OS reinstall ), all you need to do is reverse the source and destination and run the rsync command.

rsync -az /media/USB_HD_01/Backup_02_2011 /home/safeer

Note: When you provide the source directory name, if it terminates with "/" only the contents under the directory will be transferred.  But if you omit the ending "/" the same directory will be created at the destination and the contents will be copied over.

     Now backing up into a remote server

rsync -az -e ssh /home/safeer safeer@backup.safeer.in:/home/safeer/Backup_02_2011

"-e ssh" specifies the shell to be used for remote transport.

     Now let us backup remote content locally.

rsync -az -e ssh safeer@web.safeer.in:/var/www/html /media/USB_HD_01/Web_02_2011

     The syntax for a remote location is always USER@REMOTE_SERVER:REMOTE_DIRECTORY.  If USER is omitted the local username will be used, if REMOTE_DIRECTORY is omitted users home directory will be used.

     We will check a few useful options

To monitor the transfer:

-v : Verbose - provides a summary of what is happening
--progress : Displays the progress of data transfer.
--stats : Displays statistics abut the file transfer

To select and deselect files/folders:

--include PATTERN : Include files that match the given pattern
--exclude PATTERN : Exclude files that match the given pattern
--include-from FILE and --exclude-from FILE - to include and exclude files based on patterns in the file
--files-from FILE : Transfer only files listed in FILE
--filter RULE : Filter files based on RULE

     All these options are allowed multiple times.

Deletion/Update

-u : Do not transfter/overwrite if destination file is newer than source.
--delete : Delete file that exist in destination but not in source
--delete-before/--delete-during/--delete-delay/--delete-after/--delete-excluded are some options worth checking

For more detailed options and explanations, checkout the rsync man page.

Test your IMAP server with Telnet

     Internet Message Access Protocol version 4 (IMAP 4 ) is an email retrieval protocol which is a lot more robust than the Post Office Protocol version 3 ( POP3).   It provides features like name spaces,folders,sub folders,flags etc.. A discussion of all the features is beyond the scope of this article.  Here we will see how we can test the basic functionalities of an IMAP server using telnet.

     First, establish a connection to the IMAP server using telnet on port 143.

safeer@penguinpower:~$ telnet imap.safeer.in 143
Trying 10.25.0.3...
Connected to imap.safeer.in
Escape character is '^]'.
* OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE ACL ACL2=UNION] Courier-IMAP ready. Copyright 1998-2011 Double Precision, Inc.  See COPYING for distribution information.

     The last line of the output is a greeting from the IMAP server.  Further conversation with the IMAP server will be a series of requests and responses.   Requests are also known as commands and should begin with an arbitrary tag (an alpha numeric string ) followed by a command and its optional arguments, all separated by spaces.  The response, if positive will be the request tag name followed by OK and further by a server message.  This is for a single line response.  If the response is multi line, all except the last line with begin with a "*" and the last line will be request tag followed by OK and then server message.  A negative response will begin with request tag followed by a "NO" and a server message.

     To test the IMAP server at a minimum we should login,select INBOX, retrieve a message and logout.  Let us start by providing a tag followed by the command "login" followed by the IMAP username and password, all separated by spaces.

A01 LOGIN safeer fake_Password
A01 OK LOGIN Ok.

     With IMAP, you can have multiple mailboxes.  "INBOX" is the default and only one mailbox if you have not configured additional mailboxes.  Let us see the mailboxes and its sub folders using list command.  It takes two arguments, first being the folder to list and the second being a search parameter.  Since we are looking for mail boxes, we should be searching the root of the IMAP indicated by a blank "".  The search parameter is a wildcard ( "*" ) here since we want to see all folders under the root.

A02 list "" "*"
* LIST (\Marked \HasNoChildren) "." "INBOX"
1 OK LIST completed

     This is a multi line response from the server and it says there is only one mailbox "INBOX" with no sub folders.  Now let us see the  status of the "INBOX" and find out the number of messages in it.

A03 STATUS INBOX messages
* STATUS "INBOX" (MESSAGES 1)
A03 OK STATUS Completed.

     There is only one message in the INBOX. In order to view messages in the mailbox INBOX ( or any IMAP folder for that matter ) we should first select the folder.

A04 SELECT INBOX
* FLAGS (\Draft \Answered \Flagged \Deleted \Seen \Recent)
* OK [PERMANENTFLAGS (\* \Draft \Answered \Flagged \Deleted \Seen)] Limited
* 1 EXISTS
* 1 RECENT
* OK [UIDVALIDITY 1358664910] Ok
* OK [MYRIGHTS "acdilrsw"] ACL
A04 OK [READ-WRITE] Ok

     In the response, 1 EXISTS means there is only one message in INBOX and 1 RECENT means that 1 message is unread.  We can list/read the emails by using the fetch command.    Let us read the message ( since there is only one message in the INBOX, the id of the message will be 1 and that is passed as an argument to fetch command).  Let us first view the email headers.

A05 FETCH 1 body[header]
* 1 FETCH (BODY[HEADER] {501}
Return-Path: <info@safeer.in>
X-Original-To: safeer@safeer.in
Delivered-To: safeer@smtp.safeer.in
Received: from penguinpower (penguinpower [128.1.0.5])
    by smtp.safeer.in (Postfix) with ESMTP id AD10226089C
    for <safeer@safeer.in>; Thu, 10 Feb 2011 00:42:29 +0530 (IST)
Subject: POP3 Check
Message-Id: <20110210191235.AD10226089C@smtp.safeer.in>
Date: Thu, 10 Feb 2011 00:42:29 +0530 (IST)
From: info@safeer.in

)
* 1 FETCH (FLAGS (\Seen \Recent))
A05 OK FETCH completed.

     You can see the message headers.  Now let us read the message text.

A06 FETCH 1 body[text]
* 1 FETCH (BODY[TEXT] {14}
Testing POP3
)
A06 OK FETCH completed.

     We are done retrieving the message.  Note that fetch command can operate on a single message id or a range or ids in the format START:END.

    Now logout of the session:

A07  LOGOUT
* BYE Courier-IMAP server shutting down
A07 OK LOGOUT completed
Connection closed by foreign host.

    To know more about IMAP checkout the IMAP RFC at ITEF

Test your POP3 server with Telnet

     Post Office Porotocl version 3 ( POP3 )  is the protocol used to retrieve emails from a mail server.  We generally use email clients like thunderbird, outlook etc to connect to the server and read the emails.  But when you want to test your pop3 server it is useful to connect to the server through a direct tcp channel and examine its basic functionalities.  The communication with the server is through a series of requests and responses.  The requests are also know as pop3 commands and have specified syntaxes.

     To start with we have to establish a tcp session with the pop3 server.  The is done using the telnet utility to connect to the pop3 server on the pop3 port ( the default is 110, but this can be changed in the pop3 server configuration).

safeer@penguinpower:~$ telnet pop.safeer.in 110
Trying 10.25.0.2...
Connected to pop.safeer.in.
Escape character is '^]'.
+OK Dovecot ready.

     The first three lines of the output is common for any telnet session, it just says the connection is successfully established. 

     The fourth line is the response from the pop3 server.  The pop3 server has two standard responses, positive and negative.  They start with a positive or negative response code  +OK or -ERR respectively followed by a server message.  Here the response is positive as you can see, and the message says "Dovecot ready" - dovecot being the pop3 server software running on pop.safeer.in

     Once you establish the connection with the server, you have to identify yourself by providing a username and password in the following format.

USER safeer
+OK
PASS fake_Password
+OK Logged in.

     This section is self explanatory.  Now let us inspect a summary statistics of my mailbox.

STAT
+OK 3 2086

     The response code is OK, the second column is the number of messages in the mailbox and the third is the total size of the messages

     Let us list all the three messages

LIST
+OK 3 messages:
1 501
2 1035
3 550
.

     First line of the response says, the listing is successful ( +OK ) and the mailbox has 3 messages.  Subsequent lines list each messages, the first column being the message id and the second being the message size.

     Now let us view a mail, to do so issue the command RETR followed by message id.

RETR 1
+OK 501 octets
Return-Path: <info@safeer.in>
X-Original-To: safeer
Delivered-To: safeer@safeer.in
Received: from penguinpower (penguinpower [128.0.0.5])
    by smtp.safeer.in (Postfix) with SMTP id C53EFG509TH;
    Mon, 24 Jan 2011 10:49:06 +0530 (IST)
subject: POP3 server setup
Message-Id: <20110124051909.C53EFG509TH@smtp.safeer.in>
Date: Mon, 24 Jan 2011 10:49:06 +0530 (IST)
From: info@safeer.in
Mail server setup successfully
.

     So the server responded +OK followed by size of the message, and the subsequent lines prints out the email with all the headers included.

     Let us delete the message now, issue the command DELE followed by the message id.

DELE 1
+OK Marked to be deleted.
LIST
+OK 2 messages:
1 1035
2 550

     Be aware that issuing the DELE command just marks the message for deletion.  The actual deletion of message happens when the user disconnects from the current pop3 session.  So while you are in the session, it is still possible to recover your message by issuing RSET command.

RSET
+OK
LIST
+OK 3 messages:
1 501
2 1035
3 550

     We are done, now exit the session  by issuing the QUIT command.

QUIT
+OK Logging out.

     If you had messages marked for deletion, the server response would also say "messages deleted"

     To learn more about POP3, look at the POP3 RFC at IETF