Configuring Dovecot as a POP3 or IMAP server

     Dovecot can be used to retrieve mail using pop3 or imap protocols.  The minimal configuration that will get you started is pretty simple and straight forward.

First install the package, for RHEL based distros, it is dovecot.

yum install dovecot

The main configuration file for dovecot is /etc/dovecot.conf ( location changes based on you installation, file name remains as dovecot.conf )

## Dovecot supports pop3,pop3s,imap and imaps protocols. 
## Enable whichever you require.
protocols = imap pop3

## If you are using conventional unix style mboxes under
## /var/spool/mail/<username> for storing mail
mail_location = mbox:~/mail:INBOX=/var/spool/mail/%u

That is it, you can start using dovecot sevice.  Edit and save the file and restart service.

sudo /usr/sbin/service dovecot restart

To test if dovecot is working, use instructions from following tutorial

Testing your POP3 server with telnet

Managing ISO files in Linux

ISO (International organizations for standardization) format is the archive file format for optical discs. The name is derived from the file system format for optical discs, known as ISO 9660 File System. This format is also known as CDFS (Compact Disc File System). ISO files are used to either create the image of an optical disc, or to write data to it.

Here are a few tricks to create and manage ISO files.

Creating an ISO image out of a CD ROM mounted in your machine.

We can use the dd (Disc Dump) command to take the copy of an optical disc. For this first you have to insert the disc in the drive, then find out the device name of your CD-ROM.

Once the CD is mounted, you can find your CD drive in this way:

safeer@penguinpower:~$ mount|grep -i iso
/dev/sr0 on /media/UBUNTU_10 type iso9660(ro,nosuid,nodev,uhelper=udisks,uid=1000,gid=1000,iocharset=utf8,mode=0400,dmode=0500)

Here my CD device is /dev/sr0. In most of the systems where you have a single CD drive, this will be symlinked to /dev/cdrom, so you can even use that in place of /dev/sr0 ( or whatever your device is).

safeer@penguinpower:~$ ls -l /dev/cdrom
lrwxrwxrwx 1 root root 3 2010-11-06 10:23 /dev/cdrom -> sr0

Now that we know our CD device name, let us create the ISO image of the mounted disc.

safeer@penguinpower:~$ dd if=/dev/sr0 of=/home/safeer/ISO/ubuntu10.iso
1365028+0 records in
1365028+0 records out
698894336 bytes (699 MB) copied, 267.224 s, 2.6 MB/s

Where "/dev/sr0" is my CD device and "/home/safeer/ISO/ubuntu10.iso" is the name of the iso image that will be created from the CD.

Mounting an ISO file as if you are mounting a CD-ROM.

ISO file can be mounted in a UNIX file system just like inserting a CD to your CD drive. We make use of the loop device system for this purpose. Loop device is a pseudo file system available in Linux/UNIX that makes a file accessible as a block device.

First, create a mount point

safeer@penguinpower:~$ sudo mkdir /mnt/iso

Then mount the ISO file as loop device

safeer@penguinpower:~$ sudo mount -o loop -t iso9660 /home/safeer/ISO/ubuntu10.iso /mnt/iso/
safeer@penguinpower:~$ mount |grep iso
/dev/loop0 on /mnt/iso type iso9660 (rw)

Now you can browse the contents of iso /home/safeer/ISO/ubuntu10.iso via the directory /mnt/iso/.

To unmount the device:

safeer@penguinpower:~$ sudo umount /mnt/iso

Creating an ISO file from a directory.

You can use mkisofs command to create an iso. In my system mkisofs is a symlink to the program genisoimage. You can use either command for this purpose.

Here I will create an iso image of all softwares that I downloaded.

safeer@penguinpower:~$ mkisofs -o software.iso /home/safeer/Downloads/
I: -input-charset not specified, using utf-8 (detected in locale settings)
Using VPNC_000.RPM;1 for /vpnc-0.5.3-8.fc13.src.rpm (vpnc-0.5.3-6.fc11.x86_64.rpm)
Using VPNC_001.RPM;1 for /vpnc-0.5.3-6.fc11.x86_64.rpm (vpnc-0.5.3-8.fc13.x86_64.rpm)
1.03% done, estimate finish Sun Nov 7 18:28:59 2010
2.06% done, estimate finish Sun Nov 7 18:28:59 2010
................................
................................
98.68% done, estimate finish Sun Nov 7 18:29:33 2010
99.71% done, estimate finish Sun Nov 7 18:29:34 2010
Total translation table size: 0
Total rockridge attributes bytes: 0
Total directory bytes: 4914
Path table size(bytes): 36
Max brk space used 1b000
486421 extents written (950 MB)

Let us check the iso image by mounting it.

safeer@penguinpower:~$ sudo mount -t iso9660 -o loop software.iso /mnt
safeer@penguinpower:~$ cd /mnt/
safeer@penguinpower:/mnt$ ls
a0654873.tor google_c.deb skype_de.deb torrentd.tor vpnc_000.rpm vpnc_0_5.deb vpnc_con.rpm
adberdr9.deb opera_10.deb skype_ub.deb virtualb.deb vpnc_001.rpm vpnc_0_5.rpm vpnc_scr.rpm

The problem with using the default settings for mkisofs is that it truncates the filenames to be compatible with DOS file system. A few options that will help us create better image is shown below.

safeer@penguinpower:~$ mkisofs -r -R -J -l -o software.iso /home/safeer/Downloads/
I: -input-charset not specified, using utf-8 (detected in locale settings)
1.03% done, estimate finish Sun Nov 7 18:56:33 2010
....................................................
99.71% done, estimate finish Sun Nov 7 18:57:00 2010
Total translation table size: 0
Total rockridge attributes bytes: 5152
Total directory bytes: 11248
Path table size(bytes): 36
Max brk space used 1c000
486433 extents written (950 MB)

Checkout the man pages for more details on mkisofs

Configuring Postfix SMTP as your MTA

     Postfix is one of the most popuar MTAs around.  Postfix has a modular design that improve security,speed and east of configuration.  In this design, a master daemon controls individual light weight processes (modules) that performs specific tasks.  The master deamon binary is "/usr/sbin/postfix" and the daemon configuration file is /etc/postfix/mster.cf.  But this file rarely needs change unless you need to enable / disable some sub processes(modules).  The MTA configuration is in  /etc/postfix/main.cf.  There are other configuration files for specific purposes and all of them lies under the folder /etc/postfix.  Following is a minimal configuration of main.cf that will let you run a postfix mail server for your own domain.

Most of the configuration parameters in main.cf will have a commented example version, you can either uncomment and edit them, or add a new line.  I will just list out important configuration parameters with comments added.  The example configuration is for running an MTA for safeer.in domain.

## The configuration file is in "key = value" format.  The key defined previously in the file
## can be referred to as $key in the coming sections.

## The domain for which you are configuring MTA
mydomain = safeer.in

## FQDN of the mail server where postfix is being configured
myhostname = smtp.safeer.in

## The domain name that locally sent and received mail appears to be from and/or delivered to.
## When you are running mail server for a domain, set as follows, default if unset is to use hostname.
myorigin = $mydomain

## By default postfix listens only on loopback, make it listen an all interfaces.
inet_interfaces = all

## The SMTP server will accept mails to the email ids addressed to the following domains and/or hostnames.
## In our setup, this will include the mail server (smtp.safeer.in) and the mail domain safeer.in
mydestination = $myhostname, localhost.$mydomain, localhost, $mydomain

## Define trusted networks from which clients can connect to the smtp server without any restrictions.
mynetworks = 192.168.4.0/24, 10.10.5.0/24

## Allow smtp clients only from $mytnetworks and reject un authorized clients
smtpd_recipient_restrictions = permit_mynetworks, reject_unauth_destination
    
     This will get you started off with a minimal SMTP server.  Make these changes and restart your postfix server.

sudo /usr/sbin/service postfix

Ensure postfix has started listening by running

sudo netstat -atnp|grep :25|grep LISTEN

To test your server is working from command line, visit this link

Test your SMTP server with telnet

To know more about postfix configuration parameters check the official website of postfix or look at the man page by running "man 5 postconf"

Perl Snippets Part 2 - CSV to XLS

This program will convert a CSV ( Comma Separated Value ) file to an XLS ( Windows Excel ) spreadsheet. The script should be provided with the CSV file name as the first command line argument. It takes an optional second argument as the output XLS file name. If no output filename is given, the output will be saved to out.xls in the current directory.

This script requires Spreadsheet::WriteExcel perl module. On my Ubuntu box, this was installed with the command:
safeer@penguinpower:~$sudo apt-get install libspreadsheet-writeexcel-perl

#!/usr/bin/perl -w

use Spreadsheet::WriteExcel;

if (!defined $ARGV[0]) { die "You must provide a csv file"; }
else {$CSV_FILE = $ARGV[0];}

if ( defined $ARGV[1] ) { $XLS_FILE = $ARGV[1]; }
else { $XLS_FILE = "out.xls"; }

unless ( open ( CSV,$CSV_FILE))
{ die "Unable to open ".$CSV_FILE;}

my $workbook = Spreadsheet::WriteExcel->new($XLS_FILE);
if ( !defined $workbook ) {
             die "Unable to create excel file\n";
         }
$worksheet = $workbook->add_worksheet();
$col = $row = 0;

while (<CSV>)
{
chomp;
my @line = split(",",$_);
for $cell (@line){
      $worksheet->write($row, $col,$cell);
      $col++;
      }
$row++; $col = 0;
}
$workbook->close();
close (CSV);

Bash Recipes Part 1 - Changing extensions for multiple file names

I have came across many situations where a bunch of files with one filename extension has to be renamed to another extension. Here is the way to do it in bash.

We will rename a number of post script files to PDF files ( .ps to .pdf )

safeer@penguinpower:~/Documents$ ls
book0.ps book1.ps book2.ps book3.ps book4.ps
safeer@penguinpower:~/Documents$ for file in $(ls *.ps);
do
mv $file ${file/%.ps/.pdf};
done
safeer@penguinpower:~/Documents$ ls
book0.pdf book1.pdf book2.pdf book3.pdf book4.pdf

Perl Snippets Part 1 - XLS to CSV

For parsing Microsoft excel files (.xls) using Perl, we can make use of the module Spreadsheet::ParseExcel.
On my Ubuntu system this module is installed by:

safeer@penguinpower:~$sudo apt-get install libspreadsheet-parseexcel-perl

Following program will take an XLS file as input and print the csv format to standard out.

#!/usr/bin/perl -w

use strict;
use Spreadsheet::ParseExcel;

my $parser = Spreadsheet::ParseExcel->new();
if ( !defined $ARGV[0] ) { die "Please provide an excel file"; }
my $workbook = $parser->parse($ARGV[0]);
if ( !defined $workbook ) { die $parser->error(), ".\n"; }

for my $worksheet ( $workbook->worksheets() ) {

my ( $row_min, $row_max ) = $worksheet->row_range();
my ( $col_min, $col_max ) = $worksheet->col_range();

for my $row ( $row_min .. $row_max ) {
for my $col ( $col_min .. $col_max ) {

my $cell = $worksheet->get_cell( $row, $col );
if ( defined $cell ) { print $cell->value(); }
if ( $col < $col_max ) { print ","; }        
}   
print "\n";
}
}   

Configuring NTP service

     The network time protocol ( NTP ) is used to synchronize a computer system's time with an accurate time source. Interconnected and communicating systems should have their times in sync with each other to preserve the order of communication / sotring of information ( and and many more applications ).

     The ultimate source of accurate time is called a reference clock and will usually be an atomic/nuclear clocks ( usually using cesium as the element ).  Another variety of reference clocks will be gps receivers receiving time from satellites ( which in turn uses an atomic clock ). Other computer systems sync their time from reference clocks directly or from servers that sync from reference clock directly.

    NTP supports a hierarchy of time sources, the accuracy of each level of this hierarchy is referred to by the term startum.  A startum number is given to each source based on its reliability, smallest startum being most accurate.  Thus reference clocks are startum 0, the servers that sync time directly from them are startum 1, and servers that sync from these servers will be startum 2 and so on.  Basiaclly if the upstream NTP server is startum X, the server syncing from that server is startum X+1 upto a maximum value of 15.  NTP protocol/service has inherent mechanisms to determine the accuracy of upstream NTP server as well as its own by calculating different parameters like offset,resolution,jitter etc..

NTP is a UDP base protocol and uses port 123. Let us look at a basic NTP configuration

    First install the  NTP software, on ubuntu it is openntpd with main configuration file /etc/openntpd/ntpd.conf and optional include files kept under /etc/openntpd.  On a redhat system it will be ntp ( with configuration file at /etc/ntp.conf and include directory /etc/ntp.d/).

safeer@ntpc:~$sudo yum install ntp
safeer@ntpc:~$sudo /sbin/service ntpd status
 * ntpd is stopped

     First you have to determine from which NTP server you want to sync time.  This can be another server in your network, or one publicaly available.  You can even setup a standalone ntp server which syncs from its own hardware clock and provide time service to other servers in your network ( though not very much recommended )

     Now there are two ways to sync time from a server.  You can use the ntpdate binary, which will sync with the server time immediately, but this can have adverse effect on the server of the time difference between ntpserver and client and bigger.  The other option is to configure ntp service, which will allow the server to gracefully sync time.

How ntpdate works:

Check if the server is up and available:

safeer@ntpc:~$sudo ntpdate -q ntp1.safeer.in

The output will list one ore more servers behind the ntp dns name and its time details.  This indicates the sever is good, so go ahead and sync with it.

safeer@ntpc:~$ sudo ntpdate ntp1.safeer.in

You can provide multiple space separated NTP server to ntpdate.  If the command shows no error, the time will be set by now.  Check your system with date command.

Now let us see how the ntp server can be configured:

First add one or more servers:

server 1.1.1.1
server 2.2.2.2
server ntp1.safeer.in

NTPD can detect the time drift between system clock and the reference clock ( or server ) over a period of time and need to store this somewhere to adjust the system time.  This drift value has to be written to some file.  The user running NTP daemon should have write access to the file.

driftfile /var/lib/ntp/drift

     This much was about how you receive the time from upstream servers.  Now let us see what it takes to sync the server's own system clock or downstream servers. 
     First case, let us assume we just need to sync the time on our server and not provide time to any downstream servers ( ntp client mode )

    The configuration parameter to use here is "restrict" in the following syntax: restrict < host ip/hostname/network id > [mast < mask >] [flags .. ..]
     The mask is optional ( need only when u are defining access for networks and not single servers), if the flags are omitted the server will be having unrestricted access to ntp service.  Also you can use the entry "default" in place of host/network for setting default access policy.  If access is not explicitly defined for a host/network the default policy will be used.

    Since we are planning to sync only the local server we have to restrict everything except localhost from accessing ntp service. 

## This will deny ntp service to any host that done have an explicit "restrict" entry
restrict default ignore
## Allow access to localhost through loopback interface ( by omitting flags section )
restrict 127.0.0.1

If we are configuring our server to server other hosts, say the local network - 192.168.1.0/255.255.255.0 this is how you do it:

restric 192.168.1.0 mask 255.255.255.0 nomodify notrap nopeer
     The flag nomodify prevent clients from modifying servers ntp configuration, notrap prevents from sending log collection trap messages, nopeer prevents peering/associating the clients with the server to be "co-servers"

     If the upstream server becomes unavailable, the host will not be able to sync time, so as a fall back we configure the local clock as a time source, in ntp language the ip "127.127.1.0" indicates local clock

server 127.127.1.0 # local clock
fudge 127.127.1.0 stratum 10

If you want to log the ntp activity add the entry

logfile /var/log/ntp.log

Finally, let us consilidate everything in one place:

server 1.1.1.1
server 2.2.2.2
server ntp1.safeer.in

restrict default ignore
restrict 127.0.0.1

restric 192.168.1.0 mask 255.255.255.0 nomodify notrap nopeer

server 127.127.1.0 # local clock
fudge 127.127.1.0 stratum 10

driftfile /var/lib/ntp/drift
logfile /var/log/ntp.log


Before starting ntp service, run ntpdate with one of the servers listed as an initial sync, then start ntp service


safeer@ntpc:~$sudo ntpdate ntp1.safeer.in
safeer@ntpc:~$sudo /sbin/service ntpd start

For more on NTP, check official website and RFC page.

Converting time between timezones in Linux

People working with systems that are located in different timezones are always faced with the problem of converting time between timezones. On GNU Linux systems ( and most other POSIX OSs) you can manipulate the TZ environment variable to achieve this.

Generally on Linux systems, the timezone information is kept in the file /etc/localtime. This file is usually a softlink to an appropriate zone file under the location "/usr/share/zoneinfo/".

safeer@penguinpower:~$ ls -l /usr/share/zoneinfo/
drwxr-xr-x 2 root root 4096 2010-02-21 01:46 Africa
drwxr-xr-x 6 root root 12288 2010-02-21 01:46 America
drwxr-xr-x 2 root root 4096 2010-02-21 01:46 Antarctica
.............................
-rw-r--r-- 1 root root 2294 2010-01-03 01:38 EST5EDT
drwxr-xr-x 2 root root 4096 2010-02-21 01:46 Etc
drwxr-xr-x 2 root root 4096 2010-02-21 01:46 Europe
.............................
-rw-r--r-- 1 root root 118 2010-01-03 01:38 UTC
-rw-r--r-- 1 root root 1873 2010-01-03 01:38 WET
-rw-r--r-- 1 root root 2194 2010-01-03 01:38 W-SU
-rw-r--r-- 1 root root 18896 2009-12-12 20:27 zone.tab
-rw-r--r-- 1 root root 118 2010-01-03 01:38 Zulu

Some of these are directories which have timezone files under them. For eg, the directory Asia contains zone files for all the countries ( or their capital/important cities) in Asia, including India,China,Pakistan etc....

safeer@penguinpower:~$ ls -l /usr/share/zoneinfo/Asia/
-rw-r--r-- 1 root root 157 2010-01-03 01:38 Aden
-rw-r--r-- 1 root root 922 2010-01-03 01:38 Almaty
-rw-r--r-- 1 root root 8504 2010-01-03 01:38 Amman
............................
-rw-r--r-- 2 root root 657 2010-01-03 01:38 Ashgabat
-rw-r--r-- 2 root root 657 2010-01-03 01:38 Ashkhabad
-rw-r--r-- 1 root root 962 2010-01-03 01:38 Baghdad
...............
-rw-r--r-- 1 root root 1929 2010-01-03 01:38 Vladivostok
-rw-r--r-- 1 root root 1914 2010-01-03 01:38 Yakutsk
-rw-r--r-- 1 root root 2000 2010-01-03 01:38 Yekaterinburg
-rw-r--r-- 1 root root 2012 2010-01-03 01:38 Yerevan

We are going to use this directory structure for setting the TZ variable. The value of TZ variable will be the relative path of the desired timezone file with reference to "/usr/share/zoneinfo". As an example, I am going to change my system time to Singapore time.

safeer@penguinpower:~$ date
Tue Apr 6 00:44:54 IST 2010
safeer@penguinpower:~$ export TZ=Asia/Singapore
safeer@penguinpower:~$ date
Tue Apr 6 03:15:37 SGT 2010

What if you don't want to set the TZ variable but just need to see the current Singapore time?

safeer@penguinpower:~$ date
Tue Apr 6 00:48:07 IST 2010
safeer@penguinpower:~$ TZ=Asia/Singapore date
Tue Apr 6 03:18:12 SGT 2010
safeer@penguinpower:~$ date
Tue Apr 6 00:48:16 IST 2010

Here the value "Asia/Singapore" will be applicable only to the command 'date' during its execution.

We are going to use this, along with a feature of 'date' command for converting between different timezones.

Now our machine's timezone is India Standard Time (IST - Asia/Calcutta ), we need to see what will be the time in India when the Singapore time is 2:24 PM on a particular day.

safeer@penguinpower:~$ date -d 'TZ="Asia/Singapore" 2010-03-10 14:24:00'
Wed Mar 10 11:54:00 IST 2010

This shows that SGT is two and half hours ahead of IST. The -d option allows specifying a TZ variable which helped us in getting this answer. Now suppose that you want to see what will be the time in Dubai when it is 2.30 PM in Singapore. But our local timezone is in IST and should not be changed.

The way to do this is, set the TZ variable to Dubai timezone just for the particular date command. Here is how we can do it.

safeer@penguinpower:~$ TZ=Asia/Dubai date -d 'TZ="Asia/Singapore" 2010-03-10 14:24:00'
Wed Mar 10 10:24:00 GST 2010

This shows us that, Dubai time ( GST - Gulf Standard Time) is 4 hours behind Singapore time (SGT).

So the general syntax for conversion can be summarized as below:

TZ=$TO_TIMEZONE date -d 'TZ="$FROM_TIMEZONE" YYYY-MM-DD HH:MM:SS'
Where $FROM_TIMEZONE is the timezone from which the conversion should be done and $TO_TIMEZONE is the timezone to which the date should be converted. Here:

TO_TIMEZONE=Asia/Calcutta, FROM_TIMEZONE=Asia/Singapore


If you regularly do date conversions you can write a wrapper script around this command.

To learn more about TZ variable, please visit TZ page on GNU website

Base64 Encoding and Decoding

     base64 is an encoding scheme used to transfer binary/text data across networks. Though initially used predominantly for transmitting data through email, base64 is used in a wide variety of of applications nowadays.

      The encoding is done by converting input data into printable ASCII characters. The input stream is divided into 6 bit segments and converted to a printable character from a set of 64 encoding characters based on the value of the 6 bits ( 6 bits can represent a value between 0 to 63   - 64 numbers ).   In practical dividing by 6 bits is done by taking 3 consecutive bytes (3 x 8 bits =  4 x 6 bits ) of the input stream and converting it to 4 encoding characters.  If the total number of bytes in the input stream is not a multiple of three, extra 0 value bytes are padded to the end of the stream to make it multiple of three.

     The decoding of the input stream is done on the receiving end by reversing the above method. 

     In Linux, the "base64" command is used to decode/encode base64.  Let us see an example.  We will convert the "ls" command binary to base64 and back and see if the resultant binary is still working as "ls".

safeer@penguinpower:~/TMP$ cp /bin/ls .
safeer@penguinpower:~/TMP$ /home/safeer/TMP/ls
ls

Convert the binary to base64

safeer@penguinpower:~/TMP$ cat /home/safeer/TMP/ls | base64 > ls.encoded

Compare both

safeer@penguinpower:~/TMP$ wc ls ls.encoded
   503   3271 104508 ls
  1834   1834 141178 ls.encoded
  2337   5105 245686 total


safeer@penguinpower:~/TMP$ head -1 ls.encoded
f0VMRgEBAQAAAAAAAAAAAAIAAwABAAAANL4ECDQAAADckwEAAAAAADQAIAAJACgAHAAbAAYAAAA0
safeer@penguinpower:~/TMP$ tail -1 ls.encoded
AAAA6JIBAPIAAAAAAAAAAAAAAAEAAAAAAAAA
safeer@penguinpower:~/TMP$ file ls.encoded
ls.encoded: ASCII text
safeer@penguinpower:~/TMP$ file ls
ls: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24, BuildID[sha1]=0x83531f308f1fa18221be53eaf399303400c14638, stripped

Now convert the encoded data back

safeer@penguinpower:~/TMP$ cat ls.encoded |base64 -d > ls.decoded
Compare with original "ls" binary

safeer@penguinpower:~/TMP$ wc ls ls.decoded
   503   3271 104508 ls
   503   3271 104508 ls.decoded
  1006   6542 209016 total
safeer@penguinpower:~/TMP$ file ls.decoded
ls.decoded: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24, BuildID[sha1]=0x83531f308f1fa18221be53eaf399303400c14638, stripped

safeer@penguinpower:~/TMP$ chmod a+x ls.decoded
safeer@penguinpower:~/TMP$ /home/safeer/TMP/ls.decoded
ls  ls.decoded    ls.encoded

Now that you know how the conversion works, let us see some other ways to decode/encode using base64.  We will use a simple string this time for simplicity.

Bash

Encoding

safeer@penguinpower:~/TMP$ printf "safeer"|base64
c2FmZWVy

Decoding

safeer@penguinpower:~/TMP$ printf "c2FmZWVy"|base64 -d
safeer

Perl

Encoding

safeer@penguinpower:~/TMP$ perl -MMIME::Base64 -e 'print encode_base64("safeer");'
c2FmZWVy

Decoding

safeer@penguinpower:~/TMP$ perl -MMIME::Base64 -e 'print decode_base64("c2FmZWVy");'
safeer

PHP

Encoding

safeer@penguinpower:~/TMP$ php -r "print base64_encode('safeer');"
c2FmZWVy

Decoding

safeer@penguinpower:~/TMP$ php -r "echo base64_decode('c2FmZWVy');"
safeer

Python

Encoding

safeer@penguinpower:~/TMP$ python -c "import base64;print base64.b64encode('safeer')"
c2FmZWVy

Decoding

safeer@penguinpower:~/TMP$ python -c "import base64;print base64.b64decode('c2FmZWVy')"
safeer

Introduction to FFMpeg

FFMpeg is an open-source utility for recording,streaming and converting digital video and audio. We will see the usage of a few functionalities to convert between multiple file formats.

In simplest form ffmpeg can be used as:

ffmpeg -i <input filename> <outputfilename>

Eg: Convert a flash (flv)video to avi format

safeer@penguinpower:~$ffmpeg -i SELinux4EveryOne.flv SELinux4EveryOne.avi

This will throw a lot of output and then start converting the video. Once the conversion is finished play the the file with a media player like vlc or mplayer.

This should play the new avi file nicely (provided mplayer is installed):

safeer@penguinpower:~$mplayer SELinux4EveryOne.avi

Now, what if we want to extract the audio track alone from the Video? This is the command.

safeer@penguinpower:~$ffmpeg -i SELinux4EveryOne.flv SELinux4EveryOne.mp3

You can also convert flv to wav or wav to mp3

safeer@penguinpower:~$ffmpeg -i SELinux4EveryOne.flv SELinux4EveryOne.wav

safeer@penguinpower:~$ffmpeg -i SELinux4EveryOne.wav SELinux4EveryOne.mp3

A wide variety of media formats are supported by ffmpeg. To get a full list, use "ffmpeg -formats"

A few useful options:

-f <format> : Usually ffmpeg can determine the output format by the extension name itself. But incase you want to force the output format, use this option.

Eg: ffmpeg -i SELinux4EveryOne.wav -f mp3 SELinux4EveryOne.mp3

-ab : Audio bitrate ( 64k is default, 128k would be a reasonable value )
-ar : Audio frequency - 44100 Kz is default
-ac : Audio channel, a value of 1 is mono, 2 is sterio
-an : Disable audio recording.
-b : Video bit rate
-r : Video frame rate
-s : Video frame size ( Screen resolution )

Any option that you give before any input/output file will apply to that file. Also remember that you can have multiple input files each specified with "-i" option and multiple output files ins some cases.

A couple of other interesting uses of ffmpeg:

Extracting images from a video:

ffmpeg -i myvideo.avi -r 1 -s svga -f image2 img-%03d.jpg

This will extract one image per second from the video for its total length. The frame size of the images will be 800*600 ( svga ). The output file name img-%03d.jpg will be automatically expanded and used in the following way: the part "%03d" will change to 3 digit numbers padded with zeros - img-001.jpg,img-002.jpg....img-031.jpg ...

Video capturing yout desktop can be done with :

ffmpeg -f x11grab -s xga -r 50 -i :0.0 /tmp/out.avi

This will capture "1024x768" sized portion from the upper left corner of the screen. The frame rate will be 50 frames per second.

There many such uses for ffmpeg. to learn more, checkout their documentation.

Downloading Youtube videos in Linux

Lots of people who watch Youtube videos like to download them for viewing it offline later. On a Linux box, the best tool you can find is a small command line utility called "youtube-dl". This is available as package in fedora/debian/ubuntu/suse repositories. Even if this tool is not in your repository, you can download it directly from the project website. This script requires python, so make sure you have it. Unzip the downloaded archive and copy it to system path (or run from there itself - if you wish).

To download a video, you need to get its Youtube URL which is the URL that you see in the address bar of your browser when you play the video. It will look like this: http://www.youtube.com/watch?v=RANDOM_STRING. The RANDOM_STRING will contain alpha numeric characters and dash.

Let us see the actual use of the tool.

safeer@penguinpower:~$youtube-dl http://www.youtube.com/watch?v=QTw6iexyrNU

It will throw some output about the URL and then start downloading, showing the current download speed and ETA. When finished, it will report "Video data saved to QTw6iexyrNU.flv". The file will be saved in the current directory. If you want to save to another location or change the name of the file, use option "-o".

The default format for the saved video will be "flv" but if you use the -b option ( for best qulity ) the format will be mp4. So change your filename accordingly when using -o and -b together.

There are many other useful options including the option to give username and password for videos that can be accessed only by logged in Yuotube users. Use "youtube-dl -h" to see all the options.

One thing that you should remember about Youtube URLs is that if the url contains one or more "&" symbols, you need to take only the part on the left hand side of the first "&". For eg, if you got this URL "http://www.youtube.com/watch?v=QTw6iexyrNU&feature=related" from your browser, then the actual URL is "http://www.youtube.com/watch?v=QTw6iexyrNU". The rest are arguments to the Youtube website which will not be used by youtube-dl.

This tool has a windows GUI which can be found here.

Now for the people who want to do things their way, following method may be interesting. Open your browser and play the video from Youtube and wait for it to finish. Once the video is played to the full length, open a terminal keeping the Youtube page open. Then run this command on terminal.

safeer@penguinpower:~$ for i in $(find /tmp -type f -user $(whoami) 2>/dev/null);do [[ $(file $i) =~ "Flash" ]] && echo $i;done

/tmp/FlashRXaVqD

This file is the video you just watched, if you want to cross check you can run this file in some media player like vlc.

safeer@penguinpower:~$ vlc /tmp/FlashRXaVqD&

And it will play the video. Now save the file with a name and flv extension.

safeer@penguinpower:~$mv /tmp/FlashRXaVqD ~/chess-play.flv

Not a big deal, but it will save your bandwidth, if you want to first play the video and then only download it.

Chrome is developing a browser extension for Youtube downloading, I haven't tested it yet. Have a look into it if you are interested.

Disclaimer: Downloading Youtube content is subject to copy right and content policies of the concerned parties, the author or this blog will not be responsible for such issues.

Typescript - Capturing your shell activity

We know about screen capturing and recording our screen activity. There are hundreds of applications for this purpose. But what about your shell activity? Of course you can use the same screen capturing tools if you are working on the GUI window. But what if you are on a remote shell? What if you want to reuse or copy the results/commands as a backup? Obviously the graphical tools will not be of much help. This is where typescripts come into play.

A typescript is a record of whatever is printed in your terminal. That includes stdin,stdout and stderr, which is basically the commands that you type and the output that these commands generate. The command which creates the typescript is "script". You can just type the command "script" and this will start recording the shell activity to a file by the name "typescript" which will be created in the directory from which the command "script" was run. Alternatively you can specify a separate file as an argument to it. When you run the script command it actually opens a new shell, and keeps recording the shell's activity until you exit that shell and return to the parent shell from where you ran the "script"command. Let us check the usage.

safeer@penguinpower:~$ script
Script started, file is typescript
safeer@penguinpower:~$ ls MyDocuments/
Linux Windows
safeer@penguinpower:~$ mkdir MyDocuments/Cisco
safeer@penguinpower:~$ ls MyDocuments/
Cisco Linux Windows
safeer@penguinpower:~$ rm MyDocuments/Cisco
rm: cannot remove `MyDocuments/Cisco': Is a directory
safeer@penguinpower:~$ exit
exit

Script done, file is typescript

Let us examine the contents of typescript.

safeer@penguinpower:~$ cat typescript
Script started on Sunday 03 January 2010 02:05:01 AM IST
safeer@penguinpower:~$ ls MyDocuments/
Linux Windows
safeer@penguinpower:~$ mkdir MyDocuments/Cisco
safeer@penguinpower:~$ ls MyDocuments/
Cisco Linux Windows
safeer@penguinpower:~$ rm MyDocuments/Cisco
rm: cannot remove `MyDocuments/Cisco': Is a directory
safeer@penguinpower:~$ exit
exit

Script done on Sunday 03 January 2010 02:05:55 AM IST

As you can see all that is printed into the terminal is recorded in the typescript. A few useful options to use with the script:

-a : Append to the end of the type script file - if it exists already. Default behavior is to overwrite the file.
-q: Quiet mode, removes the End timestamp from the typescript.
-t: This is an interesting option, it prints the output timing data to standard error. This data can be used to replay the typescript with real-time input output delays. We will see the usage later.

You can specify an alternate typescript file as the last argument.

safeer@penguinpower:~$ script myscript
Script started, file is myscript
safeer@penguinpower:~$ echo HI
HI
safeer@penguinpower:~$ exit
exit

Script done, file is myscript

safeer@penguinpower:~$ cat myscript
Script started on Sunday 03 January 2010 02:10:28 AM IST
safeer@penguinpower:~$ echo HI
HI
safeer@penguinpower:~$ exit
exit

Script done on Sunday 03 January 2010 02:10:44 AM IST

To gather the timing information, we car try using the -t option.

safeer@penguinpower:~$ script -t myscript 2>myscript.time
Script started, file is myscript
safeer@penguinpower:~$ ls MyDocuments/
Cisco/ Linux/ Windows/
safeer@penguinpower:~$ ls MyDocuments/Cisco/
safeer@penguinpower:~$ touch MyDocuments/Cisco/gns3
safeer@penguinpower:~$ ls MyDocuments/Cisco/
gns3
safeer@penguinpower:~$ exit
exit

Script done, file is myscript

Content of time file will look like this:

safeer@penguinpower:~$ head -2 myscript.time
0.500888 27
0.253943 23

The first field indicates how much time elapsed since the previous output. The second field indicates how many characters were output this time. With this information and the typescript file we can use the command "scriptreplay" to replay our terminal activity with the same timing with which you typed on the terminal and the output was displayed. It will give you a feeling of watching a video of your terminal activity.

The usage is like this:

First get the type script to one file and the timing information obtained with "-t" option ( printed to stderr) to another file.

safeer@penguinpower:~$ script -t myscript 2>myscript.time

Now replay the typescript with the format "scriptreplay [timing file] [typescript file] ".
If no typescript filename is given, the name "typescript" is assumed.

safeer@penguinpower:~$ scriptreplay myscript.time myscript

And this will play the typescript with timing.