Bash Recipes Part 1 - Changing extensions for multiple file names

I have came across many situations where a bunch of files with one filename extension has to be renamed to another extension. Here is the way to do it in bash.

We will rename a number of post script files to PDF files ( .ps to .pdf )

safeer@penguinpower:~/Documents$ ls
book0.ps book1.ps book2.ps book3.ps book4.ps
safeer@penguinpower:~/Documents$ for file in $(ls *.ps);
do
mv $file ${file/%.ps/.pdf};
done
safeer@penguinpower:~/Documents$ ls
book0.pdf book1.pdf book2.pdf book3.pdf book4.pdf

Perl Snippets Part 1 - XLS to CSV

For parsing Microsoft excel files (.xls) using Perl, we can make use of the module Spreadsheet::ParseExcel.
On my Ubuntu system this module is installed by:

safeer@penguinpower:~$sudo apt-get install libspreadsheet-parseexcel-perl

Following program will take an XLS file as input and print the csv format to standard out.

#!/usr/bin/perl -w

use strict;
use Spreadsheet::ParseExcel;

my $parser = Spreadsheet::ParseExcel->new();
if ( !defined $ARGV[0] ) { die "Please provide an excel file"; }
my $workbook = $parser->parse($ARGV[0]);
if ( !defined $workbook ) { die $parser->error(), ".\n"; }

for my $worksheet ( $workbook->worksheets() ) {

my ( $row_min, $row_max ) = $worksheet->row_range();
my ( $col_min, $col_max ) = $worksheet->col_range();

for my $row ( $row_min .. $row_max ) {
for my $col ( $col_min .. $col_max ) {

my $cell = $worksheet->get_cell( $row, $col );
if ( defined $cell ) { print $cell->value(); }
if ( $col < $col_max ) { print ","; }        
}   
print "\n";
}
}   

Configuring NTP service

     The network time protocol ( NTP ) is used to synchronize a computer system's time with an accurate time source. Interconnected and communicating systems should have their times in sync with each other to preserve the order of communication / sotring of information ( and and many more applications ).

     The ultimate source of accurate time is called a reference clock and will usually be an atomic/nuclear clocks ( usually using cesium as the element ).  Another variety of reference clocks will be gps receivers receiving time from satellites ( which in turn uses an atomic clock ). Other computer systems sync their time from reference clocks directly or from servers that sync from reference clock directly.

    NTP supports a hierarchy of time sources, the accuracy of each level of this hierarchy is referred to by the term startum.  A startum number is given to each source based on its reliability, smallest startum being most accurate.  Thus reference clocks are startum 0, the servers that sync time directly from them are startum 1, and servers that sync from these servers will be startum 2 and so on.  Basiaclly if the upstream NTP server is startum X, the server syncing from that server is startum X+1 upto a maximum value of 15.  NTP protocol/service has inherent mechanisms to determine the accuracy of upstream NTP server as well as its own by calculating different parameters like offset,resolution,jitter etc..

NTP is a UDP base protocol and uses port 123. Let us look at a basic NTP configuration

    First install the  NTP software, on ubuntu it is openntpd with main configuration file /etc/openntpd/ntpd.conf and optional include files kept under /etc/openntpd.  On a redhat system it will be ntp ( with configuration file at /etc/ntp.conf and include directory /etc/ntp.d/).

safeer@ntpc:~$sudo yum install ntp
safeer@ntpc:~$sudo /sbin/service ntpd status
 * ntpd is stopped

     First you have to determine from which NTP server you want to sync time.  This can be another server in your network, or one publicaly available.  You can even setup a standalone ntp server which syncs from its own hardware clock and provide time service to other servers in your network ( though not very much recommended )

     Now there are two ways to sync time from a server.  You can use the ntpdate binary, which will sync with the server time immediately, but this can have adverse effect on the server of the time difference between ntpserver and client and bigger.  The other option is to configure ntp service, which will allow the server to gracefully sync time.

How ntpdate works:

Check if the server is up and available:

safeer@ntpc:~$sudo ntpdate -q ntp1.safeer.in

The output will list one ore more servers behind the ntp dns name and its time details.  This indicates the sever is good, so go ahead and sync with it.

safeer@ntpc:~$ sudo ntpdate ntp1.safeer.in

You can provide multiple space separated NTP server to ntpdate.  If the command shows no error, the time will be set by now.  Check your system with date command.

Now let us see how the ntp server can be configured:

First add one or more servers:

server 1.1.1.1
server 2.2.2.2
server ntp1.safeer.in

NTPD can detect the time drift between system clock and the reference clock ( or server ) over a period of time and need to store this somewhere to adjust the system time.  This drift value has to be written to some file.  The user running NTP daemon should have write access to the file.

driftfile /var/lib/ntp/drift

     This much was about how you receive the time from upstream servers.  Now let us see what it takes to sync the server's own system clock or downstream servers. 
     First case, let us assume we just need to sync the time on our server and not provide time to any downstream servers ( ntp client mode )

    The configuration parameter to use here is "restrict" in the following syntax: restrict < host ip/hostname/network id > [mast < mask >] [flags .. ..]
     The mask is optional ( need only when u are defining access for networks and not single servers), if the flags are omitted the server will be having unrestricted access to ntp service.  Also you can use the entry "default" in place of host/network for setting default access policy.  If access is not explicitly defined for a host/network the default policy will be used.

    Since we are planning to sync only the local server we have to restrict everything except localhost from accessing ntp service. 

## This will deny ntp service to any host that done have an explicit "restrict" entry
restrict default ignore
## Allow access to localhost through loopback interface ( by omitting flags section )
restrict 127.0.0.1

If we are configuring our server to server other hosts, say the local network - 192.168.1.0/255.255.255.0 this is how you do it:

restric 192.168.1.0 mask 255.255.255.0 nomodify notrap nopeer
     The flag nomodify prevent clients from modifying servers ntp configuration, notrap prevents from sending log collection trap messages, nopeer prevents peering/associating the clients with the server to be "co-servers"

     If the upstream server becomes unavailable, the host will not be able to sync time, so as a fall back we configure the local clock as a time source, in ntp language the ip "127.127.1.0" indicates local clock

server 127.127.1.0 # local clock
fudge 127.127.1.0 stratum 10

If you want to log the ntp activity add the entry

logfile /var/log/ntp.log

Finally, let us consilidate everything in one place:

server 1.1.1.1
server 2.2.2.2
server ntp1.safeer.in

restrict default ignore
restrict 127.0.0.1

restric 192.168.1.0 mask 255.255.255.0 nomodify notrap nopeer

server 127.127.1.0 # local clock
fudge 127.127.1.0 stratum 10

driftfile /var/lib/ntp/drift
logfile /var/log/ntp.log


Before starting ntp service, run ntpdate with one of the servers listed as an initial sync, then start ntp service


safeer@ntpc:~$sudo ntpdate ntp1.safeer.in
safeer@ntpc:~$sudo /sbin/service ntpd start

For more on NTP, check official website and RFC page.

Converting time between timezones in Linux

People working with systems that are located in different timezones are always faced with the problem of converting time between timezones. On GNU Linux systems ( and most other POSIX OSs) you can manipulate the TZ environment variable to achieve this.

Generally on Linux systems, the timezone information is kept in the file /etc/localtime. This file is usually a softlink to an appropriate zone file under the location "/usr/share/zoneinfo/".

safeer@penguinpower:~$ ls -l /usr/share/zoneinfo/
drwxr-xr-x 2 root root 4096 2010-02-21 01:46 Africa
drwxr-xr-x 6 root root 12288 2010-02-21 01:46 America
drwxr-xr-x 2 root root 4096 2010-02-21 01:46 Antarctica
.............................
-rw-r--r-- 1 root root 2294 2010-01-03 01:38 EST5EDT
drwxr-xr-x 2 root root 4096 2010-02-21 01:46 Etc
drwxr-xr-x 2 root root 4096 2010-02-21 01:46 Europe
.............................
-rw-r--r-- 1 root root 118 2010-01-03 01:38 UTC
-rw-r--r-- 1 root root 1873 2010-01-03 01:38 WET
-rw-r--r-- 1 root root 2194 2010-01-03 01:38 W-SU
-rw-r--r-- 1 root root 18896 2009-12-12 20:27 zone.tab
-rw-r--r-- 1 root root 118 2010-01-03 01:38 Zulu

Some of these are directories which have timezone files under them. For eg, the directory Asia contains zone files for all the countries ( or their capital/important cities) in Asia, including India,China,Pakistan etc....

safeer@penguinpower:~$ ls -l /usr/share/zoneinfo/Asia/
-rw-r--r-- 1 root root 157 2010-01-03 01:38 Aden
-rw-r--r-- 1 root root 922 2010-01-03 01:38 Almaty
-rw-r--r-- 1 root root 8504 2010-01-03 01:38 Amman
............................
-rw-r--r-- 2 root root 657 2010-01-03 01:38 Ashgabat
-rw-r--r-- 2 root root 657 2010-01-03 01:38 Ashkhabad
-rw-r--r-- 1 root root 962 2010-01-03 01:38 Baghdad
...............
-rw-r--r-- 1 root root 1929 2010-01-03 01:38 Vladivostok
-rw-r--r-- 1 root root 1914 2010-01-03 01:38 Yakutsk
-rw-r--r-- 1 root root 2000 2010-01-03 01:38 Yekaterinburg
-rw-r--r-- 1 root root 2012 2010-01-03 01:38 Yerevan

We are going to use this directory structure for setting the TZ variable. The value of TZ variable will be the relative path of the desired timezone file with reference to "/usr/share/zoneinfo". As an example, I am going to change my system time to Singapore time.

safeer@penguinpower:~$ date
Tue Apr 6 00:44:54 IST 2010
safeer@penguinpower:~$ export TZ=Asia/Singapore
safeer@penguinpower:~$ date
Tue Apr 6 03:15:37 SGT 2010

What if you don't want to set the TZ variable but just need to see the current Singapore time?

safeer@penguinpower:~$ date
Tue Apr 6 00:48:07 IST 2010
safeer@penguinpower:~$ TZ=Asia/Singapore date
Tue Apr 6 03:18:12 SGT 2010
safeer@penguinpower:~$ date
Tue Apr 6 00:48:16 IST 2010

Here the value "Asia/Singapore" will be applicable only to the command 'date' during its execution.

We are going to use this, along with a feature of 'date' command for converting between different timezones.

Now our machine's timezone is India Standard Time (IST - Asia/Calcutta ), we need to see what will be the time in India when the Singapore time is 2:24 PM on a particular day.

safeer@penguinpower:~$ date -d 'TZ="Asia/Singapore" 2010-03-10 14:24:00'
Wed Mar 10 11:54:00 IST 2010

This shows that SGT is two and half hours ahead of IST. The -d option allows specifying a TZ variable which helped us in getting this answer. Now suppose that you want to see what will be the time in Dubai when it is 2.30 PM in Singapore. But our local timezone is in IST and should not be changed.

The way to do this is, set the TZ variable to Dubai timezone just for the particular date command. Here is how we can do it.

safeer@penguinpower:~$ TZ=Asia/Dubai date -d 'TZ="Asia/Singapore" 2010-03-10 14:24:00'
Wed Mar 10 10:24:00 GST 2010

This shows us that, Dubai time ( GST - Gulf Standard Time) is 4 hours behind Singapore time (SGT).

So the general syntax for conversion can be summarized as below:

TZ=$TO_TIMEZONE date -d 'TZ="$FROM_TIMEZONE" YYYY-MM-DD HH:MM:SS'
Where $FROM_TIMEZONE is the timezone from which the conversion should be done and $TO_TIMEZONE is the timezone to which the date should be converted. Here:

TO_TIMEZONE=Asia/Calcutta, FROM_TIMEZONE=Asia/Singapore


If you regularly do date conversions you can write a wrapper script around this command.

To learn more about TZ variable, please visit TZ page on GNU website

Base64 Encoding and Decoding

     base64 is an encoding scheme used to transfer binary/text data across networks. Though initially used predominantly for transmitting data through email, base64 is used in a wide variety of of applications nowadays.

      The encoding is done by converting input data into printable ASCII characters. The input stream is divided into 6 bit segments and converted to a printable character from a set of 64 encoding characters based on the value of the 6 bits ( 6 bits can represent a value between 0 to 63   - 64 numbers ).   In practical dividing by 6 bits is done by taking 3 consecutive bytes (3 x 8 bits =  4 x 6 bits ) of the input stream and converting it to 4 encoding characters.  If the total number of bytes in the input stream is not a multiple of three, extra 0 value bytes are padded to the end of the stream to make it multiple of three.

     The decoding of the input stream is done on the receiving end by reversing the above method. 

     In Linux, the "base64" command is used to decode/encode base64.  Let us see an example.  We will convert the "ls" command binary to base64 and back and see if the resultant binary is still working as "ls".

safeer@penguinpower:~/TMP$ cp /bin/ls .
safeer@penguinpower:~/TMP$ /home/safeer/TMP/ls
ls

Convert the binary to base64

safeer@penguinpower:~/TMP$ cat /home/safeer/TMP/ls | base64 > ls.encoded

Compare both

safeer@penguinpower:~/TMP$ wc ls ls.encoded
   503   3271 104508 ls
  1834   1834 141178 ls.encoded
  2337   5105 245686 total


safeer@penguinpower:~/TMP$ head -1 ls.encoded
f0VMRgEBAQAAAAAAAAAAAAIAAwABAAAANL4ECDQAAADckwEAAAAAADQAIAAJACgAHAAbAAYAAAA0
safeer@penguinpower:~/TMP$ tail -1 ls.encoded
AAAA6JIBAPIAAAAAAAAAAAAAAAEAAAAAAAAA
safeer@penguinpower:~/TMP$ file ls.encoded
ls.encoded: ASCII text
safeer@penguinpower:~/TMP$ file ls
ls: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24, BuildID[sha1]=0x83531f308f1fa18221be53eaf399303400c14638, stripped

Now convert the encoded data back

safeer@penguinpower:~/TMP$ cat ls.encoded |base64 -d > ls.decoded
Compare with original "ls" binary

safeer@penguinpower:~/TMP$ wc ls ls.decoded
   503   3271 104508 ls
   503   3271 104508 ls.decoded
  1006   6542 209016 total
safeer@penguinpower:~/TMP$ file ls.decoded
ls.decoded: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24, BuildID[sha1]=0x83531f308f1fa18221be53eaf399303400c14638, stripped

safeer@penguinpower:~/TMP$ chmod a+x ls.decoded
safeer@penguinpower:~/TMP$ /home/safeer/TMP/ls.decoded
ls  ls.decoded    ls.encoded

Now that you know how the conversion works, let us see some other ways to decode/encode using base64.  We will use a simple string this time for simplicity.

Bash

Encoding

safeer@penguinpower:~/TMP$ printf "safeer"|base64
c2FmZWVy

Decoding

safeer@penguinpower:~/TMP$ printf "c2FmZWVy"|base64 -d
safeer

Perl

Encoding

safeer@penguinpower:~/TMP$ perl -MMIME::Base64 -e 'print encode_base64("safeer");'
c2FmZWVy

Decoding

safeer@penguinpower:~/TMP$ perl -MMIME::Base64 -e 'print decode_base64("c2FmZWVy");'
safeer

PHP

Encoding

safeer@penguinpower:~/TMP$ php -r "print base64_encode('safeer');"
c2FmZWVy

Decoding

safeer@penguinpower:~/TMP$ php -r "echo base64_decode('c2FmZWVy');"
safeer

Python

Encoding

safeer@penguinpower:~/TMP$ python -c "import base64;print base64.b64encode('safeer')"
c2FmZWVy

Decoding

safeer@penguinpower:~/TMP$ python -c "import base64;print base64.b64decode('c2FmZWVy')"
safeer