Bash Recipes Part 4 - Read from multiple files in parallel

     I had to download a set of files for which the download URLs where provided in one file and the corresponding file names in another file.  Now to download and name the downloaded files at the same time, I have to loop through both the files in parallel, picking one URL and its name at a time and feeding it to wget.   I used while loop combined with read command for this as follows:

while read -u5 url;read -u6 name
do
/usr/bin/wget  -c  "${url}" -O "${name}"
done 5 < url.txt 6 < name.txt

    In the last line of the loop, we are redirecting the contents of files "url.txt" and "name.txt" respectively to file descriptors 5 and 6.  The while loop iterates over two read commands, each of which reads from the file descriptors 5 and 6 one line at a time and save the line into variables "url" and "name" respectively.  These variables are used with wget to download and set the name.

     Note that file descriptor numbers 0,1 & 2 are reserved for standard input, standard output and standard error and shouldn't be used.

A simple logger for Perl

     Below is a simple logger for your custom perl applications.  The logger is created as a module and can be included in any script.  Code is explained within the program as comments

Module :


safeer@penguinpower:~$ cat DEV/Perl/MyLogger.pm

#!/usr/bin/perl -w
### Module Name is MyLogger
package MyLogger;
### Include Modules
use IO::File;
use DateTime;
use File::Basename;
use File::Spec;

### Constructor

sub new
{

my $class = shift;
### Object creation parameters
my(%params) = @_;
my $self = { };
### Identify the parent scripting using the class
(undef,$self{'caller'}) = caller;
### Whether to append current date to the log file name or not
### Off by default
$self->{'datedLogFile'} = $params{'datedLogFile'}  || 0;
### Whether to include scriptname in the log entries
### Off by default
$self->{'incScriptName'} = $params{'incScriptName'}  || 0;
### Whether to include scriptname - with full path - in the log entries
### Off by default
$self->{'incScriptWithPath'} = $params{'incScriptWithPath'}  || 0;
### Whether to include current date and time in log entries
## On by default
$self->{'incDateTime'} = $params{'incDateTime'} || 1;
### logFileBase - basename from which the logfile name should be constructed
if( defined $params{'logFileBase'} ) { 
$self->{'logFile'} = $params{'logFileBase'}; 
if( $self->{'datedLogFile'} == 1 ) { 
my $currentDate = DateTime->now->strftime('-%d-%m-%Y');
$self->{'logFile'} .= $currentDate;
}
### Append ".log" to the end of log file name
$self->{'logFile'} .= ".log";
}
else { print "No logfile defined!!" ; exit 1 ; }
### Open the file and store the file object pointer in the class
$self->{'log'} = IO::File->new;
$self->{'log'}->open("$self->{'logFile'}",">>");

bless $self,$class;
return $self;
}

### This function will do the actual logging
### It is preferred not to use this function directly
### Rather call the functions derived from this one
### Like debug,info,warn,crit
sub logme()
{
my $self = shift;
### Accept two parameters, log level ( DEBUG/INFO/WARN/CRIT )
### and log message ( can be multi line message ) 
my ($logLevel,$message) = @_ ;
### Breakup multiline messages and process line by line
@logentries = split("\n" , $message  );
foreach my $logentry ( @logentries ) {
my $logprefix = "";
### Include date and time
if( $self->{'incDateTime'} == 1 ) { 
  $timestamp = DateTime->now(time_zone=>"local")->strftime("[%d-%m-%Y %H:%M:%S]");
  $logprefix .= $timestamp ;
  }
### Add log level
$logprefix .=   "[" . $logLevel ."]" ;
### Include scriptname with path
if( $self->{'incScriptWithPath'} == 1 ) {
  my $scriptpath = File::Spec->rel2abs($self{'caller'}) ;
  $logprefix .= "[" . $scriptpath . "]"; 
  }
### Include scriptname wihtout path
elsif( $self->{'incScriptName'} == 1 ) { 
  my $scriptname = basename($self{'caller'}) ;
  $logprefix .= "[" . $scriptname . "]" ;
  }

$self->{'log'}->print( "$logprefix $logentry\n" );

}

}

### Following functions are defined for the convenience of user
### All of them calls logme() funcation  internally

### Used to register informative messages
sub info()
{
my $self = shift;
$self->logme("INFO",$_[0]);
}

### Used to register warning messages
sub warn()
{
my $self = shift;
$self->logme("WARN",$_[0]);
}

### Used to register critical messages
sub crit()
{
my $self = shift;
$self->logme("CRIT",$_[0]);
}

### Used to register debug messages
sub debug()
{
my $self = shift;
$self->logme("DEBUG",$_[0]);
}

### Destroyer
sub DESTROY
{
my $self = shift;
$self->{'log'}->close;
}


1;

Now use this module in a script


safeer@penguinpower:~/DEV/Perl$ cat loguse.pl

#!/usr/bin/perl
### Define custom library path for adding modules
use lib '/home/safeer/DEV/Perl/CustomModules';
### Include the cutom logger module
use MyLogger;
### Create an instance of custom logger
my $logger = MyLogger->new('logFileBase' => "/tmp/logPerl" , 'datedLogFile' => 1 , 'incScriptName' => 1 , 'incScriptWithPath' => 1 );
### Log all types of log messages
$logger->info("Information Message");
$logger->warn("Warning Message");
$logger->crit("Critical Message");
$logger->debug("Debug Message");

Execute the script

safeer@penguinpower:~/DEV/Perl$ ./loguse.pl

Look at the log file created

safeer@penguinpower:~$ cat /tmp/logPerl-17-07-2012.log
[17-07-2012 23:26:15][INFO][/home/safeer/DEV/Perl/loguse.pl] Information Message
[17-07-2012 23:26:15][WARN][/home/safeer/DEV/Perl/loguse.pl] Warning Message
[17-07-2012 23:26:15][CRIT][/home/safeer/DEV/Perl/loguse.pl] Critical Message
[17-07-2012 23:26:15][DEBUG][/home/safeer/DEV/Perl/loguse.pl] Debug Message