A simple logger for Perl

     Below is a simple logger for your custom perl applications.  The logger is created as a module and can be included in any script.  Code is explained within the program as comments

Module :


safeer@penguinpower:~$ cat DEV/Perl/MyLogger.pm

#!/usr/bin/perl -w
### Module Name is MyLogger
package MyLogger;
### Include Modules
use IO::File;
use DateTime;
use File::Basename;
use File::Spec;

### Constructor

sub new
{

my $class = shift;
### Object creation parameters
my(%params) = @_;
my $self = { };
### Identify the parent scripting using the class
(undef,$self{'caller'}) = caller;
### Whether to append current date to the log file name or not
### Off by default
$self->{'datedLogFile'} = $params{'datedLogFile'}  || 0;
### Whether to include scriptname in the log entries
### Off by default
$self->{'incScriptName'} = $params{'incScriptName'}  || 0;
### Whether to include scriptname - with full path - in the log entries
### Off by default
$self->{'incScriptWithPath'} = $params{'incScriptWithPath'}  || 0;
### Whether to include current date and time in log entries
## On by default
$self->{'incDateTime'} = $params{'incDateTime'} || 1;
### logFileBase - basename from which the logfile name should be constructed
if( defined $params{'logFileBase'} ) { 
$self->{'logFile'} = $params{'logFileBase'}; 
if( $self->{'datedLogFile'} == 1 ) { 
my $currentDate = DateTime->now->strftime('-%d-%m-%Y');
$self->{'logFile'} .= $currentDate;
}
### Append ".log" to the end of log file name
$self->{'logFile'} .= ".log";
}
else { print "No logfile defined!!" ; exit 1 ; }
### Open the file and store the file object pointer in the class
$self->{'log'} = IO::File->new;
$self->{'log'}->open("$self->{'logFile'}",">>");

bless $self,$class;
return $self;
}

### This function will do the actual logging
### It is preferred not to use this function directly
### Rather call the functions derived from this one
### Like debug,info,warn,crit
sub logme()
{
my $self = shift;
### Accept two parameters, log level ( DEBUG/INFO/WARN/CRIT )
### and log message ( can be multi line message ) 
my ($logLevel,$message) = @_ ;
### Breakup multiline messages and process line by line
@logentries = split("\n" , $message  );
foreach my $logentry ( @logentries ) {
my $logprefix = "";
### Include date and time
if( $self->{'incDateTime'} == 1 ) { 
  $timestamp = DateTime->now(time_zone=>"local")->strftime("[%d-%m-%Y %H:%M:%S]");
  $logprefix .= $timestamp ;
  }
### Add log level
$logprefix .=   "[" . $logLevel ."]" ;
### Include scriptname with path
if( $self->{'incScriptWithPath'} == 1 ) {
  my $scriptpath = File::Spec->rel2abs($self{'caller'}) ;
  $logprefix .= "[" . $scriptpath . "]"; 
  }
### Include scriptname wihtout path
elsif( $self->{'incScriptName'} == 1 ) { 
  my $scriptname = basename($self{'caller'}) ;
  $logprefix .= "[" . $scriptname . "]" ;
  }

$self->{'log'}->print( "$logprefix $logentry\n" );

}

}

### Following functions are defined for the convenience of user
### All of them calls logme() funcation  internally

### Used to register informative messages
sub info()
{
my $self = shift;
$self->logme("INFO",$_[0]);
}

### Used to register warning messages
sub warn()
{
my $self = shift;
$self->logme("WARN",$_[0]);
}

### Used to register critical messages
sub crit()
{
my $self = shift;
$self->logme("CRIT",$_[0]);
}

### Used to register debug messages
sub debug()
{
my $self = shift;
$self->logme("DEBUG",$_[0]);
}

### Destroyer
sub DESTROY
{
my $self = shift;
$self->{'log'}->close;
}


1;

Now use this module in a script


safeer@penguinpower:~/DEV/Perl$ cat loguse.pl

#!/usr/bin/perl
### Define custom library path for adding modules
use lib '/home/safeer/DEV/Perl/CustomModules';
### Include the cutom logger module
use MyLogger;
### Create an instance of custom logger
my $logger = MyLogger->new('logFileBase' => "/tmp/logPerl" , 'datedLogFile' => 1 , 'incScriptName' => 1 , 'incScriptWithPath' => 1 );
### Log all types of log messages
$logger->info("Information Message");
$logger->warn("Warning Message");
$logger->crit("Critical Message");
$logger->debug("Debug Message");

Execute the script

safeer@penguinpower:~/DEV/Perl$ ./loguse.pl

Look at the log file created

safeer@penguinpower:~$ cat /tmp/logPerl-17-07-2012.log
[17-07-2012 23:26:15][INFO][/home/safeer/DEV/Perl/loguse.pl] Information Message
[17-07-2012 23:26:15][WARN][/home/safeer/DEV/Perl/loguse.pl] Warning Message
[17-07-2012 23:26:15][CRIT][/home/safeer/DEV/Perl/loguse.pl] Critical Message
[17-07-2012 23:26:15][DEBUG][/home/safeer/DEV/Perl/loguse.pl] Debug Message

Perl One Liner for dumping xml files

     Whenever we work with an xml file, the first thing we like to find out is how the xml is organized.  Once we get the structure of the file, it will be easier to write code to process the xml data.  Perl developers popularly use the Dumper function from the Data::Dumper module to achieve this inside their code.  But if you have the xml file readily available and you want to take a peak into how the data is organized within the xml, you don’t have to start writing a program, instead run the following one liner with the xml file name as the last argument.

perl -MData::Dumper -MXML::Simple -e 'print Dumper(new XML::Simple->XMLin($ARGV[0]));' XML_FILE_NAME

-M - Followed by a perl module name make the module available for the perl command.  So here we are loading modules Data::Dumper and XML::Simple
-e  - Any quoted string following this will be executed as perl code.

Now examining the executable code itself,

$ARGV[0] is the first command line argument passed to the perl command after all the command "options"

XML::Simple module provide the XMLin function, which can take an xml file as argument and return a reference to a data structure containing the xml data in a more structured and accessible format.

Dumper function take any perl variable or variable reference as an argument and returns it as a structured printable string.

So the executable code,

 * dynamically calls the XMLin function made available by the XML::Simple module and
 *  passes the xml file name as its argument ( which was provided as the command line argument to perl)
 * XMLin return the xml in a data structure reference, which is
 * passed as an argument to Dumper function made available by the Dta::Dumper module, which
 * returns a printable string representation of the above reference, and
 * the print command just prints it out.

A small example

safeer@penguinpower:~$ perl -MData::Dumper -MXML::Simple -e 'print Dumper(new XML::Simple->XMLin($ARGV[0]));'  dummy.xml  
$VAR1 = { 
         'language' => { 

               'perl' => { 

                   'version' => '5.8', 

                   'content' => 'Practical Extraction and Report Language'          

                            },

          'name' => 'perl'  

                         } 

        }; 

To make life easier I have wrapped it in a bash  function and added to my bashrc

safeer@penguinpower:~$ type xmlDump 

xmlDump is a function 

xmlDump () {     

XML=${1?You should provide an xml file};   

perl -MData::Dumper -MXML::Simple -e 'print Dumper(new XML::Simple->XMLin($ARGV[0]));' $XML 

}

Authenticated website access using Perl

   This code will allow you to access a website that requires authentication via a username password pair.

Raw materials needed for this script are:

1. The URL of the sign-in page.  Shoot up your browser, go to the website you want access, go to the login screen and copy the URL address.
2. The name/number of the html login form:
   To obtain this, launch the sign in URL in a browser window, view source of the page.
   Look for the block that starts with <form, the value of the "name" property is the form name ( if exists )
   If there is no form name:
   If only one form is found in the source code, the form number will be 1
   If there are multiple forms and the login form is N-th one in the list, N is the form number.
3. The value of the "name" property corresponding to the input elements for username and password in the form.

   With these details, the below mentioned code can log you into the page.  Comments are added above the code lines, wherever necessary.

#!/usr/bin/perl -w

use strict;
## WWW::Mechanize is a wrapper on LWP:UserAgent object,
## which extents the browser capabilities of LWP.
use WWW::Mechanize;
## Needed for initializing and storing cookies for the website
use HTTP::Cookies;
## Needed if your code has to support SSL
use Crypt::SSLeay;
## Use only if you want to trouble shoot!!
use LWP::Debug qw(+);

## I want to login to Ubuntu forum, so this is my URL,
my $URL = "http://ubuntuforums.org";

## My Ubuntu forum username and password
 my $username = "your_username";
 my $password = "your_password";

## Create a browser and a cookie object
 my $browzer = WWW::Mechanize->new();
 $browzer->cookie_jar(HTTP::Cookies->new());

## Open ( HTTP GET ) the login URL
 $browzer->get($URL);

## Ubuntu forum has no form name, so using the number "2" 
## as it is the second form on the login page,
## first being a forum search box.
$browzer->form_number(2);
## vb_login_username/vb_login_password  are the input names
## used in this form
$browzer->field(vb_login_username => $username);
$browzer->field(vb_login_password => $password);
##Submit the form
$browzer->submit();

## If you are not automatically redirected to the
## forum home page after loging in,
## emulate clicking the link that says
## "Click here if your browser does not automatically
## redirect you."
 $browzer->follow_link( text =>
"Click here if your browser does not automatically redirect you."
    );

## Now that you have logged in, you can use different methods of 
## WWW::Mechanize to process the web pages.
## Eg: Save the content of the page you see after loging in 
## my $output_page = $browzer->content();
## my $output_file = "/tmp/mylogin.html"
## open(OUTFILE, ">$output_file");
## print OUTFILE "$output_page";
## close(OUTFILE);

Repair corrupted MySQL tables

     Sometimes mysql database tables get corrupted for various reasons like server reboot, multiple processes trying to write to same table etc.. If your databses is using MyISAM storage engine ( which is the default one ), you can recover these tables the following way.

     MyISAM provides a built-in tool for repairing corrupted tables.  The command is myisamchk, which can be used for varied functions like scanning database for errors, tuning tables for performance, repair corrupted tables etc.

     There is also an SQL variant for some functionalities of myisamchk, like CHECK TABLE,REPAIR TABLE etc..  While myisamchk is ran from command line, these queries should be ran from sql console.

     For me, the issue started when I tried to backup a database with mysqldump

safeer@db2 ~:$sudo /usr/bin/mysqldump oncall > oncall.sql
mysqldump: Got error: 1194: Table 'log' is marked as crashed and should be repaired when using LOCK TABLES

     Now this is a sign of problem and to start with troubleshooting, we should first check the table for errors.

     Two things before we proceed: as a best practice if possible, we should shut down the mysql server or make sure no one is accessing the tables ( obtaining a lock on the tables ).  Another thing to do is, backup the database before repairing its table.  In some cases, there is a possibility that while repairing the tables some data might be lost.

     For my case the mysql server was not a production server, rather an internal service.  So I was able to shut it down.  But backing up the databaes using mhysqldump was not an option as you saw above, so I copied out the database's data directory.  In my system the database directory is at /var/lib/mysql/oncall.

     Now let us start checking the table with myisamchk.  myisamchk requires the actual table filename with full path under mysql data directory to work with.

safeer@db2 ~:$sudo myisamchk /var/lib/mysql/oncall/log.MYI
Checking MyISAM file: /var/lib/mysql/oncall/log.MYI
Data records:  116889   Deleted blocks:       0
myisamchk: warning: Table is marked as crashed
myisamchk: warning: 3 clients are using or haven't closed the table properly
- check file-size
- check record delete-chain
- check key delete-chain
- check index reference
- check data record references index: 1
myisamchk: error: Found key at page 1439744 that points to record outside datafile
- check record links
myisamchk: error: Checksum for key:  1 doesn't match checksum for records
myisamchk: warning: Found     116889 parts                Should be: 116890 parts
MyISAM-table '/var/lib/mysql/oncall/log.MYI' is corrupted
Fix it using switch "-r" or "-o"

     Now that we know the table is corrupted, let us try fixing it.

safeer@db2 ~:$sudo myisamchk -r /var/lib/mysql/oncall/log.MYI
- recovering (with sort) MyISAM-table '/var/lib/mysql/oncall/log.MYI'
Data records: 116889
- Fixing index 1

     Check again to confirm the table is fixed

safeer@db2 ~:$sudo myisamchk /var/lib/mysql/oncall/log.MYI
Checking MyISAM file: /var/lib/mysql/oncall/log.MYI
Data records:  116889   Deleted blocks:       0
- check file-size
- check record delete-chain
- check key delete-chain
- check index reference
- check data record references index: 1
- check record links

     The table is fixed now.

     Now let us see how we can do the same task from mysql console using sql commands.  To do this, I am copying the backed up corrupted mysql files to the /var/lib/mysql/oncall location so that the db will be in the corrupted state again (This is the advantage of doing file system level db backup, you can play around it by copying data files back and forth :) ).

Select Database

mysql oncall>use oncall;

Check Table

mysql oncall>check table log;
+------------+-------+----------+-----------------------------------------------------------+
| Table      | Op    | Msg_type | Msg_text                                                  |
+------------+-------+----------+-----------------------------------------------------------+
| oncall.log | check | warning  | Table is marked as crashed                                |
| oncall.log | check | warning  | 3 clients are using or haven't closed the table properly  |
| oncall.log | check | error    | Size of datafile is: 22364116         Should be: 22364308 |
| oncall.log | check | error    | Corrupt                                                   |
+------------+-------+----------+-----------------------------------------------------------+
Repair Table

mysql> repair table log;
+------------+--------+----------+----------+
| Table      | Op     | Msg_type | Msg_text |
+------------+--------+----------+----------+
| oncall.log | repair | status   | OK       |
+------------+--------+----------+----------+
1 row in set (11.29 sec)

Check Again

mysql> check table log;
+------------+-------+----------+----------+
| Table      | Op    | Msg_type | Msg_text |
+------------+-------+----------+----------+
| oncall.log | check | status   | OK       |
+------------+-------+----------+----------+
1 row in set (7.44 sec)

     If possible always use myisamchk, it has more options that can be used if you run into problems while repairing tables.  A few options worth remembering are:

For checking:

  -F, --fast          Check only tables that haven't been closed properly.
  -m, --medium-check  Faster than extend-check, but only finds 99.99% of
                      all errors.  Should be good enough for most cases.
  -e, --extend-check  Check the table VERY throughly.  Only use this in
                      extreme cases as myisamchk should normally be able to
                      find out if the table is ok even without this switch.

For repairing:

  -r, --recover       Can fix almost anything except unique keys that aren't
                      unique.
  -o, --safe-recover  Uses old recovery method; Slower than '-r' but can
                      handle a couple of cases where '-r' reports that it
                      can't fix the data file.
  -q, --quick         Faster repair by not modifying the data file.
                      One can give a second '-q' to force myisamchk to
                      modify the original datafile in case of duplicate keys.
                      NOTE: Tables where the data file is currupted can't be
                      fixed with this option.

     For more options, checkout the man page and following mysql website links

MYISAMCHK

CHECK TABLES

Bash Recipes Part 3 - Converting between decimal and hexadecimal

     People who work with computer systems would have encountered the problem of converting between decimal and hexadecimal numbers one time or the other.  Here is a bash solution for it

function decToHex()
{
DEC=${1?You should provide an argument}
echo $DEC|grep -q -E "^[0-9]+$" || { echo "The argument is not a decimal"; return 1 ; }
printf "%x\n" ${DEC}
}

function hexToDec()
{
HEX=${1?You should provide an argument}
echo $HEX|grep -q -E "^[0-9a-fA-F]+$" || { echo "The argument is not a hexadecimal"; return 1 ; }
printf "%d\n" 0x${HEX}
}

A few examples

safeer@penguinpower:~$ decToHex 10
a
safeer@penguinpower:~$ hexToDec a
10
safeer@penguinpower:~$ decToHex 1234
4d2
safeer@penguinpower:~$ hexToDec abc
2748