Netcat as a file downloader

     The netcat utility is a multi-purpose tool for managing and manipulating TCP/IP traffic.  In this article, we will see how netcat can be used as a file downloader.  This will come in handy when you don't have utilities like wget/fetch/curl installed in our machine.

     Netcat ( "nc" is the name of the binary ) can establish tcp connection to any server/port combination and send or receive data through the established channel.  To use it as a downloader, our strategy will be:

• Establish a connection to the http port of the server.
• Send an HTTP request with the download link to the established connection
• Redirect the output of the HTTP response to a file ( which will be the download file).

     Let us try downloading apache httpd package from the url  http://apache.techartifact.com/mirror/httpd/httpd-2.4.3.tar.gz

     First, let us establish a TCP connection to port 80 of server apache.techartifact.com.  Command for this is:

/bin/nc apache.techartifact.com 80

     Second, let us construct an HTTP request. This can be done in two ways - using HTTP protocol 1.0 version and HTTP 1.1 version.

     A generic HTTP request format consists of:

• A request line ( Further contains request methode - "GET"  for download , request URI - the whole/relative download URL , protocol version - HTTP/1.0 or HTTP/1.1 )
• Multiple lines of HTTP headers ( Each HTTP header is a single line containing a header name and header value separated by a column and space )
• An empty line
• Message body

     Each of these lines will be separated by a Carriage Return ( \r ) and a Line Feed ( \n ) characters.

     Though there are many parts for an HTTP request, a bare minimum HTTP  request requires only the following:

• A request line ( for both HTTP/1.0 and 1.1 version )
• A host header ( only for HTTP/1.1 , Format is - Host: web.server.name )
• A blank line

     All separated by a CR and CF ( "\r" & "\n" )

• HTTP/1.0 request for our download URL is:  GET http://apache.techartifact.com/mirror/httpd/httpd-2.4.3.tar.gz HTTP/1.0\r\n\r\n
• HTTP/1.1 request for our download URL is:  GET http://apache.techartifact.com/mirror/httpd/httpd-2.4.3.tar.gz HTTP/1.0\r\nHost: apache.techartifact.com\r\n

     When we sent this request, the response from the  server ( if everything is good and file start getting downloaded ) will contain an http response which begins with line "HTTP/1.1 200 OK" followed by multiple header lines, then followed by a blank line ( containing "\r" ) followed by the response data ( which is the actual file to be downloaded ).  So while saving the response to a file we should strip off the http header information part (all lines between and including "HTTP/1.1 200 OK" and "\r").  This can be achieved by a simple sed command.

     To learn more about HTTP visit this link

     Let us try downloading the file with HTTP/1.0:

safeer@penguinepower:/tmp$ echo -e "GET http://apache.techartifact.com/mirror/httpd/httpd-2.4.3.tar.gz HTTP/1.0\r\n\r\n"|nc apache.techartifact.com 80|sed '/^HTTP\/1.. 200 OK\r$/,/^\r$/d' > httpd-2.4.3-with-http-1.0.tar.gz

     Now with HTTP/1.1

safeer@penguinepower:/tmp$ echo -e "GET http://apache.techartifact.com/mirror/httpd/httpd-2.4.3.tar.gz HTTP/1.1\r\nHost: apache.techartifact.com\r\n"|nc apache.techartifact.com 80 | sed '/^HTTP\/1.. 200 OK\r$/,/^\r$/d' > httpd-2.4.3-with-http-1.1.tar.gz

     Let us also download the file with wget utility directly

safeer@penguinepower:/tmp$ wget -q http://apache.techartifact.com/mirror/httpd/httpd-2.4.3.tar.gz -O httpd-2.4.3-with-wget.tar.gz
     Now compare all the files downloaded to ensure they are all the same.

safeer@penguinepower:/tmp$ du -bs httpd-2.4.3-with-*
6137268 httpd-2.4.3-with-http-1.0.tar.gz
6137268 httpd-2.4.3-with-http-1.1.tar.gz
6137268 httpd-2.4.3-with-wget.tar.gz
safeer@penguinepower:/tmp$ md5sum httpd-2.4.3-with-*
538dccd22dd18466fff3ec7948495417  httpd-2.4.3-with-http-1.0.tar.gz
538dccd22dd18466fff3ec7948495417  httpd-2.4.3-with-http-1.1.tar.gz
538dccd22dd18466fff3ec7948495417  httpd-2.4.3-with-wget.tar.gz

Let us ensure the integrity of the downloaded files by comparing their md5 with the value given in apache website

safeer@penguinepower:/tmp$ curl -s http://www.apache.org/dist/httpd/httpd-2.4.3.tar.gz.md5
538dccd22dd18466fff3ec7948495417 *httpd-2.4.3.tar.gz

Everything looks good now.

Note: This command can download from servers on which the file is actually located ( on the given port and location as in the URL ).  I haven't tested the case where the the actual file is behind a proxy and the download url redirects you to the correct location ( with an HTTP 302 message).  That situation will need some more logic.

Finding the size of a remote file from its URL

Consider the following cases,

You are about to download a file from web, and before downloading, you want to know the size of that remote file

                           OR

You have downloaded a file earlier, but you are doubtful whether the file has been downloaded fully or not.

From a browser like Firefox or Chrome, you can go to the download url and a window will popup asking you whether to save or open the file.  The same popup will also mention the size of the remote file.  While this is one way of doing it in many case you might want to do this for multiple files and/or save this information in a report or use it inside another script or task.  In such scenarios it is desirable to do this from command line, and the following script will show you how to do it.

To do this, we should have the curl utility installed, on my machine it is installed as "/usr/bin/curl".  We will first see the script and then the explanation to it.

safeer@penguinpower:~$ curl -sI DOWNLOAD_URL |awk '/Content-Length/{ print $2 }'

This will provide you the remote file size in bytes.

The basic idea is to process the HTTP headers associated with the download link, without actually downloading the file.

HTTP header of a URL contains useful information like, return status of your request, server details, content type, content length etc..  In this case, we are interested in the "Content Length" field which provides the size of the remote content ( downloadable file in this case ) in bytes.



Let us examine a typical HTTP header.  This header is for the download URL of Apache HTTPD web server.  We use curl with -s ( silent ) and -I ( info ie; header information ) flags to obtain this.

safeer@penguinpower:/tmp$ curl -sI http://apache.techartifact.com/mirror/httpd/httpd-2.4.3.tar.bz2
HTTP/1.1 200 OK
Date: Sun, 16 Dec 2012 19:46:14 GMT
Server: Apache/2.2.23 (Unix) mod_ssl/2.2.23 OpenSSL/0.9.8e-fips-rhel5 DAV/2 mod_mono/2.6.3 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635 mod_jk/1.2.35 mod_qos/9.74 mod_perl/2.0.6 Perl/v5.8.8
Last-Modified: Mon, 20 Aug 2012 13:22:55 GMT
ETag: "8708060-4591af-4c7b2684fa9c0"
Accept-Ranges: bytes
Content-Length: 4559279
Connection: close
Content-Type: application/x-tar

There are multiple fields in the header as you can see, but our interest is in the Content-Length field, which has a value of 4559279.  So as per the header, the size of the httpd package is 4559279 bytes.

Let us cross verify by downloading the file.

safeer@penguinpower:/tmp$ wget -q http://apache.techartifact.com/mirror/httpd/httpd-2.4.3.tar.bz2
safeer@penguinpower:/tmp$ du -b httpd-2.4.3.tar.bz2
4559279 httpd-2.4.3.tar.bz2

Well, the file size is indeed 4559279 bytes.


Now we know how to extract the content length from the HTTP header.  But is that all? what if the webserver is functioning but the url is not available or some other issue prevent you from downloading the header?  This may not be a problem when you are actually looking at the terminal while the command is running, but within a script or report this may not be a good idea.

To solve this, we first check the HTTP status code ie; the first line in the header to see if the response is 200 ( OK ).  Only then we return the content length, else we return a negative value so that your script / report can identify that.  In this case that value will be the negative of the HTTP response code ( when different from 200 OK ).  So the response code will tell you whether the script failed and for what reason.  This will help you in designing the fail-safe logic of the script.

safeer@penguinpower:/tmp$curl -sI http://apache.techartifact.com/mirror/httpd/httpd-2.4.3.tar.bz2 | awk '/^HTTP\/1./{if ( $2 == 200 ) { do { getline } while ( $1 != "Content-Length:" ) ; print $2} else { print "-"$2 } }
 
The awk first looks for the HTTP status code and if the code is "200 OK" the getline function advances the current record pointer to next line until the first part of the current line is "Content-Length", then prints the second part of the line which is the content length in bytes.  Otherwise, the script prints the negative of the HTTP status code.

Note: We still need to take care of the curl errors where there wont be any response from the webserver ( no dns/network connection etc.. ) in our scripts logic.  Use the exit code of curl ( a non a negative value ) for finding that out.

A logger for PHP applications

     This is a simple logger implementation that you can plug into your php applications without installing anything.

     Logger class definition, comments are provided wherever necessary.

safeer@penguinpower:~/DEV/PHP$ cat logger.php

class logger {
### Variable declaration & initialization
private $logFile = NULL;
private $filePtr = NULL;
private $incScriptName = FALSE;
private $incScriptPath = FALSE;
### Constructor, sets log file name and log entry behaviour
function __construct($options) {
### Open logfile
if( isset($options['logfile'])) 
        { 
         $logDate = date('d-m-Y');
### Final logfile name should be <LOGFILE>.<DD-MM-YYYY>.log
         $this->logFile = $options['logfile'].".".$logDate.".log"; 
### Create if log file is not existing, then open for appending
         $this->filePtr = fopen($this->logFile,"a") 
                or exit("Unable to open logfile $file\n"); 
        }
### Abort if the logfile is not set during object creation
else 
        { exit( "Logfile should be defined\n" ) ; }
### Whether logger should add calling script name to the log file
### Include script name in log entry
if( $options['scriptname'] ==  TRUE )
        { $this->incScriptName = TRUE; }
### If set include full path of script, else just script name
elseif( $options['scriptpath'] == TRUE )
        { $this->incScriptPath = TRUE; }

}
### Destructor, close the file handle when object is destroyed
function __destruct() {
if($this->filePtr) { fclose($this->filePtr); }
}
### Logging happens here
### Each line in the log file ( log entry ) will start with a log prefix
### The log prefix will start with [DATE][LOG_TYPE]
### Log type will be one of INFO/DEBUG/WARN/CRIT
private function logme($logType,$logMsg) {
$timeStamp = date('Y-m-d H:i:s');
$logPrefix = "[".$timeStamp."]";
$logPrefix .= "[".$logType."]"; 
$message = NULL;
### Add script name to log prefix based on setting
if( $this->incScriptName == TRUE and $this->incScriptPath == TRUE )
        { $logPrefix .= "[".$_SERVER['PHP_SELF']."]"; }
elseif ( $this->incScriptName == TRUE and $this->incScriptPath == FALSE )
        {  $logPrefix .= "[".basename( $_SERVER['PHP_SELF'] )."]"; }
### Break down multi-line log message and append prefix to each line
foreach ( explode("\n",$logMsg) as $msg) {
        $message .= "$logPrefix $msg\n";
        }
### Write log entries to file
fwrite($this->filePtr, $message) 
or exit("Unable to write to log ".$this->logFile);

}
### Following functions are defined for the convenience of user
### All of them calls logme() funcation  internally

### Used to register informative messages
public function info($logMsg) {
$this->logme("INFO",$logMsg);
}
### Used to register debug messages
public function debug($logMsg) {
$this->logme("DEBUG",$logMsg);
}
### Used to register warning messages
public function warn($logMsg) {
$this->logme("WARN",$logMsg);
} 
### Used to register critical messages
public function crit($logMsg) {
$this->logme("CRIT",$logMsg);
}

}
?>

     Sample use of the logger

safeer@penguinpower:~/DEV/PHP$ cat logcaller.php

### Inlcude logger class definition
include_once("logger.php");
### Define
$opts['logfile']="/tmp/testLogger";
$opts['scriptname'] = TRUE;
$opts['scriptpath'] = FALSE;
$logger = new logger($opts);
### Register all types of log messages
$logger->info("Information Message");
$logger->debug("Debug Message");
$logger->warn("Warning Message");
$logger->crit("Critical Message");
?>

Run php application and see the logging.

safeer@penguinpower:~/DEV/PHP$ php logcaller.php
safeer@penguinpower:~/DEV/PHP$ cat /tmp/testLogger.17-10-2012.log
[2012-10-10 13:31:54][INFO][logcaller.php] Information Message
[2012-10-10 13:31:54][DEBUG][logcaller.php] Debug Message
[2012-10-10 13:31:54][WARN][logcaller.php] Warning Message
[2012-10-10 13:31:54][CRIT][logcaller.php] Critical Message

Bash Recipes Part 4 - Read from multiple files in parallel

     I had to download a set of files for which the download URLs where provided in one file and the corresponding file names in another file.  Now to download and name the downloaded files at the same time, I have to loop through both the files in parallel, picking one URL and its name at a time and feeding it to wget.   I used while loop combined with read command for this as follows:

while read -u5 url;read -u6 name
do
/usr/bin/wget  -c  "${url}" -O "${name}"
done 5 < url.txt 6 < name.txt

    In the last line of the loop, we are redirecting the contents of files "url.txt" and "name.txt" respectively to file descriptors 5 and 6.  The while loop iterates over two read commands, each of which reads from the file descriptors 5 and 6 one line at a time and save the line into variables "url" and "name" respectively.  These variables are used with wget to download and set the name.

     Note that file descriptor numbers 0,1 & 2 are reserved for standard input, standard output and standard error and shouldn't be used.

A simple logger for Perl

     Below is a simple logger for your custom perl applications.  The logger is created as a module and can be included in any script.  Code is explained within the program as comments

Module :


safeer@penguinpower:~$ cat DEV/Perl/MyLogger.pm

#!/usr/bin/perl -w
### Module Name is MyLogger
package MyLogger;
### Include Modules
use IO::File;
use DateTime;
use File::Basename;
use File::Spec;

### Constructor

sub new
{

my $class = shift;
### Object creation parameters
my(%params) = @_;
my $self = { };
### Identify the parent scripting using the class
(undef,$self{'caller'}) = caller;
### Whether to append current date to the log file name or not
### Off by default
$self->{'datedLogFile'} = $params{'datedLogFile'}  || 0;
### Whether to include scriptname in the log entries
### Off by default
$self->{'incScriptName'} = $params{'incScriptName'}  || 0;
### Whether to include scriptname - with full path - in the log entries
### Off by default
$self->{'incScriptWithPath'} = $params{'incScriptWithPath'}  || 0;
### Whether to include current date and time in log entries
## On by default
$self->{'incDateTime'} = $params{'incDateTime'} || 1;
### logFileBase - basename from which the logfile name should be constructed
if( defined $params{'logFileBase'} ) { 
$self->{'logFile'} = $params{'logFileBase'}; 
if( $self->{'datedLogFile'} == 1 ) { 
my $currentDate = DateTime->now->strftime('-%d-%m-%Y');
$self->{'logFile'} .= $currentDate;
}
### Append ".log" to the end of log file name
$self->{'logFile'} .= ".log";
}
else { print "No logfile defined!!" ; exit 1 ; }
### Open the file and store the file object pointer in the class
$self->{'log'} = IO::File->new;
$self->{'log'}->open("$self->{'logFile'}",">>");

bless $self,$class;
return $self;
}

### This function will do the actual logging
### It is preferred not to use this function directly
### Rather call the functions derived from this one
### Like debug,info,warn,crit
sub logme()
{
my $self = shift;
### Accept two parameters, log level ( DEBUG/INFO/WARN/CRIT )
### and log message ( can be multi line message ) 
my ($logLevel,$message) = @_ ;
### Breakup multiline messages and process line by line
@logentries = split("\n" , $message  );
foreach my $logentry ( @logentries ) {
my $logprefix = "";
### Include date and time
if( $self->{'incDateTime'} == 1 ) { 
  $timestamp = DateTime->now(time_zone=>"local")->strftime("[%d-%m-%Y %H:%M:%S]");
  $logprefix .= $timestamp ;
  }
### Add log level
$logprefix .=   "[" . $logLevel ."]" ;
### Include scriptname with path
if( $self->{'incScriptWithPath'} == 1 ) {
  my $scriptpath = File::Spec->rel2abs($self{'caller'}) ;
  $logprefix .= "[" . $scriptpath . "]"; 
  }
### Include scriptname wihtout path
elsif( $self->{'incScriptName'} == 1 ) { 
  my $scriptname = basename($self{'caller'}) ;
  $logprefix .= "[" . $scriptname . "]" ;
  }

$self->{'log'}->print( "$logprefix $logentry\n" );

}

}

### Following functions are defined for the convenience of user
### All of them calls logme() funcation  internally

### Used to register informative messages
sub info()
{
my $self = shift;
$self->logme("INFO",$_[0]);
}

### Used to register warning messages
sub warn()
{
my $self = shift;
$self->logme("WARN",$_[0]);
}

### Used to register critical messages
sub crit()
{
my $self = shift;
$self->logme("CRIT",$_[0]);
}

### Used to register debug messages
sub debug()
{
my $self = shift;
$self->logme("DEBUG",$_[0]);
}

### Destroyer
sub DESTROY
{
my $self = shift;
$self->{'log'}->close;
}


1;

Now use this module in a script


safeer@penguinpower:~/DEV/Perl$ cat loguse.pl

#!/usr/bin/perl
### Define custom library path for adding modules
use lib '/home/safeer/DEV/Perl/CustomModules';
### Include the cutom logger module
use MyLogger;
### Create an instance of custom logger
my $logger = MyLogger->new('logFileBase' => "/tmp/logPerl" , 'datedLogFile' => 1 , 'incScriptName' => 1 , 'incScriptWithPath' => 1 );
### Log all types of log messages
$logger->info("Information Message");
$logger->warn("Warning Message");
$logger->crit("Critical Message");
$logger->debug("Debug Message");

Execute the script

safeer@penguinpower:~/DEV/Perl$ ./loguse.pl

Look at the log file created

safeer@penguinpower:~$ cat /tmp/logPerl-17-07-2012.log
[17-07-2012 23:26:15][INFO][/home/safeer/DEV/Perl/loguse.pl] Information Message
[17-07-2012 23:26:15][WARN][/home/safeer/DEV/Perl/loguse.pl] Warning Message
[17-07-2012 23:26:15][CRIT][/home/safeer/DEV/Perl/loguse.pl] Critical Message
[17-07-2012 23:26:15][DEBUG][/home/safeer/DEV/Perl/loguse.pl] Debug Message