Base64 Encoding and Decoding

     base64 is an encoding scheme used to transfer binary/text data across networks. Though initially used predominantly for transmitting data through email, base64 is used in a wide variety of of applications nowadays.

      The encoding is done by converting input data into printable ASCII characters. The input stream is divided into 6 bit segments and converted to a printable character from a set of 64 encoding characters based on the value of the 6 bits ( 6 bits can represent a value between 0 to 63   - 64 numbers ).   In practical dividing by 6 bits is done by taking 3 consecutive bytes (3 x 8 bits =  4 x 6 bits ) of the input stream and converting it to 4 encoding characters.  If the total number of bytes in the input stream is not a multiple of three, extra 0 value bytes are padded to the end of the stream to make it multiple of three.

     The decoding of the input stream is done on the receiving end by reversing the above method. 

     In Linux, the "base64" command is used to decode/encode base64.  Let us see an example.  We will convert the "ls" command binary to base64 and back and see if the resultant binary is still working as "ls".

safeer@penguinpower:~/TMP$ cp /bin/ls .
safeer@penguinpower:~/TMP$ /home/safeer/TMP/ls
ls

Convert the binary to base64

safeer@penguinpower:~/TMP$ cat /home/safeer/TMP/ls | base64 > ls.encoded

Compare both

safeer@penguinpower:~/TMP$ wc ls ls.encoded
   503   3271 104508 ls
  1834   1834 141178 ls.encoded
  2337   5105 245686 total


safeer@penguinpower:~/TMP$ head -1 ls.encoded
f0VMRgEBAQAAAAAAAAAAAAIAAwABAAAANL4ECDQAAADckwEAAAAAADQAIAAJACgAHAAbAAYAAAA0
safeer@penguinpower:~/TMP$ tail -1 ls.encoded
AAAA6JIBAPIAAAAAAAAAAAAAAAEAAAAAAAAA
safeer@penguinpower:~/TMP$ file ls.encoded
ls.encoded: ASCII text
safeer@penguinpower:~/TMP$ file ls
ls: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24, BuildID[sha1]=0x83531f308f1fa18221be53eaf399303400c14638, stripped

Now convert the encoded data back

safeer@penguinpower:~/TMP$ cat ls.encoded |base64 -d > ls.decoded
Compare with original "ls" binary

safeer@penguinpower:~/TMP$ wc ls ls.decoded
   503   3271 104508 ls
   503   3271 104508 ls.decoded
  1006   6542 209016 total
safeer@penguinpower:~/TMP$ file ls.decoded
ls.decoded: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24, BuildID[sha1]=0x83531f308f1fa18221be53eaf399303400c14638, stripped

safeer@penguinpower:~/TMP$ chmod a+x ls.decoded
safeer@penguinpower:~/TMP$ /home/safeer/TMP/ls.decoded
ls  ls.decoded    ls.encoded

Now that you know how the conversion works, let us see some other ways to decode/encode using base64.  We will use a simple string this time for simplicity.

Bash

Encoding

safeer@penguinpower:~/TMP$ printf "safeer"|base64
c2FmZWVy

Decoding

safeer@penguinpower:~/TMP$ printf "c2FmZWVy"|base64 -d
safeer

Perl

Encoding

safeer@penguinpower:~/TMP$ perl -MMIME::Base64 -e 'print encode_base64("safeer");'
c2FmZWVy

Decoding

safeer@penguinpower:~/TMP$ perl -MMIME::Base64 -e 'print decode_base64("c2FmZWVy");'
safeer

PHP

Encoding

safeer@penguinpower:~/TMP$ php -r "print base64_encode('safeer');"
c2FmZWVy

Decoding

safeer@penguinpower:~/TMP$ php -r "echo base64_decode('c2FmZWVy');"
safeer

Python

Encoding

safeer@penguinpower:~/TMP$ python -c "import base64;print base64.b64encode('safeer')"
c2FmZWVy

Decoding

safeer@penguinpower:~/TMP$ python -c "import base64;print base64.b64decode('c2FmZWVy')"
safeer

Introduction to FFMpeg

FFMpeg is an open-source utility for recording,streaming and converting digital video and audio. We will see the usage of a few functionalities to convert between multiple file formats.

In simplest form ffmpeg can be used as:

ffmpeg -i <input filename> <outputfilename>

Eg: Convert a flash (flv)video to avi format

safeer@penguinpower:~$ffmpeg -i SELinux4EveryOne.flv SELinux4EveryOne.avi

This will throw a lot of output and then start converting the video. Once the conversion is finished play the the file with a media player like vlc or mplayer.

This should play the new avi file nicely (provided mplayer is installed):

safeer@penguinpower:~$mplayer SELinux4EveryOne.avi

Now, what if we want to extract the audio track alone from the Video? This is the command.

safeer@penguinpower:~$ffmpeg -i SELinux4EveryOne.flv SELinux4EveryOne.mp3

You can also convert flv to wav or wav to mp3

safeer@penguinpower:~$ffmpeg -i SELinux4EveryOne.flv SELinux4EveryOne.wav

safeer@penguinpower:~$ffmpeg -i SELinux4EveryOne.wav SELinux4EveryOne.mp3

A wide variety of media formats are supported by ffmpeg. To get a full list, use "ffmpeg -formats"

A few useful options:

-f <format> : Usually ffmpeg can determine the output format by the extension name itself. But incase you want to force the output format, use this option.

Eg: ffmpeg -i SELinux4EveryOne.wav -f mp3 SELinux4EveryOne.mp3

-ab : Audio bitrate ( 64k is default, 128k would be a reasonable value )
-ar : Audio frequency - 44100 Kz is default
-ac : Audio channel, a value of 1 is mono, 2 is sterio
-an : Disable audio recording.
-b : Video bit rate
-r : Video frame rate
-s : Video frame size ( Screen resolution )

Any option that you give before any input/output file will apply to that file. Also remember that you can have multiple input files each specified with "-i" option and multiple output files ins some cases.

A couple of other interesting uses of ffmpeg:

Extracting images from a video:

ffmpeg -i myvideo.avi -r 1 -s svga -f image2 img-%03d.jpg

This will extract one image per second from the video for its total length. The frame size of the images will be 800*600 ( svga ). The output file name img-%03d.jpg will be automatically expanded and used in the following way: the part "%03d" will change to 3 digit numbers padded with zeros - img-001.jpg,img-002.jpg....img-031.jpg ...

Video capturing yout desktop can be done with :

ffmpeg -f x11grab -s xga -r 50 -i :0.0 /tmp/out.avi

This will capture "1024x768" sized portion from the upper left corner of the screen. The frame rate will be 50 frames per second.

There many such uses for ffmpeg. to learn more, checkout their documentation.

Downloading Youtube videos in Linux

Lots of people who watch Youtube videos like to download them for viewing it offline later. On a Linux box, the best tool you can find is a small command line utility called "youtube-dl". This is available as package in fedora/debian/ubuntu/suse repositories. Even if this tool is not in your repository, you can download it directly from the project website. This script requires python, so make sure you have it. Unzip the downloaded archive and copy it to system path (or run from there itself - if you wish).

To download a video, you need to get its Youtube URL which is the URL that you see in the address bar of your browser when you play the video. It will look like this: http://www.youtube.com/watch?v=RANDOM_STRING. The RANDOM_STRING will contain alpha numeric characters and dash.

Let us see the actual use of the tool.

safeer@penguinpower:~$youtube-dl http://www.youtube.com/watch?v=QTw6iexyrNU

It will throw some output about the URL and then start downloading, showing the current download speed and ETA. When finished, it will report "Video data saved to QTw6iexyrNU.flv". The file will be saved in the current directory. If you want to save to another location or change the name of the file, use option "-o".

The default format for the saved video will be "flv" but if you use the -b option ( for best qulity ) the format will be mp4. So change your filename accordingly when using -o and -b together.

There are many other useful options including the option to give username and password for videos that can be accessed only by logged in Yuotube users. Use "youtube-dl -h" to see all the options.

One thing that you should remember about Youtube URLs is that if the url contains one or more "&" symbols, you need to take only the part on the left hand side of the first "&". For eg, if you got this URL "http://www.youtube.com/watch?v=QTw6iexyrNU&feature=related" from your browser, then the actual URL is "http://www.youtube.com/watch?v=QTw6iexyrNU". The rest are arguments to the Youtube website which will not be used by youtube-dl.

This tool has a windows GUI which can be found here.

Now for the people who want to do things their way, following method may be interesting. Open your browser and play the video from Youtube and wait for it to finish. Once the video is played to the full length, open a terminal keeping the Youtube page open. Then run this command on terminal.

safeer@penguinpower:~$ for i in $(find /tmp -type f -user $(whoami) 2>/dev/null);do [[ $(file $i) =~ "Flash" ]] && echo $i;done

/tmp/FlashRXaVqD

This file is the video you just watched, if you want to cross check you can run this file in some media player like vlc.

safeer@penguinpower:~$ vlc /tmp/FlashRXaVqD&

And it will play the video. Now save the file with a name and flv extension.

safeer@penguinpower:~$mv /tmp/FlashRXaVqD ~/chess-play.flv

Not a big deal, but it will save your bandwidth, if you want to first play the video and then only download it.

Chrome is developing a browser extension for Youtube downloading, I haven't tested it yet. Have a look into it if you are interested.

Disclaimer: Downloading Youtube content is subject to copy right and content policies of the concerned parties, the author or this blog will not be responsible for such issues.

Typescript - Capturing your shell activity

We know about screen capturing and recording our screen activity. There are hundreds of applications for this purpose. But what about your shell activity? Of course you can use the same screen capturing tools if you are working on the GUI window. But what if you are on a remote shell? What if you want to reuse or copy the results/commands as a backup? Obviously the graphical tools will not be of much help. This is where typescripts come into play.

A typescript is a record of whatever is printed in your terminal. That includes stdin,stdout and stderr, which is basically the commands that you type and the output that these commands generate. The command which creates the typescript is "script". You can just type the command "script" and this will start recording the shell activity to a file by the name "typescript" which will be created in the directory from which the command "script" was run. Alternatively you can specify a separate file as an argument to it. When you run the script command it actually opens a new shell, and keeps recording the shell's activity until you exit that shell and return to the parent shell from where you ran the "script"command. Let us check the usage.

safeer@penguinpower:~$ script
Script started, file is typescript
safeer@penguinpower:~$ ls MyDocuments/
Linux Windows
safeer@penguinpower:~$ mkdir MyDocuments/Cisco
safeer@penguinpower:~$ ls MyDocuments/
Cisco Linux Windows
safeer@penguinpower:~$ rm MyDocuments/Cisco
rm: cannot remove `MyDocuments/Cisco': Is a directory
safeer@penguinpower:~$ exit
exit

Script done, file is typescript

Let us examine the contents of typescript.

safeer@penguinpower:~$ cat typescript
Script started on Sunday 03 January 2010 02:05:01 AM IST
safeer@penguinpower:~$ ls MyDocuments/
Linux Windows
safeer@penguinpower:~$ mkdir MyDocuments/Cisco
safeer@penguinpower:~$ ls MyDocuments/
Cisco Linux Windows
safeer@penguinpower:~$ rm MyDocuments/Cisco
rm: cannot remove `MyDocuments/Cisco': Is a directory
safeer@penguinpower:~$ exit
exit

Script done on Sunday 03 January 2010 02:05:55 AM IST

As you can see all that is printed into the terminal is recorded in the typescript. A few useful options to use with the script:

-a : Append to the end of the type script file - if it exists already. Default behavior is to overwrite the file.
-q: Quiet mode, removes the End timestamp from the typescript.
-t: This is an interesting option, it prints the output timing data to standard error. This data can be used to replay the typescript with real-time input output delays. We will see the usage later.

You can specify an alternate typescript file as the last argument.

safeer@penguinpower:~$ script myscript
Script started, file is myscript
safeer@penguinpower:~$ echo HI
HI
safeer@penguinpower:~$ exit
exit

Script done, file is myscript

safeer@penguinpower:~$ cat myscript
Script started on Sunday 03 January 2010 02:10:28 AM IST
safeer@penguinpower:~$ echo HI
HI
safeer@penguinpower:~$ exit
exit

Script done on Sunday 03 January 2010 02:10:44 AM IST

To gather the timing information, we car try using the -t option.

safeer@penguinpower:~$ script -t myscript 2>myscript.time
Script started, file is myscript
safeer@penguinpower:~$ ls MyDocuments/
Cisco/ Linux/ Windows/
safeer@penguinpower:~$ ls MyDocuments/Cisco/
safeer@penguinpower:~$ touch MyDocuments/Cisco/gns3
safeer@penguinpower:~$ ls MyDocuments/Cisco/
gns3
safeer@penguinpower:~$ exit
exit

Script done, file is myscript

Content of time file will look like this:

safeer@penguinpower:~$ head -2 myscript.time
0.500888 27
0.253943 23

The first field indicates how much time elapsed since the previous output. The second field indicates how many characters were output this time. With this information and the typescript file we can use the command "scriptreplay" to replay our terminal activity with the same timing with which you typed on the terminal and the output was displayed. It will give you a feeling of watching a video of your terminal activity.

The usage is like this:

First get the type script to one file and the timing information obtained with "-t" option ( printed to stderr) to another file.

safeer@penguinpower:~$ script -t myscript 2>myscript.time

Now replay the typescript with the format "scriptreplay [timing file] [typescript file] ".
If no typescript filename is given, the name "typescript" is assumed.

safeer@penguinpower:~$ scriptreplay myscript.time myscript

And this will play the typescript with timing.

Changing MAC address in linux

There are occasions when you may need to change your MAC (Media Access Control) address temporarily. This can be used:
1. While connected to a public wifi hot spot - it keeps your original MAC address anonymous.
2. To bypass a router/switch which has MAC address filtering.
3. To obtain a dhcp ip lease that is allocated based on MAC address.

And remember! MAC address is something that is soldered into your network adapter. You cant change it permanently; in other words, the change that you make is not persistent over reboots.

First let us find out the current MAC address of the Ethernet adapter in my machine

safeer@penguinpower:~$ ifconfig eth0|awk '/HWaddr/{print $1" "$5 }'
eth0 00:13:e4:5c:93:0a

Now I am going to change the MAC address of my Ethernet interface.

First, bring down the interface
safeer@penguinpower:~$ sudo ifconfig eth0 down

Then change the MAC addres to whatever you want
safeer@penguinpower:~$ sudo ifconfig eth0 hw ether 00:22:4d:b7:36:01

Bring up the interface
safeer@penguinpower:~$ sudo ifconfig eth0 up

Verify MAC address has changed.
safeer@penguinpower:~$ ifconfig eth0|awk '/HWaddr/{print $1" "$5 }'
eth0 00:22:4d:b7:36:01

The syntax of the command is "ifconfig [interface name] hw [encapsulation type] [MAC address] "

Every argument is self explanatory except the encapsulation type. This is "ether" (Ethernet) for both our Ethernet and wireless interfaces. For finding out encapsulation type for other interface types see ifconfig man pages. To see the encapsulation type of your network interfaces use the following command

safeer@penguinpower:~$ ifconfig -a|awk '/HWaddr/{split($3,enc,":"); print $1" "enc[2];}'
eth0 Ethernet
wlan0 Ethernet