Those who are familiar with programming languages like C/C++/Perl/PHP etc know about importing functions and variables from other file into the program. This enables the programmer to consolidate reusable code into a file and share it between multiple programs.
In a similar way, bash is also capable of loading functions and variables from other programs. It is a lot less used in bash compared to other languages, but there are a couple of places that we know off where bash uses this feature. One of the is the /etc/profile file.
There are three ways of doing this. Let us assume that our functions and variables are in a file called mydefs.inc. To include it in your code, say myscript.sh, just add one of the following to the top of myscript.sh
. ./mydefs.inc
source ./mydefs.inc
I prefer the second form as putting a period before the script may not be noticeable always. The filename should be present in your path if you are giving the filename without a path. Else you should provide a rlative or absolute path. The include file is just like any other shell script, so it should have proper executable permissions set.
Now a quick example:
safeer@penguinpower:~/DEV$ cat calculator.inc
#!/bin/bash
function sumOf()
{
echo $(($1 + $2));
}
safeer@penguinpower:~/DEV$ cat calculator.inc
#!/bin/bash:~/DEV$ cat testsum.sh
#!/bin/bash
source ${PWD}/calculator.inc
echo $(sumOf 1 2)
safeer@penguinpower:~/DEV$ ./testsum.sh
3
I defined the function sumOf in calulator.inc file and included it in testsum.sh script at the beginning and then called the function with two arguments. As expected the function returned the result of 1+2.
Sunday, November 13, 2011
Friday, September 9, 2011
Bash Recipes Part 2 - Process a string character by character
For splitting a string character by character, or doing any sort of operation per each character use the following code snippet
safeer@penguinpower:~$ NAME=safeer;for((i=0;i<${#NAME};i++)); do echo ${NAME:$i:1}; done
s
a
f
e
e
r
${#NAME} is the length of the variable $NAME
${NAME:$i:1} - This is a bash parameter substitution technique. U can extract a substring of a variable by using the syntax "${parameter:offset:length}" - This expands to up to "length" characters of parameter starting at the character specified by "offset" - The offset starts at zero.
In our case we use $i as the offset and in every iteration of the for loop we increment $i from zero to length of the string minus one ( which is the list of offsets for every single character in the string ). The length of substring is always 1 ( hence single character ). So ultimately per every iteration we extract one character from the string advancing from the beginning to the end.
Now if you want to extract the character in the reverse order, start the index from -1 and down to negative of string length.
safeer@penguinpower:~$ A=safeer;for((i=-1;i>=-${#A};i--)); do echo ${A:$i:1}; done
r
e
e
f
a
s
Thursday, June 16, 2011
Configure mutt as IMAP client
Mutt is a lightweight yet powerful command line email client. We will look at a minimal configuration for an IMAP account here.
Mutt has a global configuration file in either /etc/Muttrc or /usr/local/share/mutt/Muttrc. This is useful for configuring system wide settings. But to configure your own account you have to configure your personalized muttrc file in your home directory at ~/.muttrc.
Here is a sample muttrc file that can get yo start off with mutt. Explanations are given as comments.
set folder=imaps://my.imap.server.name:imapPort/
## Where to look for the mails
## the "+" simbol means, append this to the value of "folder" setting
set spoolfile=+Inbox
## Your imap username
set imap_user=my_imap_user
## Set your imap password - Not preferred for security reasons
## If not provided mutt will prompt you for a password
set imap_pass=my_imap_pass
## Setting sent folder ( note the "+" sign )
set record=+Sent
## Setting draft folder ( note the "+" sign )
set postponed=+Drafts
## To cache downloaded header
## Good for performance
## directory should be manually created
set header_cache=~/.mutt/cache/headers
set message_cachedir=~/.mutt/cache/bodies
## Your preferred editor for composing
set editor=vim
## How to sort your mail
set sort='last-date-received'
Mutt has a global configuration file in either /etc/Muttrc or /usr/local/share/mutt/Muttrc. This is useful for configuring system wide settings. But to configure your own account you have to configure your personalized muttrc file in your home directory at ~/.muttrc.
Here is a sample muttrc file that can get yo start off with mutt. Explanations are given as comments.
## Set your imap server name and port here.
set folder=imaps://my.imap.server.name:imapPort/
## Where to look for the mails
## the "+" simbol means, append this to the value of "folder" setting
set spoolfile=+Inbox
## Your imap username
set imap_user=my_imap_user
## Set your imap password - Not preferred for security reasons
## If not provided mutt will prompt you for a password
set imap_pass=my_imap_pass
## Setting sent folder ( note the "+" sign )
set record=+Sent
## Setting draft folder ( note the "+" sign )
set postponed=+Drafts
## To cache downloaded header
## Good for performance
## directory should be manually created
set header_cache=~/.mutt/cache/headers
set message_cachedir=~/.mutt/cache/bodies
## Your preferred editor for composing
set editor=vim
## How to sort your mail
set sort='last-date-received'
Sunday, March 20, 2011
A quick primer to GNU Screen - The terminal multiplexer
Screen is a terminal multiplexer that allows you to open up multiple virtual terminal sessions inside a single terminal. This enables the user to run multiple programs from a single terminal independently and in parallel. It also allows switching between them and scrolling through the history as if you were using real independent terminal.
Now this may not be an attraction in today's Linux desktop environment where you can open up multiple tabs of the terminal program you want to use ( like gnu-terminal, kde konsole etc). But imagine the case where your local machine don’t have GUI installed or you are connected to a remote host's terminal via ssh connection. In the case of local machines terminal, there is a limit to the number of terminals a system can have. In the case of ssh connection, you can open multiple connections but it takes up network and to manage them you need to open up an equal number of local terminals. Screen solves this limitation by allowing to spawn virtual shell within a single shell and easily navigate between them.
This is once advantage of screen but the best part is yet to come. Any program that you run from a shell is usually spawned as a child of the shell/terminal program. Hence terminating the shell/terminal session will kill any child process running in it. Screen has the ability to detach itself from the parent shell process and hook itself as a child of the init process. This makes screen resilient to logout ( for local terminals ) and network disconnects ( for remote terminals ). This is a great tool for a power user.
Now let us see how we can use screen.
First install the screen program using your favourite package manager or download from the screen website.
Simplest invocation of screen is by just typing the command "screen" on your terminal (local/remote). You might be shown an optional welcome message, press "Enter" to get rid of the message and drop tot he terminal. This will start the first virtual terminal session and will look like a regular terminal. You can run any commands/programs you want from this terminal. You can finish your work on the terminal and type exit or press Control+D as you do on a regular terminal to exit out of screen. But this do not add any value to our terminal usage.
To make the most out of screen, one should know how to spawn additional virtual terminals, detach the screen session allowing the running program to continue independently and browse through the numerous virtual terminals you open.
To do all these task you have to sent special commands otherwise known as "key bindings" to the screen application, and all of them start with an escape sequence, followed by one or more characters. The default escape sequence is "Control+a" ( press control key followed by character a ). By default screen will pass any input to the current virtual terminal, but sending an escape sequence will make screen aware that it is a special command. Let us see some useful screen commands/key bindings.
Control+a c ( Press "Control", then "a" followed by "c" ) - Open up a new virtual terminal ( hereafter called "VT") in screen.
You can create any number of VTs in this fashion, each screen will have a number starting from zero, zero being the number of the VT opened when screen is started.
Control+a " - List all VTs.
An example screen session with 13 VTs ( 0-13) is given below. There will be a selection bar on the VT you are currently on. You can use up/down arrow keys to select a different terminal or press the terminal number to select it ( this work only for terminals from 0-9 ). Pressing "Enter" will activate the selected VT.
Num Name
0 bash
1 bash
2 bash
3 bash
4 bash
5 bash
6 bash
7 bash
8 bash
9 bash
10 bash
11 bash
12 bash
Navigation commands
Control+a p - Move to previous VT ( "Control+a Backspace" can also be used )
Control+a n - Move to next VT ( "Control+a Space" can also be used )
Control+a 0-9 - Move to the numbered VT ( works only for terminals 0-9 )
Control+a Control+a - Toggle between current and previous window
Screen Command Prompt
Screen has a command prompt from which you can input commands for screen.
Control+a : - Activate the command prompt for the current VT
Providing "title TITLE_NAME" from the screen command prompt will rename the current VT (same as control+a A )
Typing help at the prompt will show you a full list of helpful commands.
Usability commands
Control+a C - Clear screen
Control+a Esc OR Control+a [ - Activate copy mode ( in copy mode you can scroll up the output , select and copy text).
Enter to copy, Control+a ] to paste
Control+a ? - Show help/key bindings
Control+a w - Show a list of virtual terminals ( displayed at the bottom of current VT, no option to select VTs )
Control+a M - Monitor for activity
control+a _ - Monitor for silence more than 30 seconds
Control+a A - Rename the current VT.
As you can see, each of these VTs have a number as well as a name. While the number increases for each VT, the name by default is the name of the shell/command spawned. While the numbers are convenient, you might lose track of which terminal is doing what. Giving it a meaningful name would help you identify the VT. When this command is given, you will be given a prompt with the default name already in it, erase it with backspace, type in your new name and press enter.
A sample screen session with three renamed VTs
Num Name
0 Mutt Email Client
1 Perl Scripting
2 Screen 3
Detach / Attach
Now that you know how to operate within a screen session, let us see how to detach/attach to en existing screen session.
Detach : You are running programs under multiple VTs inside a screen session. Now you want to disconnect from this screen for some reason ( logging out, terminating remote ssh connection etc.. ) and still let the programs under the VTs to continue. This is where you should use the detach option, which is activated by the key binding "Catrl+a d". When the key combination is pressed, screen drops you off to the shell from where you initiated the screen session. Now you can logout / disconnect and the programs will keep running.
This is the sample output you get when you detach from a screen session.
[detached from 9108.maverick]
Now to connect back, you should know the screen session's name ( do not confuse this with the VT names inside the screen session ). To list all current screen sessions, issue the command "screen -ls"
safeer@maverick:~$ screen -ls
There are screens on:
9108.pts-0.maverick (Sunday 20 March 2011 03:49:35 IST) (Detached)
1 Socket in /var/run/screen/S-safeer.
There is only one screen session open right now and the name is 9108.pts-0.maverick. The format of the screen name is [PID of screen].[TTY Numebr].[hostname]. This is the same name you saw when you where detatching the screen session.
To re attach to the disconnected session, use "screen -r <SCREEN NAME>", ie screen -r 9108.pts-0.maverick.
When re attaching the PID part is optional and can be reattached this way : "screen -r pts-0.maverick"
With a couple of screen sessions, it is easier to track which screen is what, but not always. So we can custom name the screen sessions, for that start the screen session with "screen -R <screen name>". This name replaces the [TTY].[hostname] part, but the pid part will remain. Since PID is optional when reattaching the screen, you can provide just this screen name while reattaching. See an
safeer@maverick:/var/run/screen$ screen -ls
There are screens on:
21454.SAFEER_TEST (Sunday 20 March 2011 04:03:52 IST) (Attached)
9108.pts-0.maverick (Sunday 20 March 2011 03:49:35 IST) (Detached)
2 Sockets in /var/run/screen/S-safeer.
You can re-attach to an attached session active in one terminal from another terminal after detaching it from the current terminal.
Either do it in two steps as
screen -d <SCREEN NAME>
screen -r <SCREEN NAME>
or together
screen -dr <SCREEN NAME>
Multi user mode
Multi user mode is one speciality of screen where the same or different users in a system can share the same screen session. You can control the access using ACLs, which allows restrictions based on usernames, VTs, commands and read/write/execute modes.
To enable multi user mode, we will have to use the screen command prompt invoked by "Control+a :" which will give you a prompt. Enter "multiuser on" in this prompt and multi user access will be enabled for this screen session. Once enabled, this screen session can be accessed by other users from different terminals using the command "screen -x <screen name>"
A complete discussion of all multi user mode is out of the scope of this article, so I will just list the commands that can be used inside screen command prompt.
multiuser on - Enable multi user mode.
display - Show currently connected users
acladd - add screen access for one or more users
acldel - revoke screen access for one or more users
aclchg - Add/modify permissions ( for commands,VTS,read/write/exec ) to one or more users
To know more about screen, checkout the man page and the official website of screen.
Now this may not be an attraction in today's Linux desktop environment where you can open up multiple tabs of the terminal program you want to use ( like gnu-terminal, kde konsole etc). But imagine the case where your local machine don’t have GUI installed or you are connected to a remote host's terminal via ssh connection. In the case of local machines terminal, there is a limit to the number of terminals a system can have. In the case of ssh connection, you can open multiple connections but it takes up network and to manage them you need to open up an equal number of local terminals. Screen solves this limitation by allowing to spawn virtual shell within a single shell and easily navigate between them.
This is once advantage of screen but the best part is yet to come. Any program that you run from a shell is usually spawned as a child of the shell/terminal program. Hence terminating the shell/terminal session will kill any child process running in it. Screen has the ability to detach itself from the parent shell process and hook itself as a child of the init process. This makes screen resilient to logout ( for local terminals ) and network disconnects ( for remote terminals ). This is a great tool for a power user.
Now let us see how we can use screen.
First install the screen program using your favourite package manager or download from the screen website.
Simplest invocation of screen is by just typing the command "screen" on your terminal (local/remote). You might be shown an optional welcome message, press "Enter" to get rid of the message and drop tot he terminal. This will start the first virtual terminal session and will look like a regular terminal. You can run any commands/programs you want from this terminal. You can finish your work on the terminal and type exit or press Control+D as you do on a regular terminal to exit out of screen. But this do not add any value to our terminal usage.
To make the most out of screen, one should know how to spawn additional virtual terminals, detach the screen session allowing the running program to continue independently and browse through the numerous virtual terminals you open.
To do all these task you have to sent special commands otherwise known as "key bindings" to the screen application, and all of them start with an escape sequence, followed by one or more characters. The default escape sequence is "Control+a" ( press control key followed by character a ). By default screen will pass any input to the current virtual terminal, but sending an escape sequence will make screen aware that it is a special command. Let us see some useful screen commands/key bindings.
Control+a c ( Press "Control", then "a" followed by "c" ) - Open up a new virtual terminal ( hereafter called "VT") in screen.
You can create any number of VTs in this fashion, each screen will have a number starting from zero, zero being the number of the VT opened when screen is started.
Control+a " - List all VTs.
An example screen session with 13 VTs ( 0-13) is given below. There will be a selection bar on the VT you are currently on. You can use up/down arrow keys to select a different terminal or press the terminal number to select it ( this work only for terminals from 0-9 ). Pressing "Enter" will activate the selected VT.
Num Name
0 bash
1 bash
2 bash
3 bash
4 bash
5 bash
6 bash
7 bash
8 bash
9 bash
10 bash
11 bash
12 bash
Navigation commands
Control+a p - Move to previous VT ( "Control+a Backspace" can also be used )
Control+a n - Move to next VT ( "Control+a Space" can also be used )
Control+a 0-9 - Move to the numbered VT ( works only for terminals 0-9 )
Control+a Control+a - Toggle between current and previous window
Screen Command Prompt
Screen has a command prompt from which you can input commands for screen.
Control+a : - Activate the command prompt for the current VT
Providing "title TITLE_NAME" from the screen command prompt will rename the current VT (same as control+a A )
Typing help at the prompt will show you a full list of helpful commands.
Usability commands
Control+a C - Clear screen
Control+a Esc OR Control+a [ - Activate copy mode ( in copy mode you can scroll up the output , select and copy text).
Enter to copy, Control+a ] to paste
Control+a ? - Show help/key bindings
Control+a w - Show a list of virtual terminals ( displayed at the bottom of current VT, no option to select VTs )
Control+a M - Monitor for activity
control+a _ - Monitor for silence more than 30 seconds
Control+a A - Rename the current VT.
As you can see, each of these VTs have a number as well as a name. While the number increases for each VT, the name by default is the name of the shell/command spawned. While the numbers are convenient, you might lose track of which terminal is doing what. Giving it a meaningful name would help you identify the VT. When this command is given, you will be given a prompt with the default name already in it, erase it with backspace, type in your new name and press enter.
A sample screen session with three renamed VTs
Num Name
0 Mutt Email Client
1 Perl Scripting
2 Screen 3
Detach / Attach
Now that you know how to operate within a screen session, let us see how to detach/attach to en existing screen session.
Detach : You are running programs under multiple VTs inside a screen session. Now you want to disconnect from this screen for some reason ( logging out, terminating remote ssh connection etc.. ) and still let the programs under the VTs to continue. This is where you should use the detach option, which is activated by the key binding "Catrl+a d". When the key combination is pressed, screen drops you off to the shell from where you initiated the screen session. Now you can logout / disconnect and the programs will keep running.
This is the sample output you get when you detach from a screen session.
[detached from 9108.maverick]
Now to connect back, you should know the screen session's name ( do not confuse this with the VT names inside the screen session ). To list all current screen sessions, issue the command "screen -ls"
safeer@maverick:~$ screen -ls
There are screens on:
9108.pts-0.maverick (Sunday 20 March 2011 03:49:35 IST) (Detached)
1 Socket in /var/run/screen/S-safeer.
There is only one screen session open right now and the name is 9108.pts-0.maverick. The format of the screen name is [PID of screen].[TTY Numebr].[hostname]. This is the same name you saw when you where detatching the screen session.
To re attach to the disconnected session, use "screen -r <SCREEN NAME>", ie screen -r 9108.pts-0.maverick.
When re attaching the PID part is optional and can be reattached this way : "screen -r pts-0.maverick"
With a couple of screen sessions, it is easier to track which screen is what, but not always. So we can custom name the screen sessions, for that start the screen session with "screen -R <screen name>". This name replaces the [TTY].[hostname] part, but the pid part will remain. Since PID is optional when reattaching the screen, you can provide just this screen name while reattaching. See an
safeer@maverick:/var/run/screen$ screen -ls
There are screens on:
21454.SAFEER_TEST (Sunday 20 March 2011 04:03:52 IST) (Attached)
9108.pts-0.maverick (Sunday 20 March 2011 03:49:35 IST) (Detached)
2 Sockets in /var/run/screen/S-safeer.
You can re-attach to an attached session active in one terminal from another terminal after detaching it from the current terminal.
Either do it in two steps as
screen -d <SCREEN NAME>
screen -r <SCREEN NAME>
or together
screen -dr <SCREEN NAME>
Multi user mode
Multi user mode is one speciality of screen where the same or different users in a system can share the same screen session. You can control the access using ACLs, which allows restrictions based on usernames, VTs, commands and read/write/execute modes.
To enable multi user mode, we will have to use the screen command prompt invoked by "Control+a :" which will give you a prompt. Enter "multiuser on" in this prompt and multi user access will be enabled for this screen session. Once enabled, this screen session can be accessed by other users from different terminals using the command "screen -x <screen name>"
A complete discussion of all multi user mode is out of the scope of this article, so I will just list the commands that can be used inside screen command prompt.
multiuser on - Enable multi user mode.
display - Show currently connected users
acladd - add screen access for one or more users
acldel - revoke screen access for one or more users
aclchg - Add/modify permissions ( for commands,VTS,read/write/exec ) to one or more users
To know more about screen, checkout the man page and the official website of screen.
Sunday, March 13, 2011
Block device encryption and management in Linux
The standard for Linux block level encryption is LUKS ( Linux Unified Key Setup ). DM-Crypt is the Linux kernel's device mapper module for transparently encrypting and decrypting block devices. To make use of these features, install cryptsetup package.
safeer@lin01:~$yum install cryptsetup-luks
First, format the LUKS encryption layer (on the partition). This is for standardizing the partition header and the format of the bulk data.
safeer@lin01:~$sudo /sbin/cryptsetup luksFormat /dev/sdb1
WARNING!
========
This will overwrite data on /dev/sdb1 irrevocably.
Are you sure? (Type uppercase yes): yes
Enter LUKS passphrase:
Verify passphrase:
Command successful.
The command will ask for a passphrase, make sure you remember the passphrase you provide, as this will be later used to mount the partition. Optionally you can directly provide a filename containing the passphrase as argument of above command with "--key-file <file-name>".
Open LUKS encryption layer ( device map the encrypted disk )
safeer@lin01:~$sudo /sbin/cryptsetup luksOpen /dev/sdb1 secretDisk
Enter passphrase for /dev/sdb1:
Provide the password you created during the first setup. Once authenticated, /dev/sdb1 will be devise mapped to the name you provided in the last command ( secretDisk ). To verify, run:
safeer@lin01:~$ls -l /dev/mapper/secretDisk
lrwxrwxrwx 1 root root 7 Apr 14 11:10 /dev/mapper/secretDisk -> ../dm-0
Now format the device mapper with appropriate files system, here am chosing ext3.
safeer@lin01:~$sudo /sbin/mkfs.ext3 /dev/mapper/secretDisk
I want to mount this device to /home/safeer/secretDisk. To mount it permanently, add following line to /etc/fstab
/dev/mapper/secretDisk /home/safeer/secretDisk ext3 defaults 0 0
safeer@lin01:~$sudo mount -a
df -h /home/safeer/secretDisk
During boot up, the devise mapping of encrypted disk should happen prior to mounting, this is achieved by registering the device in the file /etc/crypttab
For each crypt device, there should be one line in this file. Aminimum of two fileds is necessary here, first is the mapper name without "/dev/mapper" prefix and the second is the actual encrypted block device. The third field is passphrase file and if left empy or the value "none" is given, user will be prompted for the passphrase prior to mounting. If this has to be automated, create a file with the passphrase ( non newline character at the end) and provide the file name as the third field. The fourth field is options, which can be used to speficy encryption algorithms etc... This can be left blank in most cases.
safeer@lin01:~$grep secret /etc/crypttab
secretDisk /home/safeer/secretDisk none
To ensure things are working properly, reboot the host once.
A couple of things to know if you want more advansed setup.
LUKS allows having 8 passphrases for a partition, this means 8 users can mount the filesystem with passphrases of their own choice and eliminates the need to share a common passphrase. Check out the cryptsetup man page for options luksAddKey,luksRemoveKey and luksDump.
safeer@lin01:~$yum install cryptsetup-luks
First, format the LUKS encryption layer (on the partition). This is for standardizing the partition header and the format of the bulk data.
safeer@lin01:~$sudo /sbin/cryptsetup luksFormat /dev/sdb1
WARNING!
========
This will overwrite data on /dev/sdb1 irrevocably.
Are you sure? (Type uppercase yes): yes
Enter LUKS passphrase:
Verify passphrase:
Command successful.
The command will ask for a passphrase, make sure you remember the passphrase you provide, as this will be later used to mount the partition. Optionally you can directly provide a filename containing the passphrase as argument of above command with "--key-file <file-name>".
Open LUKS encryption layer ( device map the encrypted disk )
safeer@lin01:~$sudo /sbin/cryptsetup luksOpen /dev/sdb1 secretDisk
Enter passphrase for /dev/sdb1:
Provide the password you created during the first setup. Once authenticated, /dev/sdb1 will be devise mapped to the name you provided in the last command ( secretDisk ). To verify, run:
safeer@lin01:~$ls -l /dev/mapper/secretDisk
lrwxrwxrwx 1 root root 7 Apr 14 11:10 /dev/mapper/secretDisk -> ../dm-0
Now format the device mapper with appropriate files system, here am chosing ext3.
safeer@lin01:~$sudo /sbin/mkfs.ext3 /dev/mapper/secretDisk
I want to mount this device to /home/safeer/secretDisk. To mount it permanently, add following line to /etc/fstab
/dev/mapper/secretDisk /home/safeer/secretDisk ext3 defaults 0 0
safeer@lin01:~$sudo mount -a
df -h /home/safeer/secretDisk
During boot up, the devise mapping of encrypted disk should happen prior to mounting, this is achieved by registering the device in the file /etc/crypttab
For each crypt device, there should be one line in this file. Aminimum of two fileds is necessary here, first is the mapper name without "/dev/mapper" prefix and the second is the actual encrypted block device. The third field is passphrase file and if left empy or the value "none" is given, user will be prompted for the passphrase prior to mounting. If this has to be automated, create a file with the passphrase ( non newline character at the end) and provide the file name as the third field. The fourth field is options, which can be used to speficy encryption algorithms etc... This can be left blank in most cases.
safeer@lin01:~$grep secret /etc/crypttab
secretDisk /home/safeer/secretDisk none
To ensure things are working properly, reboot the host once.
A couple of things to know if you want more advansed setup.
LUKS allows having 8 passphrases for a partition, this means 8 users can mount the filesystem with passphrases of their own choice and eliminates the need to share a common passphrase. Check out the cryptsetup man page for options luksAddKey,luksRemoveKey and luksDump.
Subscribe to:
Posts (Atom)