Truncating a file

There are situations when you want to adjust the size of a file rather than deleting it. Especially when the file is being used by a process and you want to reduce its size without interrupting the process. Removing a file and creating another file by the same name will not work in such cases as the inode number of the file will change. This happens mainly with log files, where you want to reduce the size of the log - to save space -but don't want to stop the process that is writing to the log.

BSD has an excellent tool for this - the "truncate" command. This command can be used either to reduce/increase the size of a file. Let us consider a live example.

safeer->vps1.safeer.in ~ $ du -hs mysql-slowquery.log
476M mysql-slowquery.log

To reduce the size to 300MB

safeer->vps1.safeer.in ~ $ truncate -s 300M mysql-slowquery.log
safeer->vps1.safeer.in ~ $ du -hs mysql-slowquery.log
300M mysql-slowquery.log

Instead of specifying the exact size, if you just want to free up a 100Mb of disk space, you can do it this way:

safeer->vps1.safeer.in ~ $ truncate -s -100M mysql-slowquery.log
safeer->vps1.safeer.in ~ $ du -hs mysql-slowquery.log
200M mysql-slowquery.log

When you truncate a file by this method, the data is lost from the end of the file.

Truncate also has an option to extend a file's size, but it do not allocate any space for that. So it is not so useful.

safeer->vps1.safeer.in ~ $ truncate -s +400M mysql-slowquery.log
safeer->vps1.safeer.in ~ $ du -hs mysql-slowquery.log
200M mysql-slowquery.log

Another way of specifying size for truncation is to give a reference file. Truncate will calculate the size of the referenced file and truncate the given file to its length.

safeer->vps1.safeer.in ~ $ du -hs httpd.log mysql-slowquery.log
153M httpd.log
200M mysql-slowquery.log
safeer->vps1.safeer.in ~ $ truncate -r httpd.log mysql-slowquery.log
safeer->vps1.safeer.in ~ $ du -hs httpd.log mysql-slowquery.log
153M httpd.log
153M mysql-slowquery.log

truncate do not change the inode of the file.

safeer->vps1.safeer.in ~ $ ls -i mysql-slowquery.log
48980316 mysql-slowquery.log
safeer->vps1.safeer.in ~ $ truncate -s -132M mysql-slowquery.log
safeer->vps1.safeer.in ~ $ ls -i mysql-slowquery.log
48980316 mysql-slowquery.log

As you can see, this is a very useful command, but unfortunately we don't have a similar tool in Linux. So we can use any of the following method to emulate some truncate features.

To empty a file (truncate -s 0 )

safeer@penguinpower:~$ du -hs mysql-slowquery.log
1.1G mysql-slowquery.log
safeer@penguinpower:~$ >mysql-slowquery.log
safeer@penguinpower:~$ du -hs mysql-slowquery.log
0 mysql-slowquery.log

For resizing to an exact size you can use the dd command.

safeer@penguinpower:~$ du -hs mysql-slowquery.log
747M mysql-slowquery.log
safeer@penguinpower:~$ dd if=/dev/null seek=705 bs=1M of=mysql-slowquery.log
0+0 records in
0+0 records out
0 bytes (0 B) copied, 5.1892e-05 s, 0.0 kB/s
safeer@penguinpower:~$ du -hs mysql-slowquery.log
706M mysql-slowquery.log

Here dd argument bs specifies the unit of file size where as seek will specify the number of units to which you want to resize the file. Technically, copying /dev/null to a file will erase the content in the destination file. But with the "seek" option we specify that "seek value" X "block size" data from the beginning of the file should be preserved and the rest of the data should be erased. So in the above example, file was reduced to ~705MB (705 * 1M).

The following commands are equivalent.

truncate -s 0 mysql-slowquery.log
>mysql-slowquery.log
cat /dev/null > mysql-slowquery.log
dd if=/dev/null of=mysql-slowquery.log

ssh-agent and passwordless ssh

In addition to password authentication, ssh also provides public key authentication. Setup of public key authentication is described here . The summary of authentication is as follows:

* Generate a private key /public key pair:

ssh-keygen -t rsa

This will create a private key named id_rsa and public key named id_rsa.pub under ~/.ssh directory of the user. Here we are using RSA version 2 as the encryption protocol. Alternatively you can specify DSA ( -t dsa ) or RSA version 1 ( -t rsa1 ). If you want to create the key pair with an alternate name/location, specify it with -f option. The above command will prompt you for a passphrase for the private key. You can either give a passphrase or just press enter to have a blank passphrase (alternatively use -P '' option). The problem with having a blank passphrase is that anybody who has access to your private key file will be able to view and use it to impersonate your login. A passphrase for the private key prevents this from happening.

* Copy your public key to destination server and append it to ~/.ssh authorized_keys.

Once this is setup you can login to the destination server without using your account password. If you have provided a blank passphrase while generating public/private key you will be directly logged in. But if you have give a passphrase for you private key, you wil be prompted for that. While this gives a more secure way of logging in, for the end user it is just like replacing one password with another one. To get around this without compromising on security, we can use ssh-agent which will cache your password for repeated use.

ssh-agent is a key management program that can manage all our keys. We just need to unlock/decrypt our private keys - with the passphrase - once and add them into the agent, and for the rest of that session ssh-agent will automatically supply your key credentials whenever a new ssh connection is initiated.

Let us see the usage now.

* Start the agent.

[safeer@fcds ~]$ ssh-agent
SSH_AUTH_SOCK=/tmp/ssh-EQayeR4473/agent.4473; export SSH_AUTH_SOCK;
SSH_AGENT_PID=4474; export SSH_AGENT_PID;
echo Agent pid 4474;

Now that the agent is running you need to add your keys to the agent.

[safeer@fcds ~]$ ssh-add
Enter passphrase for /home/safeer/.ssh/id_rsa:
Identity added: /home/safeer/.ssh/id_rsa (/home/safeer/.ssh/id_rsa)
Identity added: /home/safeer/.ssh/id_dsa (/home/safeer/.ssh/id_dsa)

here you will prompted to enter the pasphrace for your keys. if all your private keys have the same passphrase, you will be prompted only once.

To see the keys added to ssh:

[safeer@fcds ~]$ ssh-add -l
2048 20:83:de:3d:7c:5f:a4:32:c2:75:c9:12:f6:7f:f2:be /home/safeer/.ssh/id_rsa (RSA)
1024 7a:29:5a:73:6b:56:d0:0d:73:4e:70:67:39:99:80:c8 /home/safeer/.ssh/id_dsa (DSA)

Since the keys are added to ssh-agent you can use private keys without being prompted for passphrase throughout this session.

To remove a specific key from the agent

[safeer@fcds ~]$ ssh-add -d /home/safeer/.ssh/id_rsa
Identity removed: /home/safeer/.ssh/id_rsa (/home/safeer/.ssh/id_rsa.pub)

If you want to remove all the keys, just use "ssh-add -d"

Recovering your kjots books

For those who don't know about kjots, it is a note taking utility that comes with the KDE desktop environment. It allows you to organize your notes into Books. Each book will have pages where you can actually save your notes. It has formating options like font selection,color,alignment etc. It allows you to export your books and/or pages to text/html formats. There are many other handy function like bookmarking searching etc.. It is a very handy and lightweight application.

I was using this for quite some time and had two books full of notes. One fine morning my KDE crashed leaving my system in hung state. When it came up after the reboot, I found that all the notes in Kjots were gone leaving two blank books in it. So I was sitting there scratching my head, suddenly it came to my mind that Kjots has an automatic backup facility and decided to try my luck.

All kde applications store their per user configuration and user data inside the local kde directory under the respective user's home directory. In most of the cases it will be .kde directory under the users home directory. But in KDE4 this wil change to .kde4. So to determine the kde directory for your user account you need to run the following command.

safeer@cms-netbook:~$ kde-config --localprefix
/home/safeer/.kde/

The kjots data is located in the path share/apps/kjots under ~/.kde. Lets examine the contents of this directory:

safeer@cms-netbook:~/.kde/share/apps/kjots$ ls
Oj6443.book Oj6443.book.2 Oj6443.book.4 Oj6443.book.6~ xA4048.book xA4048.book.10~ xA4048.book.3~ xA4048.book.6~ xA4048.book.8~
Oj6443.book.1 Oj6443.book.2~ Oj6443.book.4~ Oj6443.book.7~ xA4048.book.1 xA4048.book.2 xA4048.book.4~ xA4048.book.7 xA4048.book.9
Oj6443.book.1~ Oj6443.book.3 Oj6443.book.5 Oj6443.book.8~ xA4048.book.1~ xA4048.book.2~ xA4048.book.5~ xA4048.book.7~ xA4048.book.9~
Oj6443.book.10~ Oj6443.book.3~ Oj6443.book.5~ Oj6443.book.9~ xA4048.book.10 xA4048.book.3 xA4048.book.6 xA4048.book.8

Kjots by default assigns a random filename to each of your kjots books. This will not be the same as the name you have assigned to the book. Kjots stores its books in xml format. If you want to know which file corresponds to which book, use this command. Here the name I gave to this kjots book is "TrainingNotes".

safeer@cms-netbook:~/.kde/share/apps/kjots$ grep -A 1 -m 1 KJotsBook Oj6443.book.6~|grep Title|cut -d '>' -f2|cut -d '<' -f1
TrainingNotes

Kjots takes backup of your books every time you close it. In addition to the original book, kjots will keep 10 most recent backups of each book. The backup and book naming scheme is as follows.

Book name convention: .book
Eg: Oj6443.book

Backup book name: .book.~ with .book.1~ being the most recent book.
Eg: Oj6443.book.3~
Most recent backup for the above book will have name Oj6443.book.1~

Now that you know about the kjots backup scheme, let us see how I recovered my kjots. First find out the good backup of each books.

safeer@cms-netbook:~/.kde/share/apps/kjots$ ls -l
total 576
-rw------- 1 safeer safeer 0 2009-09-10 16:14 Oj6443.book
-rw------- 1 safeer safeer 0 2009-09-24 18:39 Oj6443.book.1~
-rw------- 1 safeer safeer 19774 2009-09-10 16:52 Oj6443.book.10~
-rw------- 1 safeer safeer 0 2009-09-19 16:37 Oj6443.book.2~
-rw------- 1 safeer safeer 0 2009-09-18 18:16 Oj6443.book.3~
-rw------- 1 safeer safeer 48278 2009-09-17 21:45 Oj6443.book.4~
-rw------- 1 safeer safeer 48278 2009-09-17 21:21 Oj6443.book.5~
-rw------- 1 safeer safeer 48278 2009-09-17 21:21 Oj6443.book.6~
-rw------- 1 safeer safeer 48278 2009-09-16 17:15 Oj6443.book.7~
-rw------- 1 safeer safeer 48278 2009-09-16 17:02 Oj6443.book.8~
-rw------- 1 safeer safeer 19774 2009-09-16 16:59 Oj6443.book.9~
-rw------- 1 safeer safeer 0 2009-09-24 18:39 xA4048.book
-rw------- 1 safeer safeer 0 2009-09-24 18:39 xA4048.book.1~
-rw------- 1 safeer safeer 39850 2009-09-10 16:52 xA4048.book.10~
-rw------- 1 safeer safeer 0 2009-09-19 16:37 xA4048.book.2~
-rw------- 1 safeer safeer 0 2009-09-18 18:16 xA4048.book.3~
-rw------- 1 safeer safeer 42450 2009-09-17 21:45 xA4048.book.4~
-rw------- 1 safeer safeer 42450 2009-09-17 21:21 xA4048.book.5~
-rw------- 1 safeer safeer 42450 2009-09-17 21:21 xA4048.book.6~
-rw------- 1 safeer safeer 42450 2009-09-16 17:15 xA4048.book.7~
-rw------- 1 safeer safeer 39850 2009-09-16 17:02 xA4048.book.8~
-rw------- 1 safeer safeer 39850 2009-09-16 16:59 xA4048.book.9~

As you can see, the most recent good copy for both the books were backed up on 2009-09-17 21:45 (this should be the time when I closed the Kjots on the previous day). All files after this has zero size. So copying this backup files to the corresponding books will get my books back. Make sure to close Kjots before doing this.

safeer@cms-netbook:~/.kde/share/apps/kjots$ cp Oj6443.book.4~ Oj6443.book
safeer@cms-netbook:~/.kde/share/apps/kjots$ cp xA4048.book.4~ xA4048.book

Relaunched Kjots and my books were back with all the contents!

VB script to move files based on creation date

My friend had this issue where he was supposed to move thousands of files sitting in one directory to separate directories based on their creation time.  Right now this script will only check the month and year of creation.  But this can be easily extended to consider up to seconds in creation time.  Explanation for each step is given as comments itself.

SplitAndMove.vbs

'Declare all varibles in advance
Dim curPath,srcDir,dstDir,myDir,objFSO,objFolder,objFiles,objFile

'Create a file sytem object as a base for manipulating files and directories
Set objFSO = CreateObject("Scripting.FileSystemObject")

'Process command line arguments
Select Case WScript.Arguments.Count
'If only one argument is given, take it as source directory
'then create a destination directory in the current working directory
       Case 1
       srcDir = WScript.Arguments.Item(0)
       curPath = objFSO.GetAbsolutePathName(".")
     dstDir = curPath & "\Split_Move_" & Day(Now()) & Month(Now()) & Year(Now()) & Hour(Now()) & Minute(Now()) & Second(Now())
       Case 2
'Two arguments given, first one is source directory
'Second one is destination directory
       srcDir = WScript.Arguments.Item(0)
     dstDir = WScript.Arguments.Item(1)
'In all other cases display Usage and exit script
       Case Else
       Wscript.Echo "Usage: " & Wscript.ScriptName & "<Source Directory> [<Destination Directory>]"
       WScript.Quit 1
End Select

'If source directory does not exist, display error message and exit script
If ( Not objFSO.FolderExists(srcDir) ) Then
   Wscript.Echo "Invalid Source Directory"
   WScript.Quit 1
End If

'If destination directory does not exist, create it.
If Not objFSO.FolderExists(dstDir) Then
   objFSO.CreateFolder(dstDir)
End If

'Get list of files in source directory
Set objFolder = objFSO.getFolder( srcDir )
Set objFiles = objFolder.files

'For each file in source directory, destination should be:
'a directory under destination directory,
'that directory's name should constitute month and year of the file's creation time.
For Each objFile in objFiles
'Construct destination directory name with month and year of file's creation time
myDir = dstDir & "\" & Month(objFile.DateCreated) & "_" & Year(objFile.DateCreated)
'If directory exists, move the file there.
If objFSO.FolderExists(myDir) Then
    objFSO.MoveFile objFile.Path , myDir & "\" & objFile.Name
'Otherwise create the directory and then move the file there.
Else
   objFSO.CreateFolder(myDir)
   objFSO.MoveFile objFile.Path , myDir & "\" & objFile.Name
End If
Next

Dashes and asterisks in linux filenames

This question was asked in an interview: "How do you delete a file that contain a leading dash/hyphen (-) in its filename ? ".

Well, in real time we rarely encounter such a scenario. But the fact is that dash is allowed to be used in a filename , but we don't usually use it at the beginning of the name. Now the problem with such a filename is that if you try to remove the file with rm command, rm will assume that the letters following the dash are command line options for the command and will throw a syntax either about a wrong option or missing operand. So the solution is to "escape" the dash at the beginning, but how can we do that? The first option is to use "--" which indicates that there are no more command line options to be read and the arguments following it should be considered as operands.

Let us try with an example;

safeer@enjoyfast-lx:~/temp$ ls
-file1
safeer@enjoyfast-lx:~/temp$ rm -file1
rm: invalid option -- 'l'
Try `rm ./-file1' to remove the file `-file1'.
Try `rm --help' for more information.

As you can see it is giving an invalid option error. Let us use the option terminator.

safeer@enjoyfast-lx:~/temp$ rm -- -file1
safeer@enjoyfast-lx:~/temp$ ls
safeer@enjoyfast-lx:~/temp$

By the way, did you notice the second line in the error message given above? Yeah, rm itself is suggesting a solution for your problem, use "./" . Giving an absolute or relative path before the filename prevents rm from interpreting the filename as an option to the command. Let us try it.

safeer@enjoyfast-lx:~/temp$ ls
-file1
safeer@enjoyfast-lx:~/temp$ rm ./-file1
safeer@enjoyfast-lx:~/temp$ ls
safeer@enjoyfast-lx:~/temp$

Now that solves the problem, but how do we create a a file with dash in name at the first place? We will have the same problem as rm with any command for creating a file, like touch,vi etc. We can use the same tricks as above here

safeer@enjoyfast-lx:~/temp$ touch -file1
touch: invalid option -- 'i'
Try `touch --help' for more information.
safeer@enjoyfast-lx:~/temp$ touch -- -file3
safeer@enjoyfast-lx:~/temp$ touch ./-file4
safeer@enjoyfast-lx:~/temp$ ls -l
total 0
-rw-r--r-- 1 safeer safeer 0 2009-07-25 00:01 -file3
-rw-r--r-- 1 safeer safeer 0 2009-07-25 00:01 -file4

Okay, now you have two files, how ill you delete them together? Let us try wild-card

safeer@enjoyfast-lx:~/temp$ rm *
rm: invalid option -- 'l'
Try `rm ./-file3' to remove the file `-file3'.
Try `rm --help' for more information.

This is because "*" expands to "-file3 -file4".So use the same trick again...

safeer@enjoyfast-lx:~/temp$ rm ./*
safeer@enjoyfast-lx:~/temp$ ls
safeer@enjoyfast-lx:~/temp$
safeer@enjoyfast-lx:~/temp$ touch ./-file{3,4}
safeer@enjoyfast-lx:~/temp$ ls
-file3 -file4
safeer@enjoyfast-lx:~/temp$ rm -- *
safeer@enjoyfast-lx:~/temp$ ls
safeer@enjoyfast-lx:~/temp$

Okay, enough fun with dash. Let us try asterisk ( * ) now.

Create a file with name "*" first.

safeer@enjoyfast-lx:~/temp$ touch *
safeer@enjoyfast-lx:~/temp$ ls
*
safeer@enjoyfast-lx:~/temp$ echo "Star" > *;echo "First" > file1;echo "Second" > file2
safeer@enjoyfast-lx:~/temp$ ls
* file1 file2

Now let us print the content of all the files.

safeer@enjoyfast-lx:~/temp$ cat *
Star
First
Second

safeer@enjoyfast-lx:~/temp$ cat * file1 file2
Star
First
Second
First
Second

In both the above cases * expands to the file list "* file1 file2". Now to print the contents of the file "*" let us try the same mechanisms.

safeer@enjoyfast-lx:~/temp$ cat -- *
Start
First
Second
safeer@enjoyfast-lx:~/temp$ cat ./*
Start
First
Second

It is not working because * will always expand unless it is escaped.

safeer@enjoyfast-lx:~/temp$ cat \*
Start

You can delete files the same way

safeer@enjoyfast-lx:~/temp$ rm \*
safeer@enjoyfast-lx:~/temp$ ls
file1 file2