Wednesday, July 18, 2012

Bash Recipes Part 4 - Read from multiple files in parallel

     I had to download a set of files for which the download URLs where provided in one file and the corresponding file names in another file.  Now to download and name the downloaded files at the same time, I have to loop through both the files in parallel, picking one URL and its name at a time and feeding it to wget.   I used while loop combined with read command for this as follows:

while read -u5 url;read -u6 name
do
/usr/bin/wget  -c  "${url}" -O "${name}"
done 5 < url.txt 6 < name.txt


    In the last line of the loop, we are redirecting the contents of files "url.txt" and "name.txt" respectively to file descriptors 5 and 6.  The while loop iterates over two read commands, each of which reads from the file descriptors 5 and 6 one line at a time and save the line into variables "url" and "name" respectively.  These variables are used with wget to download and set the name.

     Note that file descriptor numbers 0,1 & 2 are reserved for standard input, standard output and standard error and shouldn't be used.

No comments:

Post a Comment