FTP macros for automated ftp tasks

FTP macros are nothing but an ordered set of ftp commands grouped with a macro name, similar to a bash function/script. You can use the macro name like another command and execute the commands included with it. You can add a macro while inside an ftp session using the macdef command. Alternatively you can add it into the .netrc file inside your home directory. The usage of .netrc file is explained here.

Let us try creating a macro inside an ftp session. This macro will just change to a directory inside the ftp root and copy a file to local directory.

Each macro starts with a macro name on first line, followed by single ftp command on each line. The macro termination is given by a blank line.

ftp> macdef getmacro
Enter macro line by line, terminating it with a null line
cd /home/safeer/pub/site_backup/
get webdata.tar.gz


Once you create the macro run it from the ftp prompt with $ prepended to the macro name.
ftp> $getmacro
cd /home/safeer/pub/site_backup/
250 Directory successfully changed.
get webdata.tar.gz
local: webdata.tar.gz remote: webdata.tar.gz
200 PORT command successful. Consider using PASV.
150 Opening BINARY mode data connection for webdata.tar.gz (0 bytes).
226 File send OK.

A macro defined in an ftp session remains active till the close command is issued. If you want re-usable macros, you need to add them to your netrc file. The above mentioned link has the details on how to set it up. A macro defined by the name "init" (in netrc file) will be run every time you login to the corresponding ftp host.

Macro Arguments

If you want to take arguments inside your macro, use $1,$2,$3 etc... where $1 will be the first argument, $2 will be the second argument and so on. If you want to iterate through each argument of the macro, use the special variable $i. The characters "$" and "\" are special in macros and need to be escaped with "\" if you want them "as it is" in the macros.

Eg:

ftp>macdef 2args
Enter macro line by line, terminating it with a null line
cd /home/safeer/docs
get $1
put $2

Above macro will now download local.txt and upload remote.txt

ftp>$2args local.txt remote.txt
cd /home/safeer/docs
250 Directory successfully changed.
get local.txt
local: local.txt remote: file1.txt
200 PORT command successful. Consider using PASV.
150 Opening BINARY mode data connection for local.txt (47 bytes).
226 File send OK.
47 bytes received in 0.00 secs (91.1 kB/s)
put remote.txt
local: remote.txt remote: remote.txt
200 PORT command successful. Consider using PASV.
150 Ok to send data.
226 File receive OK.
47 bytes sent in 0.00 secs (454.4 kB/s)

ftp>macdef getmulti
Enter macro line by line, terminating it with a null line
cd /home/safeer/docs
get $i

Above macro will now change to the directory docs in ftp root and recursively download the files given as arguments to macro

ftp> $getmulti file1.txt file2.txt file3.txt
cd /home/safeer/docs
250 Directory successfully changed.
get file1.txt
local: file1.txt remote: file1.txt
200 PORT command successful. Consider using PASV.
150 Opening BINARY mode data connection for file1.txt (47 bytes).
226 File send OK.
47 bytes received in 0.00 secs (82.3 kB/s)
cd /home/safeer/docs
250 Directory successfully changed.
get file2.txt
local: file2.txt remote: file2.txt
200 PORT command successful. Consider using PASV.
150 Opening BINARY mode data connection for file2.txt (47 bytes).
226 File send OK.
47 bytes received in 0.00 secs (118.6 kB/s)
cd /home/safeer/docs
250 Directory successfully changed.
get file3.txt
local: file3.txt remote: file3.txt
200 PORT command successful. Consider using PASV.
150 Opening BINARY mode data connection for file3.txt (47 bytes).
226 File send OK.
47 bytes received in 0.00 secs (154.5 kB/s)
There is an maximum 8 character limit for macro names defined on command line. In netrc file this restriction is not there. Per ftp session, there is a limit of 16 macros and 4096 total characters in all defined macros.

Automating ftp logins with netrc

If you are a regular ftp user, then netrc file can help you a lot in automating your logins and a few tasks you do via ftp.

Let us first consider automating ftp logins.

Create a ".netrc" file in your home directory. The access to this file should be restricted exclusively to you.

cd ~;touch .netrc;chmod u+rw,go= .netrc

Now assume you need to connect to an ftp server "myftp.safeer.in" with username 'safeer' and password 'password#123'. To automate this, add the following line to your .netrc file.

machine myftp.safeer.in login safeer password password#123

Here machine defines the host you want to login to, login is your username for the host and password is the password for this username.

safeer@my-lptp01:~$ ftp myftp.safeer.in
Connected to myftp.safeer.in.
220 (vsFTPd 2.0.7)
331 Please specify the password.
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp>

You can define a line for each of the host you want to login. If you use the same login for many of the ftp hosts, you can use the default entry like this.

default login safeer password password#123

With this entry, if you connect to an ftp host which is not explicitly defined in netrc with a "machine" entry, the login and password from the default line will be used. If you want to disable the autologin feature of netrc while running ftp program, use the option "-n", ie; "ftp -n [ftphost]"

safeer@my-lptp01:~$ ftp -n myftp.safeer.in
Connected to myftp.safeer.in.
220 (vsFTPd 2.0.7)
ftp> pwd
530 Please login with USER and PASS.

Thats all about auto logins, now let us see how to automate tasks. We use ftp "macros" for this. FTP macros are nothing but an ordered set of ftp commands grouped with a macro name, similar to a bash function/script. You can use the macro name like another command and execute the commands included with it.

Creating a macro with name "fmacro"

macdef fmacro
cd /home/safeer/pub/site_backup
put webdata.tar.gz

Now, run the macro:

ftp> $fmacro
cd /home/safeer/pub/site_backup
250 Directory successfully changed.
put webdata.tar.gz
local: webdata.tar.gz remote: webdata.tar.gz
200 PORT command successful. Consider using PASV.
150 Ok to send data.
226 File receive OK.
ftp>

A macro defined by the name "init" (in netrc file) will be run every time you login to the corresponding ftp host. Each macro definition is associated with the login entry preceding it and terminated by a blank line. You can't define global macros in netrc file. Macros associated with the" default" entry will work for all machines that are not explicitly defined in the file. To know more about ftp macros, visit this link.

For each "machine" line you can use one or more of the variables machine,login,password in order. ie: any of the following will work.

machine myftp.safeer.in login safeer password password#123 OR
machine myftp.safeer.in login safeer OR
machine myftp.safeer.in

The last option will be used only if you want to define some macros for this host, but don't want the autologin feature. While defining the default entry, make sure that it is defined as the last entry in the file. This is necessary as netrc is processed sequentially and accepts the first entry that matches the connecting host. For more information on netrc and macros, see man pages for netrc and ftp.

HT Password Manager

HT Password Manager is a web interface to manage apache's htpasswd files. Manage multiple password files with separate per-file administrator. Administrators can add/delete/search and reset password for all users & users can change their own passwords.

Find the Sourceforge project page here: HT Password Manager

Running Subversion as Windows Service

Download the source package of subversion for windows from http://subversion.tigris.org
Unpack the Zip package to some Directory
Eg: C:\usr\svn-win32-1.4.5\

Now add the subversion binaries to system PATH. Binaries are located in “C:\usr\svn-win32-1.4.5\bin\“. To set this either you can go through the Advanced menu of system applet or use “set” command from command line.

Now you will be able to use the svn commands from command prompt. As the first step, create a root folder for all projects that will run under this server, say C:\svnroot. Now we need to run the svn server.

C:\Temp\>svnserve -d -r C:\svnroot

This will start svn server listening on default port 3690. To verify, start another command window
C:\Temp>netstat -ano|find “3690″
TCP 0.0.0.0:3690 0.0.0.0:0 LISTENING 2820
The process with PID 2820 is listening on port 3690, to verify that the process is svnserve,
C:\Temp>tasklist |find “svnserve.exe”
svnserve.exe 2820 Console 0 3,264 K

This confirms that snvserve is running and listening for connections on port 3690

This method has the drawback that the svnserve command is running in foreground which means,
*A command window will be running in the foreground all the time
*If the windows is closed or the server is rebooted the service will require to be manually started again.
*A user has to be logged in to the server and he cannot log off for another user to use the system.

So to make life easier, we will have to automate the whole thing. This means that the svnserver should be run as a windows service. Subversion has a Windows installer which will install itself as a service, bu I feel it is not that flexible and hence wanted to implement the serive in my way. Svnserve supports running as windows service with the command line switch –service. To make use of this feature we should use the Windows Service Control command line tool(sc.exe).

First, create the service with name SVNSVC
C:\Temp>sc create SVNSVC binpath= “C:\usr\svn-win32-1.4.5\bin\svnserve.exe –service -r C:\svnroot” displayname= “Subversion Server” depend= Tcpip
[SC] CreateService SUCCESS

Configure the service to start at boot time
C:\Temp>sc config svnsvc start= auto
[SC] ChangeServiceConfig SUCCESS

Start service
C:\Temp>net start svnsvc
The Subversion Server service is starting.
The Subversion Server service was started successfully.

If you mis configure the service option or run into problems, you can always remove the service from the system with the command
C:\Temp>sc delete svnsvc
[SC] DeleteService SUCCESS
and create a fresh service.

You can configure, start and stop services from the services MMC console in administrative tools instead of using the command line. But you can’t add or delete service.

Resetting MySQL root password

MySQL root user is the built in user with administrative privileges on MySQL server. Even though this is the default administrator user, you can add any number of new users with varying administrative rights on MySQL server. In many occasions system administrators tend to forget the MySQL root password, but this can be fixed with out much pain.

Stop the MySQL server. If you have start/stop service scripts installed you can do it that way:

[root@linuxbox1 ~]#service mysqld stop OR

[root@linuxbox1 ~]#/etc/rc.d/init.d/mysqld stop or whatever other script you use normally.

In some cases, this may fail if the server is overloaded, in such a case you will have to kill the mysql process

You can get the PID of the process using ps command:

[root@linuxbox1 ~]# ps aux|grep mysqld OR

looking in the mysql pid file ( found usually in MYSQL_DATA_DIRECTORY/mysql.pid or /var/run/mysqld/mysqld.pid MySQL data/installation directory is usually /var/lib/mysql ). Then kill the process

[root@linuxbox1 ~]#kill `cat /var/run/mysqld/mysqld.pid`

Now since the mysql server has stopped, start mysqld with the –skip-grant-tables option

[root@linuxbox1 ~]# /usr/bin/mysqld_safe –skip-grant-tables

Now set new root password

1) Using mysqladmin

[root@linuxbox1 ~]# mysqladmin -u root password ‘my_newpass#123‘

Note that you don.t need to provide a password to use mysqladmin. Now flush the privilege tables in mysql to take the effect of password change

[root@linuxbox1 ~]# mysqladmin -u root flush-privileges

2) Using mysql client

[root@linuxbox1 ~]# mysql -u root mysql

mysql> UPDATE user SET Password=PASSWORD(’my_newpass#123′) WHERE User=’root’;
mysql> FLUSH PRIVILEGES;

Password is reset now. Now you have to restart MySQL service. First kill the mysqld_safe process, use the ps command to get the pid of mysqld_safe. Start MySQL service as usual.

[root@linuxbox1 ~]#/etc/rc.d/init.d/mysqld start

Now you will be able to connect to MySQL with new password ‘my_newpass#123‘

[root@linuxbox1 ~]#mysql -u root -pmy_newpass#123