Hacking BSD, Part 2

O'Reilly Book Excerpts: BSD Hacks

Hacking BSD, Part 2

Editor's note: Last week's excerpts from O'Reilly's BSD Hacks offered a few administrator tricks -- finding files easily and managing disk space without reinstalling. This week's excerpt provides administrators and users alike with a hack on how to get the most out of FTP using macros and scripts.

Hack #51. Get the Most Out of FTP

Get the most out of stock ftp with macros and scripts.

In this age of GUIs and feature-rich browsers, it's easy to forget how quick and efficient command-line ftp can be. That is, until you're logged into a system that doesn't have X installed, nor a browser, nor any fancy FTP programs. If it's really your lucky day, it won't even have any manpages. And, of course, you'll need to download something.

Perhaps you find yourself using ftp all the time, always going to the same FTP servers and downloading from or uploading to the same directories. Clearly, it's time for some FTP automation.

Automating Logins

Have you ever noticed how easy it is to use FTP from a modern browser? Simply click on a hyperlink to start a download. At the command line, though, you can't even browse the FTP directory structure until you successfully log into the FTP server. Well, guess what: you always have to log into an FTP server. It's just that your web browser hides this little detail by doing it for you in the background.

You can achieve the same transparency for command-line ftp by creating a file called .netrc in your home directory and placing the following line in that file:

% more ~/.netrc
default login anonymous password genisis@istar.ca

This line will work for any FTP server on the Internet that accepts anonymous logins. (Most do, unless it's a private server.) When creating your own file, use your own email address as the password.

Test your change with this command:

% ftp ftp.freebsd.org

Compare your results to the FTP output in [Hack #71] . You should receive the same banner shown there without having to first type in a username and password.

If you're a webmaster who uses FTP to upload your new files, you do have to log in first. After all, you don't want just anyone uploading files, so you require a username and password. To automate that process, add a section to your ~.netrc that reflects the name of your server and your username and password:

machine ftp.myserver.com
login myusername
password mypassword

Since you've just inserted your password into a plain text file, it's important to change the permissions on this file so that only you can read it:

% chmod 600 ~/.netrc

If you forget to change the permissions and try to access an FTP server that requires a username and password, your login attempt should fail and result in this error message:

ftp: Error: .netrc file is readable by others.
ftp: Remove password or make file unreadable by others.

To be extra safe, exclude the password line completely. When you connect to the FTP server, your username will be provided for you, but you will still be prompted for the password.

Automating Transfers

Now, let's say that you visit ftp.freebsd.org on a regular basis and always access its pub/FreeBSD/releases/i386 directory. Rather than cding every time, you can automate that process by creating an FTP macro. Add these lines to ~/.netrc:

macdef fbsd
bin
cd /pub/FreeBSD/releases/i386

Macros are defined by macdef, and the name of the macro follows. Keep the name short but useful, as a macro is supposed to be a timesaver. Once you've declared the macro, add the FTP commands you want to execute, one line at a time. This particular macro contains the bin (or binary) command. That command is useful when downloading because it ensures all files, including non-ASCII files such as applications, will download correctly. I also included a cd command to automatically take me to my usual working directory.

TIP: It's important that a macro always ends with a blank line.

There are two ways to use your macro. If you're already connected to the FTP server, type $macroname at the ftp prompt:

ftp> $ fbsd
bin
200 Type set to I.
cd /pub/FreeBSD/releases/i386
250 "/pub/FreeBSD/releases/i386" is new cwd.

Note that each command in the macro will be executed, followed by its results.

The second way to run the macro is when you first invoke the ftp command:

% echo "$ fbsd" | ftp ftp.freebsd.org

Now, if you try that one, you'll notice that all of your commands will succeed. Then, your FTP session will abruptly end, and you'll receive your regular prompt back! Rather disappointing if you were planning on typing some more commands at the ftp prompt, but absolutely perfect if your intention is to script an entire FTP session.

Scripting an Entire Session

If you already know what you want to do, and especially if you need to do it more than once, why type in everything at the ftp prompt? Suppose you want to download the latest XFree86 distribution directly from ftp://ftp.xfree86.org/. Consider placing this macro in ~/.netrc:

macdef X
bin
bell
prompt
cd /pub/XFree86/4.3.0/source
mget *
bye

This macro assumes that this ~/.netrc file already contains the line that allows anonymous logins.

The bellcommand, which is optional, should produce a sound after each successful file transfer. The promptcommand is very important, though. By default, the FTP server expects interaction from the user. That is, when you ask to download multiple files with mget, the FTP server will wait for you to confirm every transfer by typing y. Obviously, we want to disable that behavior when we're scripting a download.

To run this macro:

% echo "$ X" | ftp ftp.xfree86.org

By default, ftp will save the downloaded files in your current working directory. If you prefer, you can specify an alternate location in your macro with the lcd (local change directory) command. For example:

lcd /usr/local/Xsource

will save the downloaded files to the /usr/local/Xsource directory. Make sure your directory exists and put the lcd line before your mget line.

A Better FTP?

No matter how hard you try to make the default FTP client user-friendly, it is still a very basic command, and you may find a little too primitive, especially if you use ftp often. If you would like to try a more convenient and user-friendly command-line tool, try ncftp, which is available as a port or package for FreeBSD, NetBSD, and OpenBSD.