Archiving PF Firewall Logs

Welcome back.

In the previous installment of this series we learned how to fine-tune the process of saving and rotating pf logs to match our preferences. Today we'll look at the problem of automating the transfer of logs from the firewall to one of the workstations connected to the internal private network segment. But, you may ask, why won't we analyze pf logs on the firewall instead? Well, while we could analyze the logs on the firewall, it is usually more convenient, efficient and safe to do it on another computer with a faster processor, more memory, and larger hard disks. We shouldn't really ask the firewall to do anything more than packet filtering. The task of log archiving and analysis is best left to another computer.

The monitoring workstation need not be connected to the internal private network segment. Logs can be sent to a trusted machine located on the outside of our network, although that is less common in small to medium-size networks. Some larger networks connecting computers at more than one physical location, though within the same organization often send logs to one central location for analysis, but still keep the copies of logs archived on-site as a security precaution.

Many administrators managing several client's networks from a single location, send logs to a single machine located inside the administrator's own network. Although this might seem a little unwise from the point of view of security, if strong encryption is used and the administrator follows all basic security precautions, the chances of hackers getting at the precious data is very low. One important drawback of sending logs outside your own network is the fact that they consume a lot of bandwidth, and bandwidth costs money.

Which method of archiving logs is best for you?

The choice of the method for remote logging and archiving depends on many factors, but in most cases, it will all boil down to the security policy of your organization, the amount of time and money you can devote to overseeing and monitoring the whole setup, as well as what you feel is most convenient to you. But in any situation, it is highly advisable that you choose a method that is the easiest to build, manage and secure within the constraints of law, money and time.

There are literally dozens of solutions for archiving logs and we cannot possibly describe them all in this article, because the exact details of every configuration are highly dependent on each site's security policies and the personal preferences of its administrators. However, since we are using OpenBSD, we can assume that certain things are going to look and work the same on most OpenBSD installations. Software tools like pf, newsyslog, or syslogd are available on all OpenBSD 3.0 (and later) systems; similarly ssh and Perl are also standard fare on all free and commercial UNIX derivatives (Linux, BSD, Mac OS X, etc.).

Figure 1: Monitoring station connected to the internal private network segment. (You can click on the figure to open a full-size view.)

For the sake of simplicity we'll use a network split into two segments: private network and DMZ (Figure 1), both protected by the same firewall, which will automatically collect and send logs to one of the workstations connected to the internal private network segment. (The details of the configuration of such network design were discussed in part 1 and part 2 of this series, and can be easily adapted to other network configurations.)

This solution is convenient and cost-effective, because the administrator can log into the monitoring station from other hosts and can use the same workstation for doing things other than log collection, archiving, and analysis. Also, because the monitoring station is connected to the network, which itself is connected to the outside world, upgrades of the station's operating system and tools are quick and easy. (The monitoring station need not be running one of UNIX variants, but our job will be much easier, if it does.)

Figure 2: Monitoring station connected to a separate network segment. (You can click on the figure to open a full-size view.)

There is a lot to like about this configuration, but the administrator should be careful to not overload the internal network with additional traffic generated when logs are transmitted from the firewall to the monitoring station. If our firewall is set up to log all traffic going in and out of the private segments and the DMZ segment, then sending the logs to the workstation connected to the internal private segment of your network (Figure 1) more than doubles the amount of data transferred in that segment. When the network becomes congested, there are two things we can do:

• if you are still using 10Mb cards, replace all of them with faster 100Mb or 10/100Mb cards; or
• connect the monitoring station to the firewall via a separate interface, this requires another network card added to the firewall machine (Figure 2).

If you want to connect the monitoring station via a separate interface, you will need to do the following things:

• add another network card to the firewall, a cheap second-hand 10Mb (compatible with OpenBSD, check the OpenBSD Platforms page) will probably be quite enough, but if you can spend 5-10 bucks more, use a 100Mb or 10/100Mb card as it certainly is more future-proof;
• connect the monitoring station to the firewall via a cross-over cable (no hub is required, because there are only two interfaces on that network segment: one in the firewall, and one in the monitoring station);
• add a new set of filtering rules for the new network segment (you can copy the rules for the private network segment, changing the segment IP numbers);
• check that everything is working.

Configuring the firewall

Before you start rebuilding your network, it would be a good idea to test the software side of this configuration without touching the hardware, that way you will have fewer things to debug. After you have the software working, reconfiguring the hardware is rather easy. Also, to avoid the risk choking your network with log transfers during the testing, either choose a time when the traffic is low or log only a small fraction of traffic, just to test things on a smaller scale before you let the whole thing run on full power.

To limit the size of logs, copy /etc/pf.conf to /etc/pf.conf_old, open /etc/pf.conf in a text editor, and add the log-all keyword to one of the pass out rules for the external interface. That way you will be logging only the outbound traffic leaving the external interface and originating from your private network. For example, change this rule

# allow TCP IPv4 connections to the outside world, keep state 
 
pass  out on $ExtIF inet proto tcp all flags S/SA modulate state

to this

# allow TCP IPv4 connections to the outside world, keep state 
 
pass  out log-all on $ExtIF inet proto tcp all flags S/SA modulate state 

You can begin logging all traffic again, when you are sure that the monitoring station is receiving logs.

While you are editing the pf configuration file, you should also add a line that prevents pf from logging packets sent from the firewall to the monitoring station to port 22, where you should have ssh listening for connections. If you don't do that, you will create a loop which will bring your network to a halt (the packets carrying the logs will be logged and sent to the monitoring station and logged again, and sent to the monitoring station, and so on ...).

The following rule should be added at the end of the pf rule section for the internal private network segment interface (you may need to modify it to match your network configuration, but don't add the log or log-all keywords):

# allow pfloggin on a remote host 
# (there are two lines below) 
 
pass  out on $PrvIF inet proto tcp  from xxx.xxx.xxx.xxx 
		to yyy.yyy.yyy.yyy port 22 modulate state 
 
pass  out on $PrvIF inet proto udp  from xxx.xxx.xxx.xxx 
		to yyy.yyy.yyy.yyy port 22 keep state

Save the modified /etc/pf.conf and reload it with

Your firewall is now logging only the outbound traffic. It is not sending the logs to the monitoring station, for that we need to use a script, which we'll write in Perl. Using Perl is not a requirement, you could use shell, AWK, or any other language, but since the Perl interpreter is installed as a part of the default OpenBSD configuration, there is little point to not use it.

Sendpflog

We are now ready to write the script that sends the logs to the firewall. For the sake of simplicity, we will try to make as little changes to the default setup as possible. The only intrusion will be one small change in /etc/newsyslog.conf and the addition of the sendpflog script. The following script will handle the rest of the work:

#!/usr/bin/perl -W 
# 
#                                   Copyright 2002 Jacek Artymiak 
#                                                License: XFree86 
#---------------------------------------------------------------- 
# section  1: open sendpflog.log -- the log used by sendpflog to 
#                                              store its messages 
 
open (LOG, ">> /var/log/sendpflog.log"); 
select (LOG); 
$|=1; 
 
#---------------------------------------------------------------- 
# section  2: define the logme function used to write messages to 
#                                                   sendpflog.log 
 
sub logme { 
 
$datetime = `date`; 
chop $datetime; 
 
$logentry = $datetime . ": sendpflog[$$]: $_[0]" . "\n"; 
 
print LOG $logentry; 
} 
 
#---------------------------------------------------------------- 
# section  3: define the loganddie function used to clean up 
#                                           before sendpflog dies 
 
sub loganddie { 
 
$datetime = `date`; 
chop $datetime; 
 
$logentry = $datetime . ": sendpflog[$$]: Fatal error: $_[0]" 
. ": Exiting ...\n"; 
 
print LOG $logentry; 
 
$logentry = $datetime 
. ": sendpflog[$$]: Closing pflog ...\n"; 
 
print LOG $logentry; 
 
close (INFILE); 
 
$logentry = $datetime 
. ": sendpflog[$$]: Closing output stream ...\n"; 
 
print LOG $logentry; 
 
close (OUTFILE); 
 
$logentry = $datetime 
. ": sendpflog[$$]: Removing sendpflog.pid ...\n"; 
 
print LOG $logentry; 
 
`rm /var/run/sendpflog.pid`; 
 
$logentry = $datetime 
. ": sendpflog[$$]: Exiting.\n"; 
 
print LOG $logentry; 
 
die ($logentry); 
} 
 
#---------------------------------------------------------------- 
# section  4: define the rotatelogs function, which closes 
#             /var/log/pflog and /var/log/sendpflog.log, 
#             sends SIGHUP to pflogd, and reopens /var/log/pflog 
#                                      and /var/log/sendpflog.log 
 
sub rotatelogs() { 
 
logme ("Closing rotated pflog."); 
 
close (INFILE); 
 
logme ("Closing sendpflog.log."); 
 
close (LOG); 
 
open (LOG, ">> /var/log/sendpflog.log"); 
select (LOG); 
$|=1; 
 
logme ("sendpflog.log rotated."); 
 
logme ("Sending SIGHUP to pflogd."); 
 
kill HUP => `cat /var/run/pflogd.pid`; 
 
logme ("Trying to open new pflog ..."); 
 
open (INFILE, "< /var/log/pflog") or 
loganddie( "Could not open pflog:" . $! . '.'); 
 
logme ("pflog opened."); 
} 
 
#---------------------------------------------------------------- 
# section  5: define the open_output function which connects to 
#             the monitoring station 
 
sub open_output { 
 
logme ("Trying to open output stream ..."); 
 
open (OUTFILE, 
'| ssh -q -2 -l username xxx.xxx.xxx.xxx "cat >> pflog"') 
or loganddie ("Could not open output stream: " . $! . '.'); 
 
sleep(5); 
select (OUTFILE); 
$|=1; 
 
logme ("Output stream opened successfully."); 
} 
 
#---------------------------------------------------------------- 
# section  6: set signal handlers 
 
$SIG{HUP}  = 'rotatelogs'; 
$SIG{INT}  = 'loganddie'; 
$SIG{QUIT} = 'loganddie'; 
$SIG{KILL} = 'loganddie'; 
$SIG{TERM} = 'loganddie'; 
$SIG{STOP} = 'loganddie'; 
$SIG{TSTP} = 'loganddie'; 
$SIG{PIPE} = 'IGNORE'; 
 
#---------------------------------------------------------------- 
# section  7: we're waking up 
 
logme ("Starting sendpflog ..."); 
 
#---------------------------------------------------------------- 
# section  8: write the current process ID (PID) to 
#                                          /var/run/sendpflog.pid 
 
logme ("Creating sendpflog.pid ..."); 
 
open (PIDFILE, "> /var/run/sendpflog.pid") or 
loganddie ("Unable to create sendpflog.pid: " . $! . '.'); 
 
logme ("Writing PID to sendpflog.pid ..."); 
 
syswrite PIDFILE, $$; 
 
logme ("Closing sendpflog.pid"); 
 
close PIDFILE; 
 
#---------------------------------------------------------------- 
# section  9: open /var/log/pflog for reading 
 
logme ("Trying to open pflog ..."); 
 
open (INFILE, "< /var/log/pflog") or 
loganddie ("Unable to open pflog: " . $! . '.'); 
 
logme ("pflog opened successfully."); 
 
#---------------------------------------------------------------- 
# section  10: read /var/log/pflogand send it to the monitoring 
#                                                         station 
 
open_output(); 
 
for (;;) { 
while (<INFILE>) { 
if (!(syswrite OUTFILE, $_)) { 
logme("The monitoring station is not responding"); 
close (OUTFILE); 
open_output(); 
} 
} 
 
sleep 1; 
seek (INFILE, 0, 1); 
}

Here is a short summary of what this script does:

• section 1: open the /var/log/sendpflog.log file to store messages generated by the script itself. They are useful for debugging purposes and this file should be the first to look at when there is something wrong going on with the script itself. The log file is located in the /var/log directory, but you can change that to another location;
• section 2: define the logme() function, which will write properly formatted logs to /var/log/sendpflog.log;
• section 3: define the loganddie() function, which logs fatal errors, closes files and removes the /var/run/sendpflog.pid PID file;
• section 4: defines the rotatelogs() function which closes and opens /var/log/pflog and /var/lof/sendpflog.log after receiving the SIGHUP signal from newsyslog;
• section 5: defines the open_output() function used to open the connection to the monitoring station. Replace username with the name of the user you wish sendpflog to login as onto the monitoring station, and replace xxx.xxx.xxx.xxx with either the hostname or the IP number of the monitoring station;
• section 6: configure signal handlers to catch most important signals;
• section 7: let the world know the script is getting ready for work;
• section 8: write the PID number of the current process to /var/run/sendpflog.pid. This file will be used by newsyslog to send the SIGHUP signal;
• section 9: open /var/log/pflog for reading;
• section 10: enter the main loop, read /var/log/pflog and send it to the monitoring station.

Make sendpflog executable, owned by root and a member of the wheel group with these commands (you need to be logged in as root):

# chmod 0700 sendpflog 
# chown root sendpflog 
# chgrp wheel sendpflog 

Move sendpflog to /root/sendpflog with

Next, open /etc/newsyslog.conf and replace the following line

with these two lines:

/var/log/pflog          600  3    250  *     ZB /var/run/sendpflogd.pid 
/var/log/sendpflog.log  600  3    250  *     ZB /var/run/sendpflogd.pid 

The changes will tell newsyslog to monitor /var/log/pflog and /var/log/sendpflog.log and rotate them when their size meets the preset criteria.

Next we need to generate a key pair used to automatically log into the monitoring station without having to give the password. This is done with the ssh-keygen utility. You can use it to generate three types of keys: rsa1 (for RSA authentication connections made using the SSH1 protocol), rsa (for RSA authenitconnections made using the SSH2 protocol), or dsa (for DSA connections made using the SSH2 protocol). If the monitoring station is running the latest release of OpenSSH, then you can use RSA authentication, older releases of SSH may need to use DSA or RSA1.

Log in as root and run ssh-keygen as follows

# ssh-keygen -t dsa 
Generating public/private dsa key pair. 
Enter file in which to save the key (/root/.ssh/id_dsa): 

Hit Enter, the computer responds with

Hit Enter, the computer responds with

Hit Enter, the computer responds with

Your identification has been saved in /root/.ssh/id_dsa. 
Your public key has been saved in /root/.ssh/id_dsa.pub. 
The key fingerprint is: 
6f:38:13:90:04:89:55:e6:50:6f:69:50:5f:77:15:e7 root@localhost 

The /root/.ssh/id_dsa.pub file is the public key which you will copy to the monitoring station to the home directory of an ordinary user which you will need to create for the sole purpose of collecting, archiving and analyzing pf logs (the name of that user must be given in section 5). Do not use the root or any privileged account for that purpose. Also, create an ordinary group that will have only one member-our unprivileged user.

After you create the new account on the monitoring station, copy /root/.ssh/id_dsa.pub to the normal user directory on the firewall, e.g. /home/joe. Then, use scp to copy id_dsa.pub to the monitoring station. Then delete id_dsa.pub from /home/joe and copy (on the monitoring station) id_dsa.pub to the log collecting user's .ssh/authorized_keys2 file. For example, if the log collecting user is called scooter, use this command:

Note that if the monitoring station is running some other implementation of SSH, the authorized_keys2 file may reside in the .ssh2 directory. Also, if you use the RSA key (generated with ssh-keygen -t rsa), the id_rsa.pub file goes into authorized_keys2 as well, while the RSA1 key, the identity.pub file (generated with ssh-keygen -t rsa1) goes into .ssh/authorized_keys.

OK, it is now time to test the connection; login on your firewall as root and issue the following command:

(Replace username and xxx.xxx.xxx.xxx with real user name on the monitoring station and the station's IP number or host name).

If all goes well, you should see a message similar to this one:

RSA key fingerprint is xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx. 
Are you sure you want to continue connecting (yes/no)?

Answer yes, and you should be logged in on the monitoring station. Be careful, if ssh asks for the password, then there is something wrong and you cannot log in with the dsa key. To fix it, generate rsa or rsa1 keys and try logging in again (if you use the rsa1 key, make sure you replace -2 in section 5 of sendpflog with -1, this tells ssh to use the SSH1 protocol instead of SSH2). One of them is bound to work (but remember that rsa is more secure than rsa1).

Configuring the monitoring station

Related Reading Network Security with OpenSSL Cryptography for Secure Communications By John Viega, Matt Messier, Pravir Chandra

While you are logged on the monitoring station, create a named pipe in the log collecting user's directory on the station with this command

Notice that the name and the location of that named pipe must be given in section 5, after the cat >> command.

You are now ready to test the connection between the firewall and the monitoring station. To do that, log on the firewall as root and start sendpflog with

Then, log on the monitoring station as the user receiving logs and type this command

Let the computers do their job for a while, browse a few web sites (to generate traffic), and then press Ctrl+C. Type

and you should see something like this (the user and group names and the file size will be different)

If that is the case, you can congratulate yourself, the logs are being transferred to the monitoring station. But if it doesn't work, you should check ssh parameters in section 5, particularly the username, the host IP number or the name of the log file.

You can now read the logs on the monitoring station with tcpdump -r ./pflog, but that is not very convenient. What we need to do is write a script that works a little like newsyslog. And this is something we'll deal with in another installment of this series.

The method of automated log transfers described in this article could be used to transfer other logs /var/log/pflog. This will be left as an exercise to the reader. I will only say that every log will have to be transferred via a separate ssh connection, and the named of the scripts, log files and named pipes should be changed to match the name of a particular log.

The sendpflog script is a part of the OpenBSD Administrator Toolbox project hosted on SourceForge.

Well, that's it for today!

Until next time...

Jacek Artymiak started his adventure with computers in 1986 with Sinclair ZX Spectrum. He's been using various commercial and Open Source Unix systems since 1991. Today, Jacek runs devGuide.net, writes and teaches about Open Source software and security, and tries to make things happen.

Read more Securing Small Networks with OpenBSD columns.

Return to BSD DevCenter.

Copyright © 2009 O'Reilly Media, Inc.