ONLamp.com: Managing Advanced PF Logs

Published on ONLamp.com (http://www.onlamp.com/)
See this if you're having trouble printing code examples

Managing Advanced PF Logs

by Jacek Artymiak
08/22/2002

Welcome back.

In part 6 and part 7 we learned how to send logs over an ssh connection and how to make that connection more secure. We also did some math to find out how much storage space we'll need to keep old logs for analysis. Today we need to roll up our sleeves again, and do some coding in Perl, because the pflog fifo pipe created on the monitoring station's hard disk in part 6 is a bit like an oil well, gushing with black stuff that needs some additional piping to turn it into an orderly flow.

Catching and taming that wild stream of data is the job of the readpflog script. Written in Perl, it runs in the background and does three basic things: 1) reads pf logs from the pflog fifo pipe, 2) archives logs on the monitoring firewall, and 3) sends them to another fifo pipe so that log analysis software can pick them up for analysis.

#!/usr/bin/perl -W
#
#                                   Copyright 2002 Jacek Artymiak
#                                                License: XFree86
#----------------------------------------------------------------
# section  1: basic setup  

use Fcntl;
use POSIX qw(:errno_h);

$rdelay = 3600;  # log archiving delay (in seconds, 0 turns off); 

#----------------------------------------------------------------
# section  2: check if readpflog.pid exists

if (-e "/home/scooter/readpflog.pid") {

    print "Looks like readpflog is already running, if it is " . 
               "not running, delete /home/scooter/readpflog.pid";
    exit;
}

#----------------------------------------------------------------
# section  3: open readpflog.log --  the log used by readpflog to 
#                                              store its messages

open (LOG, ">> /home/scooter/readpflog.log"); 
select (LOG);
$|=1;

#----------------------------------------------------------------
# section  4: define the logme function used to write messages to 
#                                                   readpflog.log

sub logme {

 $datetime = `date`;
 chop $datetime;

 $logentry = $datetime . ": readpflog[$$]: $_[0]" . "\n";

 print LOG $logentry;
}

#----------------------------------------------------------------
# section  5: define the loganddie function used to clean up 
#                                           before readpflog dies 
 
sub loganddie {

 $datetime = `date`;
 chop $datetime;

 $logentry = $datetime
                      . ": readpflog[$$]: Fatal error: $_[0]" 
        . ": Exiting ...\n";

 print LOG $logentry;

 $logentry = $datetime 
                      . ": readpflog[$$]: Closing pflog ...\n";

 print LOG $logentry;

 close (INFILE);

 $logentry = $datetime 
              . ": readpflog[$$]: Closing pflog-current ...\n";

 print LOG $logentry;

 close (OUTFILE);

 $logentry = $datetime 
              . ": readpflog[$$]: Removing readpflog.pid ...\n";

 print LOG $logentry;

 `rm ~/readpflog.pid`;

 $logentry = $datetime 
              . ": readpflog[$$]: Exiting.\n";

 print LOG $logentry;

 die ($logentry);
}

#----------------------------------------------------------------
# section  6: define the rotatelogs function, which closes and
#                                         reopens ~/readpflog.log
 
sub rotatelogs() {

 logme ("Closing readpflog.log.");

 close (LOG);

 open (LOG, ">> /home/scooter/readpflog.log");
 select (LOG);
 $|=1;

 logme ("readpflog.log rotated.");
}

#----------------------------------------------------------------
# section  7: we're waking up 

logme ("Starting readpflog ...");

#----------------------------------------------------------------
# section  8: write the current process ID (PID) to 
#                                                 ~/readpflog.pid

logme ("Creating readpflog.pid ...");

open (PIDFILE, "> /home/scooter/readpflog.pid") or 
       loganddie ("Unable to create readpflog.pid: " . $! . '.');

logme ("Writing PID to readpflog.pid ...");

syswrite PIDFILE, $$;

logme ("Closing readpflog.pid");

close PIDFILE;

#----------------------------------------------------------------
# section  9: open ~/pflog for reading

sub opensource {

    logme ("Trying to open pflog ...");

    open (INFILE, "< /home/scooter/pflog") or 
                 loganddie ("Unable to open pflog: " . $! . '.');

    logme ("pflog opened successfully.");

    select (INFILE);
    $|=1;
}

#----------------------------------------------------------------
# section 10: open ~/pflog-current for writing

sub opentarget {

    logme ("Trying to open pflog-current ...");

    open (OUTFILE, ">> /home/scooter/pflog-current") or 
         loganddie ("Unable to open pflog-current: " . $! . '.');

    logme ("pflog-current opened successfully.");

    select (OUTFILE);
    $|=1;

    alarm $rdelay;
}

#----------------------------------------------------------------
# section 11: rotate ~/pflog-* archive

sub rotatetarget {

    close (OUTFILE);

    $d_t = `date "+%Y-%m-%d-%H-%M-%S"`; 

    unless (fork) {
        system ("mv /home/scooter/pflog-current /home/scooter/pflog-" . $d_t);
        system ("gzip -9 /home/scooter/pflog-" . $d_t);
        exit;
    }

    opentarget();
}

#----------------------------------------------------------------
# section  12: opens ~/pflog-pipe fifo pipe for writing 
#

sub openpipe {

    sysopen(LPIPE, "/home/scooter/pflog-pipe", O_NONBLOCK|O_RDWR)
                                  or die "Can't open pipe: $!\n";

    select (LPIPE);
    $|=1;
}

#----------------------------------------------------------------
# section  13: set signal handlers

$SIG{HUP}  = 'rotatelogs';
$SIG{INT}  = 'loganddie';
$SIG{QUIT} = 'loganddie';
$SIG{KILL} = 'loganddie';
$SIG{TERM} = 'loganddie';
$SIG{STOP} = 'loganddie';
$SIG{TSTP} = 'loganddie';
$SIG{PIPE} = 'IGNORE';
$SIG{ALRM} = 'rotatetarget';

#----------------------------------------------------------------
# section  14: open input and output files

opensource();
opentarget();
openpipe();

#----------------------------------------------------------------
# section  15: read ~/pflog, write it to ~/pflog-current, and 
#                                                    ~/pflog-pipe

for (;;) {

    while (<INFILE>) {

        if (length ($_) != 0) {

            $buf = $_;

            if (!(syswrite OUTFILE, $buf)) {

                close (OUTFILE);
                opentarget();
            }

            if (!(syswrite LPIPE, $buf)) {

               close (LPIPE);
               openpipe();
            }
        }

    }

    sleep 1;
    seek (INFILE, 0, 1);
}

Here is a short summary of what this script does:

Section 1: Imports standard Perl modules used for non-blocking I/O, and defines the $rdelay variable used to define the delay between two consecutive pflog archiving operations; the delay is measured in seconds. One hour is 3600 seconds; be careful not to set this delay too low (below 15 seconds) -- the system may not be able to archive logs in such a short time. Also, avoid setting this variable to 0, as it will switch archiving off.
Section 2: Checks if the PID file exists and if it does, exits. You can check if readpflog is already running with ps -auxw | grep readpflog; if it is running, you will need to kill it. If it isn't, delete /home/scooter/readpf.pid and run readpflog again.
Section 3: Opens the /home/scooter/readpflog.log file to store messages generated by the script itself. The log file is located in the /home/scooter directory, but you can change that to another location.
Section 4: Defines the logme() function, which writes properly-formatted logs to /home/scooter/readpflog.log.
Section 5: Defines the loganddie() function, which logs fatal errors, closes files, and removes the /home/scooter/sendpflog.pid PID file.
Section 6: Defines the rotatelogs() function, which closes and opens /home/scooter/readpflog.log after receiving the SIGHUP signal from newsyslog.
Section 7: Lets the world know the script is getting ready for work.
Section 8: Writes the PID number of the current process to /home/scooter/readpflog.pid. This file will be used by newsyslog to send the SIGHUP signal.
Section 9: Defines the opensource() function, which opens /home/scooter/pflog for reading.
Section 10: Defines the opentarget() function, which opens /home/scooter/pflog-current for writing; this file is rotated and archived at regular intervals, defined in $rdelay.
Section 11: Defines the rotatetarget() function, which rotates /home/scooter/pflog-current; every new archive is compressed with gzip and has a name that begins with pflog- and ends with a date and time string, e.g. "pflog-2002-07-30-23-53-03.gz."
Section 12: Defines the openpipe() function, which opens /home/scooter/pflog-pipe fifo pipe for writing; this pipe can be read by tcpdump or other pflog analysis software.
Section 13: Configures signal handlers to catch most important signals.
Section 14: Opens input and output files.
Section 15: Enters the main loop, reads /home/scooter/pflog, and sends it to the dump file (/home/scooter/pflog-current) and the fifo pipe (/home/scooter/pflog-pipe).

Related Reading

Perl for System Administration
Managing multi-platform environments with Perl
By David N. Blank-Edelman

Log in as the user scooter (or whatever username you used for the user receiving logs), copy the script, and save it as readpflog. Now we need to make readpflog executable, and make it owned by scooter and a member of the scooter group with these commands (you need to be logged in as scooter):

# chmod 0700 readpflog
# chown scooter readpflog
# chgrp scooter readpflog

The user and the group need to be created on the monitoring station, with minimal privileges.

Next, we need to create the /home/scooter/pflog-pipe fifo pipe:

$ mkfifo -m 0600 pflog-pipe

In the last step, we need to add the following line to /etc/newsyslog.conf:

/home/scooter/readpflog.log        600  3    250  *     ZB /home/scooter/pflogd.pid

(Note that newsyslog will only rotate readpflog.log files, not pflog-* files.)

You might want to set $rdelay to a lower value; say, 60 seconds. If everything is working fine, you should see a list of archives similar to this one:

-rw-r--r--  1 scooter  scooter   212558 Jul 30 23:50 pflog-2002-07-30-23-50-54.gz
-rw-r--r--  1 scooter  scooter       46 Jul 30 23:51 pflog-2002-07-30-23-51-14.gz
-rw-r--r--  1 scooter  scooter       46 Jul 30 23:51 pflog-2002-07-30-23-51-24.gz
-rw-r--r--  1 scooter  scooter       46 Jul 30 23:51 pflog-2002-07-30-23-52-03.gz
-rw-r--r--  1 scooter  scooter       46 Jul 30 23:52 pflog-2002-07-30-23-52-18.gz
-rw-r--r--  1 scooter  scooter       46 Jul 30 23:52 pflog-2002-07-30-23-52-33.gz
-rw-r--r--  1 scooter  scooter       46 Jul 30 23:52 pflog-2002-07-30-23-52-48.gz
-rw-r--r--  1 scooter  scooter     4510 Jul 30 23:52 pflog-2002-07-30-23-53-03.gz
prw-r--r--  1 scooter  scooter        0 Jul 30 23:52 pflog-pipe

Now change $rdelay to a higher value, and have fun. You will most probably want to write another script, or modify readpflog (you're free to make any changes you like within the scope of the terms of the XFree86 license) to write archives to tape, or other external storage devices.

As before, you can find the listing of the script from this issue in the OpenBSD Administrator Toolbox.

Until next time!

Jacek Artymiak started his adventure with computers in 1986 with Sinclair ZX Spectrum. He's been using various commercial and Open Source Unix systems since 1991. Today, Jacek runs devGuide.net, writes and teaches about Open Source software and security, and tries to make things happen.

Read more Securing Small Networks with OpenBSD columns.

Return to the BSD DevCenter.