Centralized Logging for UNIX, Windows, and Network Devices

Corey Ramsden

There's nothing sexy about log files, but where would we be without them? They are our eyes and ears for the digital jungle. Like other senses, they can overload us with too much information from too many sources. One way to get a handle on this problem is to consolidate logging to one or more central loghosts.

There are many reasons to centralize logging. Running log file watch and reporting routines on every single host can be a cumbersome task, often requiring administrators to script out routines on several different platforms. Consolidating logs allows for the creation and maintenance of one set of watching and reporting utilities. Another major reason to centralize logging is security. Writing logs to a central host and backing them up to tape regularly is an excellent way to keep log file information safe for later review during security incident investigations.

Syslog-ng ("syslog next generation") allows you to log remote events to host-specific files that can be rotated and backed up to tape on a regular basis. Because so many operating systems, applications, and network devices support syslog functionality, we can configure almost all of our managed devices to log their activity to a central loghost.

Syslog-ng is available from:

http://www.balabit.com/products/syslog_ng/

Installation of syslog-ng on Central Loghost

We employ a simple installation of syslog-ng at our site because we have a small number of hosts and limited logging requirements. The loghost is a modest Intel x86 box with Red Hat Linux installed and locked down as tightly as possible. A loghost houses lots of sensitive information so it's important to keep as few services running on the system as possible. Preferably, it should only run two external services: syslog-ng and ssh.

The following steps will give you a basic working setup. For more information on setup and configuration, consult the online FAQ and HOWTO notes (see references). Note the step to remove the -r from the normal syslog daemon startup. This ensures that the default syslog daemon is not listening on port 514/udp and conflicting with our new syslog-ng setup.

If you are interested in running syslog-ng chroot'ed as a non-root user, this is possible in syslog-ng 1.5.x+ versions through the -C and -u flags.

1. Get the latest stable versions of libol and syslog-ng from:

http://www.balabit.com/products/syslog_ng/

./configure && make && make install

./configure && make && make install

mkdir /etc/syslog-ng
chmod 700 /etc/syslog-ng
mkdir -p /var/syslog-ng/hosts
chmod -R 700 /var/syslog-ng

touch /var/syslog-ng/hosts/[hostname].log

cd /var/syslog-ng/hosts
chmod 600 *

chmod 600 /etc/syslog-ng/*

8. Remove -r option from /etc/rc.d/init.d/syslog startup.

9. Stop/start syslog with: /etc/rc.d/init.d/syslog restart.

10. Do netstat -an to make sure it isn't listening on udp/514 anymore.

11. Start syslog-ng with: /etc/rc.d/init.d/syslog-ng start.

12. Enable syslog-ng automatically on system startup.

You may be wondering at this point why we would even leave the default syslogd/klogd setup running. We do this for the sake of conformity so that the central loghost matches the rest of the hosts in the environment as closely as possible. All hosts, including the central loghost, log events to their usual locations as well as to the central loghost. Another possibility would be to log the central loghost's own events locally to the default location (or to the local syslog-ng setup) as well as to send them to a secondary syslog-ng machine for safekeeping.

Configuration for syslog-ng.conf

There are a few important options to note in the configuration file. Be sure to set the appropriate permission bits for the files with the perm() option. Also pay close attention to the use_dns and use_fqdn options because both can cause problems if you don't have name service set up appropriately for your hosts. In our case, we elected to filter incoming log events by IP and to name files explicitly by hand to keep things simple and direct.

The source option allows you the flexibility of listening on UDP ports, TCP ports, and tunneled connections as well as internal logging streams. For creating a centralized loghost, listening on the default 514/udp syslog port is the most useful option because many managed devices are unable to change the port to which they forward syslog information. For larger networks, consider using syslog-ng in a tunneled capacity over TCP to allow remote hosts to send their events to the central logging server.

If you decide to log local events to syslog-ng as well, you'll need to specify an additional source and use that source when directing those events to the file of your choosing. Options for doing this are included in the configuration file (Listing 2).

The filter options available with syslog-ng provide great flexibility in dealing with incoming information. In our case, we've chosen only the ones for defining hosts. Notice the trailing "$" at the end of this expression:

filter                  f_unix-server1   { host(10.1.1.4$); };

Another useful option, especially for network devices, is the facility filter. For example, if you have a firewall device that only permits a few types of facility definitions in its syslog capability, you may need to designate a local facility to rules event logging and another local facility to authentication failure events.

For example, in our environment, the rule event logging is extensive and creates large log files. Rather than weed through those events looking for authorization failures to the device and other alarm events, we split them up with different local facilities and then separate them into different files when they arrive at the loghost.

Similar arrangements would also be helpful if, for example, you wanted to log account audit events (on Unix, Windows, or network devices) to a different file than the main traffic from a particular host. In syslog-ng.conf, to filter an event by a specific facility, you would specify a filter line like this:

filter                  f_local1        { facility(local1); };

log { source(src);   filter(f_firewall1); filter(f_local1); \
  destination(firewall1);
};

Log Rotation Using logrotate.d

To help manage the log files we're now collecting, we'll use the logrotate setup that comes with the Red Hat distribution. Create a file called "syslog-ng" and place it in your logrotate.d include directory. The default directory on most Linux systems is /etc/logrotate.d. Add a stanza like the one below for each host's log files. rotate 8 rotates in a cycle of eight (weeks). We're also compressing each file with compress; however, with delaycompress included, logrotate will only do so after you have two uncompressed files available (i.e., unix-server1.log and unix-server1.log.1).

If you have the disk space, you might consider using delaycompress; it is a convenient option if you frequently spot check log activity by hand because you won't need to uncompress them first:

/var/syslog-ng/hosts/unix-server1.log {
    rotate 8
    postrotate
        /bin/kill -HUP 'cat /var/run/syslog-ng.pid 2>/dev/null' \
          2>/dev/null || true
    endscript
    compress
    delaycompress
}

Now we're ready to configure each of the client machines. We chose to have each host log to itself as well as to the central loghost for the sake of redundancy and for comparing entries in the event of a security incident. So, we'll just add a line to the syslog.conf file to send all information above the info level to the loghost:

*.info                                    @loghost

For Windows hosts, several software packages are available for sending Windows eventlog entries to a syslog host. EventReporter (http://www.eventreporter.com/en) works particularly well; it is quite easy to install/configure and is inexpensive.

For whatever package you use, make sure you configure the software to include the Windows log type (Application, System, Security, etc.). This information can be very helpful when writing watch and alert scripts.

For network devices, your mileage may vary but most firewalls, routers, and switches should have a syslog capability of some sort. Just configure them to forward their events to the IP of the centralized loghost. As mentioned previously, for some of the more basic implementations, you may need to designate local facilities to different logging event types and then filter them accordingly at the central loghost.

Conclusion

Consolidating log files can be a tremendous help to systems administrators in their daily duties of log monitoring and security incident reviews. syslog-ng offers a flexible system for receiving log files from a variety of sources. If you combine that with free rotation utilities and inexpensive Windows utilities, you can centralize logging from almost every host in your network and build tools to monitor them all from one location. After it's all done, don't forget to back up those logs to tape regularly!

References

syslog-ng FAQ -- http://www.campin.net/syslog-ng/faq.html

Central loghost HOWTO -- http://www.campin.net/newlogcheck.html

Acknowledgements

I thank Rich Simon for his review and comments on this paper.

Corey Ramsden is the Director of Systems for a small online education company. He holds a degree in Political Science from Loyola College in Maryland and has 7 years of experience managing heterogeneous workstation and server environments. He can be reached at: cramsden@mindspring.com.