Filtering Port Scans with nmap-audit

Keith Resar

No matter how much a systems administrator, even those with small installations, knows about the machines and the structure of his network, there is always a surprise lurking somewhere. It may be a machine that suddenly starts running an SMTP server, or someone running a MySQL database on his office workstation. Only regular, thorough audits of the hosts and ports open on the networks can put keep the sys admin informed.

Nmap, one of the most popular and pervasive open source port scanners, expertly determines which ports are open on any given machine. The program quickly executes and generates detailed reports on these open ports. Both efficiency and documentation are desired features in such a program, unless there are hundreds or thousands of computers within the network. Unfortunately, the manual process required by the regular regiment -- running nmap, gathering its reports, and determining the necessary changes -- is labor intensive and error prone. This process, without a doubt, can't scale to large networks or heavily loaded administrators.

This is where nmap-audit fits into the picture. The program helps clean up the many reports from nmap, merge them, eliminate unnecessary data, and present the remaining information in a format that is both easy to read and able to utilize the data management power of a spreadsheet. Upon an initial complete network audit, administrators will only receive a report from nmap-audit if a change is detected in the network.

Nmap-audit is actually a wrapper around nmap, which allows multiple nmap processes to run simultaneously so the network can be traversed more quickly. The data generated by each nmap process is stored in a directory structure that ensures data integrity, even with multiple runs, yet still provides administrators easy access to the raw nmap output when necessary.

Design Goals

My organization needed a script that would help turn an excellent port scanner into a capable enterprise-wide network auditor. As such, the nmap-audit script was developed to complement nmap, leveraging the port scanner's many design features. The requirements of this script are as follows:

• Guarantee that administrators know the current state of the network.
• Perform the scan as quickly as possible, using all resources allocated to the effort.
• Perform the audit with minimal user intervention. Specifically, after the initial audit, there is to be no administrator interaction unless the state of the network has changed.
• The process must not to be administrator initiated with each audit.
• The data presentation must be in a flexible form.
• All network state data, as well as the actual nmap output, must be readily available, regardless of whether the requested information has been reported to the administrator.

Nmap-audit deploys nmap to scan multiple hosts simultaneously. Once the scans are complete, nmap-audit produces a single report that consolidates port scan data that has been filtered to eliminate commonly or already accepted network configurations. Once the audit filter is configured, the report only highlights changes to the port configuration and will not deluge the user with unnecessary information on routine conditions.

As illustrated later, nmap-audit lets the user create groups of hosts and configure pre-authorized port settings for the hosts in each group. For example, you can create a group for email servers and specify in the nmap-audit configuration not to bother alerting administrators to the obvious fact that the email servers have TCP port 25 open for SMTP. As you add more filter information, the report becomes a slimmer and more manageable summary that will only include unauthorized or unanticipated activity.

The hardest part of this process, of course, is creating the filter. However, as this article will describe, you can quickly build the audit filter through successive runs of nmap-audit and the gradual application of filtering rules.

Installation

Nmap-audit requires that nmap and the Perl module MIME::Lite be installed prior to execution. (Links are available in the references section and installation instructions are located within each of the two software archives.) After obtaining and expanding the nmap-audit archive, run perl Makefile.PL, make, and make install (as root) as with an ordinary Perl module to copy the script to a system bin directory and install the included documentation.

Usage and configuration information are available via the command perldoc nmap-audit. Also, note that an example configuration file called "nmap-audit.rc" is included in the nmap-audit distribution. This configuration should be used as a template for new installations, rather than generating a configuration file from scratch.

Initial Setup/Configuration

Before running nmap-audit, you must decide on some technical and political site-specific information. On the technical side, decide what IPs should be scanned, since only a partial network audit may be required. In a similar vein, determine which ports and protocols should be scanned. These decisions may require a balancing act between thoroughness, time constraints, and limited network/CPU resources. Ideally, all 65,535 TCP and UDP ports would be scanned on each IP in the network, regardless of whether the IP responds to an ICMP ping. Unfortunately, this thoroughness comes at a cost -- time. Depending on the aggressiveness of the scan performed, this may take as long as 16 hours per host.

On the political front, begin by obtaining management approval to perform the audit. Scanning the enterprise network can be seen as a hostile act and acceptable computing use agreements may specifically note that this should not be done. Following management approval, two other issues must be determined. How often should the audit be performed? And who will be responsible for determining whether a machine is "properly" set up?

On the first point, if the scan is performed too infrequently, network problems may continue undetected. As such, confirm that there is a policy stating how frequently these audits must be performed. Due to the low administrative overhead of performing these scans with nmap-audit, the predetermined frequency may increase over time for added network safety.

The last political point is to determine who has the final stay on a machine's configuration. In some cases, this step may appear unnecessary (such as with the MySQL server on a workstation), but in other cases this step can become quite tricky (i.e., a Web server running SNMP). Both sides have valid arguments as to whether this service should be running, and it may very well turn into the desires of the security administrator versus the Web admin. However, with a policy in place prior to the audit, such situations should be minimized.

Next, make a copy of "nmap-audit.rc" (which is the example configuration file included in the nmap-audit archive). Open nmap-audit.rc in an editor so that the following site-specific "operational" variables can be modified to suit your environment:

• EMAIL-TO, FROM, SUBJECT -- These variables must be set so nmap-audit will mail the report following its execution.
• IPS -- Use as many times as necessary, specify at most one IP address, IP range, or IP subnet per line.
• MAX-THREADS -- Determine the maximum number of nmap threads that may be run at any given time. Increase this variable with care, keeping in mind the memory requirements, network bandwidth, and CPU usage required by each nmap process.
• DETAIL -- The verbosity of the report sent following nmap-audit execution must be set to either none, low, medium, or high. During an initial audit, the detail level is best kept at low.
• DIR-NAME -- The path to the root directory under which all nmap data will be stored.
• NMAP -- The command line given for the nmap execution. Keep in mind that the output flags must not be used in conjunction with nmap-audit (this means do not use "-o" on the command line).

For more detailed information on the variables' functions and their default values, see either the documentation for nmap-audit or the well-commented example configuration file.

Creating the Audit Filter

The second part of the configuration concerns the ignore rules (the port scan data that nmap-audit should ignore). Nmap-audit comes with default rules describing certain well-known ports and other routine port scan information. Use these default rules for the initial audit, then use the output from the initial audit to write more site-specific rules.

After following the configuration instructions from the previous section, begin the audit and wait patiently for the results. As mentioned, depending on the type of scan used, each host may take as long as 16 hours to complete, though this time can easily be reduced to between 5 and 10 minutes (depending on the number of ports processed and how aggressive the scan is).

What follows is an example first audit, with the rules in nmap-audit.rc shown in Listing 1. For the sake of brevity, the output listings are not shown in their entirety. The audit, as configured in this example, will scan only the well known TCP and UDP ports. To begin the example scan, issue the command nmap-audit --config nmap-audit.rc. The screen should show output like in Figure 1.

Following nmap-audit's successful execution, examine the report mailed to the address specified in the "email-to" directive (Listing 2). At this stage, the report will be quite long; in fact, almost every host scanned is listed with open ports, which is where the attachment "scan_report.csv" comes in (Figure 2). Open this attachment in Excel to begin to shorten this list as quickly and efficiently as possible.

The first ports to examine should be those that occur most frequently, rather than the occasional outliers. To find these ports, select all the data (from the Edit menu) and then sort this data (found in the Data menu). Excel allows the data to be sorted by up to three different parameters, which I typically set to (from most to least significant): port number, protocol, and hostname, as shown in Figure 3. At this point, you should get an idea of some of the most commonly occurring open ports.

In the example data, UDP port 500 is open on about 10 machines (only shown once in the listing for this article). After some research, it was determined that many Windows 2000 servers use the ISAKMP protocol, so that a Windows server has this port open should be of no concern.

To keep ISAKMP messages from appearing in future reports, utilize the "ignore" section of the config file. This section is divided into groups, two of which are already defined in the example file (example group syntax is shown in Listing 3). Groups begin with the "group groupname" tag and end with the "end group groupname" tag, where "groupname" can be any descriptive string to help you remember the group's purpose.

Each group must contain two tag types: a type tag to define which hosts are in the group, and an ignore tag to specify what information the group ignores. The type tag may be ip, hostname, or os, depending on the group properties. In each group, multiple tag types may be used to specify all of the group members. For example, in the predefined "unix" group, you should accept operating systems such as Solaris, Linux, and FreeBSD. Each of these operating systems require its own os tag. Note that though multiple tag types may be used, the tags must all be of the same class, therefore neither an ip tag nor a hostname tag can be used in the "unix" group shown.

Ignore tags contain the actual specification of what information is to be ignored, which can vary from specific to general. All fields of the ignore tag must be supplied, whitespace delimited. If necessary, wildcards -- specified with an "*" -- can be used, although there are some tricky details so be sure to check the nmap-audit documentation for more information. After updating the ignore rule shown above, repeat this step for the other open ports that nmap detected, creating groups as necessary to maintain an organized configuration file.

Initially, port exclusion decisions will come easily, until you are eventually faced with the outliers -- individual machines that just do not have the same properties as the standard groups of servers. Unfortunately, these will likely require individual and very specific groups for nmap-audit to correctly process them.

Regular Usage of nmap-audit

After several manual iterations, as described in the preceding section, the nmap-audit report will eventually shrink to empty; this is the sign of a fully documented and audited network. From this point forward, network audits using nmap will be relatively trouble free.

Before setting this project aside, be sure to set up a cron job to execute nmap-audit at the desired frequency. When initiating a regular audit cycle, keep in mind the length of each nmap-audit run. If the cron jobs are scheduled too close together, problems will arise if a second nmap-audit job begins before the first has completed.

There are two items to consider when using nmap-audit with cron or in another non-interactive environment. Nmap-audit has a "quiet" flag that may be used to suppress status information. Furthermore, be sure to specify the full paths to files and directories both on the command line and in the configuration file. Otherwise, a process that worked flawlessly during the initial audit may suddenly fail once nmap-audit has been set to autopilot.

When there is a change in the network, the nmap-audit reports may again contain data. Since these reports should be quite small, there may be no need to manipulate the nmap-audit output in Excel -- a simple glance at the body of the email message should be sufficient to determine whether this is an acceptable network change that may be ignored in future reports, or if this indicates a problem.

Since all data gathered by nmap is retained, the historical data can be quickly retrieved either via nmap-audit or by reading the actual nmap output, should your site need to investigate a network security incident. One point to remember is that nmap-audit is a wrapper around nmap and, as such, the verbose data generated by nmap will always exist on the scanning machine. To extract the signal from the noise, the information generated by nmap is filtered through nmap-audit prior to producing the report. As such, the raw nmap scan files will still exist should future examination be necessary.

Future Script Goals

The current implementation of nmap-audit is ideal for ensuring that no undocumented services are running on machines throughout the network. A new issue that has begun to present itself on our network requires a slightly different focus for nmap-audit. Rather than merely noting additional services on the machine, information on services that have disappeared will also be documented.

Resources

Nmap -- http://www.insecure.org/nmap/; documentation is available in the nmap man page.

MIME::Lite -- http://search.cpan.org/dist/MIME-Lite/

Nmap-audit -- http://heavyk.org/nmap-audit/; documentation is available via the nmap-audit perldoc.

Keith Resar is a sys admin for the U.S. Government in Washington, D.C.. Prior to that, he was employed by the IT group at Carleton College in Northfield, Minnesota, where he also earned a B.A. in Computer Science.