Watch Multiple Log Files on Multiple Machines
This page describes, in Unix manual page style,
a Perl program
available for downloading from this site
which allows a system administrator to watch entries, as
they are added, in any number of log files on one or more
machines on a network. Logs which transfer to new files
are automatically followed, and an option allows translation
of numeric Internet addresses into the corresponding hostnames
where possible. Log items can be relayed to one or more
other hosts on the network, permitting a machine's local
log files to be monitored there.
logtail -- Monitor multiple log files on multiple machines
logtail [ -dnqu ] [ -ehostname ] [ -l[port] ] [ -pinterval ]
[ -rtime ] [ -stime ] [ -tport ] [ -vlevel ] [ -wcols ]
Long-term survivors in the system administration game know that
an excellent way to spot little emerging mouselike problems
before they mature into moose-sized disasters is keeping an eye
on the various system log files. In days of yore this was quite
simple since almost everything of interest went into the master
Unix logfile and, if a networked system supported the "loghost"
facility, logs for a number of machines on a network could be
consolidated onto a single server.
As more and more services have come into use, log files have
proliferated. Today, an administrator might want to keep an eye
- Master system log
- FTP transfer log
- Web (HTTP) server access log
- Uninterruptible Power Source (UPS) log
- Network Time Protocol synchronisation log
- Automated backup system log
- Firewall suspicious access log
- Logs from local services such as Speak Freely
Look Who's Listening and Echo servers
and various others. Many of the widely-used dæmons for these
functions do not use the common Unix log mechanism, insisting
instead on appending entries to their own private log files. In
the case of commercial firewall, backup, and UPS dæmons, source
code is not usually available, so modifying them to use the
system log is not an option.
When there was a single log file, monitoring it couldn't be
tail -f /var/adm/messages
(or whatever the log file is called on your system). The
-f option causes tail to start at the end of the
file and periodically monitor it for growth, printing any
additions to the file. Unfortunately, tail only monitors
a single file, so the only way to watch multiple files is to
launch separate jobs each running tail on an individual
file--yuck. Worse, to prevent log files from growing without
bound, consuming all available disc space, many programs that
write logs provide a mechanism, automatic or semi-automatic, to
periodically cycle the log to a new file, renaming the previous
log file. Unfortunately, tail -f does not detect that
this has happened, and just waits forever at the end of the old
log, not knowing a new file is now being written. If servers are
run on a number of different machines (or you simply want to
watch Unix log files on all the hosts you're responsible for to
keep an eye out for developing disc problems, etc.), you would
then have to rsh copies of tail -f on every machine
you wanted to monitor, and then make sure the process got
restarted when a machine rebooted.
Enter logtail, which allows you to monitor any number of
log files on any number of machines (assuming they support Perl
with networking and use ASCII log files). Logs from any machine
can be echoed to as many other machines as you like. Log files
are closed and re-opened at a user-defined interval, so cycling
of log files into new files is followed automatically.
- Log messages received from other hosts are normally prefixed with
the hostname followed by a colon, for example,
``wombat:''. Specifying the -d option
causes the full domain name to be shown:
``wombat.marsupials.au:''. If host name lookup on your
machine does not return full domain names for local
hosts, this option will have no effect.
- Messages from logfiles on this host are relayed to a copy
of logtail running on machine hostname.
If you want to only relay messages, and not also print
them locally, specify the -q (Quiet) option as
well. Messages are sent to the named host on the port
given by the -t option, defaulting to port 5741
if no -t option is present. If no copy of
logtail is running on hostname and
listening on the proper port, relayed messages will
silently go to packet heaven with no indication they're
being lost. You can relay to as many other machines as
you wish by specifying multiple -e options; the
same port is used for relays to all hosts.
- Listen for messages relayed from other hosts on the specified
port, or port 5741 if no port is given.
Note that you must specify the -l option in
order to receive messages relayed from other hosts;
otherwise logtail only examines local log files.
- When the -r option is specified, logtail normally
uses fork to spawn a child process to perform
each host name lookup. This blocks processing of other
items until the possibly lengthy host name lookup is
complete. The very act of creating a large number of
child processes may itself, however, cause problems on
certain implementations of Unix. In particular, some
systems leak resources when processes are created and
destroyed which, in time, may bring the system to its
knees. The -n option, specified along with the
-r option, causes logtail to perform host
lookups in the main process instead of spawning a child
process for each. While this may serve as a
work-around on lightly loaded systems, the forced
serialisation of lookups and log item processing will
generally prove intolerable on servers with moderate to
heavy load. Conversely, if your accesses are
exclusively from local machines whose names can be
looked up rapidly, the -n option may reduce
overhead by avoiding the creation of processes for
lookups which can be completed without significant
- To avoid filling up disc drives with huge, ever-growing log files,
many dæmons periodically, either automatically or at
the direction of a user, cycle their log files; the
existing log file is closed and renamed, logging
continues in a newly created file, and the oldest log
file is deleted. When a log file is cycled,
logtail has no way to directly discover this has
occurred. If it continues to watch the old file, it
will not find entries written into its replacement. To
mitigate this situation, logtail periodically
closes and re-opens the files it monitors. This allows
logtail to begin monitoring the new file instead
of the now-renamed older cycle. By default
logtail performs this close and reopen operation
every 15 minutes. To change the frequency, specify the
-pinterval option with interval
the desired time in minutes. Specifying an
interval of 0 completely disables the close and
reopen mechanism. If you know none of the files being
monitored are ever cycled, this will eliminate the
overhead of periodically closing and reopening them.
- Quiet: don't print log items on the local machine. If you want to
relay log items to another host or hosts with the
-e option but not print them locally, specify
the -q option. This makes sense for "headless
servers" administered from other workstations. You can
specify the -q option without the -e
option, but doing so simply consumes system resources
while producing no output whatsoever.
- When the -r option is specified, logtail attempts to
replace any sequences which resemble numeric Internet
addresses (IP addresses) with the corresponding host
name, looked up with gethostbyaddr.
For example, an HTTP log item such as:
172.27.151.132 - - [05/Nov/1997:17:05:12 +0100] "GET / HTTP/1.0" 200 2668
might be translated into:
roswell.area51.af.mil - - [05/Nov/1997:17:05:12 +0100] "GET / HTTP/1.0" 200 2668
If the host name lookup is unsuccessful, the original
numeric address remains intact.
Host names corresponding to numeric IP addresses are
saved by logtail in an internal cache so
multiple accesses from a given host do not require
repeatedly querying the system's host name lookup
mechanism. Entries in the name cache are purged after
time minutes of inactivity, with 10 minutes the
default if no time is given.
At first glance, this host name lookup may appear to be
the niftiest single feature of logtail, but
before you turn it on, ponder carefully the
consequences for the system running the program. If,
for example, you're monitoring the HTTP access log of a
heavily loaded Web site, the -r option may
result in thousands, perhaps millions, of calls to
gethostbyaddr every hour, most of which will
require multiple Internet accesses to resolve the host
name from the IP address. In order to make these
requests without blocking the processing of other
items, logtail performs each of these lookups in
a separate Unix process, but if your system's resolver
is single-threaded, everything may still grind to a
halt due to the resolver bottleneck. Worse, your own
resolver and those of upstream Internet sites may grow
to enormous size due to all the requests cluttering up
its own internal cache.
As long as the volume of log items is relatively small
(say, a couple per minute), and/or the vast majority of
references are to local hosts whose names can be looked
up almost instantaneously, the -r option can
substantially increase the comprehensibility of the
log. But if you switch it on while monitoring a
million-hit-per-day Web site, you're almost certain to
end up regretting it.
- logtail will sleep time seconds between checks for
growth of the files it's monitoring. The default, one
second, is the same used by the -f option of
tail. If the files you're monitoring are only
updated infrequently, a longer sleep time can reduce
the overhead of running logtail. If you've
specified the -l option to listen for messages
from other hosts, choose a sleep time appropriate for
the fastest-arriving stream of relayed messages. If
the sleep time is too long, relayed messages may be
lost if the network input queue overflows.
- Transmit messages relayed to other hosts with the -e option
on port. Note that the same port is used to
relay to all hosts named in -e option
specifications. If no -t option is given, the
default port of 5741 is used.
- Print how-to-call information and a summary of options.
- Verbose: print internal debugging information. You probably won't
find this interesting unless you're modifying the
program. The greater the level, the more
detailed (and voluminous) the debugging information
- Wrap output lines at column cols. By default, lines are
wrapped at column 79. A cols specification of 0
disables line wrapping. logtail attempts to
wrap lines so as to preserve their meaning. Lines are
broken at white space whenever possible. If this
cannot be done (frequently the case in HTTP logs which
contain lengthy CGI GET requests), an attempt is then
made to break the line at a comma or ampersand, major
delimiters in such strings. Finally, if the line still
cannot be broken, it is split at a slash or plus sign.
If none of these delimiters succeeds in breaking the
line, it is simply output unbroken. Continuation lines
are indented to distinguish them from the first line of
the next log item.
logtail is a Perl script. In order to use it, you must
have Perl installed on your system. logtail was developed
using Perl 4.0, patch level 36.
If you're running logtail on a non-System V Unix (for
example, BSD-derived systems such as SunOS 4.x), you may have to
change the definitions of $AF_INET and
$SOCK_DGRAM to agree with those in your system's
/usr/include/sys/socket.h file. Please refer to the
comments at the beginning of logtail.pl which explain this
in more detail and include a small C program you can run to
determine the correct settings for your machine.
The host name lookup (-r option) and relaying between
hosts (-e and -l options) make extensive use of
Unix networking and process control facilities. They are
unlikely in the extreme to work on non-Unix systems, at least not
without a great deal of modification.
The -r option, which attempts to translate numeric
Internet addresses into host names, can cripple a heavily loaded
system by a flood of name server requests. Read the discussion
of whether this option is appropriate to use on your system in
the OPTIONS section above before enabling it.
When logtail begins monitoring a new cycle of a file,
entries written between the time the file was cycled and the time
logtail transferred to monitoring it are not printed; only
subsequently added entries will be seen. This is deliberate: the
HTTP log of a busy Web site, for example, can grow to a very
substantial size in the interval between being cycled and
logtail's discovering it. Processing all the backed up
items may be very time-consuming, and relaying them to other
hosts may run into maximum network packet size restrictions which
would require substantial additional complexity to work around.
Since logtail is intended to provide an overview of system
activity, a few overlooked messages on those infrequent occasions
when files are cycled doesn't seem unreasonable.
Each of the logfile arguments named on the command line is
monitored for growth and any additional material is printed.
Output from logtail is written to standard output, and may
fork(2), gethostbyaddr(3N), perl(1), rsh(1), tail(1)
This software is in the public domain. Permission to use, copy,
modify, and distribute this software and its documentation for
any purpose and without fee is hereby granted, without any
conditions or restrictions. This software is provided "as is"
without express or implied warranty.
by John Walker
November 7th, 1997