The next tool in the series, reportnew, is the oldest (first written
in February 1999) and one of the largest and most complex (currently
almost 4,500 lines of code, though it was only 500 lines in June 2003).
Much of that gain has been in the last year; in May 2025 it was just
over 2,400 lines. In this post I’ll describe why I wrote it, what
it does (so far as I am aware) uniquely, and how I use it.
Origins of reportnew
I had originally looked at using swatch (or “simple
watcher,” now called swatchdog), which was first developed in 1993
to help automate UNIX syslog monitoring. It had a per-log-file
configuration with a mix of “watchfor” and “ignore” directives that
take regular expressions, and actions that include displaying the
match (“echo”), sending email (“mail”), beeping (“bell”), sending
messages to screens of logged-in users (“write”), and executing a
command (“exec”) or piping the output to a command (“pipe”). I decided
I wanted to do things slightly differently–I wanted the matching and
exclusion via regular expressions, I wanted email notifications, but I
wanted one setup for all my logs and I wanted it to run periodically
rather than continuously (see “How It Works,” below). My goal: to have
something that would identify anomalous logs and bring them to my
attention to see if anything of consequence was occurring, whether
that was a failure of some kind, a misconfiguration, a problem caused
by an update, or something indicative of an attack.
I also wanted support for the logs produced by the mail and DNS
software I was using at the time (Dan Bernstein’s qmail and
djbdns), which used a log format known as cyclog (subsequently
multilog). The cyclog log files consisted of a directory named for the log
file, containing the current active log in a file named current and
rotated logs in filenames that were named as two decimal numbers separated by a
dot, the first component seconds and the second microseconds.
Timestamps in the logs used the same format. The multilog format
changed the timestamps to TAI64N hex strings which were an @-sign
followed by 24 characters of lowercase hex digits, and the rotated
logs were named the same way. I wanted my log monitoring program to
support these file formats and to convert the times to human-readable
strings. I implemented cyclog support on August 11, 1999 and multilog
support on August 26, 1999 (and Bernstein made the switch from cyclog
to multilog with the release of daemontools 0.60 on August 24,
1999).
A few years later, I decided that I also wanted to be able to build a
single config for all my hosts, which I first implemented in February
2003 with “begin-host:
The original implementation only permitted a single “match”, “exclude”, and “action” statement per log, which proved too limiting, so I added support for an arbitrary number of them in March 2013.
But the key unique (so far as I am aware) feature that I added was support for UNIX process accounting logs in July 2013.
Process Accounting
*BSD systems optionally will run process accounting, which adds a log entry for every process that runs on the system containing the command executed, the user who ran it, the time at which it was executed, and how long it ran and how much CPU it used, along with the process ID. Historically, this functionality was built for billing purposes so that users (or their departments) would be billed for how much CPU and memory usage. It’s also a way to see what commands are executed by which users at which times, as well as to see, via process flags in the accounting records, whether the process forked another child process, crashed and dumped core, and, on OpenBSD, whether there were any pledge or unveil violations or other memory violations.
The initial implementation of supporting process accounting
logs in reportnew was a bit of a hack–it used the system
lastcomm command to output to a temp file (which is in
reverse chronological order), reversed it to make it chronological,
and used the timestamps to find the last entry already reported
as the point at which to start checking. In 2024, I updated this
to parse OpenBSD process accounting logs directly, which showed
how unreliable the prior method had been, since process accounting
records aren’t written until the process terminates–so they aren’t
all necessarily in chronological order.
In the course of building the functionality to parse the binary
records directly for Linux (and macOS), I wrote a perl implementation
of lastcomm (but which displays the records chronologically instead
of in reverse) and ensured that its output matched the system command
for each line. I also used that tool in developing direct support of
process accounting files on Linux and macOS, and gave it the ability
to generate output in either BSD or Linux format. That lastcomm.pl
script is included with reportnew.
I define rules for my process accounting logs so that I get alerts
when any command is executed by a user that shouldn’t exist, when a
user (such as a system account) executes a command it shouldn’t
execute, when there are flags associated with a command that shouldn’t
be there (pledge and unveil violations, memory errors, core dumps),
and when commands are executed outside of expected time windows (see
below). This functionality helped me identify three bugs in
OpenBSD that were corrected in the latest release (7.9): one that caused
periodic crashes in the spamd daemon (fixed in CVS revision 1.164), one that caused periodic exits of the
sensorsd daemon (fixed in CVS revision 1.70), and one that caused the calendar command’s
mail functionality to fail due to an unveil violation (fixed in CVS
revision 1.41).
These rules are made feasible in part by defining macros for different categories of commands that are related to each other, so that they can be used in exclude rules rather than listing every single command uniquely for each user. The default config file defines command category macros for OpenBSD, Linux, and macOS, with the macOS ones being the most extensive (but still incomplete).
Macros
In late 2023 I added macro functionality (the macros described in the
blog post on distribute.pl were patterned after these), of the
form macro_name="macro_value"[:post-processing-tag], which were divided into pre-processing
macros (expanded inside match and exclude rules) and post-processing
macros (used for enriching output by either having the macro name appended
to matches of the value in output (tag “append”) or the value being replaced
by the macro name (tag “substitute”). I use append macros for things like
IP addresses and MAC addresses, and substitute macros for things like certificate
fingerprints and SSH key fingerprints, to make output in notifications more
understandable. And, as already noted, they are indispensable for making
use of process accounting log monitoring, to exclude expected activity and keep
the number of alerts manageable and relevant.
In mid-2025 I extended the functionality to allow importing macro
files into the config from external files, and optionally to require
those macro files to be signed with a signify signing key. This
allows me to keep my main configs locked (using syslock, see blog
post) on each OpenBSD host with system immutable flags while still
updating particular macros in a secure manner by signing them and
distributing them to the hosts that use them (using rsync-client.pl,
see blog
post).
Session Matching
In 2017 I switched my mail servers from Postfix to OpenBSD’s OpenSMTPD. One side-effect
of that change was that mail logs now spread a lot of information that had previously
appeared in individual log entries across multiple log entries, which were identified
by hex identifiers that showed the logs were part of the same session. I lived with it
for a while, but eventually got annoyed by log alerts that lacked sufficient context
and required me to go find the relevant context, so in early 2020 I added session
matching functionality to reportnew by extending the match and exclude rule syntax
to add “session-with” to “match” and “session-without” to “exclude”. Here is an example
from the default config that reports entire OpenSMTPD sessions when any line in the
session indicates a failure or an unexpected certificate:
log: /var/log/maillog
match: session-with /[a-f0-9]{16} (?:mta|smtp)/
exclude: session-without /([a-f0-9]{16}) (?:mta|smtp) (?:.*reject|warn|error|panic|deny|fail|timeout|fatal|invalid|bad signature|tls_ciphers=(?!%%TLS_1_3%%|%%TLS_1_2%%)|cert-check result=\"(?:valid|verified|unverified)\" fingerprint=\"(?!%%knownfingerprint1%%|%%knownfingerprint2%%).*\"))/
action: notify admin@domain
The way this works is that the “match” rule collects all logs which contain hex strings followed by either “mta” or “smtp”, which all of the logs for each mail session do. The “session-with” is an instruction to collect matches for subsequent “session-without” comparisons. Next, the “session-without” regular expression is used to identify what is of interest (in this case, various kinds of error or failure), and any sessions without a string that matches are excluded. The capture group is used to find all of the collected lines from the “session-with” matches that correspond, and they are reported together.
Linux and macOS Support
In August 2025 I finally made a long-overdue switch from using VMware
to using Proxmox for my home virtual server hosting. This prompted me
to make all of my tools work better (or at all) on Linux. In September
2025 I updated the old lastcomm-based process accounting checking to
work for Linux’s output format and I added support for Linux journal
files for monitoring so that I could monitor the same logs and daemons
I monitor on OpenBSD on Linux. It was this that also prompted me to
write the lastcomm.pl script described above in the process accounting
section, which I used to implement direct process accounting log
processing for Linux and macOS as well. From that point on, I’ve made
a point of supporting OpenBSD, Linux, and macOS for all subsequent
features.
How It Works
In the “Origins of reportnew” section above, I mentioned that I wanted
it to run periodically, not continuously. Here’s a quick explanation
of how that works. reportnew maintains a state file (in
/etc/reportnew/reportnew.size, though this location is configurable)
with one line for each log monitored which includes the offset and
SHA256 of the first line in the log, as well as the last check time
for that log file. On each run of reportnew, it looks for rotated
logs with a modification time later than the last check time for that
log, and, if the earliest of those has a first line SHA256 hash
matching the stored value, starts from the saved offset. If it doesn’t
match, then it starts from the beginning of that file. Processing
then proceeds through all of the subsequent rotated log files and
through the end of the current log file in order. If there are no new
rotated log files and the current log file is shorter than the stored
offset, reportnew assumes the file has been truncated or replaced
and restarts from the beginning of the file. On the first execution
for a new log, all rotated logs are processed. While a determined
attacker could modify this state file to disrupt its function, this
implementation handles normal file system events including log
rotation and truncation.
Use of LLMs for Security Assessment and Feature Development
As with all of my scripts, I’ve used large language models (LLMs) to do security
assessments on the code. With reportnew, I also used LLMs to add features I had
been thinking about for years but had never gotten around to implementing. The
first and largest of these (accounting for most of the increase in size in
November 2025) was privilege separation. The idea behind privilege separation
is to divide the functions of the script into those which require root
access (e.g., opening log files) and those which do not, and having the
script fork a child process that drops all privileges and makes requests
of the privileged parent process for the specific functions that require
root access. I designed this functionality using Claude and implemented it in
phases over the course of six days during November 2025.
Privilege separation is enabled by either using “privsep: yes” in the config
file or using a -p option to the command, and works on OpenBSD, Linux, and
macOS. On OpenBSD, pledge functionality is also used to reduce capabilities
to the minimum required for both the privileged parent and the nonprivileged
child processes. Note: reportnew can be run by a nonprivileged user on
world-readable logs, which includes process accounting logs on OpenBSD and macOS
but not on Linux, where they require privileged access.
The next feature added using LLMs followed shortly after this and was
related–the ability to execute external scripts based on matching log
entries. This was implemented to drop privileges if run as root and
privilege separation is not in use, and to execute as the
nonprivileged process if privilege separation is in use or if run as
non-root. Further, scripts would only be executed if signed and
located in /etc/reportnew/scripts. It’s also intended that any
scripts only have write access to /etc/reportnew/scripts/logs, which
for the sample scripts provided is enforced in those scripts
themselves on OpenBSD using unveil.
A final feature added using LLMs is time windows – the ability to define
rules to be applicable only during particular times. This was implemented
by adding an optional “times:” field to a match/exclude/action triplet,
the value of which is either a named time range or its negation. The named
time ranges are defined in the config using a “define_time:” field which
specifies a name and one or more time specifications which are explained
with numerous examples in the sample config and the README on Github.
The way I use this feature is to divide match/exclude/action rules into
two sets–rules for times when I expect certain activity, and rules for
times in which I don’t expect that activity. For my _rsyncu user activity
using the SSH keys associated with backups, I expect that to happen during
backup windows, so I ignore it with “exclude” during those times; but I
don’t expect it to occur at any other times so I don’t exclude it during
those times. For my own logins, I don’t expect them to occur outside of
my normal waking hours, so I report them if they occur at other times.
Similarly for process accounting rules and specific commands that I
expect to be limited to specific time windows.
One final feature developed with the help of LLMs is the large set of macOS command category macros that are provided with the sample config. I began by building macros based on what I knew, but found there were a very large number of macOS processes that I wasn’t familiar with. I gave a collection of process accounting logs to Claude and asked it to group items into categories for admin users, non-admin users, and root. As I come across new alerts from my own laptop from time to time, I give those to Claude for categorization. The current set in the default config is a subset of macOS and third-party software that resulted from my own use, but is likely fairly reliable for most of the system service accounts. For an individual user, an admin user, or root, you’re likely to see unwanted alerts if you use software that I don’t use.
Getting Started and Further Reading
reportnew is available on my website and on Github. A sample
reportnew.conf file is provided and the Github README has more
information about usage and implementation.
Next up will be the final post before the wrap-up, which will be about
file integrity monitoring with sigtree.pl, the largest script and
the one that I added privilege separation to during the last week of
2025 using Claude, which took 83 revised designs before completion.