If you missed the overview post, you can see it here. This one is about managing immutable and append-only files on *BSD, Linux, and macOS.
Immutable and Append-Only Files
BSD-derived operating systems (including macOS) and Linux both support the concept of files being made immutable, so that neither their contents nor attributes can be changed. They also both support files being made append-only, so that the existing contents cannot be changed except by adding more data to the end. They do it in slightly different ways.
BSD Implementation
On BSD-derived systems, these features are controlled using file
system flags which have “system” and “user” variants; the former can
only be set and unset by the root user, while the latter can be set
and unset by any user on files they own. The flags are changed using
the chflags command, and their names are schg (system immutable),
uchg (user immutable), sappnd (system append-only), and uappnd
(user append-only). BSD-derived systems also take the further step for
the system-level flags that they can be set but not unset after the
system has booted from single-user mode to multi-user mode. This is
controlled by the system security level (kern.securelevel) which is
raised automatically by the init process from 0 (“Insecure Mode,”
also known as single-user mode) at initial boot to 1 (“Secure Mode,”
the default multi-user mode). The system security level may also be
raised with the sysctl command but cannot be lowered except by
shutting the system down to return to single-user mode
(securelevel=0), which shuts down system daemons and drops network
connections, leaving the system accessible only by the system console
until it reboots. The effect of this is that even the root user
cannot modify the contents or attributes of files with schg (and can
only append to files with sappnd) without shutting down the system.
Linux Implementation
On Linux, by contrast, only the root user can set or unset immutable
or append-only attributes, using the chattr command with +i/-i
or +a/-a to set or unset immutable or append-only attributes,
respectively. But there is no system security level that prevents
unsetting these attributes at any time after they’ve been set. The
Linux attributes are equivalent to the BSD user flags, but limited to
the root user. I have used syslock less on Linux than on OpenBSD,
but have used it fairly broadly on Proxmox and Kali Linux.
macOS Implementation
On macOS, the BSD flags are present but by default the system is
always in “Insecure Mode” (securelevel=0), and Apple has added
additional file flags, notably restricted, which it uses on
operating system commands and libraries as part of its “System
Integrity Protection” (SIP) feature added in OS X El Capitan (10.11)
in 2015. The restricted flag cannot be unset even when the system
securelevel=0, but only when the system is booted into Recovery
mode. While the restricted flag is, like the immutable and
append-only flags, managed by the chflags command, it is not
supported by my tools and it’s just mentioned here to note that macOS
has implemented a very similar capability in a different way. I have
the least experience in using syslock on macOS, and because of that
and the fact that system binaries are protected by this alternative
mechanism, I use it less broadly there.
BSD Security Levels
When I first learned about file system immutable flags, the default
system startup file on OpenBSD named /etc/rc.securelevel contained
lines to set the system securelevel to 1, but that now happens
automatically in the boot sequence by the init process, and the
default rc.securelevel example file in /etc/examples consists only
of comments and is no longer installed by default. All of my systems
have that config file installed to set the securelevel, for reasons
which will be explained shortly.
OpenBSD supports two other settings for
kern.securelevel besides 0
and 1, which are -1 (“Permanently Insecure Mode”) and 2 (“Highly Secure
Mode”). Permanently Insecure Mode prevents the securelevel from being
raised to 1 automatically; it instead goes to 0 and remains there.
Highly Secure Mode features all of the restrictions of Secure Mode, plus
also restricts changes to host firewalls with OpenBSD’s
pf packet filter, allowing
only changes to what IP addresses are in tables but no changes to
filtering or NAT rules. The details of each level are documented in the
OpenBSD securelevel man page.
These restrictions are intended to reduce the impact and blast radius of
both system compromises where an attacker gains root and to reduce the
impact of administration errors.
Security Control, Administrative Safeguard, or Security Theater
The latter goal – reducing the impact of administration errors – is
one that the BSD, Linux, and macOS default settings all support, but the
former goal – reducing the impact and blast radius of system
compromises where an attacker gains root privileges – is only evident
for the BSD system immutable and append-only flags, where even the
root user cannot unset them
so long as the system is in Secure or Highly Secure mode. Some have
argued that these flags are also mere speed bumps or error prevention
(or “security theater”),
on the grounds that they are easily bypassed, which can be done by
returning the system to Insecure Mode. There are two main ways for a
user with root access to do that, which are (1) using console access to
access a root shell in single-user mode after shutdown, which may not be
particularly easy as a remotely connected attacker likely does not have
console access, or (2) modifying configuration files that allow command
execution before the system raises the system securelevel, and rebooting
the system. There are many configuration files and commands executed on
the system during the boot sequence while the system is in Insecure
Mode, and if any of those can be modified to either execute commands or
prevent the system from going into Secure Mode, there is a path to
unsetting the system immutable and append-only flags for the attacker
(at the cost of a potentially noisy reboot). (I have made use of this
path myself in the past while testing immutable flags and getting myself
stuck.) It’s because many files would have to be set immutable to close
off that second reboot path that some have called even the system flags
“security theater,” but I think it is both feasible and it can be a
genuine security control. The key is making it practical to lock enough
of the right files, which is the problem I’ve tried to solve with these
tools.
Origins of syslock/sysunlock
My syslock/sysunlock tool (two opposite functions in a single perl
script, from here on I’ll generally just refer to syslock except
when the distinction matters) is designed to make the management of
all of these implementations of immutable and append-only file system
controls feasible, straightforward, and usable, at least as an error
prevention method and at best as a security control. I originally
wrote it after coming across a simple shell script of the same name
by George
Shaffer, but
it now looks quite different, supporting system and user immutable and
append-only flags on OpenBSD and macOS, and Linux’s near equivalent
(+i/-i and +a/-a). My recommendation for anyone starting out
on a BSD system is to begin with the user flags, which are trivial to
unlock and cause no permanent damage if you need to make changes
quickly. Once comfortable with the group structure, you can consider
adding system flags for files that rarely need to change. I’ll
describe that group structure next.
syslock Groups
The main feature of syslock that makes it usable is that it is
configured to place lists of files and directories into groups, and
those groups can be locked or unlocked with a single command (syslock -g <groupname>). The groups are defined like tags associated
with a list of files and directories, so that a given file or
directory can be in multiple groups. The default configs supplied with
the tool include group names such as etc (files in /etc),
etcrare (files in /etc that are rarely modified), and fstab
(/etc/fstab gets its own group as it’s a painful file to clobber by
accident). There are relatively self-explanatory group names like
binaries, libraries, and system, and then there’s presecure,
which covers the files and directories that are potential targets for
someone trying to find a way to bypass system immutable flags (e.g.,
startup scripts like /etc/rc, /etc/rc.local, /etc/rc.d, and
others that may be less obvious like /etc/sysctl.conf). (There is
no presecure group defined in the Linux or macOS example configs.)
For BSD and macOS, you can choose to make user immutable or system
immutable your default, but you can also use groups to explicitly call
out a set of files as the opposite of your default, with the group
names schg and uchg (and similarly with sappnd and uappnd
groups). These groups are specially handled so that they can be
specified in combination with another group name (e.g., etc:uchg or
acct-logs:uappnd) to identify the files and directories that are in
both.
Since it is possible to enable the schg flag in Secure Mode but not
to disable it, by default neither syslock nor sysunlock will touch
those files while in Secure Mode (sysunlock can’t unlock them), but
the -f (force) option to syslock will lock them. A group can have
both a system and a user subset of files regardless of what default is
configured. For example, if your default is schg but you want some
commonly modified files in /etc to be uchg, you could put them in
that group along with the etc group. Then, if you unlocked files in
Insecure Mode with -g etc, both the schg and uchg files in the
etc group would be unlocked, while if you used sysunlock -g etc
in Secure Mode, only the uchg files would be unlocked. If you were
in Insecure Mode but only wanted to lock the uchg files in the etc
group, you can use -g etc:uchg.
The intention of groups is to provide a mechanism for unlocking
specifically what is needed to perform a specific task. The default
and sample config files supplied with the tool include groups for
other tools that will be covered in this series of blog posts,
including reportnew (blog post June 16), rsync (for rsync-tools,
blog post June 9), and sigtree (blog post June 19), as well as for
process accounting log files (to make them append-only), mail servers,
web servers, and DNS servers (the latter two in the BSD default config
only). Log files and DNS servers provide two examples for where you’d
want to use different flag-specific subgroups. For log files, you
want the active log file to be append-only, but you want the rotated
and archived log files to be immutable; you also want them to use user
rather than system flags if the logs are subject to rotation via
newsyslog or other log rotation mechanism. Thus the sample configs
put the live process accounting log in the uappnd group and the
rotated process accounting logs in the uchg group. The rotation
process needs to unlock both of those subgroups before rotation, and
lock them again after rotation. For DNS configuration files, you might
want to lock your zone files with schg if they don’t change
frequently and aren’t changed by any automation, but if you use
automated DNSSEC signing, you’d lock the signed zone files with uchg
so that they can be unlocked before signing and relocked after
signing. It’s worth noting that logging to a separate machine is a
much better security control than using uappnd (or Linux +a) flags
on log files–but I do both.
Overlapping Groups
Note that it is possible to define groups that overlap with each other
in various ways, and while this is generally acceptable and unlikely
to cause any issues on Linux or if only a single type of flag is used
on BSD or macOS, it can create issues if overlapping groups have a mix
of flag types such that multiple flags get set on any files or
directories. The default and sample configs do not contain any group
definitions that create this problem, and syslock will generate
warnings if it detects cases of conflicting flag types in the
configuration, but will not detect all possible cases (e.g., where
symlinks are involved).
OpenBSD-specific: KARL Features
There is also an implicit group of files identified in the config file
by a leading “!” prefix character, which designates system files that
are part of OpenBSD’s Kernel Address Randomized Link (KARL) feature,
where the kernel and key binaries (currently libc, libcrypto, ld.so,
sshd, sshd-session, sshd-auth, and ssh-agent – the list may increase
between releases) are relinked in random order at reboot. These files
are locked or unlocked by using a -s (for system) option. (This was
perhaps a poor choice of option letter and name, as it is distinct
from the “system” group which is intended to capture key operating
system files and what needs to be unlocked for upgrading and patching,
though I typically unlock everything before upgrading or
patching. This is just to ensure that nothing the system needs to
install is blocked, which can leave a system in an inconsistent and
not fully operable state, but the “system” group should actually work
for this purpose.) Since I use system immutable flags as my default
on most systems, and on the presecure group on all systems, my
practice for patches and upgrades is to shut the system down (enter
Insecure Mode), unlock everything (sysunlock with no arguments),
perform the upgrade (with an extra step) or patch, then lock
everything (syslock with no arguments), then unlock what’s needed
for Karl (sysunlock -s), and then exit, which starts up system
daemons and networking, and returns to Secure Mode, without rebooting
the kernel or resetting the system uptime counter. In my
rc.securelevel file I have the following line:
echo -n " running syslock"; (/bin/sleep 10; /usr/local/bin/syslock -swf) &
The -s and -f options have already been explained, but -w means
to wait for KARL relinking to complete before locking, so as not to
prevent that process from occurring. Typically, I do not unlock with
-s for most instances when I shut down to make changes, and so I do
prevent KARL relinking in those cases, and it doesn’t lead to any
system inconsistencies–it just preserves the previously established
link ordering.
The extra step I mentioned above for an upgrade is that I comment out
the line in rc.securelevel that re-locks the system because at the
completion of a sysupgrade process there will be a reboot after
which I will want to make many additional file changes using
sysmerge to update system configuration files, update packages I
have installed, remove unused binaries and old libraries, and so
forth. I then remove the comment and reboot again. (Also, if a patch
with syspatch rebuilds the kernel, I’ll reboot with the new kernel
after the return to Secure Mode and the KARL process completes.)
Audit Feature
Both syslock and sysunlock have an audit (-a) option, which will
report which files are currently not in the expected state. That is,
syslock -a will tell you all the files which should be locked per
the config but are unlocked, and sysunlock -a will tell you all the
files which should be unlocked per the config but are locked. This
can also be applied to any specific group with -g <groupname>. If used on a BSD system in Secure Mode, the audit
can be restricted to what would actually be changed at the current
security level using the -o (operational restrictions) option. A
-q (quiet) option will suppress all output and just return 0 for
success or 1 (error) for failure; this was created to allow a check to
see if all files in a particular group are unlocked for an
installation to occur for my install.pl tool (to be covered in a
June 12 blog post). With -q, the audit will finish and return at the
first discrepancy found.
Path Prefixes
The configuration file syntax allows listed paths to use three other prefixes in addition to the OpenBSD-specific “!” referenced above. These prefixes, which only have effect on directory paths (and generate a warning but are otherwise ignored on other file types) are:
+ Do not recurse through subdirectories. (No prefix, the default,
means lock the directory and every thing in it, recursively.)
- Do not lock the directory itself, just its contents.
= Lock the directory and its file contents, but not subdirectories.
These cannot be used in combination (though some combinations, like
=-, might conceivably be useful), and while I initially made use of
the first two path prefixes in my configs, they ended up not being
particularly useful with the patterns of groups I developed. I
recently added the = path prefix to address a Linux-specific case.
Linux-specific: Bootloader Considerations
On many Linux systems (notably on Proxmox which uses Debian), the grub
bootloader rewrites files in /boot/grub, (specifically grub.cfg
and grubenv) on reboots. If those files are immutable, the reboot
will fail to update those files. The default Linux config creates a
grub group for /boot/grub and its contents, and uses the = path
prefix on a separate group for /boot and its other
subdirectories. The grub group can then be unlocked before a reboot,
while the broader group is unlocked for kernel updates, and both
locked again afterward. The grub group unlocks slightly more than
strictly necessary (everything in /boot/grub rather than the
directory and the two specific files), but handles the common case
cleanly.
Use of Large Language Models (LLMs)
I’ve made use of LLMs, initially for performing security assessments
with suggested improvements that I’d implement selectively and by
hand, but subsequently for working out the details of design for
prospective changes, writing code, and identifying the causes of
bugs. Each of these capabilities has significantly improved since the
initial use for code assessments in the summer of 2025. Specific
capabilities implemented by Claude include adding append-only flag
support, improving error messages and warnings, code refactoring,
enhancing the audit features and adding the -o option, and drafting
the Github README. Most recently, while I was in the process of
writing this blog post, I revisited an issue I had run into involving
overlapping groups with differing flags, and thought that a new path
prefix that ignored directories might be a good way to handle it. In
addition to that specific use case, for which Claude suggested the use
of = (to mean “lock the directory and files at the same level”), I
asked whether any combinations of existing path prefixes might be
useful or if any other new prefixes might make sense. Claude suggested
that -= or =- might be meaningful and useful (don’t lock the
directory itself, lock the files in it but not the subdirectories),
but I chose not to add the additional complexity without a specific
use case. In the process I identified a bug in how path prefixes were
being handled in previous Claude-generated code related to perl taint
handling, and Claude fixed it along with the implementation of =.
I’ve used ChatGPT and Gemini in addition to Claude for security assessments on the code, and they’ve each identified different issues. While each has found real bugs, my impression is that I’ve seen more false positives from ChatGPT and Gemini than from Claude, though they’ve also found real bugs in code written by Claude.
I’ve found LLMs quite useful for security auditing, refactoring, and error message improvements, as well as writing new capabilities that are relatively straightforward. In the case of implementing append-only flags, it took multiple design, build, and test cycles to get the code to production quality. Overall, forcing myself to explain design choices clearly (and sometimes reconsider them) in the process of using LLMs has been extremely valuable, and has also been a benefit of writing this blog post. I used Claude to identify key topics that should be mentioned in this series, but did not rely on Claude to outline or write the post. Claude and a human editor reviewed this post (and the opening overview post) identify typos and suggest edits.
I’ll include a section like this in each post in the series.
Getting Started and Further Reading
The sample config files supplied with syslock cover OpenBSD, Linux
(with some Proxmox specifics in comments), and macOS, and provide a
reasonable starting point. As noted above, I recommend starting with
uchg groups on BSD and macOS, which gives you immediate error
prevention value and a chance to discover what needs to be unlocked
for your specific regular workflows before committing to more
restrictive schg flags. For Linux, there’s only one kind of
immutable attribute and it’s changeable without a shutdown, so you can
start immediately with the sample config.
As noted above, the syslock audit feature is integrated with my
install.pl tool and syslock groups more generally are a key
feature of both that tool and its distribute.pl counterpart, both of
which will be covered in a blog post on June 12. The wrap-up post on
June 23 will show how all of these tools fit together into a coherent
security architecture.
syslock is available on my
website and on
Github, along with sample
configs for OpenBSD, Linux, and macOS.