Now that I’ve covered two building blocks –
syslock for managing
immutable
flags
and rsync-tools for moving files around – in this post I’ll discuss
more recent tools that put those together in order to manage the
distribution of files across my systems. I originally looked at using
something like puppet or
chef to manage configurations
on my hosts, but they both seemed overly complex for my needs and
neither was built to handle my environment with immutable files. I
decided to build a tool to install updates, thinking that perhaps
I could set it to run automatically on a reboot by calling it in
/etc/rc.securelevel before
the system securelevel is raised, but in practice I’ve just used
it manually.
distribute.pl
This script packages up components into gzipped tar files (unless
they are already in tar files as signed packages in OpenBSD format),
determines what hosts they need to be delivered to, and then ships
them off using rsync (with rrsync on the receiving end to limit
access to only /var/install, which is where install.pl will expect
to find items to install). Originally the list of files that the
script would distribute was specified in the code, but I split it out
to a separate config file as I made more frequent changes. There are
three types of file that can be distributed: plain files (which can be
any type of file), packages (OpenBSD format), and “custom.” The last
of these is intended for files that require some custom processing
before delivery, which is currently implemented by adding additional
subroutines to the code but could be extended in the future to allow
the use of external code, perhaps similar to something I implemented
for reportnew, which will be covered in the next blog post. For
individual perl scripts, config files, and X.509 certificates, I use
“plain”; for OpenBSD packages and all of the scripts discussed in this
blog series, I use “package,” and I have two implemented “custom”
distributions: (1) “doas,” which takes a doas.conf.template file
containing all of the information required to generate doas.conf
files for all of my hosts, and then distributes the resulting
generated config file for each host. (2) “ip-address,” which I use
when the dynamically assigned IPv4 addresses on my WAN connections
change to update all of the config files on remote hosts which
reference those addresses. One of the things that happens when my
IPv4 addresses change is that I lose IPv4 access to my cloud-hosted
servers; distribute.pl will automatically use IPv6 to distribute the
changed config files in this case. (It also takes -4 and -6
options to force IPv4 or IPv6.)
Each entry in the config file specifies a name (the name used to specify
the file in an argument to the script), a file name (the source
location), a destination location, a type (plain, package, or custom),
and the hosts to which that file should be distributed (which may be
“all” or “all except” followed by a list of one or more hosts). An
optional “syslock-groups” field identifies which
syslock groups must be
unlocked for the installation to occur. For packages, instead of a
source pathname, the “file” field is the name of the package followed by
a hyphen and the letters “PKG” – distribute.pl will look in the default
OpenBSD location where new ports are built to find the most recent
version of signed package to distribute; you can override the location
in the config file.
For plain and custom files,
distribute.pl will prompt for
the passphrase for the signing key (default is
$DOMAIN-$YEAR-pkg.sec in
/etc/signify and can be
overridden in the config file or with the
-k option to specify an
arbitrary key or -p option to
use the prior year key for your domain, but use of other format key
names will generate a warning). I protect my signing key passphrase by
keeping it in KeePassXC. KeePassXC is protected with a passphrase (which
could be intercepted by a keylogger) but also by a YubiKey as a key file
equivalent, which KeePassXC queries using the HMAC-SHA1 challenge
response function. The key file data never leaves the YubiKey. I have a
primary and secondary YubiKey configured with the same HMAC-SHA1 secret
so that I don’t lose access to my passwords if I lose my primary
YubiKey.
For packages, no additional signature is required (but see below for
what will generate warnings from
install.pl.)
If you want to distribute a config file to only a few of a set of hosts
that would normally receive it, you can specify an individual or
comma-separated list of hosts on the command line using
-h.
The config file may define macros of the form
macro_name="macro_value"
which can be used in the config file as
%%macro_name%%; there are
also special variables $HOST, $SIGNIFY_PUB_KEY, and
$SIGNIFY_PUB_KEY_NEXT which can be used; the latter two are for
distributing the public key files themselves to other hosts. The script
will warn beginning on December 1 that the next year’s key should be
distributed.
Packaged with distribute.pl
is gendoas.pl, the script
which generates doas.conf
files from a template file which specifies which sections are applicable
to which hosts using comments of the form “# hosts: sudo
was the ability to have a single
sudoers file for all hosts;
this gives me a close equivalent, I have only one file to maintain.
(It’s worth noting that doas
can be used on Linux with the
opendoas package.)
install.pl
The other component of this system is
install.pl, which does the
actual installation of files in
/var/install, provided there
are public keys for those signing keys in
/etc/signify. Packages will
be installed from any source signed with such a key, but plain and
custom files will generate a warning and user prompt if the key is not
the current or prior year key for your own domain. The signatures are
checked both before and after reading the
tar file to mitigate
time-of-check/time-of-use race conditions. If
syslock exists on the system,
then install.pl will unlock
the “local” group by default and any other specified
syslock groups. But if the
system is a *BSD system running at securelevel > 0, the install will
not proceed unless the -f
(force) option is used, since it would be unable to unlock anything with
system immutable flags. With
-f it will attempt to proceed
on the assumption that any system immutable flags that might be in the
way have already been unlocked and will see if there are any user
immutable flags in the relevant groups and unlock them if necessary, and
then will re-lock anything that it unlocked at the end (i.e., if system
immutable flags would have been in the way but were manually unlocked,
they will be left unlocked).
Packages are installed using OpenBSD’s
pkg_add if the destination
system is an OpenBSD system, but otherwise
install.pl includes a minimal
implementation of both
pkg_add and
pkg_delete functionality so
that OpenBSD-format packages can be installed on Linux and macOS systems
if they are packages that work on any architecture. The
pkg_delete functionality is
used to remove an old version of a package before installing the new one
(it will not delete installed configuration files unless they have
remained unchanged from the default). This custom implementation also
checks to see if there is a version of a config file in the package with
“macos.” or “linux.” on the front of it, and installs the appropriate
config file for the operating system in use. For perl modules, instead
of installing in
/usr/local/libdata/perl5/site_perl
(OpenBSD), it will install in
/usr/local/lib/site_perl
(Linux) or
/Library/Perl/Updates/<perlversion>
(macOS; I might want to add recognition of use of Homebrew in the
future). Like on OpenBSD, the installed package is registered in a
directory in /var/db/pkg
listing the contents of what was installed. One feature of OpenBSD’s
pkg_add that is NOT
implemented is a check for file collisions from an installation, but it
will not overwrite an existing installed config except in the case where
you are upgrading from an older version to a newer one and the existing
config has not changed since it was installed (i.e., it has the same
size and SHA256 hash). The upshot of this is that once you modify the
config it won’t get clobbered by a new installation even if the package
registration isn’t present on the system. The new config always goes
into the
/usr/local/share/examples/<package>
directory.
Included with install.pl is
pkg_info.pl, a minimal
implementation of OpenBSD’s
pkg_info, which lists
installed packages or displays information about a particular installed
package.
One last thing that install.pl does is add an entry to
/etc/CHANGELOG with the datee
showing what exactly was installed.
Signify.pm
A perl module that is used for the cryptographic functions in
install.pl and distribute.pl is Signify.pm, a wrapper for
OpenBSD’s signify command to allow signing and verification of
individual files with detached signatures (a separate file containing
the signature) and to allow signing and verification of gzip files,
where the signature is embedded in the gzip header (as produced by
OpenBSD’s pkg_sign). This works on Linux (with the signify-openbsd
package) and macOS (with signify-osx, available via
Homebrew). Although in some scripts I used to use GPG for signatures,
I switched to signify at some point after it became available (in
OpenBSD 5.5) as it is much lighter weight and uses Ed25519 digital
signatures. While this is currently a source of strong protection
against classical (non-quantum) attacks, it is not resistant to
quantum cryptography and should soon be supplemented by ML-DSA or
SLH-DSA. There is an open source ml-signify in Rust that uses
ML-DSA, but there is no official OpenBSD port of it; it’s something to
look at in the future. (This is presently an OpenBSD ecosystem
limitation, ML-KEM support is not yet present in LibreSSL, for
example.) This perl module is a component used by all of the
remaining tools that will be covered in this series.
Use of LLMs for Security Assessment
All of these scripts have been reviewed for security issues with LLMs,
which led to improved error checking (in particular to correct a latent
bug in Signify::sign_gzip
where the success of copying the signed gzip was not checked), a fix to
a bug in the code for finding the latest version of a package. Claude
was particularly helpful in working out the details of how to integrate
with syslock in a more useful
way using a variant of the -a
(audit) flag combined with
-q, which returns as soon as
it finds the first inconsistency with the desired state, used to verify
that there are user immutable flags to unlock, as well as in expanding
and improving the minimal implementation of
pkg_add built into
install.pl so that it sets
file permissions as well as timestamps, and properly installs per-OS
config files.
Getting Started and Further Reading
These tools are available on my
website and on
Github. A
sample distribute.conf config
file is provided and the Github
README contains more details
about usage and implementation. All of these were written initially for
OpenBSD but also work on Linux and macOS. A quick way to get started is
to put install.pl on a system
and then use it to install the OpenBSD package versions of the tools,
which will install the tools, the config file appropriate to the
operating system, and all example config files (if any) in
/usr/local/share/examples/<package>.
The next post will cover
reportnew, a tool for log
monitoring that is most noteworthy for handling process accounting logs
on OpenBSD, Linux, and macOS. That post and the next one, on
sigtree.pl, involve the most
extensive use of LLMs in designing and implementing privilege separation
into those tools.