24 min to read
Puppety Goodness - Rootless Splunk SC4S
How to take the pain out of deploying SC4S rootlessly using Puppet
Since we now know that Cisco weren’t just trying to pay their Splunk bill, and that their investment means they likely intend to keep developing the platform for the immediate future, it’s about time to talk about how to get data into Splunk using Puppet.
Splunk Connect for Syslog (or SC4S for short) is a containerised, Splunkified syslog-ng server intended to buffer and forward data to one or more Splunk clusters. Data comes in like it would for regular old syslog, SC4S handles it, then ships it out to one or more Splunk HTTP Event Collectors (HECs) to be snarfed up and indexed.
Where the HECs are in your Splunk cluster, how they’re configured, etc. is out of scope for this article, but you will want to be familiar enough with how that all works as you’ll need to know a) where your HECs are in your cluster, and b) what your HEC token is so they won’t shun the non-believer.
Now that you’re armed with the necessary bits of Splunk config, let’s get into the meat of the Puppety goodness.
Table of Contents
- Assumptions
- Architecture
- Puppet modules
- Building a role and profile for SC4S
- Configuring data in Hiera
- Extra configuration
- Puppet code
Assumptions
This article was written with Puppet 8 in mind. It should be compatible with Puppet 7 as well as syntax has stayed relatively the same, but YMMV the further you stray from these waters. If you go off the beaten path and encounter dependency monsters or stray horrors of unimplemented or deprecated features, best of luck (or poke me until I update the article, IDK).
We’ll be using the roles and profiles method when developing our Puppet code as we’re planning to put together something to deploy directly in your environment rather than designing a module for Forge. Profiles help us break up complex configurations into logical blocks, and then roles allow us to tie those together into a full picture of what a node is intended to look like. See Puppet’s documentation for more inforomation.
I’m also assuming your control repository looks pretty much like Puppet’s example control repository. This means Hiera data is under ./data, profiles are under ./site-modules/profile/manifests, roles are under ./site-modules/role/manifests, and the Puppetfile is in the root of the repo where it normally belongs. In the node Hiera we create, I’m also assuming you’re using Hiera to apply roles to your nodes, rather than manually in your site manifest or some other method.
Architecture
Typically SC4S is deployed onto a Linux machine with close to out of the box configuration (which sadly means a small bucket of security poor life choices, such as disabling SELinux or Firewalld and running containers as root). To be fair though, hardening systems is a problem everywhere and we’re at the absolute least going to make sure SC4S is rootless which requires a slight architecture change from the standard next-next-next deployment. Further down under extra configuration we go into some quick and easy goodness with SELinux and Firewalld as well to make life easy.
Our configuration will make use of HAProxy - a very handy piece of software that allows to spin up arbitrary listeners that are really fast at proxying traffic to arbitrary backend servers. HAProxy provides us with a lean, powerful way of load balancing traffic between HEC endpoints, ensuring we don’t end up with data duplication sending all traffic to all nodes all the time, and that if something goes down we don’t just end up with a massive hole in our logs.
As a neat side effect, HAProxy listeners are also quite capable of masquerading inbound, publically-facing ports to whichever ports you like on the other side. This preemptively solves one of the major hurdles of running an SC4S container rootless, which is that unprivileged users may not bind privileged ports (including 514 for syslog).
When our container and proxy are deployed, our configuration at a high level will look something like this:
Puppet modules
This article relies on several modules from Puppet forge. Make sure you add puppetlabs-haproxy and southalc-podman and their dependencies to your Puppetfile. If you are implementing SELinux and Firewalld, make sure you have puppet-firewalld and puppet-selinux along with their dependencies as well.
Building a role and profile for SC4S
Profile skeleton
First we build a skeleton for the Puppet profile.
# @summary Deploy SC4S on a node
#
# @param sen_hec_token
# Sensitive paramater representing the Splunk HTTP Event Collector token used by SC4S to communicate with Splunk.
#
# @example
# include profile::sc4s
#
class profile::sc4s (
Sensitive[String[1]] $sen_hec_token,
) {
}
We declare a single input variable, sen_hec_token, which is marked as Sensitive. This means that the value is redacted in Puppet logs and reports. While I don’t go into how to manage sensitive information in Puppet, later on when this variable is used you’ll also notice it being unwrapped. This is the other side of encapsulating your variable where the original data is returned allowing you to use it in Puppet code. In addition to this, you should also look into securing sensitive data in Puppet so when you store secrets in your hiera, they’re at least encrypted in such a way that Puppet’s private keys are the only way to decrypt them again.
Make sure to set this variable in your node data, yaml or eyaml, as it will be needed when you push this code to production.
User creation
While you can run containers on Linux as root, best practice is to run containers rootless where possible. This is done to prevent an adversary who compromises a container from then going on to take control of the rest of the system by abusing the container orchestrator’s permissions. While we have to trust an external vendor somewhere and at some point, in this case pulling their totally up to date image from GitHub, it’s best to still restrict the container to the minimum set of permissions possible.
The UID and GID of our new user and group don’t matter too much, but they do need to be consistent across your configuration. 10000 has been chosen for this example as it’s way outside the system range and the ranges users and groups are normally be assigned in, so it’s unlikely to clash with anything else.
group { 'sc4s':
ensure => present,
gid => 10000,
}
user { 'sc4s':
ensure => present,
home => '/opt/sc4s',
uid => 10000,
gid => 10000,
password => '!!',
}
The SC4S user’s home directory is set to the same location we’re going to drop all the container’s content and local mounts so that we don’t have any confusion of a service account having a standard user’s home directory under /home (it’s a service account and shouldn’t be logging on like a regular user), and so we can limit where it can access as much as possible.
Preparing directories and files
First we set up and permission the directories SC4S will need when it’s up and running. We create SC4S’ home directory /opt/sc4s first as Puppet’s declarative language is designed to manage every resource by name. This means it doesn’t handle recursive directory creation like an imperative automation system would.
The three additional directories - local, archive, and tls - are mount points on the local filesystem that will be presented to the SC4S container for persistent storage.
file { ['/opt/sc4s', '/opt/sc4s/local', '/opt/sc4s/archive', '/opt/sc4s/tls']:
ensure => directory,
owner => 'sc4s',
group => 'sc4s',
mode => '0750',
}
Finally for preparation work on the node, we declare a template for the SC4S environment file. This file is picked up by the SC4S container when it starts up and provides extra parameters that drive things like where the HTTP Event collector it should send data to lives and what HEC token to use with it.
$env_file = @(EOF)
<%- |
String[1] $hec_token,
Stdlib::IP::Address $network_address,
| -%>
SC4S_DEST_SPLUNK_HEC_DEFAULT_URL=http://<%= $network_address %>:8090
SC4S_DEST_SPLUNK_HEC_DEFAULT_TOKEN=<%= $hec_token %>
SC4S_DEST_SPLUNK_HEC_DEFAULT_TLS_VERIFY=no
file { '/opt/sc4s/env_file':
owner => 'sc4s',
group => 'sc4s',
mode => '0600',
content => inline_epp($env_file, {
hec_token => $enc_hec_token.unwrap,
network_address => $facts['networking']['ip'],
}),
}
One wafer thin include
Our first real piece of configuration is an include for Podman. While all of Podman’s configuration can be performed inside a Puppet profile, it’s essentially a couple of stubs and one giant flexible hash with some arrays mixed in for good measure. As the module explicitly supports it, this means we can simply include the podman module and then deal with the rest of the Podman configuration later in hiera.
include podman
Configuring HAProxy
HAProxy similarly to Podman can be mostly built out with a simple include as it handles most of the complex configuration itself. Aside from including the module, we declare a HTTP listener on port 8090. You can change the listener’s port and the port in the environment file template above if you like, but 8090 shouldn’t interfere with anything Splunk related.
The listener is set to round robin mode, which means it will sequentially send each new connection to the next member of the pool. This prevents SC4S from sending all data to all indexers at all times which will cause data duplication in Splunk and consume expensive license allocation for the month.
We also create a second listener to handle catching syslog traffic arriving on the front end which will pass traffic arriving on 514/tcp and 514/udp through to SC4S on port 2514 via the sc4s container balancer member we create directly below.
include haproxy
haproxy::listen {
default:
collected_exported => false,
ipaddress => $facts['networking']['ip'],
;
'splunk':
ports => '8090',
options => {
'option' => [ 'tcplog' ],
'balance' => 'roundrobin',
},
;
'inbound_syslog':
ports => '514',
;
}
haproxy::balancermember { 'sc4s_container':
listening_service => 'inbound_syslog',
server_names => $facts['networking']['fqdn'],
ipaddresses => $facts['networking']['ip'],
ports => '2514',
options => 'check',
}
The HEC balancer members are declared as a resource collector using the spaceship operator (<<| |>>). This allows other nodes in your environment to export haproxy::balancermember resources with the same tag which nodes with this profile will then detect and add to their own configurations.
A neat feature of collectors which I could not find documentation for at the time of writing is that they also accept defaults which can be defined inside curly braces just like a normal resource. In this case, we set defaults for the listening service we created above, the Splunk port used by HEC, and some common HAProxy options.
Haproxy::Balancermember <<| tag == "splunk_haproxy_backend" |>> {
listening_service => 'splunk',
ports => '8088',
options => 'ssl verify none check',
}
In this instance, I’ve turned off ssl verification as I built this in a lab scenario with no PKI, but if you have certificate trust you can also modify your configuration to validate certificates.
Refreshing SC4S when config changes
The last configuration for the SC4S profile is to set up a relationship between the SC4S environment file and the SC4S container. This is done using the notify operator (~>) which tells Puppet that if the environment file changes, to notify the Podman container. When the container receives a notification from another resource that it’s changed, it refreshes itself.
File['/opt/sc4s/env_file'] ~> Podman::Container['sc4s']
Traditionally this would be done inside resources by setting requires and notify statements, and this still does work, however is is preferred instead to define relationships at the bottom of a profile or class to keep your Puppet code clean and readable rather than having to play a game of spot the resource scrolling up and down your code. When referencing declared resources in this way the names of the resource types are capitalised, and the resources themselves are called by name inside square brackets.
If you want to take control of other SC4S configuration files under /opt/sc4s/local at a later date, make sure to add a refresh for the container on those as well as it will ensure if the configuration changes that SC4S will load it immediately.
Now that we have a working profile, we need to define a role, and then finally fill out hiera data to fill out the content it will drive for us.
Creating a role for SC4S
Once your profile is good to go, you’ll want to add it to a new role alongside any other relevant profiles for your SOE, or other services you want to run on SC4S nodes (represented by the ellipsis below which you will replace).
# @summary Deploy an SC4S server node
class role::sc4s_server () {
...
include profile::sc4s_server
}
Configuring data in Hiera
All of our Hiera data is node data for this deployment, but if you have another broader scope Hiera config to drop the HAProxy and SC4S Hiera that just covers the SC4S nodes in your environment, that works too.
First up, if your site manifest loads classes from Hiera the way mine does, define your role at the top of the file. If you apply roles elsewhere with another method, follow your own internal doco for how to handle that.
---
classes:
- role::sc4s
Next up we configure HAProxy. We don’t want to manage chrooting HAProxy’s directory under /var/lib/haproxy/ as it messes with HAProxy’s ability to generate its own stats file. We also tune defaults to turn on HTTP mode and tune global options to trust the operating system’s bind and server ciphers for TLS. This is particularly important if you’re hardening your environment and want to have HAProxy follow your system configuration.
haproxy::chroot_dir_manage: false
haproxy::defaults_options:
mode: 'http'
haproxy::global_options:
ssl-default-bind-ciphers: 'PROFILE=SYSTEM'
ssl-default-server-ciphers: 'PROFILE=SYSTEM'
Then comes the big bit - configuring Podman. We first carve off a subuid to limit what UIDs can be used with our rootless container. A good explanation of what this mechanism is and how it works can be found here. We also define a volume that will be used for syslog-ng to buffer data while it queues to process and send to Splunk.
podman::manage_subuid: true
podman::subuid:
'10000':
subuid: 1230000
count: 65535
podman::volumes:
splunk-sc4s-var:
user: 'sc4s'
We then declare our container. If you find this familiar, this is because it’s pretty much exactly what you find in Splunk’s Systemd unit file for running SC4S under Podman. Puppet will be managing the container directly however, so putting together a formal Systemd service is unnecessary and makes defining the container and its configuration a lot more straightforward.
podman_containers:
sc4s:
user: 'sc4s'
image: 'gchr.io/splunk/splunk-connect-for-syslog/container3:latest'
flags:
env:
- 'SC4S_CONTAINER_HOST="sc4s"'
env-file: '/opt/sc4s/env_file'
health-cmd: '/healthcheck.sh'
health-interval: '10s'
health-retries: '6'
health-timeout: '6s'
name: 'SC4S'
publish:
- '2514:514'
- '2514:514/udp'
- '6514:6514'
volume:
- 'splunk-sc4s-var:/var/lib/syslog-ng'
- '/opt/sc4s/local:/etc/syslog-ng/conf.d/local:z'
- '/opt/sc4s/archive:/var/lib/syslog-ng/archive:z'
- '/opt/sc4s/tls:/etc/syslog-ng/tls:z'
service_flags:
timeout: '60'
require:
- Podman::Volume[splunk-sc4s-var]
Finally we set lookup options for HAProxy to merge our custom Hiera above with the down into the values used by the HAProxy Puppet module. This means we don’t have to reproduce the entire hash of default and global options here in the node data. As node data is defined in Hiera at the most-specific level in standard configurations, our settings will trump any below them in the heirarchy.
lookup_options:
haproxy::defaults_options:
merge:
strategy: deep
merge_hash_arrays: true
haproxy::global_options:
merge:
strategy: deep
merge_hash_arrays: true
Extra configuration
SELinux configuration
If you have SELinux enabled (which you really should), make sure to also add the following SELinux config to your SC4S profile before you declare your HAProxy listener:
selinux::boolean { 'haproxy_connect_any': }
selinux::port { 'allow-haproxy-frontend':
ensure => present,
seltype => 'http_port_t',
protocol => 'tcp',
port => 8090,
}
This configuration allows HAProxy to bind 8090/tcp and listen for incoming connections.
You’ll also want to make sure that the directories you’re exposing to SC4S via Podman are using the container_file_t SELinux type, otherwise every time Puppet runs, it will try to update it to a user type and restart your container. For the directories we created before, this changes them in the manifest to look like this:
file {
default:
ensure => directory,
owner => 'sc4s',
group => 'sc4s',
mode => '0750',
;
'/opt/sc4s':
;
['/opt/sc4s/local', '/opt/sc4s/archive', '/opt/sc4s/tls']:
seltype => 'container_file_t',
;
}
If you decide to have Puppet take control of the SC4S configuration files such as the metadata that controls which sourcetypes and indexes are applied to processed data, you’ll also want to set this SELinux type on those files as well.
Local firewall configuration
Similarly, if you’re using firewalld, you will also need to add these firewall rules to your profile for inbound SC4S traffic:
include firewalld
firewalld_port { 'Allow sc4s on tcp 8088 in the public zone':
ensure => present,
zone => 'public',
protocol => 'tcp',
port => 8088,
}
firewalld_service {
default:
ensure => present,
zone => 'public',
;
'Allow syslog in the public zone':
service => 'syslog',
;
'Allow TLS syslog in the public zone':
service => 'syslog-tls',
;
}
Depending on your configuration, you may also need an outbound rule. If you wish to accept tcp syslog, you’ll also want a port rule to allow 514/tcp as the Firewalld syslog service assumes 514/udp for syslog only and 6514/tcp for syslog-tls.
HEC node configuration
For each HEC-enabled Splunk node (usually a collection of indexers, heavy forwarders, etc.), add an exported resource for the HAProxy member. This will be picked up by the collector in the SC4S profile and added to its load balancer members.
class profile::some_splunk_node_with_a_hec (){
...
@@haproxy::balancermember { $facts['networking']['fqdn']:
listening_service => 'inbound_syslog',
server_names => $facts['networking']['fqdn'],
ipaddresses => $facts['networking']['ip'],
tag => 'splunk_haproxy_backend',
}
}
Puppet code
Here’s what the full profile looks like (minus documentation above the class definition):
class profile::sc4s (
Sensitive[String[1]] $sen_hec_token,
) {
group { 'sc4s':
ensure => present,
gid => 10000,
}
user { 'sc4s':
ensure => present,
home => '/opt/sc4s',
uid => 10000,
gid => 10000,
password => '!!',
}
file { ['/opt/sc4s', '/opt/sc4s/local', '/opt/sc4s/archive', '/opt/sc4s/tls']:
ensure => directory,
owner => 'sc4s',
group => 'sc4s',
mode => '0750',
}
$env_file = @(EOF)
<%- |
String[1] $hec_token,
Stdlib::IP::Address $network_address,
| -%>
SC4S_DEST_SPLUNK_HEC_DEFAULT_URL=http://<%= $network_address %>:8090
SC4S_DEST_SPLUNK_HEC_DEFAULT_TOKEN=<%= $hec_token %>
SC4S_DEST_SPLUNK_HEC_DEFAULT_TLS_VERIFY=no
| EOF
file { '/opt/sc4s/env_file':
owner => 'sc4s',
group => 'sc4s',
mode => '0600',
content => inline_epp($env_file, {
hec_token => $enc_hec_token.unwrap,
network_adress => $facts['networking']['ip'],
}),
}
include podman
include haproxy
haproxy::listen {
default:
collected_exported => false,
ipaddress => $facts['networking']['ip'],
;
'splunk':
ports => '8090',
options => {
'option' => [ 'tcplog' ],
'balance' => 'roundrobin',
},
;
'inbound_syslog':
ports => '514',
;
}
haproxy::balancermember { 'sc4s_container':
listening_service => 'inbound_syslog',
server_names => $facts['networking']['fqdn'],
ipaddresses => $facts['networking']['ip'],
ports => '2514',
options => 'check',
}
Haproxy::Balancermember <<| tag == "splunk_haproxy_backend" |>> {
listening_service => 'splunk',
ports => '8088',
options => 'ssl verify none check',
}
File['/opt/sc4s/env_file'] ~> Podman::Container['sc4s']
}
Here’s the hiera for my SC4S instance (minus any unrelated config particular to my home cluster):
---
classes:
- role::sc4s
haproxy::chroot_dir_manage: false
haproxy::defaults_options:
mode: 'http'
haproxy::global_options:
ssl-default-bind-ciphers: 'PROFILE=SYSTEM'
ssl-default-server-ciphers: 'PROFILE=SYSTEM'
podman::manage_subuid: true
podman::subuid:
'10000':
subuid: 1230000
count: 65535
podman::volumes:
splunk-sc4s-var:
user: 'sc4s'
podman_containers:
sc4s:
user: 'sc4s'
image: 'gchr.io/splunk/splunk-connect-for-syslog/container3:latest'
flags:
env:
- 'SC4S_CONTAINER_HOST="sc4s"'
env-file: '/opt/sc4s/env_file'
health-cmd: '/healthcheck.sh'
health-interval: '10s'
health-retries: '6'
health-timeout: '6s'
name: 'SC4S'
publish:
- '2514:514'
- '2514:514/udp'
- '6514:6514'
volume:
- 'splunk-sc4s-var:/var/lib/syslog-ng'
- '/opt/sc4s/local:/etc/syslog-ng/conf.d/local:z'
- '/opt/sc4s/archive:/var/lib/syslog-ng/archive:z'
- '/opt/sc4s/tls:/etc/syslog-ng/tls:z'
service_flags:
timeout: '60'
require:
- Podman::Volume[splunk-sc4s-var]
lookup_options:
haproxy::defaults_options:
merge:
strategy: deep
merge_hash_arrays: true
haproxy::global_options:
merge:
strategy: deep
merge_hash_arrays: true