Updated on September 9, 2020 at 12:23 am
As I continue to tweak my network infrastructure one of the items on my list is to have Splunk forward somewhere offsite in case my whole local network gets ransacked. At least I might have logs of what happened.
Setting up two servers for forwarding and receiving is relatively straightforward using the Splunk web interface. However the important detail for me was making sure the forwarding was happening securely to the remote Splunk server.
As I mentioned setting up the initial forwarding and receiving is fairly easy and can be accomplished in a few steps with the web interface.
First, you need to set up your local input on your local server for capturing logs, in my case this is syslog for the moment. So I have a UDP port listening on 514 for incoming data.
Second, you need to configure forwarding under “Forwarding and Receiving”. Simply point the local forwarding server at your remote receiving Splunk server on port 9997 which I believe is the standard port used in this scenario. Now in my case I also set Forwarding defaults to store a local copy of events.
Third, on your remote server you need to configure receiving which is simple enough, you just set the receiving port to 9997.
Now that will get you going in short order, however you will be transmitting logs to the remote server in the clear and in my case that is over the internet. So we want to get this connection to the remote server secured and we will do that using SSL, however this needs to be performed by editing a couple of .conf files and can’t be set up through the web interface.
We will start with the local server first (the forwarder). We will need to edit the outputs.conf file. This will be located under “splunkinstalldir/etc/system/local/” and need to have something similar to what is below:
[tcpout]
defaultGroup = default-autolb-group
indexAndForward = 1
[tcpout:default-autolb-group]
compressed = true
server = splunk01.somewhere.net :9997
sslCertPath = /opt/splunk/etc/auth/mycerts/splunkservercertificate.pem
sslPassword = password
sslRootCAPath = /opt/splunk/etc/auth/mycerts/splunkcacertificate.pem
sslVerifyServerCert = false
And on the remote server under the same file path as above, we will need to edit the inputs.conf file as below:
[default]
host = splunk01
[splunktcp-ssl:9997]
compressed = true
[SSL]
password = password
rootCA = /opt/splunk/etc/auth/mycerts/splunkcacertificate.pem
serverCert = /opt/splunk/etc/auth/mycerts/splunkservercertificate.pem
Once you have modified these two files and restarted Splunk on both servers, you should now be forwarding your logs securely to your remote server.
A couple things to note:
Once you restart Splunk the password field will be encrypted in both .conf files. Also you will either want to create your own self-signed certs or use certs created by a trusted certificate authority.