How to build a dedicated syslog server

Don’t let cloud migration muck up your monitoring

Now that you’ve finally migrated all of your company’s servers and systems to the cloud, you can sit back and relax, with only the local networking hardware to worry about. 

Unfortunately, the local administration and monitoring utilities were hosted on those now-cloudy servers. This is the point at which random pieces of networking hardware will delight in crashing just to torment you, as they know that you'll have no idea why they went down or how to resolve it without the network logging and monitoring system you used to have on hand.

Advertisement - Article continues below

In this tutorial we’ll set up a Raspberry Pi to act as a syslog server, providing a persistent central repository for the logs from all your switches, firewalls and the like. The Pi is perfect for this, as syslog servers need minimal computing power and memory, and it’s so small that it can be located alongside your core switches without getting in the way. 

While this tutorial will focus on using the Pi for this task, all the instructions from step 2 onwards will apply to any Ubuntu Server installation, so you can deploy this on a local hypervisor or in the cloud just as easily.

Step 1 - Install Ubuntu server on your Raspberry Pi

Ubuntu Server is available for Raspberry Pi from the Pi 2 onwards. Instructions on how to deploy Ubuntu server to a Raspberry Pi are available on Ubuntu’s website here

Advertisement
Advertisement - Article continues below

We used a 32GB microSD card, but a smaller or larger one can be used as desired, down to 8GB if your logging requirements are modest.

Advertisement - Article continues below

Once you’ve transferred your Ubuntu image to the SD card, boot the Raspberry Pi for the first time with a monitor and keyboard attached, using a cabled network connection. Once it has booted, a message will appear showing the IP address of the device (assuming you have a DHCP server on the network), and the default login (ubuntu/ubuntu).

Step 2 - Set up users and networking

So the devices we monitor can send their logs to the syslog server, it will need a static IP address. Whilst you could assign it a reserved address on your DHCP server, this would make it dependent on said DHCP server in order to function. As we want our log server to be completely self-sufficient, we’ll assign it a static address and configure it to use that independently of a DHCP server.

Log in from the console using the default credentials. You should first remove cloud-init, as this feature is not needed for this device:

Advertisement - Article continues below

sudo apt purge cloud-init -y

sudo rm -rf /etc/cloud

Next, we’ll configure the static networking details. Use the following commands to tidy up the left over cloud-init network config and create your own.

sudo rm /etc/netplan/50-cloud-init.yaml

sudo nano /etc/netplan/01-static.yaml

Fill in the config file as per the example below, making sure to use the IP address on your local network that you have assigned to the server, and the appropriate gateway and DNS servers for your network.

For reasons of uptime and stability in case of emergencies, you may wish to leave Google’s public DNS servers (8.8.8.8 and 8.8.4.4) in place there

network:

    ethernets:

        eth0:

            addresses:

            - 192.168.1.248/24

            dhcp4: false

            gateway4: 192.168.1.1

            nameservers:

                addresses:

                - 8.8.4.4

                - 8.8.8.8

                search:

                - workgroup

    version: 2

Advertisement
Advertisement - Article continues below

Save the file by hitting Ctrl-O and exit nano with Ctrl-X.

Next, we need to enable password-based SSH logins. This will enable remote access without setting up SSH keys at this point. This server should never handle confidential information, but you may re-enable SSH key based access later should you so choose. Use the following command to open the ssh config file:

Advertisement - Article continues below

sudo nano /etc/ssh/sshd_config

Scroll down to line 56, and change password authentication to yes as shown in the screenshot below. Save the file and exit as before.

Now restart the SSH server:

sudo service sshd restart

Add a user for yourself and delete the default user “ubuntu”. Here, we’ve arbitrarily chosen the username “thorsten” because we’ve written a lot of tutorials and are bored with “admin”, but you should probably just use your regular username. You can add accounts for all the IT admins on your network at this point if you wish.

sudo useradd thorsten -g users -G sudo -s /bin/bash

sudo userdel ubuntu

Set a password for this user with the following command

sudo passwd thorsten

Finally, change the hostname to something more appropriate. We used “syslog” for ours.

sudo nano /etc/hostname

Save the file as before, then reboot the Pi using the following command.

sudo reboot

Once it’s booted, log back in. You can disconnect the keyboard and monitor and continue remotely via ssh from your desktop at this point, if you wish. Log in as the user you just created.

Step 3 - Set up software

Firstly, it's a good idea to make sure the system is up to date at this point by running the following commands.

Advertisement - Article continues below

sudo apt update 

sudo apt upgrade -y

Now it's time to set up the syslog server software. We will be using the rsyslog software, which is installed by default with Ubuntu server, so we do not need to install any extra packages. However, the default configuration disables the network functionality of the software, so we need to edit the config file to re-enable this feature. 

Run the following command to edit the config file.

Advertisement
Advertisement - Article continues below

sudo nano /etc/rsyslog.conf

Uncomment lines 17-18 and 20-21 to enable UDP and TCP syslog reception, then save the file as before and restart the software with the following command.

sudo service rsyslog restart

Step 4 - Setting up devices and assigning log files.

Next, we need to configure the software so it knows which log files it should use to save events sent to it from other devices. We’ll be creating a separate log file for each device that sends its logs to this server.

As network devices can be a bit unpredictable in the way they identify themselves in their log entries, the easiest way to determine this is to set up the device to send its logs to the syslog server and then watch the main syslog, which receives all log events not exclusively directed into other files. Start echoing this file to the terminal on the server by running the following command.

Advertisement - Article continues below

sudo tail -f /var/log/syslog

Now pick one of the devices that will be sending its logs to this server, and enter this server’s IP address as the remote syslog server for the device. For the purposes of this tutorial, we will be setting up logging for a Synology NAS.

From the NAS admin page, we configure the remote syslog server as shown in the screenshot below. We used the default UDP setting, however this syslog server is set up for both TCP and UDP connections. 

Apply the settings to your device, then go back to the syslog server terminal and examine the messages that appear in the syslog. 

After the time and date, the next field is the name that the device uses when logging events. We will use this to feed those logs into a seperate file. In this case the name is “DS211J”.

First, create a subfolder for the device logs to live in:

Advertisement
Advertisement - Article continues below
Advertisement - Article continues below

sudo mkdir /var/log/devices

sudo chown -R syslog.adm /var/log/devices

Next, we need to create a new config file for rsyslog to tell it what to do with these log entries. Use the command below to open a new file for editing.

sudo nano /etc/rsyslog.d/40-devices.conf

Each device needs a pair of lines, one to tell the syslog server what to do with the log entries from this device, and one to tell it to keep processing rules in this file. These are shown in the screenshot below. We named the new log file to match the hostname sent by the device, but the file can be named any way you wish.

Save the file and exit, then restart rsyslog so that it will read the new config file:

sudo service rsyslog restart 

Now send a test message from the device to the syslog server, or wait for it to send some events on its own. The new log file will be created the first time an event matching the pattern in the config file is received. Check that the event is correctly recorded in the new file using the following command, changing the file name to match whatever you called this log file.

Advertisement - Article continues below

sudo tail /var/log/devices/ds211j.log

Additional devices can be configured in the same way.

Step 5 - Keeping things tidy

The last step is to set up a logrotate configuration for the new file. Whilst you can have one logrotate file for all your devices, it’s easier to keep track of things if you have one per device. Use the command below to open a new file for editing

sudo nano /etc/logrotate.d/ds211j

Add the following code into the file, adjusting the file name to match your new log file.

/var/log/devices/ds211j.log {

    rotate 5

    weekly

    compress

    postrotate

    /usr/bin/killall -HUP syslogd

    endscript

}

This will rotate the log files every week, compress the older ones, and keep 5 weeks worth of log files. You can set the rotation pattern that suits your needs.

Optional configuration

If you plan to have a lot of devices logging to this server, or want to keep the logs for a long time, it would be a good idea to add some dedicated storage for the log files, such as a USB memory stick. This can be formatted in FAT32 if you wish, so that it can be read on any computer system if required. 

Advertisement - Article continues below

This has the added benefit of preventing the logs from filling up the root partition, which is a risk with syslog servers that handle large numbers of devices and events.

You can also set up an off-site mirror of the log files if you wish, by adding a scheduled job on the server to ship the logs to a remote system, using rsync or similar.

Featured Resources

The case for a marketing content hub

Transform your digital marketing to deliver customer expectations

Download now

Fast, flexible and compliant e-signatures for global businesses

Be at the forefront of digital transformation with electronic signatures

Download now

Why CEOS should care about the move to SAP S/4HANA

And how they can accelerate business value

Download now

IT faces new security challenges in the wake of COVID-19

Beat the crisis by learning how to secure your network

Download now
Advertisement
Advertisement

Most Popular

Visit/operating-systems/microsoft-windows/355812/microsoft-warns-against-installing-windows-10-may-2020
Microsoft Windows

Microsoft warns users not to install Windows 10's May update

28 May 2020
Visit/security/data-breaches/355777/easyjet-faces-class-action-lawsuit-over-data-breach
data breaches

EasyJet faces class-action lawsuit over data breach

26 May 2020
Visit/security/cyber-security/355797/microsoft-bans-trend-micros-rootkit-buster-from-windows-10
cyber security

Microsoft bans Trend Micro driver from Windows 10 for "cheating" hardware tests

27 May 2020