skip to content

System: Controlling what logs where with rsyslog.conf

The rsyslog facility is a mystery to most. We're just happy to use the logs provided and don't worry too much about how it all works. Here we take a look under the hood to see what's actually going on and what control we have over our log files.

The following instructions relate to rsyslog version 5.8 and the new version 8.4 which introduces some syntax changes.

Default rsyslog.conf settings

Your default /etc/rsyslog.conf file should contain something like the following:

auth,authpriv.* /var/log/auth.log *.*;auth,authpriv.none -/var/log/syslog daemon.* -/var/log/daemon.log kern.* -/var/log/kern.log lpr.* -/var/log/lpr.log mail.* -/var/log/mail.log user.* -/var/log/user.log mail.info -/var/log/mail.info mail.warn -/var/log/mail.warn mail.err /var/log/mail.err *.=debug;\ auth,authpriv.none;\ news.none;mail.none -/var/log/debug *.=info;*.=notice;*.=warn;\ auth,authpriv.none;\ cron,daemon.none;\ mail,news.none -/var/log/messages *.emerg :omusrmsg:*

The purpose of these settings is to sample a stream of incoming messages and route them as appropriate into different log files (or by other means such as email or system-wide alerts).

The first column is a filter to capture a subset of messages and pipe them into a specific log file, or take other action. The destination log files appear on the right.

For the curious, omusrmsg stands for "User Message Output Module" and combined with '*' will send a console alert to all logged in users. For example, when the system is going down for reboot.

Security Levels

The available security levels are defined in RFC 5424 as follows:

Code Rsyslog Severity
0 emerg, panic Emergency: system is unusable
1 alert Alert: action must be taken immediately
2 crit Critical: critical conditions
3 err, error Error: error conditions
4 warning, warn Warning: warning conditions
5 notice Notice: normal but significant condition
6 info Informational: informational messages
7 debug Debug: debug-level messages

So when we read from rsyslog.conf that /var/log/mail.info receives messages from the 'mail' facility with a security level of "info or higher" what we mean is that the mail.info log also collects entries destined for: mail.notice, mail.warn, mail.err, mail.crit, mail.alert and mail.emerg (so higher in severity), but not mail.debug which is less severe than 'info'.

How the filters work

As described earlier, the column on the left is a filter, and in that column you have entries using the format:

<facility>.<security level>

For example:

auth,authpriv.*
matches all messages from the auth or authpriv facilities.
*.*;auth,authpriv.none
matches all messages from all facilites, but excluding any from auth or authpriv.
mail.info
matches messages from the mail facility with security levels of 'info' or higher.
mail.warn
matches messages from the mail facility with security levels of 'warn' or higher.
*.=debug;
matched messages from all facilities with the 'debug' security level and no other.

You can deduce from this that some messages will appear in multiple log files, which is something we want to keep an eye on and control.

For example, with the above settings, an incoming message with status mail.info will appear in three (3) log files: syslog, mail.log and mail.info. While a message with status mail.err will be logged to five (5) log files, namely: syslog, mail.log, mail.info, mail.warn, and mail.err.

And a message logged from iptables with --log-level 7 will arrive with a status of kern.debug and be logged to three (3) separate log files: syslog, kern.log and debug.

You can monitor these files in real-time from the command-line:

cd /var/log tail -fn0 syslog mail.err mail.warn mail.info mail.log kern.log debug

Prefix your commands with sudo if you're not running as root or don't have read access.

Less duplication in the mail logs

Focusing on the 'mail' facility for now, we don't really need that many copies of every log entry. So let's make some changes.

Old settings

*.*;auth,authpriv.none -/var/log/syslog mail.* -/var/log/mail.log mail.info -/var/log/mail.info mail.warn -/var/log/mail.warn mail.err /var/log/mail.err

New settings

*.*;auth,authpriv,mail.none -/var/log/syslog mail.* -/var/log/mail.log mail.warn /var/log/mail.err

Now we have just two logfiles for 'mail'. The mail.log for all entries, and mail.err for 'warn', 'error', and any more severe messages.

Security level Old settings log targets New settings
mail.info syslog, mail.log, mail.info mail.log
mail.warn syslog, mail.log, mail.info, mail.warn mail.log, mail.err
mail.err syslog, mail.log, mail.info, mail.warn, mail.err mail.log, mail.err

What a difference this makes already on an active mail server!

Less duplication in the iptables logs

Similar to the situation to mail, we have an issue with iptables logging to multiple locations.

At the end of our firewall script we have the following catch-all to LOG details of any access attempts that weren't already either ACCEPT'ed or REJECT'ed by earlier rules:

... $IPTABLES -A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7 $IPTABLES --policy INPUT DROP

With a log level of 7 (debug) this rule generates log messages with a status of kern.debug. And with the default configuration rsyslog will write them to three (3) locations: syslog, kern.log and debug. Let's clean that up.

Old settings

*.*;auth,authpriv.none -/var/log/syslog kern.* -/var/log/kern.log *.=debug;\ auth,authpriv.none;\ news.none;mail.none -/var/log/debug

New settings (old syntax)

*.*;auth,authpriv,kern.none -/var/log/syslog kern.* -/var/log/kern.log kern.debug ~ *.=debug;\ auth,authpriv.none;\ news.none;mail.none -/var/log/debug

New settings (new syntax)

*.*;auth,authpriv,kern.none /var/log/syslog kern.* /var/log/kern.log kern.debug stop *.=debug;\ auth,authpriv.none;\ news.none;mail.none /var/log/debug

We've excluded kernel messages from the syslog, and put a stop on kern.debug messages to prevent them being picked up by subsequent filters. This ensures that our iptables LOG messages now only appear in kern.log and not in the syslog or debug logs.

The syntax for the stop has changed from tilde (~) to the word 'stop' in the latest version.

You'll notice that the minuses have also been removed in the new syntax. They are used in the old version to prevent excess disk writes, but not needed and not used in the new version of rsyslog.

An alternative approach used by some is:

kern.debug /var/log/iptables.log & stop kern.* /var/log/kern.log

This creates a new log file just for iptables debugging.

Remember that any log files you add will need to be rotated using logrorate or a similar facility so they can't grow to fill up the partition.

Restarting rsyslog

For any configuration changes to take affect you need to restart the rsyslog daemon

Under the old 'init' system:

service rsyslog restart

And under the new 'systemd' system:

systemctl restart rsyslog.service

Creating a basic filter

Here's a quick example showing how you can split off certain entries into a new log file. In this case we're separating out spamd log entries from the main mail.log.

Create a file /etc/rsyslog.d/spamd.conf with:

Old syntax

:programname,isequal,"spamd" /var/log/spamd.log & ~

New syntax

if $programname == 'spamd' then /var/log/spamd.log & stop

This will now intercept any log entries matching spamd[PID] and pipe them into a new spamd.log.

After this you should also update /etc/logrotate.d/rsyslog to recognise and rotate the new log file along with the standard mail logs:

... /var/log/mail.err /var/log/mail.log /var/log/spamd.log { ... } ...

More advanced filtering

There are now three types of filters available in rsyslog. We've been working with the "traditional" selectors, but there are also RainerScript-based filters and property-based filters which you can read up on here. They allow for regular expression matching among other things.

rsyslog errors after upgrade

On a separate topic, you may find that after upgrading rsyslog you start seeing errors similar to the following:

May 24 06:25:33 hostname rsyslogd: [origin software="rsyslogd" swVersion="8.4.2" x-pid="1812" x-info="http://www.rsyslog.com"] rsyslogd was HUPed May 24 06:26:39 hostname rsyslogd0: action 'action 17' resumed (module 'builtin:ompipe') [try http://www.rsyslog.com/e/0 ] May 24 06:26:39 hostname rsyslogd-2359: action 'action 17' resumed (module 'builtin:ompipe') [try http://www.rsyslog.com/e/2359 ] May 24 06:55:58 hostname rsyslogd-2007: action 'action 17' suspended, next retry is Sun May 24 06:56:28 2015 [try http://www.rsyslog.com/e/2007 ]

These stem from rsyslog trying to write to a destination that doesn't exist. In particular you're probably not running an xconsole so shouldn't be piping anything to /dev/xconsole.

The fix is to comment out the /dev/xconsole section of rsyslog.conf as follows:

#daemon.*;mail.*;\ # news.err;\ # *.=debug;*.=info;\ # *.=notice;*.=warn |/dev/xconsole

And then restart the rsyslog daemon.

References

< System

Post your comment or question
top