System: Controlling what logs where with rsyslog.conf
The rsyslog facility is a mystery to most. We're just happy to use the logs provided and don't worry too much about how it all works. Here we take a look under the hood to see what's actually going on and what control we have over our log files.
The following instructions relate to rsyslog version 5.8 and the new version 8.4 which introduces some syntax changes.
Default rsyslog.conf settings
Your default /etc/rsyslog.conf file should contain something like the following:
auth,authpriv.* /var/log/auth.log
*.*;auth,authpriv.none -/var/log/syslog
daemon.* -/var/log/daemon.log
kern.* -/var/log/kern.log
lpr.* -/var/log/lpr.log
mail.* -/var/log/mail.log
user.* -/var/log/user.log
mail.info -/var/log/mail.info
mail.warn -/var/log/mail.warn
mail.err /var/log/mail.err
*.=debug;\
auth,authpriv.none;\
news.none;mail.none -/var/log/debug
*.=info;*.=notice;*.=warn;\
auth,authpriv.none;\
cron,daemon.none;\
mail,news.none -/var/log/messages
*.emerg :omusrmsg:*
The purpose of these settings is to sample a stream of incoming messages and route them as appropriate into different log files (or by other means such as email or system-wide alerts).
The first column is a filter to capture a subset of messages and pipe them into a specific log file, or take other action. The destination log files appear on the right.
For the curious, omusrmsg stands for "User Message Output Module" and combined with '*' will send a console alert to all logged in users. For example, when the system is going down for reboot.
Security Levels
The available security levels are defined in RFC 5424 as follows:
Code | Rsyslog | Severity |
---|---|---|
0 | emerg, panic | Emergency: system is unusable |
1 | alert | Alert: action must be taken immediately |
2 | crit | Critical: critical conditions |
3 | err, error | Error: error conditions |
4 | warning, warn | Warning: warning conditions |
5 | notice | Notice: normal but significant condition |
6 | info | Informational: informational messages |
7 | debug | Debug: debug-level messages |
So when we read from rsyslog.conf that /var/log/mail.info receives messages from the 'mail' facility with a security level of "info or higher" what we mean is that the mail.info log also collects entries destined for: mail.notice, mail.warn, mail.err, mail.crit, mail.alert and mail.emerg (so higher in severity), but not mail.debug which is less severe than 'info'.
How the filters work
As described earlier, the column on the left is a filter, and in that column you have entries using the format:
<facility>.<security level>
For example:
- auth,authpriv.*
- matches all messages from the auth or authpriv facilities.
- *.*;auth,authpriv.none
- matches all messages from all facilites, but excluding any from auth or authpriv.
- mail.info
- matches messages from the mail facility with security levels of 'info' or higher.
- mail.warn
- matches messages from the mail facility with security levels of 'warn' or higher.
- *.=debug;
- matched messages from all facilities with the 'debug' security level and no other.
You can deduce from this that some messages will appear in multiple log files, which is something we want to keep an eye on and control.
For example, with the above settings, an incoming message with status mail.info will appear in three (3) log files: syslog, mail.log and mail.info. While a message with status mail.err will be logged to five (5) log files, namely: syslog, mail.log, mail.info, mail.warn, and mail.err.
And a message logged from iptables with --log-level 7 will arrive with a status of kern.debug and be logged to three (3) separate log files: syslog, kern.log and debug.
You can monitor these files in real-time from the command-line:
cd /var/log
tail -fn0 syslog mail.err mail.warn mail.info mail.log kern.log debug
Prefix your commands with sudo if you're not running as root or don't have read access.
Less duplication in the mail logs
Focusing on the 'mail' facility for now, we don't really need that many copies of every log entry. So let's make some changes.
Old settings
*.*;auth,authpriv.none -/var/log/syslog
mail.* -/var/log/mail.log
mail.info -/var/log/mail.info
mail.warn -/var/log/mail.warn
mail.err /var/log/mail.err
New settings
*.*;auth,authpriv,mail.none -/var/log/syslog
mail.* -/var/log/mail.log
mail.warn /var/log/mail.err
Now we have just two logfiles for 'mail'. The mail.log for all entries, and mail.err for 'warn', 'error', and any more severe messages.
Security level | Old settings log targets | New settings |
---|---|---|
mail.info | syslog, mail.log, mail.info | mail.log |
mail.warn | syslog, mail.log, mail.info, mail.warn | mail.log, mail.err |
mail.err | syslog, mail.log, mail.info, mail.warn, mail.err | mail.log, mail.err |
What a difference this makes already on an active mail server!
Less duplication in the iptables logs
Similar to the situation to mail, we have an issue with iptables logging to multiple locations.
At the end of our firewall script we have the following catch-all to LOG details of any access attempts that weren't already either ACCEPT'ed or REJECT'ed by earlier rules:
...
$IPTABLES -A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7
$IPTABLES --policy INPUT DROP
With a log level of 7 (debug) this rule generates log messages with a status of kern.debug. And with the default configuration rsyslog will write them to three (3) locations: syslog, kern.log and debug. Let's clean that up.
Old settings
*.*;auth,authpriv.none -/var/log/syslog
kern.* -/var/log/kern.log
*.=debug;\
auth,authpriv.none;\
news.none;mail.none -/var/log/debug
New settings (old syntax)
*.*;auth,authpriv,kern.none -/var/log/syslog
kern.* -/var/log/kern.log
kern.debug ~
*.=debug;\
auth,authpriv.none;\
news.none;mail.none -/var/log/debug
New settings (new syntax)
*.*;auth,authpriv,kern.none /var/log/syslog
kern.* /var/log/kern.log
kern.debug stop
*.=debug;\
auth,authpriv.none;\
news.none;mail.none /var/log/debug
We've excluded kernel messages from the syslog, and put a stop on kern.debug messages to prevent them being picked up by subsequent filters. This ensures that our iptables LOG messages now only appear in kern.log and not in the syslog or debug logs.
The syntax for the stop has changed from tilde (~) to the word 'stop' in the latest version.
You'll notice that the minuses have also been removed in the new syntax. They are used in the old version to prevent excess disk writes, but not needed and not used in the new version of rsyslog.
An alternative approach used by some is:
kern.debug /var/log/iptables.log
& stop
kern.* /var/log/kern.log
This creates a new log file just for iptables debugging.
Remember that any log files you add will need to be rotated using logrorate or a similar facility so they can't grow to fill up the partition.
Restarting rsyslog
For any configuration changes to take affect you need to restart the rsyslog daemon
Under the old 'init' system:
service rsyslog restart
And under the new 'systemd' system:
systemctl restart rsyslog.service
Creating a basic filter
Here's a quick example showing how you can split off certain entries into a new log file. In this case we're separating out spamd log entries from the main mail.log.
Create a file /etc/rsyslog.d/spamd.conf with:
Old syntax
:programname,isequal,"spamd" /var/log/spamd.log
& ~
New syntax
if $programname == 'spamd' then /var/log/spamd.log
& stop
This will now intercept any log entries matching spamd[PID] and pipe them into a new spamd.log.
After this you should also update /etc/logrotate.d/rsyslog to recognise and rotate the new log file along with the standard mail logs:
...
/var/log/mail.err
/var/log/mail.log
/var/log/spamd.log
{
...
}
...
More advanced filtering
There are now three types of filters available in rsyslog. We've been working with the "traditional" selectors, but there are also RainerScript-based filters and property-based filters which you can read up on here. They allow for regular expression matching among other things.
rsyslog errors after upgrade
On a separate topic, you may find that after upgrading rsyslog you start seeing errors similar to the following:
May 24 06:25:33 hostname rsyslogd: [origin software="rsyslogd" swVersion="8.4.2" x-pid="1812" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
May 24 06:26:39 hostname rsyslogd0: action 'action 17' resumed (module 'builtin:ompipe') [try http://www.rsyslog.com/e/0 ]
May 24 06:26:39 hostname rsyslogd-2359: action 'action 17' resumed (module 'builtin:ompipe') [try http://www.rsyslog.com/e/2359 ]
May 24 06:55:58 hostname rsyslogd-2007: action 'action 17' suspended, next retry is Sun May 24 06:56:28 2015 [try http://www.rsyslog.com/e/2007 ]
These stem from rsyslog trying to write to a destination that doesn't exist. In particular you're probably not running an xconsole so shouldn't be piping anything to /dev/xconsole.
The fix is to comment out the /dev/xconsole section of rsyslog.conf as follows:
#daemon.*;mail.*;\
# news.err;\
# *.=debug;*.=info;\
# *.=notice;*.=warn |/dev/xconsole
And then restart the rsyslog daemon.
References
Related Articles - Log Files
- SQL Using a PostgreSQL foreign data wrapper to analyze log files
- System Controlling what logs where with rsyslog.conf
- System Logging sFTP activity for chrooted users
- System Analyzing Apache Log Files
- System Blocking Unwanted Spiders and Scrapers
- System Bash script to generate broken links report
- System Referer Spam from Microsoft Bing
- System Referer Spam from Live Search
- System Fake Traffic from AVG