How to Use rsyslog for Advanced Log Management

rsyslog is the default system logging daemon on most Linux distributions. While many administrators only interact with logs through journalctl, rsyslog provides powerful filtering, forwarding, and processing capabilities essential for production servers.

Understanding rsyslog Configuration

# Main config file
cat /etc/rsyslog.conf

# Additional configuration
ls /etc/rsyslog.d/

# Configuration uses facility.priority syntax:
# Facilities: auth, authpriv, cron, daemon, kern, mail, syslog, user, local0-local7
# Priorities: emerg, alert, crit, err, warning, notice, info, debug

# Examples:
# auth.*                /var/log/auth.log      — All auth messages
# *.err                 /var/log/errors.log    — All errors
# mail.warning          /var/log/mail.warn     — Mail warnings and above

Custom Log Routing

# Create a rule to separate application logs
# /etc/rsyslog.d/50-myapp.conf

# Log messages from local0 facility to a dedicated file
local0.*    /var/log/myapp/application.log

# Filter by program name
if $programname == "myapp" then /var/log/myapp/myapp.log
& stop    # Do not process this message further

# Filter by message content
:msg, contains, "SQL Error" /var/log/myapp/sql-errors.log

Log Templates

# Custom log format
template(name="DetailedFormat" type="string"
  string="%timegenerated:::date-rfc3339% %HOSTNAME% %syslogtag%%msg%
")

# Use the template
local0.* /var/log/myapp/app.log;DetailedFormat

# JSON format for log aggregation
template(name="JSONFormat" type="string"
  string="{"timestamp":"%timegenerated:::date-rfc3339%","host":"%HOSTNAME%","severity":"%syslogseverity-text%","facility":"%syslogfacility-text%","tag":"%syslogtag%","message":"%msg:::json%"}
")

local0.* /var/log/myapp/app.json;JSONFormat

Remote Log Forwarding

# Send logs to a remote syslog server
# /etc/rsyslog.d/60-remote.conf

# Forward over UDP (traditional, faster)
*.* @logserver.example.com:514

# Forward over TCP (reliable)
*.* @@logserver.example.com:514

# Forward over TCP with TLS
*.* @@(o)logserver.example.com:6514

# Forward only specific messages
auth.* @@logserver.example.com:514
*.err @@logserver.example.com:514

Log Rotation Integration

# /etc/logrotate.d/myapp
/var/log/myapp/*.log {
    daily
    rotate 30
    compress
    delaycompress
    missingok
    notifempty
    create 640 syslog adm
    sharedscripts
    postrotate
        /usr/lib/rsyslog/rsyslog-rotate
    endscript
}

Performance Tuning

# Buffer messages for batch writing
# /etc/rsyslog.d/00-performance.conf

# Main queue settings
main_queue(
  queue.size="100000"
  queue.type="LinkedList"
  queue.workerThreads="4"
)

# Async file writing
module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat"
  asyncWriting="on"
  flushInterval="1"
)

Troubleshooting rsyslog

# Check rsyslog status
systemctl status rsyslog

# Validate configuration
rsyslogd -N1

# Debug mode
rsyslogd -dn

# View rsyslog internal stats
cat /var/log/rsyslog-stats.log

# Restart after config changes
sudo systemctl restart rsyslog