Sending Syslog to Loki

Getting Syslog messages into Loki #

So, now that we’ve figured out Where our logs were going¹… lets push the rest of the logs into the stack

Ship them to promtail with some buffering #

The rsyslog site’s guide for shipping logs² recommends the following configuration for nodes:

*.*  action(type="omfwd" target="192.0.2.2" port="10514"
            protocol="tcp" action.resumeRetryCount="100" 
            queue.type="linkedList" queue.size="10000")

Click here to see the full rsyslog reference config ↕

# this is the simplest forwarding action:
*.* action(type="omfwd" target="192.0.2.1" port="10514" protocol="tcp")
# it is equivalent to the following obsolete legacy format line:
*.* @@192.0.2.1:10514 
# do NOT use this any longer!

# Note: if the remote system is unreachable, processing will
# block here and discard messages after a while

# so a better use is
*.*  action(type="omfwd" target="192.0.2.2" port="10514" protocol="tcp" action.resumeRetryCount="100" queue.type="linkedList" queue.size="10000")
# this will de-couple the sending from the other logging actions,
# and prevent delays when the remote system is not reachable. Also,
# it will try to connect 100 times before it discards messages as
# undeliverable.
# the rest below is more or less a plain vanilla rsyslog.conf as 
# many distros ship it - it's more for your reference...
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none      /var/log/messages
# The authpriv file has restricted access.
authpriv.*                                    /var/log/secure
# Log all the mail messages in one place.
mail.*                                        /var/log/maillog
# Log cron stuff
cron.*                                        /var/log/cron
# Everybody gets emergency messages
*.emerg                                       :omusrmsg:*
# Save news errors of level crit and higher in a special file.
uucp,news.crit                                /var/log/spooler
# Save boot messages also to boot.log
local7.*                                      /var/log/boot.log

So, why not just make a few simple template files to push out to nodetypes

Gitlab rsyslog snippit ↕

/etc/rsyslog.d/wpl-gitlablogs.conf_{Lang: syslog}

*.*  action(type="omfwd"
       protocol="tcp" target="elky.nfo.wolfspyre.io" port="5520"
       Template="RSYSLOG_SyslogProtocol23Format"
       TCP_Framing="octet-counted" KeepAlive="on"
       action.resumeRetryCount="-1"
       queue.type="linkedlist" queue.size="50000")
# udp/tcp 5520 = syslog-ng listener for Gitlab
# tcp 15520 = promtail listener  for Gitlab

CoreDNS rsyslog snippit ↕

/etc/rsyslog.d/wpl-corednslog.conf_{Lang: syslog}

*.*  action(type="omfwd"
       protocol="tcp" target="elky.nfo.wolfspyre.io" port="5553"
       Template="RSYSLOG_SyslogProtocol23Format"
       TCP_Framing="octet-counted" KeepAlive="on"
       action.resumeRetryCount="-1"
       queue.type="linkedlist" queue.size="50000")
# udp/tcp 5553 = syslog-ng listener for CoreDNS
# tcp 15553 = promtail listener  for CoreDNS

Generic rsyslog snippit ↕

/etc/rsyslog.d/wpl-prom-syslog.conf_{Lang: syslog}

*.*  action(type="omfwd"
       protocol="tcp" target="elky.nfo.wolfspyre.io" port="16514"
       Template="RSYSLOG_SyslogProtocol23Format"
       TCP_Framing="octet-counted" KeepAlive="on"
       action.resumeRetryCount="-1"
       queue.type="linkedlist" queue.size="50000")
# udp/tcp 6514 = syslog-ng listener for syslog
# tcp 16514 = promtail listener  for syslog

It REALLY helps to point promtail at the right loki instance if you wish loki to ingest aforementioned logs. sigh ↩︎
https://www.rsyslog.com/sending-messages-to-a-remote-syslog-server/ ↩︎

Getting Syslog messages into Loki #

Ship them to promtail with some buffering #

Cookie-Cutter file templates #