The WolfspyreLabs Blog/ 2022/ October/ Sending Syslog to Loki/ Sending Syslog to Loki Getting Syslog messages into Loki #So, now that we’ve figured out Where our logs were going1… lets push the rest of the logs into the stack Ship them to promtail with some buffering #The rsyslog site’s guide for shipping logs2 recommends the following configuration for nodes: *.* action(type="omfwd" target="192.0.2.2" port="10514" protocol="tcp" action.resumeRetryCount="100" queue.type="linkedList" queue.size="10000") Click here to see the full rsyslog reference config โ # this is the simplest forwarding action: *.* action(type="omfwd" target="192.0.2.1" port="10514" protocol="tcp") # it is equivalent to the following obsolete legacy format line: *.* @@192.0.2.1:10514 # do NOT use this any longer! # Note: if the remote system is unreachable, processing will # block here and discard messages after a while # so a better use is *.* action(type="omfwd" target="192.0.2.2" port="10514" protocol="tcp" action.resumeRetryCount="100" queue.type="linkedList" queue.size="10000") # this will de-couple the sending from the other logging actions, # and prevent delays when the remote system is not reachable. Also, # it will try to connect 100 times before it discards messages as # undeliverable. # the rest below is more or less a plain vanilla rsyslog.conf as # many distros ship it - it's more for your reference... # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none;cron.none /var/log/messages # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* /var/log/maillog # Log cron stuff cron.* /var/log/cron # Everybody gets emergency messages *.emerg :omusrmsg:* # Save news errors of level crit and higher in a special file. uucp,news.crit /var/log/spooler # Save boot messages also to boot.log local7.* /var/log/boot.log Cookie-Cutter file templates #So, why not just make a few simple template files to push out to nodetypes Gitlab rsyslog snippit โ /etc/rsyslog.d/wpl-gitlablogs.confLang: syslog *.* action(type="omfwd" protocol="tcp" target="elky.nfo.wolfspyre.io" port="5520" Template="RSYSLOG_SyslogProtocol23Format" TCP_Framing="octet-counted" KeepAlive="on" action.resumeRetryCount="-1" queue.type="linkedlist" queue.size="50000") # udp/tcp 5520 = syslog-ng listener for Gitlab # tcp 15520 = promtail listener for Gitlab CoreDNS rsyslog snippit โ /etc/rsyslog.d/wpl-corednslog.confLang: syslog *.* action(type="omfwd" protocol="tcp" target="elky.nfo.wolfspyre.io" port="5553" Template="RSYSLOG_SyslogProtocol23Format" TCP_Framing="octet-counted" KeepAlive="on" action.resumeRetryCount="-1" queue.type="linkedlist" queue.size="50000") # udp/tcp 5553 = syslog-ng listener for CoreDNS # tcp 15553 = promtail listener for CoreDNS Generic rsyslog snippit โ /etc/rsyslog.d/wpl-prom-syslog.confLang: syslog *.* action(type="omfwd" protocol="tcp" target="elky.nfo.wolfspyre.io" port="16514" Template="RSYSLOG_SyslogProtocol23Format" TCP_Framing="octet-counted" KeepAlive="on" action.resumeRetryCount="-1" queue.type="linkedlist" queue.size="50000") # udp/tcp 6514 = syslog-ng listener for syslog # tcp 16514 = promtail listener for syslog It REALLY helps to point promtail at the right loki instance if you wish loki to ingest aforementioned logs. sigh ↩︎ https://www.rsyslog.com/sending-messages-to-a-remote-syslog-server/ ↩︎