rsyslog presentation

发表于   |   分类于   |  

rsyslog message flow

message enter rsyslog with the help of input modules. Then, they are passed to ruleset, where rules are conditionally applied. when a rule matches, the message is transferred to an action, which then does something to the message, e.g. write it to a file or a database, or forward it to a remote host.

rsyslog overview

  • all the features supported: http://www.rsyslog.com/doc/master/features.html
  • rsyslog defaults to using TCP on port 514, while syslog uses udp and 514.
  • /var/log persistently store the process and kenel log
  • system-journald collects:
    • messages from kernel
    • early stage of the boot process
    • standard output and error of daemons
    • syslog
  • rsyslog service:
    • sorts syslog messages by type and priority.
    • write them to /var/log
    • most syslogs are stored in /var/log/messages
    • cat /etc/rsyslog.conf can show all the facilities.
    • config files locates in /etc/rsyslog.conf and /etc/rsyslog.d/*.conf
  • logrotate is to rotate /var/log/messages to /var/log/messages-20160920
    • after 4 weeks, old log file is discarded to free disk space.
    • cron jobs are run daily to check if there’s log to discard
    • most log files are rotated weekly
  • logger: useful to test changes to rsyslog configuration.
    • logger -p local7.notice "Log entry created on serverX"

how to config in /etc/rsyslog.conf

Filters

facility/priority-based filters

FACILITY.PRIORITY
FACILITY: kern (0), user (1), mail (2), daemon (3), auth (4), syslog (5), lpr (6), news (7), uucp (8), cron (9), authpriv (10), ftp (11), and local0 through local7 (16 - 23).
PRIORITY: debug (7), info (6), notice (5), warning (4), err (3), crit (2), alert (1), and emerg (0).
EXAMPLE:

  • kern.*
  • mail.crit
  • cron.!info,!debug

    property-based filters

    syntax::PROPERTY, [!]COMPARE_OPERATION, "STRING"
    PROPERTIES: fetch from man rsyslog.conf, under Available properties.
    COMPARE_OPERATION: contains, isequal, startswith, regex, ereregex, isempty
    EXAMPLE:
  • :msg, contains, "error"
  • :hostname, isequal, "host1"
  • :msg, !regex, "fatal .* error"

    expression-based filters

    syntax: if EXPRESSION then ACTION else ACTION
    EXAMPLE:
    1
    2
    3
    4
    5
    6
    7
    if $programname == 'prog1' then {
    	   action(type="omfile" file="/var/log/prog1.log")
       if $msg contains 'test' then
    	   action(type="omfile" file="/var/log/prog1test.log")
       else
         action(type="omfile" file="/var/log/prog1notest.log")
    }

RainerScript

complex filter, use rsyslog’s own script language, called rainerscript
HOW TO USE rainerscript(official doc):
http://www.rsyslog.com/doc/rainerscript.html
can use set and unset to define and delete user defined variable.

ACTION

saving syslog messages to log files

FILTER PATH
examples: cron.* /var/log/cron.log
FILTER -PATH
by default, rsyslog sync logs everytime after a syslog message is generated. use - to omit syncing.
can improve performance when programs run very verbose log messages.
FILTER ?DynamicFile
DynamicFile: a name of defined template that modify output paths, can use - to disable syncing.
if it’s set as /dev/console, syslog messages can be printed on standard output.

sending syslog messages over network

@[(zNUMBER)]HOST:[PORT]
@: use UDP protocol
@@: use TCP protocol
zNUMBER: z means using zlib to compress the syslog messages. NUMBER means the compress level(1-9). below 90 bytes are never compressed
EXAMPLE:
*.* @192.168.0.1
*.* @(z9)example.com:6514

can also send syslog messages to specific user
executing a program

FILTER ^EXECUTABLE; TEMPLATE
syslog message filtered by FILTER, formatted withTEMPLATE, are passed as a parameter to EXECUTABLE program. Then it’s executed.

storing syslog messages in a database

:PLUGIN:DB_HOST,DB_NAME,DB_USER,DB_PASSWORD;[TEMPLATE]
currently, rsyslog only supports mysql and postgresql. have to install rsyslog-mysql and rsyslog-pgsql packages. config in /etc/rsyslog.conf to load the module:
$ModLoad ommysql # Output module for MySQL support $ModLoad ompgsql # Output module for PostgreSQL support

discard syslog messages

FILTER ~

specifying multiple actions

FILTER ACTION & ACTION & ACTION
EXAMPLE:
kern.=crit user1 & ^test-program; temp & @192.168.0.1

TEMPLATES

templates definition should always precede rule definition in /etc/rsyslog.conf, or it’ll be ignored.
Any output that is generated by rsyslog can be formatted with the use of template.
template TEMPLATE_NAME,"text %PROPERTY% more text", [OPTION]
OPTION: currently only sql and stdsql are supported.
also provide some predefined templates.
EXAMPLE:
$template verbose, "%syslogseverity%, %syslogfacility%, %timegenerated%, %HOSTNAME%, %syslogtag%, %msg%\n"
$template dbFormat,"insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')", sql
option sql tells the database writer to format the message as an mysql sql query.

#####generate dynamic file names:
$template DynamicFile,"/var/log/test_logs/%timegenerated%-test.log"
*.* ?DynamicFile
? is an action definition to mark the template

#####PROPERTIES in template follows syntax:
%PROPERTY_NAME[:FROM_CHAR:TO_CHAR:OPTION]%
re can be used here too, to set “R” as the “FROM_CHAR”, and set re as the “TO_CHAR”.
property options can refer to man rsyslog.conf under property options part.
%timegenerated:1:10:date-rfc3339%: format the first 10 characters of time stamp as RFC 3999.

log rotation: container does not support yet

rsyslog image (7.4.7)

basic feature:

  • installs from atomic command

    • atomic install rhel7/rsyslog

      • execute /bin/install.sh in image to make some directory and mount some dirs
        1
        docker run --rm --privileged -v /:/host -e HOST=/host -e IMAGE=rhel7/rsyslog -e NAME=rsyslog rhel7/rsyslog /bin/install.sh

    • atomic run rhel7/rsyslog

      • mount some dirs installed in the previous step, and then run the /bin/rsyslog.sh to start rsyslog service
        1
        docker run -d --privileged --name rsyslog --net=host -v /etc/pki/rsyslog:/etc/pki/rsyslog -v /etc/rsyslog.conf:/etc/rsyslog.conf -v /etc/rsyslog.d:/etc/rsyslog.d -v /var/log:/var/log -v /var/lib/rsyslog:/var/lib/rsyslog -v /run/log:/run/log -v /etc/machine-id:/etc/machine-id -v /etc/localtime:/etc/localtime -e IMAGE=rhel7/rsyslog -e NAME=rsyslog --restart=always rhel7/rsyslog /bin/rsyslog.sh

  • configure from host: no need to go inside the container

  • restarting the service after changing configuration:
    • docker stop rsyslog
    • docker rm rsyslog
    • atomic run rhel7/rsyslog
  • super privileged container: container has root access to host

test simply:

  • docker ps to check whether it’s running or not
  • tail -f /var/log/messages to monitor the syslogs
  • use logger "test info", then to see the syslogs.

disable journald persistent logging:

  • why?: if rsyslog is working, no need to use journald logs any more.
  • how?: modify /etc/systemd/journald.conf, change Storage=volatile.
  • what happened?: journald still works, but store logs to ramdisk. rsyslog can still capture and process the journald logs.

rsyslog container does not support logrotate yet

how to update:

  • docker pull rhel7/rsyslog
  • if there’s a new image downloaded, run below
    • docker stop rhel7/rsyslog
    • docker rm rhel7/rsyslog
    • atomic install rhel7/rsyslog
    • atomic run rhel7/rsyslog