Configuring rsyslog on the Centralized Logging Server The steps in this procedure must be followed on the system that you intend to use as your centralized logging sever. All steps in this procedure must be run while logged in as the root user. The file is read line-by-line and any line read is passed to rsyslog’s rule engine. The rule engine applies filter conditions and selects which actions needs to be carried out. Empty lines are not processed, as they would result in empty syslog records. They are simply ignored. As new lines are written they are taken from the file and processed.

Rsyslog offers four different types “filter conditions”:

• “traditional” severity and facility based selectors
• property-based filters
• expression-based filters
• BSD-style blocks (not upward compatible)

Selectors¶

Selectors are the traditional way of filtering syslog messages. Theyhave been kept in rsyslog with their original syntax, because it iswell-known, highly effective and also needed for compatibility withstock syslogd configuration files. If you just need to filter based onpriority and facility, you should do this with selector lines. They arenot second-class citizens in rsyslog and offer the best performancefor this job.

The selector field itself again consists of two parts, a facility and apriority, separated by a period (”.’‘). Both parts are case insensitiveand can also be specified as decimal numbers, but don’t do that, youhave been warned. Both facilities and priorities are described insyslog(3). The names mentioned below correspond to the similarLOG_-values in /usr/include/syslog.h.

The facility is one of the following keywords: auth, authpriv, cron,daemon, kern, lpr, mail, mark, news, security (same as auth), syslog,user, uucp and local0 through local7. The keyword security should not beused anymore and mark is only for internal use and therefore should notbe used in applications. Anyway, you may want to specify and redirectthese messages here. The facility specifies the subsystem that producedthe message, i.e. all mail programs log with the mail facility(LOG_MAIL) if they log using syslog.

The priority is one of the following keywords, in ascending order:debug, info, notice, warning, warn (same as warning), err, error (sameas err), crit, alert, emerg, panic (same as emerg). The keywords error,warn and panic are deprecated and should not be used anymore. Thepriority defines the severity of the message.

The behavior of the original BSD syslogd is that all messages of thespecified priority and higher are logged according to the given action.Rsyslogd behaves the same, but has some extensions.

In addition to the above mentioned names the rsyslogd(8) understandsthe following extensions: An asterisk (“*’‘) stands for all facilitiesor all priorities, depending on where it is used (before or after theperiod). The keyword none stands for no priority of the given facility.You can specify multiple facilities with the same priority pattern inone statement using the comma (”,’‘) operator. You may specify as muchfacilities as you want. Remember that only the facility part from such astatement is taken, a priority part would be skipped.

Multiple selectors may be specified for a single action using thesemicolon (”;’‘) separator. Remember that each selector in the selectorfield is capable to overwrite the preceding ones. Using this behavioryou can exclude some priorities from the pattern.

Rsyslogd has a syntax extension to the original BSD source, that makesits use more intuitively. You may precede every priority with an equalssign (“=’‘) to specify only this single priority and not any of theabove. You may also (both is valid, too) precede the priority with anexclamation mark (”!’‘) to ignore all that priorities, either exact thisone or this and any higher priority. If you use both extensions than theexclamation mark must occur before the equals sign, just use itintuitively.

Property-Based Filters¶

Property-based filters are unique to rsyslogd. They allow to filter onany property, like HOSTNAME, syslogtag and msg. A list of allcurrently-supported properties can be found in the property replacerdocumentation (but keep in mind that only theproperties, not the replacer is supported). With this filter, eachproperties can be checked against a specified value, using a specifiedcompare operation.

A property-based filter must start with a colon in column 0. This tellsrsyslogd that it is the new filter type. The colon must be followed bythe property name, a comma, the name of the compare operation to carryout, another comma and then the value to compare against. This valuemust be quoted. There can be spaces and tabs between the commas.Property names and compare operations are case-sensitive, so “msg”works, while “MSG” is an invalid property name. In brief, the syntax isas follows:

:property,[!]compare-operation,'value'

Compare-Operations¶

The following compare-operations are currently supported:

Checks if the value is found exactly at the beginning of the propertyvalue. For example, if you search for “val” with

:msg,startswith,'val'

it will be a match if msg contains “values are in this message” but itwon’t match if the msg contains “There are values in this message” (inthe later case, “contains” would match). Please note that “startswith” isby far faster than regular expressions. So even once they areimplemented, it can make very much sense (performance-wise) to use“startswith”.

You can use the bang-character (!) immediately in front of acompare-operation, the outcome of this operation is negated. Forexample, if msg contains “This is an informative message”, the followingsample would not match:

:msg,contains,'error'

but this one matches:

:msg,!contains,'error'

Using negation can be useful if you would like to do some genericprocessing but exclude some specific events. You can use the discardaction in conjunction with that. A sample would be:

FabFilter Pro-L 2 introduces four new limiting algorithms. Modern is the new 'best for all' algorithm that produces even better results than the Transparent and Allround algorithms from version 1. Limiter waves l2 tutorial. In this brief tutorial you will learn about the differences between the Waves L1 and L2 Ultramaximizer plugins and see how to apply them to your tracks. Some mastering engineers even use multiple limiters. Nevo sometimes starts out with an L2 to boost the level slightly before routing the signal to an L3. “Because the L2 is wideband,” he says, “it applies limiting to the whole frequency range. But the L3 allows me to control the different frequencies in a less dynamic way.

Do not overlook the tilde in line 2! In this sample, all messagesare written to the file allmsgs-including-informational.log. Then, allmessages containing the string “informational” are discarded. That meansthe config file lines below the “discard line” (number 2 in our sample)will not be applied to this message. Then, all remaining lines will alsobe written to the file allmsgs-but-informational.log.

Value Part¶

Value is a quoted string. It supports some escape sequences:

” - the quote character (e.g. “String with ”Quotes””)

- the backslash character (e.g. “C:tmp”)

Escape sequences always start with a backslash. Additional escapesequences might be added in the future. Backslash characters must beescaped. Any other sequence then those outlined above is invalid and maylead to unpredictable results.

Probably, “msg” is the most prominent use case of property basedfilters. It is the actual message text. If you would like to filterbased on some message content (e.g. the presence of a specific code),this can be done easily by:

:msg,contains,'ID-4711'

This filter will match when the message contains the string “ID-4711”.Please note that the comparison is case-sensitive, so it would not matchif “id-4711” would be contained in the message.

:msg,regex,'fatal.*error'

This filter uses a POSIX regular expression. It matches when the stringcontains the words “fatal” and “error” with anything in between (e.g.“fatal net error” and “fatal lib error” but not “fatal error” as twospaces are required by the regular expression!).

Getting property-based filters right can sometimes be challenging. Inorder to help you do it with as minimal effort as possible, rsyslogdspits out debug information for all property-based filters during theirevaluation. To enable this, run rsyslogd in foreground and specify the“-d” option.

Boolean operations inside property based filters (like ‘message contains“ID17” or message contains “ID18”’) are currently not supported (exceptfor “not” as outlined above). Please note that while it is possible toquery facility and severity via property-based filters, it is far moreadvisable to use classic selectors (see above) for those cases.

Expression-Based Filters¶

Expression based filters allow filtering on arbitrary complexexpressions, which can include boolean, arithmetic and stringoperations. Expression filters will evolve into a full configurationscripting language. Unfortunately, their syntax will slightly changeduring that process. So if you use them now, you need to be prepared tochange your configuration files some time later. However, we try toimplement the scripting facility as soon as possible (also in respect tostage work needed). So the window of exposure is probably not too long.

Expression based filters are indicated by the keyword “if” in column 1of a new line. They have this format:

“if” and “then” are fixed keywords that mus be present. “expr” is a(potentially quite complex) expression. So the expressiondocumentation for details.“action-part-of-selector-line” is an action, just as you know it (e.g.“/var/log/logfile” to write to that file).

BSD-style Blocks¶

Note: rsyslog v7+ does no longer support BSD-style blocksfor technical reasons. So it is strongly recommended not touse them.

Rsyslogd supports BSD-style blocks inside rsyslog.conf. Each block oflines is separated from the previous block by a program or hostnamespecification. A block will only log messages corresponding to the mostrecent program and hostname specifications given. Thus, a block whichselects ‘ppp’ as the program, directly followed by a block that selectsmessages from the hostname ‘dialhost’, then the second block will onlylog messages from the ppp program on dialhost.

A program specification is a line beginning with ‘!prog’ and thefollowing blocks will be associated with calls to syslog from thatspecific program. A program specification for ‘foo’ will also match anymessage logged by the kernel with the prefix ‘foo: ’. Alternatively, aprogram specification ‘-foo’ causes the following blocks to be appliedto messages from any program but the one specified. A hostnamespecification of the form ‘+hostname’ and the following blocks will beapplied to messages received from the specified hostname. Alternatively,a hostname specification ‘-hostname’ causes the following blocks to beapplied to messages from any host but the one specified. If the hostnameis given as ‘@’, the local hostname will be used. (NOT YET IMPLEMENTED)A program or hostname specification may be reset by giving the programor hostname as ‘*’.

Please note that the “#!prog”, “#+hostname” and “#-hostname” syntaxavailable in BSD syslogd is not supported by rsyslogd. By default, nohostname or program is set.

Examples¶

Right now, you need to specify numerical values if you would like tocheck for facilities and severity. These can be found in RFC 5424.If you don’t like that,you can of course also use the textual property - just be sure to usethe right one. As expression support is enhanced, this will change. Forexample, if you would like to filter on message that have facilitylocal0, start with “DEVNAME” and have either “error1” or “error0” intheir message content, you could use the following filter:

Please note that the above must all be on one line! And if you wouldlike to store all messages except those that contain “error1” or“error0”, you just need to add a “not”:

If you would like to do case-insensitive comparisons, use “contains_i”instead of “contains” and “startswith_i” instead of “startswith”.Note that regular expressions are currently NOT supported inexpression-based filters. These will be added later when functionsupport is added to the expression engine (the reason is that regularexpressions will be a separate loadable module, which requires some moreprequisites before it can be implemented).

This documentation is part of the rsyslogproject.Copyright © 2008-2014 by Rainer Gerhardsand Adiscon. Released under the GNU GPLversion 2 or higher.

Templates are a key feature of rsyslog. They allow to specify any formata user might want. They are also used for dynamic file name generation.Every output in rsyslog uses templates - this holds true for files, usermessages and so on. The database writer expects its template to be aproper SQL statement - so this is highly customizable too. You might askhow does all of this work when no templates at all are specified. Goodquestion ;). The answer is simple, though. Templates compatible with thestock syslogd formats are hardcoded into rsyslogd. So if no template isspecified, we use one of those hardcoded templates. Search for“template_” in syslogd.c and you will find the hardcoded ones.

Templates are specified by template() statements. They can also bespecified via $template legacy statements.

Note: key elements of templates are rsyslog properties. See thersyslog properties reference for a list of whichare available.

The template() statement¶

The template() statement is used to define templates. Note that it is astatic statement, that means all templates are defined when rsyslogreads the config file. As such, templates are not affected byif-statements or config nesting.

The basic structure of the template statement is as follows:

In addition to this simpler syntax, list templates (to be describedbelow) support an extended syntax:

Each template has a parameter name, which specifies the templatename, and a parameter type, which specifies the template type. Thename parameter must be unique, and behaviour is unpredictable if it isnot. The type parameter specifies different template types.Different types simply enable different ways to specify the templatecontent. The template type does not affect what an (output) plugincan do with it. So use the type that best fits your needs (from a configwriting point of view!). The following types are available:

• list
• subtree
• string
• plugin

The various types are described below.

list¶

In this case, the template is generated by a list of constant andvariable statements. These follow the template spec in curly braces.This type is also primarily meant for use with structure-aware outputs,like ommongodb. However, it also works perfectly with text-basedoutputs. We recommend to use this mode if more complex propertysubstitutions need to be done. In that case, the list-based templatesyntax is much clearer than the simple string-based one.

The list template contains the template header (with type=”list”)and is followed by constant and property statements, given incurly braces to signify the template statement they belong to. As thename says, constant statements describe constant text andproperty describes property access. There are many options toproperty, described further below. Most of these options are used toextract only partial property contents or to modify the text obtained(like to change its case to upper or lower case, only).

To grasp the idea, an actual sample is:

This sample is probably primarily targeted at the usual file-basedoutput.

constant statement¶

This provides a way to specify constant text. The text is usedliterally. It is primarily intended for text-based output, so that someconstant text can be included. For example, if a complex template isbuilt for file output, one usually needs to finish it by a newline,which can be introduced by a constant statement. Here is an actualsample of that use case from the rsylsog testbench:

The following escape sequences are recognized inside the constant text:

• - single backslash
• n - LF
• ooo - (three octal digits) - represents character with thisnumerical value (e.g. 101 equals “A”). Note that three octal digitsmust be given (in contrast to some languages, where between one andthree are valid). While we support octal notation, we recommend touse hex notation as this is better known.
• xhh - (where h is a hex digit) - represents character with thisnumerical value (e.g. x41 equals “A”). Note that two hexadecimaldigits must be given (in contrast to some languages where one or twoare valid).
• .. some others .. list needs to be extended

Note: if an unsupported character follows a backslash, this is treatedas an error. Behaviour is unpredictable in this case.

To aid usage of the same template both for text-based outputs andstructured ones, constant text without an “outname” parameter will beignored when creating the name/value tree for structured outputs. So ifyou want to supply some constant text e.g. to mongodb, you must includean outname, as can be seen here:

The “constant” statement supports the following parameters:

• value - the constant value to use
• outname - output field name (for structured outputs)

property statement¶

This statement is used to include property text. It can access allproperties. Also, options permit to specify picking only part of aproperty or modifying it. It supports the following parameters:

• name - the name of the property to access
• outname - output field name (for structured outputs)
• dateformat - date format to use (only for date-related properties)
• caseconversion - permits to convert case of the text. Supportedvalues are “lower” and “upper”
• controlcharacters - specifies how to handle control characters.Supported values are “escape”, which escapes them, “space”, whichreplaces them by a single space, and “drop”, which simply removesthem from the string.
• securepath - used for creating pathnames suitable for use in dynafiletemplates
• format - specify format on a field basis. Supported values are:
  • “csv” for use when csv-data isgenerated
  • “json” which formats properjson content (but without a field header)
  • “jsonf” which formats as acomplete json field
  • “jsonr” which avoids doubleescaping the value but makes it safe for a json field
  • “jsonfr” which is thecombination of “jsonf” and “jsonr”.
• position.from - obtain substring starting from this position (1 isthe first position)
• position.to - obtain substring up to this position
• position.relativeToEnd - the from and to position is relative to theend of the string instead of the usual start of string. (availablesince rsyslog v7.3.10)
• fixedwidth - changes behaviour of position.to so that it pads thesource string with spaces up to the value of position.to if the sourcestring is shorter. “on” or “off” (default) (available since rsyslogv8.13.0)
• field.number - obtain this field match
• field.delimiter - decimal value of delimiter character for fieldextraction
• regex.expression - expression to use
• regex.type - either ERE or BRE
• regex.nomatchmode - what to do if we have no match
• regex.match - match to use
• regex.submatch - submatch to use
• droplastlf - drop a trailing LF, if it is present
• mandatory - signifies a field as mandatory. If set to “on”, thisfield will always be present in data passed to structured outputs,even if it is empty. If “off” (the default) empty fields will not bepassed to structured outputs. This is especially useful for outputsthat support dynamic schemas (like ommongodb).
• spifno1stsp - expert options for RFC3164 template processing

subtree¶

Available since rsyslog 7.1.4

In this case, the template is generated based on a complete (CEE)subtree. This type of template is most useful for outputs that know howto process hierarchical structure, like ommongodb. With that type, theparameter subtree must be specified, which tells which subtree touse. For example template(name=”tpl1” type=”subtree” subtree=”$!”)includes all CEE data, while template(name=”tpl2” type=”subtree”subtree=”$!usr!tpl2”) includes only the subtree starting at $!usr!tpl2.The core idea when using this type of template is that the actual datais prefabricated via set and unset script statements, and the resultingstructure is then used inside the template. This method MUST be used ifa complete subtree needs to be placed directly into the object’s root.With all other template types, only subcontainers can be generated. Notethat subtree type can also be used with text-based outputs, like omfile.HOWEVER, you do not have any capability to specify constant text, and assuch cannot include line breaks. As a consequence, using this templatetype for text outputs is usually only useful for debugging or veryspecial cases (e.g. where the text is interpreted by a JSON parser lateron).

Use case¶

A typical use case is to first create a custom subtree and then includeit into the template, like in this small example:

Here, we assume that $msg contains various fields, and the data from afield is to be extracted and stored - together with the message - asfield content.

string¶

This closely resembles the legacy template statement. It has a mandatoryparameter string, which holds the template string to be applied. Atemplate string is a mix of constant text and replacement variables (seeproperty replacer). These variables are taken from message or otherdynamic content when the final string to be passed to a plugin isgenerated. String-based templates are a great way to specify textualcontent, especially if no complex manipulation to properties isnecessary.

This is a sample for a string-based template:

The text between percent signs (‘%’) is interpreted by the rsyslogproperty replacer. In a nutshell,it contains the property to use as well as options for formattingand further processing. This is very similar to what the propertyobject in list templates does (it actually is just a different language toexpress most of the same things).

Everything outside of the percent signs is constant text. In theabove case, we have mostly spaces between the property values. At theend of the string, an escape sequence is used.

Escape sequences permit to specify nonprintable characters. They workvery similar to escape sequences in C and many other languages. Theyare initiated by the backslash characters and followed by one or morecharacters that specify the actual character. For example 7 is theUS-ASCII BEL character and n is a newline. The set is similar towhat C and perl support, but a bit more limited.

plugin¶

In this case, the template is generated by a plugin (which is thencalled a “strgen” or “string generator”). The format is fixed as it iscoded. While this is inflexible, it provides superior performance, andis often used for that reason (not that “regular” templates are slow -but in very demanding environments that “last bit” can make adifference). Refer to the plugin’s documentation for further details.For this type, the parameter plugin must be specified and mustcontain the name of the plugin as it identifies itself. Note that theplugin must be loaded prior to being used inside a template.Config example:

options¶

The <options> part is optional. It carries options influencing thetemplate as a whole and is a part of the template parameters. See detailsbelow. Be sure NOT to mistake template options with property options -the latter ones are processed by the property replacer and apply to aSINGLE property, only (and not the whole template).Template options are case-insensitive. Currently defined are:

option.sql - format the string suitable for a SQL statement in MySQLformat. This will replace single quotes (“’”) and the backslashcharacter by their backslash-escaped counterpart (“’” and “”)inside each field. Please note that in MySQL configuration, theNO_BACKSLASH_ESCAPES mode must be turned off for this format to work(this is the default).

option.stdsql - format the string suitable for a SQL statement thatis to be sent to a standards-compliant sql server. This will replacesingle quotes (“’”) by two single quotes (“’‘”) inside each field. Youmust use stdsql together with MySQL if in MySQL configuration theNO_BACKSLASH_ESCAPES is turned on.

option.json - format the string suitable for a json statement. Thiswill replace single quotes (“’”) by two single quotes (“’‘”) inside eachfield.

option.casesensitive - treat property name references as casesensitive. The default is “off”, where all property name references arefirst converted to lowercase during template definition. With thisoption turned “on”, property names are looked up as defined in thetemplate. Use this option if you have JSON ($!*), local (!.*),or global ($!*) properties which container uppercase letters. Thenormal Rsyslog properties are case-insensitive, so this option is notneeded for properly referencing those properties.

The use the options option.sql, option.stdsql, andoption.json are mutually exclusive. Using more than one at the sametime can cause unpredictable behaviour.

Either the sql or stdsql option must be specified when atemplate is used for writing to a database, otherwise injection mightoccur. Please note that due to the unfortunate fact that several vendorshave violated the sql standard and introduced their own escape methods,it is impossible to have a single option doing all the work. So youyourself must make sure you are using the right format. If you choosethe wrong one, you are still vulnerable to sql injection.Please note that the database writer checks that the sql option ispresent in the template. If it is not present, the write database actionis disabled. This is to guard you against accidentally forgetting it andthen becoming vulnerable to SQL injection. The sql option can also beuseful with files - especially if you want to import them into adatabase on another machine for performance reasons. However, do NOT useit if you do not have a real need for it - among others, it takes sometoll on the processing time. Not much, but on a really busy system youmight notice it.

The default template for the write to database action has the sql optionset. As we currently support only MySQL and the sql option matches thedefault MySQL configuration, this is a good choice. However, if you haveturned on NO_BACKSLASH_ESCAPES in your MySQL config, you need tosupply a template with the stdsql option. Otherwise you will becomevulnerable to SQL injection.

Examples¶

Standard Template for Writing to Files¶

The equivalent string template looks like this:

Note that the template string itself must be on a single line.

Standard Template for Forwarding to a Remote Host (RFC3164 mode)¶

The equivalent string template looks like this:

Note that the template string itself must be on a single line.

Standard Template for write to the MySQL database¶

The equivalent string template looks like this:

Note that the template string itself must be on a single line.

Creating Dynamic File Names for omfile¶

Templates can be used to generate actions with dynamic file names.For example, if you would like to split syslog messages from different hoststo different files (one per host), you can define the following template:

Legacy example:

This template can then be used when defining an action. Itwill result in something like “/var/log/system-localhost.log”

legacy format¶

In pre v6-versions of rsyslog, you need to use the $templatestatement to configure templates. They provide the equivalent to string-and plugin-based templates. The legacy syntax continuous to work in v7,however we recommend to avoid legacy format for newly written configfiles. Legacy and current config statements can coexist within the sameconfig file.

The general format is

where “name” is the template name and“param” is a single parameter that specifies template content. Theoptional “options” part is used to set template options.

string¶

The parameter is the same string that with the current-style format youspecify in the string parameter, for example:

Note that list templates are not available in legacy format, so you needto use complex property replacer constructs to do complex things.

plugin¶

This is equivalent to the “plugin”-type template directive. Here, theparameter is the plugin name, with an equal sign prepended. An exampleis:

Reserved Template Names¶

Template names beginning with “RSYSLOG_” are reserved for rsyslog use.Do NOT use them if, otherwise you may receive a conflict in the future(and quite unpredictable behaviour). There is a small set of pre-definedtemplates that you can use without the need to define it:

• RSYSLOG_TraditionalFileFormat - the “old style” default log fileformat with low-precision timestamps
• RSYSLOG_FileFormat - a modern-style logfile format similar toTraditionalFileFormat, both with high-precision timestamps andtimezone information
• RSYSLOG_TraditionalForwardFormat - the traditional forwarding formatwith low-precision timestamps. Most useful if you send messages toother syslogd’s or rsyslogd below version 3.12.5.
• RSYSLOG_SysklogdFileFormat - sysklogd compatible log file format. Ifused with options: $SpaceLFOnReceiveon,$EscapeControlCharactersOnReceiveoff,$DropTrailingLFOnReceptionoff, the log format will conform to sysklogd log format.
• RSYSLOG_ForwardFormat - a new high-precision forwarding format verysimilar to the traditional one, but with high-precision timestampsand timezone information. Recommended to be used when sendingmessages to rsyslog 3.12.5 or above.
• RSYSLOG_SyslogProtocol23Format - the format specified in IETF’sinternet-draft ietf-syslog-protocol-23, which is assumed to becomethe new syslog standard RFC. This format includes severalimprovements. The rsyslog message parser understands this format, soyou can use it together with all relatively recent versions ofrsyslog. Other syslogd’s may get hopelessly confused if receivingthat format, so check before you use it. Note that the format isunlikely to change when the final RFC comes out, but this may happen.
• RSYSLOG_DebugFormat - a special format used for troubleshootingproperty problems. This format is meant to be written to a log file.Do not use for production or remote forwarding.

Legacy String-based Template Samples¶

This section provides some default templates in legacy format, as usedin rsyslog previous to version 6. Note that this format is stillsupported, so there is no hard need to upgrade existing configurations.However, it is strongly recommended that the legacy constructs are notused when crafting new templates. Note that each $template statement ison a single line, but probably broken across several lines fordisplay purposes by your browsers. Lines are separated by empty lines.Keep in mind, that line breaks are important in legacy format.

See Also¶

This documentation is part of the rsyslog project.Copyright © 2008-2014 by Rainer Gerhardsand Adiscon. Released under the GNU GPLversion 2 or higher.