This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

db-parser: Process message content with a pattern database (patterndb)

1 - Classifying log messages

The AxoSyslog application can compare the contents of the received log messages to predefined message patterns. By comparing the messages to the known patterns, AxoSyslog is able to identify the exact type of the messages, and sort them into message classes. The message classes can be used to classify the type of the event described in the log message. The message classes can be customized, and for example, can label the messages as user login, application crash, file transfer, and so on events.

To find the pattern that matches a particular message, AxoSyslog uses a method called longest prefix match radix tree. This means that AxoSyslog creates a tree structure of the available patterns, where the different characters available in the patterns for a given position are the branches of the tree.

To classify a message, AxoSyslog selects the first character of the message (the text of message, not the header), and selects the patterns starting with this character, other patterns are ignored for the rest of the process. After that, the second character of the message is compared to the second character of the selected patterns. Again, matching patterns are selected, and the others discarded. This process is repeated until a single pattern completely matches the message, or no match is found. In the latter case, the message is classified as unknown, otherwise the class of the matching pattern is assigned to the message.

To make the message classification more flexible and robust, the patterns can contain pattern parsers: elements that match on a set of characters. For example, the NUMBER parser matches on any integer or hexadecimal number (for example, 1, 123, 894054, 0xFFFF, and so on). Other pattern parsers match on various strings and IP addresses. For the details of available pattern parsers, see Using pattern parsers.

The functionality of the pattern database is similar to that of the logcheck project, but it is much easier to write and maintain the patterns used by syslog-ng, than the regular expressions used by logcheck. Also, it is much easier to understand AxoSyslog pattens than regular expressions.

Pattern matching based on regular expressions is computationally very intensive, especially when the number of patterns increases. The solution used by AxoSyslog can be performed real-time, and is independent from the number of patterns, so it scales much better. The following patterns describe the same message: Accepted password for bazsi from 10.50.0.247 port 42156 ssh2

A regular expression matching this message from the logcheck project: Accepted (gssapi(-with-mic|-keyex)?|rsa|dsa|password|publickey|keyboard-interactive/pam) for [^[:space:]]+ from [^[:space:]]+ port [0-9]+( (ssh|ssh2))?

An AxoSyslog database pattern for this message: Accepted @QSTRING:auth_method: @ for@QSTRING:username: @from @QSTRING:client_addr: @port @NUMBER:port:@ ssh2

For details on using pattern databases to classify log messages, see Using pattern databases.

1.1 - The structure of the pattern database

The pattern database is organized as follows:

Pattern database structure

  • The pattern database consists of rulesets. A ruleset consists of a Program Pattern and a set of rules: the rules of a ruleset are applied to log messages if the name of the application that sent the message matches the Program Pattern of the ruleset. The name of the application (the content of the ${PROGRAM} macro) is compared to the Program Patterns of the available rulesets, and then the rules of the matching rulesets are applied to the message. (If the content of the ${PROGRAM} macro is not the proper name of the application, you can use the program-template() option to specify it.)

  • The Program Pattern can be a string that specifies the name of the application or the beginning of its name (for example, to match for sendmail, the program pattern can be sendmail, or just send), and the Program Pattern can contain pattern parsers. Note that pattern parsers are completely independent from the AxoSyslog parsers used to segment messages. Additionally, every rule has a unique identifier: if a message matches a rule, the identifier of the rule is stored together with the message.

  • Rules consist of a message pattern and a class. The Message Pattern is similar to the Program Pattern, but is applied to the message part of the log message (the content of the ${MESSAGE} macro). If a message pattern matches the message, the class of the rule is assigned to the message (for example, Security, Violation, and so on).

  • Rules can also contain additional information about the matching messages, such as the description of the rule, an URL, name-value pairs, or free-form tags.

  • Patterns can consist of literals (keywords, or rather, keycharacters) and pattern parsers.

1.2 - How pattern matching works

How pattern matching works

The followings describe how patterns work. This information applies to program patterns and message patterns alike, even though message patterns are used to illustrate the procedure.

Patterns can consist of literals (keywords, or rather, keycharacters) and pattern parsers. Pattern parsers attempt to parse a sequence of characters according to certain rules.

When a new message arrives, AxoSyslog attempts to classify it using the pattern database. The available patterns are organized alphabetically into a tree, and AxoSyslog inspects the message character-by-character, starting from the beginning. This approach ensures that only a small subset of the rules must be evaluated at any given step, resulting in high processing speed. Note that the speed of classifying messages is practically independent from the total number of rules.

For example, if the message begins with the Apple string, only patterns beginning with the character A are considered. In the next step, AxoSyslog selects the patterns that start with Ap, and so on, until there is no more specific pattern left. The AxoSyslog application has a strong preference for rules that match the input string completely.

Note that literal matches take precedence over pattern parser matches: if at a step there is a pattern that matches the next character with a literal, and another pattern that would match it with a parser, the pattern with the literal match is selected. Using the previous example, if at the third step there is the literal pattern Apport and a pattern parser Ap@STRING@, the Apport pattern is matched. If the literal does not match the incoming string (for example, Apple), AxoSyslog attempts to match the pattern with the parser. However, if there are two or more parsers on the same level, only the first one will be applied, even if it does not perfectly match the message.

If there are two parsers at the same level (for example, Ap@STRING@ and Ap@QSTRING@), it is random which pattern is applied (technically, the one that is loaded first). However, if the selected parser cannot parse at least one character of the message, the other parser is used. But having two different parsers at the same level is extremely rare, so the impact of this limitation is much less than it appears.

1.3 - Artificial ignorance

Artificial ignorance is a method used to detect anomalies. When applied to log analysis, it means that you ignore the regular, common log messages — these are the result of the regular behavior of your system, and therefore are not too concerning. However, new messages that have not appeared in the logs before can signal important events, and should be therefore investigated. “By definition, something we have never seen before is anomalous” (Marcus J. Ranum). 

The AxoSyslog application can classify messages using a pattern database: messages that do not match any pattern are classified as unknown. This provides a way to use artificial ignorance to review your log messages. You can periodically review the unknown messages — AxoSyslog can send them to a separate destination, and add patterns for them to the pattern database. By reviewing and manually classifying the unknown messages, you can iteratively classify more and more messages, until only the really anomalous messages show up as unknown.

Obviously, for this to work, a large number of message patterns are required. The radix-tree matching method used for message classification is very effective, can be performed very fast, and scales very well. Basically the time required to perform a pattern matching is independent from the number of patterns in the database. For sample pattern databases, see Downloading sample pattern databases.

2 - Using pattern databases

To classify messages using a pattern database, include a db-parser() statement in your syslog-ng.conf configuration file using the following syntax:

Declaration:

   parser <identifier> {
        db-parser(file("<database_filename>"));
    };

Note that using the parser in a log statement only performs the classification, but does not automatically do anything with the results of the classification.

Example: Defining pattern databases

The following statement uses the database located at /opt/syslog-ng/var/db/patterndb.xml.

   parser pattern_db {
        db-parser(
            file("/opt/syslog-ng/var/db/patterndb.xml")
        );
    };

To apply the patterns on the incoming messages, include the parser in a log statement:

   log {
        source(s_all);
        parser(pattern_db);
        destination( di_messages_class);
    };

By default, AxoSyslog tries to apply the patterns to the body of the incoming messages, that is, to the value of the $MESSAGE macro. If you want to apply patterns to a specific field, or to an expression created from the log message (for example, using template functions or other parsers), use the message-template() option. For example:

   parser pattern_db {
        db-parser(
            file("/opt/syslog-ng/var/db/patterndb.xml")
            message-template("${MY-CUSTOM-FIELD-TO-PROCESS}")
        );
    };

By default, AxoSyslog uses the name of the application (content of the ${PROGRAM} macro) to select which rules to apply to the message. If the content of the ${PROGRAM} macro is not the proper name of the application, you can use the program-template() option to specify it. For example:

   parser pattern_db {
        db-parser(
            file("/opt/syslog-ng/var/db/patterndb.xml")
            program-template("${MY-CUSTOM-FIELD-TO-SELECT-RULES}")
        );
    };

Note that the program-template() option is available in AxoSyslog version 3.21 and later.

Example: Using classification results

The following destination separates the log messages into different files based on the class assigned to the pattern that matches the message (for example, Violation and Security type messages are stored in a separate file), and also adds the ID of the matching rule to the message:

   destination di_messages_class {
        file(
            "/var/log/messages-${.classifier.class}"
            template("${.classifier.rule_id};${S_UNIXTIME};${SOURCEIP};${HOST};${PROGRAM};${PID};${MESSAGE}\n")
            template-escape(no)
        );
    };

Note that if you chain pattern databases, that is, use multiple databases in the same log path, the class assigned to the message (the value of ${.classifier.class}) will be the one assigned by the last pattern database. As a result, a message might be classified as unknown even if a previous parser successfully classified it. For example, consider the following configuration:

   log {
        ...
        parser(db_parser1);
        parser(db_parser2);
        ...
    };

Even if db_parser1 matches the message, db_parser2 might set ${.classifier.class} to unknown. To avoid this problem, you can use an ‘if’ statement to apply the second parser only if the first parser could not classify the message:

   log {
        ...
        parser{ db-parser(file("db_parser1.xml")); };
        if (match("^unknown$" value(".classifier.class"))) {
            parser { db-parser(file("db_parser2.xml")); };
        };
        ...
    };

For details on how to create your own pattern databases see The pattern database format.

Drop unmatched messages

If you want to automatically drop unmatched messages (that is, discard every message that does not match a pattern in the pattern database), use the drop-unmatched() option in the definition of the pattern database:

   parser pattern_db {
        db-parser(
            file("/opt/syslog-ng/var/db/patterndb.xml")
            drop-unmatched(yes)
        );
    };

Note that the drop-unmatched() option is available in AxoSyslog version 3.11 and later.

2.1 - Using parser results in filters and templates

The results of message classification and parsing can be used in custom filters and templates, for example, in file and database templates. The following built-in macros allow you to use the results of the classification:

  • The .classifier.class macro contains the class assigned to the message (for example, violation, security, or unknown).

  • The .classifier.rule_id macro contains the identifier of the message pattern that matched the message.

  • The .classifier.context_id macro contains the identifier of the context for messages that were correlated. For details on correlating messages, see Correlating log messages using pattern databases.

Example: Using classification results for filtering messages

To filter on a specific message class, create a filter that checks the .classifier_class macro, and use this filter in a log statement.

   filter fi_class_violation {
        match(
            "violation"
            value(".classifier.class")
            type("string")
        );
    };

   log {
        source(s_all);
        parser(pattern_db);
        filter(fi_class_violation);
        destination(di_class_violation);
    };

Filtering on the unknown class selects messages that did not match any rule of the pattern database. Routing these messages into a separate file allows you to periodically review new or unknown messages.

To filter on messages matching a specific classification rule, create a filter that checks the .classifier.rule_id macro. The unique identifier of the rule (for example, e1e9c0d8-13bb-11de-8293-000c2922ed0a) is the id attribute of the rule in the XML database.

   filter fi_class_rule {
        match(
            "e1e9c0d8-13bb-11de-8293-000c2922ed0a"
            value(".classifier.rule_id")
            type("string")
        );
    };

Pattern database rules can assign tags to messages. These tags can be used to select tagged messages using the tags() filter function.

The message-segments parsed by the pattern parsers can also be used as macros as well. To accomplish this, you have to add a name to the parser, and then you can use this name as a macro that refers to the parsed value of the message.

Example: Using pattern parsers as macros

For example, you want to parse messages of an application that look like "Transaction: <type>", where <type> is a string that has different values (for example, refused, accepted, incomplete, and so on). To parse these messages, you can use the following pattern:

   'Transaction: @ESTRING::.@'

Here the @ESTRING@ parser parses the message until the next full stop character. To use the results in a filter or a filename template, include a name in the parser of the pattern, for example:

   'Transaction: @ESTRING:TRANSACTIONTYPE:.@'

After that, add a custom template to the log path that uses this template. For example, to select every accepted transaction, use the following custom filter in the log path:

   match("accepted" value("TRANSACTIONTYPE"));

2.2 - Downloading sample pattern databases

To simplify the building of pattern databases, Axoflow has released sample databases. You can download sample pattern databases from the Axoflow GitHub page.

Note that these pattern databases are only samples and experimental databases. They are not officially supported, and may or may not work in your environment.

The pattern databases are available under the Creative Commons Attribution-Share Alike 3.0 (CC by-SA) license. This includes every pattern database written by community contributors or the Axoflow staff. It means that:

  • You are free to use and modify the patterns for your needs.

  • If you redistribute the pattern databases, you must distribute your modifications under the same license.

  • If you redistribute the pattern databases, you must make it obvious that the source of the original pattern databases is the GitHub page.

For legal details, the full text of the license is available here.

If you create patterns that are not available in the GitHub repository, consider sharing them with the community. To do this, open a GitHub issue, or contact us.

2.3 - Correlating log messages using pattern databases

The AxoSyslog application can correlate log messages identified using pattern databases. Alternatively, you can also correlate log messages using the grouping-by() parser. For details, see Correlating messages using the grouping-by() parser.

Log messages are supposed to describe events, but applications often separate information about a single event into different log messages. For example, the Postfix email server logs the sender and recipient addresses into separate log messages, or in case of an unsuccessful login attempt, the OpenSSH server sends a log message about the authentication failure, and the reason of the failure in the next message. Of course, messages that are not so directly related can be correlated as well, for example, login-logout messages, and so on.

To correlate log messages with AxoSyslog, you can add messages into message-groups called contexts. A context consists of a series of log messages that are related to each other in some way, for example, the log messages of an SSH session can belong to the same context. As new messages come in, they may be added to a context. Also, when an incoming message is identified it can trigger actions to be performed, for example, generate a new message that contains all the important information that was stored previously in the context.

(For details on triggering actions and generating messages, see Triggering actions for identified messages.)

There are two attributes for pattern database rules that determine if a message matching the rule is added to a context: context-scope and context-id. The context-scope attribute acts as an early filter, selecting messages sent by the same process (${HOST}${PROGRAM}${PID} is identical), application (${HOST}${PROGRAM} is identical), or host, while the context-id actually adds the message to the context specified in the id. The context-id can be a simple string, or can contain macros or values extracted from the log messages for further filtering. Starting with AxoSyslog version 3.5, if a message is added to a context, AxoSyslog automatically adds the identifier of the context to the .classifier.context_id macro of the message.

Another parameter of a rule is the context-timeout attribute, which determines how long a context is stored, that is, how long AxoSyslog waits for related messages to arrive.

Note the following points about timeout values:

  • When a new message is added to a context, AxoSyslog will restart the timeout using the context-timeout set for the new message.

  • When calculating if the timeout has already expired or not, AxoSyslog uses the timestamps of the incoming messages, not system time elapsed between receiving the two messages (unless the messages do not include a timestamp, or the keep-timestamp(no) option is set). That way AxoSyslog can be used to process and correlate already existing log messages offline. However, the timestamps of the messages must be in chronological order (that is, a new message cannot be older than the one already processed), and if a message is newer than the current system time (that is, it seems to be coming from the future), AxoSyslog will replace its timestamp with the current system time.

    Example: How AxoSyslog calculates context-timeout

    Consider the following two messages:

        <38>1990-01-01T14:45:25 customhostname program6[1234]: program6 testmessage
    <38>1990-01-01T14:46:25 customhostname program6[1234]: program6 testmessage

    If the context-timeout is 10 seconds and AxoSyslog receives the messages within 1 second, the timeout event will occour immediately, because the difference of the two timestamp (60 seconds) is larger than the timeout value (10 seconds).

  • Avoid using unnecessarily long timeout values on high-traffic systems, as storing the contexts for many messages can require considerable memory. For example, if two related messages usually arrive within seconds, it is not needed to set the timeout to several hours.

Example: Using message correlation

   <rule xml:id="..." context-id="ssh-session" context-timeout="86400" context-scope="process">
        <patterns>
            <pattern>Accepted @ESTRING:usracct.authmethod: @for @ESTRING:usracct.username: @from @ESTRING:usracct.device: @port @ESTRING:: @@ANYSTRING:usracct.service@</pattern>
        </patterns>
    ...
    </rule>

For details on configuring message correlation, see the context-id, context-timeout, and context-scope attributes of pattern database rules.

2.3.1 - Referencing earlier messages of the context

When using the <value> element in pattern database rules together with message correlation, you can also refer to fields and values of earlier messages of the context by adding the @<distance-of-referenced-message-from-the-current> suffix to the macro. For example, if there are three log messages in a context, and you are creating a generated message for the third log message, the ${HOST}@1 expression refers to the host field of the current (third) message in the context, the ${HOST}@2 expression refers to the host field of the previous (second) message in the context, ${PID}@3 to the PID of the first message, and so on. For example, the following message can be created from SSH login/logout messages (for details on generating new messages, see Triggering actions for identified messages): An SSH session for ${SSH_USERNAME}@1 from ${SSH_CLIENT_ADDRESS}@2 closed. Session lasted from ${DATE}@2 to ${DATE}.

Example: Referencing values from an earlier message

The following action can be used to log the length of an SSH session (the time difference between a login and a logout message in the context):

   <actions>
        <action>
            <message>
                <values>
                    <value name="MESSAGE">An SSH session for ${SSH_USERNAME}@1 from ${SSH_CLIENT_ADDRESS}@2 closed. Session lasted from ${DATE}@2 to ${DATE} </value>
                </values>
            </message>
        </action>
    </actions>

If you do not know in which message of the context contains the information you need, you can use the grep, the context-lookup, or the context-values template functions.

Example: Using the grep template function

The following example selects the message of the context that has a username name-value pair with the root value, and returns the value of the auth_method name-value pair.

   $(grep ("${username}" == "root") ${auth_method})

To perform calculations on fields that have numerical values, see Numerical operations.

3 - Triggering actions for identified messages

The AxoSyslog application can generate (trigger) messages automatically if certain events occur, for example, a specific log message is received, or the correlation timeout of a message expires. Basically, you can define messages for every pattern database rule that are emitted when a message matching the rule is received. Triggering messages is often used together with message correlation, but can also be used separately. When used together with message correlation, you can also create a new correlation context when a new message is received.

The generated message is injected into the same place where the db-parser() statement is referenced in the log path. To post the generated message into the internal() source instead, use the inject-mode() option in the definition of the parser.

Example: Sending triggered messages to the internal() source

To send the generated messages to the internal source, use the inject-mode(internal) option:

   parser p_db {
        db-parser(
            file("mypatterndbfile.xml")
            inject-mode(internal)
        );
    };

To inject the generated messages where the pattern database is referenced, use the inject-mode(pass-through) option:

   parser p_db {
        db-parser(
            file("mypatterndbfile.xml")
            inject-mode(pass-through)
        );
    };

The generated message must be configured in the pattern database rule. It is possible to create an entire message, use macros and values extracted from the original message with pattern database, and so on.

Example: Generating messages for pattern database matches

When inserted in a pattern database rule, the following example generates a message when a message matching the rule is received.

   <actions>
        <action>
            <message>
                <values>
                    <value name="MESSAGE">A log message from ${HOST} matched rule number $.classifier.rule_id</value>
                </values>
            </message>
        </action>
    </actions>

To inherit the properties and values of the triggering message, set the inherit-properties attribute of the <message> element to TRUE. That way the triggering log message is cloned, including name-value pairs and tags. If you set any values for the message in the <action> element, they will override the values of the original message.

Example: Generating messages with inherited values

The following action generates a message that is identical to the original message, but its $PROGRAM field is set to overriding-original-program-name

   <actions>
        <action>
            <message inherit-properties='TRUE'>
                <values>
                    <value name="PROGRAM">overriding-original-program-name</value>
                </values>
            </message>
        </action>
    </actions>

Example: Creating a new context from an action

In AxoSyslog version 3.8 and newer, you can create a new context as an action. For details, see Element: create-context.

The following example creates a new context whenever the rule matches. The new context receives 1000 as ID, and program as scope, and the content set in the <message> element of the >element.

   <rule provider='test' id='12' class='violation'>
      <patterns>
        <pattern>simple-message-with-action-to-create-context</pattern>
      </patterns>
      <actions>
        <action trigger='match'>
          <create-context context-id='1000' context-timeout='60' context-scope='program'>
            <message inherit-properties='context'>
              <values>
                <value name='MESSAGE'>context message</value>
              </values>
            </message>
          </create-context>
        </action>
      </actions>
    </rule>

For details on configuring actions, see the description of the pattern database format.

3.1 - Conditional actions

To limit when a message is triggered, use the condition attribute and specify a filter expression: the action will be executed only if the condition is met. For example, the following action is executed only if the message was sent by the host called myhost.

   <action condition="'${HOST}' == 'myhost'">

You can use the same operators in the condition that can be used in filters. For details, see Comparing macro values in filters.

The following action can be used to log the length of an SSH session (the time difference between a login and a logout message in the context):

   <actions>
        <action>
            <message>
                <values>
                    <value name="MESSAGE">An SSH session for ${SSH_USERNAME}@1 from ${SSH_CLIENT_ADDRESS}@2 closed. Session lasted from ${DATE}@2 ${DATE} </value>
                </values>
            </message>
        </action>
    </actions>

Example: Actions based on the number of messages

The following example triggers different actions based on the number of messages in the context. This way you can check if the context contains enough messages for the event to be complete, and execute a different action if it does not.

   <actions>
        <action condition='"$(context-length)" >= "4"'>
            <message>
                <values>
                    <value name="PROGRAM">event</value>
                    <value name="MESSAGE">Event complete</value>
                </values>
            </message>
        </action>
        <action condition='"$(context-length)" < "4"'>
            <message>
                <values>
                    <value name="PROGRAM">error</value>
                <value name="MESSAGE">Error detected</value>
                </values>
            </message>
        </action>
    </actions>

3.2 - External actions

To perform an external action when a message is triggered, for example, to send the message in an email, you have to route the generated messages to an external application using the program() destination.

Example: Sending triggered messages to external applications

The following sample configuration selects the triggered messages and sends them to an external script.

  1. Set a field in the triggered message that is easy to identify and filter. For example:

        <values>
        <value name="MESSAGE">A log message from ${HOST} matched rule number $.classifier.rule_id</value>
        <value name="TRIGGER">yes</value>
    </values>

  2. Create a destination that will process the triggered messages.

        destination d_triggers {
        program("/bin/myscript"; );
    };

  3. Create a filter that selects the triggered messages from the internal source.

        filter f_triggers {
        match("yes" value ("TRIGGER") type(string));
    };

  4. Create a logpath that selects the triggered messages from the internal source and sends them to the script:

        log { source(s_local); filter(f_triggers); destination(d_triggers); };

  5. Create a script that will actually process the generated messages, for example:

        #!/usr/bin/perl
    while (<>) {
        # body of the script to send emails, snmp traps, and so on
    }

3.3 - Actions and message correlation

Certain features of generating messages can be used only if message correlation is used as well. For details on correlating messages, see Correlating log messages using pattern databases.

  • The AxoSyslog application automatically fills the fields for the generated message based on the scope of the context, for example, the HOST and PROGRAM fields if the context-scope is program.

  • When used together with message correlation, you can also refer to fields and values of earlier messages of the context by adding the @<distance-of-referenced-message-from-the-current> suffix to the macro. For details, see Referencing earlier messages of the context.

    Example: Referencing values from an earlier message

    The following action can be used to log the length of an SSH session (the time difference between a login and a logout message in the context):

       <actions>
        <action>
            <message>
                <values>
                    <value name="MESSAGE">An SSH session for ${SSH_USERNAME}@1 from ${SSH_CLIENT_ADDRESS}@2 closed. Session lasted from ${DATE}@2 to ${DATE} </value>
                </values>
            </message>
        </action>
    </actions>

  • You can use the name-value pairs of other messages of the context. If you set the inherit-properties attribute of the generated message to context, AxoSyslog collects every name-value pair from each message stored in the context, and includes them in the generated message. This means that you can refer to a name-value pair without having to know which message of the context included it. If a name-value pair appears in multiple messages of the context, the value in the latest message will be used. To refer to an earlier value, use the @<distance-of-referenced-message-from-the-current> suffix format.

        <action>
        <message inherit-properties='context'>

    Example: Using the inherit-properties option

    For example, if inherit-properties is set to context, and you have a rule that collects SSH login and logout messages to the same context, you can use the following value to generate a message collecting the most important information form both messages, including the beginning and end date.

        <value name="MESSAGE">An SSH session for ${SSH_USERNAME} from ${SSH_CLIENT_ADDRESS} closed. Session lasted from ${DATE}@2 to $DATE pid: $PID.</value>

    The following is a detailed rule for this purpose.

        <patterndb version='4' pub_date='2015-04-13'>
        <ruleset name='sshd' id='12345678'>
            <pattern>sshd</pattern>
                <rules>
                    <!-- The pattern database rule for the first log message -->
                    <rule provider='me' id='12347598' class='system'
                        context-id="ssh-login-logout" context-timeout="86400"
                        context-scope="process">
                    <!-- Note the context-id that groups together the
                    relevant messages, and the context-timeout value that
                    determines how long a new message can be added to the
                    context  -->
                        <patterns>
                            <pattern>Accepted @ESTRING:SSH.AUTH_METHOD: @for @ESTRING:SSH_USERNAME: @from @ESTRING:SSH_CLIENT_ADDRESS: @port @ESTRING:: @@ANYSTRING:SSH_SERVICE@</pattern>
                            <!-- This is the actual pattern used to identify
                            the log message. The segments between the @
                            characters are parsers that recognize the variable
                            parts of the message - they can also be used as
                            macros.  -->
                        </patterns>
                    </rule>
                    <!-- The pattern database rule for the fourth log message -->
                    <rule provider='me' id='12347599' class='system' context-id="ssh-login-logout" context-scope="process">
                        <patterns>
                             <pattern>pam_unix(sshd:session): session closed for user @ANYSTRING:SSH_USERNAME@</pattern>
                        </patterns>
                        <actions>
                            <action>
                                <message inherit-properties='context'>
                                    <values>
                                        <value name="MESSAGE">An SSH session for ${SSH_USERNAME} from ${SSH_CLIENT_ADDRESS} closed. Session lasted from ${DATE}@2 to $DATE pid: $PID.</value>
                                        <value name="TRIGGER">yes</value>
                                        <!-- This is the new log message
                                        that is generated when the logout
                                        message is received. The macros ending
                                        with @2 reference values of the
                                        previous message from the context. -->
                                    </values>
                                </message>
                            </action>
                        </actions>
                    </rule>
                </rules>
        </ruleset>
    </patterndb>

  • It is possible to generate a message when the context-timeout of the original message expires and no new message is added to the context during this time. To accomplish this, include the trigger="timeout" attribute in the action element:

        <action trigger="timeout">

    Example: Sending alert when a client disappears

    The following example shows how to combine various features of AxoSyslog to send an email alert if a client stops sending messages.

    • Configure your clients to send MARK messages periodically. It is enough to configure MARK messages for the destination that forwards your log messages to your AxoSyslog server (mark-mode(periodical)).

    • On your AxoSyslog server, create a pattern database rule that matches on the incoming MARK messages. In the rule, set the context-scope attribute to host, and the context-timeout attribute to a value that is higher than the mark-freq value set on your clients (by default, mark-freq is 1200 seconds, so set context-timeout at least to 1500 seconds, but you might want to use a higher value, depending on your environment).

    • Add an action to this rule that sends you an email alert if the context-timeout expires, and the server does not receive a new MARK message (<action trigger="timeout">).

    • On your AxoSyslog server, use the pattern database in the log path that handles incoming log messages.

4 - Creating pattern databases

4.1 - Using pattern parsers

Pattern parsers attempt to parse a part of the message using rules specific to the type of the parser. Parsers are enclosed between @ characters. The syntax of parsers is the following:

  • a beginning @ character,

  • the type of the parser written in capitals,

  • optionally a name,

  • parameters of the parser, if any, and

  • a closing @ character.

Example: Pattern parser syntax

A simple parser:

   @STRING@

A named parser:

   @STRING:myparser_name@

A named parser with a parameter:

   @STRING:myparser_name:*@

A parser with a parameter, but without a name:

   @STRING::*@

Patterns and literals can be mixed together. For example, to parse a message that begins with the Host: string followed by an IP address (for example, Host: 192.168.1.1), the following pattern can be used: Host:@IPv4@.

Example: Using the STRING and ESTRING parsers

For example, look at the following message: user=joe96 group=somegroup.

  • @STRING:mytext:@ parses only to the first non-alphanumeric character (=), parsing only user, so the value of the ${mytext} macro will be user

  • @STRING:mytext:=@ parses the equation mark as well, and proceeds to the next non-alphanumeric character (the whitespace), resulting in user=joe96

  • @STRING:mytext:= @ will parse the whitespace as well, and proceed to the next non-alphanumeric non-equation mark non-whitespace character, resulting in user=joe96 group=somegroup

Of course, usually it is better to parse the different values separately, like this: "user=@STRING:user@ group=@STRING:group@".

If the username or the group may contain non-alphanumeric characters, you can either include these in the second parameter of the parser (as shown at the beginning of this example), or use an ESTRING parser to parse the message till the next whitespace: "user=@ESTRING:user: @group=@ESTRING:group: @".

4.1.1 - Pattern parsers

The following parsers are available in AxoSyslog. The internal parsers (for example, @NUMBER@) automatically associate type information to the parsed name-value pair. For details on data types, see Specifying data types in value-pairs.

@ANYSTRING@

Parses everything to the end of the message, you can use it to collect everything that is not parsed specifically to a single macro. In that sense its behavior is similar to the greedy() option of the CSV parser.

@DOUBLE@

An obsolete alias of the @FLOAT@ parser.

@EMAIL@

This parser matches an email address. The parameter is a set of characters to strip from the beginning and the end of the email address. That way email addresses enclosed between other characters can be matched easily (for example, <[email protected]> or "[email protected]". Characters that are valid for a hostname are not stripped from the end of the hostname. This includes a trailing period if present.

For example, the @EMAIL:email:"[<]> parser will match any of the following email addresses: <[email protected]>, [[email protected]], "[email protected]", and set the value of the email macro to [email protected].

@ESTRING@

This parser has a required parameter that acts as the stopcharacter: the parser parses everything until it finds the stopcharacter. For example, to stop by the next " (double quote) character, use @ESTRING::"@. You can use the colon (:) as stopcharacter as well, for example: @ESTRING:::@. You can also specify a stopstring instead of a single character, for example, @ESTRING::stop_here.@. The @ character cannot be a stopcharacter, nor can line-breaks or tabs.

@FLOAT@

A floating-point number that may contain a dot (.) character. (Up to AxoSyslog 3.1, the name of this parser was @DOUBLE@.)

@HOSTNAME@

Parses a generic hostname. The hostname may contain only alphanumeric characters (A-Z,a-z,0-9), hypen (-), or dot (.).

@IPv4@

Parses an IPv4 IP address (numbers separated with a maximum of 3 dots).

@IPv6@

Parses any valid IPv6 IP address.

@IPvANY@

Parses any IP address.

@LLADDR@

Parses a Link Layer Address in the xx:xx:xx:... form, where each xx is a 2 digit HEX number (an octet). The parameter specifies the maximum number of octets to match and defaults to 20. The MACADDR parser is a special wrapper using the LLADDR parser. For example, the following parser parses maximally 10 octets, and stores the results in the link-level-address macro:

   @LLADDR:link-level-address:10@

@MACADDR@

Parses the standard format of a MAC-48 address, consisting of is six groups of two hexadecimal digits, separated by colons. For example, 00:50:fc:e3:cd:37.

@NLSTRING@

This parser parses everything until the next new-line character (more precisely, until the next Unix-style LF or Windows-style CRLF character). For single-line messages, NLSTRING is equivalent with ANYSTRING. For multi-line messages, NLSTRING parses to the end of the current line, while ANYSTRING parses to the end of the message. Using NLSTRING is useful when parsing multi-line messages, for example, Windows logs. For example, the following pattern parses information from Windows security auditing logs.

   <pattern>Example-PC\Example: Security Microsoft Windows security auditing.: [Success Audit] A new process has been created.
    
        Subject:
        Security ID: @LNSTRING:.winaudit.SubjectUserSid@
        Account Name: @LNSTRING:.winaudit.SubjectUserName@
        Account Domain: @LNSTRING:.winaudit.SubjectDomainName@
        Logon ID: @LNSTRING:.winaudit.SubjectLogonId@
    
        Process Information:
        New Process ID: @LNSTRING:.winaudit.NewProcessId@
        New Process Name: @LNSTRING:.winaudit.NewProcessName@
        Token Elevation Type: @LNSTRING:.winaudit.TokenElevationType@
        Creator Process ID: @LNSTRING:.winaudit.ProcessId@
        Process Command Line: @LNSTRING:.winaudit.CommandLine@
    
        Token Elevation Type indicates the type of token that was assigned to the new process in accordance with User Account Control policy.</pattern>

@NUMBER@

A sequence of decimal (0-9) numbers (for example, 1, 0687, and so on). Note that if the number starts with the 0x characters, it is parsed as a hexadecimal number, but only if at least one valid character follows 0x. A leading hyphen (-) is accepted for non-hexadecimal numbers, but other separator characters (for example, dot or comma) are not. To parse floating-point numbers, use the @FLOAT@ parser.

@OPTIONALSET@

Parse any combination of the specified characters. For example, specifying a whitespace character parses any number of whitespaces, and can be used to process paddings (for example, log messages of the Squid application have whitespace padding after the username).

For example, the @OPTIONALSET:: "@ parser will parse any combination of whitespaces and double-quotes.

Available in 3.31 and later.

@PCRE@

Use Perl-Compatible Regular Expressions (as implemented by the PCRE library), after the identification of the potential patterns has happened by the radix implementation.

Syntax: @PCRE:name:regexp@

@QSTRING@

Parse a string between the quote characters specified as parameter. Note that the quote character can be different at the beginning and the end of the quote, for example: @QSTRING::"@ parses everything between two quotation marks ("), while @QSTRING:\&lt;\&gt;@ parses from an opening bracket to the closing bracket. The @ character cannot be a quote character, nor can line-breaks or tabs.

@SET@

Parse any combination of the specified characters until another character is found. For example, specifying a whitespace character parses any number of whitespaces, and can be used to process paddings (for example, log messages of the Squid application have whitespace padding after the username).

For example, the @SET:: "@ parser will parse any combination of whitespaces and double-quotes.

Available in AxoSyslog 3.4 and later.

@STRING@

A sequence of alphanumeric characters (0-9, A-z), not including any whitespace. Optionally, other accepted characters can be listed as parameters (for example, to parse a complete sentence, add the whitespace as parameter, like: @STRING:: @). Note that the @ character cannot be a parameter, nor can line-breaks or tabs.

4.2 - What's new in the pattern database format V5

The V5 database format has the following differences compared to the V4 format:

  • The <ruleset> element can now store multiple reference URLs using the new <rule_urls> element. For details, see Element: ruleset.

  • In an <action>, you can now initialize a new context. As a result, the <message> element is not required. For details, see Element: create-context.

  • The inherit-properties attribute is deprecated, use the inherit-mode attribute instead. For details, see Element: action.

4.3 - The pattern database format

Pattern databases are XML files that contain rules describing the message patterns. For sample pattern databases, see Downloading sample pattern databases.

The following scheme describes the V5 format of the pattern database. This format is backwards-compatible with the earlier formats.

For a sample database containing only a single pattern, see Example: A pattern database containing a single rule.

Example: A pattern database containing a single rule

The following pattern database contains a single rule that matches a log message of the ssh application. A sample log message looks like:

   Accepted password for sampleuser from 10.50.0.247 port 42156 ssh2

The following is a simple pattern database containing a matching rule.

   <patterndb version='5' pub_date='2010-10-17'>
        <ruleset name='ssh' id='123456678'>
            <pattern>ssh</pattern>
                <rules>
                    <rule provider='me' id='182437592347598' class='system'>
                        <patterns>
                            <pattern>Accepted @QSTRING:SSH.AUTH_METHOD: @ for@QSTRING:SSH_USERNAME: @from\ @QSTRING:SSH_CLIENT_ADDRESS: @port @NUMBER:SSH_PORT_NUMBER:@ ssh2</pattern>
                        </patterns>
                    </rule>
                </rules>
        </ruleset>
    </patterndb>

Note that the rule uses macros that refer to parts of the message, for example, you can use the ${SSH_USERNAME} macro refer to the username used in the connection.

The following is the same example, but with a test message and test values for the parsers.

   <patterndb version='4' pub_date='2010-10-17'>
        <ruleset name='ssh' id='123456678'>
            <pattern>ssh</pattern>
                <rules>
                    <rule provider='me' id='182437592347598' class='system'>
                        <patterns>
                            <pattern>Accepted @QSTRING:SSH.AUTH_METHOD: @ for@QSTRING:SSH_USERNAME: @from\ @QSTRING:SSH_CLIENT_ADDRESS: @port @NUMBER:SSH_PORT_NUMBER:@ ssh2</pattern>
                        </patterns>
                        <examples>
                            <example>
                                <test_message>Accepted password for sampleuser from 10.50.0.247 port 42156 ssh2</test_message>
                                <test_values>
                                    <test_value name="SSH.AUTH_METHOD">password</test_value>
                                    <test_value name="SSH_USERNAME">sampleuser</test_value>
                                    <test_value name="SSH_CLIENT_ADDRESS">10.50.0.247</test_value>
                                    <test_value name="SSH_PORT_NUMBER">42156</test_value>
                                </test_values>
                           </example>
                        </examples>
                    </rule>
                </rules>
        </ruleset>
    </patterndb>

4.3.1 - Element: patterndb

Location

/patterndb

Description

The container element of the pattern database.

Attributes

  • version: The schema version of the pattern database. The current version is 4.

  • pubdate: The publication date of the XML file.

Children

  • ruleset

Example

   <patterndb version='4' pub_date='2010-10-25'>

4.3.2 - Element: ruleset

Location

/patterndb/ruleset

Description

A container element to group log patterns for an application or program. A <patterndb> element may contain any number of <ruleset> elements.

Attributes

  • name: The name of the application. Note that the function of this attribute is to make the database more readable, syslog-ng uses the <pattern> element to identify the applications sending log messages.

  • id: A unique ID of the application, for example, the md5 sum of the name attribute.

Children

  • patterns

  • rules

  • actions

  • tags

  • description: OPTIONAL — A description of the ruleset or the application.

  • url: OPTIONAL — An URL referring to further information about the ruleset or the application.

  • rule_urls: OPTIONAL — To list multiple URLs referring to further information about the ruleset or the application, enclose the <url> elements into an <urls> element.

Example

   <ruleset name='su' id='480de478-d4a6-4a7f-bea4-0c0245d361e1'>

4.3.3 - Element: patterns

Location

/patterndb/ruleset/patterns

Description

A container element. A <patterns> element may contain any number of `elements.

Attributes

N/A

Children

  • pattern: The name of the application — syslog-ng matches this value to the ${PROGRAM} header of the syslog message to find the rulesets applicable to the syslog message.

    Specifying multiple patterns is useful if two or more applications have different names (that is, different ${PROGRAM} fields), but otherwise send identical log messages.

    It is not necessary to use multiple patterns if only the end of the ${PROGRAM} fields is different, use only the beginning of the ${PROGRAM} field as the pattern. For example, the Postfix email server sends messages using different process names, but all of them begin with the postfix string.

    You can also use parsers in the program pattern if needed, and use the parsed results later. For example: `stfix\@ESTRING:.postfix.component:[@»

Example

   <patterns>
        <pattern>firstapplication</pattern>
        <pattern>otherapplication</pattern>
    </patterns>

Using parsers in the program pattern:

   <pattern>postfix\@ESTRING:.postfix.component:[@</pattern>

4.3.4 - Element: rules

Location

/patterndb/ruleset/rules

Description

A container element for the rules of the ruleset.

Attributes

N/A

Children

  • rule

Example

   <rules>
        <rule provider='me' id='182437592347598' class='system'>
            <patterns>
                <pattern>Accepted @QSTRING:SSH.AUTH_METHOD: @ for@QSTRING:SSH_USERNAME: @from\ @QSTRING:SSH_CLIENT_ADDRESS: @port @NUMBER:SSH_PORT_NUMBER:@ ssh2</pattern>
            </patterns>
        </rule>
    </rules>

4.3.5 - Element: rule

Location

/patterndb/ruleset/rules/rule

Description

An element containing message patterns and how a message that matches these patterns is classified.

The element may contain any number of elements.

Attributes

  • provider: The provider of the rule. This is used to distinguish between who supplied the rule, that is, if it has been created by Axoflow, or added to the XML by a local user.

  • id: The globally unique ID of the rule.

  • class: The class of the rule — this class is assigned to the messages matching a pattern of this rule.

Children

  • patterns

Example

   <rule provider='example' id='f57196aa-75fd-11dd-9bba-001e6806451b' class='violation'>

The following example specifies attributes for correlating messages as well. For details on correlating messages, see Correlating log messages using pattern databases.

   <rule provider='example' id='f57196aa-75fd-11dd-9bba-001e6806451b' class='violation' context-id='same-session' context-scope='process' context-timeout='360'>

4.3.6 - Element: patterns

Location

/patterndb/ruleset/rules/rule/patterns

Description

An element containing the patterns of the rule. If a element contains multiple *lements, the class of the *»>ssigned to every syslog message matching any of the patterns.

Attributes

N/A

Children

  • pattern: A pattern describing a log message. This element is also called message pattern. For example:

        <pattern>+ ??? root-</pattern>

  • description: OPTIONAL — A description of the pattern or the log message matching the pattern.

  • urls

  • values

  • examples

Example

   <patterns>
        <pattern>Accepted @QSTRING:SSH.AUTH_METHOD: @ for@QSTRING:SSH_USERNAME: @from\ @QSTRING:SSH_CLIENT_ADDRESS: @port @NUMBER:SSH_PORT_NUMBER:@ ssh2</pattern>
    </patterns>

4.3.7 - Element: urls

Location

/patterndb/ruleset/rules/rule/patterns/urls

Description

OPTIONAL — An element containing one or more URLs referring to further information about the patterns or the matching log messages.

Attributes

N/A

Children

  • url: OPTIONAL — An URL referring to further information about the patterns or the matching log messages.

Example

N/A

4.3.8 - Element: values

Location

/patterndb/ruleset/rules/rule/patterns/values

Description

OPTIONAL — Name-value pairs that are assigned to messages matching the patterns, for example, the representation of the event in the message according to the Common Event Format (CEF) or Common Event Exchange (CEE). The names can be used as macros to reference the assigned values.

Attributes

N/A

Children

  • value: OPTIONAL — Contains the value of the name-value pair that is assigned to the message.

    The <value> element of name-value pairs can include template functions. For details, see Using template functions, for examples, see if.

    You can associate types with values using the "type" attribute, for example, integer is a type-cast that associates $foobar with an integer type. For details on data types, see Specifying data types in value-pairs.

    <value name="foobar" type="integer">$PID</value>

    db-parser()’s internal parsers (for example, @NUMBER@) automatically associate type information to the parsed name-value pair.

    When used together with message correlation, the <value> element of name-value pairs can include references to the values of earlier messages from the same context. For details, see Correlating log messages using pattern databases.

  • name: The name of the name-value pair. It can also be used as a macro to reference the assigned value.

Example

   <values>
        <value name=".classifier.outcome">/Success</value>
    </values>

4.3.9 - Element: actions

Location

/patterndb/ruleset/actions

Description

OPTIONAL — A container element for actions that are performed if a message is recognized by the pattern. For details on actions, see Triggering actions for identified messages.

Attributes

N/A

Children

  • action

Example

Example: Generating messages for pattern database matches

When inserted in a pattern database rule, the following example generates a message when a message matching the rule is received.

   <actions>
        <action>
            <message>
                <values>
                    <value name="MESSAGE">A log message from ${HOST} matched rule number $.classifier.rule_id</value>
                </values>
            </message>
        </action>
    </actions>

To inherit the properties and values of the triggering message, set the inherit-properties attribute of the <message> element to TRUE. That way the triggering log message is cloned, including name-value pairs and tags. If you set any values for the message in the <action> element, they will override the values of the original message.

Example: Generating messages with inherited values

The following action generates a message that is identical to the original message, but its $PROGRAM field is set to overriding-original-program-name

   <actions>
        <action>
            <message inherit-properties='TRUE'>
                <values>
                    <value name="PROGRAM">overriding-original-program-name</value>
                </values>
            </message>
        </action>
    </actions>

4.3.10 - Element: action

Location

/patterndb/ruleset/actions/action

Description

OPTIONAL — A container element describing an action that is performed when a message matching the rule is received.

Attributes

  • condition: An AxoSyslog filter expression. The action is performed only if the message matches the filter. The filter can include macros and name-value pairs extracted from the message. When using actions together with message-correlation, you can also use the $(context-length) macro, which returns the number of messages in the current context. For example, this can be used to determine if the expected number of messages has arrived to the context: condition='"$(context-length)" > "5"'

  • rate: Specifies maximum how many messages should be generated in the specified time period in the following format: <number-of-messages>/<period-in-seconds>. For example: 1/60 allows 1 message per minute. Rates apply within the scope of the context, that is, if context-scope="host" and rate="1/60", then maximum one message is generated per minute for every host that sends a log message matching the rule. Excess messages are dropped. Note that when applying the rate to the generated messages, AxoSyslog uses the timestamps of the log messages, similarly to calculating the context-timeout. That way rate is applied correctly even if the log messages are processed offline.

  • trigger: Specifies when the action is executed. The trigger attribute has the following possible values:

    • match: Execute the action immediately when a message matching the rule is received.

    • timeout: Execute the action when the correlation timer (context-timeout) of the pattern database rule expires. This is available only if actions are used together with correlating messages.

Children

  • create-context

  • message: A container element storing the message to be sent when the action is executed. Currently AxoSyslog sends these messages to the internal() destination.

    • For details on the message context, see Correlating log messages using pattern databases and Actions and message correlation. For details on triggering messages, see Triggering actions for identified messages

      inherit-mode: This attribute controls which name-value pairs and tags are propagated to the newly generated message.

      • context: AxoSyslog collects every name-value pair from each message stored in the context, and includes them in the generated message. If a name-value pair appears in multiple messages of the context, the value in the latest message will be used. Note that tags are not merged, the generated message will inherit the tags assigned to the last message of the context.

      • last-message: Only the name-value pairs appearing in the last message are copied. If the context contains only a single message, then it is the message that triggered the action.

      • none: An empty message is created, without inheriting any tags or name-value pairs.

      This option is available in AxoSyslog 3.8 and later.

    • inherit-properties: This attribute is deprecated. Use the inherit-mode attribute instead.

      If set to TRUE, the original message that triggered the action is cloned, including its name-value pairs and tags.

      If set to context, AxoSyslog collects every name-value pair from each message stored in the context, and includes them in the generated message. If a name-value pair appears in multiple messages of the context, the value in the latest message will be used. Note that tags are not merged, the generated message will inherit the tags assigned to the last message of the context.

      For details on the message context, see Correlating log messages using pattern databases and Actions and message correlation. For details on triggering messages, see Triggering actions for identified messages

      This option is available in AxoSyslog 5.3.2 and later.

  • values: A container element for values and fields that are used to create the message generated by the action.

    • value: Sets the value of the message field specified in the name attribute of the element. For example, to specify the body of the generated message, use the following syntax:

      
    <value name="MESSAGE">A log message matched rule number $.classifier.rule_id</value>

      Note that currently it is not possible to add DATE, FACILITY, or SEVERITY fields to the message.

      When the action is used together with message correlation, the AxoSyslog application automatically adds fields to the message based on the context-scope parameter. For example, using context-scope="process" automatically fills the HOST, PROGRAM, and PID fields of the generated message.

    • name: Name of the message field set by the value element.

Example: Generating messages for pattern database matches

When inserted in a pattern database rule, the following example generates a message when a message matching the rule is received.

   <actions>
        <action>
            <message>
                <values>
                    <value name="MESSAGE">A log message from ${HOST} matched rule number $.classifier.rule_id</value>
                </values>
            </message>
        </action>
    </actions>

To inherit the properties and values of the triggering message, set the inherit-properties attribute of the <message> element to TRUE. That way the triggering log message is cloned, including name-value pairs and tags. If you set any values for the message in the <action> element, they will override the values of the original message.

Example: Generating messages with inherited values

The following action generates a message that is identical to the original message, but its $PROGRAM field is set to overriding-original-program-name

   <actions>
        <action>
            <message inherit-properties='TRUE'>
                <values>
                    <value name="PROGRAM">overriding-original-program-name</value>
                </values>
            </message>
        </action>
    </actions>

4.3.11 - Element: create-context

Location

/patterndb/ruleset/actions/action/create-context

Description

OPTIONAL — Creates a new correlation context from the current message and its associated context. This can be used to “split” a context.

Available in AxoSyslog version 3.8 and later.

Attributes

Children

  • message: A container element storing the message that is added to the new context when the action is executed.

    • inherit-mode: This attribute controls which name-value pairs and tags are propagated to the newly generated message.

      • context: AxoSyslog collects every name-value pair from each message stored in the context, and includes them in the generated message. If a name-value pair appears in multiple messages of the context, the value in the latest message will be used. Note that tags are not merged, the generated message will inherit the tags assigned to the last message of the context.

      • last-message: Only the name-value pairs appearing in the last message are copied. If the context contains only a single message, then it is the message that triggered the action.

      • none: An empty message is created, without inheriting any tags or name-value pairs.

      For details on the message context, see Correlating log messages using pattern databases and Actions and message correlation. For details on triggering messages, see Triggering actions for identified messages

Example

The following example creates a new context whenever the rule matches. The new context receives 1000 as ID, and program as scope, and the content set in the <message> element of the >element.

   <rule provider='test' id='12' class='violation'>
      <patterns>
        <pattern>simple-message-with-action-to-create-context</pattern>
      </patterns>
      <actions>
        <action trigger='match'>
          <create-context context-id='1000' context-timeout='60' context-scope='program'>
            <message inherit-properties='context'>
              <values>
                <value name='MESSAGE'>context message</value>
              </values>
            </message>
          </create-context>
        </action>
      </actions>
    </rule>

4.3.12 - Element: tags

Location

/patterndb/ruleset/tags

Description

OPTIONAL — An element containing custom keywords (tags) about the messages matching the patterns. The tags can be used to label specific events (for example, user logons). It is also possible to filter on these tags later (for details, see Tagging messages). Starting with AxoSyslog 3.2, the list of tags assigned to a message can be referenced with the ${TAGS} macro.

Attributes

N/A

Children

  • tag: OPTIONAL — A keyword or tags applied to messages matching the rule.

Example

   <tags><tag>UserLogin</tag></tags>

4.3.13 - Element: example

Location

/patterndb/ruleset/rules/rule/patterns/examples/example

Description

OPTIONAL — A container element for a sample log message.

Attributes

N/A

Children

  • test_message: OPTIONAL — A sample log message that should match this pattern. For example:

        <test_message program="myapplication">Content filter has been enabled</test_message>

    • program: The program pattern of the test message. For example:

          <test_message program="proftpd">ubuntu (::ffff:192.168.2.179[::ffff:192.168.2.179]) - FTP session closed.</test_message>

  • test_values: OPTIONAL — A container element to test the results of the parsers used in the pattern.

    • test_value: OPTIONAL — The expected value of the parser when matching the pattern to the test message. For example:

          <test_value name=".dict.ContentFilter" type="string">enabled</test_value>

Example

   <examples>
        <example>
            <test_message>Accepted password for sampleuser from 10.50.0.247 port 42156 ssh2</test_message>
            <test_values>
                <test_value name="SSH_AUTH_METHOD">password</test_value>
                <test_value name="SSH_USERNAME">sampleuser</test_value>
                <test_value name="SSH_CLIENT_ADDRESS">10.50.0.247</test_value>
                <test_value name="SSH_PORT_NUMBER" type="integer">42156</test_value>
            </test_values>
        </example>
    </examples>

4.3.14 - Element: examples

Location

/patterndb/ruleset/rules/rule/patterns/examples

Description

OPTIONAL — A container element for sample log messages that should be recognized by the pattern. These messages can be used also to test the patterns and the parsers.

Attributes

N/A

Children

  • example

Example

   <examples>
        <example>
            <test_message>Accepted password for sampleuser from 10.50.0.247 port 42156 ssh2</test_message>
            <test_values>
                <test_value name="SSH.AUTH_METHOD">password</test_value>
                <test_value name="SSH_USERNAME">sampleuser</test_value>
                <test_value name="SSH_CLIENT_ADDRESS">10.50.0.247</test_value>
                <test_value name="SSH_PORT_NUMBER">42156</test_value>
            </test_values>
        </example>
    </examples>