This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

The pattern database format

Pattern databases are XML files that contain rules describing the message patterns. For sample pattern databases, see Downloading sample pattern databases.

The following scheme describes the V5 format of the pattern database. This format is backwards-compatible with the earlier formats.

For a sample database containing only a single pattern, see Example: A pattern database containing a single rule.

Example: A pattern database containing a single rule

The following pattern database contains a single rule that matches a log message of the ssh application. A sample log message looks like:

   Accepted password for sampleuser from 10.50.0.247 port 42156 ssh2

The following is a simple pattern database containing a matching rule.

   <patterndb version='5' pub_date='2010-10-17'>
        <ruleset name='ssh' id='123456678'>
            <pattern>ssh</pattern>
                <rules>
                    <rule provider='me' id='182437592347598' class='system'>
                        <patterns>
                            <pattern>Accepted @QSTRING:SSH.AUTH_METHOD: @ for@QSTRING:SSH_USERNAME: @from\ @QSTRING:SSH_CLIENT_ADDRESS: @port @NUMBER:SSH_PORT_NUMBER:@ ssh2</pattern>
                        </patterns>
                    </rule>
                </rules>
        </ruleset>
    </patterndb>

Note that the rule uses macros that refer to parts of the message, for example, you can use the ${SSH_USERNAME} macro refer to the username used in the connection.

The following is the same example, but with a test message and test values for the parsers.

   <patterndb version='4' pub_date='2010-10-17'>
        <ruleset name='ssh' id='123456678'>
            <pattern>ssh</pattern>
                <rules>
                    <rule provider='me' id='182437592347598' class='system'>
                        <patterns>
                            <pattern>Accepted @QSTRING:SSH.AUTH_METHOD: @ for@QSTRING:SSH_USERNAME: @from\ @QSTRING:SSH_CLIENT_ADDRESS: @port @NUMBER:SSH_PORT_NUMBER:@ ssh2</pattern>
                        </patterns>
                        <examples>
                            <example>
                                <test_message>Accepted password for sampleuser from 10.50.0.247 port 42156 ssh2</test_message>
                                <test_values>
                                    <test_value name="SSH.AUTH_METHOD">password</test_value>
                                    <test_value name="SSH_USERNAME">sampleuser</test_value>
                                    <test_value name="SSH_CLIENT_ADDRESS">10.50.0.247</test_value>
                                    <test_value name="SSH_PORT_NUMBER">42156</test_value>
                                </test_values>
                           </example>
                        </examples>
                    </rule>
                </rules>
        </ruleset>
    </patterndb>

1 - Element: patterndb

Location

/patterndb

Description

The container element of the pattern database.

Attributes

  • version: The schema version of the pattern database. The current version is 4.

  • pubdate: The publication date of the XML file.

Children

  • ruleset

Example

   <patterndb version='4' pub_date='2010-10-25'>

2 - Element: ruleset

Location

/patterndb/ruleset

Description

A container element to group log patterns for an application or program. A <patterndb> element may contain any number of <ruleset> elements.

Attributes

  • name: The name of the application. Note that the function of this attribute is to make the database more readable, syslog-ng uses the <pattern> element to identify the applications sending log messages.

  • id: A unique ID of the application, for example, the md5 sum of the name attribute.

Children

  • patterns

  • rules

  • actions

  • tags

  • description: OPTIONAL — A description of the ruleset or the application.

  • url: OPTIONAL — An URL referring to further information about the ruleset or the application.

  • rule_urls: OPTIONAL — To list multiple URLs referring to further information about the ruleset or the application, enclose the <url> elements into an <urls> element.

Example

   <ruleset name='su' id='480de478-d4a6-4a7f-bea4-0c0245d361e1'>

3 - Element: patterns

Location

/patterndb/ruleset/patterns

Description

A container element. A <patterns> element may contain any number of `elements.

Attributes

N/A

Children

  • pattern: The name of the application — syslog-ng matches this value to the ${PROGRAM} header of the syslog message to find the rulesets applicable to the syslog message.

    Specifying multiple patterns is useful if two or more applications have different names (that is, different ${PROGRAM} fields), but otherwise send identical log messages.

    It is not necessary to use multiple patterns if only the end of the ${PROGRAM} fields is different, use only the beginning of the ${PROGRAM} field as the pattern. For example, the Postfix email server sends messages using different process names, but all of them begin with the postfix string.

    You can also use parsers in the program pattern if needed, and use the parsed results later. For example: `stfix\@ESTRING:.postfix.component:[@»

Example

   <patterns>
        <pattern>firstapplication</pattern>
        <pattern>otherapplication</pattern>
    </patterns>

Using parsers in the program pattern:

   <pattern>postfix\@ESTRING:.postfix.component:[@</pattern>

4 - Element: rules

Location

/patterndb/ruleset/rules

Description

A container element for the rules of the ruleset.

Attributes

N/A

Children

  • rule

Example

   <rules>
        <rule provider='me' id='182437592347598' class='system'>
            <patterns>
                <pattern>Accepted @QSTRING:SSH.AUTH_METHOD: @ for@QSTRING:SSH_USERNAME: @from\ @QSTRING:SSH_CLIENT_ADDRESS: @port @NUMBER:SSH_PORT_NUMBER:@ ssh2</pattern>
            </patterns>
        </rule>
    </rules>

5 - Element: rule

Location

/patterndb/ruleset/rules/rule

Description

An element containing message patterns and how a message that matches these patterns is classified.

The element may contain any number of elements.

Attributes

  • provider: The provider of the rule. This is used to distinguish between who supplied the rule, that is, if it has been created by Axoflow, or added to the XML by a local user.

  • id: The globally unique ID of the rule.

  • class: The class of the rule — this class is assigned to the messages matching a pattern of this rule.

Children

  • patterns

Example

   <rule provider='example' id='f57196aa-75fd-11dd-9bba-001e6806451b' class='violation'>

The following example specifies attributes for correlating messages as well. For details on correlating messages, see Correlating log messages using pattern databases.

   <rule provider='example' id='f57196aa-75fd-11dd-9bba-001e6806451b' class='violation' context-id='same-session' context-scope='process' context-timeout='360'>

6 - Element: patterns

Location

/patterndb/ruleset/rules/rule/patterns

Description

An element containing the patterns of the rule. If a element contains multiple *lements, the class of the *»>ssigned to every syslog message matching any of the patterns.

Attributes

N/A

Children

  • pattern: A pattern describing a log message. This element is also called message pattern. For example:

        <pattern>+ ??? root-</pattern>
    
  • description: OPTIONAL — A description of the pattern or the log message matching the pattern.

  • urls

  • values

  • examples

Example

   <patterns>
        <pattern>Accepted @QSTRING:SSH.AUTH_METHOD: @ for@QSTRING:SSH_USERNAME: @from\ @QSTRING:SSH_CLIENT_ADDRESS: @port @NUMBER:SSH_PORT_NUMBER:@ ssh2</pattern>
    </patterns>

7 - Element: urls

Location

/patterndb/ruleset/rules/rule/patterns/urls

Description

OPTIONAL — An element containing one or more URLs referring to further information about the patterns or the matching log messages.

Attributes

N/A

Children

  • url: OPTIONAL — An URL referring to further information about the patterns or the matching log messages.

Example

N/A

8 - Element: values

Location

/patterndb/ruleset/rules/rule/patterns/values

Description

OPTIONAL — Name-value pairs that are assigned to messages matching the patterns, for example, the representation of the event in the message according to the Common Event Format (CEF) or Common Event Exchange (CEE). The names can be used as macros to reference the assigned values.

Attributes

N/A

Children

  • value: OPTIONAL — Contains the value of the name-value pair that is assigned to the message.

    The <value> element of name-value pairs can include template functions. For details, see Using template functions, for examples, see if.

    You can associate types with values using the "type" attribute, for example, integer is a type-cast that associates $foobar with an integer type. For details on data types, see Specifying data types in value-pairs.

    <value name="foobar" type="integer">$PID</value>
    

    db-parser()’s internal parsers (for example, @NUMBER@) automatically associate type information to the parsed name-value pair.

    When used together with message correlation, the <value> element of name-value pairs can include references to the values of earlier messages from the same context. For details, see Correlating log messages using pattern databases.

  • name: The name of the name-value pair. It can also be used as a macro to reference the assigned value.

Example

   <values>
        <value name=".classifier.outcome">/Success</value>
    </values>

9 - Element: actions

Location

/patterndb/ruleset/actions

Description

OPTIONAL — A container element for actions that are performed if a message is recognized by the pattern. For details on actions, see Triggering actions for identified messages.

Attributes

N/A

Children

  • action

Example

Example: Generating messages for pattern database matches

When inserted in a pattern database rule, the following example generates a message when a message matching the rule is received.

   <actions>
        <action>
            <message>
                <values>
                    <value name="MESSAGE">A log message from ${HOST} matched rule number $.classifier.rule_id</value>
                </values>
            </message>
        </action>
    </actions>

To inherit the properties and values of the triggering message, set the inherit-properties attribute of the <message> element to TRUE. That way the triggering log message is cloned, including name-value pairs and tags. If you set any values for the message in the <action> element, they will override the values of the original message.

Example: Generating messages with inherited values

The following action generates a message that is identical to the original message, but its $PROGRAM field is set to overriding-original-program-name

   <actions>
        <action>
            <message inherit-properties='TRUE'>
                <values>
                    <value name="PROGRAM">overriding-original-program-name</value>
                </values>
            </message>
        </action>
    </actions>

10 - Element: action

Location

/patterndb/ruleset/actions/action

Description

OPTIONAL — A container element describing an action that is performed when a message matching the rule is received.

Attributes

  • condition: An AxoSyslog filter expression. The action is performed only if the message matches the filter. The filter can include macros and name-value pairs extracted from the message. When using actions together with message-correlation, you can also use the $(context-length) macro, which returns the number of messages in the current context. For example, this can be used to determine if the expected number of messages has arrived to the context: condition='"$(context-length)" > "5"'

  • rate: Specifies maximum how many messages should be generated in the specified time period in the following format: <number-of-messages>/<period-in-seconds>. For example: 1/60 allows 1 message per minute. Rates apply within the scope of the context, that is, if context-scope="host" and rate="1/60", then maximum one message is generated per minute for every host that sends a log message matching the rule. Excess messages are dropped. Note that when applying the rate to the generated messages, AxoSyslog uses the timestamps of the log messages, similarly to calculating the context-timeout. That way rate is applied correctly even if the log messages are processed offline.

  • trigger: Specifies when the action is executed. The trigger attribute has the following possible values:

    • match: Execute the action immediately when a message matching the rule is received.

    • timeout: Execute the action when the correlation timer (context-timeout) of the pattern database rule expires. This is available only if actions are used together with correlating messages.

Children

  • create-context

  • message: A container element storing the message to be sent when the action is executed. Currently AxoSyslog sends these messages to the internal() destination.

    • For details on the message context, see Correlating log messages using pattern databases and Actions and message correlation. For details on triggering messages, see Triggering actions for identified messages

      inherit-mode: This attribute controls which name-value pairs and tags are propagated to the newly generated message.

      • context: AxoSyslog collects every name-value pair from each message stored in the context, and includes them in the generated message. If a name-value pair appears in multiple messages of the context, the value in the latest message will be used. Note that tags are not merged, the generated message will inherit the tags assigned to the last message of the context.

      • last-message: Only the name-value pairs appearing in the last message are copied. If the context contains only a single message, then it is the message that triggered the action.

      • none: An empty message is created, without inheriting any tags or name-value pairs.

      This option is available in AxoSyslog 3.8 and later.

    • inherit-properties: This attribute is deprecated. Use the inherit-mode attribute instead.

      If set to TRUE, the original message that triggered the action is cloned, including its name-value pairs and tags.

      If set to context, AxoSyslog collects every name-value pair from each message stored in the context, and includes them in the generated message. If a name-value pair appears in multiple messages of the context, the value in the latest message will be used. Note that tags are not merged, the generated message will inherit the tags assigned to the last message of the context.

      For details on the message context, see Correlating log messages using pattern databases and Actions and message correlation. For details on triggering messages, see Triggering actions for identified messages

      This option is available in AxoSyslog 5.3.2 and later.

  • values: A container element for values and fields that are used to create the message generated by the action.

    • value: Sets the value of the message field specified in the name attribute of the element. For example, to specify the body of the generated message, use the following syntax:

      
          <value name="MESSAGE">A log message matched rule number $.classifier.rule_id</value>
      

      Note that currently it is not possible to add DATE, FACILITY, or SEVERITY fields to the message.

      When the action is used together with message correlation, the AxoSyslog application automatically adds fields to the message based on the context-scope parameter. For example, using context-scope="process" automatically fills the HOST, PROGRAM, and PID fields of the generated message.

    • name: Name of the message field set by the value element.

Example: Generating messages for pattern database matches

When inserted in a pattern database rule, the following example generates a message when a message matching the rule is received.

   <actions>
        <action>
            <message>
                <values>
                    <value name="MESSAGE">A log message from ${HOST} matched rule number $.classifier.rule_id</value>
                </values>
            </message>
        </action>
    </actions>

To inherit the properties and values of the triggering message, set the inherit-properties attribute of the <message> element to TRUE. That way the triggering log message is cloned, including name-value pairs and tags. If you set any values for the message in the <action> element, they will override the values of the original message.

Example: Generating messages with inherited values

The following action generates a message that is identical to the original message, but its $PROGRAM field is set to overriding-original-program-name

   <actions>
        <action>
            <message inherit-properties='TRUE'>
                <values>
                    <value name="PROGRAM">overriding-original-program-name</value>
                </values>
            </message>
        </action>
    </actions>

11 - Element: create-context

Location

/patterndb/ruleset/actions/action/create-context

Description

OPTIONAL — Creates a new correlation context from the current message and its associated context. This can be used to “split” a context.

Available in AxoSyslog version 3.8 and later.

Attributes

Children

  • message: A container element storing the message that is added to the new context when the action is executed.

    • inherit-mode: This attribute controls which name-value pairs and tags are propagated to the newly generated message.

      • context: AxoSyslog collects every name-value pair from each message stored in the context, and includes them in the generated message. If a name-value pair appears in multiple messages of the context, the value in the latest message will be used. Note that tags are not merged, the generated message will inherit the tags assigned to the last message of the context.

      • last-message: Only the name-value pairs appearing in the last message are copied. If the context contains only a single message, then it is the message that triggered the action.

      • none: An empty message is created, without inheriting any tags or name-value pairs.

      For details on the message context, see Correlating log messages using pattern databases and Actions and message correlation. For details on triggering messages, see Triggering actions for identified messages

Example

The following example creates a new context whenever the rule matches. The new context receives 1000 as ID, and program as scope, and the content set in the <message> element of the >element.

   <rule provider='test' id='12' class='violation'>
      <patterns>
        <pattern>simple-message-with-action-to-create-context</pattern>
      </patterns>
      <actions>
        <action trigger='match'>
          <create-context context-id='1000' context-timeout='60' context-scope='program'>
            <message inherit-properties='context'>
              <values>
                <value name='MESSAGE'>context message</value>
              </values>
            </message>
          </create-context>
        </action>
      </actions>
    </rule>

12 - Element: tags

Location

/patterndb/ruleset/tags

Description

OPTIONAL — An element containing custom keywords (tags) about the messages matching the patterns. The tags can be used to label specific events (for example, user logons). It is also possible to filter on these tags later (for details, see Tagging messages). Starting with AxoSyslog 3.2, the list of tags assigned to a message can be referenced with the ${TAGS} macro.

Attributes

N/A

Children

  • tag: OPTIONAL — A keyword or tags applied to messages matching the rule.

Example

   <tags><tag>UserLogin</tag></tags>

13 - Element: example

Location

/patterndb/ruleset/rules/rule/patterns/examples/example

Description

OPTIONAL — A container element for a sample log message.

Attributes

N/A

Children

  • test_message: OPTIONAL — A sample log message that should match this pattern. For example:

        <test_message program="myapplication">Content filter has been enabled</test_message>
    
    • program: The program pattern of the test message. For example:

          <test_message program="proftpd">ubuntu (::ffff:192.168.2.179[::ffff:192.168.2.179]) - FTP session closed.</test_message>
      
  • test_values: OPTIONAL — A container element to test the results of the parsers used in the pattern.

    • test_value: OPTIONAL — The expected value of the parser when matching the pattern to the test message. For example:

          <test_value name=".dict.ContentFilter" type="string">enabled</test_value>
      

Example

   <examples>
        <example>
            <test_message>Accepted password for sampleuser from 10.50.0.247 port 42156 ssh2</test_message>
            <test_values>
                <test_value name="SSH_AUTH_METHOD">password</test_value>
                <test_value name="SSH_USERNAME">sampleuser</test_value>
                <test_value name="SSH_CLIENT_ADDRESS">10.50.0.247</test_value>
                <test_value name="SSH_PORT_NUMBER" type="integer">42156</test_value>
            </test_values>
        </example>
    </examples>

14 - Element: examples

Location

/patterndb/ruleset/rules/rule/patterns/examples

Description

OPTIONAL — A container element for sample log messages that should be recognized by the pattern. These messages can be used also to test the patterns and the parsers.

Attributes

N/A

Children

  • example

Example

   <examples>
        <example>
            <test_message>Accepted password for sampleuser from 10.50.0.247 port 42156 ssh2</test_message>
            <test_values>
                <test_value name="SSH.AUTH_METHOD">password</test_value>
                <test_value name="SSH_USERNAME">sampleuser</test_value>
                <test_value name="SSH_CLIENT_ADDRESS">10.50.0.247</test_value>
                <test_value name="SSH_PORT_NUMBER">42156</test_value>
            </test_values>
        </example>
    </examples>