As log volumes increase and data structures become more intricate, traditional syslog-ng™ filters struggle to keep up. FilterX, a powerful new capability of AxoRouter and AxoSyslog, is designed to make filtering and modifying log messages easier and faster. Whether you’re dealing with deeply nested JSON fields or OpenTelemetry logs, FilterX gives you the flexibility and control needed to classify, curate, and route even the most complex log formats. While used extensively in Axoflow’s flagship AxoRouter security data curation product to address complex enterprise use cases, FilterX is also appropriate for use in more traditional settings. In this blog, we explore simple examples that you can easily incorporate into your syslog-ng configurations used by AxoSyslog (Axoflow’s drop-in replacement for syslog-ng).

What is FilterX?

FilterX (developed by Axoflow) is a replacement for syslog-ng filter statements, parsers, and rewrite rules. It has a syntax and rich set of operators similar to popular scripting languages that allows you to filter, parse, manipulate, and rewrite variables and complex data structures.

FilterX is a consistent and comprehensive reimplementation of several core features with improved performance, proper typing support, and the ability to handle multi-level typed objects, and is currently implemented in AxoRouter and AxoSyslog. FilterX does not deprecate the traditional syslog-ng filter configuration elements of AxoSyslog – if you currently don’t need the enhanced functionality of FilterX, you can continue to use the traditional components.

Why is FilterX needed?

• Syslog-ng is well-known for being able to process syslog messages at a very high throughput. To some extent, this high performance comes from the fact that it’s optimized to handle most data as strings. However, as systems become increasingly complex and produce more and more logs, handling this data in properly typed data structures is also becoming more prevalent. This makes sense since higher quality data makes it far easier (and also faster) to run queries, create specific alerts, and find the data you need.
  
  To address this issue, we’ve added typing support to certain data elements (see this post on my personal blog for details), but that doesn’t cover everything. To properly and efficiently handle complex data structures – for example, the deeply nested fields of OpenTelemetry logs, protocol buffers, and JSON – we needed a new approach.
• Traditional filter/parser/rewrite statements, while useful and powerful, make manipulating complex data quite difficult because these statements are completely independent, and it’s difficult to pass data between them.
• When optimizing data for storage or analysis (for example, for a SIEM for which you pay based on the amount of ingested data) you want to reduce the noise and delete unneeded data (for example, empty JSON fields). Until now, that wasn’t easy to do.
• JSON data is becoming more prevalent, and while syslog-ng can handle JSON, it is cumbersome to work with and legacy syslog-ng implementations perform far more slowly using traditional configuration elements.

FilterX basics

FilterX blocks consist of a list of FilterX statements, the result of every statement is either true or false. If a message matches all FilterX statements, it passes the FilterX block to the next element of the log path, for example, the destination.

For example, consider a log path that:

• selects messages that contain the word deny,
• limits messages to those sent from the host example, and 
• sets (rewrites) the PROGRAM field to example-program.

This is how it looks using traditional filter and rewrite rules:

log {
    source(...);
    filter {
        host("example"); #This is a regexp search
        message("deny");
    };
    rewrite{
        set("example-program" value("PROGRAM"));
    };
    destination(...);
};

And this is the same using FilterX statements:

log {
    source(...);
    filterx {
        ${HOST} == "example";
        ${MESSAGE} =~ "deny";
        ${PROGRAM} = "example-program";
    };
    destination(...);
};

It’s way more readable that way, and we aren’t even using parsers (yet).

What is FilterX good for

FilterX helps you route and process your log messages. If you are familiar with the syslog-ng configuration syntax, the easiest way to understand the benefits of FilterX is via examples. For this post, I’ve collected three:

• Parsing relevant parts of a message and discarding the rest
• Filter Syslog messages that have a JSON body based on a field in its JSON content
• Handle OpenTelemetry logs

For the examples, let’s suppose that the incoming log message has the following JSON-formatted content (received from a Zscaler device):

{
  "LogTimestamp": "Mon Sep 02 14:34:06 2024",
  "ConnectionID": "6N9BHIHZrwrandom,dUPdoZAgr6vJKlv588GG",
  "Exporter": "unset",
  "TimestampRequestReceiveStart": "2020-03-01T22:39:30.679Z",
  "TimestampRequestReceiveHeaderFinish": "2020-03-01T22:39:30.679Z",
  "TimestampRequestReceiveFinish": "2020-03-01T22:39:30.680Z",
  "TimestampRequestTransmitStart": "2020-03-01T22:39:30.680Z",
  "TimestampRequestTransmitFinish": "2020-03-02T02:28:53.277Z",
  "TimestampResponseReceiveStart": "2020-03-01T22:39:30.707Z",
  "TimestampResponseReceiveFinish": "2020-03-02T02:28:53.309Z",
  "TimestampResponseTransmitStart": "2020-03-01T22:39:30.707Z",
  "TimestampResponseTransmitFinish": "2020-03-02T02:28:51.762Z",
  "TotalTimeRequestReceive": 1193,
  "TotalTimeRequestTransmit": 13762597414,
  "TotalTimeResponseReceive": 13762601379,
  "TotalTimeResponseTransmit": 13761054628,
  "TotalTimeConnectionSetup": 1037,
  "TotalTimeServerResponse": -13762570100,
  "Method": "GET",
  "Protocol": "HTTPS",
  "Host": "randomhost.example.com",
  "URL": "/remoteDesktopGateway",
  "UserAgent": "",
  "XFF": "",
  "NameID": "example@randomexample.com",
  "StatusCode": 101,
  "RequestSize": 2246,
  "ResponseSize": 3823185,
  "ApplicationPort": 443,
  "ClientPublicIp": "256.256.256.256",
  "ClientPublicPort": 00000,
  "ClientPrivateIp": "",
  "Customer": "testhost",
  "ConnectionStatus": "zfce_mt_remote_disconnect",
  "ConnectionReason": "BRK_MT_CLOSED_FROM_ASSISTANT"
}

Parsing message parts

Suppose you only need a few fields of that message (ConnectionID, Host, URL, and StatusCode) and want to discard the rest. In this first example, we’ll use regular expressions (ugly and slow – but flexible; we’ll do proper JSON parsing in the next example).

We parse the fields we need using the regexp_search FilterX function, and set the MESSAGE field to the results of this search.

filterx {
	${MESSAGE} = json(regexp_search(${MESSAGE}, '"ConnectionID":"(?[a-z,A-Z,0-9]*)".*"Host":"(?[a-z,A-Z,0-9,.]*)".*"URL":"(?[\/a-z,A-Z,0-9,.]*)".*"StatusCode":(?[0-9]*)'));
};

The content of the MESSAGE field is a JSON with only the extracted fields:

{
  "connectionid": "6N9BHIHZrwrandom,dUPdoZAgr6vJKlv588GG",
  "host": "randomhost.example.com",
  "url": "/remoteDesktopGateway",
  "statuscode": "101"
}

Filter JSON content

The following example parses the Zscaler message as JSON (using a proper JSON parser, instead of the regular expressions of the previous example), and selects messages where the StatusCode field is not 200.

filterx {
  json(${MESSAGE})["StatusCode"] != 200;
};

Note: you can also refer to elements of an object using dot notation, for example:

json(${MESSAGE}).StatusCode != 200;

You can achieve the same result without FilterX using traditional filter and parser blocks – so why is this better? Because in that case the traditional JSON parser would automatically add every parsed field to name-value pairs, which isn’t needed in this use case, and significantly increases the size of the message. FilterX solves the task without any overhead, while making the code far easier to read.

Route and modify OpenTelemetry logs

For the third use case, we’d like to route OpenTelemetry log records based on the name of the application that sent the message. This information is available only in the content of the log record, usually under the service.name key of the resource object. To do this, we will receive the log using the opentelemetry() source of AxoSyslog, and then use FilterX to check the value of the service.name key.

log {
    source { opentelemetry(); };
    filterx {
        # Input mapping that converts the incoming otlp data into a structured variable
        declare resource = otel_resource(${.otel_raw.resource});
        # Filtering
        resource["service.name"] == "name-of-your-application";
    };
    destination {
        # your opentelemetry destination settings
    };
};

If you have other metadata about the message in the resource object (for example, hostname, role of the host, or similar), you can use those for more specific filtering. 

Note that OpenTelemetry is a protobuf based format, which is very difficult to modify using the traditional syslog-ng syntax. FilterX maps the OpenTelemetry data structure into manageable objects, and provides the tools to modify it easily. We’ll give you detailed examples in a follow-up blog post, until then you can read how to handle OpenTelemetry logs with filterx in the AxoSyslog documentation.

Summary

By providing a clean, intuitive syntax and the ability to handle complex, multi-layered data structures, FilterX is a must-have tool for modern syslog-ng users, simplifying everything from JSON parsing to OpenTelemetry log management. FilterX is an exclusive capability included in AxoRouter and AxoSyslog, and is not available in syslog-ng.

While still under development, you can try out a sneak preview of FilterX in AxoSyslog 4.8.1.

Feedback is most welcome on Discord and GitHub. To learn more about FilterX, check out the FilterX chapter of the AxoSyslog documentation.

Legal notice: syslog-ng is a trademark of One Identity LLC.

Why AxoSyslog

AxoSyslog is a fork and drop-in replacement of syslog-ng. Here are some reasons you might want to give AxoSyslog a try:

• Developed and supported by the original creators of syslog-ng.
• Improved performance.
• Better cloud-native support, including Kubernetes integration and ARM support.
• Rapid development, bugfixes, and support by Axoflow.
• It is the foundation of the syslog and other data collection functionality in AxoRouter, our curation solution that collects, aggregates, transforms, and routes all kinds of telemetry and security data automatically – at carrier-grade scale.
• Integrates with the Axoflow Platform that provides metrics, management, and alerts for your observability pipeline.