Why Policy-Based Routing Beats Static Rules

Data Pipeline Management, also called Telemetry Pipelines, has become an industry standard for handling security data. The reasons are clear: the aging, hard to maintain systems with immense volume of data are recipes for disaster. In a time when you struggle with both missing data from the system and alert fatigue it is hard to prioritize. Axoflow as a Security Data Pipeline solves these problems at once by improving the security data quality automatically, providing volume reduction and high-quality, actionable data. This space is thriving with different companies with similar promises, so let’s see how Axoflow is different? How does declarative Policy-Based Routing beat static rules all the time?

Declarative management

Managing large infrastructures is not a new problem. Modern Infrastructure as a Code and other automation tools, and even Kubernetes were born to solve such challenges. While generalist tools are highly customizable, you have to spend considerable effort to understand them and build your own environment. But there is an important lesson learned from those systems: Most popular tools use declarative language to describe configuration.

The key difference between the imperative and declarative methods is that:

• when using imperative methods, you're dictating instructions, while when using
• declarative methods, the system knows enough to figure out instructions on its own.

As security functions are in almost all cases orthogonal to the company's traditional business, they use their own solutions and tools to provide the safety net for all other business units. In a situation like this, a reliable, easy to manage specialized tool can significantly reduce the effort required.

Labels

Why are labels a crucial part of policy based routing? In a declarative environment you are managing the state of the system. But how do you define the state of streaming data? That’s the point where labels play a huge role. Each entity in the Axoflow Platform has labels. The log sources (hosts), the routers, and even the data streams. This information declares a state like: Palo Alto logs are received from three different Hosts on Axorouter-1. A stream of logs then will have those identifiers as labels attached them:

As labels are such a crucial part of the system, we’ll dedicate a separate blog post for them. Until then, watch our webinar called "Show me your labels and I'll tell you who you are" on that topic.

Flows

Labels define the data that the platform receives. So how do we use them in our policies (or Flows as we call them at Axoflow)? A Flow specifies what data should be sent to which destination, and what transformations should be applied. Let’s see how an example Flow looks like if we want to send Palo Alto logs to Splunk?

As Axoflow does automatic parsing and labeling, the Flow can refer to the product label to select Palo Alto logs in Step 1. The next step of the Flow is reduction. Reduction reduces the noise in the data by removing redundant and unneeded information, and it also reduces the size of the data at the same time, reducing ingestion and storage costs as a side effect. (We explained how data reduction works in detail in the Classify security data in transit: improve data quality and reduce costs blog post.)

The Flow points to a Splunk destination and Axoflow - behind the curtains - automatically converts the data to the format optimized for Splunk, and fills all the necessary fields like sourcetype and index. The following table explains different sourcetypes and indexes Axoflow applies for Palo Alto logs:

‍

That’s all! This is a production ready configuration to handle all your Palo Alto logs. And of course this is a trivial example and you can fine tune or generalize your flows as much as you want:

• Do you want specific devices’ log to different places? Use the device names or other identifiers! 
• Do you want to drop a field? Use the unset processing step!

We will provide more examples and tricks in the following blog post.

Flows vs. rule-based solutions

Just for fun let’s compare to traditional rule based solutions. See what kind of tasks you should do by yourself?

• The industry “best practice” before Axoflow was to dedicate ports to different types of logs. So first you would dedicate separate ports to different logs, and configure the tool to listen on them. (i.e: UDP 514)
• Now you have a dedicated port you can apply parsing on the data stream. This is either provided by some community marketplace, or you write your parsing instructions yourself. Palo Alto has a dynamic format based on the subtype of the log, so you end up with 5-10 conditional parsing.
• Make sure that your tool validates the incoming data and provides error handling, so if accidentally something else sends data to this port it does not get dropped or misclassified.
• Fix up the content. This could mean a lot of different things, starting from adjusting timezones to even repairing malformatted content.
• Enrich with contextual information. Server location, responsible team, or anything else that would be useful during incident response, but you didn’t have the time nor the tools to manage those.
• Format the message for the destination and fill in the metadata to improve threat detection accuracy.

And remember, this was only one type of logs to one specific destination. You have to configure all of these combinations which will exponentially grow your configuration that you have to maintain. Those benefits are amazing time savers during the first deployment, but PBR (Policy-Based Routing) can dramatically reduce your day to day operations efforts as well.

Day 2 operations

These benefits apply to you if you have a considerable amount of entities you are collecting data from. Our customers have infrastructures ranging from a couple of thousands to hundreds of thousands hosts. Infrastructures are changing, especially large ones with thousands or hundreds of thousands of machines. Axoflow makes it easy (or using our classification database, even effortless) to:

• Onboard / Offboard new appliances or 
• Handle appliance upgrades and log format changes (as illustrated in the figure below)
• Detect malfunctions in the pipeline
• Be aware and never loose unidentified logs
• Create policies that last for years not months

Here is what happens in the pipeline during a traditional upgrade:

‍

And here is the seamless upgrade flow provided by Axoflow:

Conclusion

Policy-Based Routing might sound like “magic” at first but it is actually a strictly deterministic system. It is easy to start with and easy to learn the fine bits as well. In the following articles I will describe the different usage of labels: How to use dynamic routing based on the content of the data and a lot more! So stay tuned to become an expert in Security Data management without hassle.