Why Use OpenTelemetry gRPC for Log Data Transport

Every organization is collecting logs, for observability, for security analysis, for troubleshooting, for compliance, and that can amount to terabytes a day — so efficiently transporting all that data can quickly become a bottleneck.

Traditionally, log data has been transported over the syslog protocol for decades, while HTTP gained popularity in the last ten years or so, mostly for sending data to SIEM and analytics tools. Now, newer technologies like OpenTelemetry and gRPC are redefining how telemetry data moves between systems. In this post, we’ll explore why gRPC outperforms HTTP and gives better reliability than syslog for log data transport, and how that impacts pipelines and components that support multiple protocols, such as Google Cloud Pub/Sub and OpenTelemetry pipelines.

Note that this post is about transporting data as-is. For details on data filtering and reduction to cut data volume and ingestion or storage costs, see our other posts like Top 4 tricks to reduce SIEM data volume, or Classify security data in transit: improve data quality and reduce costs.

HTTP vs. gRPC: A Quick Overview

HTTP has been around since the early 1990s. It’s human-readable, ubiquitous, and easy to debug. Most REST APIs use HTTP/1.1 with JSON for payloads — a simple, flexible setup for web-based communication, but not necessarily designed for high-throughput machine-to machine-communication. Because of its historical presence, tools and libraries are readily available, making it easy to implement the client side in most programming languages. Most SIEMs and log analytics or observability tools have HTTP endpoints for data ingestion.

gRPC, on the other hand, was designed by Google specifically for efficient, high-performance, machine-to-machine communication originally for remote procedure calls, but it also excels on streaming large amounts of data. It builds on top of HTTP/2: its powerful single-connection multiplexing and built-in compression, and uses Protocol Buffers (protobuf) for compact binary serialization. Tooling and libraries are available for most modern programming languages and they provide the performance benefits out-of-the-box. Protocol Buffers enforces a contract-first development for the server and client, where a single source of truth (.proto file) is used to define the API between the two. This requires an additional code generation step, but it ensures that there will be no mistakes in the communication. Modern SIEMs support gRPC for the highest-performance data ingestion.

In essence, HTTP is great for humans; gRPC is built for computers.

Key Advantages of gRPC Over HTTP

Performance and Efficiency

• Compact Payloads: Protocol Buffers compress data into binary form, significantly reducing message size — often by 50–90% compared to JSON.
  
• Lower CPU Overhead: Binary serialization/deserialization is faster than parsing text-based JSON.
  
• Fewer Handshakes: Persistent connections and multiplexed streams mean less time wasted on setting up new requests.

This efficiency translates directly into faster and cheaper data transport, especially for log data that can reach millions of messages per second.

Low Latency

gRPC uses HTTP/2 multiplexing, allowing multiple concurrent messages over a single TCP connection. That eliminates head-of-line blocking — the issue where one slow request delays all others.

With streaming RPCs, gRPC can continuously push or pull data without reopening connections. For log ingestion, this means near-real-time delivery and lower end-to-end latency.

Scalability and Reliability

gRPC’s built-in features — load balancing, timeouts, and flow control — make it naturally suited for distributed environments.

When network conditions fluctuate, gRPC’s flow control mechanisms and retries maintain stability without overwhelming endpoints, ensuring steady log delivery even under bursty conditions.

Feeding data to Google Cloud Pub/Sub

Google Cloud Pub/Sub is a popular destination to collect security and observability data, and supports data ingestion using both REST (HTTP) and gRPC APIs.

gRPC as a Transport for Pub/Sub

The gRPC API in Pub/Sub provides:

• Lower publish latency: your data gets into Pub/Sub faster
  
• Higher throughput per connection: sending the same data via gRPC consumes less bandwidth
• More efficient streaming for both publishers and subscribers

When sending security data — which can be both high-volume and latency-sensitive — these benefits translate into tangible improvements.

Efficiency in Log Data Transmission

With gRPC, clients (for example, your AxoRouters) can:

• Reuse persistent connections for continuous log publishing.
  
• Receive acknowledgments faster, reducing retry overhead.
  
• Send larger batches without connection resets.

This is especially valuable for high-volume data pipelines: Pub/Sub clients using gRPC achieve higher message throughput and lower CPU usage compared to REST clients.
For organizations handling gigabytes of logs per minute, that means less resource consumption, faster ingestion, and more predictable delivery.

gRPC in Telemetry Pipelines: OTLP/gRPC vs. OTLP/HTTP

What Is OTLP?

The OpenTelemetry Protocol (OTLP) is fast becoming the new standard for transmitting telemetry data — logs, metrics, and traces — between exporters, collectors, and backends. It supports multiple transports:

• OTLP/HTTP (using JSON or Protobuf over HTTP)
  
• OTLP/gRPC (using Protobuf over gRPC)

Both achieve the same goal, but differ in performance characteristics.

In production-scale telemetry systems, OTLP/gRPC allows collectors and agents to continuously stream data with minimal overhead. Unfortunately, many vendors that support OTLP support only OTLP/HTTP in their cloud offerings (for example, Clickhouse or Grafana Loki). To use OTLP/gRPC, you have to use their self-hosted versions.

OTLP/gRPC Benefits in Real Deployments

• Lower CPU and memory use for telemetry agents and collectors.
  
• Reduced network congestion via efficient flow control.
  
• Faster pipeline recovery when nodes reconnect after failures.
  
• Improved scalability across distributed systems.

When you switch from OTLP/HTTP to OTLP/gRPC you’re optimizing the entire data plane for sustained, reliable throughput.

For optimal performance, Axoflow collectors transport data using OTLP/gRPC between edge hosts and AxoRouters, and also to destinations if it’s supported.

OTLP/gRPC vs. Traditional Syslog

Before OpenTelemetry and gRPC-based telemetry protocols became mainstream, Syslog was the de facto standard for log forwarding in many infrastructures. It’s simple, mature, and widely supported, and still in widespread use — but it was never without its shortcomings.

Syslog at a Glance

Syslog is a text-based, UDP or TCP-based protocol designed for centralized log aggregation. Applications send messages to a Syslog daemon (for example, rsyslog or AxoSyslog), which then forwards them to collectors or SIEM tools.

While this simplicity made it mainstream for decades, syslog has always had its drawbacks, and struggled to meet the demands of modern, real-time observability pipelines.

Key Advantages of OTLP/gRPC Over Syslog

1. Structured and Typed Data

• Syslog: Unstructured or semi-structured text messages; requires parsing to extract fields.
  
• OTLP/gRPC: Uses Protocol Buffers (Protobuf), ensuring strongly typed, structured log data.

This structure means data arrives ready for processing.

2. Reliable and Efficient Transport

• Syslog (UDP): Fire-and-forget; messages can be lost under network congestion, high load, and other cases. For details, see our post Why syslog over UDP loses messages and how to avoid that.
  
• Syslog (TCP): Reliable but incurs overhead per connection and lacks flow control by default. Certain syslog implementations (Like AxoSyslog) have solutions for flow-control.
  
• OTLP/gRPC: Built on HTTP/2 with flow control, retries, and acknowledgments.

This guarantees delivery under load, even across unreliable networks — critical for large-scale telemetry ingestion.

3. Batching and Backpressure Handling

Syslog transmits messages line by line. There’s no notion of batching (one of the main advantages of feeding SIEMs via their HTTP APIs), backpressure management is non-existent for UDP and is a simple halting of the connection for TCP. OTLP/gRPC, by contrast, supports batching, bidirectional streaming, allowing continuous log transfer while dynamically adjusting to downstream capacity. The server can tell the client when to try to reconnect in case of a resource overload.

The result: fewer dropped logs and smoother pipeline performance during spikes.

4. Data acknowledgment

TCP guarantees that the data has reached the kernel buffer of the server, but it can not guarantee that the application has read it, or if it was processed successfully. This is a huge gap, which can cost you lost data. Ensuring that the data is correctly processed can only happen when the client and the server applications communicate this with each other. This must be implemented for each application for syslog, but it is a built-in feature for gRPC.

5. Rich Metadata and Context

Syslog messages are often limited to the bare minimum: a timestamp, severity, hostname, program name, and message body. (Sidenote: a surprisingly large ratio of vendors fail to comply even with this simple schema and send broken messages.) 

OTLP/gRPC logs can include structured metadata about the sender host and application. This contextual data makes it easier to route and correlate logs with metrics and traces in end-to-end observability workflows.

6. Performance, Load-Balancing and Compression

OTLP/gRPC’s binary encoding and built-in compression drastically reduces the payload size compared to syslog. However, the application level acknowledgement introduces another level of two-way communication, which can slow down the throughput.

Load balancing is trivial with UDP and widely supported with gRPC (because of HTTP/2 load balancers), but it is really hard with TCP. One TCP connection cannot be split into multiple without interpreting the data on-wire. This is possible with software that knows syslog in and out (AxoSyslog), but it is usually not implemented in generic load balancers.

Single connection communication is the only aspect where syslog beats gRPC, but the load-balancing and out-of-the-box parallelized nature of gRPC can overcome this.

Conclusion

Switching from syslog or HTTP to gRPC can dramatically improve the efficiency, reliability, and scalability of log data transport.

Whether you’re publishing to Google Cloud Pub/Sub or managing a logging or security data pipeline, gRPC offers clear advantages:

• Faster serialization with Protobuf
  
• Lower network and CPU overhead
  
• Built-in streaming, load-balancing and flow control
  
• Better type safety and resilience
  
• Better metadata support

As observability systems and security data volumes continue to grow in scale and complexity, choosing the right transport protocol becomes more than a technical detail — it’s a performance multiplier. The Axoflow security data layer out-of-the-box classifies, reduces, normalizes, and enriches security data so you don’t have to. Up to 70% faster investigations. More than 50% reduction in SIEM spend. Request a sandbox to test it yourself!