Specifying data types in value-pairs
Prior to version 4.0, AxoSyslog handled every data as strings, and allowed you to convert the strings into other types of data that only certain destinations data formats supported. For example, SQL, MongoDB, JSON, or AMQP support data types like numbers or dates. The AxoSyslog application allows you to specify the data type in templates (this is also called type-hinting or type-casting). If the destination driver supports data types, AxoSyslog converts the incoming data to the specified data type. For example, this allows you to store integer numbers as numbers in MongoDB, instead of strings.
Starting with AxoSyslog 4.0, each name-value pair is a (name, type, value) triplet, and several components of AxoSyslog have typing support, for example,
json-parser() and the
$(format-json) template function. For details, see the list of supported data types.
Using explicit type-hinting
You can explicitly type-cast a AxoSyslog template to a specific type. To use type-hinting, enclose the macro or template containing the data with the type:
<datatype>("<macro>"), for example:
int("$PID"). See the Type-hinting examples and the list of supported data types for details.
WarningHazard of data loss! If AxoSyslog cannot convert the data into the specified type, an error occurs, and AxoSyslog drops the message by default. To change how AxoSyslog handles data-conversion errors, see Global options.
Not every destination or other component supports data types. For details, see the list of AxoSyslog components that support data types.
The following example stores the MESSAGE, PID, DATE, and PROGRAM fields of a log message in a MongoDB database. The DATE and PID parts are stored as numbers instead of strings.
mongodb( value-pairs(pair("date", datetime("$UNIXTIME")) pair("pid", int64("$PID")) pair("program", "$PROGRAM")) pair("message", "$MESSAGE")) ) );
The following example formats the same fields into JSON.
$(format-json date=datetime($UNIXTIME) pid=int64($PID) program=$PROGRAM message=$MESSAGE)
The following example formats the MESSAGE field as a JSON list.
Data types in AxoSyslog
The AxoSyslog application currently supports the following data-types.
boolean: Converts the data to a boolean value. Anything that begins with a
1is converted to true, anything that begins with an
0is converted to false.
datetime: Use it only with UNIX timestamps, anything else will likely result in an error. This means that currently you can use only the
$UNIXTIMEmacro for this purpose.
double: A floating-point number.
json: A JSON snippet. (Available in AxoSyslog 4.0 and later.)
list: The data as a list. For details, see the list manipulation template functions in Template functions of AxoSyslog.
null: An unset value.
integer: A 32-bit or 64-bit integer, determined by the destination. For example, mongodb uses int32 if the number is less than MAXINT32 and int64 otherwise.
string: The data as a string.
Components that support data types
In AxoSyslog 4.0 and later, the following AxoSyslog components that support data types. Other components treat every data as strings.
Comparisons in filter expressions: the previously numeric operators are type-aware. The exact comparison depends on the types associated with the values you compare. For details, see Comparing macro values in filters.
When using the
json-parser(), AxoSyslog converts all elements of the JSON object to name-value pairs. Any type information originally present in the incoming JSON object is retained, and automatically propagated to other AxoSyslog components (for example, a destination) if they support types.
- Elements without a type are treated as strings.
- JSON lists (arrays) are converted to AxoSyslog lists, so you can manipulate them using the
You can set the type of the field. Where you can use of templates in
groupset(), you can use type-casting, and the type information is properly promoted. For details, see Specifying data types in value-pairs.
db-parser()rules can associate types with values using the
"type"attribute, for example:
<value name="foobar" type="integer">$PID</value>
integeris a type-cast that associates
$foobarwith an integer type.
db-parser()’s internal parsers (for example,
@NUMBER@) automatically associate type information to the parsed name-value pair.
add-contextual-data(): Name-value pairs that are populated using
add-contextual-data()propagate type information, similarly to
map-value-pairs()propagates type information.
SQL type support: The
sql()driver supports types, so that columns with specific types are stored as those types.
Template type support: You can cast templates explicitly to a specific type. Templates also propagate type information from macros, template functions, and values in the template string.
python()typing: All Python components (sources, destinations, parsers, and template functions) support all data types, except
On-disk serialized formats (that is, disk buffer): Version 4.0 and newer are compatible with messages serialized with an earlier version, and the format is compatible for downgrades as well. This means that even if a newer version of AxoSyslog serialized a message, older versions and associated tools are able to read it (but drop the type information of course).