About disk queue files
Normal and reliable queue files
The key difference between disk queue files that employ the
reliable(yes) option and not is the strategy they employ. Reliable disk queues guarantee that all the messages passing through them are written to disk first, and removed from the queue only after the destination has confirmed that the message has been successfully received. This prevents message loss, for example, due to AxoSyslog crashes. Of course, using the
reliable(yes) option introduces a significant performance penalty as well.
Both reliable and normal disk-buffers employ an in-memory output queue (set in
quot-size()) and an in-memory overflow queue (set in
flow-control-window-bytes() for reliable disk-buffers, or
flow-control-window-size() for normal disk-buffers). The difference between reliable and normal disk-buffers is that when the reliable disk-buffer uses one of its in-memory queues, it also stores the message on the disk, whereas the normal disk-buffer stores the message only in memory. The normal disk-buffer only uses the disk if the in-memory output buffer is filled up completely. This approach has better performance (due to fewer disk I/O operations), but also carries the risk of losing a maximum of
flow-control-window-size() number of messages in case of an unexpected power failure or application crash.
Size of the queue files
Disk queue files grow. Each may take up to
capacity-bytes() bytes on the disk. Due to the nature of reliable queue files, all the messages traversing the queue are written to disk, constantly increasing the size of the queue file.
The disk-buffer file’s size should be considered as the configured
capacity-bytes() at any point of time, even if it does not have messages in it. Truncating the disk-buffer file can slow down disk I/O operations, so AxoSyslog does not always truncate the file when it would be possible (see the
truncate-size-ratio() option). If a large disk-buffer file is not desirable, you should set the
capacity-bytes() option to a smaller value. Note that AxoSyslog version 4.0 and later doesn’t truncate disk-buffer files by default.
Starting with AxoSyslog version 4.0, you can preallocate disk-buffer files.
NoteThe disk-buffer file’s size does not strictly correlate to the number of stored messages. If you want to get information about the disk-buffer, use
dqtool(for more information, see Getting the status information of disk-buffer files).
NoteIf a queue file becomes corrupt, AxoSyslog starts a new one. This might lead to the queue files consuming more space in total than their maximal configured size and the number of configured queue files multiplied together.
Preallocating disk-buffer files
By default, AxoSyslog doesn’t reserve the disk space for the disk-buffer file, since in a properly configured and sized environment the disk-buffer is practically empty, so a large preallocated disk-buffer file is just a waste of disk space. But a preallocated buffer can prevent other data from using the intended buffer space (and elicit a warning from the OS if disk space is low), preventing message loss if the buffer is actually needed. To avoid this problem, when using AxoSyslog 4.0 or later, you can preallocate the space for your disk-buffer files by setting
In addition to making sure that the required disk space is available when needed, preallocated disk-buffer files provide radically better (3-4x) performance as well: in case of an outage the amount of messages stored in the disk-buffer is continuously growing, and using large continuous files is faster, than constantly waiting on a file to change its size.
If you are running AxoSyslog on a dedicated host (always recommended for any high-volume settings), use