How to get information about disk-buffer files

• 1: Information about disk-buffer files
• 2: Getting the status information of disk-buffer files
• 3: Getting the list of disk-buffer files
• 4: Printing the content of disk-buffer files
• 5: Orphan disk-buffer files
• 6: How to process messages from an orphan disk-buffer file using a separate syslog-ng OSE instance

1 - Information about disk-buffer files

This section describes information about disk-buffer files used in AxoSyslog.

The following list contains information about how disk-buffer files are used in AxoSyslog:

• You can configure disk-buffer() for a remote destination in the destination() statement.

  For more information about an example of configuring disk-buffer() for a remote destination in the destination() statement, see disk-buffer().

• By default, AxoSyslog creates disk-buffer files under /opt/syslog-ng/var directory, unless dir() option is set in disk-buffer().

• The filenames are generated automatically by AxoSyslog with the extensions .qf for a normal disk-buffer and .rqf for a reliable disk-buffer.

• The disk-buffer file stores processed log messages in the format in which they would have been sent out to the destination, but doesn’t store information about the destination.

2 - Getting the status information of disk-buffer files

Purpose

This section describes getting the status information of the disk-buffer files used in AxoSyslog.

Command syntax

The basic command syntax for getting the status information of the disk-buffer files used in AxoSyslog looks like the following:

   /opt/syslog-ng/bin/dqtool info DISK-BUFFER_FILE

Example commands

The following example commands describe how you can get the status information of two different types of disk-buffer files (namely, empty normal disk-buffer files, and non-empty reliable disk-buffer queue files).

Example commands for empty, normal disk-buffer files, and non-empty, reliable disk-buffer queue files

• Empty, normal disk-buffer file

      /opt/syslog-ng/bin/dqtool info /opt/syslog-ng/var/syslog-ng-00000.qf/var/lib/syslog-ng/syslog-ng-00000.qfDisk-buffer state loaded; filename='/opt/syslog-ng/var/syslog-ng-00000.qf/var/lib/syslog-ng/syslog-ng-00000.qf', number_of_messages='0'
  

• Non-empty, reliable disk-buffer queue file

      /opt/syslog-ng/bin/dqtool info /opt/syslog-ng/var/syslog-ng-00000.rqfReliable disk-buffer state loaded; filename='/opt/syslog-ng/var/syslog-ng-00000.rqf', number_of_messages='10'
  

One-liner command to get the state of disk-buffer files in the default directory

You can use the following one-liner command to get the state of disk-buffer files in the default directory:

   for qfile in /opt/syslog-ng/var/*.?(r)qf ; do /opt/syslog-ng/bin/dqtool info $qfile 2>&1 ; done

3 - Getting the list of disk-buffer files

Purpose

This section describes getting the list of disk-buffer files used in AxoSyslog.

The AxoSyslog application stores information (namely, the IP:PORT or DNS:PORT of the destinations, and the name of the disk-buffer file) about disk-buffer files in its persist file.

Example: command for listing the disk-buffer files in use

The following command will list the disk-buffer files in use:

   /opt/syslog-ng/bin/persist-tool dump /opt/syslog-ng/var/syslog-ng.persist/var/lib/syslog-ng/syslog-ng.persist | awk -F '["=]' '/(qfile\(|\.queue)/ { gsub(/[ \t]+/, "", $5); gsub(/^[0-9A-Fa-f]{8}/, "", $5); "echo "$5"|xxd -r -p"|& getline QUEUE; printf("%s ==> %s\n",$1,QUEUE)}'

The example output will look like the following:

   afsocket_dd_qfile(stream,10.21.10.20:601)  ==> /opt/syslog-ng/var/syslog-ng-00000.rqf

   xxd: command not found

4 - Printing the content of disk-buffer files

Purpose

This section describes printing the content of the disk-buffer files used in AxoSyslog.

Command syntax

The command syntax for printing the content of the disk-buffer files used in AxoSyslog looks like the following:

   /opt/syslog-ng/bin/dqtool cat DISK-BUFFER_FILE

Short example output for printed content

Example: short output that shows the printed content of the disk-buffer files used in AxoSyslog

The following short output example shows the printed content of the disk-buffer files used in AxoSyslog:

   /opt/syslog-ng/bin/dqtool cat /opt/syslog-ng/var/syslog-ng-00000.rqf
    
    Reliable disk-buffer state loaded; filename='/opt/syslog-ng/var/syslog-ng-00000.rqf', queue_length='2952', size='-437712'
    Jul 31 12:33:48.226 10.21.10.10 <382019-07-31T12:33:36 localhost prg00000[1234]: seq: 0000000838, thread: 0000, runid: 1564569216, stamp: 2019-07-31T12:33:36 PADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADD
    ...

5 - Orphan disk-buffer files

Purpose

This section describes orphan disk-buffer files used in AxoSyslog.

Orphan disk-buffer files

In certain situations (for example, after modifying the disk-buffer configuration or losing the persist information), AxoSyslog creates a new disk-buffer file instead of using the already existing one. In these situations, the already existing disk-buffer file becomes a so-called orphan disk-buffer file.

Discovering the new disk-buffer files (orphan disk-buffer files)

To discover orphan disk-buffer files, get the list of disk-buffer files from the persist file, then compare the list with the contents of the disk-buffer files’ saving directory.

For more information about how you can get the list of disk-buffer files from the persist file, see Getting the list of disk-buffer files).

Example: difference between the list of disk-buffer files from the persist file and the content of the disk-buffer files’ saving directory

The following examples show the difference between the list of disk-buffer files from the persist file and the content of the disk-buffer files’ saving directory.

Disk-buffer file list from persist file:

   afsocket_dd_qfile(stream,10.21.10.112:514) = { "queue_file": "/opt/syslog-ng/var/syslog-ng-00001.rqf" }

Disk-buffer files’ saving directory content:

   # ls -l /opt/syslog-ng/var//var/lib/syslog-ng/*qf
    -rw------- 1 root root 2986780 Jul 31 12:30 /opt/syslog-ng/var/syslog-ng-00000.qf/var/lib/syslog-ng/syslog-ng-00000.qf
    -rw------- 1 root root 2000080 Jul 31 12:31 /opt/syslog-ng/var/syslog-ng-00000.rqf
    -rw------- 1 root root    4096 Aug  1 11:09 /opt/syslog-ng/var/syslog-ng-00001.rqf

The disk-buffer files syslog-ng-00000.qf and syslog-ng-00000.rqf don’t exist in the persist file. These two files are the orphan disk-buffer files.

For more information about orphan disk-buffer files and how to process the messages in orphan disk-buffer files using a separate AxoSyslog instance, see How to process messages from an orphan disk-buffer file using a separate syslog-ng OSE instance.

6 - How to process messages from an orphan disk-buffer file using a separate syslog-ng OSE instance

Purpose

This section describes how to read messages from an orphan disk-buffer file by using a separate AxoSyslog process running parallel to the already running AxoSyslog instance.

Orphan disk-buffer files

In certain situations (for example, after modifying the disk-buffer configuration or losing the persist information), AxoSyslog creates a new disk-buffer file instead of using the already existing one. In these situations, the already existing disk-buffer file becomes a so-called orphan disk-buffer file.

Processing the messages from an orphan disk-buffer file by using a separate AxoSyslog instance

When AxoSyslog creates orphan disk-buffer files, you can start a separate AxoSyslog instance parallel to the AxoSyslog instance already running, and use the following resolution process to process the messages in the orphan disk-buffer file.

To process the messages from an orphan disk-buffer file using a separate AxoSyslog instance,

1. Identify the orphan disk-buffer files and make a record of them. For more information, see How to get information about disk-buffer files.

   It is important to know the type of the disk-buffer file. Disk-buffer file types can be normal (.qf) or reliable (.rqf).

   In the examples during this process, the /opt/syslog-ng/var/syslog-ng-00005.rqf orphan reliable disk-buffer file is used.

2. Determine the destination of the logs. The content of the disk-buffer may help you determine the logs’ destination. For more information, see How to get information about disk-buffer files.

   In the examples during this process, the destination 10.21.10.20 is used with the standard network() port 514.

3. Create a directory for the temporary instance. In the examples during this process, the /tmp/qdisk directory is used.

       mkdir /tmp/qdisk
   

   Warning

   Make sure that there is sufficient disk space in the directory. The minimum recommended disk space in the directory is equal to the size of the orphan disk-buffer file.

4. Create the configuration file /tmp/qdisk/qdisk.conf for the temporary instance with the following content.

   Example: creating the /tmp/qdisk/qdisk.conf configuration file for the temporary instance

       @version:7.0
       @include "scl.conf"
   
       options {
         keep-hostname(yes);
         keep-timestamp(yes);
       };
   
       destination d_destination {
       #    ADD YOUR DESTINATION HERE
   
       };
   
       log {
         destination(d_destination);
       };
   

5. Add your destination statement with disk-buffer() to the configuration file. You can copy the destination statement from your running AxoSyslog configuration.

   Warning

   Add the dir() option and set the disk-buffer file’s destination directory to the temporary directory (that is, /tmp/qdisk) in your destination statement.

6. Start the temporary AxoSyslog instance in the foreground.

       syslog-ng -Fe -f /tmp/qdisk/qdisk.conf -R /tmp/qdisk/qdisk.persist -c /tmp/qdisk/qdisk.ctl
   

   The AxoSyslog application will log to the console, so you will see any potential error that may occur during startup.

   The following example output displays that an empty disk-buffer file has been created and the connection to the remote destination has been established.

   Example: output displaying newly created empty disk-buffer file and connection established to remote destination

       Follow-mode file source not found, deferring open; filename='/no_such_file_or.dir'
       Reliable disk-buffer state saved; filename='/tmp/qdisk/syslog-ng-00000.rqf', qdisk_length='0'
       No server license found, running in client mode;
       syslog-ng starting up; version='7.0.20', cfg-fingerprint='eaa03b9efb88b87d7c1b0ce7efd042ed8ac0c013', cfg-nonce-ndx='0', cfg-signature='c0327a7f7e6418ce0399a75089377dfb662bb072'
       FIPS information; FIPS-mode='disabled'
       Syslog connection established; fd='7', server='AF_INET(10.21.10.20:514)', local='AF_INET(0.0.0.0:0)'
   

7. To stop AxoSyslog, press CTRL+C.

8. Overwrite the empty disk-buffer file with the orphan disk-buffer file.

       mv /opt/syslog-ng/var/syslog-ng-00005.rqf /tmp/qdisk/syslog-ng-00000.rqf
   

9. Start AxoSyslog using the command used in Start the temporary AxoSyslog instance in the foreground step.

       syslog-ng -Fe -f /tmp/qdisk/qdisk.conf -R /tmp/qdisk/qdisk.persist -c /tmp/qdisk/qdisk.ctl
   

10. Open another terminal and check the progress by using one of the following methods.

    • Checking the number of stored logs in the disk-buffer (that is, the last number from the output).

          /opt/syslog-ng/sbin/syslog-ng-ctl stats -c /tmp/qdisk/qdisk.ctl | grep 'dst.*queued'
      

    • Checking the status of the disk-buffer file.

          /opt/syslog-ng/bin/dqtool info /tmp/qdisk/syslog-ng-00000.rqf
      

      An empty disk-buffer file will look similar to this:

      Example: empty disk-buffer file status message

      When checking the status of the disk-buffer files, the terminal will display a similar status message for an empty disk-buffer file:

          Reliable disk-buffer state loaded; filename='/tmp/qdisk/syslog-ng-00000.rqf', queue_length='0', size='0'
      

11. Press CTRL+C to stop AxoSyslog.

12. Check the state of the orphan disk-buffer file. For more information, see How to get information about disk-buffer files.

13. If you have more than one orphan disk-buffer file, repeat the steps following the AxoSyslog stop (that is, the steps beginning from overwriting the empty disk-buffer file with the orphan disk-buffer file) for each orphan disk-buffer file.

14. Remove the temporary directory.

    Example: command for removing the temporary directory

    The following command removes the /mp/qdisk temporary directory:

        rm -rf /tmp/qdisk