Introduction
This article will provide information about detecting problems related to alarms and proposed solutions.
Alarmflow Worker
To detect and resolve issues, we need to know the working architecture of our service.
When logs enter Logsign, the parser service performs the parsing process. The parsed logs are then passed to the logsign-rule-flow service, which enriches them and checks them against the alert rule conditions. When a log matches an alert rule, rule-flow publishes the match to the alarmflow queue on NATS. The logsign-alarmflow-worker service consumes this queue, applies the white list and silence checks, and triggers the alarm.
After an alert triggers, the worker publishes its document to the indexer for display in the user interface. Configured actions are queued in Redis. Standard actions use a list; managed security actions use a hash.
Let's explain the topic with an example.
Suppose you are waiting for a log to trigger your alarm in the Logsign user interface, but you have observed that the alarm has not been triggered despite the log being created. There may be more than one reason for this situation.
Open Settings > Alerts & Incidents > Alert Rules and edit the rule. Action Column is required. Check whether the selected field exists in the event. The worker compares its value with the rule's White List. A listed value suppresses the alert. A missing field does not by itself stop the alert document from being created. When the Incident module is enabled, an empty value prevents incident creation.
Our second control point is Silence Time, which prevents the same alarm from triggering again in case of repetition. Once an alarm triggers, the same alarm with the same Silence Columns values does not trigger again until the Silence Time has passed. Make sure that the untriggered alarm is not still within its Silence Time.
Check parser processing on Settings > Maintenance > Event Stats. Review the Parser chart for received, processed, failed, and unknown events.
On the same page, check the Drop chart for the parser_dropped series. If it rises, investigate parser processing and capacity before changing the service count.
Our last control point should be the Logsign-alarmflow-worker service. After connecting to the Logsign server's terminal screen, we should check the service with the following command:
journalctl -u logsign-alarmflow-worker -f
If there are any expressions like errors in the Logsign alarmflow-worker service, they should be examined. Since the rule matching itself is done by the rule-flow service, you can check its logs the same way with the following command:
journalctl -u logsign-rule-flow -f