Introduction
This article will provide detailed technical information and network troubleshooting regarding the services in the Leaf Module.
Services
There are two services in the Leaf module: nats-server and nats-subscriber. While Leaf servers only have the nats-server service, the center server has both the nats-server and nats-subscriber services.
nats-server: It enables the connection and communication between Leaf servers and the center server. The following port permissions are required for communication between Leaf servers and the center server:
Leaf -> Center: 4222, 7422, 8222
Center -> Leaf: 4222, 8222
nats-subscriber: It is only available on the center server, and logs coming from Leaf servers first enter this service. Then, the nats-subscriber service directs the logs to the parser service. One nats-subscriber service will be sufficient for every 3000 EPS.
parser: Since logs coming to Leaf servers are processed in the parser service, normally one parser service is required for 200-300 EPS. However, as the logs coming from Leaf servers to the center server undergo parsing, they are directly posted to Elasticsearch via the parser service without being processed in the center server's parser. Therefore, one parser service on the center server will be sufficient for every 1000-1500 EPS coming from Leaf.
Nats Commands
nats s ls
The amount of log sent within the current threshold period is displayed in the messages section and its size in the size section. The last message part shows when the last log from the relevant leaf came to the center.
nats s report
If there is any delay, the amount of these delayed logs can be viewed in the lag section.
nats s info
When the command is applied, general information for the selected leaf is displayed.
In addition, by tunneling the 8222 port, nats interface can be accessed with any browser and all details about the leaf structure can be examined.
All the information in the interface can be accessed via the cli command with the curl command.
Leafz Section
In the leafz section, the details of all registered Leaf servers on the center can be viewed. The "rtt" (round trip time) in this section provides clear information about the network health between the Leaf and the center. If this value is high, there may be delays in the network, indicating that the logs will reach the center with a delay.
Network Monitoring
One of the most important aspects of the Leaf module is having a healthy network and sufficient bandwidth. Delays on the network or insufficient bandwidth for the current EPS (events per second) can cause the system to completely stop.
To perform network monitoring, the "iperf" and "iperf3" packages should be installed using the following commands.
apt-get update && apt-get install iperf && apt-get install iperf3 -y
With the command below, the bandwidth can be measured by sending a request to the center via the leaf server from port 7422.
iperf -c 10.10.2.85 -p 7422
The command can be repeated multiple times to obtain real-time bandwidth information.
The required minimum bandwidth values based on the EPS can be checked in the Logsign Leaf System Requirements article.
When there is insufficient bandwidth, delays may occur in the log flow. These delays can be checked by monitoring the logs of the nats-server service on the Leaf server. In the screenshot below, a Slow Consumer error is received, indicating that there is a delay.
On the Center, the nats s report command can be used to find out whether there is a delay in incoming logs and the amount of delayed logs.
In cases where the bandwidth is severely insufficient, even if there appears to be a connection between the Leaf and the center, the nats-subscriber service may interrupt the log flow by giving a timeout error because it cannot subscribe to the relevant Leaf server. The error related to this will be reflected in the service logs shortly after restarting the nats-subscriber service.