Introduction
This article will provide information about the configurations found in the Data Management panel and the index structure used in the cluster architecture.
Data Retention
Depending on the disk capacity and hardware adequacy of your Logsign structure, you can keep your index, archive, and signed logs for certain periods.
Delete Live Reports Before: The duration of the index to be stored with Elasticsearch.
Delete Incident Before: The duration of storing cases opened in the incident management panel.
Delete Archived Data Before: The duration of your logs to be stored as an archive.
Delete Signed Data Before: The duration of your logs to be stored as signed.
Compression Format: The compression format of your archive and signed logs.
Compression Level: The compression ratio, which increases disk gain but also increases decompression time.
Rotate Period: After storing the data for a certain period for Logsign archive and signed, the compression process is initiated. You can update the period with this configuration.
Set Raw Message: Sometimes, you may want to see the original (raw) version of your logs received with Logsign. For this process, raw message opening is required. As a result of this process, the disk space the logs occupy will double, and hardware usage for indexing will also be affected. We recommend keeping this feature off except when necessary.
Policies
In the option below, when your system disk usage reaches 90%, the logging will stop to prevent critical problems in the system. You can track disk usage rates from the indicator panel and also activate the disk usage warning notification.
AllinOne Logsign Index Policy
The configurations under the Index Policy are related to the structure of the index logs stored with Elasticsearch.
Logsign Events Index Shard Count : 1 Logsign Events Index Replica Count : 0 |
It is the index stored for health control services on Logsign Events. 0 is defined in order not to keep a backup as a replica. |
Positions Index Shard Count :1 Positions Index Replica Count :0 |
Position Index: It finds the positions of pollers such as SMB, SFTP, MSSQL and is used to index them again. |
Status Index Shard Count :1 Status Index Replica Count :0 |
It is the area where the health control status of resources is determined. |
Sample Index Shard Count :1 Sample Index Replica Count :0 |
This is the area where the policies required for the Log Capture Tool are determined. |
Incidents Index Shard Count :1 Incidents Index Replica Count :0 |
It is the index area where the cases in the event management panel are stored. |
Index Shard Count :1 Index Replica Count :0 |
It is the index area where logs are stored on a daily basis. |
Index Merge Factor :1000 Index Refresh Interval :60 |
Elasticsearch writes many logs to the index instantly; However, the hardware does not send instantaneous logs to the user interface for effective use, it is expected that 1000 logs are collected or 60 seconds are expected to display the logs in the user interface. |
The above visual shows the configurations of Logsign, which is a cluster-based structure. If you are using a cluster-based structure, your nodes that make up the cluster are distributed across different shards on each node. This distribution of shards on nodes aims to provide faster response times for Logsign queries and to balance the Elasticsearch performance, in a way.
Replica shards are provided to you for redundancy, which is one of the basic purposes of the cluster structure. Replica shards take a clone of the existing shard, and if any node becomes unreachable, the replica shard performs the operation instead of the lost shard.
Let us explain this situation with an example. In the image above, there are shards between p0-p5 and r0-r5. P0-p5 is the shard where queries with primary shards are run. R0-r5 is replica shard, in case of loss between p0-p5, r0-r5 clone shards will be used .