Introduction
This article will provide information about the configurations found in the Data Management panel and the index structure used in the cluster architecture. The settings described below are under Settings > Data Management.
Data Retention
Depending on the disk capacity and hardware adequacy of your Logsign structure, you can keep your index, archive, and signed logs for certain periods.
Archive and signed log retention is configured on the Settings > Data Management > Data Retention page. The Persist Settings section of this page contains the following fields:
Compression Format: The compression format of your archive and signed logs. Options: None, Bzip, Gzip (default: Gzip).
Compression Level: The compression ratio, which increases disk gain but also increases decompression time. Values 1 to 9 can be selected (default: 9).
Delete Archived Data Before: The duration of your logs to be stored as an archive (default: 365).
Delete Signed Data Before: The duration of your logs to be stored as signed (default: 730).
Rotate Period: After storing the data for a certain period for Logsign archive and signed, the compression process is initiated. You can update the period with this configuration. Options: Daily (default), 1 Hour, 2 Hours, 3 Hours, 4 Hours, 5 Hours, 6 Hours, 8 Hours, 12 Hours.
Above the Persist Settings section, you can create a Data Retention Policy with Add to define separate Delete Signed Data Before and Delete Archived Data Before values for a group of sources.
Index and incident retention: The retention period of the index stored with Elasticsearch and of incidents is no longer set on the Data Retention page. These values are on the Settings > Data Management > Index Management page: Total Data Retention Period (Days) (event indices older than this are deleted, default: 7) and Incident Data Retention Period (Days) (incident indices older than this are deleted, default: 7). On the same page, Frozen Data Retention Period (Days) and Closed Data Retention Period (Days) can be used to freeze or close older event indices (0 = disabled).
Set Raw Msg: Sometimes, you may want to see the original (raw) version of your logs received with Logsign. For this process, raw message opening is required. This option is no longer a global setting; it is enabled per source with the Set Raw Msg checkbox in the source form under Settings > Integrations > Data Collection (default: off). As a result of this process, the disk space the logs occupy will double, and hardware usage for indexing will also be affected. We recommend keeping this feature off except when necessary.
Policies
The Settings > Data Management > Policies page contains the Health Check Eps Average field (default: 30) and the Incident Policy section with Incident Time Period (hours) (0 to 168, default: 168) and Incident Count Limit (0 to 200, default: 200).
There is no disk usage option on this page. Disk usage is checked automatically by the health check service: when disk usage reaches 90%, offline report processing is paused; when it reaches 95%, data collection stops to prevent critical problems in the system. Both resume automatically when disk usage drops below 90%. You can track disk usage rates from the indicator panel.
Index Policy
The configurations under the Index Policy are related to the structure of the index logs stored with Elasticsearch. The values below are the defaults of an all-in-one installation.
|
Logsign Events Index Shard Count : 1 Logsign Events Index Replica Count : 0 |
It is the index stored for health control services on Logsign Events. 0 is defined in order not to keep a backup as a replica. |
|
Positions Index Shard Count :1 Positions Index Replica Count :0 |
Position Index: It finds the positions of pollers such as SMB, SFTP, MSSQL and is used to index them again. |
|
Incidents Index Shard Count :1 Incidents Index Replica Count :0 |
It is the index area where the cases in the event management panel are stored. |
|
Index Shard Count :1 Index Replica Count :0 |
It is the index area where logs are stored on a daily basis. |
|
Index Merge Factor :1000 Index Refresh Interval :60s |
Elasticsearch writes many logs to the index instantly; However, the hardware does not send instantaneous logs to the user interface for effective use, it is expected that 1000 logs are collected or 60 seconds are expected to display the logs in the user interface. |
|
Search Query Execute Preference : _primary_first Report Query Execute Preference : _primary_first |
The Elasticsearch shard preference used when search and report queries are run. |
The above visual shows the configurations of Logsign, which is a cluster-based structure. If you are using a cluster-based structure, your nodes that make up the cluster are distributed across different shards on each node. This distribution of shards on nodes aims to provide faster response times for Logsign queries and to balance the Elasticsearch performance, in a way.
Replica shards are provided to you for redundancy, which is one of the basic purposes of the cluster structure. Replica shards take a clone of the existing shard, and if any node becomes unreachable, the replica shard performs the operation instead of the lost shard.
Let us explain this situation with an example. In the image above, there are shards between p0-p5 and r0-r5. P0-p5 is the shard where queries with primary shards are run. R0-r5 is replica shard, in case of loss between p0-p5, r0-r5 clone shards will be used .