Introduction
This article will provide information about the Elasticsearch service and index management.
Elasticsearch
Elasticsearch is a database service that works in a NoSQL structure and provides full-text search support.
It is important that Elasticsearch provides fast and stable results in the large data it stores. Instead of searching the queries we run in the text, it searches in the blocks inside the index, providing faster results within seconds.
How Logsign Search Works?
When a log is transferred to Elasticsearch, the data is indexed with the Apache Lucene infrastructure. During the transfer process, Elasticsearch indexes which document (row) the log is stored in. Thus, when we run a query, instead of searching for the query in all indexes, it searches only the related logs in the address, allowing for faster results and more effective use of server hardware.
Documents are indexed as JSON (JavaScript Object Notation).
Let's explain with an example query.
When you call a query with Source.UserName:umit.gunes, the stages shown in the picture below occur.
Logsign creates new indexes with the name log_YYYYMMDD for your logs with 24-hour cycles and stores the logs collected during the day in these indexes.
Your indexes are stored for the number of days specified in the Total Data Retention Period (Days) field under Settings › Data Management › Index Management (Index Management Configuration). Indexes older than this period are deleted from the system. On the same screen, Frozen Data Retention Period (Days) freezes older indexes (they stay searchable with low memory usage) and Closed Data Retention Period (Days) closes older indexes (they must be reopened before they can be searched). A value of 0 disables freezing or closing. The frozen period cannot be greater than the closed period, and neither can be greater than the total period.
Index Policy
The Index Policy section on the Settings › Data Management › Policies page is used to define the shard and replica numbers (Index Shard Count and Index Replica Count). Our logs, which are stored as JSON in Elasticsearch indexes, are located in the shards. If you are using a cluster structure, you will see shards configured in your system as 6 or more.
When your shards increase with the increase in your logs and the slow response of your queries, and the shard size reaches an average of 25GB, the process of increasing the number of shards is carried out. You can see the effects of this process in the next index formation.
Index Management
It is recommended to close our indexes when you apply a Logsign update or routine restart of the Logsign server.
You can do this from the interface under Settings › Data Management › Index Management. In the Index List, select the indexes and click Close Selected. After the update or restart is completed, select the same indexes and click Open Selected.
You can also use the following command on the Logsign server for this process.
AllinOne Logsign:
curl -XPOST 'http://localhost:9200/log_*/_close'
If you are using Cluster Logsign, you need to update it with the internal interface address or hostname.
curl -XPOST 'http://2.2.2.2:9200/log_*/_close'
To open the indexes after the update process is completed, you can use the following command.
AllinOne Logsign:
curl -XPOST 'http://localhost:9200/log_*/_open'
Cluster Logsign:
curl -XPOST 'http://2.2.2.2:9200/log_*/_open'