This article will provide information about the Elasticsearch service and index management.
Elasticsearch is a database service that works in a NoSQL structure and provides full-text search support.
It is important that Elasticsearch provides fast and stable results in the large data it stores. Instead of searching the queries we run in the text, it searches in the blocks inside the index, providing faster results within seconds.
How Logsign Search Works?
When a log is transferred to Elasticsearch, the data is indexed with the Apache Lucene infrastructure. During the transfer process, Elasticsearch indexes which document (row) the log is stored in. Thus, when we run a query, instead of searching for the query in all indexes, it searches only the related logs in the address, allowing for faster results and more effective use of server hardware.
Let's explain with an example query.
When you call a query with Source.UserName:umit.gunes, the stages shown in the picture below occur.
Logsign creates new indexes with the name log_YYYYMMDD for your logs with 24-hour cycles and stores the logs collected during the day in these indexes.
Your indexes are stored for the number of days specified in the Delete Live Report Before configuration in the Data Retention panel, and the indexes outside the specified day are deleted.
The configurations in the Index Policy panel are used to define the shard and replica numbers. Our logs, which are stored as JSON in Elasticsearch indexes, are located in the shards. If you are using a cluster structure, you will see shards configured in your system as 6 or more.
When your shards increase with the increase in your logs and the slow response of your queries, and the shard size reaches an average of 25GB, the process of increasing the number of shards is carried out. You can see the effects of this process in the next index formation.
It is recommended to close our indexes when you apply a Logsign update or routine restart of the Logsign server.
You can use the following command on the Logsign server for this process.
curl -XPOST 'http://localhost:9200/log_*/_close'
If you are using Cluster Logsign, you need to update it with the internal interface address or hostname.
curl -XPOST 'http://188.8.131.52:9200/log_*/_close'
To open the indexes after the update process is completed, you can use the following command.
curl -XPOST 'http://localhost:9200/log_*/_open'