Introduction
This article will provide information on converting archive data that you have stored as live data.
Archive Data
We define our data that has undergone parsing and can be compressed and stored for a long time in the /opt/var/log/archive directory as archive data. With Elasticsearch service, we can store our live data for a certain period depending on hardware specifications and disk capacity, and if data needs to be stored beyond this period, it is kept as archive. By default, your archive data is kept for 365 days.
Offline Report
The offline report process involves writing our compressed data stored in the archive back to the Elasticsearch service as an index, and we call this operation reindexing. When this process is completed, you can perform fast and easy searches and prepare reports.
The offline report process is faster in a logsign that works in a cluster structure. In cluster structures, the offline report process splits the data into pieces on multiple servers using spark technology and transfers the data back to the index. This is the reason why fast offline report process is achieved in cluster structures.
In logsign operating as single (allinone), the offline worker service works, and with this service, your archive data is transferred back to the Elasticsearch service and turned into live data.
To start the Offline Report process, access the Reports -> Offline Reports panel.
Before starting the offline report process, a query should be created for the logs that need to be taken.
In the offline report process, you cannot use mini query, full-text search, * queries, only queries as key=value.
In the following example, a query is created to find the successful login activities of the umit.gunes user.
Source.UserName:umit.gunes EventMap.SubType:"Login"
By selecting the time range and clicking the Check Archive button, the archive data is checked.
With this information, we can understand that we have a file with a data size of 2.23 GB in the specified date range, and the query will search within 21,560,485 data.
To start the process, we complete the process with the Create Offline Report button.
When the offline report is started, you can follow the process in the Status Running section.
If you need multiple offline reports, you can also create the offline report process at the same time. Logsign will queue your offline report process and start the next offline report when the process is completed.
You can monitor the offline report process instantly before it is 100% complete. Click the Search button for this process.