Logsign Unified SecOps Platform, with more than 200 prepared integrations, collects logs in real-time from hundreds of data sources. Optimization is required on Logsign Unified SecOps Platform as the logs continuously increase. Logsign Unified SecOps Platform DPM (Data Policy Manager) enables you to optimize log management.
- To collect logs from a source or grouped log sources,
- To manage the redundancy durations of the logs,
- To facilitate the management of our storage capacity,
You can create data policies with Logsign Unified SecOps Platform DPM and increase the data collection, storage, and indexing performance.
Every source uses a data policy. A new source starts with the system policy named Default, which does not filter anything. You can assign another policy to the source, or adjust the policy of a single source from the source's own settings in Data Collection.
Click the Settings > Integrations > Data Policies tab on the Logsign Unified SecOps Platform WEB interface. You will see the Data Policy adjusted as default under the name Default on the page. The data load of the incidents on the relevant source will be relieved if you create a separate Data Policy for each source.
Each policy in the list has Edit, Delete and Apply buttons. The Default policy is a system policy, so it has no Delete button: you can edit it but you cannot delete it. When you change a policy, click Apply to push the saved version to the sources that already use it. The Apply Policy window lists those sources (Device Type, Device Name, Device Ip), and Apply All Sources updates all of them. Apply does not assign the policy to sources that use a different policy; if no source uses the policy, the window does not open.
When you click the + Policy button on the upper right corner of the page, you'll see the screen where you can configure the Data Policy you want to create.
Policy Name: You can determine a name relevant to the source with which it will be correlated. (Example: WindowsDataPolicy). Only letters, digits, spaces and the characters _ . - / are accepted, and the name must be unique.
For Input
In this section, you can drop the incident movements you do not want on Logsign Unified SecOps Platform at the first level, before they are parsed, indexed or stored. Filtering here is done with Regexp (regular expression, a pattern language for matching text). The rules are applied to the raw log line. Starting with version 6.4.115, the input filters also apply to API (poller) sources.
Include By Regexp: When you tick the box next to it, an input box and an Add button appear below. Type the expression, click Add, and it is added to the Regexp List. You can add more than one expression. An entry that is not a valid regular expression is rejected with the message "Please enter a valid regex expression!". A plain word (ex: malware) is also a valid expression. Characters such as ., |, $, ( and [ have a special meaning, so escape them with a backslash when you mean the character itself. For example, to match the IP address 10.0.0.10 exactly, write 10\.0\.0\.10.
You can use the Regexp examples below:
"192\.168\.1\.1": When you need to enter the IP address
"wareztool.": It can be matched with expressions such as "wareztools, wareztoolz, wareztoolx"
"Li[cs]ense" : It checks two expressions: "License or Lisense"
"malware|trojan": It looks for a match with the words "malware or trojan"
"cain (download|indir)": It checks the expression "cain download or cain indir"
"h3808063@mvrht\.com" : It looks for the e-mail address h3808063@mvrht.com
The examples above are written without the quotation marks when you enter them, and they are not limited.
Please note: The Include By Regexp list is only evaluated when Exclude By Regexp is also active and has at least one expression. An incident movement that matches an include expression is always kept, even if it also matches an exclude expression.
Exclude By Regexp: When you tick the box next to it, an input box and an Add button appear below. The incident movements that match the expressions in this list are not written on Logsign Unified SecOps Platform.
When the Exclude By Regexp box is not ticked, Logsign Unified SecOps Platform accepts all incident movements, so you do not need to fill in Include By Regexp. However,
- if you have a source on your system and you are familiar with the incident movements coming from this source; and
- if you do not want to select the irrelevant incident movements coming in between,
you can add .* (an expression that matches every line) to Exclude By Regexp, and then add only the incident movements you want to keep to Include By Regexp. A single * is not a valid regular expression and is rejected.
Syslog Redundancy Period: Enter a value in seconds. When the same log line arrives more than once from a syslog source within this period, only the first one is accepted. Leaving the value empty or 0 turns the check off.
Field Redundancy
The rules in this section are applied when the log is parsed, so an incident movement dropped here is not indexed or stored as JSON. Click the Add button to open the Add Field Redundancy window.
Period: You can give values in terms of seconds in this section. For the fields you will choose on the Fields row, we can select only one incident movement if more than one of the same within x seconds.
Fields: Select the fields that are compared. Incident movements whose values in these fields are the same within the period are treated as duplicates.
Collect Fields: When you tick the box next to it, the Collection Fields row appears.
Collection Fields: Select the fields whose values are collected from the duplicate incident movements during the period.
When you click the Save button in the window, the rule is added to the table with the Period, Fields and Collection Fields columns. You can create more than one rule by repeating the same process.
For Indexing
In this section, you can determine the incident movements you would like to be written on the Index.
Filter Index Fields: If the box next to it is clicked, the relevant setting will be activated, and the Index Fields row will appear.
Index Fields: Only the fields you select here are written on the Index. The time, severity, event source and data type fields are always kept.
Include Logs: When you tick the box next to it, an input box and an Add button appear below. The entries are SystemID values.
To find the SystemID of an incident movement, open it on the Search page; the value is in the Event.SystemID field of the log details. Type the SystemID, click Add, and repeat for each SystemID you want to add.
Please note: The Add button of these lists checks every entry as a regular expression, so a single * is currently rejected. Add the SystemID values one by one.
Exclude Logs: It can be used to prevent the undesired and irrelevant incident movements from the relevant source to be written on the Index. The incident movements whose SystemID is in this list are not indexed. A SystemID that is also in the Include Logs list is always indexed.
Include By Regexp: When you tick the box next to it, an input box and an Add button appear below. An incident movement that matches one of these expressions is indexed even if it matches an exclude rule.
You can use the Regexp examples below:
"192\.168\.1\.1": When you need to enter the IP address
"wareztool.": It can be matched with expressions such as "wareztools, wareztoolz, wareztoolx"
"Li[cs]ense" : It checks two expressions: "License or Lisense"
"malware|trojan": It looks for a match with the word "malware or trojan"
"cain (download|indir)": It checks the expression "cain download or cain indir"
"h3808073@mvrht\.com" : It looks for the e-mail address h3808073@mvrht.com
Exclude By Regexp: When you tick the box next to it, an input box and an Add button appear below. The incident movements that match these expressions are not written on the Index.
When the Exclude By Regexp box is not ticked, you do not need to fill in the Include By Regexp section.
Include by Key-Value: When you tick the box, select a field in Add Key, type its value in Add Value and click Add. After adding the pairs, click Save As List. All pairs in one list must match; you can save more than one list. An incident movement that matches a list here is indexed even if it matches an exclude list.
Exclude by Key-Value: It allows you to select field-based incident movements you do not want to be written on Index from the relevant source. Lists are created in the same way as above. Key-value lists compare exact values and do not support *.
For JSON(Parsed) Store
This section shows the configurations we can make for the extracted incident movements on the Logsign Unified SecOps Platform archive.
Filter Index Fields: If the box next to it is clicked, you'll activate the relevant setting, and the Index Fields row will appear.
Index Fields: It is a feature you can keep on the archive only the selected fields from the relevant source. You do not store the irrelevant incident movements by not holding the areas you did not choose on the archive.
Exclude Logs: The incident movements whose SystemID is in this list are not kept on the JSON archive.
Include Logs: This list is evaluated together with Exclude Logs. When both are active, only the incident movements whose SystemID is in the Include Logs list are kept on the JSON archive. SystemIDs are found and added in the same way as in the For Indexing section.
Include by Key-Value: It is evaluated when Exclude by Key-Value is also active. An incident movement that matches a list here is kept on JSON.
Exclude by Key-Value: It allows you to select field-based incident movements you do not want to be kept on JSON from the relevant source. Key-value lists compare exact values and do not support *.
For RAW Store
In this field, you will see the configurations you can make on the raw versions (unprocessed, straight from the source) of the incident movements.
Include By Regexp: When you tick the box next to it, an input box and an Add button appear below. It is evaluated when Exclude By Regexp is also active; an incident movement that matches one of these expressions is written on the RAW file.
You can use the Regexp examples below:
"192\.168\.1\.1": When you need to enter the IP address
"wareztool.": It can be matched with expressions such as "wareztools, wareztoolz, wareztoolx"
"Li[cs]ense" : It checks two expressions: "License or Lisense"
"malware|trojan": It looks for a match with the word "malware or trojan"
"cain (download|indir)": It checks the expression "cain download or cain indir"
"h3808073@mvrht\.com" : It looks for the e-mail address h3808073@mvrht.com
Exclude By Regexp: When you tick the box next to it, an input box and an Add button appear below. It is the section where you can select the incident movements you do not want to be written on the RAW file from the relevant source.
When the Exclude By Regexp box is not ticked, you do not need to fill in the Include By Regexp section.
When you finish, click Save at the bottom of the page.
Persist groups
The For Persist section and the Persist group file option are no longer part of the data policy screen. Source-specific storage groups are now managed under Settings > Data Management > Data Retention, where you create a group with a Group Name and select its Sources.