Introduction
This article will provide information about Data Policy Management and entry-level log filtering.
Entry Filter
The entry filter with Logsign DPM varies depending on the sources received through syslog and poller. In this article, entry-level log filtering process will be explained for a source that logs through syslog.
When a filter is applied at the entry level (For Input), Logsign filters the logs before any action is taken, allowing log dropping without affecting the system hardware too much.
When you create a rule at the entry level, the filtered log:
- Will not be written to disk,
- Will not be checked by your alarm rules,
- Will not provide list and behavior control,
- Will not enter the UEBA radar,
- Will not write Archive, Signed, and Index logs.
In summary, applying a filter at the entry level does not allow access to the system input.
As DPM has a rule at the entry level, the rules must be written in regex format.
Before starting the filtering process, you need to note the raw forms of the logs to be filtered, and you can enable the rawmsg module for this purpose.
Let's continue with an example related to this topic.
The above logs are the logs generated when a service account and source login occur from a Microsoft Windows source, and we can drop these logs at the entry level.
We note the raw form of the log.
Rawmsg:
Apr 27 14:22:01 EDU_WEF.logsign.academy Microsoft-Windows-Security-Auditing[808]: {"EventTime":1682594521102595,"Hostname":"EDU_WEF.logsign.academy","Keywords":-9214364837600034816,"EventType":"AUDIT_SUCCESS","SeverityValue":2,"Severity":"INFO","EventID":4624,"SourceName":"Microsoft-Windows-Security-Auditing","ProviderGuid":"{54849625-5478-4994-A5BA-3E3B0328C30D}","Version":2,"Task":12544,"OpcodeValue":0,"RecordNumber":9914359,"ActivityID":"{54E0D701-767D-0006-06D7-E0547D76D901}","ProcessID":808,"ThreadID":1272,"Channel":"Security","Message":"An account was successfully logged on.\r\n\r\nSubject:\r\n\tSecurity ID:\t\tS-1-0-0\r\n\tAccount Name:\t\t-\r\n\tAccount Domain:\t\t-\r\n\tLogon ID:\t\t0x0\r\n\r\nLogon Information:\r\n\tLogon Type:\t\t3\r\n\tRestricted Admin Mode:\t-\r\n\tVirtual Account:\t\tNo\r\n\tElevated Token:\t\tYes\r\n\r\nImpersonation Level:\t\tImpersonation\r\n\r\nNew Logon:\r\n\tSecurity ID:\t\tS-1-5-21-1652294852-620328662-553045647-1000\r\n\tAccount Name:\t\tEDU_DC01$\r\n\tAccount Domain:\t\tLOGSIGN.ACADEMY\r\n\tLogon ID:\t\t0x96FD3BC\r\n\tLinked Logon ID:\t\t0x0\r\n\tNetwork Account Name:\t-\r\n\tNetwork Account Domain:\t-\r\n\tLogon GUID:\t\t{ea95f93f-7cf6-4aae-47fe-35a5e3b7a503}\r\n\r\nProcess Information:\r\n\tProcess ID:\t\t0x0\r\n\tProcess Name:\t\t-\r\n\r\nNetwork Information:\r\n\tWorkstation Name:\t-\r\n\tSource Network Address:\t-\r\n\tSource Port:\t\t-\r\n\r\nDetailed Authentication Information:\r\n\tLogon Process:\t\tKerberos\r\n\tAuthentication Package:\tKerberos\r\n\tTransited Services:\t-\r\n\tPackage Name (NTLM only):\t-\r\n\tKey Length:\t\t0\r\n\r\nThis event is generated when a logon session is created. It is generated on the computer that was accessed.\r\n\r\nThe subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.\r\n\r\nThe logon type field indicates the kind of logon that occurred. The most common types are 2 (interactive) and 3 (network).\r\n\r\nThe New Logon fields indicate the account for whom the new logon was created, i.e. the account that was logged on.\r\n\r\nThe network fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.\r\n\r\nThe impersonation level field indicates the extent to which a process in the logon session can impersonate.\r\n\r\nThe authentication information fields provide detailed information about this specific logon request.\r\n\t- Logon GUID is a unique identifier that can be used to correlate this event with a KDC event.\r\n\t- Transited services indicate which intermediate services have participated in this logon request.\r\n\t- Package name indicates which sub-protocol was used among the NTLM protocols.\r\n\t- Key length indicates the length of the generated session key. This will be 0 if no session key was requested.","Category":"Logon","Opcode":"Info","SubjectUserSid":"S-1-0-0","SubjectUserName":"-","SubjectDomainName":"-","SubjectLogonId":"0x0","TargetUserSid":"S-1-5-21-1652294852-620328662-553045647-1000","TargetUserName":"EDU_DC01$","TargetDomainName":"LOGSIGN.ACADEMY","TargetLogonId":"0x96fd3bc","LogonType":"3","LogonProcessName":"Kerberos","AuthenticationPackageName":"Kerberos","WorkstationName":"-","LogonGuid":"{ea95f93f-7cf6-4aae-47fe-35a5e3b7a503}","TransmittedServices":"-","LmPackageName":"-","KeyLength":"0","ProcessName":"-","IpAddress":"-","IpPort":"-","ImpersonationLevel":"%%1833","RestrictedAdminMode":"-","TargetOutboundUserName":"-","TargetOutboundDomainName":"-","VirtualAccount":"%%1843","TargetLinkedLogonId":"0x0","ElevatedToken":"%%1842","EventReceivedTime":"2023-04-27 14:22:01","SourceModuleName":"lgs_eventlog","SourceModuleType":"im_msvistalog"}
This is a technical document that explains how to use regular expressions (regex) to filter unwanted logs from a log file. It also mentions how to use an online regex platform, such as regex101, to check for errors in the regex code. The example in the document shows how to filter logs with Event ID 4624 and a user name that ends with "$". The document also explains how to write the regex code in Go Lang and how to test it using the online platform regex101.
Unwanted columns and values should be identified within the log.
In this filtering, our filter rule will be the Event ID is 4624 and the user name ends with $. This way, we will drop successful session logs that end with $.
When the regex code is written correctly, an example log will be marked, indicating that the regex code is usable.
EventID\"\:4624.*?TargetUserName\"\:\".*\$\"
We prepared the regex code above in two parts.
EventID":4624 -
Since " and : are special characters, we need to add the \ character.
We prepared the first code as EventID\"\:4624
We are adding \ before special characters of TargetUserName":"EDU_DC01$.
TargetUserName\"\:\"EDU_DC01\$\"
In logs coming from multiple sources, EDU_DC01 can be changed to EDU_DC02 or a different source name. Therefore, we need to write it in a generic way.
TargetUserName\"\:\".*\$\"
We added an extra expression for the case where any expression comes after the red highlighted part. We completed the code for the second rule as well.
We used .*? expression to pass all expressions between Event.ID:4624 and TargetUserName and to pass directly to TargetUserName.
Regex code: EventID\"\:4624.*?TargetUserName\"\:\".*\$\"
Let's make this example a little different. Let's examine the case where the Event ID is 4624 or 4634 and the username ends with $.
We used () and | expressions for the OR operation.
Regex code: EventID\"\:(4624|4634).*?TargetUserName\"\:\".*\$\"
We can define the regex code to DPM.
After pasting the code into "Exclude By Regexp", we add the code with "Add".
We save it to DPM.
We can check by looking at the Filter Stats graph on the Eps Stats panel.
You can check that the logs filtered by a different method are not coming.