We first need to analyze and interpret the log that we will write Regex. Dividing it into sections and clarifying which sections we will take will strengthen our hand in the preparation process.
Oct 30 10:01:58 1.1.1.44 01-23-45-67-89:ab sapd[8634]: <33312> <NOTI> |AP Logsign-Turkey@1.1.1.77 sapd| |ap| zone0 Recv ERROR from 10.10.0.10 TunDev none Error RC_ERROR_IKEP1
We can start by writing the date format written in this log that we will use as an example. However, this date format can be in many different structures, it is necessary to prepare a different structure for each of them.
Oct 30 10:01:58 > (?P<timestamp>\w{3} \d{2} \d{2}:\d{2}:\d{2})
2024-10-30T10:01:58 > (?P<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2})
2024-10-30 22:01:58 > (?P<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})
2024-10-30T10:01:58.467641Z > (?P<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d+Z)
Note: The time value is parsed with Python, and the fractional seconds part (%f) accepts at most 6 digits. In the example 10:01:58.467641Z there are 6 digits, so it works. If the fractional part in your log has 7 or more digits, leave the part after the dot outside the capture group.
2024-10-30T10:01:58.467641Z > 2024-10-30T10:01:58
(?P<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2})\.\d+Z can be written like this. In this case, select the %Y-%m-%dT%H:%M:%S time format for the group. After that you can continue with the classic structure. You can leave a space and shape the regex according to your log type.
This is how you can get a reference.
Each time you extend the regex, click Check Regex. The regex is tested against the log sample, and every captured group is listed as Group 1, Group 2 and so on. Every listed group must be assigned to a field before the Add button becomes active. If you assign a time field such as Time.Generated, you also need to select its Time Format. For this reason, only put the parts you want to map inside parentheses.
Then we can gradually get all the logs that exist and that we think will be useful to us.
There is a simple way to use it, based on taking all the space until you see a gap, and you can use it a lot.
(\S+) when you do it this way, it will get the whole log until you see a space. However, in rawmsg we have a space after time. We need to pass this space as \s+ as well.
The structure that should be used for the space should be \s+ when the existing group ends.
We can use the same structure in the section we specify as MAC address, but the scenario changes here because there is a special character in the Service.Name section that follows.
Here I used a structure like (\w+), which corresponds to letters, numbers and underscores. It will take the existing characters until it gets to the section outside these and will stop when it gets to a special character.
Then, in the field that I will specify as Service.ID, I need to use a structure like this: \[(\d+)\]. The equivalent of this is to take the decimal part in square brackets.
However, if it was not just a number, then we would have to use a capital S instead of d.
In the next part of the log, there is a number inside the special characters <>. At this stage, we will use the same decimal structure, but we need to put <> where there are square brackets.
We do not forget to put \s+ in the spaces.
Then there is a word in <> instead of a decimal. When it is a word, the structure we use is (\w+). If there is both a word and a number here, we can do (\S+).
We need to use this structure \|(.+?)\| to get the section inside the pipe '|'. When you give the space in between and type this regex, it will get all the data in the pipe.
We do the same for the field in the next section and continue.
Generally, the information in the last parts of the log can be interpreted as details and can be extracted in one go. In this example, we have extracted the field in the last part of the log with the (.*) method that we can get all at once.
Structures and log types may vary in Regex. With a little practice, you can get the most accurate results quickly.
Do not hesitate to get support from artificial intelligence when writing regex. Since our regex structure is based on grouping logic, it is useful to print regex to AI by grouping it.
In a scenario where you throw the whole log and print a regex, you will usually get an error. In this case, instead of directing the AI to correct it over and over again, you can print it in groups and observe which area has an error yourself, manually correct that error or ask it to write a new regex directly for that section.
Under Mapping, you can choose Static to assign a single EventMap value to the logs. At this stage, you need to take reference from the Log Message List
However, if you want to make a mapping with a file (File option under Mapping), you can find what you need to do specific to that method under Parse with JSON.