We first need to analyze and interpret the log that we will write Regex. Dividing it into sections and clarifying which sections we will take will strengthen our hand in the preparation process.
Oct 30 10:01:58 1.1.1.44 01-23-45-67-89:ab sapd[8634]: <33312> <NOTI> |AP Logsign-Turkey@1.1.1.77 sapd| |ap| zone0 Recv ERROR from 10.10.0.10 TunDev none Error RC_ERROR_IKEP1
We can start by writing the date format written in this log that we will use as an example. However, this date format can be in many different structures, it is necessary to prepare a different structure for each of them.
Oct 30 10:01:58 > (?P<timestamp>\w{3} \d{2} \d{2}:\d{2}:\d{2})
2024-10-30T10:01:58 > (?P<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2})
2024-10-30 22:01:58 > (?P<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})
2024-10-30T10:01:58.467641Z > (?P<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d+Z)
Note: In Python, having more than 6 numbers in the field I specified 10:01:58.467641Z will give an error. If there are 7 numbers in this section in the log example, you need to fix it or you need to remove the part after the dot.
2024-10-30T10:01:58.467641Z > 2024-10-30T10:01:58.
(?P<timestamp>\d{4}-\d{2}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2})\.\d+Z can be written like this. After that you can continue with the classic structure. You can leave a space and shape the regex according to your log type.
This is how you can get a reference.
Then we can gradually get all the logs that exist and that we think will be useful to us.
There is a simple way to use it, based on taking all the space until you see a gap, and you can use it a lot.
(\S+) when you do it this way, it will get the whole log until you see a space. However, in rawmsg we have a space after time. We need to pass this space as \s+ as well.
The structure that should be used for the space should be \s+ when the existing group ends.
We can use the same structure in the section we specify as MAC address, but the scenario changes here because there is a special character in the Service.Name section that follows.
Here I used a structure like (\w+), which corresponds to letters, numbers and underscores. It will take the existing characters until it gets to the section outside these and will stop when it gets to a special character.
Then, in the field that I will specify as Service.ID, I need to use a structure like this: \[(\d+)\]. The equivalent of this is to take the decimal part in square brackets.
However, if it was not just a number, then we would have to use a capital S instead of d.
In the next part of the log, there is a number inside the special characters <>. At this stage, we will use the same decimal structure, but we need to put <> where there are square brackets.
We do not forget to put \s+ in the spaces.
Then there is a word in <> instead of a decimal. When it is a word, the structure we use is (\w+). If there is both a word and a number here, we can do (\S+).
We need to use this structure \|(.+?)\| to get the section inside the pipe '|'. When you give the space in between and type this regex, it will get all the data in the pipe.
We do the same for the field in the next section and continue.
Generally, the information in the last parts of the log can be interpreted as details and can be extracted in one go. In this example, we have extracted the field in the last part of the log with the (.*) method that we can get all at once.
Structures and log types may vary in Regex. With a little practice, you can get the most accurate results quickly.
Do not hesitate to get support from artificial intelligence when writing regex. Since our regex structure is based on grouping logic, it is useful to print regex to AI by grouping it.
In a scenario where you throw the whole log and print a regex, you will usually get an error. In this case, instead of directing the AI to correct it over and over again, you can print it in groups and observe which area has an error yourself, manually correct that error or ask it to write a new regex directly for that section.
You can use Static mapping in logs. At this stage, you need to take reference from the Log Message List
However, if you want to make a mapping with file, you can find what you need to do specific to that method under Parse with JSON.