Inquiry on Log File Structure and Consolidation Impact

[IamTenacious]

Hello @georgemakrakis ,

I have been working with the Zeek-IEC104 plugin and noticed that it generates 29 different types of log files. I would like to understand the rationale behind maintaining multiple log files instead of consolidating the data into a single log. Could you please explain why the plugin is designed this way and what the potential impact might be if I were to merge all the logs into a single log file? Specifically, I am interested in any performance implications, data organization considerations, and how this might affect the usability and analysis of the log data.

[IamTenacious]

@georgemakrakis please reply

[georgemakrakis]

@IamTenacious thank you for the insightful comment. This was a choice I made at the time to be able to distinguish between the basic information about the protocol in the iec104.log file and then each individual information object code log file will include the "payload" information that can be used to identify the precise action occurred in the monitored environment, What I would like to integrate is an enable/disable option for the individual information object code log files, to give the user the choice to utilize them on demand.

I am not sure about the implications about including all the possible information into a single log file, but I think the current structure of the main.zeek file might help you to get started with that. I have also yet to conduct any performance analysis regarding this parser to evaluate the impact of the current approach in terms of data organization, usability and analysis of the log data.

[IamTenacious]

@georgemakrakis Thank you for your insightful explanation. I understand that currently, the logs are separated to distinguish between basic protocol information and detailed payload information for each information object code. The proposed enable/disable option for generating individual log files sounds like a great addition.

Just to confirm, with this upcoming feature, if I choose to consolidate all the logs into a single iec104.log file, there shouldn't be any significant issues in terms of performance or data organization, correct? I appreciate the flexibility this feature will provide and want to ensure that merging logs won't negatively impact my analysis or the overall performance.

Thank you for your continued support and for considering this feature enhancement.

[georgemakrakis]

@IamTenacious I am not sure if the proposed change will impact the performance of the analyzer, since I have not done that yet as mentioned above. I would appreciate it though, if you could provide any metrics and benchmarks that should be met, so when I perform this action, I will know how far or close we fall.