-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Duplicated logs ingested into Sentinel with OCI (Azure Functions) Data Connector #10863
Comments
Hello team, |
Hey @fa-clavis, Thanks for flagging this issue, we will investigate this issue and get back to you with some updates. Thanks! |
Hello @v-sudkharat, any updates regarding this issue? |
Hey @fa-clavis, We are checking the function code for duplicate ingestion of data, just want to know, it's possible to you to share OCI demo account with us? so we can check our changes. Or please let us also know if you have any test workspace into your environment where you can test our change. Thanks! |
Hello @v-sudkharat , I talked with the client here, and they don't have a test environment available for OCI, however, they prefer to do the troubleshooting in a quick meet. |
Hi @fa-clavis, let us check with our team to validate some changes, which required the OCI env. once it gets validated will update you. Thanks! |
Hello @v-sudkharat, great news! |
Hi @fa-clavis, Could you please share few events generated with same id_g with us via a mail - [email protected] |
Hi @fa-clavis, Just want to check with you, Is there any multiple function app deployed into the environment and point towards the same workspace? |
@fa-clavis, Waiting for your response on above comment, so based on that we can reach out to respective team. Thanks! |
Hi @fa-clavis, Any response for us? |
Hello @v-sudkharat, sorry for the delay. |
@fa-clavis, thank you for the update, we are connecting with our concern data connector team for this issue, we will keep you updated. Thanks! |
Hello @v-sudkharat, any updates regarding this issue? |
Hi @fa-clavis, We are still in connect with appropriate data connector team for this issue, and we will once get back to you with some update from them. |
Hello @v-sudkharat, alright, no problem! |
Hi @fa-clavis, With your shared data I can see there are slight change in time oracle_ingestedtime_t column for id 8e8b4a6c-0877-4ff8-b51b-4ac04fecdc59 so, can you check the same in oracle side as well if you are getting different oracle_ingestedtime in RAW logs of OCI : |
Hello @v-sudkharat, yes it is correct on the OCI side. However, this doesn't explain why I would have that same event in Sentinel, 542 times as shown in the “count_logs_query_data.csv” file that I sent through e-mail. As I've been mentioning for a while now, there is probably a problem with the loop within the function app as it keeps iterating through the same log somehow. Because there is no problem on the OCI side, it's a truly 100% MS problem. |
Hi @fa-clavis, Thank you for your response. We have reached out to our concern team for this issue and shared the issue details, to troubleshoot on this issue team suggested to open an azure support case. |
Hi @fa-clavis, Could you please confirm did you raise a support case in azure? so our support team can prioritize this case. Thanks! |
Hello @v-sudkharat, |
Hi @fa-clavis, The support case helps our concern team to get the required access of workspace/data to troubleshoot the root cause of issue, which cannot be shared or done via a GitHub as it's as open source, and it also help to prioritize/escalate the issue to respective teams to gets resolve. |
Hello @v-sudkharat, alright, thanks for your explanation. |
@fa-clavis, Thanks! |
Hi @fa-clavis, Any update on case? |
Hello @v-sudkharat, unfortunately, I still don't have any updates with the team because of some complexity that involves the company budget and another internal team's approval inside the company. |
@fa-clavis, Ok. Thanks! |
Describe the bug
Hello team,
When I was going to model and create some analytics rules for a client, I noticed that there were multiple logs with the same "id_g" field populated into the OCI_Logs_CL table.
This "id_g" field (in OCI logs it's the eventID) is the unique identifier for which every alert receives, so there shouldn't be multiple logs with the same ID in Sentinel.
Source: https://docs.oracle.com/en-us/iaas/Content/Audit/Reference/logeventreference.htm
I have installed the default Azure Functions with the ARM template, while I have crosschecked multiple eventIDs in the OCI audit logs, I'm not sure why the same log is being populated 2,3,5, 30 times or more.
See the attached screenshots to understand the issue better.
P.S. I suspected something was odd with the connector because the amount of logs that's being ingested because the amount of logs that are produced in OCI is much less than the client's AWS environment.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
The expected behavior is to ingest each eventID once and not multiple times.
Screenshots
Additional context
To install and configure the data connector, I used the following resources:
While the log is unique within the Audit Log service, I strongly believe there is something wrong with the Function App or the Python script where the cursor isn't being set up correctly as it pages through the logs.
I believe the problem is somewhere near this line in the main.py file:
Azure-Sentinel/Solutions/Oracle Cloud Infrastructure/Data Connectors/AzureFunctionOCILogs/main.py
Line 131 in 296f272
The text was updated successfully, but these errors were encountered: