Mimecast_Elastic integration #2157

djordje-adzemovic-devtech · 2021-11-15T08:21:46Z

What does this PR do?

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
If I'm introducing a new feature, I have modified the Kibana version constraint in my package's manifest.yml file to point to the latest Elastic stack release (e.g. ^7.13.0).

Author's Checklist

[ ]

How to test this PR locally

Related issues

Screenshots

cla-checker-service · 2021-11-15T08:21:50Z

💚 CLA has been signed

elasticmachine · 2021-11-15T08:25:28Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-11-26T11:12:24.912+0000
Duration: 21 min 39 sec
Commit: 913f122

Test stats 🧪

Test	Results
Failed	0
Passed	65
Skipped	0
Total	65

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

djordje-adzemovic-devtech · 2021-11-15T09:55:11Z

/test

elasticmachine · 2021-11-15T10:12:51Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

marc-gr · 2021-11-15T10:13:27Z

/test

marc-gr

Left some comments. In addition could be interesting to add some system tests (you can take a look at the tests in the google_workspace integration for reference), this way we can ensure the httpjson config is set as expected on top of validating the pipelines.

marc-gr · 2021-11-15T10:16:04Z

packages/mimecast/data_stream/audit_events/elasticsearch/ingest_pipeline/default.yml

+ separator: "\\."
+ target_field: file.parts
+ if: 'ctx?.file?.name != null && ctx?.event?.action == "threat-intel-feed-download"'
+# - join:


I think this comment slipped in.

packages/mimecast/data_stream/audit_events/elasticsearch/ingest_pipeline/default.yml

marc-gr · 2021-11-15T10:29:55Z

packages/mimecast/docs/README.md

@@ -0,0 +1,5 @@
+# Mimecast/Elastic Integration
+
+This is a new integration created using the [elastic-package](https:/elastic/elastic-package) tool.


Would be nice to add some information about the integration in the readme, anything relevant to its setup, prerequisites, ... along with the list of fields and a sample event. You can take a look at the README template found in _dev/build/docs/README.md and use the one in some other packages as a guide.

marc-gr · 2021-11-15T10:32:13Z

packages/mimecast/data_stream/ttp_ap_logs/_dev/test/pipeline/test-ttp-ap-logs.log-expected.json

+ "from": "\u003c\u003e",
+ "message_id": "\u003cDB8P194MB0824EE4C8D360CCE3DEB0243A1B89@DB8P194MB0824.EURP194.PROD.OUTLOOK.COM\u003e",
+ "attachments": {
+ "hash": "2fb26be55ac710e4d9f80677ba24ae212dbb36bd934a0569fe521839e9f5d16e",


Hashes, user info, ips, etc.. would be nice if you could add a processor to copy them into the related.* fields, it is very useful for cross referencing and making searches easier. https://www.elastic.co/guide/en/ecs/current/ecs-related.html

marc-gr · 2021-11-15T10:33:05Z

packages/mimecast/data_stream/ttp_ap_logs/_dev/test/pipeline/test-ttp-ap-logs.log-expected.json

+ "name": "Inbound - Safe file with On-Demand Sandbox"
+ },
+ "event": {
+ "action": "user release, none",


I am not 100% sure spaces and commas are desired in the event.action field, maybe processing it a bit to clean it up would be nice

ebeahan · 2021-11-18T22:08:28Z

@djordje-adzemovic-devtech the Mimecast integration is looking awesome!

@jamiehynds had mentioned you've adopted some fields proposed over in the ECS email RFC.

Not sure if it impacts any work here, but I wanted to call out that I've changed the proposed mappings a bit based on feedback from several users. A summary of the changes is covered in elastic/ecs#1593 (comment)

djordje-adzemovic-devtech · 2021-11-19T13:11:00Z

Thank you @ebeahan ! Should not be such a big deal, and will be implemented.

marc-gr · 2021-11-23T09:26:19Z

/test

marc-gr · 2021-11-23T14:56:25Z

/test

marc-gr · 2021-11-24T08:28:41Z

/test

marc-gr · 2021-11-24T08:49:24Z

/test

marc-gr · 2021-11-24T08:53:43Z

/test

marc-gr · 2021-11-24T15:22:06Z

/test

marc-gr · 2021-11-24T15:29:27Z

/test

marc-gr · 2021-11-24T16:23:06Z

/test

marc-gr · 2021-11-25T09:19:03Z

/test

marc-gr · 2021-11-25T09:50:24Z

Great work! Could you add a screenshot of the dashboards and the setup page? Would help a lot :)

marc-gr

Some small changes aside from the screenshots and I think it should be good to go

marc-gr · 2021-11-25T09:26:15Z

packages/mimecast/changelog.yml

+ changes:
+ - description: Initial draft of the package
+ type: enhancement
+ link: https:/elastic/integrations/pull/0 # FIXME Replace with the real PR link


Suggested change

link: https:/elastic/integrations/pull/0 # FIXME Replace with the real PR link

link: https:/elastic/integrations/pull/2157

marc-gr · 2021-11-25T09:53:19Z

packages/mimecast/data_stream/audit_events/elasticsearch/ingest_pipeline/default.yml

+---
+description: Pipeline for processing sample logs
+processors:
+ # Generic event/ecs fields we always want to populated


Suggested change

# Generic event/ecs fields we always want to populated

# Generic event/ecs fields we always want to populate

marc-gr · 2021-11-25T10:11:27Z

packages/mimecast/data_stream/audit_events/elasticsearch/ingest_pipeline/default.yml

+ - set:
+ field: event.created
+ value: "{{mimecast.date}}"
+ if: 'ctx?.mimecast?.date != null && ctx?.mimecast?.time != null'
+ - append: 
+ field: event.created
+ value: "{{mimecast.time}}"
+ if: 'ctx?.mimecast?.date != null && ctx?.mimecast?.time != null'
+ - join: 
+ field: event.created
+ separator: " "
+ if: 'ctx?.mimecast?.date != null && ctx?.mimecast?.time != null'


Suggested change

- set:

field: event.created

value: "{{mimecast.date}}"

if: 'ctx?.mimecast?.date != null && ctx?.mimecast?.time != null'

- append:

field: event.created

value: "{{mimecast.time}}"

if: 'ctx?.mimecast?.date != null && ctx?.mimecast?.time != null'

- join:

field: event.created

separator: " "

if: 'ctx?.mimecast?.date != null && ctx?.mimecast?.time != null'

- join:

field: event.created

separator: " "

if: 'ctx?.mimecast?.date != null && ctx?.mimecast?.time != null'

marc-gr · 2021-11-25T10:20:55Z

packages/mimecast/data_stream/siem_logs/elasticsearch/ingest_pipeline/default.yml

+ ### RECEIPT LOGS
+ - rename:
+ field: mimecast.acc
+ target_field: cloud.account.id


This field might be populated in some cases, would be necessary to re-map to a custom field to avoid issues (it is failing right now in CI, for example one or more errors found in documents stored in logs-mimecast.siem_logs-ep data stream: [0] found error.message in event: field [cloud.account.id] already exists

djordje-adzemovic-devtech · 2021-11-25T11:41:03Z

Here are dashboards:

djordje-adzemovic-devtech · 2021-11-25T12:10:57Z

Setup page screenshots:

marc-gr · 2021-11-26T10:26:44Z

/test

marc-gr · 2021-11-26T10:27:02Z

/test

marc-gr · 2021-11-26T11:11:16Z

/test

* Mimecast package initial build * First Data stream added * Package commited * Mimecast Elastic package - everything should be ok now * Correct test which fails * CR requested changes * Nullcheck and formatting fixes Co-authored-by: Marc Guasch <[email protected]>

djordje-adzemovic-devtech added 4 commits October 7, 2021 13:24

Mimecast package initial build

25d40b1

Merge branch 'master' of https:/elastic/integrations

f679895

First Data stream added

99d0528

Package commited

db9ba76

djordje-adzemovic-devtech marked this pull request as draft November 15, 2021 08:21

djordje-adzemovic-devtech marked this pull request as ready for review November 15, 2021 08:50

marc-gr self-requested a review November 15, 2021 10:12

marc-gr added enhancement New feature or request Team:Security-External Integrations labels Nov 15, 2021

marc-gr requested changes Nov 15, 2021

View reviewed changes

djordje-adzemovic-devtech requested a review from marc-gr November 22, 2021 14:27

Mimecast Elastic package - everything should be ok now

ec6f9d4

Correct test which fails

2d3c5db

marc-gr requested changes Nov 25, 2021

View reviewed changes

marc-gr reviewed Nov 25, 2021

View reviewed changes

CR requested changes

bc699bb

Nullcheck and formatting fixes

913f122

marc-gr approved these changes Nov 26, 2021

View reviewed changes

marc-gr merged commit 0346221 into elastic:master Nov 26, 2021

andrewkroh added the Integration:mimecast Mimecast label Feb 11, 2022

andrewkroh mentioned this pull request May 2, 2022

[production] Promote many packages from snapshot elastic/package-storage#4302

Merged

andrewkroh added the New Integration label Aug 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mimecast_Elastic integration #2157

Mimecast_Elastic integration #2157

djordje-adzemovic-devtech commented Nov 15, 2021 •

edited

Loading

cla-checker-service bot commented Nov 15, 2021 •

edited

Loading

elasticmachine commented Nov 15, 2021 •

edited

Loading

Build stats

Test stats 🧪

djordje-adzemovic-devtech commented Nov 15, 2021

elasticmachine commented Nov 15, 2021

marc-gr commented Nov 15, 2021

marc-gr left a comment

marc-gr Nov 15, 2021

marc-gr Nov 15, 2021

marc-gr Nov 15, 2021

marc-gr Nov 15, 2021

ebeahan commented Nov 18, 2021

djordje-adzemovic-devtech commented Nov 19, 2021

marc-gr commented Nov 23, 2021

marc-gr commented Nov 23, 2021

marc-gr commented Nov 24, 2021

marc-gr commented Nov 24, 2021

marc-gr commented Nov 24, 2021

marc-gr commented Nov 24, 2021

marc-gr commented Nov 24, 2021

marc-gr commented Nov 24, 2021

marc-gr commented Nov 25, 2021

marc-gr commented Nov 25, 2021

marc-gr left a comment

marc-gr Nov 25, 2021

marc-gr Nov 25, 2021

marc-gr Nov 25, 2021

marc-gr Nov 25, 2021

djordje-adzemovic-devtech commented Nov 25, 2021

djordje-adzemovic-devtech commented Nov 25, 2021

marc-gr commented Nov 26, 2021

marc-gr commented Nov 26, 2021

marc-gr commented Nov 26, 2021

		@@ -0,0 +1,5 @@
		# Mimecast/Elastic Integration

		This is a new integration created using the [elastic-package](https:/elastic/elastic-package) tool.

	link: https:/elastic/integrations/pull/0 # FIXME Replace with the real PR link
	link: https:/elastic/integrations/pull/2157

	# Generic event/ecs fields we always want to populated
	# Generic event/ecs fields we always want to populate

Mimecast_Elastic integration #2157

Mimecast_Elastic integration #2157

Conversation

djordje-adzemovic-devtech commented Nov 15, 2021 • edited Loading

What does this PR do?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

cla-checker-service bot commented Nov 15, 2021 • edited Loading

elasticmachine commented Nov 15, 2021 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

djordje-adzemovic-devtech commented Nov 15, 2021

elasticmachine commented Nov 15, 2021

marc-gr commented Nov 15, 2021

marc-gr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ebeahan commented Nov 18, 2021

djordje-adzemovic-devtech commented Nov 19, 2021

marc-gr commented Nov 23, 2021

marc-gr commented Nov 23, 2021

marc-gr commented Nov 24, 2021

marc-gr commented Nov 24, 2021

marc-gr commented Nov 24, 2021

marc-gr commented Nov 24, 2021

marc-gr commented Nov 24, 2021

marc-gr commented Nov 24, 2021

marc-gr commented Nov 25, 2021

marc-gr commented Nov 25, 2021

marc-gr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

djordje-adzemovic-devtech commented Nov 25, 2021

djordje-adzemovic-devtech commented Nov 25, 2021

marc-gr commented Nov 26, 2021

marc-gr commented Nov 26, 2021

marc-gr commented Nov 26, 2021

djordje-adzemovic-devtech commented Nov 15, 2021 •

edited

Loading

cla-checker-service bot commented Nov 15, 2021 •

edited

Loading

elasticmachine commented Nov 15, 2021 •

edited

Loading