add universal events archiver function

[raylrui]

Feature/ universal-event-archiver

Features

Build a function to archive all events that go over the OrcaBus event bus, and write it as JSON to the dedicated S3 bucket Formatting the S3 key with year/month/day partitioning.
S3 object will be like: s3://{bucket_name}/events/{year}/{month}/{day}/{event_type}_{totalSeconds.microsecond}.json

To dos

• need to check if all events follow the event_schemas, as we need retrieve the event type (or request title, request type)
• time zone use utc or local time?
• output bucket?
• optional: DLQ problems? (need sqs queue for dlq)

[reisingerf]

Ah, very nice!
Already looks very good!

[reisingerf]

OK, looks great!

Final suggestion / discusstion:
I know it's not really "stateful", but it's very closely linked to the event bus and we would not want to miss any events just because of "random" redeployments of the stateless resources. Therefore I'd suggest to:

• move it under the stateful stack
• add it to the event bus construct instead of keeping it separate

@victorskl your thoughts?

[raylrui]

if we need add custom archiver, we can turn on Optional flag addCustomEventArchiver in the event bus construct with params of vpc, SG, destination Bucket:

addCustomEventArchiver?: boolean;
vpcProps?: VpcLookupOptions;
lambdaSecurityGroupName?: string;
archiveBucket?: string;

[victorskl]

@raylrui Is this PR ready for review? Or, would you intend to complete remainder TODO items?

Could you also pls help me do favour; after you finalised, can you pls tick/press the "Re-request review" circle button, next to the reviewer name, pls. That way, I will get notification and, I will attend to it, asap. Thank you.

[raylrui]

yes, @victorskl apology or that. So far this PR is ready for review, I will click the button, but nothing will happen when merged, as we don't open that flag and no bucket params defined.
I will make another PR if we wanna open that archiver and have bucket ready.

[victorskl]

Thanks Ray, no apology need. I'll review in a tick, next & feedback.

[reisingerf]

@victorskl FYI: just had a quick catch-up with Ray. He will add the archive bucket to the construct and create his own SG.
He'll also extend the eventbusConstruct.test.ts to check whether the addCustomEventArchiver flag is working as expected.

[victorskl]

Comment to attend. And, could you do:

• rebase or merge on top of current main
• turn addCustomEventArchiver flag on
• follow up archival bucket and, tests

Then, pls re-request review for me. Tq

[victorskl]

ditto fix filename:

index: 'universal_event_archiver.py',

[raylrui]

I thought it should be a noun, like "archiver" sth, but happy to have any suggestion

[victorskl]

Current filename is fine. It is just the typo.

[victorskl]

Need fix filename typo:

universal_event_archiver.py

[victorskl]

Also. Would be good, if you could unit test this handler. See token service cognitor as an example, if any. You can leverage botocore stubber.

[victorskl]

And sanitize_string. Generally, all public functions.

[raylrui]

sure, I will look it through first and make update in unit test for next commit

[reisingerf]

LGTM

[raylrui]

Hi team, recent updates on the archiver:

1. add unit test for archiver function with Stubber
2. refactor folder and code structure, and add makefile to make it easier to maintain
3. open the addCustomEventArchiver flag and define specific bucket event-archive-bucketand lambdaSG OrcaBusEventArchiveSecurityGroup in config
4. extend the cdk unit testing with addCustomEventArchiver and without addCustomEventArchiver in eventbusConstruct.test.ts
5. add NagSuppressions.addResourceSuppressionsByPath for SharedStack in deployment.test.ts for the permission between archive lambda and archive bucket, as the use of AWS managed policies and wildcard permission break some default AwsSolutions Rules Pack checks.

[victorskl]

thkz for update, Ray. I will review in tick after lunch. ETA 3pm

[victorskl]

Nicely done, Ray.! Go for it.

[reisingerf]

Let's roll!

[raylrui]

I will try to test it after deployment finish...