Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Filemanager: S3 events persistence #37

Merged
merged 53 commits into from
Nov 16, 2023
Merged

Filemanager: S3 events persistence #37

merged 53 commits into from
Nov 16, 2023

Conversation

brainstorm
Copy link
Member

  • Serializes S3 event attributes to DB backend.
  • Using Axum (and AWS SDK) for local development, will transition to lambda Rust runtime crates later.

brainstorm and others added 30 commits June 22, 2023 12:53
@brainstorm
Sketch out traits for this service, gather requirements from @reisingerf
e451288 
 documented in main.rs
@brainstorm
Add example data fixtures for S3 and GDS
29fe2a7
@brainstorm
Document audit requirements sketched by @reisingerf
d403c7b
@brainstorm
Migrate skel to real app 🌅
f34d5f7
@brainstorm
Add minor docs for filemanager cmdline local ops
9a95299
@brainstorm
Clarify that chain of custody is outside the scope of FM, should be t…
41d3cc9 
…ackled by native cloudwatch/cloudtrail
@brainstorm
Add async-trait, we should probably come up with a generic traits def…
cd6bb94 
…inition... probably worth revisiting until we have traits well defined on the real app and we can add them in retrospective review
@brainstorm
Add those settings, otherwise VSCode gives up searching for Rust work…
da2d3e2 
…spaces: webtoon/psd#83
@brainstorm
Add S3 events+SQS setup script
080982f
@brainstorm
[ci skip] Do not assume edits to /etc/hosts (localstack->localhost)
7357719
@brainstorm
Make sure this is fully reproducible for @mmalenic
d5521b1
@brainstorm
Correct migration syntax
346b441
@brainstorm
Add S3 events parsing with AWS SDK for Rust, lambda runtime (and http…
342d81e 
…) will come later on, first we run locally with Axum
@brainstorm
Run the s3-sqs script for localstack setup first of all...
f9513e3
@brainstorm
Add SQS example url for localstack (local development)
e19c0f6
@brainstorm
Add AWS SDK crates for Rust
c3c1b92
@brainstorm @mmalenic
Insert SQS values to DB
310fa0c 
Co-authored-by: Marko Malenic <[email protected]>
@brainstorm
More error mapping and show SQS messages
d5008e9
@brainstorm
Cargo lock
1cf0d18
@brainstorm
Fix SQL object_key vs key migration errors
8f75762
@brainstorm
Cargo update for time-rs
0df9fe0
@brainstorm
Cargo fmt
9d68ec0
@brainstorm
Fix sql keyword syntax error, first S3 event ingested successfully
f5e9875
@brainstorm @mmalenic
Complete S3 events object mapping to database.
877d054 
Co-authored-by: Marko Malenic <[email protected]>
@mmalenic
Fix errors and add into response impl
f209548
@mmalenic
Sketch traits for ingesting events
fd3ae04
@mmalenic
Remove database dependency from queue struct and return EventMessages…
4c71f32 
… instead
@mmalenic
refactor: add very basic outline of filemanager crates
a74f262
@mmalenic
refactor: move code into multiple crates, add postgres tables
f050939
@mmalenic
refactor: simplify events module so that it is only concerned with th…
d358905 
…e event message struct
@brainstorm brainstorm marked this pull request as ready for review September 25, 2023 05:19
@brainstorm
Copy link
Member Author

brainstorm commented Nov 16, 2023 •
edited
Loading

@mmalenic I was about to merge this gigantic PR but thought about getting your approval first.

@brainstorm
[ci skip] Bump up CDK deps
e5c9ce4
@brainstorm
[ci skip] Getting the following error from the log group README step:
1a3f87d 
% ./deploy/aws-get-filemanager-logs.sh -c awslocal

Parameter validation failed:
Invalid length for parameter logGroupName, value: 0, valid min length: 1
@brainstorm
When following the README as-is, localstack does not have any logGrou…
046d6c2 
…p because no lambda invoke has occurred yet? /cc @mmalenic
mmalenic
mmalenic approved these changes Nov 16, 2023
View reviewed changes
Copy link
Member

@mmalenic mmalenic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, go for it.

In relation to the log groups, yes it fails if no events have been created yet. Maybe this needs a clearer note in the README.

@brainstorm brainstorm merged commit 330f9d4 into main Nov 16, 2023
1 of 2 checks passed
@brainstorm brainstorm deleted the traits branch November 16, 2023 19:23
@brainstorm
Copy link
Member Author

In relation to the log groups, yes it fails if no events have been created yet. Maybe this needs a clearer note in the README.

More than a note: I meant that if a newcomer to the codebase reads the README, it should have the least amount of gotchas. Following the steps in sequence should yield a deterministic result and have a locally deployed dev setup as a result.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mmalenic mmalenic mmalenic approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@brainstorm @mmalenic