Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(filesystem source): Define the Filesystem source interface. #335

Merged
merged 15 commits into from
Feb 16, 2022

Conversation

pangzhenzhou
Copy link
Contributor

@pangzhenzhou pangzhenzhou commented Feb 15, 2022

What's changed and what's your intention?

A simple definition of the filesystem source interface, in our system we consider s3 as part of the filesystem. Currently, the filesystem includes two basic interfaces Directory and EntrySubscriber.

  1. Directory defines a limited number of reading operations and the ability to sense changes in the underlying directory files
  2. EntrySubscriber defines read operations on a single file.

Compared to other message sources, Directory is like Topic, Entry is like Partition.

Checklist

  • I have written necessary docs and comments
  • I have added necessary unit tests and integration tests

Refer to a related PR or issue link (optional)

@codecov
Copy link

codecov bot commented Feb 15, 2022

Codecov Report

Merging #335 (44c2f26) into main (fc24f97) will increase coverage by 0.03%.
The diff coverage is 85.61%.

Impacted file tree graph

@@             Coverage Diff              @@
##               main     #335      +/-   ##
============================================
+ Coverage     74.23%   74.27%   +0.03%     
  Complexity     2681     2681              
============================================
  Files           863      864       +1     
  Lines         48830    48991     +161     
  Branches       1591     1591              
============================================
+ Hits          36250    36388     +138     
- Misses        11767    11790      +23     
  Partials        813      813              
Flag Coverage Δ
java 62.00% <ø> (ø)
rust 79.52% <85.61%> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
rust/connector/src/lib.rs 100.00% <ø> (ø)
rust/connector/src/filesystem/file_common.rs 85.61% <85.61%> (ø)
rust/meta/src/storage/metastore.rs 94.48% <0.00%> (-0.66%) ⬇️
rust/storage/src/store.rs 15.00% <0.00%> (-0.52%) ⬇️
rust/meta/src/storage/sled_metastore.rs 96.77% <0.00%> (-0.11%) ⬇️
rust/stream/src/task/stream_manager.rs 52.83% <0.00%> (ø)
...st/stream/src/executor/aggregation/agg_executor.rs 84.96% <0.00%> (+0.75%) ⬆️
rust/stream/src/executor/local_simple_agg.rs 95.49% <0.00%> (+1.11%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fc24f97...44c2f26. Read the comment docs.

rust/connector/Cargo.toml Outdated Show resolved Hide resolved
rust/connector/src/filesystem/file_common.rs Outdated Show resolved Hide resolved
rust/connector/src/filesystem/file_common.rs Show resolved Hide resolved
Copy link
Contributor

@tabVersion tabVersion left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Signed-off-by: tabVersion <[email protected]>
@pangzhenzhou pangzhenzhou merged commit 338c6bb into main Feb 16, 2022
@pangzhenzhou pangzhenzhou deleted the zhenzhou/s3_source branch February 16, 2022 09:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants