feat(frontend): add virtual StreamTableScan node #1024

skyzh · 2022-03-17T08:00:07Z

Signed-off-by: Alex Chi [email protected]

What's changed and what's your intention?

The StreamScan will be converted into chain node and its two inputs when to_prost.

Checklist

I have written necessary docs and comments
I have added necessary unit tests and integration tests

Refer to a related PR or issue link (optional)

Signed-off-by: Alex Chi <[email protected]>

rust/frontend/test_runner/tests/testdata/basic_query.yaml

BugenZhao · 2022-03-17T08:03:48Z

rust/frontend/src/optimizer/plan_node/stream_scan.rs

+/// `StreamScan` is a virtual plan node to represent a stream scan. It will be converted to chain +
+/// upstream table scan + batch table scan when converting to MView creation request.


Just FYI, the source of a streaming plan might be either Source or Chain, depending on whether the object to scan is a Source or MaterializedView.

... So we need both SourceScan node and TableScan node?

Under the design of materialized sources, there will all be tables (mviews)

Well, I don't think this design is really settled down. I'll separate StreamScan to two nodes: SourceScan and TableScan...

Signed-off-by: Alex Chi <[email protected]>

fuyufjh · 2022-03-17T08:09:10Z

rust/frontend/test_runner/tests/testdata/basic_query.yaml

@@ -69,7 +69,7 @@
 BatchScan { table: "t", columns: ["v1"] }
 stream_plan: |
 StreamProject { exprs: [$0], expr_alias: [Some("v1")] }
- StreamTableSource { logical: LogicalScan { table: "t", columns: ["v1"] } }
+ StreamScan { logical: "LogicalScan { table: \"t\", columns: [\"v1\"] }" }


Emmm, still strange to me... Can we keep the original format 😇

Oh, I know the problem... Please don't use logical, but display the fields you want.

e.g.

impl fmt::Display for BatchSort { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { f.debug_struct("BatchSort") .field("order", self.order()) .finish() } }

That's true! I was thinking of #1016 previously, and I found that we have to stick to our own println instead of using utilities provided by std::fmt.

Signed-off-by: Alex Chi <[email protected]>

fuyufjh

The Display format is still strange... I'll fix that later. LGTM for the rest

codecov · 2022-03-17T08:35:59Z

Codecov Report

Merging #1024 (cfd852d) into main (0f23d89) will increase coverage by 0.08%.
The diff coverage is 47.61%.

@@             Coverage Diff              @@
##               main    #1024      +/-   ##
============================================
+ Coverage     71.05%   71.14%   +0.08%     
  Complexity     2766     2766              
============================================
  Files           977      978       +1     
  Lines         57587    57702     +115     
  Branches       1790     1790              
============================================
+ Hits          40921    41053     +132     
+ Misses        15775    15758      -17     
  Partials        891      891

Flag	Coverage Δ
java	`61.03% <ø> (ø)`
rust	`74.97% <47.61%> (+0.10%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
rust/frontend/src/handler/create_mv.rs	`0.00% <0.00%> (ø)`
rust/frontend/src/optimizer/plan_node/mod.rs	`40.35% <0.00%> (-2.25%)`	⬇️
...tend/src/optimizer/plan_node/stream_source_scan.rs	`0.00% <0.00%> (ø)`
...ntend/src/optimizer/plan_node/stream_table_scan.rs	`75.00% <75.00%> (ø)`
...t/frontend/src/optimizer/plan_node/logical_scan.rs	`100.00% <100.00%> (ø)`
rust/meta/src/hummock/hummock_manager_tests.rs	`93.57% <0.00%> (-1.53%)`	⬇️
rust/storage/src/hummock/error.rs	`18.18% <0.00%> (-0.87%)`	⬇️
rust/storage/src/hummock/local_version_manager.rs	`77.45% <0.00%> (-0.55%)`	⬇️
...rage/src/hummock/mock/mock_hummock_meta_service.rs	`70.68% <0.00%> (ø)`
... and 7 more

📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more

BugenZhao · 2022-03-17T09:49:08Z

rust/frontend/src/optimizer/plan_node/stream_source_scan.rs

 fn to_stream_prost_body(&self) -> ProstStreamNode {
- // TODO: support real serialization
- ProstStreamNode::SourceNode(Default::default())
+ ProstStreamNode::MergeNode(Default::default())


Should be SourceNode?

BugenZhao · 2022-03-17T09:49:59Z

rust/frontend/src/optimizer/plan_node/logical_scan.rs

- StreamTableSource::new(self.clone()).into()
+ StreamTableScan::new(self.clone()).into()


We should dispatch to different physical plan according to whether the "table" is mview or source.
Perhaps we should do this in binder?

feat(frontend): add virtual StreamScan node

28929e0

Signed-off-by: Alex Chi <[email protected]>

skyzh requested review from fuyufjh and st1page March 17, 2022 08:00

github-actions bot added the type/feature label Mar 17, 2022

fuyufjh reviewed Mar 17, 2022

View reviewed changes

rust/frontend/test_runner/tests/testdata/basic_query.yaml Outdated Show resolved Hide resolved

BugenZhao reviewed Mar 17, 2022

View reviewed changes

fix display

55193dc

Signed-off-by: Alex Chi <[email protected]>

skyzh requested a review from fuyufjh March 17, 2022 08:07

fuyufjh reviewed Mar 17, 2022

View reviewed changes

add source scan

cfd852d

Signed-off-by: Alex Chi <[email protected]>

skyzh requested a review from fuyufjh March 17, 2022 08:17

skyzh changed the title ~~feat(frontend): add virtual StreamScan node~~ feat(frontend): add virtual StreamTableScan node Mar 17, 2022

fuyufjh approved these changes Mar 17, 2022

View reviewed changes

skyzh merged commit 20f0a43 into main Mar 17, 2022

skyzh deleted the skyzh/virtual-stream-scan branch March 17, 2022 08:38

skyzh mentioned this pull request Mar 17, 2022

frontend: implement ToStreamProst for stream plan nodes #961

Closed

8 tasks

BugenZhao reviewed Mar 17, 2022

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(frontend): add virtual StreamTableScan node #1024

feat(frontend): add virtual StreamTableScan node #1024

skyzh commented Mar 17, 2022 •

edited

Loading

BugenZhao Mar 17, 2022

skyzh Mar 17, 2022

fuyufjh Mar 17, 2022

skyzh Mar 17, 2022

fuyufjh Mar 17, 2022

fuyufjh Mar 17, 2022 •

edited

Loading

skyzh Mar 17, 2022 •

edited

Loading

fuyufjh left a comment

codecov bot commented Mar 17, 2022

BugenZhao Mar 17, 2022

BugenZhao Mar 17, 2022

		/// `StreamScan` is a virtual plan node to represent a stream scan. It will be converted to chain +
		/// upstream table scan + batch table scan when converting to MView creation request.

		StreamTableSource::new(self.clone()).into()
		StreamTableScan::new(self.clone()).into()

feat(frontend): add virtual StreamTableScan node #1024

feat(frontend): add virtual StreamTableScan node #1024

Conversation

skyzh commented Mar 17, 2022 • edited Loading

What's changed and what's your intention?

Checklist

Refer to a related PR or issue link (optional)

BugenZhao Mar 17, 2022

Choose a reason for hiding this comment

skyzh Mar 17, 2022

Choose a reason for hiding this comment

fuyufjh Mar 17, 2022

Choose a reason for hiding this comment

skyzh Mar 17, 2022

Choose a reason for hiding this comment

fuyufjh Mar 17, 2022

Choose a reason for hiding this comment

fuyufjh Mar 17, 2022 • edited Loading

Choose a reason for hiding this comment

skyzh Mar 17, 2022 • edited Loading

Choose a reason for hiding this comment

fuyufjh left a comment

Choose a reason for hiding this comment

codecov bot commented Mar 17, 2022

Codecov Report

BugenZhao Mar 17, 2022

Choose a reason for hiding this comment

BugenZhao Mar 17, 2022

Choose a reason for hiding this comment

skyzh commented Mar 17, 2022 •

edited

Loading

fuyufjh Mar 17, 2022 •

edited

Loading

skyzh Mar 17, 2022 •

edited

Loading