feat(steam): support stream change log #17132

xxhZs · 2024-06-05T10:30:20Z

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

add an changed log exec
use cte to create changed log

Checklist

I have written necessary rustdoc comments
I have added necessary unit tests and integration tests
I have added test labels as necessary. See details.
I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
All checks passed in ./risedev check (or alias, ./risedev c)
My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

in this pr, we support the stream changelog. It can send the 'op' on the stream downstream as a separate column

After this operation, we will discard the upstream pk and recreate a self-incrementing id _change_log_row_id. And the table upstream of changelog cannot have the same name as changelog_op.
Syntactically, we use a cte-like syntax
example

CREATE TABLE t1(v1 int, v2 int);
CREATE MATERIALIZED VIEW mv1 AS WITH sub AS changelog from t1 SELECT * FROM sub;
INSERT INTO t1 VALUES(1,1),(2,2);
UPDATE t1 SET v2 = 100 WHERE v1 = 1;
DELETE FROM t1 WHERE v1 = 2;
SELECT * FROM mv1;

the result is

v1 v2 changelog_op
1 1 1
2 2 1
2 2 4
2 100 3
2 2 2

save

save save fi fmt support

fix

fix ci fmt fix ci

src/frontend/src/optimizer/plan_node/logical_changed_log.rs

src/frontend/src/optimizer/plan_node/generic/changed_log.rs

chenzl25 · 2024-06-06T06:34:18Z

src/frontend/src/optimizer/plan_node/logical_changed_log.rs

+ input: PlanRef,
+ input_col_change: ColIndexMapping,
+ ) -> (Self, ColIndexMapping) {
+ let changed_log = Self::new(input, self.core.need_op, true);


Let's move this logic to logical_rewrite_for_stream, because it has done more things than rewrite_with_input should contain.

e2e_test/streaming/changed_log.slt

hzxa21 · 2024-06-13T07:29:06Z

proto/stream_plan.proto

@@ -287,6 +287,10 @@ message FilterNode {
 expr.ExprNode search_condition = 1;
 }

+message ChangedLogNode {
+ bool need_op = 1;


I think this is used for projection push-down to prune out op column if it is not used in downstream. Can we have some documentation to explain in which case need_op is false?

Can we also add documentation here in the pb in addition to the rust codes as well?

src/frontend/src/binder/query.rs

src/sqlparser/src/ast/query.rs

src/frontend/src/binder/query.rs

src/sqlparser/tests/sqlparser_common.rs

src/stream/src/executor/changed_log.rs

hzxa21 · 2024-06-13T08:14:07Z

src/tests/sqlsmith/src/sql_gen/query.rs

@@ -133,7 +133,7 @@ impl<'a, R: Rng> SqlGenerator<'a, R> {
 let from = None;
 let cte = Cte {
 alias: alias.clone(),
- query,
+ query: Some(query),


How hard it is to support changelog cte in sqlsmith as well?

will do it in next pr

hzxa21 · 2024-06-13T08:44:34Z

I did a simple test with this PR:

dev=> create table t (v1 int primary key, v2 int);
CREATE_TABLE
dev=> create materialized view mv as select * from t;
CREATE_MATERIALIZED_VIEW
dev=> create materialized view changelog as with c as changedlog from mv select * from c;
CREATE_MATERIALIZED_VIEW

dev=> describe changelog;
        Name         |          Type           | Is Hidden | Description 
---------------------+-------------------------+-----------+-------------
 v1                  | integer                 | false     | 
 v2                  | integer                 | false     | 
 op                  | smallint                | false     | 
 _changed_log_row_id | serial                  | true      | 
 primary key         | v1, _changed_log_row_id |           | 
 distribution key    | _changed_log_row_id     |           | 
 table description   | changelog               |           | 
(7 rows)

Is it expected that the pk of changelog mv is v1, _changed_log_row_id instead of just _changed_log_row_id?

xxhZs · 2024-06-14T10:24:15Z

I did a simple test with this PR:

dev=> create table t (v1 int primary key, v2 int);
CREATE_TABLE
dev=> create materialized view mv as select * from t;
CREATE_MATERIALIZED_VIEW
dev=> create materialized view changelog as with c as changedlog from mv select * from c;
CREATE_MATERIALIZED_VIEW

dev=> describe changelog;
        Name         |          Type           | Is Hidden | Description 
---------------------+-------------------------+-----------+-------------
 v1                  | integer                 | false     | 
 v2                  | integer                 | false     | 
 op                  | smallint                | false     | 
 _changed_log_row_id | serial                  | true      | 
 primary key         | v1, _changed_log_row_id |           | 
 distribution key    | _changed_log_row_id     |           | 
 table description   | changelog               |           | 
(7 rows)

Is it expected that the pk of changelog mv is v1, _changed_log_row_id instead of just _changed_log_row_id?

fixed ,now pk is only _changed_log_row_id

hzxa21 · 2024-06-17T06:08:21Z

src/frontend/src/optimizer/plan_node/generic/change_log.rs

+ let keys = vec![self.schema().len() - 1];
+ Some(keys)
+ } else {
+ None


IIUC, change_log's stream key must be not None and must be its row_id. In which case will we reach this branch?

hzxa21 · 2024-06-17T06:30:09Z

Tried with another example:

dev=> create table t (v1 int primary key, v2 int);
CREATE_TABLE
dev=> create materialized view mv as select * from t;
CREATE_MATERIALIZED_VIEW
dev=> create materialized view changelog as with c as changelog from mv select v2 from c;
CREATE_MATERIALIZED_VIEW
dev=> describe changelog
dev-> ;
        Name        |        Type        | Is Hidden | Description 
--------------------+--------------------+-----------+-------------
 v2                 | integer            | false     | 
 mv.v1              | integer            | true      | 
 _change_log_row_id | serial             | true      | 
 primary key        | _change_log_row_id |           | 
 distribution key   | _change_log_row_id |           | 
 table description  | changelog          |           | 
(6 rows)

Is it expected to include mv.v1 here?

hzxa21 · 2024-06-17T06:40:40Z

Another failing example:

dev=> create table t (v1 int primary key, v2 int);
CREATE_TABLE
dev=> create materialized view mv as select * from t;
CREATE_MATERIALIZED_VIEW
dev=> create materialized view changelog as with c as changelog from mv select v2, op from c;
CREATE_MATERIALIZED_VIEW
dev=>  create materialized view changelog3 as with c as changelog from changelog select * from c;
ERROR:  Failed to run the query

Caused by:
  Invalid input syntax: column "op" specified more than once

chaining changelog on top of another changelog MV

fix ci

src/frontend/src/optimizer/plan_node/logical_change_log.rs

wenym1 · 2024-06-26T07:07:42Z

src/stream/src/executor/change_log.rs

+ };
+ yield Message::Chunk(new_chunk);
+ }
+ m => yield m,


For watermark message, should we also convert it to a new kind of row, or just discard the watermark? The semantic of watermark will be broken if we just simply yield the original watermark.

gitguardian · 2024-06-26T09:10:56Z

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secret in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
9425213	Triggered	Generic Password	`0b21756`	ci/scripts/e2e-sink-test.sh	View secret

🛠 Guidelines to remediate hardcoded secrets

Understand the implications of revoking this secret by investigating where it is used in your code.
Replace and store your secret safely. Learn here the best practices.
Revoke and rotate this secret.
If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

following these best practices for managing and storing secrets including API keys and other credentials
install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}

hzxa21

Rest LGTM. Please update the Release Note as well.

hzxa21 · 2024-06-25T09:51:40Z

proto/stream_plan.proto

@@ -287,6 +287,10 @@ message FilterNode {
 expr.ExprNode search_condition = 1;
 }

+message ChangedLogNode {
+ bool need_op = 1;


Can we also add documentation here in the pb in addition to the rust codes as well?

… (#18240)

xxhZs added 4 commits May 28, 2024 14:17

save

de9b71c

save

save

c17f5b5

save

03d1383

save save fi fmt support

add test

759b4c9

github-actions bot added type/feature ci/run-e2e-single-node-tests labels Jun 5, 2024

rename

5195a4e

fix

xxhZs force-pushed the xxh/stream-subscription branch from 33d9a2e to 5195a4e Compare June 5, 2024 10:49

xxhZs marked this pull request as ready for review June 5, 2024 10:49

xxhZs requested review from chenzl25 and hzxa21 June 5, 2024 10:49

xxhZs force-pushed the xxh/stream-subscription branch 2 times, most recently from ba7bc10 to d940439 Compare June 6, 2024 03:29

fix ci

10a7707

fix ci fmt fix ci

xxhZs force-pushed the xxh/stream-subscription branch from d940439 to 10a7707 Compare June 6, 2024 03:53

Merge branch 'main' into xxh/stream-subscription

76e205c

chenzl25 reviewed Jun 6, 2024

View reviewed changes

src/frontend/src/optimizer/plan_node/logical_changed_log.rs Outdated Show resolved Hide resolved

src/frontend/src/optimizer/plan_node/generic/changed_log.rs Outdated Show resolved Hide resolved

chenzl25 reviewed Jun 6, 2024

View reviewed changes

src/frontend/src/optimizer/plan_node/generic/changed_log.rs Outdated Show resolved Hide resolved

chenzl25 reviewed Jun 6, 2024

View reviewed changes

fix comm

f4df317

hzxa21 added the user-facing-changes Contains changes that are visible to users label Jun 12, 2024

hzxa21 reviewed Jun 13, 2024

View reviewed changes

xxhZs added 2 commits June 14, 2024 16:07

fix comm

2a1d9bf

fmt

5dd77bf

hzxa21 reviewed Jun 17, 2024

View reviewed changes

hzxa21 mentioned this pull request Jun 18, 2024

feat(sink): Support Deletes for ClickHouse ReplacingMergeTree #17283

Merged

9 tasks

xxhZs added 2 commits June 18, 2024 15:48

fix comm

027adea

fmt

ab6422a

fix ci

xxhZs force-pushed the xxh/stream-subscription branch from 0e6ee46 to ab6422a Compare June 19, 2024 05:03

chenzl25 reviewed Jun 20, 2024

View reviewed changes

src/frontend/src/optimizer/plan_node/logical_change_log.rs Outdated Show resolved Hide resolved

fix com

267e6c9

xxhZs mentioned this pull request Jun 21, 2024

feat(sink): minor improvements for starrocks sink #17321

Closed

9 tasks

wenym1 reviewed Jun 26, 2024

View reviewed changes

xxhZs requested a review from a team as a code owner June 26, 2024 08:59

Merge branch 'main' into xxh/stream-subscription

0b21756

xxhZs force-pushed the xxh/stream-subscription branch from bbba5e0 to 0b21756 Compare June 26, 2024 09:10

rename

056daf7

hzxa21 approved these changes Jun 26, 2024

View reviewed changes

watermark

7c8d8dc

xxhZs added this pull request to the merge queue Jun 26, 2024

Merged via the queue into main with commit 61e9e52 Jun 26, 2024
39 of 41 checks passed

xxhZs deleted the xxh/stream-subscription branch June 26, 2024 11:53

neverchanje mentioned this pull request Jun 26, 2024

Document the AS CHANGELOG clause risingwavelabs/risingwave-docs#2313

Closed

hzxa21 changed the title ~~feat(steam): support stream changed log~~ feat(steam): support stream change log Aug 26, 2024

xxhZs added a commit that referenced this pull request Aug 26, 2024

feat(steam): support stream changed log (#17132)

8b31283

xxhZs mentioned this pull request Aug 26, 2024

cherry-pick fix: add stream change log to 1.10 (#17583)(#17515)(#17132) #18240

Merged

9 tasks

tabVersion pushed a commit that referenced this pull request Aug 26, 2024

cherry-pick fix: add stream change log to 1.10 (#17583)(#17515)(#17132)…

a4cc5ea

… (#18240)

BugenZhao mentioned this pull request Sep 18, 2024

risingwave 2.0.0 risingwavelabs/homebrew-risingwave#44

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(steam): support stream change log #17132

feat(steam): support stream change log #17132

xxhZs commented Jun 5, 2024 •

edited by hzxa21

Loading

chenzl25 Jun 6, 2024 •

edited

Loading

hzxa21 Jun 13, 2024

hzxa21 Jun 25, 2024

hzxa21 Jun 13, 2024

xxhZs Jun 14, 2024

hzxa21 commented Jun 13, 2024

xxhZs commented Jun 14, 2024

hzxa21 Jun 17, 2024

hzxa21 commented Jun 17, 2024

hzxa21 commented Jun 17, 2024

wenym1 Jun 26, 2024

gitguardian bot commented Jun 26, 2024 •

edited

Loading

hzxa21 left a comment

hzxa21 Jun 25, 2024

feat(steam): support stream change log #17132

feat(steam): support stream change log #17132

Conversation

xxhZs commented Jun 5, 2024 • edited by hzxa21 Loading

What's changed and what's your intention?

Checklist

Documentation

Release note

chenzl25 Jun 6, 2024 • edited Loading

Choose a reason for hiding this comment

hzxa21 Jun 13, 2024

Choose a reason for hiding this comment

hzxa21 Jun 25, 2024

Choose a reason for hiding this comment

hzxa21 Jun 13, 2024

Choose a reason for hiding this comment

xxhZs Jun 14, 2024

Choose a reason for hiding this comment

hzxa21 commented Jun 13, 2024

xxhZs commented Jun 14, 2024

hzxa21 Jun 17, 2024

Choose a reason for hiding this comment

hzxa21 commented Jun 17, 2024

hzxa21 commented Jun 17, 2024

wenym1 Jun 26, 2024

Choose a reason for hiding this comment

gitguardian bot commented Jun 26, 2024 • edited Loading

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

hzxa21 left a comment

Choose a reason for hiding this comment

hzxa21 Jun 25, 2024

Choose a reason for hiding this comment

xxhZs commented Jun 5, 2024 •

edited by hzxa21

Loading

chenzl25 Jun 6, 2024 •

edited

Loading

gitguardian bot commented Jun 26, 2024 •

edited

Loading