feat(Sink): support redis sink #11999
Conversation
support redis sink fmt
fuyufjh
requested review from
fuyufjh,
tabVersion,
xiangjinwu,
st1page and
wenym1
Codecov Report
@@ Coverage Diff @@
## main #11999 +/- ##
==========================================
+ Coverage 68.75% 68.84% +0.09%
==========================================
Files 1495 1496 +1
Lines 250189 250508 +319
==========================================
+ Hits 172015 172465 +450
+ Misses 78174 78043 -131
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 23 files with indirect coverage changes 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
|
related to #11995 ? Should it be |
It is indeed a new encoding but we may need a better name than
Is the default really useful? Maybe we shall require the user to provide templates when using this new encoding, or design the default carefully - for example, the value itself may contain Talking about implementation, As for the issue #11995, I am still investigating a good way to refactor them. Right now we can focus on certain use cases first, for example kafka/kinesis coupled to json and redis coupled to this new encoding. |
| type K = String; | ||
| type V = Vec<u8>; |
There was a problem hiding this comment.
IIRC, ToRedisArg accepts bytes for both key and value. any reason for limiting key to string?
There was a problem hiding this comment.
Inserting k v into redis in string format is only supported here.
src/connector/src/sink/redis.rs
Outdated
| } | ||
|
|
||
| fn check_string_format(format: &str, set: &HashSet<String>) -> Result<()> { | ||
| let re = Regex::new(r"\{([^}]*)\}").unwrap(); |
There was a problem hiding this comment.
will it handle format containing escaped string? such as '{\"a\": \{{col_a}\}}'
There was a problem hiding this comment.
Not yet, escaped characters are treated as strings, and {} can only have column names in it, not other characters.
| @@ -107,6 +109,70 @@ impl SinkFormatterImpl { | |||
| ))) | |||
| } | |||
| } | |||
|
|
|||
| pub fn new_with_redis( | |||
There was a problem hiding this comment.
The syntax for create redis sink with key value template format may follow the new create sink ... format XXX encode XXX syntax.
If this PR is urgent, we can move the template row encoder to a separate PR, and in this PR we only use the json encoder for both key and value.
There was a problem hiding this comment.
Yes, I will create a new pull request to implement the new syntax. I think we can initially implement it in the current way, and then create a new pull request for modifications.
I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
Support redis sink. Data in RW will be saved in Redis in the form of string key-value pairs. By default, our key is the primary key, and the value includes all other values. Users can also input their desired format in string format. Therefore, even in the case of 'append only', we still need to provide the primary key.
Please refer to the README for specific instructions.
Checklist
./risedev check(or alias,./risedev c)Documentation
Release note
If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.