feat(sink): support encode jsonb data as dynamic json type in sink #17693

yuhao-su · 2024-07-16T06:47:11Z

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

close #11699

Checklist

I have written necessary rustdoc comments
I have added necessary unit tests and integration tests
I have added test labels as necessary. See details.
I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
All checks passed in ./risedev check (or alias, ./risedev c)
My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Add a option for ENCODE JSON in WITH clause .

jsonb.handling.mode can be set to either string or dynamic

string: encode the jsonb type as a string. example: {"k": 2} -> "{\"k\": 2}". Here a json object jsonb was encoded to a string.
dynamic: dynamically encode the jsonb type as json type. example: {"k": 2} -> {"k": 2}. Here a json object jsonb was encoded to a json object.

Release note

If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

yuhao-su · 2024-07-16T06:48:12Z

src/connector/src/sink/encoder/json.rs

+ CustomJsonType::None => match config.encode_jsonb_mode {
+ EncodeJsonbMode::String => {
+ json!(jsonb_ref.to_string())


yuhao-su · 2024-07-16T06:48:45Z

src/connector/src/sink/encoder/json.rs

+ let encode_jsonb_obj_config = JsonEncoderConfig {
+ time_handling_mode: TimeHandlingMode::Milli,
+ date_handling_mode: DateHandlingMode::String,
+ timestamp_handling_mode: TimestampHandlingMode::String,
+ timestamptz_handling_mode: TimestamptzHandlingMode::UtcString,


Can we move this refactor to its own PR?

stacked on #17706

xiangjinwu · 2024-07-16T06:50:52Z

src/connector/src/sink/encoder/mod.rs

+/// How the jsonb type is encoded.
+///
+/// - `string`: encode jsonb as string. `[1, true, "foo"] -> "[1, true, \"foo\"]"`
+/// - `object`: encode jsonb as json object. `[1, true, "foo"] -> [1, true, "foo"]`


This is not accurate. [1, true, "foo"] is a json array rather than a json object. This is why the original issue calls it dynamic type: it may be one of json null / boolean / number / string / array / object.

So rename it to dynamic?

Okay. I use object bc the function name is datum_to_json_object()
`

Because the top level json returned by this function is always an object containing multiple fields, corresponding to a table row containing multiple columns.

yuhao-su · 2024-07-16T07:01:34Z

We didn't have timestamptz.handling.mode or encode.jsonb.mode for NATS or snowflake. Should we add those? cc @xiangjinwu

xiangjinwu · 2024-07-16T07:04:58Z

We didn't have timestamptz.handling.mode or encode.jsonb.mode for NATS or snowflake. Should we add those? cc @xiangjinwu

NATS: Yes. But it has not been migrated to the format ... encode ... (...) syntax yet.
Snowflake: I am not aware of this sink. What is the current data type mapping?

yuhao-su · 2024-07-16T19:42:35Z

Snowflake: I am not aware of this sink. What is the current data type mapping?

Currently it uses default behavior that is encode jsonb as string.

It's my understanding users usually create a table with 1 column to store raw data from json file. And create more tables from the source table. It makes more sense to encode RW as dynamic json type. But snowflake user can handle string by PARSE_JSON https://docs.snowflake.com/en/user-guide/tutorials/json-basics-tutorial#prerequisites
cc @chenzl25 @xzhseh

But it has not been migrated to the format ... encode ... (...) syntax yet.

We can do this later.

xzhseh · 2024-07-17T04:23:29Z

Snowflake: I am not aware of this sink. What is the current data type mapping?

Currently it uses default behavior that is encode jsonb as string.

It's my understanding users usually create a table with 1 column to store raw data from json file. And create more tables from the source table. It makes more sense to encode RW as dynamic json type. But snowflake user can handle string by PARSE_JSON https://docs.snowflake.com/en/user-guide/tutorials/json-basics-tutorial#prerequisites cc @chenzl25 @xzhseh

But it has not been migrated to the format ... encode ... (...) syntax yet.

We can do this later.

I don't think there is any type mapping setup for Snowflake sink as for now..

every column is by default handled/interpreted at Snowflake side, and users need to make sure the data types/column names are matched between the two, though for some types there are no precise one-to-one mapping.

xiangjinwu

Rest LGTM. Thank you!

src/connector/src/sink/encoder/json.rs

src/connector/src/sink/encoder/mod.rs

xiangjinwu · 2024-07-17T06:52:44Z

src/connector/src/sink/encoder/json.rs

@@ -628,11 +639,12 @@ mod tests {
 assert_eq!(date_value, json!(719163));

 let from_epoch_config = JsonEncoderConfig {
- time_handling_mode: TimeHandlingMode::Milli,
+ time_handling_mode: TimeHandlingMode::String,


Irrelevant change?

Ah some merge issue. But this test is for date_handling_mode sotime_handling_mode won't matter.

…17693)

yuhao-su added the user-facing-changes Contains changes that are visible to users label Jul 16, 2024

yuhao-su requested review from xiangjinwu and xxhZs July 16, 2024 06:47

github-actions bot added the type/feature label Jul 16, 2024

yuhao-su commented Jul 16, 2024

View reviewed changes

xiangjinwu reviewed Jul 16, 2024

View reviewed changes

yuhao-su added 4 commits July 16, 2024 13:46

refactor

b069613

init

a67df1a

fix test

21a2689

rename object

95a153c

yuhao-su force-pushed the yuhao/sink-jsonb-dynamic branch from f3e7d61 to 95a153c Compare July 16, 2024 19:29

yuhao-su changed the base branch from main to yuhao/json-encoder-config July 16, 2024 19:32

yuhao-su requested a review from xiangjinwu July 16, 2024 19:33

yuhao-su changed the title ~~feat(sink): support encode jsonb data as json object in sink~~ feat(sink): support encode jsonb data as dynamic json type in sink Jul 16, 2024

yuhao-su added 3 commits July 16, 2024 15:18

fix comment

3ede229

improve test

118d103

error msg

f58e94b

xiangjinwu reviewed Jul 17, 2024

View reviewed changes

src/connector/src/sink/encoder/json.rs Outdated Show resolved Hide resolved

src/connector/src/sink/encoder/mod.rs Outdated Show resolved Hide resolved

Base automatically changed from yuhao/json-encoder-config to main July 17, 2024 05:15

yuhao-su added 2 commits July 17, 2024 01:10

improve

a275a29

resolve conflict

ca2e7ab

yuhao-su enabled auto-merge July 17, 2024 06:23

xiangjinwu approved these changes Jul 17, 2024

View reviewed changes

yuhao-su added this pull request to the merge queue Jul 17, 2024

Merged via the queue into main with commit 30fd4d8 Jul 17, 2024
31 of 32 checks passed

yuhao-su deleted the yuhao/sink-jsonb-dynamic branch July 17, 2024 07:22

neverchanje mentioned this pull request Jul 17, 2024

Document: feat(sink): support encode jsonb data as dynamic json type in sink risingwavelabs/risingwave-docs#2385

Closed

yuhao-su added a commit that referenced this pull request Aug 2, 2024

feat(sink): support encode jsonb data as dynamic json type in sink (#…

35488ce

…17693)

yuhao-su mentioned this pull request Aug 2, 2024

feat: cherry pick sink dynamic json type #17902

Merged

9 tasks

ShanlanLi mentioned this pull request Aug 30, 2024

2385 document featsink support encode jsonb data as dynamic json type in sink risingwavelabs/risingwave-docs#2548

Merged

2 tasks

BugenZhao mentioned this pull request Sep 18, 2024

risingwave 2.0.0 risingwavelabs/homebrew-risingwave#44

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(sink): support encode jsonb data as dynamic json type in sink #17693

feat(sink): support encode jsonb data as dynamic json type in sink #17693

yuhao-su commented Jul 16, 2024 •

edited

Loading

yuhao-su Jul 16, 2024

yuhao-su Jul 16, 2024

xiangjinwu Jul 16, 2024

yuhao-su Jul 16, 2024

yuhao-su Jul 16, 2024

xiangjinwu Jul 16, 2024

yuhao-su Jul 16, 2024

xiangjinwu Jul 16, 2024

yuhao-su Jul 16, 2024

xiangjinwu Jul 17, 2024

yuhao-su commented Jul 16, 2024

xiangjinwu commented Jul 16, 2024

yuhao-su commented Jul 16, 2024 •

edited

Loading

xzhseh commented Jul 17, 2024

xiangjinwu left a comment

xiangjinwu Jul 17, 2024

yuhao-su Jul 17, 2024

feat(sink): support encode jsonb data as dynamic json type in sink #17693

feat(sink): support encode jsonb data as dynamic json type in sink #17693

Conversation

yuhao-su commented Jul 16, 2024 • edited Loading

What's changed and what's your intention?

Checklist

Documentation

Release note

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuhao-su commented Jul 16, 2024

xiangjinwu commented Jul 16, 2024

yuhao-su commented Jul 16, 2024 • edited Loading

xzhseh commented Jul 17, 2024

xiangjinwu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuhao-su commented Jul 16, 2024 •

edited

Loading

yuhao-su commented Jul 16, 2024 •

edited

Loading