-
Notifications
You must be signed in to change notification settings - Fork 570
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(sink): support doris sink #12336
Conversation
add doris_connector add support http sink doris
modfify jdbc m
389166a
to
e1c8834
Compare
c9c53b3
to
72cb0fb
Compare
72cb0fb
to
205f4bd
Compare
.write(row_json_string.into()) | ||
.await?; | ||
} | ||
Op::UpdateDelete => {} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it safe to ignore update delete here? Does doris support upsert semantic?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, we will only support upsert in Doris when UNIQUE_KEYS are present. In such cases, insert will also be supported for overwriting.
Codecov Report
@@ Coverage Diff @@
## main #12336 +/- ##
==========================================
- Coverage 69.71% 69.57% -0.15%
==========================================
Files 1427 1429 +2
Lines 237449 238100 +651
==========================================
+ Hits 165531 165651 +120
- Misses 71918 72449 +531
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 7 files with indirect coverage changes 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In my local testing, RW timestamp
value will be written as null in doris. Please fix this issue.
Rest LGTM! Great job!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also, please help to write a simple release note about doris sink by describing the SQL syntax, the WITH options and some extra notes (e.g. the type mapping and the unsupported types). You can refer to this PR's description as an example.
modify
3dd8b4a
to
b571f08
Compare
I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
We support doris sink with https://doris.apache.org/docs/dev/data-operate/import/import-way/stream-load-manual.
Currently, it is not possible to automatically align the element names and types in Doris, so users need to ensure that the struct in both the RW (Read-Write) and Doris are identical. If they are not identical, it may result in a failed insert into Doris (i.e., this row's struct will be null or set to default values)
Checklist
./risedev check
(or alias,./risedev c
)Documentation
Release note
SQL Example:
append-only:
upsert:
WITH options:
connector
: required. We usedoris
url
: required. The http connection port (not the mysql connection port) for the fe of doris.username
: optional. User name for the dors icredential. Needs to be created in doris.password
: optional. Password for the doris credential. Needs to be created in doris.database
: optional. Database for the doris.table
: optional. Table for the doris.Notes:
struct
in Doris have the samename
andtype
as those in thestruct
in RW. Otherwise, it may lead to an inability to insert thisstruct
.UNIQUE KEY