Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[HUDI-6504] [DOCS] Updated roadmap #9142

Merged
merged 2 commits into from
Jul 7, 2023
Merged

[HUDI-6504] [DOCS] Updated roadmap #9142

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
33bcc10
updated roadmap
nfarah86 Jul 7, 2023
e539e6f
Update roadmap.md
yihua Jul 7, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 40 additions & 33 deletions website/src/pages/roadmap.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,59 +10,66 @@ down by areas on our [stack](blog/2021/07/21/streaming-data-lake-platform/#hudi-

## Future Releases

Next major release : [0.13.0](https://issues.apache.org/jira/projects/HUDI/versions/12352101) (Jan 2023)
Next major release : [0.14.0](https://issues.apache.org/jira/projects/HUDI/versions/12352700) (July 2023)

| Release | Timeline |
|--------------------------------------------------------------------------|----------|
| [0.12.2](https://issues.apache.org/jira/projects/HUDI/versions/12352249) | Dec 2022 |
| [0.13.0](https://issues.apache.org/jira/projects/HUDI/versions/12352101) | Jan 2023 |
| [1.0.0](https://issues.apache.org/jira/projects/HUDI/versions/12351210) | TBD |
| [0.14.0](https://issues.apache.org/jira/projects/HUDI/versions/12352700) | July 2023 |
| [1.0.0](https://issues.apache.org/jira/projects/HUDI/versions/12351210) | Oct 2023 |

## Transactional Database Layer

| Feature | Target Release | Tracking |
|----------------------------------------------------------------|----------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Metaserver for all metadata | 0.13.0 | [Strawman design](https://docs.google.com/presentation/d/1QBgLw11TM2Qf1KUESofGrQDb63EuggNCpPaxc82Kldo/edit#slide=id.ged084e5bf6_0_278), [RFC-36](https:/apache/hudi/pull/4718), [HUDI-3345](https://issues.apache.org/jira/browse/HUDI-3345) |
| Supervised table service planning and execution | 0.13.0 | [RFC-43](https:/apache/hudi/pull/4309), [HUDI-4147](https://issues.apache.org/jira/browse/HUDI-4147) |
| Support of Change Data Capture (CDC) with Hudi change logs | 0.13.0 | [RFC-51](https://github.com/apache/hudi/blob/master/rfc/rfc-51/rfc-51.md), [HUDI-3478](https://issues.apache.org/jira/browse/HUDI-3478) |
| Record-level index to speed up UUID-based upserts and deletes | 0.13.0 | [RFC-08](https://cwiki.apache.org/confluence/display/HUDI/RFC-08++Record+level+indexing+mechanisms+for+Hudi+datasets), [HUDI-53](https://issues.apache.org/jira/browse/HUDI-53) |
| Consistent hashing index for dynamic buckets | 0.13.0 | [RFC-42](https:/apache/hudi/blob/master/rfc/rfc-42/rfc-42.md), [HUDI-3000](https://issues.apache.org/jira/browse/HUDI-3000) |
| Secondary index to improve query performance | 0.13.0 | [RFC-52](https://github.com/apache/hudi/pull/5370), [HUDI-3907](https://issues.apache.org/jira/browse/HUDI-3907) |
| Reducing write amplification with Log Compaction in MOR | 0.13.0 | [RFC-48](https:/apache/hudi/pull/5041), [HUDI-3580](https://issues.apache.org/jira/browse/HUDI-3580) |
| Eager conflict detection for Optimistic Concurrency Control | 0.13.0 | [RFC-56](https:/apache/hudi/pull/6003), [HUDI-1575](https://issues.apache.org/jira/browse/HUDI-1575) |
| Index Function for Optimizing Query Performance | 0.13.0 | [RFC-63](https:/apache/hudi/pull/7235), [HUDI-512](https://issues.apache.org/jira/browse/HUDI-512) |
| Lock-Free message queue to improve writing efficiency | 0.13.0 | [RFC-53](https:/apache/hudi/blob/master/rfc/rfc-53/rfc-53.md), [HUDI-3963](https://issues.apache.org/jira/browse/HUDI-3963) |
| Efficient bootstrap and migration of existing non-Hudi dataset | 1.0.0 onward | [HUDI-1265](https://issues.apache.org/jira/browse/HUDI-1265) |
| Lock free concurrency control | 1.0.0 onward | [HUDI-3187](https://issues.apache.org/jira/browse/HUDI-3187) |
| Non-blocking/Lock-free updates during clustering | 1.0.0 onward | [HUDI-1042](https://issues.apache.org/jira/browse/HUDI-1042) |
| Time Travel updates, deletes | 1.0.0 onward ||
| General purpose support for multi-table transactions | 1.0.0 onward ||
| Support for primary key-less table | 0.14.0 | [HUDI-4699](https://issues.apache.org/jira/browse/HUDI-4699) |
| Efficient bootstrap and migration of existing non-Hudi dataset | 0.14.0 | [HUDI-1265](https://issues.apache.org/jira/browse/HUDI-1265) |
| Record-level index to speed up UUID-based upserts and deletes | 0.14.0 | [RFC-08](https://cwiki.apache.org/confluence/display/HUDI/RFC-08++Record+level+indexing+mechanisms+for+Hudi+datasets), [HUDI-53](https://issues.apache.org/jira/browse/HUDI-53) |
|1.x Storage format | 1.0.0 | [HUDI-6242](https://issues.apache.org/jira/browse/HUDI-6242) |
| Writer performance improvements | 1.0.0 |[HUDI-3249](https://issues.apache.org/jira/browse/HUDI-3249) |
| Non-blocking concurrency control | 1.0.0 | [HUDI-3187](https://issues.apache.org/jira/browse/HUDI-3187), [HUDI-1042](https://issues.apache.org/jira/browse/HUDI-1042), [RFC-66](https:/apache/hudi/pull/7907) |
| Time Travel updates, deletes | 1.0.0 ||
| General purpose support for multi-table transactions | 1.0.0 ||
| A more effective HoodieMergeHandler for COW table with parquet | 1.0.0 | [RFC-68](https:/apache/hudi/blob/f1afb1bf04abdc94a26d61dc302f36ec2bbeb15b/rfc/rfc-68/rfc-68.md) |
| Secondary indexes to improve query performance | 1.0.0 | [RFC-52](https:/apache/hudi/pull/5370), [HUDI-3907](https://issues.apache.org/jira/browse/HUDI-3907) |
| Index Function for Optimizing Query Performance | 1.0.0 | [RFC-63](https:/apache/hudi/pull/7235), [HUDI-512](https://issues.apache.org/jira/browse/HUDI-512) |
|Logical partitioning via indexing | 1.0.0 | [HUDI-512](https://issues.apache.org/jira/browse/HUDI-512) |
|Streaming CDC/Incremental read improvement | 1.0.0 | [HUDI-2749](https://issues.apache.org/jira/browse/HUDI-2749) |
| Supervised table service planning and execution | 1.1.0 | [RFC-43](https:/apache/hudi/pull/4309), [HUDI-4147](https://issues.apache.org/jira/browse/HUDI-4147) |


## Programming APIs

|Feature| Target Release |Tracking|
|------------|----------------|-----------|
| Redesign and optimization of record payload abstraction | 0.13.0 | [RFC-46](https:/apache/hudi/blob/master/rfc/rfc-46/rfc-46.md), [HUDI-3217](https://issues.apache.org/jira/browse/HUDI-3217) |
| Optimized storage layout for cloud object stores | 0.13.0 | [RFC-60](https:/apache/hudi/pull/5113), [HUDI-3625](https://issues.apache.org/jira/browse/HUDI-3625) |
| Support of verification with multiple event_time fields | 1.0.0 onward | [RFC-59](https:/apache/hudi/pull/6382), [HUDI-4569](https://issues.apache.org/jira/browse/HUDI-4569) |
| MOR snapshot reader | 0.14.0 |[HUDI-6452](https:/apache/hudi/pull/9066) |
| APIs/Abstractions, Record mergers | 1.0.0 | [HUDI-6243](https://issues.apache.org/jira/browse/HUDI-6243), [HUDI-3217](https://issues.apache.org/jira/browse/HUDI-3217)|
| New Hudi Table Format APIs for Query Integrations | 1.0.0 | [RFC-64](https:/apache/hudi/pull/7080), [HUDI-4141](https://issues.apache.org/jira/browse/HUDI-4141) |
| Snapshot view management | 1.0.0 | [RFC-61](https:/apache/hudi/pull/6576), [HUDI-4677](https://issues.apache.org/jira/browse/HUDI-4677) |
| Optimized storage layout for cloud object stores | 1.0.0 | [RFC-60](https:/apache/hudi/pull/5113), [HUDI-3625](https://issues.apache.org/jira/browse/HUDI-3625) |
| Support of verification with multiple event_time fields | 1.0.0 | [RFC-59](https:/apache/hudi/pull/6382), [HUDI-4569](https://issues.apache.org/jira/browse/HUDI-4569) |


## Execution Engine Integration

| Feature | Target Release | Tracking |
|------------------------------------------------------------------------------------------------|--------------|-----------------------------------------------------------------------------------------------------------------|
| Integrate column stats index with all query engines | 0.13.0 | [RFC-58](https:/apache/hudi/pull/6345), [HUDI-4552](https://issues.apache.org/jira/browse/HUDI-4552) |
| Upgrade to Spark 3 as the default profile | 0.13.0 | [HUDI-3431](https://issues.apache.org/jira/browse/HUDI-3431) |
| Spark datasource V2 read | 1.0.0 onward | [HUDI-4449](https://issues.apache.org/jira/browse/HUDI-4449) |
| Materialized Views with incremental updates using Flink | 1.0.0 onward ||
| SQL DML support for Presto/Trino connectors (could be accelerated based on community feedback) | 1.0.0 onward ||
| Explore other execution engines/runtimes (Ray, native Rust, Python) | 1.0.0 onward ||
| Support for Spark 3.4.0 | 0.14.0 | [HUDI-6198](https://issues.apache.org/jira/browse/HUDI-6198) |
| Integrate column stats index with all query engines | 0.14.0 | [RFC-58](https:/apache/hudi/pull/6345), [HUDI-4552](https://issues.apache.org/jira/browse/HUDI-4552) |
| Upgrade to Spark 3 as the default profile | 0.14.0 | [HUDI-3431](https://issues.apache.org/jira/browse/HUDI-3431) |
| Performance for Spark-SQL writes | 0.14.0 | [HUDI-6315](https://issues.apache.org/jira/browse/HUDI-6315), [HUDI-6376](https://issues.apache.org/jira/browse/HUDI-6376) |
| Presto/Trino queries with new format | 1.0.0 |[HUDI-3210](https://issues.apache.org/jira/browse/HUDI-4394), [HUDI-4394](https://issues.apache.org/jira/browse/HUDI-4394), [HUDI-4552](https://issues.apache.org/jira/browse/HUDI-4552) |
| Materialized Views with incremental updates using Flink | 1.0.0 ||
| Explore other execution engines/runtimes (Ray, native Rust, Python) | 1.0.0 ||
| Spark datasource V2 read | 1.1.0 | [HUDI-4449](https://issues.apache.org/jira/browse/HUDI-4449) |


## Platform Services

| Feature | Target Release | Tracking |
|-----------------------------------------------------------------------------------------------------|--------------|-----------------------------------------------------------------------------------------------------------------------------------------|
| Protobuf source for DeltaStreamer | 0.13.0 | [RFC-57](https:/apache/hudi/blob/master/rfc/rfc-57/rfc-57.md), [HUDI-4399](https://issues.apache.org/jira/browse/HUDI-4399) |
| Improving Hudi CLI features and usability | 0.13.0 | [HUDI-1388](https://issues.apache.org/jira/browse/HUDI-1388) |
| Hudi integration with Snowflake | 1.0.0 onward | [RFC-41](https:/apache/hudi/pull/4074), [HUDI-2832](https://issues.apache.org/jira/browse/HUDI-2832) |
| Support for reliable, event based ingestion from cloud stores - GCS, Azure and the others | 1.0.0 onward | [HUDI-1896](https://issues.apache.org/jira/browse/HUDI-1896) |
| Mutable, Transactional caching for Hudi Tables (could be accelerated based on community feedback) | 1.0.0 onward | [Strawman design](https://docs.google.com/presentation/d/1QBgLw11TM2Qf1KUESofGrQDb63EuggNCpPaxc82Kldo/edit#slide=id.gf7e0551254_0_5) |
| Hudi Reverse streamer | 1.0.0 | [RFC-70](https:/apache/hudi/pull/9040) |
| Partition TTL management | 1.1.0 | [RFC-65](https:/apache/hudi/pull/8062) |
| Diagnostic Reporter | 1.1.0 | [RFC-62](https:/apache/hudi/pull/6600) |
| Hudi integration with Snowflake | 1.1.0 | [RFC-41](https:/apache/hudi/pull/4074), [HUDI-2832](https://issues.apache.org/jira/browse/HUDI-2832) |
| Support for reliable, event based ingestion from cloud stores - GCS, Azure and the others | 1.1.0 | [HUDI-1896](https://issues.apache.org/jira/browse/HUDI-1896) |
| Mutable, Transactional caching for Hudi Tables (could be accelerated based on community feedback) | 1.1.0 | [Strawman design](https://docs.google.com/presentation/d/1QBgLw11TM2Qf1KUESofGrQDb63EuggNCpPaxc82Kldo/edit#slide=id.gf7e0551254_0_5), [HUDI-6489](https://issues.apache.org/jira/browse/HUDI-6489) |