Skip to content

Commit

Permalink
[HUDI-6504] [DOCS] Updated roadmap (#9142)
Browse files Browse the repository at this point in the history 
---------

Co-authored-by: Y Ethan Guo <[email protected]>
  • Loading branch information
@nfarah86 @yihua
nfarah86 and yihua authored Jul 7, 2023
1 parent 58677ff commit 6b3dadc
Showing 1 changed file with 40 additions and 33 deletions.
73 changes: 40 additions & 33 deletions website/src/pages/roadmap.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,59 +10,66 @@ down by areas on our [stack](blog/2021/07/21/streaming-data-lake-platform/#hudi-

## Future Releases

Next major release : [0.13.0](https://issues.apache.org/jira/projects/HUDI/versions/12352101) (Jan 2023)
Next major release : [0.14.0](https://issues.apache.org/jira/projects/HUDI/versions/12352700) (July 2023)

| Release | Timeline |
|--------------------------------------------------------------------------|----------|
| [0.12.2](https://issues.apache.org/jira/projects/HUDI/versions/12352249) | Dec 2022 |
| [0.13.0](https://issues.apache.org/jira/projects/HUDI/versions/12352101) | Jan 2023 |
| [1.0.0](https://issues.apache.org/jira/projects/HUDI/versions/12351210) | TBD |
| [0.14.0](https://issues.apache.org/jira/projects/HUDI/versions/12352700) | July 2023 |
| [1.0.0](https://issues.apache.org/jira/projects/HUDI/versions/12351210) | Oct 2023 |

## Transactional Database Layer

| Feature | Target Release | Tracking |
|----------------------------------------------------------------|----------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Metaserver for all metadata | 0.13.0 | [Strawman design](https://docs.google.com/presentation/d/1QBgLw11TM2Qf1KUESofGrQDb63EuggNCpPaxc82Kldo/edit#slide=id.ged084e5bf6_0_278), [RFC-36](https:/apache/hudi/pull/4718), [HUDI-3345](https://issues.apache.org/jira/browse/HUDI-3345) |
| Supervised table service planning and execution | 0.13.0 | [RFC-43](https:/apache/hudi/pull/4309), [HUDI-4147](https://issues.apache.org/jira/browse/HUDI-4147) |
| Support of Change Data Capture (CDC) with Hudi change logs | 0.13.0 | [RFC-51](https://github.com/apache/hudi/blob/master/rfc/rfc-51/rfc-51.md), [HUDI-3478](https://issues.apache.org/jira/browse/HUDI-3478) |
| Record-level index to speed up UUID-based upserts and deletes | 0.13.0 | [RFC-08](https://cwiki.apache.org/confluence/display/HUDI/RFC-08++Record+level+indexing+mechanisms+for+Hudi+datasets), [HUDI-53](https://issues.apache.org/jira/browse/HUDI-53) |
| Consistent hashing index for dynamic buckets | 0.13.0 | [RFC-42](https:/apache/hudi/blob/master/rfc/rfc-42/rfc-42.md), [HUDI-3000](https://issues.apache.org/jira/browse/HUDI-3000) |
| Secondary index to improve query performance | 0.13.0 | [RFC-52](https://github.com/apache/hudi/pull/5370), [HUDI-3907](https://issues.apache.org/jira/browse/HUDI-3907) |
| Reducing write amplification with Log Compaction in MOR | 0.13.0 | [RFC-48](https:/apache/hudi/pull/5041), [HUDI-3580](https://issues.apache.org/jira/browse/HUDI-3580) |
| Eager conflict detection for Optimistic Concurrency Control | 0.13.0 | [RFC-56](https:/apache/hudi/pull/6003), [HUDI-1575](https://issues.apache.org/jira/browse/HUDI-1575) |
| Index Function for Optimizing Query Performance | 0.13.0 | [RFC-63](https:/apache/hudi/pull/7235), [HUDI-512](https://issues.apache.org/jira/browse/HUDI-512) |
| Lock-Free message queue to improve writing efficiency | 0.13.0 | [RFC-53](https:/apache/hudi/blob/master/rfc/rfc-53/rfc-53.md), [HUDI-3963](https://issues.apache.org/jira/browse/HUDI-3963) |
| Efficient bootstrap and migration of existing non-Hudi dataset | 1.0.0 onward | [HUDI-1265](https://issues.apache.org/jira/browse/HUDI-1265) |
| Lock free concurrency control | 1.0.0 onward | [HUDI-3187](https://issues.apache.org/jira/browse/HUDI-3187) |
| Non-blocking/Lock-free updates during clustering | 1.0.0 onward | [HUDI-1042](https://issues.apache.org/jira/browse/HUDI-1042) |
| Time Travel updates, deletes | 1.0.0 onward ||
| General purpose support for multi-table transactions | 1.0.0 onward ||
| Support for primary key-less table | 0.14.0 | [HUDI-4699](https://issues.apache.org/jira/browse/HUDI-4699) |
| Efficient bootstrap and migration of existing non-Hudi dataset | 0.14.0 | [HUDI-1265](https://issues.apache.org/jira/browse/HUDI-1265) |
| Record-level index to speed up UUID-based upserts and deletes | 0.14.0 | [RFC-08](https://cwiki.apache.org/confluence/display/HUDI/RFC-08++Record+level+indexing+mechanisms+for+Hudi+datasets), [HUDI-53](https://issues.apache.org/jira/browse/HUDI-53) |
|1.x Storage format | 1.0.0 | [HUDI-6242](https://issues.apache.org/jira/browse/HUDI-6242) |
| Writer performance improvements | 1.0.0 |[HUDI-3249](https://issues.apache.org/jira/browse/HUDI-3249) |
| Non-blocking concurrency control | 1.0.0 | [HUDI-3187](https://issues.apache.org/jira/browse/HUDI-3187), [HUDI-1042](https://issues.apache.org/jira/browse/HUDI-1042), [RFC-66](https:/apache/hudi/pull/7907) |
| Time Travel updates, deletes | 1.0.0 ||
| General purpose support for multi-table transactions | 1.0.0 ||
| A more effective HoodieMergeHandler for COW table with parquet | 1.0.0 | [RFC-68](https:/apache/hudi/blob/f1afb1bf04abdc94a26d61dc302f36ec2bbeb15b/rfc/rfc-68/rfc-68.md) |
| Secondary indexes to improve query performance | 1.0.0 | [RFC-52](https:/apache/hudi/pull/5370), [HUDI-3907](https://issues.apache.org/jira/browse/HUDI-3907) |
| Index Function for Optimizing Query Performance | 1.0.0 | [RFC-63](https:/apache/hudi/pull/7235), [HUDI-512](https://issues.apache.org/jira/browse/HUDI-512) |
|Logical partitioning via indexing | 1.0.0 | [HUDI-512](https://issues.apache.org/jira/browse/HUDI-512) |
|Streaming CDC/Incremental read improvement | 1.0.0 | [HUDI-2749](https://issues.apache.org/jira/browse/HUDI-2749) |
| Supervised table service planning and execution | 1.1.0 | [RFC-43](https:/apache/hudi/pull/4309), [HUDI-4147](https://issues.apache.org/jira/browse/HUDI-4147) |


## Programming APIs

|Feature| Target Release |Tracking|
|------------|----------------|-----------|
| Redesign and optimization of record payload abstraction | 0.13.0 | [RFC-46](https:/apache/hudi/blob/master/rfc/rfc-46/rfc-46.md), [HUDI-3217](https://issues.apache.org/jira/browse/HUDI-3217) |
| Optimized storage layout for cloud object stores | 0.13.0 | [RFC-60](https:/apache/hudi/pull/5113), [HUDI-3625](https://issues.apache.org/jira/browse/HUDI-3625) |
| Support of verification with multiple event_time fields | 1.0.0 onward | [RFC-59](https:/apache/hudi/pull/6382), [HUDI-4569](https://issues.apache.org/jira/browse/HUDI-4569) |
| MOR snapshot reader | 0.14.0 |[HUDI-6452](https:/apache/hudi/pull/9066) |
| APIs/Abstractions, Record mergers | 1.0.0 | [HUDI-6243](https://issues.apache.org/jira/browse/HUDI-6243), [HUDI-3217](https://issues.apache.org/jira/browse/HUDI-3217)|
| New Hudi Table Format APIs for Query Integrations | 1.0.0 | [RFC-64](https:/apache/hudi/pull/7080), [HUDI-4141](https://issues.apache.org/jira/browse/HUDI-4141) |
| Snapshot view management | 1.0.0 | [RFC-61](https:/apache/hudi/pull/6576), [HUDI-4677](https://issues.apache.org/jira/browse/HUDI-4677) |
| Optimized storage layout for cloud object stores | 1.0.0 | [RFC-60](https:/apache/hudi/pull/5113), [HUDI-3625](https://issues.apache.org/jira/browse/HUDI-3625) |
| Support of verification with multiple event_time fields | 1.0.0 | [RFC-59](https:/apache/hudi/pull/6382), [HUDI-4569](https://issues.apache.org/jira/browse/HUDI-4569) |


## Execution Engine Integration

| Feature | Target Release | Tracking |
|------------------------------------------------------------------------------------------------|--------------|-----------------------------------------------------------------------------------------------------------------|
| Integrate column stats index with all query engines | 0.13.0 | [RFC-58](https:/apache/hudi/pull/6345), [HUDI-4552](https://issues.apache.org/jira/browse/HUDI-4552) |
| Upgrade to Spark 3 as the default profile | 0.13.0 | [HUDI-3431](https://issues.apache.org/jira/browse/HUDI-3431) |
| Spark datasource V2 read | 1.0.0 onward | [HUDI-4449](https://issues.apache.org/jira/browse/HUDI-4449) |
| Materialized Views with incremental updates using Flink | 1.0.0 onward ||
| SQL DML support for Presto/Trino connectors (could be accelerated based on community feedback) | 1.0.0 onward ||
| Explore other execution engines/runtimes (Ray, native Rust, Python) | 1.0.0 onward ||
| Support for Spark 3.4.0 | 0.14.0 | [HUDI-6198](https://issues.apache.org/jira/browse/HUDI-6198) |
| Integrate column stats index with all query engines | 0.14.0 | [RFC-58](https:/apache/hudi/pull/6345), [HUDI-4552](https://issues.apache.org/jira/browse/HUDI-4552) |
| Upgrade to Spark 3 as the default profile | 0.14.0 | [HUDI-3431](https://issues.apache.org/jira/browse/HUDI-3431) |
| Performance for Spark-SQL writes | 0.14.0 | [HUDI-6315](https://issues.apache.org/jira/browse/HUDI-6315), [HUDI-6376](https://issues.apache.org/jira/browse/HUDI-6376) |
| Presto/Trino queries with new format | 1.0.0 |[HUDI-3210](https://issues.apache.org/jira/browse/HUDI-4394), [HUDI-4394](https://issues.apache.org/jira/browse/HUDI-4394), [HUDI-4552](https://issues.apache.org/jira/browse/HUDI-4552) |
| Materialized Views with incremental updates using Flink | 1.0.0 ||
| Explore other execution engines/runtimes (Ray, native Rust, Python) | 1.0.0 ||
| Spark datasource V2 read | 1.1.0 | [HUDI-4449](https://issues.apache.org/jira/browse/HUDI-4449) |


## Platform Services

| Feature | Target Release | Tracking |
|-----------------------------------------------------------------------------------------------------|--------------|-----------------------------------------------------------------------------------------------------------------------------------------|
| Protobuf source for DeltaStreamer | 0.13.0 | [RFC-57](https:/apache/hudi/blob/master/rfc/rfc-57/rfc-57.md), [HUDI-4399](https://issues.apache.org/jira/browse/HUDI-4399) |
| Improving Hudi CLI features and usability | 0.13.0 | [HUDI-1388](https://issues.apache.org/jira/browse/HUDI-1388) |
| Hudi integration with Snowflake | 1.0.0 onward | [RFC-41](https:/apache/hudi/pull/4074), [HUDI-2832](https://issues.apache.org/jira/browse/HUDI-2832) |
| Support for reliable, event based ingestion from cloud stores - GCS, Azure and the others | 1.0.0 onward | [HUDI-1896](https://issues.apache.org/jira/browse/HUDI-1896) |
| Mutable, Transactional caching for Hudi Tables (could be accelerated based on community feedback) | 1.0.0 onward | [Strawman design](https://docs.google.com/presentation/d/1QBgLw11TM2Qf1KUESofGrQDb63EuggNCpPaxc82Kldo/edit#slide=id.gf7e0551254_0_5) |
| Hudi Reverse streamer | 1.0.0 | [RFC-70](https:/apache/hudi/pull/9040) |
| Partition TTL management | 1.1.0 | [RFC-65](https:/apache/hudi/pull/8062) |
| Diagnostic Reporter | 1.1.0 | [RFC-62](https:/apache/hudi/pull/6600) |
| Hudi integration with Snowflake | 1.1.0 | [RFC-41](https:/apache/hudi/pull/4074), [HUDI-2832](https://issues.apache.org/jira/browse/HUDI-2832) |
| Support for reliable, event based ingestion from cloud stores - GCS, Azure and the others | 1.1.0 | [HUDI-1896](https://issues.apache.org/jira/browse/HUDI-1896) |
| Mutable, Transactional caching for Hudi Tables (could be accelerated based on community feedback) | 1.1.0 | [Strawman design](https://docs.google.com/presentation/d/1QBgLw11TM2Qf1KUESofGrQDb63EuggNCpPaxc82Kldo/edit#slide=id.gf7e0551254_0_5), [HUDI-6489](https://issues.apache.org/jira/browse/HUDI-6489) |

0 comments on commit 6b3dadc

Please sign in to comment.