[HUDI-6504] [DOCS] Updated roadmap #9142

nfarah86 · 2023-07-07T14:55:11Z

Change Logs

updated the roadmap for 0.14.0 and 1.x.

nsivabalan · 2023-07-07T14:57:55Z

website/src/pages/roadmap.md


 | Release | Timeline |
 |--------------------------------------------------------------------------|----------|
-| [0.12.2](https://issues.apache.org/jira/projects/HUDI/versions/12352249) | Dec 2022 |
-| [0.13.0](https://issues.apache.org/jira/projects/HUDI/versions/12352101) | Jan 2023 |
+| [0.14.0](https://issues.apache.org/jira/projects/HUDI/versions/12352700) | July 2023 |
 | [1.0.0](https://issues.apache.org/jira/projects/HUDI/versions/12351210) | TBD |


lets make Oct as timeline for 1..x

nsivabalan · 2023-07-07T14:58:30Z

website/src/pages/roadmap.md

-| Eager conflict detection for Optimistic Concurrency Control | 0.13.0 | [RFC-56](https:/apache/hudi/pull/6003), [HUDI-1575](https://issues.apache.org/jira/browse/HUDI-1575) |
-| Index Function for Optimizing Query Performance | 0.13.0 | [RFC-63](https:/apache/hudi/pull/7235), [HUDI-512](https://issues.apache.org/jira/browse/HUDI-512) |
+| Support for primary key-less table | 0.14.0 | [HUDI-4699](https://issues.apache.org/jira/browse/HUDI-4699) |
+| Metaserver for all metadata | 1.0.0 | [Strawman design](https://docs.google.com/presentation/d/1QBgLw11TM2Qf1KUESofGrQDb63EuggNCpPaxc82Kldo/edit#slide=id.ged084e5bf6_0_278), [RFC-36](https:/apache/hudi/pull/4718), [HUDI-3345](https://issues.apache.org/jira/browse/HUDI-3345) |


move all of 0.14.0 to the beginning under each section

Major part of metaserver is already implemented and released in 0.13.0. We should remove it.

fixed @yihua

nsivabalan · 2023-07-07T14:59:59Z

website/src/pages/roadmap.md

-| Support of verification with multiple event_time fields | 1.0.0 onward | [RFC-59](https:/apache/hudi/pull/6382), [HUDI-4569](https://issues.apache.org/jira/browse/HUDI-4569) |
+| New Hudi Table Spec API for Query Integrations | 1.0.0 | [RFC-64](https:/apache/hudi/pull/7080) |
+| Snapshot view management | 1.0.0 | [RFC-61](https:/apache/hudi/pull/6576), [HUDI-4678](https://issues.apache.org/jira/browse/HUDI-4678) |
+| Engine specific record representation for better performance | 1.0.0 | [RFC-46](https:/apache/hudi/blob/master/rfc/rfc-46/rfc-46.md), [HUDI-3217](https://issues.apache.org/jira/browse/HUDI-3217) |


sorry, I only gave this title.
may be we can consider this as well
"Optimizing performance w/ engine based record representation"

This is released in 0.13.0.

nsivabalan · 2023-07-07T15:01:21Z

website/src/pages/roadmap.md

-| Redesign and optimization of record payload abstraction | 0.13.0 | [RFC-46](https:/apache/hudi/blob/master/rfc/rfc-46/rfc-46.md), [HUDI-3217](https://issues.apache.org/jira/browse/HUDI-3217) |
-| Optimized storage layout for cloud object stores | 0.13.0 | [RFC-60](https:/apache/hudi/pull/5113), [HUDI-3625](https://issues.apache.org/jira/browse/HUDI-3625) |
-| Support of verification with multiple event_time fields | 1.0.0 onward | [RFC-59](https:/apache/hudi/pull/6382), [HUDI-4569](https://issues.apache.org/jira/browse/HUDI-4569) |
+| New Hudi Table Spec API for Query Integrations | 1.0.0 | [RFC-64](https:/apache/hudi/pull/7080) |


looks like we don't have anything on this title for 0.14.0.
@yihua @codope @xushiyan : can we think of something under this title for 014.0 ?

We landed this in 0.14.0: MOR snapshot reader to integrate with query engines without using Hadoop APIs #9066 . Is it worth mentioning?

nsivabalan · 2023-07-07T15:02:09Z

website/src/pages/roadmap.md

+| Mutable, Transactional caching for Hudi Tables (could be accelerated based on community feedback) | 1.0.0 onward | [Strawman design](https://docs.google.com/presentation/d/1QBgLw11TM2Qf1KUESofGrQDb63EuggNCpPaxc82Kldo/edit#slide=id.gf7e0551254_0_5) |
+| Hudi Reverse streamer | 1.0.0 onward | [RFC-70](https:/apache/hudi/pull/9040) | 


We also discussed to include perf improvement for spark-sql writes as a line item. can we add that in

it's under execution engine- should i move it elsewhere?

Yes, under execution engine.

yihua · 2023-07-07T15:09:14Z

website/src/pages/roadmap.md

-| Eager conflict detection for Optimistic Concurrency Control | 0.13.0 | [RFC-56](https:/apache/hudi/pull/6003), [HUDI-1575](https://issues.apache.org/jira/browse/HUDI-1575) |
-| Index Function for Optimizing Query Performance | 0.13.0 | [RFC-63](https:/apache/hudi/pull/7235), [HUDI-512](https://issues.apache.org/jira/browse/HUDI-512) |
+| Support for primary key-less table | 0.14.0 | [HUDI-4699](https://issues.apache.org/jira/browse/HUDI-4699) |
+| Metaserver for all metadata | 1.0.0 | [Strawman design](https://docs.google.com/presentation/d/1QBgLw11TM2Qf1KUESofGrQDb63EuggNCpPaxc82Kldo/edit#slide=id.ged084e5bf6_0_278), [RFC-36](https:/apache/hudi/pull/4718), [HUDI-3345](https://issues.apache.org/jira/browse/HUDI-3345) |


Major part of metaserver is already implemented and released in 0.13.0. We should remove it.

yihua · 2023-07-07T15:09:58Z

website/src/pages/roadmap.md

+| Supervised table service planning and execution | 1.0.0 | [RFC-43](https:/apache/hudi/pull/4309), [HUDI-4147](https://issues.apache.org/jira/browse/HUDI-4147) |
+| Record-level index to speed up UUID-based upserts and deletes | 0.14.0 | [RFC-08](https://cwiki.apache.org/confluence/display/HUDI/RFC-08++Record+level+indexing+mechanisms+for+Hudi+datasets), [HUDI-53](https://issues.apache.org/jira/browse/HUDI-53) |
+| Secondary index to improve query performance | 1.0.0 | [RFC-52](https:/apache/hudi/pull/5370), [HUDI-3907](https://issues.apache.org/jira/browse/HUDI-3907) |
+| Index Function for Optimizing Query Performance | 1.0.0 | [RFC-63](https:/apache/hudi/pull/7235), [HUDI-512](https://issues.apache.org/jira/browse/HUDI-512) |
 | Lock-Free message queue to improve writing efficiency | 0.13.0 | [RFC-53](https:/apache/hudi/blob/master/rfc/rfc-53/rfc-53.md), [HUDI-3963](https://issues.apache.org/jira/browse/HUDI-3963) |


This is done in 0.13.0.

yihua · 2023-07-07T15:12:05Z

website/src/pages/roadmap.md

 | Lock free concurrency control | 1.0.0 onward | [HUDI-3187](https://issues.apache.org/jira/browse/HUDI-3187) |
 | Non-blocking/Lock-free updates during clustering | 1.0.0 onward | [HUDI-1042](https://issues.apache.org/jira/browse/HUDI-1042) |
 | Time Travel updates, deletes | 1.0.0 onward ||
 | General purpose support for multi-table transactions | 1.0.0 onward ||
+| Lockless multi-writer support | 1.0.0 | [RFC-66](https:/apache/hudi/pull/7907) |
+| A more effective HoodieMergeHandler for COW table with parque | 1.0.0 | [RFC-68](https:/apache/hudi/blob/f1afb1bf04abdc94a26d61dc302f36ec2bbeb15b/rfc/rfc-68/rfc-68.md) |


Suggested change

| A more effective HoodieMergeHandler for COW table with parque | 1.0.0 | [RFC-68](https:/apache/hudi/blob/f1afb1bf04abdc94a26d61dc302f36ec2bbeb15b/rfc/rfc-68/rfc-68.md) |

| A more effective HoodieMergeHandler for COW table with parquet | 1.0.0 | [RFC-68](https:/apache/hudi/blob/f1afb1bf04abdc94a26d61dc302f36ec2bbeb15b/rfc/rfc-68/rfc-68.md) |

yihua · 2023-07-07T15:12:55Z

website/src/pages/roadmap.md

-| Redesign and optimization of record payload abstraction | 0.13.0 | [RFC-46](https:/apache/hudi/blob/master/rfc/rfc-46/rfc-46.md), [HUDI-3217](https://issues.apache.org/jira/browse/HUDI-3217) |
-| Optimized storage layout for cloud object stores | 0.13.0 | [RFC-60](https:/apache/hudi/pull/5113), [HUDI-3625](https://issues.apache.org/jira/browse/HUDI-3625) |
-| Support of verification with multiple event_time fields | 1.0.0 onward | [RFC-59](https:/apache/hudi/pull/6382), [HUDI-4569](https://issues.apache.org/jira/browse/HUDI-4569) |
+| New Hudi Table Spec API for Query Integrations | 1.0.0 | [RFC-64](https:/apache/hudi/pull/7080) |


We landed this in 0.14.0: MOR snapshot reader to integrate with query engines without using Hadoop APIs #9066 . Is it worth mentioning?

yihua · 2023-07-07T15:13:33Z

website/src/pages/roadmap.md

-| Support of verification with multiple event_time fields | 1.0.0 onward | [RFC-59](https:/apache/hudi/pull/6382), [HUDI-4569](https://issues.apache.org/jira/browse/HUDI-4569) |
+| New Hudi Table Spec API for Query Integrations | 1.0.0 | [RFC-64](https:/apache/hudi/pull/7080) |
+| Snapshot view management | 1.0.0 | [RFC-61](https:/apache/hudi/pull/6576), [HUDI-4678](https://issues.apache.org/jira/browse/HUDI-4678) |
+| Engine specific record representation for better performance | 1.0.0 | [RFC-46](https:/apache/hudi/blob/master/rfc/rfc-46/rfc-46.md), [HUDI-3217](https://issues.apache.org/jira/browse/HUDI-3217) |


This is released in 0.13.0.

yihua · 2023-07-07T15:15:21Z

website/src/pages/roadmap.md

+| Support for Spark 3.4.0 | 0.14.0 | [HUDI-6198](https://issues.apache.org/jira/browse/HUDI-6198) |
+| Integrate column stats index with all query engines | 0.14.0 | [RFC-58](https:/apache/hudi/pull/6345), [HUDI-4552](https://issues.apache.org/jira/browse/HUDI-4552) |
+| Upgrade to Spark 3 as the default profile | 0.14.0 | [HUDI-3431](https://issues.apache.org/jira/browse/HUDI-3431) |
+| Performance for Spark-SQL writes | 0.14.0 | [HUDI-6315](https://issues.apache.org/jira/browse/HUDI-6315), [HUDI-6376](https://issues.apache.org/jira/browse/HUDI-6376) |
 | Spark datasource V2 read | 1.0.0 onward | [HUDI-4449](https://issues.apache.org/jira/browse/HUDI-4449) |
 | Materialized Views with incremental updates using Flink | 1.0.0 onward ||
 | SQL DML support for Presto/Trino connectors (could be accelerated based on community feedback) | 1.0.0 onward ||


We don't plan to do this (SQL DML support in Presto/Trino) in near future. We should remove this entry.

yihua · 2023-07-07T15:15:47Z

website/src/pages/roadmap.md

@@ -61,8 +64,10 @@ Next major release : [0.13.0](https://issues.apache.org/jira/projects/HUDI/versi

 | Feature | Target Release | Tracking |
 |-----------------------------------------------------------------------------------------------------|--------------|-----------------------------------------------------------------------------------------------------------------------------------------|
-| Protobuf source for DeltaStreamer | 0.13.0 | [RFC-57](https:/apache/hudi/blob/master/rfc/rfc-57/rfc-57.md), [HUDI-4399](https://issues.apache.org/jira/browse/HUDI-4399) |
-| Improving Hudi CLI features and usability | 0.13.0 | [HUDI-1388](https://issues.apache.org/jira/browse/HUDI-1388) |
+| Improving Hudi CLI features and usability | 0.14.0 | [HUDI-1388](https://issues.apache.org/jira/browse/HUDI-1388) |


This is done in 0.13.0.

updated roadmap based on siva and ethan's feedback

nfarah86 · 2023-07-07T16:09:35Z

cc @yihua updated with items from rfc 69

nsivabalan · 2023-07-07T17:06:34Z

Can you add a jira and link it in title please.

website/src/pages/roadmap.md

yihua

LGTM with a few edits

nsivabalan

few minor comments

nfarah86 · 2023-07-07T17:43:21Z

Jira: https://issues.apache.org/jira/browse/HUDI-6504

nsivabalan reviewed Jul 7, 2023

View reviewed changes

yihua reviewed Jul 7, 2023

View reviewed changes

updated roadmap

33bcc10

updated roadmap based on siva and ethan's feedback

nfarah86 force-pushed the updateRoadmap branch from f13b72e to 33bcc10 Compare July 7, 2023 16:07

nsivabalan requested changes Jul 7, 2023

View reviewed changes

website/src/pages/roadmap.md Outdated Show resolved Hide resolved

website/src/pages/roadmap.md Outdated Show resolved Hide resolved

Update roadmap.md

e539e6f

yihua approved these changes Jul 7, 2023

View reviewed changes

yihua changed the title ~~updated roadmap~~ [DOCS] Updated roadmap Jul 7, 2023

yihua added the docs label Jul 7, 2023

nsivabalan reviewed Jul 7, 2023

View reviewed changes

nsivabalan approved these changes Jul 7, 2023

View reviewed changes

nsivabalan changed the title ~~[DOCS] Updated roadmap~~ [HUDI-6504] [DOCS] Updated roadmap Jul 7, 2023

yihua merged commit 6b3dadc into apache:asf-site Jul 7, 2023
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[HUDI-6504] [DOCS] Updated roadmap #9142

[HUDI-6504] [DOCS] Updated roadmap #9142

nfarah86 commented Jul 7, 2023

nsivabalan Jul 7, 2023

nfarah86 Jul 7, 2023

nsivabalan Jul 7, 2023

yihua Jul 7, 2023

nfarah86 Jul 7, 2023

nfarah86 Jul 7, 2023

nsivabalan Jul 7, 2023

nfarah86 Jul 7, 2023

yihua Jul 7, 2023

nsivabalan Jul 7, 2023

yihua Jul 7, 2023

nsivabalan Jul 7, 2023

nfarah86 Jul 7, 2023

yihua Jul 7, 2023

yihua Jul 7, 2023

yihua Jul 7, 2023

yihua Jul 7, 2023

yihua Jul 7, 2023

yihua Jul 7, 2023

yihua Jul 7, 2023 •

edited

Loading

yihua Jul 7, 2023

nfarah86 commented Jul 7, 2023

nsivabalan commented Jul 7, 2023

yihua left a comment

nsivabalan left a comment

nfarah86 commented Jul 7, 2023

		\| Mutable, Transactional caching for Hudi Tables (could be accelerated based on community feedback) \| 1.0.0 onward \| [Strawman design](https://docs.google.com/presentation/d/1QBgLw11TM2Qf1KUESofGrQDb63EuggNCpPaxc82Kldo/edit#slide=id.gf7e0551254_0_5) \|
		\| Hudi Reverse streamer \| 1.0.0 onward \| [RFC-70](https:/apache/hudi/pull/9040) \|

	\| A more effective HoodieMergeHandler for COW table with parque \| 1.0.0 \| [RFC-68](https:/apache/hudi/blob/f1afb1bf04abdc94a26d61dc302f36ec2bbeb15b/rfc/rfc-68/rfc-68.md) \|
	\| A more effective HoodieMergeHandler for COW table with parquet \| 1.0.0 \| [RFC-68](https:/apache/hudi/blob/f1afb1bf04abdc94a26d61dc302f36ec2bbeb15b/rfc/rfc-68/rfc-68.md) \|

[HUDI-6504] [DOCS] Updated roadmap #9142

[HUDI-6504] [DOCS] Updated roadmap #9142

Conversation

nfarah86 commented Jul 7, 2023

Change Logs

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yihua Jul 7, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nfarah86 commented Jul 7, 2023

nsivabalan commented Jul 7, 2023

yihua left a comment

Choose a reason for hiding this comment

nsivabalan left a comment

Choose a reason for hiding this comment

nfarah86 commented Jul 7, 2023

yihua Jul 7, 2023 •

edited

Loading