PIP-31: Transactional Streaming #2664

kuangye098 · 2018-09-27T03:56:32Z

This is the master issue for tracking implementing PIP-31: https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#heading=h.40v9f3564x7t

V1: #8657
V2: #8633

sijie · 2018-09-30T07:55:00Z

@kuangye098 it is a nice feature request. I am wondering if you are interested in implementing this feature, we can collaborate on this if you are interested in contribution.

The producer pulls the pre-processing transaction message.

you mean "produce", instead of "pull", right?

kuangye098 · 2018-09-30T08:06:30Z

@sijie You are right, I may have described it incorrectly.

kuangye098 · 2018-09-30T08:16:19Z

@sijie I can try to implement it, because in other projects, I have implemented it.

sijie · 2018-09-30T08:25:27Z

@kuangye098 that’s awesome. Since it is big feature, it would be good that you can write a PIP (pulsar improvement proposal) to describe your proposal of changes. Then we can discuss in details.

Here is wiki page for all the pulsar PIPs. You can check out how other people wrote them https:/apache/pulsar/wiki

kuangye098 · 2018-09-30T08:42:07Z

@sijie OK，I will read it carefully and then write a PIP to describe the proposal of changes.

sijie · 2018-09-30T18:34:35Z

great thank you @kuangye098

codelipenghui · 2019-01-02T03:39:59Z

@sijie Is this the same as you mentioned before(Fast Transaction Message)?

ConcurrencyPractitioner · 2019-02-28T04:03:48Z

Hi @sijie BTW do you know how we create a PIP on Github? For example, do we need permission granted or something first before we create one.

sijie · 2019-02-28T04:13:56Z

@ConcurrencyPractitioner :

You don't need any permissions to start a PIP, but you do need permissions to post the PIP to Pulsar Github Wiki.

So you can just write down a PIP in gist and share it to dev@ mailing list. Any pulsar committer can help you re-post your gist to Pulsar wiki.

sijie · 2019-03-27T07:02:42Z

Here is the PIP for supporting transactions: https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#heading=h.40v9f3564x7t

I will break down this PIP into tasks and update the tasks here. This issue will be used as an anchor for all transaction related tasks.

sijie · 2019-04-22T09:28:36Z

Broke the PIP into tasks and updated them in the description.

sijie · 2019-04-22T09:28:59Z

Marked this as for 2.5.0. since we are close to release 2.4.0 soon.

…a store and an in-memory implementation Master Issue: apache#2664 *Motivation* Setup the project structure for developing transaction coordinator. *Changes* - setup the project structure - introduce common classes: `TxnID`, `TxnStatus` - introduce `TransactionMetadataStore` for TC to store and manage the transaction status - enable checkstyle for `pulsar-transaction` *NOTES* All the changes are isolated into `pulsar-transaction` module. Hence it will not block any other changes or releases.

MarvinCai · 2019-04-29T02:17:06Z

I'm very interesting in this PIP and I could help with some subtasks.

…-memory implementation Master Issue: apache#2664 *Motivation* Setup the project structure for developing transaction buffer. *Changes* - setup the project structure - introduce common classes: `TxnID`, `TxnStatus` - introduce `TransactionBuffer` for broker to store and manage entries for transactions - enable checkstyle for `pulsar-transaction` *NOTES* All the changes are isolated into `pulsar-transaction` module. Hence it will not block any other changes or releases.

sijie · 2019-04-29T05:57:22Z

@MarvinCai : @tuteng and I start working on the TC part, and @zymap is already working on the TB part. Maybe you can help us in the acknowledgement part. I will try to do the api and protocol and coordinating the work around.

…a store and an in-memory implementation (#4161) * [transaction][coordinator] add the interfaces for transaction metadata store and an in-memory implementation Master Issue: #2664 *Motivation* Setup the project structure for developing transaction coordinator. *Changes* - setup the project structure - introduce common classes: `TxnID`, `TxnStatus` - introduce `TransactionMetadataStore` for TC to store and manage the transaction status - enable checkstyle for `pulsar-transaction` *NOTES* All the changes are isolated into `pulsar-transaction` module. Hence it will not block any other changes or releases.

…-memory implementation (#4165) Master Issue: #2664 *Motivation* Setup the project structure for developing transaction buffer. *Changes* - setup the project structure - introduce common classes: `TxnID`, `TxnStatus` - introduce `TransactionBuffer` for broker to store and manage entries for transactions - enable checkstyle for `pulsar-transaction` *NOTES* All the changes are isolated into `pulsar-transaction` module. Hence it will not block any other changes or releases.

…cknowledgement path (#4265) Master Issue: #2664 Motivation: Add acknowledgeMessage, commit, abort for transaction in PersistentSubscription. Changes: Will put message in Pending_ACK status when acknowledgeMessage is called with TxnID. No real status class introduced, only added collection to hold messages in Pending_ACK status. Current PR only keep Pending_ACK state in memory, in subsequent PR will also persistent these pending acks so we can recover from broker failure. Add commitTxn to put message to Deleted status. Add abortTxn to put message to Pending status. For normal acknowledgeMessage and redeliverUnacknowledgedMessages, will check to see if message if message is in Pending_ACK first. If true, will **ignore** those acks/redeliverys. Add unit test.

Master Issue: #2664 Fix https:/streamnative/pulsar/issues/1304 ### Motivation Currently, the consumer can't receive transaction messages. ### Modifications Support process the commit marker in the topic partition and fetch transaction messages from TransactionBuffer.

Master Issue: apache#2664 Fix https:/streamnative/pulsar/issues/1304 ### Motivation Currently, the consumer can't receive transaction messages. ### Modifications Support process the commit marker in the topic partition and fetch transaction messages from TransactionBuffer.

Fix https:/streamnative/pulsar/issues/1307 Master Issue: #2664 ### Motivation Handle acknowledge in the transaction. ### Modifications 1. Register subscription to the transaction metadata store. 2. Make the ack command carry the transaction information. 3. When committing a transaction, commit every subscription in the transaction.

Master Issue: apache#2664 Fix https:/streamnative/pulsar/issues/1304 ### Motivation Currently, the consumer can't receive transaction messages. ### Modifications Support process the commit marker in the topic partition and fetch transaction messages from TransactionBuffer.

Fix https:/streamnative/pulsar/issues/1307 Master Issue: apache#2664 ### Motivation Handle acknowledge in the transaction. ### Modifications 1. Register subscription to the transaction metadata store. 2. Make the ack command carry the transaction information. 3. When committing a transaction, commit every subscription in the transaction.

Master Issue: apache#2664 Fix https:/streamnative/pulsar/issues/1304 ### Motivation Currently, the consumer can't receive transaction messages. ### Modifications Support process the commit marker in the topic partition and fetch transaction messages from TransactionBuffer.

Fix https:/streamnative/pulsar/issues/1307 Master Issue: apache#2664 ### Motivation Handle acknowledge in the transaction. ### Modifications 1. Register subscription to the transaction metadata store. 2. Make the ack command carry the transaction information. 3. When committing a transaction, commit every subscription in the transaction.

Master Issue: apache#2664 Fix https:/streamnative/pulsar/issues/1304 ### Motivation Currently, the consumer can't receive transaction messages. ### Modifications Support process the commit marker in the topic partition and fetch transaction messages from TransactionBuffer.

Fix https:/streamnative/pulsar/issues/1307 Master Issue: apache#2664 ### Motivation Handle acknowledge in the transaction. ### Modifications 1. Register subscription to the transaction metadata store. 2. Make the ack command carry the transaction information. 3. When committing a transaction, commit every subscription in the transaction.

sijie · 2020-11-19T20:12:12Z

Created a separate issue for tracking the remaining tasks at #8633.

The transaction feature will be released as a developer-preview feature as part of 2.7.0.

codelipenghui · 2020-11-21T03:47:18Z

Create transaction V1 issue to track the tasks that release at 2.7.0, so move this issue to 2.8.0

codelipenghui · 2021-05-21T13:33:38Z

Close this issue, #8633 is tracking the remaining tasks.

sijie added area/client type/feature The PR added a new feature or issue requested a new feature labels Sep 30, 2018

sijie added the triage/week-40 label Oct 3, 2018

sijie added triage/week-8 and removed triage/week-40 labels Feb 21, 2019

sijie self-assigned this Mar 6, 2019

sijie changed the title ~~Support for Transactional Message~~ PIP-31: Transactional Streaming Mar 27, 2019

sijie added this to the 2.5.0 milestone Apr 22, 2019

sijie mentioned this issue Apr 28, 2019

[transaction][coordinator] add the interfaces for transaction metadata store and an in-memory implementation #4161

Merged

sijie mentioned this issue Apr 29, 2019

[transaction][buffer] add interfaces for transaction buffer and an in-memory implementation #4165

Merged

MarvinCai mentioned this issue May 12, 2019

[transaction][acknowledge] Introduce in-memory PENDING_ACK state in acknowledgement path #4265

Merged

gaoran10 mentioned this issue Aug 7, 2020

[Transaction] Support consume transaction messages. #7781

Merged

1 task

sijie removed the triage/week-35 label Aug 10, 2020

This was referenced Aug 14, 2020

[WIP] [Transaction] Support Ack Transaction Message #7816

Closed

[Transaction] Handle Acknowledge In The Transaction #7856

Merged

tuhaihe mentioned this issue Sep 11, 2020

2020-9-9: Pulsar China User Group Meeting streamnative/pulsar-user-group-loc-cn#2

Open

codelipenghui removed this from the 2.7.0 milestone Nov 19, 2020

sijie mentioned this issue Nov 19, 2020

PIP-31: Transactional Streaming V2 #8633

Open

4 tasks

sijie added this to the 2.7.0 milestone Nov 19, 2020

sijie assigned codelipenghui Nov 19, 2020

sijie mentioned this issue Nov 19, 2020

ISSUE-8633: PIP-31: Transactional Streaming V2 streamnative/pulsar-archived#1715

Closed

4 tasks

codelipenghui modified the milestones: 2.7.0, 2.8.0 Nov 21, 2020

hpvd mentioned this issue Nov 26, 2020

[Doc] hardware recommodations (Cores, RAM, storage) for dev #8671

Closed

cbornet mentioned this issue Dec 4, 2020

Transaction support cbornet/pulsar-grpc#1

Open

This was referenced Jan 19, 2021

ISSUE-8633: PIP-31: Transactional Streaming V2 streamnative/pulsar-archived#2026

Closed

ISSUE-8633: PIP-31: Transactional Streaming V2 streamnative/pulsar-archived#2027

Closed

ISSUE-8633: PIP-31: Transactional Streaming V2 streamnative/pulsar-archived#2028

Open

codelipenghui closed this as completed May 21, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PIP-31: Transactional Streaming #2664

PIP-31: Transactional Streaming #2664

kuangye098 commented Sep 27, 2018 •

edited by codelipenghui

Loading

sijie commented Sep 30, 2018

kuangye098 commented Sep 30, 2018 •

edited

Loading

kuangye098 commented Sep 30, 2018

sijie commented Sep 30, 2018

kuangye098 commented Sep 30, 2018

sijie commented Sep 30, 2018

codelipenghui commented Jan 2, 2019

ConcurrencyPractitioner commented Feb 28, 2019

sijie commented Feb 28, 2019

sijie commented Mar 27, 2019

sijie commented Apr 22, 2019

sijie commented Apr 22, 2019

MarvinCai commented Apr 29, 2019

sijie commented Apr 29, 2019

sijie commented Nov 19, 2020

codelipenghui commented Nov 21, 2020

codelipenghui commented May 21, 2021

PIP-31: Transactional Streaming #2664

PIP-31: Transactional Streaming #2664

Comments

kuangye098 commented Sep 27, 2018 • edited by codelipenghui Loading

sijie commented Sep 30, 2018

kuangye098 commented Sep 30, 2018 • edited Loading

kuangye098 commented Sep 30, 2018

sijie commented Sep 30, 2018

kuangye098 commented Sep 30, 2018

sijie commented Sep 30, 2018

codelipenghui commented Jan 2, 2019

ConcurrencyPractitioner commented Feb 28, 2019

sijie commented Feb 28, 2019

sijie commented Mar 27, 2019

sijie commented Apr 22, 2019

sijie commented Apr 22, 2019

MarvinCai commented Apr 29, 2019

sijie commented Apr 29, 2019

sijie commented Nov 19, 2020

codelipenghui commented Nov 21, 2020

codelipenghui commented May 21, 2021

kuangye098 commented Sep 27, 2018 •

edited by codelipenghui

Loading

kuangye098 commented Sep 30, 2018 •

edited

Loading