services/submitter: Prototype of Transaction Submission Service

[bartekn]

What

This commit moves transaction submitter module from an old SDF project and adapts it to the current stellar/go API.

Why

A service like this can be useful for organizations outside SDF. Also, in SDF we have multiple tx submitters in different projects, using a single, well-tested and battle proven code can save time and prevent some common mistakes.

How it works

TransactionSubmitter is the main process responsible for submitting transactions. A transaction can be in one of four states:

• pending - transaction waiting to be submitted,
• sending - transaction loaded and queued to be submitted,
• sent - transaction successfully sent (added to ledger)
• error - error while sending.

Every second TransactionSubmitter loads TransactionPerSecond transactions with pending state from a DB and attempts to send them. In a DB transaction it changes their state to sending.

Next, one of Channels is selected to actually submit tx with it's sequence number. Channels are used one after another. If tx is successfully included in the ledger the state is changed to sent, otherwise to error. In case of bad_seq error channel state is automatically reloaded.

Known limitations

The above algorithm is simple and prevents many common mistakes like attempting to send a transaction again on error case (which ex. can be a success but the 200 OK response was not delivered due to network issues).

Below is a list of possible improvements to the system:

• Make resubmitting an automatic process. Currently the system requires a manual inspection of errored transactions. This is probably fine when the system is maintained by an experienced developer but it's not always a case. We could improve the algorithm to detect when the error is actually a final state:
  • Add timebounds.max_time and monitor ledgers to check if tx actually made it to the ledger in a given time.
  • There's a state in which a tx is loaded to be sent but is not actually sent (before POST /transactions call). Maybe we need an intermediate state? This state can be safely reverted to pending on system crash.
• Currently the fees are hardcoded. We should probably allow admins to set max fee and monitor /fee_stats endpoint in Horizon. As with previous points, txs that did not make it into ledger because of low fee should be resubmitted.
• Idea from @JakeUrban: in init stage, channel accounts can be created using SEP-5 derivation and sponsored by a master account. This will simplify the process to just generating a master seed which is great.
• Currently the system is only supporting Postgresql. When the feature set of V1 is complete we can create an interface based on PostgresStore to support other DB engines.
• Also @JakeUrban: It should be possible to use the system as a Go package but also it should expose JSON-RPC to be used by non-Go applications.

[JakeUrban]

Thanks for getting this started, I'm super excited about the project.

Error Handling

I think the next step to take here is to implement automatic resubmission on recoverable errors. Recoverable errors are:

• 504 status code
• 429 status code
• tx_too_late transaction result code
• tx_bad_seq transaction result code
• tx_insufficient_fee transaction result code

Errors that are not recoverable:

• 500 status code
• tx_insufficient_balance result code
• all payment operation result codes (except success)

All other errors indicate a bug in the package.

Interface Design

I also think we need to simplify the interface exposed to the developer using this package. From their perspective, there should be three transaction statuses:

• pending
• success
• error

There should be no distinction between a transaction that has not been picked up from the store, one that has been retrieved from the store but not submitted, and one that has been submitted but has not been included in a ledger.

[JakeUrban]

What

Refactors @bartekn's initial implementation of the transaction submitter to better manage concurrent transaction submission.

• Each channel account is given a go routine that pull from a buffered queue of transactions
• Every second, the submitter checks if the queue is full, and if its not, fills the queue with more pending transactions

This PR also adds the following:

• Signs the transactions with the root & channel account
• Uses fee bump transactions so channel accounts don't need to hold XLM

Why

The existing implementation assigns transactions to channels that could still be working on submitting previously assigned transactions. This would cause sequence number errors and channel accounts would constantly have to reload their state, reconstruct transactions, and resubmit them.

Using a buffered channel allows for:

1. the submitter to only query the DB when the channel is not full
2. The channel accounts to never be idle -- unless there are no pending transactions in the database