(Reverted by #4882) Asynchronously write batches to NuDB.

[mtrippled]

High Level Overview of Change

For NuDB, use BatchWriter for all writes. This is identical behavior to that of RocksDB.
Increase batch size.

This PR is related to 2 others that, when combined, increase throughput significantly:
#4504
#4505

Context of Change

This improves transaction throughput. The bulk of writes to the nodestore happen during the "accept" phase of consensus as a new ledger is built. The number of objects increases directly with transaction volume, and write latency directly affects consensus interval duration. Batching writes and performing them in the background minimizes this impact. The new behavior is identical to that of RocksDB, which has used BatchWriter in production for several years already. Increasing the batch size should allow all ledgers for the forseeable future to be persisted this way without blocking consensus.

Type of Change

• [X ] New feature (non-breaking change which adds functionality)

This change is part of a suite of throughput-enhancing pull requests. They have been tested in conjunction with one another.

[intelliot]

note: Howard and John will try to look this week

[HowardHinnant]

Looks good to me.

[ximinez]

LGTM

[intelliot]

note: the current plan is for this to be squashed and merged together with the other 2 PRs in this milestone, after all of them have been approved in code review (by 2+ reviewers each).

[intelliot]

correction: the 3 PRs will be merged at about the same time, but they will be 3 separate commits on develop.

[intelliot]

Since this was reverted in 1.12.0-rc3, this PR is being reopened for additional review, testing, and potential improvements, with the expectation that it will be merged again soon (e.g. for the 1.13 or 2.0 release).

[intelliot]

note: still prefer to merge this at the same time as #4504 #4505

[intelliot]

Note: This PR introduces a bug which #4762 was expected to fix, but it apparently does not. Thus, the proposal of #4882 which just reverts #4503.

[JoelKatz]

Having looked at this code and looked at its effects, I think it should be reverted if it hasn't been already.

My vague recollection was that the batch writing code was added because earlier databases had a habit of stalling thread that called in to write even under low load. If load gets too high, there is no choice but to impose a delay on each write because otherwise data to be written will back up in memory without limit. This code imposes a limit on how much data can back up in memory and then it stalls writer threads.

However, NuDB has built in logic to do exactly this already. And it does it much better than this code can because NuDB does it in a way that is both aware of, and connected to, the database internals.

For example, NuDB knows precisely what its write capabilities are and can throttle only if the write rate exceeds its capabilities. The batch writer just has a fixed buffer threshold. NuDB's write logic drops redundant writes without ever buffering them or risking delaying a thread because of them. The batch writer doesn't do that and so can hold multiple copies of the same object. And NuDB adds delayed writes to a cache that will allow the data to be fetched immediately. The batch writer doesn't do that and so may trigger extra work to fetch the same object twice.

If there is a demonstrated need to reduce the amount of writes delayed, the better option is to increase NuDB's burst size which is already a configurable option. Even better might be a change to NuDB to allow the burst size to be raised automatically, say, to the lesser of its configured value or half the size of the previous write batch.

[intelliot]

This has already been reverted by #4882, but thank you for looking into this, because otherwise we were considering re-merging this again (in conjunction with #4762).