Replay

[pictographer]

@jbearer wrote:
In the process of finding a bug that had caused a panic in the deployed Goerli faucet, I ended up writing a simple utility to replay a list of ledger events in a local wallet. By using the same mnemonic phrase that the Goerli faucet used, I was able to replicate the exact sequence of event handling that caused the crash, and reproduce the exact same panic, but locally and independent of the Goerli deployment. From there, I was able to point at a local copy of Seahorse and methodically add logging until I found the bug, less than a half hour later.

Right now, this replay tool is very simple. It doesn't do any analysis or provide any additional logging of its own. It merely gives you a quick way to reproduce a problem locally, and you can then go about adding instrumentation and whatever else you might need to do to debug the issue. So the workflow is like

• a bug happens in production
• immediately hit https://eqs.goerli.cape.tech/get_events_since/0 and save the result, which is a JSON serialized list of events
• run wallet-replay with mnemonic of the production wallet that failed and the saved event list
• if you suspect the bug is in a dependency (like Seahorse, in this case) point at a local copy of the dependency and add instrumentation
• debug until you understand the problem

Since we spend so much effort making things deterministic anyways (this applies to both the wallet, and the espresso validators) and since the EQS/EsQS necessarily saves an ordered list of every event that happens since genesis, this kind of debugging tool can potentially be made extremely powerful for fixing bugs that occur in production or bugs that are otherwise difficult to reproduce. Maybe a line of work for the ATA sometime in the future could be building out this tooling, not just for the CAPE wallet but for the generic wallet and for other services like the validators, and making it easier to get useful information out of the tool than changing all of your Cargo.tomls to point at a local copy of a dependency and then adding logging statements.