Fullnode is panicking and exiting

[pgarg66]

Problem

Fullnode in testnet sometimes exit with panicked at 'fullnode exit: Any', src/libcore/result.rs:997:5. The panic can be seen in the dashboard.

t             at src/libcore/panicking.rs:85
   9: core::option::expect_failed
             at src/libcore/option.rs:1038
  10: solana::leader_schedule_utils::slot_leader_at
  11: solana::replay_stage::ReplayStage::start_leader
thread 'main' panicked at 'fullnode exit: Any', src/libcore/result.rs:997:5
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:39
   1: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:70
   2: std::panicking::default_hook::{{closure}}
             at src/libstd/sys_common/backtrace.rs:58
             at src/libstd/panicking.rs:200
   3: std::panicking::default_hook
             at src/libstd/panicking.rs:215
   4: solana_metrics::metrics::set_panic_hook::{{closure}}::{{closure}}
   5: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:482
   6: std::panicking::continue_panic_fmt
             at src/libstd/panicking.rs:385
   7: rust_begin_unwind
             at src/libstd/panicking.rs:312
   8: core::panicking::panic_fmt
             at src/libcore/panicking.rs:85
   9: core::result::unwrap_failed
  10: solana_fullnode::main
  11: std::rt::lang_start::{{closure}}
  12: std::panicking::try::do_call
             at src/libstd/rt.rs:49
             at src/libstd/panicking.rs:297
  13: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:92
  14: std::rt::lang_start_internal
             at src/libstd/panicking.rs:276
             at src/libstd/panic.rs:388
             at src/libstd/rt.rs:48
  15: main
  16: __libc_start_main
  17: _start

Proposed Solution

[pgarg66]

Also, seeing this panic in the dashboard.

panicked at 'epoch state must exist', src/libcore/option.rs:1038:5

[sagar-solana]

Working on a fix for this via #3164. Need to root cause the problem and fix that. Current solution doesn't fix the root.