Logs failing to flush in Logdna::Client due to concurrency issue in rails

[matthagy]

Found that any uses of Logdna::Client within the boot sequence of rails results in subsequent logs never flushing.

For background, using logdna 1.5.0 w/ rails 6.1.7.5 on ruby 3.2.2. Configuring Logdna::Ruby following the instructions in TAMING RUBY ON RAILS LOGGING WITH LOGRAGE AND MEZMO in config/application.rb. As long as Rails.logger isn't used within the subsequent application setup steps, then logging works as expected. But if any other subsequent step of the application setup process attempts to log, then all subsequent logs fail to be emitted.

Tracked this down to Logdna::Client#schedule_flush. By adding logging, I found that the associated Concurrent::ScheduledTask in @scheduled_flush remains in the pending state indefinitely. Further logging shows that the ScheduledTask is associated with a Concurrent::CachedThreadPool that contains a single thread and that thread has is marked as dead.

Guessing that as part of the rails boot up process, the default executor is terminated and therefore the @scheduled_flush never completes. Not that using the @work_thread_pool thread pool doesn't fix the issue; the same behavior is observed. By not logging anything during rails configurations/boot, a ScheduledTask isn't created while the application is still booting, and therefore this issue doesn't occur.

I think this issue could be addressed by tracking the time at which @scheduled_flush is created and then in subsequent invocations of schedule_flush checking if it has been pending far longer than expected. If it is sufficiently stale, then the outstanding ScheduledTask would be cancelled and a new one created. There may be more robust/elegant ways to address this because I'm not particularly familiar with ruby/rails thread management.