[CT-461] Capture exceptions at top level of threads and cli

[nathaniel-may]

Is there an existing feature request for this?

• I have searched the existing issues

Describe the Feature

discovered while coding up #4326. Somewhere in here when an exception is raised, the thread just hangs and you have to step through a debugger to read the exception message. Exceptions should bubble all the way to the top even in threaded environments.

you can recreate this by making fields_to_json throw.

[gshank]

I saw a similar problem when adding the event_status dictionary to ParsedNodeDefaults. It hung executing a test in test/integration/059... because the SourceDefinition didn't have that attribute. Did not see anywhere the actual error, but since I had just added that attribute I guessed, added it to SourceDefinition, and it fixed it.

[nathaniel-may]

Estimation:

• guess is that when an exception is thrown inside a thread, it's not caught and handled correctly
• rpc server would benefit from this fix as well. Possibly fix existing tickets?
• This is basically one big unknown unknown
• Might be a better fit for the execution team
• suggesting a 2 day spike instead of estimating.

[gshank]

Note: this failure was experienced while running tests. On a particular type of test failure, the tests would hang and not complete. I was able to recreate this failure by removing "NodeInfoMixin" from the ParsedSourceDefinition class and executing: ".tox/py-integration/bin/python -m pytest -nauto --full-trace tests/functional/sources". (The "--full-trace" was suggested by pytest.) The resulting stacktrace started in /_pytest/main.py, and ended in threading.py, in the wait function, on the line "gotit = waiter.acquire(True, timeout)".

I was able to reduce it to a single test by running pytest -v --full-trace tests/functional/sources/test_source_fresher_state.py::TestSourceFresherNothingToDo::test_source_fresher_nothing_to_do. The stack trace went from the run_queue function in task/runnable.py, "self.job_queue.join()" to the join function in "graph/queue.py", line "self.inner.join()", to the joing function in the python queue.py file, "self.all_testsk_done.wait()", and then in the wait function in threading.py "waiter.acquire()".

Since this is deep in the execution team area, it might make more sense for them to take a look at it.

[gshank]

Closing this issue in favor of new ticket #5602, as requested.