Task remains in "running" state when last attempt timed out

[mikecote]

This is outlined in the documentation under limitations, just very strange once we have a Task Manager UI.

Steps to reproduce:

• Create a custom task definition with a runner that waits for a never resolving promise await new Promise(resolve => {});
• Schedule a task of that type
• Monitor the .kibana_task_manager document and once it reaches attempts: 3 (or 2?), you will see it remain in running status

NOTE: The default timeouts and retry can be long, you can modify those in the task definition (to ex: timeout: '10s') and in the task runner (to ex: 30s) for backoff multiple to test this.

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[gmmorris]

I've done some investigation and it looks like we can use the updatebyQuery to solve this.
The query filters out tasks whose attempts have exceeded the max attempts, but instead we can pick these up as well and in the updateFields function, instead of just blindly setting to claiming we can write some painless code that will instead set the status to failed.

kibana/x-pack/plugins/task_manager/server/queries/mark_available_tasks_as_claimed.ts

Lines 103 to 108 in a8b1b9e

	export const updateFields = (fieldUpdates: {
	[field: string]: string | number | Date;
	}): ScriptClause => ({
	source: Object.keys(fieldUpdates)
	.map((field) => `ctx._source.task.${field}=params.${field};`)
	.join(' '),

[ymao1]

Waiting on this issue until #80371 is investigated/resolved. My concern is that if there is some natural limit to the updateByQuery that is preventing older docs from being returned/updated/sorted, removing the clause to limit the search by maxAttempts would return even more newer, potentially failed docs, exacerbating the issue of not picking up older documents.

[pmuellr]

It seems that we found the issue with the zombie idle tasks in TM, so we can proceed here. There's a proposed fix in PR #80692, I suspect it will not affect this issue, seems like we can merge the fixes independently.