Rework scheduler/IPC layer API to not be a horrifying mess

[andyross]

The scheduler internals were rewritten a while back and they're tight and featureful and I love them. The scheduler public API has been stable and clean for a while, with only a handful of new features over the years which are straightforward and orthogonal.

But the scheduler internal API used by the IPC layer remains a ridiculous tangled archaeology of successive design changes.

We can pend and unpend threads. We can pend the current thread and atomically release a lock. We can pend the current thread without a lock held. We can pend the current thread with only the legacy irq_lock() held, even though nothing in tree uses it anymore. We can unpend threads from wait queues or directly. We can unpend the first thread from a wait queue or all of them at once. We can abort a current timeout while unpending or ignore it.

And all those permutations are implemented in terms of each other in complicated ways that lead to mistakes, like the lack of synchronization discovered in #32136

And most of the complexity is silly, being there just for the benefit of one or two callers that could be easily ported to a cleaner API.

[pabigot]

How does this relate to #29668, which is something I'd like to see completed.

[andyross]

It's the same problem. I wasn't completely on board with @andrewboie 's desire for a firmly documented API here (the IPC layer is inherently quite closely coupled to the kernel, after all) but certainly wasn't opposed. This is capturing the underlying problem, not that specific solution I guess. My personal preference would be to start with a cleanup of the layer architecture we have and then see later if that fits into something we want to bless long-term as an internal API.

[pabigot]

OK. I do think it's worth having a clearly documented API; questions about what atomic, spin_lock, pend, etc. really promise to do in various configurations (and why we think the implementation is correct) will come up repeatedly even if the primary audience is kernel maintainers (and those that aspire or are forced to become such because they need special IPC capabilities, as in #29618).

I'm fine with a cleanup approach as long as it includes documentation.