net_buf issue leads to unwanted elem free

[XavierChapron]

Describe the bug
Current net_buf structure shared the first word in a union between sys_snode_t node; and struct net_buf *frags;.
This has been introduced a very long time ago : 553af7c
However, net_buf allocation are handled by a LIFO, and the current documentation of k_lifo_put(), clearly states that the first word of the element must be reserved for the kernel, which is not the case here.

zephyr/include/kernel.h

Line 2302 in 2637c0e

* This routine adds a data item to @a lifo. A LIFO queue data item must be

It appears, that this shared first word works great in many cases as the kernel mostly use it only when the element is in the queue and therefore the element is not allocated at this time.
However, in a edge case (probably when the queue is empty and a new element is added), the last element is modified, even if it is currently in used.
At that moment, the value of node is written, hence the frags value is not NULL anymore, and bad things happens in my use case.

To Reproduce
I've currently see this using mcumgr and smp_bt, however I expect it might be reproducible more easily when the edge case as been localized more accurately.

Impact
On my case, this leads to a mess in the net_buf pool, that leads to bt_conn ref not being properly released, witch means bt_le_adv_resume() is never called in bt_conn_unref().

Environment:

• Zephyr SDK,
• nrf52840,
• zephyr 2.4 but as neither net_buf nor queue implementations s changed since, I really think main is also affected.

[XavierChapron]

This is a eventually a duplicate of #32579.
See #38830 (comment) for more details.