[BUG]: DocaSourceStage could potentially overwrite data in ring buffer before downstream stages have a chance to read them

[dagardner-nv]

Version

24.10

Which installation method(s) does this occur on?

Docker, Source

Describe the bug.

DocaSourceStage creates a large ring buffer which can hold up to 67M packets.
However there is not guarantee that downstream stages (DocaConvertStage) will be able to consume packet data prior to being overwritten.

Worse is that a downstream stage could also get a dirty read where the wrong packet, or a partially written packet data is read leading to data corruption.

Minimum reproducible example

No response

Relevant log output

Click here to see error details

[Paste the error here, it will be hidden by default]

Full env printout

Click here to see environment details

[Paste the results of print_env.sh here, it will be hidden by default]

Other/Misc.

No response

Code of Conduct

• I agree to follow Morpheus' Code of Conduct
• I have searched the open bugs and have found no duplicates for this bug report

[dagardner-nv]

I'm thinking there are a few approaches here:

1. Copy and send from the source stage, I took a quick pass at this approach but it came at the cost of performance by ~10x
2. Perform a new malloc/move, really a variation on option 1 but we remove the ring buffer, and avoid the cost of the copy. The tricky bit here is that since the source doesn't know how many packets it will receive, it needs to malloc up to MAX_PKT_RECEIVE and then shrink to the amount actually used.
3. Distribute references to the semaphores, allowing the convert stage to release them. In the event that the convert stage isn't keeping up, this would eventually block the source stage.
4. Include some sort of packet_id with the RawPacketMessage allowing the convert message to determine if the packet has been overwritten. This would likely still require the semaphores to prevent a partial read.