Websocket flow control is broken (deadlock+memory leak on websocket.receive)

[alexmnazarenko]

Long story short

Hello! I've ran into serious problem while implementing custom RPC over websockets (aiohttp on both sides). On a dense stream of larger messages consumer just freezes (while eating gigabytes of memory). Sender is "fine" (await ws.send_bytes() proceeds).

It seems to be correlated with message size, but I'm not sure. Surfaces randomly, but quite often on larger (64k+) messages. Who is client and who is server isn't important, so I'll stick to sender/consumer terms.

The problem manifested itself only in actual distributed test, if sender-consumer are both on localhost everything works fine.

Why do I think that issue is somewhere in flow control - the usecase was fine on older versions of aiohttp before it was implemented.

Any feedback is much appreciated )

Expected behaviour

• Consumer doesn't hang
• Sender blocks if needed

Actual behaviour

• On consumer's side: await ws.receive (or async for msg in ws) hangs at some point, memory grows to infinity and beyond
• On sender's side: everything looks fine (but you'll never get any response as consumer is dead)
• Even more interesting: very rarely consumer shows an error and disconnects

Steps to reproduce

Original program was quite convoluted, so I've made a simple example reproducing the issue.

Consumer script (implemented as server):

import aiohttp.web

async def consumer(request):
    ws = aiohttp.web.WebSocketResponse()
    await ws.prepare(request)

    msg_index = 0
    async for msg in ws:
        msg_index += 1
        print("Block", msg.type, msg_index)
    return ws


def main():
    app = aiohttp.web.Application()
    app.router.add_get("/ws", consumer)
    aiohttp.web.run_app(app, port=9090)

if __name__ == '__main__':
    main()

Sender script (implemented as client):

import asyncio
import aiohttp
import uuid

async def main():
    # can use larger number for increase the chance to hang, but beware of consumer's memory use.
    # with provided constant values it'll peak at ~1.3Gb
    BLOCK_COUNT = 5000
    BLOCK_SIZE_MULTIPLIER = 2**14
    # uuid4 is 16 bytes, 256k block total size
    BLOCK_SIZE = 16 * BLOCK_SIZE_MULTIPLIER
    async with aiohttp.ClientSession() as session:
        # TODO: use your own IP :)
        async with session.ws_connect("http://xxx.xxx.xxx.xxx:9090/ws") as ws:
            for block_idx in range(BLOCK_COUNT):
                await ws.send_bytes(uuid.uuid4().bytes * BLOCK_SIZE_MULTIPLIER)
                print("Block", block_idx + 1, "sent")
    
if __name__ == '__main__':
  asyncio.get_event_loop().run_until_complete(main())

Sender prints everything up to "Block 5000 sent". Consumer often, but not always stops somewhere much eariler (seen 293, 3097 - completely random).

Your environment

• 2 machines, Ubuntu 16.04
• Python 3.6
• aiohttp 2.0.7
• Gbit LAN

[alexmnazarenko]

Has anyone been able to reproduce it yet? I hope I'm not crazy )

[fafhrd91]

I will check in couple days

[alexmnazarenko]

Update: error may happen even on localhost connection, but probability is really low

[alexmnazarenko]

Update2: using uvloop doesn't help either (but it's faster, so failure rate is lower)

[fafhrd91]

I am planing to look into this problem before 2.1 release.

Btw did you try aiohttp from master?

[alexmnazarenko]

Will try it right now, ETA ~15min )

[alexmnazarenko]

Update: with aiohttp-master still hangs and leaks, pretty reliably

[alexmnazarenko]

Any news? 2.1 behaves the same. I really hope to look into it myself at the end of month, but kinda caught in the deadlines crossfire for now.

[fafhrd91]

Sorry, I didn't have enough time. Will try to look into this problem this week.

[pjknkda]

@alexmnazarenko Could you test this issue with #1962? I've tested it and the MR seems to fix the issue in my machine.

[alexmnazarenko]

@pjknkda Works like a charm, tested it on multiple machines ) Thank you!

[alexmnazarenko]

@fafhrd91 Can we expect a release with this fix anytime soon?

[fafhrd91]

yes, but I'd like to check #1955 as well

[kummahiih]

This seems to be security critical. Any idea of how long it takes to fix this?

[asvetlov]

Everything is fixed, at least 3.2 has no such problems

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a [new issue] for related bugs.
If you feel like there's important points made in this discussion, please include those exceprts into that [new issue].
[new issue]: https:/aio-libs/aiohttp/issues/new