Skip to content

Commit

Permalink
Fix errors cascade on Websocket.prepare() error
Browse files Browse the repository at this point in the history
This fix the error cascade that happened when opening the logs endpoint
not as websocket. This was problematic because it returned to the http
client a 500 error instead of the actual error code and message

To reproduce
* start aleph supervisor
* open in Firefox: http://localhost:4020/vm/3fc0aa9569da840c43e7bd2033c3c580abb46b007527d6d20f2d4e98e867f7af
* then in another tab http://localhost:4020/control/machine/3fc0aa9569da840c43e7bd2033c3c580abb46b007527d6d20f2d4e98e867f7af/logs

Error that happened before
2023-11-16 16:16:58,089 | ERROR | Error handling request
Traceback (most recent call last):
  File "/home/ubuntu/remote-aleph/src/aleph/vm/orchestrator/views/operator.py", line 147, in stream_logs
    await ws.prepare(request)
  File "/home/ubuntu/.virtualenvs/aleph-vm/lib/python3.10/site-packages/aiohttp/web_ws.py", line 137, in prepare
    protocol, writer = self._pre_start(request)
  File "/home/ubuntu/.virtualenvs/aleph-vm/lib/python3.10/site-packages/aiohttp/web_ws.py", line 232, in _pre_start
    headers, protocol, compress, notakeover = self._handshake(request)
  File "/home/ubuntu/.virtualenvs/aleph-vm/lib/python3.10/site-packages/aiohttp/web_ws.py", line 149, in _handshake
    raise HTTPBadRequest(
aiohttp.web_exceptions.HTTPBadRequest: Bad Request

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/remote-aleph/src/aleph/vm/orchestrator/views/operator.py", line 157, in stream_logs
    await ws.close()
  File "/home/ubuntu/.virtualenvs/aleph-vm/lib/python3.10/site-packages/aiohttp/web_ws.py", line 337, in close
    raise RuntimeError("Call .prepare() first")
RuntimeError: Call .prepare() first

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/.virtualenvs/aleph-vm/lib/python3.10/site-packages/aiohttp/web_protocol.py", line 433, in _handle_request
    resp = await request_handler(request)
  File "/home/ubuntu/.virtualenvs/aleph-vm/lib/python3.10/site-packages/aiohttp/web_app.py", line 504, in _handle
    resp = await handler(request)
  File "/home/ubuntu/.virtualenvs/aleph-vm/lib/python3.10/site-packages/aiohttp/web_middlewares.py", line 117, in impl
    return await handler(request)
  File "/home/ubuntu/remote-aleph/src/aleph/vm/orchestrator/supervisor.py", line 46, in server_version_middleware
    resp: web.StreamResponse = await handler(request)
  File "/home/ubuntu/remote-aleph/src/aleph/vm/orchestrator/views/operator.py", line 159, in stream_logs
    execution.vm.fvm.log_queues.remove(queue)
ValueError: list.remove(x): x not in list
  • Loading branch information
olethanh authored and hoh committed Nov 17, 2023
1 parent 7d2060e commit 5daf33a
Showing 1 changed file with 3 additions and 4 deletions.
7 changes: 3 additions & 4 deletions src/aleph/vm/orchestrator/views/operator.py
Original file line number Diff line number Diff line change
Expand Up @@ -143,14 +143,12 @@ async def stream_logs(request: web.Request):
queue: asyncio.Queue = asyncio.Queue(maxsize=1000)
try:
ws = web.WebSocketResponse()
await ws.prepare(request)
try:
await ws.prepare(request)

# Limit the number of queues per VM
if len(execution.vm.fvm.log_queues) > 20:
logger.warning("Too many log queues, dropping the oldest one")
execution.vm.fvm.log_queues.pop(0)

execution.vm.fvm.log_queues.append(queue)

while True:
Expand All @@ -161,7 +159,8 @@ async def stream_logs(request: web.Request):
finally:
await ws.close()
finally:
execution.vm.fvm.log_queues.remove(queue)
if queue in execution.vm.fvm.log_queues:
execution.vm.fvm.log_queues.remove(queue)
queue.empty()


Expand Down

0 comments on commit 5daf33a

Please sign in to comment.