Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RuntimeError: no running event loop #8555

Closed
1 task done
tomplus opened this issue Jul 30, 2024 · 16 comments · Fixed by #8558
Closed
1 task done

RuntimeError: no running event loop #8555

tomplus opened this issue Jul 30, 2024 · 16 comments · Fixed by #8558
Labels
Milestone

Comments

@tomplus
Copy link

tomplus commented Jul 30, 2024

Describe the bug

It looks like a breaking change in the latest version.

In version 3.9.5 it works:

import aiohttp
connector = aiohttp.TCPConnector()

(in a more complex example I get a warning (DeprecationWarning: The object should be created within an async function))

but in 3.10 I get an exception:

Traceback (most recent call last):
  File "/tmp/a1.py", line 5, in <module>
    connector = aiohttp.TCPConnector()
  File "/home/tomek/work/github/kubernetes-client/kubernetes_asyncio-new/VENV/lib/python3.10/site-packages/aiohttp/connector.py", line 784, in __init__
    super().__init__(
  File "/home/tomek/work/github/kubernetes-client/kubernetes_asyncio-new/VENV/lib/python3.10/site-packages/aiohttp/connector.py", line 234, in __init__
    loop = loop or asyncio.get_running_loop()
RuntimeError: no running event loop

To Reproduce

code

import aiohttp
connector = aiohttp.TCPConnector()

Expected behavior

If the change is intentional it would be very useful to describe that in a changelog.
If not I expect the same behavior like in the previous version.

Logs/tracebacks

Traceback (most recent call last):
  File "/tmp/a1.py", line 5, in <module>
    connector = aiohttp.TCPConnector()
  File "/home/tomek/work/github/kubernetes-client/kubernetes_asyncio-new/VENV/lib/python3.10/site-packages/aiohttp/connector.py", line 784, in __init__
    super().__init__(
  File "/home/tomek/work/github/kubernetes-client/kubernetes_asyncio-new/VENV/lib/python3.10/site-packages/aiohttp/connector.py", line 234, in __init__
    loop = loop or asyncio.get_running_loop()
RuntimeError: no running event loop


### Python Version

```console
$ python --version
3.10

aiohttp Version

$ python -m pip show aiohttp
Name: aiohttp
Version: 3.10.0

multidict Version

$ python -m pip show multidict
Name: multidict
Version: 6.0.4

yarl Version

$ python -m pip show yarl
Name: yarl
Version: 1.8.2

OS

Linux

Related component

Client

Additional context

No response

Code of Conduct

  • I agree to follow the aio-libs Code of Conduct
@tomplus tomplus added the bug label Jul 30, 2024
@bdraco
Copy link
Member

bdraco commented Jul 30, 2024

It looks like this is caused by #8512 which is a backport of #5278 from master

@bdraco
Copy link
Member

bdraco commented Jul 30, 2024

It looks like it was deprecated in 3.9.5 and now no longer works in 3.10.0 https:/aio-libs/aiohttp/blob/3.9/aiohttp/helpers.py#L296 but #5278 didn't have a changelog message to say it would stop working.

In 3.9.5 it would create a new non-running event loop

>>> import aiohttp
>>> print(aiohttp.__version__)
3.9.5
>>> connector = aiohttp.TCPConnector()
>>> print(connector._loop)
<_UnixSelectorEventLoop running=False closed=False debug=False>
>>> 

@bdraco
Copy link
Member

bdraco commented Jul 30, 2024

Based on the above, I'd say the change is intentional but could use a changelog entry.

@bdraco
Copy link
Member

bdraco commented Jul 30, 2024

Creating a ClientSession, CookieJar, or Connector without a running asyncio event loop will now explicitly raise in 3.10.0 because it calls asyncio.get_running_loop()

@aslonnie
Copy link

is this going to be fixed in 3.10.1 ? or is this breakage intentional?

@bdraco
Copy link
Member

bdraco commented Jul 31, 2024

The only planned change is to improve the changelog. As it was deprecated over 5 years ago, there is no plan to extend the deprecation period.

Edit: This practice is deprecated in Python as well

https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.get_event_loop
Screenshot 2024-08-01 at 2 27 12 PM

@TheDurableDane
Copy link

This is a breaking change (intentional or not), so shouldn't the version be bumped to 4.0.0? Regardless of the length of the deprecation period.

@webknjaz
Copy link
Member

webknjaz commented Aug 1, 2024

https://hynek.me/articles/semver-will-not-save-you/#taking-responsibility

@TheDurableDane
Copy link

@webknjaz That's a bit passive-aggressive. I know I shouldn't blindly trust any package I use, but sematic versioning best pratices would be a nice touch for such a widely used package as aiohttp. I don't think you are excused just because you can link to a blog post that says it's the user's fault.

@bdraco
Copy link
Member

bdraco commented Aug 1, 2024

I thought about a partial revert of #8512 (originally #5278) on 3.10/3.11, but I'm not sold on that idea because Python has deprecated calling asyncio.get_event_loop() without a running event loop. After all, the behavior is confusing. Even if we did revert this, it's kicking the can down the road until it becomes an error.

https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.get_event_loop
Screenshot 2024-08-01 at 2 27 12 PM

can-anyscale pushed a commit to ray-project/ray that referenced this issue Aug 1, 2024
aiohttp's new 3.10.0 stops supporting creating ClientSession in non async context any more.

aio-libs/aiohttp#8555 (comment)

---------

Signed-off-by: Rui Qiao <[email protected]>
@webknjaz
Copy link
Member

webknjaz commented Aug 2, 2024

@TheDurableDane you're right the end-users should be able to rely on the documented expectations. Could you point out where we promise that the first version component alone corresponds to SemVer's major version? My observation is that major has always two components.
Besides, SemVer is talking about public APIs and my understanding is that those haven't changed.
And finally, people who make releases may interpret things differently, compared to those consuming them. I keep seeing this problem where the users invent something in their heads and think this always matches the mental models of others. That's not how it works. SemVer by its nature ultimately represents how the maintainers see changes, not necessarily how every single user perceive them.

@Dreamsorcerer
Copy link
Member

Just to add to bdraco's conclusion, I think the old code may have been dangerously wrong in some situations.
If you create TCPConnector as a global, the get_event_loop() call would create a new loop and store the reference for later.
If you then run your application with asyncio.run() or web.run_app(), it will create a new loop to execute the application in.
You then have your application running in one loop and TCPConnector trying to do callbacks etc. in another loop which isn't even running.

@bdraco
Copy link
Member

bdraco commented Aug 2, 2024

0001-Restore-the-ability-to-create-objects-without-a-runn.patch

Attached is a patch for anyone who arrives at this issue in the future because a security
issue requires them to update to 3.10.x or later and needs a quick solution until they
can adjust their object creation.

@kyrlon
Copy link

kyrlon commented Sep 10, 2024

0001-Restore-the-ability-to-create-objects-without-a-runn.patch

Attached is a patch for anyone who arrives at this issue in the future because a security issue requires them to update to 3.10.x or later and needs a quick solution until they can adjust their object creation.

Can confirm this works on my Windows 10 machine:

OS: Windows 10
Python version: Python3.12
aiohttp==3.10.5

however got this working by manual alterations.

  1. I cannot for the life of me build from source (or pip install via py -m pip install git+https:/aio-libs/aiohttp.git), so that was my first issue
Note: Here is the log on the build attempt...

Collecting git+https:/aio-libs/aiohttp.git
  Cloning https:/aio-libs/aiohttp.git to c:\users\kyrlon\appdata\local\temp\1\pip-req-build-n53k6a8a
  Running command git clone --filter=blob:none --quiet https:/aio-libs/aiohttp.git 'C:\Users\kyrlon\AppData\Local\Temp\1\pip-req-build-n53k6a8a'
  Resolved https:/aio-libs/aiohttp.git to commit ffcf9dc4ea157adc5b7b5b31b6cc69f37d533122
  Running command git submodule update --init --recursive -q
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting aiohappyeyeballs>=2.3.0 (from aiohttp==4.0.0a2.dev0)
  Using cached aiohappyeyeballs-2.4.0-py3-none-any.whl (12 kB)
Collecting aiosignal>=1.1.2 (from aiohttp==4.0.0a2.dev0)
  Using cached aiosignal-1.3.1-py3-none-any.whl (7.6 kB)
Collecting frozenlist>=1.1.1 (from aiohttp==4.0.0a2.dev0)
  Using cached frozenlist-1.4.1-cp311-cp311-win_amd64.whl (50 kB)
Collecting multidict<7.0,>=4.5 (from aiohttp==4.0.0a2.dev0)
  Using cached multidict-6.1.0-cp311-cp311-win_amd64.whl (28 kB)
Collecting yarl<2.0,>=1.11.0 (from aiohttp==4.0.0a2.dev0)
  Using cached yarl-1.11.1-cp311-cp311-win_amd64.whl (110 kB)
Requirement already satisfied: idna>=2.0 in c:\users\kyrlon\appdata\local\programs\python\python311\lib\site-packages (from yarl<2.0,>=1.11.0->aiohttp==4.0.0a2.dev0) (3.7)
Building wheels for collected packages: aiohttp
  Building wheel for aiohttp (pyproject.toml): started
  Building wheel for aiohttp (pyproject.toml): finished with status 'error'
  error: subprocess-exited-with-error

Building wheel for aiohttp (pyproject.toml) did not run successfully.
exit code: 1

[90 lines of output]


  • Accelerated build *

running bdist_wheel
running build
running build_py
creating build
creating build\lib.win-amd64-cpython-311
creating build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\abc.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\base_protocol.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\client.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\client_exceptions.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\client_proto.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\client_reqrep.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\client_ws.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\compression_utils.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\connector.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\cookiejar.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\formdata.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\hdrs.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\helpers.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\http.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\http_exceptions.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\http_parser.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\http_websocket.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\http_writer.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\locks.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\log.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\multipart.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\payload.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\pytest_plugin.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\resolver.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\streams.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\tcp_helpers.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\test_utils.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\tracing.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\typedefs.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\web.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\web_app.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\web_exceptions.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\web_fileresponse.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\web_log.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\web_middlewares.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\web_protocol.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\web_request.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\web_response.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\web_routedef.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\web_runner.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\web_server.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\web_urldispatcher.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\web_ws.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\worker.py -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp_init_.py -> build\lib.win-amd64-cpython-311\aiohttp
running egg_info
writing aiohttp.egg-info\PKG-INFO
writing dependency_links to aiohttp.egg-info\dependency_links.txt
writing requirements to aiohttp.egg-info\requires.txt
writing top-level names to aiohttp.egg-info\top_level.txt
reading manifest file 'aiohttp.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'aiohttp' anywhere in distribution
warning: no previously-included files matching '.pyc' found anywhere in distribution
warning: no previously-included files matching '
.pyd' found anywhere in distribution
warning: no previously-included files matching '.so' found anywhere in distribution
warning: no previously-included files matching '
.lib' found anywhere in distribution
warning: no previously-included files matching '.dll' found anywhere in distribution
warning: no previously-included files matching '
.a' found anywhere in distribution
warning: no previously-included files matching '*.obj' found anywhere in distribution
warning: no previously-included files found matching 'aiohttp*.html'
no previously-included directories found matching 'docs_build'
adding license file 'LICENSE.txt'
writing manifest file 'aiohttp.egg-info\SOURCES.txt'
copying aiohttp_cparser.pxd -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp_find_header.pxd -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp_helpers.pyi -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp_helpers.pyx -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp_http_parser.pyx -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp_http_writer.pyx -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp_websocket.pyx -> build\lib.win-amd64-cpython-311\aiohttp
copying aiohttp\py.typed -> build\lib.win-amd64-cpython-311\aiohttp
running build_ext
building 'aiohttp._websocket' extension
creating build\temp.win-amd64-cpython-311
creating build\temp.win-amd64-cpython-311\Release
creating build\temp.win-amd64-cpython-311\Release\aiohttp
"C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.40.33807\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -IC:\Users\kyrlon\AppData\Local\Programs\Python\Python311\include -IC:\Users\kyrlon\AppData\Local\Programs\Python\Python311\Include "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.40.33807\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\VS\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\cppwinrt" /Tcaiohttp/_websocket.c /Fobuild\temp.win-amd64-cpython-311\Release\aiohttp/_websocket.obj
_websocket.c
c1: fatal error C1083: Cannot open source file: 'aiohttp/_websocket.c': No such file or directory
error: command 'C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.40.33807\bin\HostX86\x64\cl.exe' failed with exit code 2
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for aiohttp
Failed to build aiohttp
ERROR: Could not build wheels for aiohttp, which is required to install pyproject.toml-based projects

[notice] A new release of pip is available: 23.1.2 -> 24.2
[notice] To update, run: python.exe -m pip install --upgrade pip

  1. At the time of this post, I was using aiohttp==3.10.5, so when I did attempt to patch, it failed. Manual modifications was a slow and patient process (inspecting the patch with a text editor), but afterwards was able run such an example on my current system mentioned from this article:
import sys
import os
import json
import asyncio
import aiohttp


# Initialize connection pool
conn = aiohttp.TCPConnector(limit_per_host=100, limit=0, ttl_dns_cache=300)
PARALLEL_REQUESTS = 100
results = []
urls = ['https://jsonplaceholder.typicode.com/todos/1' for i in range(4000)] #array of urls

async def gather_with_concurrency(n):
    semaphore = asyncio.Semaphore(n)
    session = aiohttp.ClientSession(connector=conn)

    # heres the logic for the generator
    async def get(url):
        async with semaphore:
            async with session.get(url, ssl=False) as response:
                obj = json.loads(await response.read())
                results.append(obj)
    await asyncio.gather(*(get(url) for url in urls))
    await session.close()
if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    loop.run_until_complete(gather_with_concurrency(PARALLEL_REQUESTS))
    conn.close()

    print(f"Completed {len(urls)} requests with {len(results)} results")

@zeocax
Copy link

zeocax commented Sep 23, 2024

This may help

import aiohttp
import asyncio

async def create_session():
    return aiohttp.ClientSession()

aiohttp_client_session = asyncio.get_event_loop().run_until_complete(create_session())

@Dreamsorcerer
Copy link
Member

As mentioned above, very likely to break your application (now or in the future).

anoadragon453 added a commit to matrix-org/sygnal that referenced this issue Oct 2, 2024
Since `aiohttp==0.20.0`, instantiating `aiohttp.ClientSession` must be
done from an async function. Calling this from a non-async function
relied on deprecated behaviour in Python, which broke in recent
releases.
aio-libs/aiohttp#8555 (comment)
explains the situation.

Rather than pin `aiohttp` to an older version, let's just fix the
behaviour.
anoadragon453 added a commit to matrix-org/sygnal that referenced this issue Oct 2, 2024
Since `aiohttp==3.10.0`, instantiating `aiohttp.ClientSession` must be
done from an async function. Calling this from a non-async function
relied on deprecated behaviour in Python, which broke in recent
releases.
aio-libs/aiohttp#8555 (comment)
explains the situation.

Rather than pin `aiohttp` to an older version, let's just fix the
behaviour.
anoadragon453 added a commit to matrix-org/sygnal that referenced this issue Oct 2, 2024
Since `aiohttp==3.10.0`, instantiating `aiohttp.ClientSession` must be
done from an async function. Calling this from a non-async function
relied on deprecated behaviour in Python, which broke in recent
releases.
aio-libs/aiohttp#8555 (comment)
explains the situation.

Rather than pin `aiohttp` to an older version, let's just fix the
behaviour.
anoadragon453 added a commit to matrix-org/sygnal that referenced this issue Oct 2, 2024
Since `aiohttp==3.10.0`, instantiating `aiohttp.ClientSession` must be
done from an async function. Calling this from a non-async function
relied on deprecated behaviour in Python, which broke in recent
releases.
aio-libs/aiohttp#8555 (comment)
explains the situation.

Rather than pin `aiohttp` to an older version, let's just fix the
behaviour.
anoadragon453 added a commit to matrix-org/sygnal that referenced this issue Oct 2, 2024
Since `aiohttp==3.10.0`, instantiating `aiohttp.ClientSession` must be
done from an async function. Calling this from a non-async function
relied on deprecated behaviour in Python, which broke in recent
releases.
aio-libs/aiohttp#8555 (comment)
explains the situation.

Rather than pin `aiohttp` to an older version, let's just fix the
behaviour.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

8 participants