Python 3.11: '_Code' object has no attribute 'co_positions' #377

Yun-Kim · 2022-10-11T17:55:06Z

I have a celery task that raises an exception:

@self.app.task
def fn_exception():
    raise Exception("Task class is failing")

When the task is applied, instead of being caught and sending a task failure signal, I get an attribute error from a biliard object: AttributeError: '_Code' object has no attribute 'co_positions'. co_positions is new in Python 3.11 for codeobjects, so I'm wondering if this is a billiards compatibility problem with Python 3.11 or if it's something else entirely with celery.

Here is the traceback:

.riot/venv_py3110_celery~505_redis~35/lib/python3.11/site-packages/celery/app/task.py:773: in apply
    ret = tracer(task_id, args, kwargs, request)
.riot/venv_py3110_celery~505_redis~35/lib/python3.11/site-packages/celery/app/trace.py:422: in trace_task
    I, R, state, retval = on_error(task_request, exc, uuid)
.riot/venv_py3110_celery~505_redis~35/lib/python3.11/site-packages/celery/app/trace.py:358: in on_error
    R = I.handle_error_state(
.riot/venv_py3110_celery~505_redis~35/lib/python3.11/site-packages/celery/app/trace.py:165: in handle_error_state
    return {
.riot/venv_py3110_celery~505_redis~35/lib/python3.11/site-packages/celery/app/trace.py:224: in handle_failure
    self._log_error(task, req, einfo)
.riot/venv_py3110_celery~505_redis~35/lib/python3.11/site-packages/celery/app/trace.py:252: in _log_error
    logger.log(policy.severity, policy.format.strip(), context,
../.pyenv/versions/3.11-dev/lib/python3.11/logging/__init__.py:1559: in log
    self._log(level, msg, args, **kwargs)
../.pyenv/versions/3.11-dev/lib/python3.11/logging/__init__.py:1634: in _log
    self.handle(record)
../.pyenv/versions/3.11-dev/lib/python3.11/logging/__init__.py:1644: in handle
    self.callHandlers(record)
../.pyenv/versions/3.11-dev/lib/python3.11/logging/__init__.py:1706: in callHandlers
    hdlr.handle(record)
../.pyenv/versions/3.11-dev/lib/python3.11/logging/__init__.py:978: in handle
    self.emit(record)
.riot/venv_py3110_mock_pytest700_pytest-mock_coverage_pytest-cov_opentracing_hypothesis/lib/python3.11/site-packages/_pytest/logging.py:331: in emit
    super().emit(record)
../.pyenv/versions/3.11-dev/lib/python3.11/logging/__init__.py:1118: in emit
    self.handleError(record)
../.pyenv/versions/3.11-dev/lib/python3.11/logging/__init__.py:1110: in emit
    msg = self.format(record)
../.pyenv/versions/3.11-dev/lib/python3.11/logging/__init__.py:953: in format
    return fmt.format(record)
.riot/venv_py3110_mock_pytest700_pytest-mock_coverage_pytest-cov_opentracing_hypothesis/lib/python3.11/site-packages/_pytest/logging.py:92: in format
    return super().format(record)
../.pyenv/versions/3.11-dev/lib/python3.11/logging/__init__.py:695: in format
    record.exc_text = self.formatException(record.exc_info)
../.pyenv/versions/3.11-dev/lib/python3.11/logging/__init__.py:645: in formatException
    traceback.print_exception(ei[0], ei[1], tb, None, sio)
../.pyenv/versions/3.11-dev/lib/python3.11/traceback.py:124: in print_exception
    te = TracebackException(type(value), value, tb, limit=limit, compact=True)
../.pyenv/versions/3.11-dev/lib/python3.11/traceback.py:688: in __init__
    self.stack = StackSummary._extract_from_extended_frame_gen(
../.pyenv/versions/3.11-dev/lib/python3.11/traceback.py:416: in _extract_from_extended_frame_gen
    for f, (lineno, end_lineno, colno, end_colno) in frame_gen:
../.pyenv/versions/3.11-dev/lib/python3.11/traceback.py:353: in _walk_tb_with_full_positions
    positions = _get_code_position(tb.tb_frame.f_code, tb.tb_lasti)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

code = <billiard.einfo._Code object at 0x7f4db0e40a10>, instruction_index = 428

    def _get_code_position(code, instruction_index):
        if instruction_index < 0:
            return (None, None, None, None)
>       positions_gen = code.co_positions()
E       AttributeError: '_Code' object has no attribute 'co_positions'

Note: I am running billard==3.6.4.0, celery==5.2.7, redis==3.5 with Python 3.11.

The text was updated successfully, but these errors were encountered:

auvipy · 2022-10-12T07:22:29Z

try to down grade your python or use celery 5.3.0b1 with billiard 4.0.2

Yun-Kim · 2022-10-12T18:16:14Z

Hi @auvipy, I tried using celery==5.3.0b1, billiard==4.0.2 but am getting the same error 😢

I also checked billiard.einfo.py with version billiard==4.0.2 but it looks like the _Code class doesn't have the attribute co_positions as required by Python's traceback module. Will this be added anytime soon? Thanks!

auvipy · 2022-10-13T08:16:26Z

you can contribute to fix that

The new property `co_postions` needs to be pulled through for Exception handling in python 3.11 to work. Addresses celery#377

Yun-Kim · 2022-11-07T18:08:17Z

@eltonkl thanks for getting the fix! @auvipy When can we expect a release with the fix included?

auvipy · 2022-11-08T04:05:24Z

ping me again if not released in a week

Yun-Kim · 2022-11-15T17:18:51Z

@auvipy do you have any updates?

Celery breaks with an error. See celery/billiard#377

* Dependencies versions updated Removed tests for `celery<5`. Support for these expired Aug 2021. https://docs.celeryq.dev/en/master/history/whatsnew-5.1.html#long-term-support-policy Remove support for Python 3.7, as newest django (4.1) supports Python 3.8+ only * PostgreSQL version updated to 11 Django 4.1 support PostgreSQL versions 11+ * Default python version set to 3.8 * Python 3.11 not supported yet Celery breaks with an error. See celery/billiard#377

fredley · 2022-11-22T11:46:02Z

@auvipy another week, another ping (do say if you'd rather not be reminded).

BoPeng · 2022-12-02T20:43:45Z

Encountered the same problem so pinging @auvipy. :-)

freemindcore · 2022-12-10T03:21:19Z

@auvipy another week, another ping (do say if you'd rather not be reminded).
Thanks!

thedrow · 2022-12-13T15:52:47Z

@auvipy We can release a new version of billiard without releasing Kombu/Celery yet.

auvipy · 2022-12-13T16:43:49Z

@auvipy We can release a new version of billiard without releasing Kombu/Celery yet.

100%

auvipy · 2022-12-13T16:44:11Z

I will release ASAP

auvipy · 2022-12-14T12:25:02Z

released a new version

masarliev · 2022-12-14T12:29:01Z

@auvipy celery 5.2.7 requireds billiard<4.0. Can you release 3.6.5.0(may be) with this fix?

auvipy · 2022-12-14T13:03:59Z

this release is for celery 5.3 mainly. we still do not back port python 3.11 support to celery 5.2.x

The celery tests failed on Python 3.11. This is most likely due to this issue in billiard, a celery dependency, about it not working on Python 3.11 because of the error reported in the CI: celery/billiard#377 It's been fixed in billiard 4.1.0, but celery is locked on billiard version lower than 4, so it cannot use this version with the fix. This issue does not arise on the Docker tests, because they use Python 3.9.16. I've moved the error test span event assertions to the error test that is available in the functional tests, and removed the unit test. That way, the build will run successfully.

* Add tests for errors in Celery tasks I noticed there were no tests for the error scenario in the Celery package. This commit adds a basic test, based on the previous test and how I see other packages test the error status on the span. Part of #987 * Record exception in Celery instrumentation In addition to setting the status on the span, also record the exception on the span. This adds an event to the span with more details about the error, following the format other instrumentations also use. * Update CHANGELOG with Celery record exception * Fix lint code formatting issues * Move Celery error tests to the functional tests The celery tests failed on Python 3.11. This is most likely due to this issue in billiard, a celery dependency, about it not working on Python 3.11 because of the error reported in the CI: celery/billiard#377 It's been fixed in billiard 4.1.0, but celery is locked on billiard version lower than 4, so it cannot use this version with the fix. This issue does not arise on the Docker tests, because they use Python 3.9.16. I've moved the error test span event assertions to the error test that is available in the functional tests, and removed the unit test. That way, the build will run successfully. * Remove duplicate entry in changelog This was added in a recent merge commit on this PR branch. * Remove unused test code With the move of the tests for tasks with errors to the functional tests, remove the unit test's error task and unused imports. --------- Co-authored-by: Srikanth Chekuri <[email protected]>

sparrowt · 2023-03-07T16:16:22Z

released a new version

That's https:/celery/billiard/releases/tag/v4.1.0 for anyone else who hit this and is wondering how to get the fix.

However...

this release is for celery 5.3 mainly. we still do not back port python 3.11 support to celery 5.2.x

https://pypi.org/project/celery/#history suggests that all celery 5.3.x versions are still pre-release, so from your comment above it seems that there is no support (or plan for it) for python 3.11 in any current official 'release' of celery? 😢

auvipy · 2023-03-08T06:30:14Z

that can be used with 5.3beta versions as of now

sparrowt · 2023-03-13T10:06:34Z

Yes I see that, however we do not plan to use a beta version of celery in production - it's a shame that the current 5.2.x series is being left behind in terms of python 3.11 support.

Nusnus · 2023-03-13T10:25:34Z

Yes I see that, however we do not plan to use a beta version of celery in production - it's a shame that the current 5.2.x series is being left behind in terms of python 3.11 support.

I understand the need for Python 3.11 support my friend @sparrowt, and it will come, but we first need to concentrate on releasing the official Celery 5.3 release (no beta, no pre-releases, official), so we can close the circle on that aspect and move on forward.
As much as Python 3.11 support would be awesome, releasing a stable 5.3 version is much higher on the list of priorities.

I am sure @auvipy and myself would happy to support any contribution towards adding Python 3.11, but at the moment our hands are full and we cannot depreioritize other issues to push this instead.

christianbundy · 2023-04-03T22:50:12Z

How to live dangerously (i.e. Python 3.11 with Celery 5.2):

diff --git a/.venv/lib/python3.11/site-packages/billiard/einfo.py b/.venv/lib/python3.11/site-packages/billiard/einfo.py
index 236f400c7..84ff7529d 100644
--- a/.venv/lib/python3.11/site-packages/billiard/einfo.py
+++ b/.venv/lib/python3.11/site-packages/billiard/einfo.py
@@ -24,6 +24,13 @@ class _Code(object):
         self.co_nlocals = code.co_nlocals
         self.co_stacksize = code.co_stacksize
         self.co_varnames = ()
+        if sys.version_info >= (3, 11):
+            self._co_positions = list(code.co_positions())
+
+    if sys.version_info >= (3, 11):
+        @property
+        def co_positions(self):
+            return self._co_positions.__iter__


 class _Frame(object):

THE SOFTWARE IS PROVIDED “AS IS” AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

Yun-Kim mentioned this issue Oct 14, 2022

Support Python 3.11 DataDog/dd-trace-py#4149

Closed

imjoehaines mentioned this issue Oct 26, 2022

Add Python 3.11 to CI bugsnag/bugsnag-python#324

Merged

fredley added a commit to fredley/billiard that referenced this issue Nov 1, 2022

Add Code.co_positions for python 3.11 support

07a0975

The new property `co_postions` needs to be pulled through for Exception handling in python 3.11 to work. Addresses celery#377

fredley mentioned this issue Nov 1, 2022

Add Code.co_positions for python 3.11 support #378

Closed

eltonkl mentioned this issue Nov 1, 2022

Add missing attribute co_positions for Python 3.11 #379

Merged

maciej-gol added a commit to maciej-gol/tenant-schemas-celery that referenced this issue Nov 18, 2022

Python 3.11 not supported yet

59522db

Celery breaks with an error. See celery/billiard#377

auvipy closed this as completed Dec 14, 2022

github-actions bot mentioned this issue Dec 14, 2022

Upstream issue celery/billiard#377 closed DataDog/dd-trace-py#4788

Closed

rgangopadhya mentioned this issue Apr 3, 2023

Release 5.3 celery/celery#7605

Closed

12 tasks

jzmiller1 mentioned this issue May 4, 2023

python 3.11 support celery/celery#8227

Closed

4 tasks

sentrivana mentioned this issue May 8, 2023

AttributeError: '_Code' object has no attribute 'co_positions' getsentry/sentry-python#1798

Closed

mfocko mentioned this issue May 10, 2023

Sentry reporting tracebacks in pod logs with AttributeError: '_Code' object has no attribute 'co_positions' packit/packit-service#1872

Closed

grzanka mentioned this issue Jun 5, 2023

Python 3.11 tests are silently failing yaptide/yaptide#425

Closed

u1735067 mentioned this issue Jul 18, 2023

[1.6] Add support for Python 3.11, drop support for Python 3.7 nautobot/nautobot#4118

Merged

6 tasks

marcuspp mentioned this issue Aug 8, 2023

ICMSLST-2135 - Gov Notify uktrade/icms#1149

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python 3.11: '_Code' object has no attribute 'co_positions' #377

Python 3.11: '_Code' object has no attribute 'co_positions' #377

Yun-Kim commented Oct 11, 2022

auvipy commented Oct 12, 2022

Yun-Kim commented Oct 12, 2022

auvipy commented Oct 13, 2022

Yun-Kim commented Nov 7, 2022

auvipy commented Nov 8, 2022

Yun-Kim commented Nov 15, 2022

fredley commented Nov 22, 2022

BoPeng commented Dec 2, 2022

freemindcore commented Dec 10, 2022

thedrow commented Dec 13, 2022

auvipy commented Dec 13, 2022

auvipy commented Dec 13, 2022

auvipy commented Dec 14, 2022

masarliev commented Dec 14, 2022

auvipy commented Dec 14, 2022

sparrowt commented Mar 7, 2023 •

edited

Loading

auvipy commented Mar 8, 2023

sparrowt commented Mar 13, 2023

Nusnus commented Mar 13, 2023 •

edited

Loading

christianbundy commented Apr 3, 2023

Python 3.11: '_Code' object has no attribute 'co_positions' #377

Python 3.11: '_Code' object has no attribute 'co_positions' #377

Comments

Yun-Kim commented Oct 11, 2022

auvipy commented Oct 12, 2022

Yun-Kim commented Oct 12, 2022

auvipy commented Oct 13, 2022

Yun-Kim commented Nov 7, 2022

auvipy commented Nov 8, 2022

Yun-Kim commented Nov 15, 2022

fredley commented Nov 22, 2022

BoPeng commented Dec 2, 2022

freemindcore commented Dec 10, 2022

thedrow commented Dec 13, 2022

auvipy commented Dec 13, 2022

auvipy commented Dec 13, 2022

auvipy commented Dec 14, 2022

masarliev commented Dec 14, 2022

auvipy commented Dec 14, 2022

sparrowt commented Mar 7, 2023 • edited Loading

auvipy commented Mar 8, 2023

sparrowt commented Mar 13, 2023

Nusnus commented Mar 13, 2023 • edited Loading

christianbundy commented Apr 3, 2023

sparrowt commented Mar 7, 2023 •

edited

Loading

Nusnus commented Mar 13, 2023 •

edited

Loading