Can't connect to redis sentinel

[aspedrosa]

Apache Airflow version

2.7.1

What happened

When running either airflow celery worker or airflow celery flower following error happens

2023-09-28T13:10:32+01:00	[2023-09-28 13:10:32,666: INFO/MainProcess] Connected to sentinel://XXX.XXX.XXX.XXX:XXXX/0
2023-09-28T13:10:32+01:00	�[1;33m[2023-09-28 13:10:32,667: WARNING/MainProcess] /usr/local/lib/python3.9/site-packages/celery/worker/consumer/consumer.py:507: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
2023-09-28T13:10:32+01:00	whether broker connection retries are made during startup in Celery 6.0 and above.
2023-09-28T13:10:32+01:00	If you wish to retain the existing behavior for retrying connections on startup,
2023-09-28T13:10:32+01:00	you should set broker_connection_retry_on_startup to True.
2023-09-28T13:10:32+01:00	  warnings.warn(
2023-09-28T13:10:32+01:00	
2023-09-28T13:10:32+01:00	[2023-09-28 13:10:32,670: INFO/MainProcess] mingle: searching for neighbors
2023-09-28T13:10:33+01:00	[2023-09-28 13:10:33,681: INFO/MainProcess] mingle: all alone
2023-09-28T13:10:33+01:00	[2023-09-28 13:10:33,703: INFO/MainProcess] celery@XXXX-airflow-celery-worker-XXXX-dev-dsv01 ready.
2023-09-28T13:10:33+01:00	�[1;35m[2023-09-28 13:10:33,707: CRITICAL/MainProcess] Unrecoverable error: TypeError("read_response() got an unexpected keyword argument 'push_request'")
2023-09-28T13:10:33+01:00	Traceback (most recent call last):
2023-09-28T13:10:33+01:00	  File "/usr/local/lib/python3.9/site-packages/celery/worker/worker.py", line 202, in start
2023-09-28T13:10:33+01:00	    self.blueprint.start(self)
2023-09-28T13:10:33+01:00	  File "/usr/local/lib/python3.9/site-packages/celery/bootsteps.py", line 116, in start
2023-09-28T13:10:33+01:00	    step.start(parent)
2023-09-28T13:10:33+01:00	  File "/usr/local/lib/python3.9/site-packages/celery/bootsteps.py", line 365, in start
2023-09-28T13:10:33+01:00	    return self.obj.start()
2023-09-28T13:10:33+01:00	  File "/usr/local/lib/python3.9/site-packages/celery/worker/consumer/consumer.py", line 340, in start
2023-09-28T13:10:33+01:00	    blueprint.start(self)
2023-09-28T13:10:33+01:00	  File "/usr/local/lib/python3.9/site-packages/celery/bootsteps.py", line 116, in start
2023-09-28T13:10:33+01:00	    step.start(parent)
2023-09-28T13:10:33+01:00	  File "/usr/local/lib/python3.9/site-packages/celery/worker/consumer/consumer.py", line 742, in start
2023-09-28T13:10:33+01:00	    c.loop(*c.loop_args())
2023-09-28T13:10:33+01:00	  File "/usr/local/lib/python3.9/site-packages/celery/worker/loops.py", line 97, in asynloop
2023-09-28T13:10:33+01:00	    next(loop)
2023-09-28T13:10:33+01:00	  File "/usr/local/lib/python3.9/site-packages/kombu/asynchronous/hub.py", line 373, in create_loop
2023-09-28T13:10:33+01:00	    cb(*cbargs)
2023-09-28T13:10:33+01:00	  File "/usr/local/lib/python3.9/site-packages/kombu/transport/redis.py", line 1341, in on_readable
2023-09-28T13:10:33+01:00	    self.cycle.on_readable(fileno)
2023-09-28T13:10:33+01:00	  File "/usr/local/lib/python3.9/site-packages/kombu/transport/redis.py", line 566, in on_readable
2023-09-28T13:10:33+01:00	    chan.handlers[type]()
2023-09-28T13:10:33+01:00	  File "/usr/local/lib/python3.9/site-packages/kombu/transport/redis.py", line 910, in _receive
2023-09-28T13:10:33+01:00	    ret.append(self._receive_one(c))
2023-09-28T13:10:33+01:00	  File "/usr/local/lib/python3.9/site-packages/kombu/transport/redis.py", line 920, in _receive_one
2023-09-28T13:10:33+01:00	    response = c.parse_response()
2023-09-28T13:10:33+01:00	  File "/usr/local/lib/python3.9/site-packages/redis/client.py", line 796, in parse_response
2023-09-28T13:10:33+01:00	    response = self._execute(conn, try_read)
2023-09-28T13:10:33+01:00	  File "/usr/local/lib/python3.9/site-packages/redis/client.py", line 772, in _execute
2023-09-28T13:10:33+01:00	    return conn.retry.call_with_retry(
2023-09-28T13:10:33+01:00	  File "/usr/local/lib/python3.9/site-packages/redis/retry.py", line 46, in call_with_retry
2023-09-28T13:10:33+01:00	    return do()
2023-09-28T13:10:33+01:00	  File "/usr/local/lib/python3.9/site-packages/redis/client.py", line 773, in <lambda>
2023-09-28T13:10:33+01:00	    lambda: command(*args, **kwargs),
2023-09-28T13:10:33+01:00	  File "/usr/local/lib/python3.9/site-packages/redis/client.py", line 794, in try_read
2023-09-28T13:10:33+01:00	    return conn.read_response(disconnect_on_error=False, push_request=True)
2023-09-28T13:10:33+01:00	TypeError: read_response() got an unexpected keyword argument 'push_request'
2023-09-28T13:10:34+01:00	[2023-09-28 13:10:34 +0100] [23] [INFO] Handling signal: term
2023-09-28T13:10:34+01:00	[2023-09-28 13:10:34 +0100] [25] [INFO] Worker exiting (pid: 25)
2023-09-28T13:10:34+01:00	[2023-09-28 13:10:34 +0100] [24] [INFO] Worker exiting (pid: 24)
2023-09-28T13:10:34+01:00	[2023-09-28 13:10:34 +0100] [23] [INFO] Shutting down: Master

What you think should happen instead

This can be fixed by pinning redis to 5.0.1, which includes the following fix redis/redis-py#2922, in the 2.7 constraints files.

How to reproduce

Use redis sentinel as a broker.

I had the following setting
broker_url = sentinel://X.X.X.X:26379/0;sentinel://X.X.X.X:26379/0;sentinel://X.X.X.X:26379/0

Operating System

Container with Oracle Linux 8

Versions of Apache Airflow Providers

No response

Deployment

Other Docker-based deployment

Deployment details

Custom image where we install airflow with pip install apache-airflow[redis,celery]==1.7.1 --contrains https://raw.githubusercontent.com/apache/airflow/constraints-2.7.1/constraints-3.9.txt

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[Taragolis]

This limitation comes from apache-airflow-providers-redis.
There is no changes in constraints after airflow version released (only some critical things), so if you do not intend to use redis provider you need to follow general recommendation and run:

pip install "apache-airflow==2.7.1" "redis==5.0.1"

[Taragolis]

airflow/airflow/providers/redis/provider.yaml

Lines 41 to 47 in 21990ed

	dependencies:
	- apache-airflow>=2.4.0
	# We limit redis to <5.0.0 because of incompatibility with celery. Both Celery and Kombu limited it
	# and deferred fixing it for later, we should bump the limit once they do. Also !=4.5.5 matches celery
	# limits and prevents installing 4.5.5 which is broken.
	# https:/celery/celery/pull/8442, https:/celery/kombu/pull/1776
	- redis>=4.5.2,<5.0.0,!=4.5.5