[build] Fix base OS compilation issue caused by incompatibility of urllib with requests. #2830

oleksandrivantsiv · 2023-05-05T10:51:53Z

What I did

Fix SONiC image compilation issue:

[2023-05-03T17:54:00.146Z] + sudo cp files/build_templates/docker_image_ctl.j2 ./fs********-mellanox/usr/share/sonic/templates/docker_image_ctl.j2
[2023-05-03T17:54:00.146Z] + sudo LANG=C DOCKER_HOST= ch******** ./fs********-mellanox /usr/local/bin/generate_shutdown_order.py
[2023-05-03T17:54:00.146Z] Traceback (most recent call last):
[2023-05-03T17:54:00.146Z]   File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 214, in _retrieve_server_version
[2023-05-03T17:54:00.146Z]     return self.version(api_version=False)["ApiVersion"]
[2023-05-03T17:54:00.147Z]   File "/usr/local/lib/python3.9/dist-packages/docker/api/daemon.py", line 181, in version
[2023-05-03T17:54:00.147Z]     return self._result(self._get(url), json=True)
[2023-05-03T17:54:00.147Z]   File "/usr/local/lib/python3.9/dist-packages/docker/utils/decorators.py", line 46, in inner
[2023-05-03T17:54:00.147Z]     return f(self, *args, **kwargs)
[2023-05-03T17:54:00.147Z]   File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 237, in _get
[2023-05-03T17:54:00.147Z]     return self.get(url, **self._set_request_timeout(kwargs))
[2023-05-03T17:54:00.147Z]   File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 600, in get
[2023-05-03T17:54:00.147Z]     return self.request("GET", url, **kwargs)
[2023-05-03T17:54:00.147Z]   File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 587, in request
[2023-05-03T17:54:00.147Z]     resp = self.send(prep, **send_kwargs)
[2023-05-03T17:54:00.147Z]   File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 701, in send
[2023-05-03T17:54:00.147Z]     r = adapter.send(request, **kwargs)
[2023-05-03T17:54:00.147Z]   File "/usr/local/lib/python3.9/dist-packages/requests/adapters.py", line 486, in send
[2023-05-03T17:54:00.147Z]     resp = conn.urlopen(
[2023-05-03T17:54:00.147Z]   File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 790, in urlopen
[2023-05-03T17:54:00.147Z]     response = self._make_request(
[2023-05-03T17:54:00.147Z]   File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 496, in _make_request
[2023-05-03T17:54:00.147Z]     conn.request(
[2023-05-03T17:54:00.147Z] TypeError: request() got an unexpected keyword argument 'chunked'

How I did it

Pin urllib to version < 2 as it was suggested in docker/docker-py#3113.

How to verify it

Run SONiC image compilation.

Previous command output (if the output of a command-line utility has changed)

New command output (if the output of a command-line utility has changed)

liushilongbuaa · 2023-05-05T11:54:42Z

It didn't work.

…requests >= 2.29.0. - Pin urllib to older version from 2.0.0 Signed-off-by: Oleksandr Ivantsiv <[email protected]> Co-authored-by: [email protected]

oleksandrivantsiv · 2023-05-05T13:25:39Z

@liushilongbuaa now it is fixed

anamehra · 2023-05-05T17:57:31Z

We are seeing this failure on 202205 as well, please add the fix there too.
@abdosi , FYI.

qiluo-msft · 2023-05-05T21:27:12Z

I see a new release hours ago: https:/docker/docker-py/releases/tag/6.1.0
Could you check whether it really "Support for requests 2.29.0+ and urllib3 2.x"?

oleksandrivantsiv · 2023-05-07T23:50:52Z

@qiluo-msft requests 2.29.0+ and urllib3 2.x doesn't work in the SONiC environment. I got the same error.
The current fix seems to be the correct one.
Redis-py fixed the issue in the same way: redis/redis-py#2748

liushilongbuaa · 2023-05-08T02:51:51Z

https://dev.azure.com/mssonic/build/_build/results?buildId=268185&view=results
Yesterdays' PR check passed. Maybe it is fixed?

keboliu · 2023-05-08T04:01:52Z

https://dev.azure.com/mssonic/build/_build/results?buildId=268185&view=results Yesterdays' PR check passed. Maybe it is fixed?

Hi @liushilongbuaa I still see the same issue in my local build environment.

liushilongbuaa · 2023-05-08T04:22:40Z

It is already fixed by docker's new release.

Installed /__w/1/s/.eggs/docker_image_py-0.1.12-py3.9.egg
Searching for docker>=4.4.4
Reading https://pypi.org/simple/docker/
Downloading https://files.pythonhosted.org/packages/b6/21/6b7450980b84c435ac4b58ad6b291c99b506d07a261bafe2bd96ecbc68bd/docker-6.1.1-py3-none-any.whl#sha256=8308b23d3d0982c74f7aa0a3abd774898c0c4fba006e9c3bde4f68354e470fe2
Best match: docker 6.1.1

liushilongbuaa · 2023-05-08T04:44:58Z

setup.py

@@ -221,6 +221,7 @@
 },
 install_requires=[
 'click==7.0',
+ 'urllib3<2',
 'click-log>=0.3.2',
 'docker>=4.4.4',


docker>=6.1.1

liushilongbuaa · 2023-05-08T04:45:27Z

setup.py

@@ -221,6 +221,7 @@
 },
 install_requires=[
 'click==7.0',
+ 'urllib3<2',


we don't need to limit urllib3 now.

Hi @liushilongbuaa seems it's not fixed, I still see same issue in my local build, and it also failed the checker of PR sonic-net/sonic-swss#2763.

stephenxs · 2023-05-08T09:16:35Z

A similar issue is also observed in the swss vstest: https://dev.azure.com/mssonic/be1b070f-be15-4154-aade-b1d3bfb17054/_apis/build/builds/268784/logs/95

…requests >= 2.29.0. (#2830) #### What I did Fix SONiC image compilation issue: ``` [2023-05-03T17:54:00.146Z] + sudo cp files/build_templates/docker_image_ctl.j2 ./fs********-mellanox/usr/share/sonic/templates/docker_image_ctl.j2 [2023-05-03T17:54:00.146Z] + sudo LANG=C DOCKER_HOST= ch******** ./fs********-mellanox /usr/local/bin/generate_shutdown_order.py [2023-05-03T17:54:00.146Z] Traceback (most recent call last): [2023-05-03T17:54:00.146Z] File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 214, in _retrieve_server_version [2023-05-03T17:54:00.146Z] return self.version(api_version=False)["ApiVersion"] [2023-05-03T17:54:00.147Z] File "/usr/local/lib/python3.9/dist-packages/docker/api/daemon.py", line 181, in version [2023-05-03T17:54:00.147Z] return self._result(self._get(url), json=True) [2023-05-03T17:54:00.147Z] File "/usr/local/lib/python3.9/dist-packages/docker/utils/decorators.py", line 46, in inner [2023-05-03T17:54:00.147Z] return f(self, *args, **kwargs) [2023-05-03T17:54:00.147Z] File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 237, in _get [2023-05-03T17:54:00.147Z] return self.get(url, **self._set_request_timeout(kwargs)) [2023-05-03T17:54:00.147Z] File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 600, in get [2023-05-03T17:54:00.147Z] return self.request("GET", url, **kwargs) [2023-05-03T17:54:00.147Z] File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 587, in request [2023-05-03T17:54:00.147Z] resp = self.send(prep, **send_kwargs) [2023-05-03T17:54:00.147Z] File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 701, in send [2023-05-03T17:54:00.147Z] r = adapter.send(request, **kwargs) [2023-05-03T17:54:00.147Z] File "/usr/local/lib/python3.9/dist-packages/requests/adapters.py", line 486, in send [2023-05-03T17:54:00.147Z] resp = conn.urlopen( [2023-05-03T17:54:00.147Z] File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 790, in urlopen [2023-05-03T17:54:00.147Z] response = self._make_request( [2023-05-03T17:54:00.147Z] File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 496, in _make_request [2023-05-03T17:54:00.147Z] conn.request( [2023-05-03T17:54:00.147Z] TypeError: request() got an unexpected keyword argument 'chunked' ``` #### How I did it Pin urllib to version < 2 as it was suggested in docker/docker-py#3113. #### How to verify it Run SONiC image compilation.

anamehra · 2023-05-11T16:25:18Z

Hi @abdosi , please help to get this one in 202205.

rlhui · 2023-05-12T04:39:49Z

Hi @abdosi , please help to get this one in 202205.

@yxieca please help on this one, thanks

msosyak · 2023-05-12T08:07:23Z

@qiluo-msft What is needed to trigger sonic-utilities submodule update in sonic-buildimage repo? Could you help with it, please?

yxieca · 2023-05-17T22:21:58Z

@oleksandrivantsiv @liushilongbuaa do we still need to cherry-pick this change? Do we need this change at all?

liushilongbuaa · 2023-05-18T03:27:31Z

Yes. We need.
Local build maybe fails. But it doesn't affect our pipeline.

yxieca · 2023-05-20T23:22:42Z

Yes. We need. Local build maybe fails. But it doesn't affect our pipeline.

I thought locking docker to 6.1.1 should have fixed it? @liushilongbuaa

liushilongbuaa · 2023-05-22T05:07:03Z

Yes. We need. Local build maybe fails. But it doesn't affect our pipeline.

I thought locking docker to 6.1.1 should have fixed it? @liushilongbuaa

docker 6.1.1 will fix the issue.
So, Line:225 change to docker>=6.1.0 may be better.

yxieca · 2023-05-26T17:13:59Z

Removed cherry-pick flag according to Shilong's comment.

…requests >= 2.29.0. (sonic-net#2830) #### What I did Fix SONiC image compilation issue: ``` [2023-05-03T17:54:00.146Z] + sudo cp files/build_templates/docker_image_ctl.j2 ./fs********-mellanox/usr/share/sonic/templates/docker_image_ctl.j2 [2023-05-03T17:54:00.146Z] + sudo LANG=C DOCKER_HOST= ch******** ./fs********-mellanox /usr/local/bin/generate_shutdown_order.py [2023-05-03T17:54:00.146Z] Traceback (most recent call last): [2023-05-03T17:54:00.146Z] File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 214, in _retrieve_server_version [2023-05-03T17:54:00.146Z] return self.version(api_version=False)["ApiVersion"] [2023-05-03T17:54:00.147Z] File "/usr/local/lib/python3.9/dist-packages/docker/api/daemon.py", line 181, in version [2023-05-03T17:54:00.147Z] return self._result(self._get(url), json=True) [2023-05-03T17:54:00.147Z] File "/usr/local/lib/python3.9/dist-packages/docker/utils/decorators.py", line 46, in inner [2023-05-03T17:54:00.147Z] return f(self, *args, **kwargs) [2023-05-03T17:54:00.147Z] File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 237, in _get [2023-05-03T17:54:00.147Z] return self.get(url, **self._set_request_timeout(kwargs)) [2023-05-03T17:54:00.147Z] File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 600, in get [2023-05-03T17:54:00.147Z] return self.request("GET", url, **kwargs) [2023-05-03T17:54:00.147Z] File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 587, in request [2023-05-03T17:54:00.147Z] resp = self.send(prep, **send_kwargs) [2023-05-03T17:54:00.147Z] File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 701, in send [2023-05-03T17:54:00.147Z] r = adapter.send(request, **kwargs) [2023-05-03T17:54:00.147Z] File "/usr/local/lib/python3.9/dist-packages/requests/adapters.py", line 486, in send [2023-05-03T17:54:00.147Z] resp = conn.urlopen( [2023-05-03T17:54:00.147Z] File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 790, in urlopen [2023-05-03T17:54:00.147Z] response = self._make_request( [2023-05-03T17:54:00.147Z] File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 496, in _make_request [2023-05-03T17:54:00.147Z] conn.request( [2023-05-03T17:54:00.147Z] TypeError: request() got an unexpected keyword argument 'chunked' ``` #### How I did it Pin urllib to version < 2 as it was suggested in docker/docker-py#3113. #### How to verify it Run SONiC image compilation.

oleksandrivantsiv self-assigned this May 5, 2023

[build] Fix base OS compilation issue caused by incompatibility with …

bfcaa8c

…requests >= 2.29.0. - Pin urllib to older version from 2.0.0 Signed-off-by: Oleksandr Ivantsiv <[email protected]> Co-authored-by: [email protected]

oleksandrivantsiv force-pushed the request-fixed-version branch 3 times, most recently from ec0596f to bfcaa8c Compare May 5, 2023 12:50

keboliu added the Request for 202211 Branch label May 5, 2023

oleksandrivantsiv changed the title ~~[build] Fix base OS compilation issue caused by incompatibility with requests >= 2.29.0~~ [build] Fix base OS compilation issue caused by incompatibility urllib with requests >= 2.29.0 May 5, 2023

abdosi added the Request for 202205 Branch label May 5, 2023

oleksandrivantsiv force-pushed the request-fixed-version branch from bfcaa8c to 92f8aa7 Compare May 7, 2023 21:15

oleksandrivantsiv changed the title ~~[build] Fix base OS compilation issue caused by incompatibility urllib with requests >= 2.29.0~~ [build] Fix base OS compilation issue caused by incompatibility of older versions of urllib with requests. May 7, 2023

oleksandrivantsiv requested a review from qiluo-msft May 7, 2023 21:41

oleksandrivantsiv marked this pull request as draft May 7, 2023 21:42

oleksandrivantsiv marked this pull request as ready for review May 7, 2023 22:13

oleksandrivantsiv marked this pull request as draft May 7, 2023 22:43

oleksandrivantsiv force-pushed the request-fixed-version branch from 92f8aa7 to bfcaa8c Compare May 7, 2023 23:26

oleksandrivantsiv changed the title ~~[build] Fix base OS compilation issue caused by incompatibility of older versions of urllib with requests.~~ [build] Fix base OS compilation issue caused by incompatibility of urllib with requests. May 7, 2023

oleksandrivantsiv marked this pull request as ready for review May 7, 2023 23:46

keboliu requested a review from liushilongbuaa May 8, 2023 02:00

liushilongbuaa reviewed May 8, 2023

View reviewed changes

stephenxs mentioned this pull request May 8, 2023

[Dynamic Buffer Calc] Enhance the logic to check maximum headroom exceeding to cover corner scenarios sonic-net/sonic-swss#2763

Merged

qiluo-msft approved these changes May 8, 2023

View reviewed changes

qiluo-msft merged commit 71ef4f1 into sonic-net:master May 8, 2023

StormLiangMS added the Included in 202211 Branch label May 9, 2023

yxieca removed Request for 202205 Branch Request for 202211 Branch Included in 202211 Branch labels May 26, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[build] Fix base OS compilation issue caused by incompatibility of urllib with requests. #2830

[build] Fix base OS compilation issue caused by incompatibility of urllib with requests. #2830

oleksandrivantsiv commented May 5, 2023 •

edited

Loading

liushilongbuaa commented May 5, 2023

oleksandrivantsiv commented May 5, 2023

anamehra commented May 5, 2023

qiluo-msft commented May 5, 2023

oleksandrivantsiv commented May 7, 2023

liushilongbuaa commented May 8, 2023

keboliu commented May 8, 2023

liushilongbuaa commented May 8, 2023 •

edited

Loading

liushilongbuaa May 8, 2023

liushilongbuaa May 8, 2023

keboliu May 8, 2023

stephenxs commented May 8, 2023

anamehra commented May 11, 2023

rlhui commented May 12, 2023

msosyak commented May 12, 2023

yxieca commented May 17, 2023

liushilongbuaa commented May 18, 2023

yxieca commented May 20, 2023

liushilongbuaa commented May 22, 2023

yxieca commented May 26, 2023

[build] Fix base OS compilation issue caused by incompatibility of urllib with requests. #2830

[build] Fix base OS compilation issue caused by incompatibility of urllib with requests. #2830

Conversation

oleksandrivantsiv commented May 5, 2023 • edited Loading

What I did

How I did it

How to verify it

Previous command output (if the output of a command-line utility has changed)

New command output (if the output of a command-line utility has changed)

liushilongbuaa commented May 5, 2023

oleksandrivantsiv commented May 5, 2023

anamehra commented May 5, 2023

qiluo-msft commented May 5, 2023

oleksandrivantsiv commented May 7, 2023

liushilongbuaa commented May 8, 2023

keboliu commented May 8, 2023

liushilongbuaa commented May 8, 2023 • edited Loading

liushilongbuaa May 8, 2023

Choose a reason for hiding this comment

liushilongbuaa May 8, 2023

Choose a reason for hiding this comment

keboliu May 8, 2023

Choose a reason for hiding this comment

stephenxs commented May 8, 2023

anamehra commented May 11, 2023

rlhui commented May 12, 2023

msosyak commented May 12, 2023

yxieca commented May 17, 2023

liushilongbuaa commented May 18, 2023

yxieca commented May 20, 2023

liushilongbuaa commented May 22, 2023

yxieca commented May 26, 2023

oleksandrivantsiv commented May 5, 2023 •

edited

Loading

liushilongbuaa commented May 8, 2023 •

edited

Loading