Don't require pyarrow on 3.9, and show a warning on custom components #2452

akrolsmir · 2020-12-11T01:45:08Z

Fixes #2356 #2001

Current Approach (12/16):

In Pipfile, specify that PyArrow is only installed on Python < 3.9
In all Streamlit versions, dynamically import pyarrow when Custom Components are created; show a nice error message if pyarrow is not installed
Once PyArrow supports Python 3.9 (ETA 2020 Jan), revert this PR, then update docs to say Streamlit supports 3.9 officially
(Future) Consider resolving watchdog error by excluding watchdog on Mac, permanently?

Tested locally on 3.9 and 3.8, verified components still work on 3.8.
Get the wheel file: https://share.streamlit.io/streamlit/core-previews/pr-2452

.
.
.
.
.

Old Proposal (12/14):

Create separate wheelfiles for python 3.6-3.8 and python 3.9
Exclude pyarrow dependency from python 3.9
In all Streamlit versions, dynamically import pyarrow when Custom Components are created; show a nice error message if pyarrow is not installed
(Future) Also fix up watchdog error with different wheels for Mac?

Notes (12/11):

Actually, 1. doesn't look so easy anymore. I tried editing the setup.py file thinking it would get run on each pip install, but actually it only gets run when we package the wheel file.
The wheel file itself does not contain setup.py, but rather has a hardcoded list of dependencies for pip to install (you can unzip the whl and look under streamlit-0.72.0.dist-info/METADATA). So the dependencies are locked in by the time we upload the wheel file; and when users pip install streamlit, there's no code that we control that decides which requirements to install.
By that same reasoning, it's not clear whether the watchdog fix in setup.py works for our users either???
Would love to know if I'm wrong on any of the above (eg if pip install does not install from an uploaded wheel, but rather a source folder including setup.py?)

Notes (12/10):
From @tvst:
A temporary patch for the PyArrow thing could look like this:

When you pip-install Steramlit, we can check the Python version (during the installation process). If Python 3.9, don't include Arrow.
Then when you try to use Components, put a try/catch around the import Arrow statement. If it fails, show a message with a super nice user-readable explanation of whats up and how to solve it.

~~This PR is still completely untested; waiting for PR Preview to make a wheelfile.~~ Wheelfile here
Could use some Product workshopping on the specific error text we show
Biggest worry is this comment:

Using Python 3.8 does not solve the problem. Seems to me, pyarrow does not provide a wheel for Mac OS 11 yet.

If true, we'd want to change up our setup.py to try and detect Mac OS 11???

akrolsmir · 2020-12-12T06:30:09Z

Preview:

polm · 2020-12-12T06:57:18Z

Regarding this comment:

Using Python 3.8 does not solve the problem. Seems to me, pyarrow does not provide a wheel for Mac OS 11 yet.

This bit me too with some packages I maintain. The new Mac architecture can use packages for the old Mac architecture, but pip needed some changes to realize that, so it only works with pip 20.3 or higher. See this issue for more details.

Pip 20.3 was released Dec 1, so after the comment you quoted. So if someone on 3.8 or lower on the new Mac has issues you can probably just tell them to upgrade pip.

karriebear · 2020-12-12T19:48:53Z

Curious did you give the extras_require a try? It looks like we could add a dependency based on python version but haven't tried yet. https://discuss.python.org/t/adding-a-default-extra-require-environment/4898/7. If this works, I would personally prefer this approach.

My concerns with the current approach is mainly regarding the release and distribution process. Concerns are easily testable but would want to confirm if we have a 3.9 version on pypi, would pip install be smart enough to get the right wheel for each environment. For releases, should we test both wheels? May have to modify our release scripts to ensure the right wheel is installed for each testing, etc.

akrolsmir · 2020-12-14T06:55:41Z

Curious did you give the extras_require a try? It looks like we could add a dependency based on python version but haven't tried yet. https://discuss.python.org/t/adding-a-default-extra-require-environment/4898/7. If this works, I would personally prefer this approach.

I have not -- the problem with extras_require afaict is that it's part of setuptools, meaning that it would only work for the source distribution, aka when users install from our .tar.gz using pip install streamlit --no-binary. Our wheel (aka bdist, binary distribution) would presumably strip out the extras_require as it takes out all of the setuptools code, compiling the requirements into the METADATA file.

I guess we could change our guides to recommend pip install streamlit --no-binary, or alternatively stop distributing wheel files altogether so pip install streamlit would always pick the .tar.gz? (Until pyarrow starts supporting 3.9). One potential problem: do our users need to install any additional dependencies to compile the .tar.gz into a working version of Streamlit?

My concerns with the current approach is mainly regarding the release and distribution process. Concerns are easily testable but would want to confirm if we have a 3.9 version on pypi, would pip install be smart enough to get the right wheel for each environment. For releases, should we test both wheels? May have to modify our release scripts to ensure the right wheel is installed for each testing, etc.

Yeah, this is my primary concern too. I don't think we want to maintain multiple wheels long-term, this is really just a workaround for the pyarrow 3.9 thing, and we should take it out as soon as pyarrow provides wheels for 3.9. Until then, though, I think the added maintenance burden would be worth making the pip install streamlit experience suck less for our users.

karriebear · 2020-12-14T15:27:51Z

I have not -- the problem with extras_require afaict is that it's part of setuptools, meaning that it would only work for the source distribution, aka when users install from our .tar.gz using pip install streamlit --no-binary. Our wheel (aka bdist, binary distribution) would presumably strip out the extras_require as it takes out all of the setuptools code, compiling the requirements into the METADATA file.

From my understanding, the setuptools code is what creates our wheel so if you pass extras_require it should put the conditional dependency into the METADATA file. Based on the example in the link we should get a line like below added which should hopefully only install numpy if python version > 3.4.
Requires-Dist: numpy ; python_version>"3.4"

I think the issue with your first attempt was that you were evaluating the conditional requirements on compile time whereas adding the conditional requirement as a parameter to the setuptools command will pass the conditional requirement to evaluate on install time.

karriebear · 2020-12-14T15:40:20Z

Did a very not thorough test for extras_require and it looks to do what we want. Draft PR for it #2464

akrolsmir · 2020-12-16T15:58:52Z

Note for posterity: Karrie was absolutely right and I was wrong above; extra_requires would have worked perfectly by updating the METADATA file to conditionally install pyarrow. I got a bit confused about the difference between install_requires and extra_requires, the former might work too. But the easiest solution of all turned out to be to not muck with Setuptools; Pipenv already provides a way of specifying requirements conditional on Python versions, so let's just use that.

karriebear · 2020-12-16T16:50:04Z

lib/streamlit/components/v1/components.py

+
+And if you're using Streamlit Sharing, add "pyarrow" to your requirements.txt."""
+ )
+


probably not worth it but just noting this down, it looks all custom components will not work regardless or not if the component uses tables.

Good point. The alternate approach would be to try-except specifically the areas that call on arrow_table, or to move this try-catch into arrow_table.py altogether.

In my head, cutting the boundary at "Custom Components" rather than "Custom Components using tables" was easier for users to reason about, and I imagine those who run into this issue (those inclined to install components) are technical enough not to have a huge problem with this, plus AFAICT components are not that widely, so I didn't feel like optimizing this further.

Incidentally, would be nice to have easy metrics on "What % of Streamlit Apps use Components" and "What % of Streamlit Apps are on Python 3.9".

lib/streamlit/components/v1/components.py

Co-authored-by: karrie <[email protected]>

asaini · 2020-12-16T19:59:57Z

lib/streamlit/components/v1/components.py

+PyArrow. Unfortunately, PyArrow does not yet support Python 3.9.
+
+You can either switch to Python 3.8 with an environment manager like PyEnv, or
+[install Streamlit with conda](https://discuss.streamlit.io/t/note-installation-issues-with-python-3-9-and-streamlit/6946/7):


@akrolsmir : I'm a little confused by reading this line. Are we telling the user to

install Python 3.8 (and streamlit) using conda as en environment manager
OR

are we saying that by using conda the user can use Python 3.9 and PyArrow

Can we make the distinction clear in the comment?

install Python 3.8 (and streamlit) using any environment manager or use conda

Cool, LGTM 👍

asaini · 2020-12-16T21:46:37Z

LGTM 👍

* develop: Extract st.container, columns, container out of delta_generator.py (streamlit#2487) Remove the unused variables inside protos (streamlit#2486) Stop running cypress-flaky-approval for each PR (streamlit#2490) Dynamically import git, and fail gracefully if missing (streamlit#2482) Bump React to 17.0.1 (streamlit#2453) Fix emojis (streamlit#2480) Add missing copyright headers (streamlit#2478) Stop requiring watchdog when installing Streamlit on Macs (streamlit#2470) Minor tweak to PyArrow warning message (streamlit#2472) Use a set literal (streamlit#2476) Add type annotations to DeltaGenerator mixins (streamlit#2475) Update change log Up version to 0.73.0 Don't require pyarrow on 3.9, and show a warning on custom components (streamlit#2452)

Don't require pyarrow on 3.9, and show a warning on custom components

d3f6329

akrolsmir requested a review from a team December 11, 2020 01:45

karriebear self-assigned this Dec 11, 2020

akrolsmir added the product review needed label Dec 11, 2020

akrolsmir added 5 commits December 11, 2020 22:02

Move arrow_table to a dynamic dependency

ae5109b

Build a separate wheel for py39

55f605c

Switch to StreamlitAPIException

0badcdc

Remove unneeded cmd line params

83214c6

Prevent 3.9 on the universal wheel

2db2a48

akrolsmir changed the title ~~Don't require pyarrow on 3.9, and show a warning on custom components~~ Build wheels for Python 3.9 without pyarrow Dec 12, 2020

karriebear added the release-blocker label Dec 15, 2020

akrolsmir added 3 commits December 16, 2020 07:13

Instead of building multiple wheels, use extras_require

b5427df

... Need to actually require them

d6302a3

Even better: Pipfile provides a way to gate python versions

241b774

akrolsmir changed the title ~~Build wheels for Python 3.9 without pyarrow~~ Don't require pyarrow on 3.9, and show a warning on custom components Dec 16, 2020

karriebear approved these changes Dec 16, 2020

View reviewed changes

Update lib/streamlit/components/v1/components.py

1be6e4d

Co-authored-by: karrie <[email protected]>

akrolsmir mentioned this pull request Dec 16, 2020

Stop requiring watchdog when installing Streamlit on Macs #2470

Merged

asaini reviewed Dec 16, 2020

View reviewed changes

karriebear merged commit 451b7cc into streamlit:develop Dec 16, 2020

akrolsmir mentioned this pull request Dec 17, 2020

Minor tweak to PyArrow warning message #2472

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Don't require pyarrow on 3.9, and show a warning on custom components #2452

Don't require pyarrow on 3.9, and show a warning on custom components #2452

akrolsmir commented Dec 11, 2020 •

edited

Loading

akrolsmir commented Dec 12, 2020

polm commented Dec 12, 2020

karriebear commented Dec 12, 2020

akrolsmir commented Dec 14, 2020 •

edited

Loading

karriebear commented Dec 14, 2020

karriebear commented Dec 14, 2020

akrolsmir commented Dec 16, 2020

karriebear Dec 16, 2020

akrolsmir Dec 16, 2020

asaini Dec 16, 2020

karriebear Dec 16, 2020

asaini Dec 16, 2020

asaini commented Dec 16, 2020


		And if you're using Streamlit Sharing, add "pyarrow" to your requirements.txt."""
		)

Don't require pyarrow on 3.9, and show a warning on custom components #2452

Don't require pyarrow on 3.9, and show a warning on custom components #2452

Conversation

akrolsmir commented Dec 11, 2020 • edited Loading

akrolsmir commented Dec 12, 2020

polm commented Dec 12, 2020

karriebear commented Dec 12, 2020

akrolsmir commented Dec 14, 2020 • edited Loading

karriebear commented Dec 14, 2020

karriebear commented Dec 14, 2020

akrolsmir commented Dec 16, 2020

karriebear Dec 16, 2020

Choose a reason for hiding this comment

akrolsmir Dec 16, 2020

Choose a reason for hiding this comment

asaini Dec 16, 2020

Choose a reason for hiding this comment

karriebear Dec 16, 2020

Choose a reason for hiding this comment

asaini Dec 16, 2020

Choose a reason for hiding this comment

asaini commented Dec 16, 2020

akrolsmir commented Dec 11, 2020 •

edited

Loading

akrolsmir commented Dec 14, 2020 •

edited

Loading