Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Isolate sasl as a dependency #109

Closed
jtcohen6 opened this issue Sep 30, 2020 · 3 comments · Fixed by #127
Closed

Isolate sasl as a dependency #109

jtcohen6 opened this issue Sep 30, 2020 · 3 comments · Fixed by #127

Comments

@jtcohen6
Copy link
Contributor

jtcohen6 commented Sep 30, 2020

Right now, pip install dbt-spark is not working on Windows because:

  • dbt-spark requires PyHive[hive]
  • hive extra requires sasl
  • pip install sasl is failing on Windows machines

On my personal machine:

Python 3.8.5
pip 20.2.3

Here's one of the stacktraces I've seen, though they vary by user/machine:

Building wheels for collected packages: sasl
  Building wheel for sasl (setup.py) ... error
  ERROR: Command errored out with exit status 1:
...
Complete output (27 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build\lib.win-amd64-3.7
  creating build\lib.win-amd64-3.7\sasl
  copying sasl\__init__.py -> build\lib.win-amd64-3.7\sasl
  running egg_info
  writing sasl.egg-info\PKG-INFO
  writing dependency_links to sasl.egg-info\dependency_links.txt
  writing requirements to sasl.egg-info\requires.txt
  writing top-level names to sasl.egg-info\top_level.txt
  reading manifest file 'sasl.egg-info\SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  writing manifest file 'sasl.egg-info\SOURCES.txt'
  copying sasl\saslwrapper.cpp -> build\lib.win-amd64-3.7\sasl
  copying sasl\saslwrapper.h -> build\lib.win-amd64-3.7\sasl
  copying sasl\saslwrapper.pyx -> build\lib.win-amd64-3.7\sasl
  running build_ext
  building 'sasl.saslwrapper' extension
  creating build\temp.win-amd64-3.7
  creating build\temp.win-amd64-3.7\Release
  creating build\temp.win-amd64-3.7\Release\sasl
  C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -Isasl -Ic:\users\User\appdata\local\programs\python\python37\include -Ic:\users\User\appdata\local\programs\python\python37\include "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE" "-IC:\Program Files (x86)\Windows Kits\10\include\wdf\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\wdf\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\wdf\um" "-IC:\Program Files (x86)\Windows Kits\10\include\wdf\winrt" /EHsc /Tpsasl/saslwrapper.cpp /Fobuild\temp.win-amd64-3.7\Release\sasl/saslwrapper.obj 
  saslwrapper.cpp
  c:\users\User\appdata\local\programs\python\python37\include\pyconfig.h(59): fatal error C1083: Cannot open include file: 'io.h': No such file or directory   
  error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\BIN\\x86_amd64\\cl.exe' failed with exit status 2
----------------------------------------
ERROR: Failed building wheel for sasl
  Running setup.py clean for sasl
Failed to build sasl
Installing collected packages: pure-sasl, thrift-sasl, sasl, PyHive, dbt-spark
    Running setup.py install for sasl ... error
@jtcohen6
Copy link
Contributor Author

Workaround: there are pre-built sasl wheels for python 3.6 + 3.7 (nothing for 3.8): https://www.lfd.uci.edu/~gohlke/pythonlibs/#sasl

Download, then:

pip install C:\users\User\Downloads\sasl‑0.2.1‑cp37‑cp37m‑win_amd64.whl
pip install dbt-spark

@jtcohen6 jtcohen6 changed the title Failed installation on Windows Isolate sasl as a dependency Nov 19, 2020
@jtcohen6
Copy link
Contributor Author

jtcohen6 commented Nov 19, 2020

It looks like pip install sasl does not work with py39.

  • We have an open issue to support python 3.9 in dbt-core: test dbt with Python 3.9 dbt-core#2822
  • IMO the best resolution to this issue is moving PyHive[hive] into an extras_require, only for use when connecting via the http or thrift methods—similar to what we did with pyodbc.

@Fokko
Copy link
Contributor

Fokko commented Nov 19, 2020

@charlottevdscheun encountered the same issue, back in the days when she was still on Windows :)

I don't have any objections to moving the pyhive stuff to extras_require, similar to odbc.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants