Running "pip install ." fails on Windows #755

sschuberth · 2017-09-13T08:55:05Z

I'm not 100% sure this is supposed to work, but as it does on Linux, I believe it should work on Windows, too. Running pip install . on a fresh checkout of ScanCode gives

Processing c:\dev\scancode-toolkit
Exception:
Traceback (most recent call last):
  File "c:\python27\lib\site-packages\pip\basecommand.py", line 209, in main
    status = self.run(options, args)
  File "c:\python27\lib\site-packages\pip\commands\install.py", line 299, in run
    requirement_set.prepare_files(finder)
  File "c:\python27\lib\site-packages\pip\req\req_set.py", line 360, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "c:\python27\lib\site-packages\pip\req\req_set.py", line 577, in _prepare_file
    session=self.session, hashes=hashes)
  File "c:\python27\lib\site-packages\pip\download.py", line 798, in unpack_url
    unpack_file_url(link, location, download_dir, hashes=hashes)
  File "c:\python27\lib\site-packages\pip\download.py", line 676, in unpack_file_url
    shutil.copytree(link_path, location, symlinks=True)
  File "c:\python27\lib\shutil.py", line 208, in copytree
    raise Error, errors
Error: [('C:\\Dev\\scancode-toolkit\\tests\\scancode\\data\\unicodepath\\uc\\unicodepath\\???', 'c:\\users\\sebast~1\\appdata\\local\\temp\\pip-wizgww-build\\tests\\scancode\\data\\unicodepath\\uc\\unicodepath\\???', "[Errno 22] invalid   mode ('rb') or filename: 'C:\\\\Dev\\\\scancode-toolkit\\\\tests\\\\scancode\\\\data\\\\unicodepath\\\\uc\\\\unicodepath\\\\???'"), ('C:\\Dev\\scancode-toolkit\\tests\\scancode\\data\\unicodepath\\uc\\unicodepath\\???a',   'c:\\users\\sebast~1\\appdata\\local\\temp\\pip-wizgww-build\\tests\\scancode\\data\\unicodepath\\uc\\unicodepath\\???a', "[Errno 22] invalid mode ('rb') or filename:   'C:\\\\Dev\\\\scancode-toolkit\\\\tests\\\\scancode\\\\data\\\\unicodepath\\\\uc\\\\unicodepath\\\\???a'")]
You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

Note that I'm deliberately not calling configure as I'm working on a generic way to detect Python project dependencies, and the generic way to install dependencies is to run pip install AFAIK.

The text was updated successfully, but these errors were encountered:

pombredanne · 2017-09-13T15:23:39Z

@sschuberth Thanks! It should definitely work. Weirdly enough this seems to be related to the new test files added for #688 that git can handle ok on Windows but pip chokes on. The other weird stuff is that the pip install . is exactly what ./configure runs....

So a possible cause could be the version of pip and setuptools you have: can you try with pip 9.0.1?

sschuberth · 2017-09-13T15:31:41Z

Just tried, same problem with pip 9.0.1:

Processing c:\dev\scancode-toolkit
Exception:
Traceback (most recent call last):
  File "c:\python\lib\site-packages\pip\basecommand.py", line 215, in main
    status = self.run(options, args)
  File "c:\python\lib\site-packages\pip\commands\install.py", line 324, in run
    requirement_set.prepare_files(finder)
  File "c:\python\lib\site-packages\pip\req\req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "c:\python\lib\site-packages\pip\req\req_set.py", line 620, in _prepare_file
    session=self.session, hashes=hashes)
  File "c:\python\lib\site-packages\pip\download.py", line 809, in unpack_url
    unpack_file_url(link, location, download_dir, hashes=hashes)
  File "c:\python\lib\site-packages\pip\download.py", line 686, in unpack_file_url
    shutil.copytree(link_path, location, symlinks=True)
  File "c:\python\lib\shutil.py", line 208, in copytree
    raise Error, errors
Error: [('C:\\Dev\\scancode-toolkit\\tests\\scancode\\data\\unicodepath\\uc\\unicodepath\\???', 'c:\\users\\sebast~1\\appdata\\local\\temp\\pip-ibgtvi-build\\tests\\scancode\\data\\unicodepath\\uc\\unicodepath\\???', "[Errno 22] invalid mode ('rb') or filename: 'C:\\\\Dev\\\\scancode-toolkit\\\\tests\\\\scancode\\\\data\\\\unicodepath\\\\uc\\\\unicodepath\\\\???'"), ('C:\\Dev\\scancode-toolkit\\tests\\scancode\\data\\unicodepath\\uc\\unicodepath\\???a', 'c:\\users\\sebast~1\\appdata\\local\\temp\\pip-ibgtvi-build\\tests\\scancode\\data\\unicodepath\\uc\\unicodepath\\???a', "[Errno 22] invalid mode ('rb') or filename: 'C:\\\\Dev\\\\scancode-toolkit\\\\tests\\\\scancode\\\\data\\\\unicodepath\\\\uc\\\\unicodepath\\\\???a'")]

pombredanne · 2017-09-13T15:52:55Z

@sschuberth Thanks. The fix is easy: I will tar the test files so they are not "visible" in a checkout and only extracted at test run time and this should likely fix this.
You could check that this will work by deleting locally tests\scancode\data\unicodepath\ and running pip again

pombredanne · 2017-09-13T15:53:20Z

@sschuberth which version of Python do you run btw?

sschuberth · 2017-09-13T16:05:10Z

I'm using Python 2.7.12,

Confirmed that after manually deleting tests\scancode\data\unicodepath\ running pip install . works (well, except that it requires Microsoft Visual C++ 9.0 now, which I don't have installed, to compile some native extension, I guess).

Any idea why the Windows CI did not also break?

pombredanne · 2017-09-13T16:41:35Z

@sschuberth good. No idea why the Ci works OK. They may have something special in their env.
Now all the deps should be available as prebuilt wheels on Pypi afaik. So no compiler should be needed.... Can you tell me which one failed to install?

sschuberth · 2017-09-13T18:25:58Z

The package that failed to install due to the missing compiler is called just url.

* this ensures that the repo can be cloned on Windows Signed-off-by: Philippe Ombredanne <[email protected]>

pombredanne · 2017-09-19T23:08:40Z

I fixed the pip install issue from a clone. As for the installation of packages that have native code that needs to be compiled there is no easy way out on Windows short of installing a compiler (or point pip to use the pre-built packages in thirdparty/ which is exactly the same as running configure ...)

* this ensures that the repo can be cloned on Windows Signed-off-by: Philippe Ombredanne <[email protected]>

Signed-off-by: Philippe Ombredanne <[email protected]>

* do not map the legacy "url" to anything #755 * allow to build a Package from data and not only from a file * do not return full location for the metafile but only filename #782 * ensure only one download_urls is created #779 * improve the mapping of "parties" * other misc code improvement and TODOs * add new tests for #755 and other samples Signed-off-by: Philippe Ombredanne <[email protected]>

Signed-off-by: Philippe Ombredanne <[email protected]>

* do not map the legacy "url" to anything #755 * allow to build a Package from data and not only from a file * do not return full location for the metafile but only filename #782 * ensure only one download_urls is created #779 * improve the mapping of "parties" * other misc code improvement and TODOs * add new tests for #755 and other samples Signed-off-by: Philippe Ombredanne <[email protected]>

pombredanne · 2017-10-20T15:28:02Z

@sschuberth this should be all set now, right?

sschuberth · 2017-10-20T15:55:32Z

Unfortunately, it still does not seem to work with the current develop branch. I'm getting:

$ pip install .
Processing c:\dev\scancode-toolkit

Exception:
Traceback (most recent call last):
  File "c:\python\lib\site-packages\pip\basecommand.py", line 215, in main
    status = self.run(options, args)
  File "c:\python\lib\site-packages\pip\commands\install.py", line 324, in run
    requirement_set.prepare_files(finder)
  File "c:\python\lib\site-packages\pip\req\req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "c:\python\lib\site-packages\pip\req\req_set.py", line 620, in _prepare_file
    session=self.session, hashes=hashes)
  File "c:\python\lib\site-packages\pip\download.py", line 809, in unpack_url
    unpack_file_url(link, location, download_dir, hashes=hashes)
  File "c:\python\lib\site-packages\pip\download.py", line 686, in unpack_file_url
    shutil.copytree(link_path, location, symlinks=True)
  File "c:\python\lib\shutil.py", line 208, in copytree
    raise Error, errors
Error: [('C:\\Dev\\scancode-toolkit\\tests\\scancode\\data\\unicodepath\\uc\\unicodepath\\???', 'c:\\users\\sebast~1\\appdata\\local\\temp\\pip-xn6i8z-build\\tests\\scancode\\data\\unicodepath\\uc\\unicodepath\\???', "[Errno 22] invalid mode ('rb') or filename: 'C:\\\\Dev\\\\scancode-toolkit\\\\tests\\\\scancode\\\\data\\\\unicodepath\\\\uc\\\\unicodepath\\\\???'"), ('C:\\Dev\\scancode-toolkit\\tests\\scancode\\data\\unicodepath\\uc\\unicodepath\\???a', 'c:\\users\\sebast~1\\appdata\\local\\temp\\pip-xn6i8z-build\\tests\\scancode\\data\\unicodepath\\uc\\unicodepath\\???a', "[Errno 22] invalid mode ('rb') or filename: 'C:\\\\Dev\\\\scancode-toolkit\\\\tests\\\\scancode\\\\data\\\\unicodepath\\\\uc\\\\unicodepath\\\\???a'")]

$ pip --version
pip 9.0.1 from c:\python\lib\site-packages (python 2.7)

pombredanne · 2017-10-20T16:06:05Z

Thank you! I now hate windows :D but we will fix this eventually !

pombredanne · 2018-10-30T19:50:00Z

So this should work nicely now... but as explained in #1214 this requires you to have a proper C/C++ compiler installed.

sschuberth · 2018-10-30T20:07:38Z

Unfortunately, I'm still getting (almost) the same error with latest develop (commit 81f177e):

$ pip install .
Processing c:\users\sebastian\development\github\nexb\scancode-toolkit
Exception:
Traceback (most recent call last):
  File "c:\python\lib\site-packages\pip\basecommand.py", line 215, in main
    status = self.run(options, args)
  File "c:\python\lib\site-packages\pip\commands\install.py", line 324, in run
    requirement_set.prepare_files(finder)
  File "c:\python\lib\site-packages\pip\req\req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "c:\python\lib\site-packages\pip\req\req_set.py", line 620, in _prepare_file
    session=self.session, hashes=hashes)
  File "c:\python\lib\site-packages\pip\download.py", line 809, in unpack_url
    unpack_file_url(link, location, download_dir, hashes=hashes)
  File "c:\python\lib\site-packages\pip\download.py", line 686, in unpack_file_url
    shutil.copytree(link_path, location, symlinks=True)
  File "c:\python\lib\shutil.py", line 208, in copytree
    raise Error, errors
Error: [('C:\\Users\\sebastian\\Development\\GitHub\\nexB\\scancode-toolkit\\tests\\scancode\\data\\unicodepath\\uc\\unicodepath\\???', 'c:\\users\\sebastian\\appdata\\local\\temp\\pip-xjsp7m-build\\tests\\scancode\\data\\unicodepath\\uc\\unicodepath\\???', "[Errno 22] invalid mode ('rb') or filename: 'C:\\\\Users\\\\sebastian\\\\Development\\\\GitHub\\\\nexB\\\\scancode-toolkit\\\\tests\\\\scancode\\\\data\\\\unicodepath\\\\uc\\\\unicodepath\\\\???'"), ('C:\\Users\\sebastian\\Development\\GitHub\\nexB\\scancode-toolkit\\tests\\scancode\\data\\unicodepath\\uc\\unicodepath\\???a', 'c:\\users\\sebastian\\appdata\\local\\temp\\pip-xjsp7m-build\\tests\\scancode\\data\\unicodepath\\uc\\unicodepath\\???a', "[Errno 22] invalid mode ('rb') or filename: 'C:\\\\Users\\\\sebastian\\\\Development\\\\GitHub\\\\nexB\\\\scancode-toolkit\\\\tests\\\\scancode\\\\data\\\\unicodepath\\\\uc\\\\unicodepath\\\\???a'")]
You are using pip version 9.0.1, however version 18.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

pombredanne · 2018-10-30T20:22:33Z

@sschuberth you are cheating! 😆
I just noticed that you are using some bash shell on Windows? (may be the Linux subsystem?) this would not be supported at all for now (help welcomed but this has never been tested).

But that may not be at all the issue here. Is this from a fresh checkout or download? have you created your own virtualenv? or not? it may be that there is some mixing up between various things there.

sschuberth · 2018-10-30T20:30:01Z

Indeed I am using Git Bash, but since the Python I'm calling is a native Windows one and I'm not passing any paths as arguments, using cmd does not make any difference:

C:\Users\sebastian\Development\GitHub\nexB\scancode-toolkit>pip install .
Processing c:\users\sebastian\development\github\nexb\scancode-toolkit
Exception:
Traceback (most recent call last):
  File "c:\python\lib\site-packages\pip\basecommand.py", line 215, in main
    status = self.run(options, args)
  File "c:\python\lib\site-packages\pip\commands\install.py", line 324, in run
    requirement_set.prepare_files(finder)
  File "c:\python\lib\site-packages\pip\req\req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "c:\python\lib\site-packages\pip\req\req_set.py", line 620, in _prepare_file
    session=self.session, hashes=hashes)
  File "c:\python\lib\site-packages\pip\download.py", line 809, in unpack_url
    unpack_file_url(link, location, download_dir, hashes=hashes)
  File "c:\python\lib\site-packages\pip\download.py", line 686, in unpack_file_url
    shutil.copytree(link_path, location, symlinks=True)
  File "c:\python\lib\shutil.py", line 208, in copytree
    raise Error, errors
Error: [('C:\\Users\\sebastian\\Development\\GitHub\\nexB\\scancode-toolkit\\tests\\scancode\\data\\unicodepath\\uc\\unicodepath\\???', 'c:\\users\\sebastian\\appdata\\local\\temp\\pip-iwscpj-build\\tests\\scancode\\data\\unicodepath\\uc\\unicodepath\\???', "[Errno 22] invalid mode ('rb') or filename: 'C:\\\\Users\\\\sebastian\\\\Development\\\\GitHub\\\\nexB\\\\scancode-toolkit\\\\tests\\\\scancode\\\\data\\\\unicodepath\\\\uc\\\\unicodepath\\\\???'"), ('C:\\Users\\sebastian\\Development\\GitHub\\nexB\\scancode-toolkit\\tests\\scancode\\data\\unicodepath\\uc\\unicodepath\\???a', 'c:\\users\\sebastian\\appdata\\local\\temp\\pip-iwscpj-build\\tests\\scancode\\data\\unicodepath\\uc\\unicodepath\\???a', "[Errno 22] invalid mode ('rb') or filename: 'C:\\\\Users\\\\sebastian\\\\Development\\\\GitHub\\\\nexB\\\\scancode-toolkit\\\\tests\\\\scancode\\\\data\\\\unicodepath\\\\uc\\\\unicodepath\\\\???a'")]
You are using pip version 9.0.1, however version 18.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

Sorry 😉

pombredanne · 2018-10-30T20:34:44Z

so the thing is that you have not "activated" the local virtualenv...
Assuming you are using a checkout and that you have been running configure --clean and then configure then, if you run in a CMD bin\activate ... then pip install . this should work....

sschuberth · 2018-10-30T21:03:58Z

After running

configure --clean
configure
bin\activate
pip install .

in cmd I still get

Processing c:\users\sebastian\development\github\nexb\scancode-toolkit
Could not install packages due to an EnvironmentError: [('C:\\Users\\sebastian\\Development\\GitHub\\nexB\\scancode-toolkit\\tests\\scancode\\data\\unicodepath\\uc\\unicodepath\\???', 'c:\\users\\sebastian\\appdata\\local\\temp\\pip-req-build-3g3fcb\\tests\\scancode\\data\\unicodepath\\uc\\unicodepath\\???', "[Errno 22] invalid mode ('rb') or filename: 'C:\\\\Users\\\\sebastian\\\\Development\\\\GitHub\\\\nexB\\\\scancode-toolkit\\\\tests\\\\scancode\\\\data\\\\unicodepath\\\\uc\\\\unicodepath\\\\???'"), ('C:\\Users\\sebastian\\Development\\GitHub\\nexB\\scancode-toolkit\\tests\\scancode\\data\\unicodepath\\uc\\unicodepath\\???a', 'c:\\users\\sebastian\\appdata\\local\\temp\\pip-req-build-3g3fcb\\tests\\scancode\\data\\unicodepath\\uc\\unicodepath\\???a', "[Errno 22] invalid mode ('rb') or filename: 'C:\\\\Users\\\\sebastian\\\\Development\\\\GitHub\\\\nexB\\\\scancode-toolkit\\\\tests\\\\scancode\\\\data\\\\unicodepath\\\\uc\\\\unicodepath\\\\???a'")]

pombredanne · 2018-10-30T21:09:27Z

Thank you for testing! I will find a fix.

pombredanne · 2018-10-30T23:28:30Z

Here is the skinny: this is a bug in pip that I would need to report and might never be fixed in Python 2. It gets confused by having specific unicode paths (that are in the test files). pip is trying to move these around when doing a pip install .

If you use a pip install -e . this will work and pip will not try to copy files around.

I will report this to pypa/pip

pombredanne · 2018-10-31T02:21:14Z

I was able to create a minimum package that shows the error. And I submitted a pip bug pypa/pip#5972

sschuberth · 2018-10-31T06:59:20Z

I will report this to pypa/pip

Thanks for this!

might never be fixed in Python 2

Speaking of it, what's the status / current plan to use Python 3 for ScanCode as I haven't seen any updates on #295 recently?

pombredanne · 2018-10-31T07:25:33Z

Speaking of it, what's the status / current plan to use Python 3 for ScanCode?

IMHO the port could be a 3.1 right after a 3.0 release? what do you think?
We will support both Python 2 and 3 for a good while anyway IMHO

pombredanne · 2018-10-31T07:26:28Z

IMHO the port could be a 3.1 right after a 3.0 release--> I mean by these the scancode 3.1 right after the 3.0 release. many 3's are confusing ;)

sschuberth · 2018-10-31T07:30:18Z

Well, if support of Python 3 would require any breaking changes to ScanCode, it would make more sense to me to do this with the new major 3.0 release of ScanCode rather than with a minor 3.1 release (in the spirit of semantic versioning). On the other hand, I could understand if you don't want to delay ScanCode 3.0 any further "just" because of Python 3.0 support. After all, it's all up to you 😉

pombredanne · 2019-02-18T17:04:31Z

FYI, still no update on the Pypa side. pypa/pip#5972
But I tagged this as Python3 as this would be something that should go away in Python3.

pombredanne · 2021-09-24T12:08:56Z

This is no longer an issue now that we only support Python 3 and dropped Python 2 support

pombredanne added bug installation and packaging labels Sep 13, 2017

pombredanne added a commit that referenced this issue Sep 19, 2017

Use archive for weird test paths for #755

ee13af0

* this ensures that the repo can be cloned on Windows Signed-off-by: Philippe Ombredanne <[email protected]>

pombredanne mentioned this issue Sep 19, 2017

Ensure fsencode/fsdecode works on ASCII FS. Use tarball for unicode test files #758

Merged

pombredanne added the fixed pending review label Sep 19, 2017

pombredanne added a commit that referenced this issue Sep 22, 2017

Use archive for weird test paths for #755

d5f3116

* this ensures that the repo can be cloned on Windows Signed-off-by: Philippe Ombredanne <[email protected]>

pombredanne added a commit that referenced this issue Sep 22, 2017

Ensure tarball with unicode path extracts on Windows #755

2e4e345

Signed-off-by: Philippe Ombredanne <[email protected]>

pombredanne added a commit that referenced this issue Sep 22, 2017

Skip some unicode-related tests on Windows #755

0ed7c87

Signed-off-by: Philippe Ombredanne <[email protected]>

pombredanne added a commit that referenced this issue Oct 1, 2017

Improve VCS url splits #755

6741eaa

Signed-off-by: Philippe Ombredanne <[email protected]>

pombredanne added a commit that referenced this issue Oct 3, 2017

Improve VCS url splits #755

a308072

Signed-off-by: Philippe Ombredanne <[email protected]>

pombredanne removed the fixed pending review label Aug 6, 2018

sschuberth changed the title ~~Running pip install fails on Windows~~ Running "pip install ." fails on Windows Oct 18, 2018

sschuberth mentioned this issue Oct 18, 2018

Installing scancode-toolkit from PyPI fails due to pefile issue #1214

Closed

pombredanne mentioned this issue Oct 31, 2018

Failed to run "pip install ." on Windows when package data contains unicode paths pypa/pip#5972

Closed

pombredanne added the Python3 label Feb 18, 2019

pombredanne closed this as completed Sep 24, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Running "pip install ." fails on Windows #755

Running "pip install ." fails on Windows #755

sschuberth commented Sep 13, 2017

pombredanne commented Sep 13, 2017

sschuberth commented Sep 13, 2017 •

edited

Loading

pombredanne commented Sep 13, 2017

pombredanne commented Sep 13, 2017

sschuberth commented Sep 13, 2017

pombredanne commented Sep 13, 2017

sschuberth commented Sep 13, 2017

pombredanne commented Sep 19, 2017 •

edited by sschuberth

Loading

pombredanne commented Oct 20, 2017

sschuberth commented Oct 20, 2017

pombredanne commented Oct 20, 2017

pombredanne commented Oct 30, 2018

sschuberth commented Oct 30, 2018

pombredanne commented Oct 30, 2018

sschuberth commented Oct 30, 2018

pombredanne commented Oct 30, 2018

sschuberth commented Oct 30, 2018

pombredanne commented Oct 30, 2018

pombredanne commented Oct 30, 2018

pombredanne commented Oct 31, 2018

sschuberth commented Oct 31, 2018 •

edited

Loading

pombredanne commented Oct 31, 2018

pombredanne commented Oct 31, 2018

sschuberth commented Oct 31, 2018

pombredanne commented Feb 18, 2019

pombredanne commented Sep 24, 2021

Running "pip install ." fails on Windows #755

Running "pip install ." fails on Windows #755

Comments

sschuberth commented Sep 13, 2017

pombredanne commented Sep 13, 2017

sschuberth commented Sep 13, 2017 • edited Loading

pombredanne commented Sep 13, 2017

pombredanne commented Sep 13, 2017

sschuberth commented Sep 13, 2017

pombredanne commented Sep 13, 2017

sschuberth commented Sep 13, 2017

pombredanne commented Sep 19, 2017 • edited by sschuberth Loading

pombredanne commented Oct 20, 2017

sschuberth commented Oct 20, 2017

pombredanne commented Oct 20, 2017

pombredanne commented Oct 30, 2018

sschuberth commented Oct 30, 2018

pombredanne commented Oct 30, 2018

sschuberth commented Oct 30, 2018

pombredanne commented Oct 30, 2018

sschuberth commented Oct 30, 2018

pombredanne commented Oct 30, 2018

pombredanne commented Oct 30, 2018

pombredanne commented Oct 31, 2018

sschuberth commented Oct 31, 2018 • edited Loading

pombredanne commented Oct 31, 2018

pombredanne commented Oct 31, 2018

sschuberth commented Oct 31, 2018

pombredanne commented Feb 18, 2019

pombredanne commented Sep 24, 2021

sschuberth commented Sep 13, 2017 •

edited

Loading

pombredanne commented Sep 19, 2017 •

edited by sschuberth

Loading

sschuberth commented Oct 31, 2018 •

edited

Loading