setuptools >= 20.2 may break applications using pyparsing

[sekrause]

I'm developing an application that uses pyparsing and after upgrading setuptools to the newest version I noticed some tests failing. In my main parser module I define an alias for the ParseBaseException which I then use in other parts of the application to catch exceptions:

# definition of the ParseException
ParseException = pyparsing.ParseBaseException

# importing this alias in another module
from ...filterreader.parser import ParseException

Now my tests were failing because the ParseException was never actually caught. Some investigation by comparing the id() of the objects showed that the ParseException alias was no longer the same object as pyparsing.ParseBaseException. This was because the module pyparsing at the time of the alias definition was not the same pyparsing module which is later used for parsing. Looking at sys.module I can see that I have two pyparsing modules:

pyparsing: <module 'pyparsing' from 'D:\\Project\\python34.zip\\pyparsing.pyc'>
pkg_resources.extern.pyparsing: <module 'pyparsing' from 'D:\\Project\\python34.zip\\pyparsing.pyc'>

At the time of the alias definition id(pyparsing) is equal to the id() of pkg_resources.extern.pyparsing. When I later import pyparsing I get the other module. This whole problem only happens when I use the application packaged by cx_Freeze, so maybe some kind of race condition happens when importing from a ZIP file. I'm using 64 bit Python 3.4.4 on Windows.

The first version of setuptools where I can see this problem is 20.2, until 20.1 everything is fine. Looking at the source I can see that starting with 20.2 setuptools also includes its own pyparsing copy, so most likely that change is related to my problem.

[jaraco]

Hi Sebastian. Sorry to hear this change is causing you problems. Unfortunately, removing pyparsing from the vendored packages isn't an easy operation.

It's quite possible that the extern technique isn't viable in the cx_Freeze environment, but I'm not even sure where to start. Can you describe the steps necessary to create a (preferably minimal) cx_Freeze package that demonstrates this behavior?

[warsaw]

We so badly need a common, principled way to vendorize! pip's approach seems to be the current best in breed.

[jaraco]

@warsaw: Agreed. The setuptools implementation rose out of #229, which was originally based on the vendoring of packaging, which was implemented by dstufft, presumably in the same spirit as the vendoring used in pip, but had the downside of making imports very ugly and error prone. I haven't reviewed pip's current vendoring scheme, so I'd welcome someone to review it and assess the viability.

[sekrause]

I'm currently trying to investigate this problem a bit more, but creating a mininal example environment with cx_Freeze seems to be difficult.

Theoretically having two pyparsing modules should to be just fine, as long as they have different, stable module names. There problem here seems to be that at some point the pkg_resources.extern.pyparsing module is just named pyparsing and then gets renamed at some point.

[sekrause]

Also note that at no point the pyparsing from pkg_resources/_vendor/pyparsing.py is actually used in my frozen application. The vendored package of pyparsing is 2.0.6 while I also include pyparsing 2.1.5 directly.

Both the modules pyparsing and pkg_resources.extern.pyparsing are later 2.1.5.

[jaraco]

There problem here seems to be that at some point the pkg_resources.extern.pyparsing module is just named pyparsing and then gets renamed at some point.

That seems incorrect to me. Throughout the pkg_resources code, only pkg_resources.extern.pyparsing is imported. Can you demonstrate what you mean or explain it in another way?

Both the modules pyparsing and pkg_resources.extern.pyparsing are later 2.1.5.

The way the vendoring code works is it prefers the vendored version if available but will use the canonical version if the vendored version is not available and it installs this resolved package into the extern namespace. So I would expect on a system where pkg_resources._vendor.pyparsing is missing or broken, to resolve pkg_resources.extern.pyparsing as the root pyparsing (i.e. 2.1.5).

Looking at sys.module I can see that I have two pyparsing modules.

This is expected. Due to the preference for the vendored package, a preference which @dstufft said was preferable because it meant that pkg_resources could more reliably depend on a specific version, there would have to be two modules.

Now my tests were failing because the ParseException was never actually caught.

Can you confirm that the uncaught ParseExceptions are coming from calls in pkg_resources? If so, you won't be able to rely on pyparsing.ParseException to catch those, but you'll need to catch the exception from the extern namespace (pkg_resources.extern.pyparsing.ParseException).