working with third party libs without stubs

[chadrik]

Hi,
mypy is crashing in what I think is a pretty straight-forward situation. This seems to me like a pretty important UX issue, related to python/typing#84

I'm running mypy on a library that contains type annotations. It depends on another library that also contains type annotations, but no stubs. Additionally, because the project supports python 2.x, I'm installing the latest version of the typing module.

In order to find the type-info in the dependency, which was pip-installed into a virtual env, I have to add the site-packages directory of the virtual env to MYPYPATH, but doing so causes mypy to get tripped up by the typing module that lives there, which aborts the whole operation:

(.env2) chad$ python3 -m mypy --py2 --follow-imports=silent my_awesome_package
.env2/lib/python2.7/site-packages/typing.py:1355: error: Cannot determine consistent method resolution order (MRO) for "TupleMeta"
.env2/lib/python2.7/site-packages/typing.py:1410: error: Cannot determine consistent method resolution order (MRO) for "CallableMeta"
.env2/lib/python2.7/site-packages/typing.py:1601: error: Cannot determine consistent method resolution order (MRO) for "_ProtocolMeta"
.env2/lib/python2.7/site-packages/typing.py:1985: error: INTERNAL ERROR -- please report a bug at https:/python/mypy/issues version: 0.521
.env2/lib/python2.7/site-packages/typing.py:1985: note: please use --show-traceback to print a traceback when reporting a bug

I've ensured that python3 and the venv python both have the latest typing:

(.env2) chad$ python3 -m pip install -U typing
Requirement already up-to-date: typing in /usr/local/lib/python3.6/site-packages
(.env2) chad$ python -m pip install -U typing
Requirement already up-to-date: typing in ./.env2/lib/python2.7/site-packages

Here's the traceback:

Traceback (most recent call last):
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.6/site-packages/mypy/__main__.py", line 11, in <module>
    main(None)
  File "/usr/local/lib/python3.6/site-packages/mypy/main.py", line 50, in main
    res = type_check_only(sources, bin_dir, options)
  File "/usr/local/lib/python3.6/site-packages/mypy/main.py", line 97, in type_check_only
    options=options)
  File "/usr/local/lib/python3.6/site-packages/mypy/build.py", line 196, in build
    graph = dispatch(sources, manager)
  File "/usr/local/lib/python3.6/site-packages/mypy/build.py", line 1801, in dispatch
    process_graph(graph, manager)
  File "/usr/local/lib/python3.6/site-packages/mypy/build.py", line 2044, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/usr/local/lib/python3.6/site-packages/mypy/build.py", line 2139, in process_stale_scc
    graph[id].semantic_analysis()
  File "/usr/local/lib/python3.6/site-packages/mypy/build.py", line 1696, in semantic_analysis
    self.manager.semantic_analyzer.visit_file(self.tree, self.xpath, self.options, patches)
  File "/usr/local/lib/python3.6/site-packages/mypy/semanal.py", line 295, in visit_file
    self.accept(d)
  File "/usr/local/lib/python3.6/site-packages/mypy/semanal.py", line 3645, in accept
    node.accept(self)
  File "/usr/local/lib/python3.6/site-packages/mypy/nodes.py", line 565, in accept
    return visitor.visit_func_def(self)
  File "/usr/local/lib/python3.6/site-packages/mypy/semanal.py", line 418, in visit_func_def
    self.analyze_function(defn)
  File "/usr/local/lib/python3.6/site-packages/mypy/semanal.py", line 627, in analyze_function
    defn.body.accept(self)
  File "/usr/local/lib/python3.6/site-packages/mypy/nodes.py", line 815, in accept
    return visitor.visit_block(self)
  File "/usr/local/lib/python3.6/site-packages/mypy/semanal.py", line 1522, in visit_block
    self.accept(s)
  File "/usr/local/lib/python3.6/site-packages/mypy/semanal.py", line 3645, in accept
    node.accept(self)
  File "/usr/local/lib/python3.6/site-packages/mypy/nodes.py", line 859, in accept
    return visitor.visit_assignment_stmt(self)
  File "/usr/local/lib/python3.6/site-packages/mypy/semanal.py", line 1581, in visit_assignment_stmt
    self.process_namedtuple_definition(s)
  File "/usr/local/lib/python3.6/site-packages/mypy/semanal.py", line 2126, in process_namedtuple_definition
    named_tuple = self.check_namedtuple(s.rvalue, name)
  File "/usr/local/lib/python3.6/site-packages/mypy/semanal.py", line 2157, in check_namedtuple
    return self.build_namedtuple_typeinfo('namedtuple', [], [], {})
  File "/usr/local/lib/python3.6/site-packages/mypy/semanal.py", line 2275, in build_namedtuple_typeinfo
    info = self.basic_new_typeinfo(name, fallback)
  File "/usr/local/lib/python3.6/site-packages/mypy/semanal.py", line 2251, in basic_new_typeinfo
    info.mro = [info] + basetype_or_fallback.type.mro
TypeError: can only concatenate list (not "NoneType") to list

Next, I used pip to uninstall typing and re-ran mypy. This prevents mypy from crashing, but it still gets tripped up by other dependencies in site-packages, this time on one which contains no typing information whatsoever.

(.env2) chad$ python3 -m mypy --py2 --show-traceback --follow-imports=silent my_awesome_package
.env2/lib/python2.7/site-packages/schematics/datastructures.py:15: error: Cannot determine consistent method resolution order (MRO) for "OrderedDict

If I don't include site-packages in MYPYPATH, I get more details about type errors in my own package, so it appears this error is causing mypy to abort additional phases of type checking.

It's unclear what I should try next. Any help would be greatly appreciated. Thanks!

[gvanrossum]

Looks like you manipulated MYPYPATH to point to the stdlib so that mypy tries to analyze the real (runtime-only) typing.py module instead of its typing.pyi stub in typeshed. There's a warning in the docs specifically against doing this.

[chadrik]

I set it to the virtual-env's site-packages directory:

export MYPYPATH=.env2/lib/python2.7/site-packages/

As I mentioned, there is a typing.py there, but removing it just raised a different error.

[ethanhs]

This is a duplicate of #1876, you may also want to look at related #3350. We specifically do not support pointing to site-packages intentionally, as it causes typing to be picked up from there, which we want to avoid.

[chadrik]

So the gist is that developers need to maintain a set of stubs for their library even if they're already using type annotations within their code as intended. It's kind of a bummer. Really hoping to see some movement on python/typing#84.

[ethanhs]

@chadrik I have started writing a formal PEP. I hope to have a very early draft viable for commenting by the end of this weekend.

[chadrik]

@ethanhs glad to hear it!