Allow __init__ with signature but no return type

[JukkaL]

Code:

class Visitor:
    def __init__(self, a: int):
        pass

Error:

x.py: In member "__init__" of class "Visitor":
x.py, line 2: Cannot define return type for "__init__"

The return type is Any (implicitly), and Any should be a valid return type for __init__.

This was reported by Guido.

[gvanrossum]

Hm, I think the return type for init should be None. A "return " inside an init is almost certainly caused by a misunderstanding. (Maybe you're thinking of new, which does return the object it created?)

[JukkaL]

Ah, the issue is that the error message is wrong. The message should say that the return type must be None? The message may be a relict from times before Python-compatible syntax.

[gvanrossum]

But why do I have to specify the return type at all? Can't it default to
None here?

On Wednesday, March 18, 2015, Jukka Lehtosalo [email protected]
wrote:

Closed #604 #604 via 182a70b
182a70b
.

—
Reply to this email directly or view it on GitHub
#604 (comment).

--Guido van Rossum (on iPad)

[JukkaL]

Because of consistency. Consider __init__ that takes no arguments, other than self:

def __init__(self):
    ...

Would this be dynamically typed or statically typed? The current approach is to treat any function that has no annotation (no argument or return types) as dynamically typed, and statically typed (i.e. type checked) otherwise. The None return type marks the function as type checked.

Same goes for functions in general:

def f():   # Dynamically typed
    ...
def f() -> None:   # Statically typed
    ...
def g(x: int):  # Statically typed, Any return type
    ...

[gvanrossum]

Hm. I think it'd suck if we'd have to develop the habit of adding explicit
"-> None" to all init methods just to make mypy happy.

On Thu, Mar 19, 2015 at 8:11 PM, Jukka Lehtosalo [email protected]
wrote:

Because of consistency. Consider init that takes no arguments, other
than self:

def init(self):
...

Would this be dynamically typed or statically typed? The current approach
is to treat any function that has no annotation (no argument or return
types) as dynamically typed, and statically typed (i.e. type checked)
otherwise. The None return type marks the function as type checked.

Same goes for functions in general:

def f(): # Dynamically typed
...
def f() -> None: # Statically typed
...
def g(x: int): # Statically typed, Any return type
...

—
Reply to this email directly or view it on GitHub
#604 (comment).

--Guido van Rossum (python.org/~guido)

[JukkaL]

Okay, let's reopen this. Here's a proposed new spec. Let me know if it matches your thinking.

-> None would only be needed for type checked __init__ methods that take no arguments:

def __init__(self) -> None:
    ...

(In my Python corpus, these account for about 15% of all __init__ methods. __init__ methods seem to cover about 10% of all methods.)

An __init__ method with any argument types would be type checked, and the return type would always be None implicitly, but you can also give an explicit None return type if you want.

What about other functions and methods that have arguments with types but don't have an explicit return type? Should they always default to an Any return type? I see a few potential alternatives to this:

1. Default to a None return type, similar to __init__.
2. Default to a None return type if there is no return statement with an explicit value in the body. Give an error otherwise.
3. Default to a None return type if there is no return statement with an explicit value in the body. Default to an Any return type otherwise.
4. Try to infer the return type automatically. I don't like this much because it adds complexity, reduces consistency and "explicit is better than implicit" in this case, in my opinion.

[gvanrossum]

It's a tricky issue, since we need to balance the desire to catch errors in the body of the function (what if a return type was intended but accidentally omitted?) with the need to derive the most useful return type for the benefit of checking call sites, as well as making the system be pleasant to use. I think for methods in general the status quo is fine, but (2) and (3) have some slight advantage, and some even slighter downside.

However I still think __init__ is a special case -- its return value is determined by how Python uses it, not by what the user might want it to return. I think the absence of -> None should never result in an error message here (and in fact -> something_else should be considered an error, even -> Any). The heuristic over whether a function is considered type-checked or not is somewhat unfortunate but I can live with the requirement to add a dummy -> None to type-check an __init__ method that has no arguments besides self.

[JukkaL]

I'm leaning towards using the same convention everywhere (2) -- missing return type (if the signature has argument types) is the same as -> None. One reason is that in many examples of PEP 484 / mypy code I've seen, people tend to omit return types, even though their intention was probably not to have an Any return type -- they just did the easiest/shortest thing. Programmers are lazy, and we probably shouldn't make the rare case (Any return) easier to write than the common case (None return) when we have a choice.

Still, a convention of using an explicit -> None even when it's not needed (except for __init__, where it isn't useful, as you argued) may be a reasonable thing. However, perhaps it shouldn't be enforced.

Here's some more stats (these are actually pretty interesting): In my corpus about half of all methods have signature (self) -- but only a small fraction of these are __init__ methods. Approx. 2/3 of (self) methods seem to have None as the expected return type. Top-level functions that take no arguments but have a None return type are much less frequent.

[gvanrossum]

I'm assuming those argument-less None-returning methods are some kind of pattern to set/clear specific flags? E.g. set_debug(). Would be interesting to look at these a bit more, since the stats look a bit suspicious. OTOH it's understandable that there aren't many top-level functions like that -- changing global state is frowned upon more than changing instance state.

[gvanrossum]

I also realized that I'm not sure what your proposal is, exactly. Do you propose to assume ->None for all functions and methods that don't have an explicit return annotation? Or only for those that have at least one argument annotation? Or only for those that have no "return " in their body? (And how would the latter differ?)

[JukkaL]

Sorry for being vague. Here's a more explicit version of my proposal:

If a function or method has at least one argument annotation and no return type annotation, the return type would implicitly be -> None. If there is a return statement with a value within the function, it would be handled identically to a function with n explicit -> None return (it would typically result in an error).

The body of a function is not type checked, if it has no argument type annotations and no return type annotation. All the argument types and the return type are implicitly Any for a such function. This way the function can be called in statically typed code and the return value can be used in any way.

Here's a sample of 40 method names with only self as an argument from my corpus:

    def _writemarkers(self):
    def testFilterOnAnyCategory_OneUsedAndOneUnusedCategory(self):
    def getreply(self):
    def setUp(self):
    def test_source(self):
    def test_istitle(self):
    def ABD_method(self):
    def MakeCopiesButton_clicked(self):
    def NextButton_clicked(self):
    def __init__(self):
    def test_handle_expt(self):
    def getwidth(self):
    def testTest(self):
    def test_read_longlink(self):
    def addpackers(self):
    def test_using_mapping(self):
    def __do_layout(self):
    def __getstate__(self):
    def flush(self):
    def slotMapSelectionChanged(self):
    def tearDown(self):
    def test_unicode_filenames(self):
    def _find_swig_libdir(self):
    def tearDown(self):
    def testFloats(self):
    def sendObject(self):
    def test_failing_reload(self):
    def testReverseSortOrderWithGrandchildren(self):
    def __init__(self):
    def copy(self):
    def testChangeDueDate(self):
    def __len__(self):
    def testTwoAttachmentsCompat(self):
    def testEllipsis(self):
    def test_lineno(self):
    def IsExpandable(self):
    def modifiedFiles(self):
    def test_zones(self):
    def redo(self):
    def __init__(self):

I didn't look at the implementations, but I'd guess about 70% of the methods would have -> None as the return type. Note that mostly that is due to test methods. Whether test cases should be statically typed is a matter of taste, but type checking can verify that your type annotations are consistent with the way you use your code in tests.

[gvanrossum]

OK, the proposal looks good. Unit tests are a rather special case in a sense -- the TestCase class encourages having lots of parameter-less None-returning functions (all test functions, of course, but also setUp and tearDown, and often code shared between tests is also written in this style). Do you have a sample that excludes tests?

[JukkaL]

Here's a sample where test, setUp and tearDown methods are filtered out:

    def setlist(self):
    def GetMeasuringFont(self):
    def __unicode__(self):
    def isAnyItemCollapsable(self):
    def make_html_file(self):
    def setup_shlib_compiler(self):
    def __init__(self):
    def ShouldInheritColours(self):
    def __set_event(self):
    def InitMisc(self):
    def tagBody(self):
    def current_frames_with_threads(self):
    def __iter__(self):
    def _open(self):
    def makerepairinstructions(self):
    def __repr__(self):
    def __getMainWindow(self):
    def createRecurrence(self):
    def defaults(self):
    def nextCoverpageButton_clicked(self):
    def GetText(self):
    def createTests(self):
    def getWidget(self):
    def showcontents(self):
    def onFinish(self):
    def getfp(self):
    def __init__(self):
    def accept(self):
    def __repr__(self):
    def getparams(self):
    def _check_choice(self):
    def get_clock_seq_hi_variant(self):
    def is_namespace(self):
    def getChildNodes(self):
    def is_regression(self): return _ml.CvKNearest_is_regression(self)
    def _get_float_longitude(self):
    def get_yolks(self):
    def initUi(self):
    def GetType(self):
    def __init__(self):

Now it's actually a bit tricky to figure out which of these will return a value, but it seems that a fairly small fraction would have -> None return type, maybe ~25%. On average, the explicit -> None return type would be needed on roughly 1 line out of 200 in my corpus.

[nfergu]

As far as I can understand it, what was proposed here doesn't seem to have been implemented.

Consider this test case (from check-classes.test)

[case testConstructorWithImplicitReturnValueType]
import typing
class A:
    def __init__(self, x: int): pass
[out]
main: In member "__init__" of class "A":
main, line 3: The return type of "__init__" must be None

This suggests that an explicit return type of None must be specified, even though the __init__ method has arguments.

This appears to go against the proposal here which was (as far as I understand it) to only enforce this for __init__ methods that have not arguments.

As @gvanrossum points out above, it's a bit cumbersome to have to add "-> None" to all init methods just to make mypy happy.

Can anyone comment?

[JukkaL]

We decided to require the explicit -> None for consistency. It's a bit of extra work but fairly trivial for any significant codebase.

[henryJack]

It's a real shame to have to add -> None to every __init__ method. I'm not sure why this is needed for consistency as adding a return command will already give you a pylint error,

http://pylint-messages.wikidot.com/messages:e0101

and a TypeError at runtime,

https://docs.python.org/2/reference/datamodel.html#object.__init__

...Is there a way to, at least, suppress these errors?

EDIT: Actually, I think I can live with this and agree with the reasoning. Thanks for the awesome tool!

[henryJack]

You can also use the following regexp's to find and replace offending __init__ methods. Helpful if you have a large code base!

For init's on a single line --> find: (def __init__\(self,?.*\)), replace: $1 -> None

For init's accross multiple lines --> find: (def __init__\(self[^)]*)\)(?!\s->), replace: $1) -> None

[henryJack]

Has there been any discussion about an automypy tool analogous to the autopep8 tool?

[JukkaL]

There has been a lot of discussion about various tools for automatically generating annotations. A tool that would automatically improve annotations by inserting missing -> None types could be useful. Another somewhat related thing that could be nice would be automatic addition of string literal escapes when they are needed for forward references.

[rostislav-simonik]

@gvanrossum I'm little bit confused. Is it possible to return value by __init__ method ? Am I missing something.

[gvanrossum]

No, __init__ cannot return a value, hence you must specify -> None here.

[rostislav-simonik]

Oh, ok. I don't understand why mypy requires explicit type declaration for __init__ ?
I assume the goal is to provide type validation with minimal code burden. Shouldn't be mypy clever on this? Is there any risk mitigation that I don't see? Thanks for explanation.

[Michael0x2a]

@rostislav-simonik -- I would imagine because if you have an __init__ function that accepts no parameters, it's ambiguous if it should be typechecked or not without that annotation (and being able to clearly delineate between typed and untyped functions and mix the two in a sane way is one of the core features of PEP 484).

Basically, we add a little bit of redundancy in exchange for consistency -- as the Zen of Python says, "In the face of ambiguity, refuse the temptation to guess".

[onlined]

What is the status? Is final decision made? Is somebody assigned with the implementation? I can work on it, if there is no one available.

[JukkaL]

@onlined We are happy to accept a PR that implements the proposal. Nobody is working on this yet.

[ethanhs]

This was resolved in #5677. Thank you @onlined!