A tale about support for type hints for the addition function in Python, or Here’s how difficult it can be to go to IT …

Type hints are great! But not so long ago I played devil’s advocate: I argued that these hints can actually be annoying, especially for old-school Python programmers.

I think many were skeptical about this, and therefore let’s look at one completely fictional situation. Unless otherwise stated, everything in it is fictional. While editing the text, I realized that in attempts 4–6 there are even more errors than expected, but I will not rewrite again.

So exactly You support a popular third-party library slowadd. It has a lot of helper functions, decorators, classes and metaclasses, but here is the main function:

def slow_add(a, b):
    time.sleep(0.1)
    return a + b

Types

You have always worked with traditional duck typing: if a And b do not add up, the function throws an exception. But you’ve just dropped support for Python 2, so users are demanding type hints.

Attempt #1

You add simple hints:

def slow_add(a: int, b: int) -> int:
    time.sleep(0.1)
    return a + b

All tests pass, the codebase is mypy compliant, and in the release notes you write “Type hint support added!”.

Attempt #2

Users are flooding GitHub Issues with complaints! MyPy doesn’t work because slow_add floating point numbers are passed. This breaks the assembly, and due to internal organizational policies that always increase type hint coverage, you cannot roll back to the old version. Your users’ weekends are corrupted.

You investigate the problem and it turns out that for the type chain ints -> float -> complex MyPy supports duck typing compatibility. Cool!

Here is the new release:

def slow_add(a: complex, b: complex) -> complex:
    time.sleep(0.1)
    return a + b

It’s funny that this is a MyPy note and not a PEP standard…

Attempt #5

Users are grateful for the speed, but after a couple of days, one of them asks why Decimal no longer supported. You are replacing the type complex on Decimal and your other MyPy tests crash.

Python 3 introduced numeric abstract base classesso the ideal option is to simply hint types as numbers.Number.

But MyPy doesn’t consider integers, floats, or decimals as numbers. After reading about typingyou guess: the point is Decimals And Union:

def slow_add(
    a: Union[complex, Decimal], b: Union[complex, Decimal]
) -> Union[complex, Decimal]:
    time.sleep(0.1)
    return a + b

Oh no! Now MyPy is complaining that you can’t add other types of numbers to Decimal. Well, that’s not what you wanted anyway… A little more reading and you’ll try the overload:

@overload
def slow_add(a: Decimal, b: Decimal) -> Decimal:
    ...

@overload
def slow_add(a: complex, b: complex) -> complex:
    ...

def slow_add(a, b):
    time.sleep(0.1)
    return a + b

And now strict MyPy complains about missing in slow_add type annotation. After reading about this problem Do you understand that @overload only useful to users, and the body of the function is no longer checked in mypy. It’s good that in the discussion of the problem there was an example of the implementation of the solution to the problem:

T = TypeVar("T", Decimal, complex)

def slow_add(a: T, b: T) -> T:
    time.sleep(0.1)
    return a + b

Attempt #4

You are rolling out a new release. After a few days, more and more users start complaining. A very enthusiastic user explains a particularly critical use case with tuples: slow_add((1, ), (2, )). And I don’t want to add each type over and over again, there must be a better way!

You learn protocols, type variables, and only positional parameters… uv… a lot of things, but now something should be perfect:

T = TypeVar("T")

class Addable(Protocol):
    def __add__(self: T, other: T, /) -> T:
        ...

def slow_add(a: Addable, b: Addable) -> Addable:
    time.sleep(0.1)
    return a + b

Let’s digress a little

You roll out a new release again, noting in the Release Notes that “any type you add is now supported.”

The tuple user again immediately complains and says that hints don’t work for tuples longer, like this: slow_add((1, 2), (3, 4)). This is odd, since you’ve checked multiple tuple lengths.

After debugging the user environment by running back and forth on GitHub, you saw that pyright throws the code above as an error, but MyPy does not, even in strict mode. Presumably MyPy behaves correctly, so you continue to bliss, ignoring the fundamental error.

If MyPy is not working correctly, then Pyright should clearly throw an error. I reported this to both projects, and decision details, if you’re interested, the maintainer explained. Unfortunately, these details were not taken into account until “Attempt #7”.

Attempt #5

A week later, a user reports a problem: the latest release says that “any type you add is now supported”, but they have a bunch of classes that you can implement only by using __radd__and the new version throws typing errors.

You are trying several approaches. This solves the problem best:

T = TypeVar("T")

class Addable(Protocol):
    def __add__(self: T, other: T, /) -> T:
        ...

class RAddable(Protocol):
    def __radd__(self: T, other: Any, /) -> T:
        ...

@overload
def slow_add(a: Addable, b: Addable) -> Addable:
    ...

@overload
def slow_add(a: Any, b: RAddable) -> RAddable:
    ...

def slow_add(a: Any, b: Any) -> Any:
    time.sleep(0.1)
    return a + b

It’s annoying that now MyPy doesn’t have a consistent approach to the function body. And it has not yet been fully expressed the condition that when b – RAddable, a should not be the same type, because Python type annotations do not yet support type exclusions.

Attempt #6

A couple of days later, a new user complains that they are getting type hint errors when trying to raise the output of our main function to a power: pow(slow_add(1, 1), slow_add(1, 1)). It’s not that bad, you quickly realize that the problem is with protocol annotations. In fact, it is not protocols that need to be annotated, but type variables:

T = TypeVar("T")

class Addable(Protocol):
    def __add__(self: T, other: T, /) -> T:
        ...

A = TypeVar("A", bound=Addable)

class RAddable(Protocol):
    def __radd__(self: T, other: Any, /) -> T:
        ...

R = TypeVar("R", bound=RAddable)

@overload
def slow_add(a: A, b: A) -> A:
    ...

@overload
def slow_add(a: Any, b: R) -> R:
    ...

def slow_add(a: Any, b: Any) -> Any:
    time.sleep(0.1)
    return a + b

Attempt #7

The tuple user is back with us! Now it says that MyPy in strict mode complains about the expression slow_add((1,), (2,)) == (1, 2):

Non-overlapping equality check (left operand type: “Tuple[int]”, right operand type: “Tuple[int, int]”)

You understand that you cannot guarantee anything about the type returned from an arbitrary __add__ or __radd__ values, and therefore you begin to generously scatter Any:

class Addable(Protocol):
    def __add__(self: "Addable", other: Any, /) -> Any:
        ...

class RAddable(Protocol):
    def __radd__(self: "RAddable", other: Any, /) -> Any:
        ...

@overload
def slow_add(a: Addable, b: Any) -> Any:
    ...

@overload
def slow_add(a: Any, b: RAddable) -> Any:
    ...

def slow_add(a: Any, b: Any) -> Any:
    time.sleep(0.1)
    return a + b

Attempt #8

Users just went crazy! The nice automatic type guesses that were in the last release of their IDE are now gone! Well, you can’t type hint everything, but you could include built-in type hints and, May besome types of the standard library, for example Decimal:

You decide you can rely on some MyPy duck types, but check this code:

@overload
def slow_add(a: complex, b: complex) -> complex:
    ...

And you realize that MyPy throws an error on something like slow_add(1, 1.0).as_integer_ratio(). So what you end up doing is:

class Addable(Protocol):
    def __add__(self: "Addable", other: Any, /) -> Any:
        ...

class RAddable(Protocol):
    def __radd__(self: "RAddable", other: Any, /) -> Any:
        ...

@overload
def slow_add(a: int, b: int) -> int:
    ...

@overload
def slow_add(a: float, b: float) -> float:
    ...

@overload
def slow_add(a: complex, b: complex) -> complex:
    ...

@overload
def slow_add(a: str, b: str) -> str:
    ...

@overload
def slow_add(a: tuple[Any, ...], b: tuple[Any, ...]) -> tuple[Any, ...]:
    ...

@overload
def slow_add(a: list[Any], b: list[Any]) -> list[Any]:
    ...

@overload
def slow_add(a: Decimal, b: Decimal) -> Decimal:
    ...

@overload
def slow_add(a: Fraction, b: Fraction) -> Fraction:
    ...

@overload
def slow_add(a: Addable, b: Any) -> Any:
    ...

@overload
def slow_add(a: Any, b: RAddable) -> Any:
    ...

def slow_add(a: Any, b: Any) -> Any:
    time.sleep(0.1)
    return a + b

As already mentioned, MyPy does not use overload signatures and compares them with the function body, so you must check all these type hints for accuracy yourself, manually.

Attempt #9

A few months later, the user says he’s using an embedded version of Python that doesn’t have Decimal. Why would your package import it at all? So now the code looks like this:

from __future__ import annotations

import time
from typing import TYPE_CHECKING, Any, Protocol, TypeVar, overload

if TYPE_CHECKING:
    from decimal import Decimal
    from fractions import Fraction

class Addable(Protocol):
    def __add__(self: "Addable", other: Any, /) -> Any:
        ...

class RAddable(Protocol):
    def __radd__(self: "RAddable", other: Any, /) -> Any:
        ...

@overload
def slow_add(a: int, b: int) -> int:
    ...

@overload
def slow_add(a: float, b: float) -> float:
    ...

@overload
def slow_add(a: complex, b: complex) -> complex:
    ...

@overload
def slow_add(a: str, b: str) -> str:
    ...

@overload
def slow_add(a: tuple[Any, ...], b: tuple[Any, ...]) -> tuple[Any, ...]:
    ...

@overload
def slow_add(a: list[Any], b: list[Any]) -> list[Any]:
    ...

@overload
def slow_add(a: Decimal, b: Decimal) -> Decimal:
    ...

@overload
def slow_add(a: Fraction, b: Fraction) -> Fraction:
    ...

@overload
def slow_add(a: Addable, b: Any) -> Any:
    ...

@overload
def slow_add(a: Any, b: RAddable) -> Any:
    ...

def slow_add(a: Any, b: Any) -> Any:
    time.sleep(0.1)
    return a + b

TL;DR

Turning even the simplest function that relies on duck typing into a useful function with type hinting can be excruciatingly difficult. Please always show empathy when asking someone to update the code in the way you think it should work.

While writing this post, I learned a lot about type hints. Please try to find edge cases where my type hints are wrong or could get better. This is a good exercise.

Another amendment: I gave up on fixes late at night, but smart people spotted the bugs! I have a “tenth attempt” to fix them. But pyright complains because my overloads overlap. But I don’t think there is a way to express what you want in annotations without overlap.

Mypy complains that sometimes previously posted user code throws an error comparison-overlap, Interestingly. But it looks like the lack of custom code overlaps here can be seen by pyright.

I’ll describe pyright and mypy issues on Github, although they may be mostly architecture driven, i.e. being a limitation of the current state of Python’s type hints in general:

T = TypeVar("T")

class Addable(Protocol):
    def __add__(self: "Addable", other: Any, /) -> Any:
        ...

class SameAddable(Protocol):
    def __add__(self: T, other: T, /) -> T:
        ...

class RAddable(Protocol):
    def __radd__(self: "RAddable", other: Any, /) -> Any:
        ...

class SameRAddable(Protocol):
    def __radd__(self: T, other: Any, /) -> T:
        ...

SA = TypeVar("SA", bound=SameAddable)
RA = TypeVar("RA", bound=SameRAddable)

@overload
def slow_add(a: SA, b: SA) -> SA:
    ...

@overload
def slow_add(a: Addable, b: Any) -> Any:
    ...

@overload
def slow_add(a: Any, b: RA) -> RA:
    ...

@overload
def slow_add(a: Any, b: RAddable) -> Any:
    ...

def slow_add(a: Any, b: Any) -> Any:
    time.sleep(0.1)
    return a + b

That’s how difficult it can be in IT. But we will help you master the necessary theory, gain useful experience and, if difficulties do not scare you, get a job in the field of information technology: