[Python-ideas] Runtime types vs static types

[Koos Zevenhoven]

There has been some discussion here and there concerning the differences
between runtime types and static types (mypy etc.). What I write below is
not really an idea or proposal---just a perspective, or a topic that people
may want to discuss. Since the discussion on this is currently very fuzzy
and scattered and not really happening either AFAICT (I've probably missed
many discussions, though). Anyway, I thought I'd give it a shot:

Clearly, there needs to be some sort of distinction between runtime
classes/types and static types, because static types can be more precise
than Python's dynamic runtime semantics. For example, Iterable[int] is an
iterable that contains integers. For a static type checker, it is clear
what this means. But at runtime, it may be impossible to figure out whether
an iterable is really of this type without consuming the whole iterable and
checking whether each yielded element is an integer. Even that is not
possible if the iterable is infinite. Even Sequence[int] is problematic,
because checking the types of all elements of the sequence could take a
long time.

Since things like isinstance(it, Iterable[int]) cannot guarantee a proper
answer, one easily arrives at the conclusion that static types and runtime
classes are just two separate things and that one cannot require that all
types support something like isinstance at runtime.

On the other hand, there are many runtime things that can or could be done
using (type) annotations, for example:

Multidispatch (example with hypothetical syntax below):

@overload
def concatenate(parts: Iterable[str]) -> str:
    return "".join(parts)

@overload
def concatenate(parts: Iterable[bytes]) -> bytes:
    return b"".join(parts)

@overload
def concatenate(parts: Iterable[Iterable]) -> Iterable:
    return itertools.chain(*parts)

or runtime type checking:

@check_types
def load_from_file(filename: Union[os.PathLike, str, bytes]):
    with open(filename) as f:
        return do_stuff_with(f.read())

which would automatically give a nice error message if, say, a file object
is given as argument instead of a path to a file.

However useful (and efficient) these things might be, the runtime type
checks are problematic, as discussed above.

Furthermore, other differences between runtime and static typing may emerge
(or have emerged), which will complicate the matter further. For instance,
the runtime __annotations__ of classes, modules and functions may in some
cases contain something completely different from what a type checker
thinks the type should be.

These and other incompatibilities between runtime and static typing will
create two (or more) different kinds of type-annotated Python:
runtime-oriented Python and Python with static type checking. These may be
incompatible in both directions: a static type checker may complain about
code that is perfectly valid for the runtime folks, and code written for
static type checking may not be able to use new Python techniques that make
use of type hints at runtime. There may not even be a fully functional
subset of the two "languages". Different libraries will adhere to different
standards and will not be compatible with each other. The split will be
much worse and more difficult to understand than Python 2 vs 3, peoples
around the world will suffer like never before, and programming in Python
will become a very complicated mess.


One way of solving the problem would be that type annotations are only a
static concept, like with stubs or comment-based type annotations. This
would also be nice from a memory and performance perspective, as evaluating
and storing the annotations would not occupy memory (although both issues
and some more might be nicely solved by making the annotations lazily
ealuated). However, leaving out runtime effects of type annotations is not
the approach taken, and runtime introspection of annotations seems to have
some promising applications as well. And for many cases, the traditional
Python class actually acts very nicely as both the runtime and static type.

So if type annotations will be both for runtime and for static checking,
how to make everything work for both static and runtime typing?

Since a writer of a library does not know what the type hints will be used
for by the library users, it is very important that there is only one way
of making type annotations which will work regardless of what the
annotations are used for in the end. This will also make it much easier to
learn Python typing.

Regarding runtime types and isinstance, let's look at the Iterable[int]
example. For this case, there are a few options:

1) Don't implement isinstance

This is problematic for runtime uses of annotations.

2) isinstance([1, '2', 'three'], Iterable[int]) returns True

This is in fact now the case. This is ok for many runtime situations, but
lacks precision compared to the static version. One may want to distinguish
between Iterable[int] and Iterable[str] at runtime (e.g. the multidispatch
example above).

3) Check as much as you can at runtime

There could be something like Reiterable, which means the object is not
consumed by iterating over it, so one could actually check if all elements
are instances of int. This would be useful in some situations, but not
available for every object. Furthermore, the check could take an arbitrary
amount of time so it is not really suitable for things like multidispatch
or some matching constructs etc., where the performance overhead of the
type check is really important.

4) Do a deeper check than in (2) but trust the annotations

For example, an instance of a class that has a method like

def __iter__(self) -> Iterator[int]:
    some code

could be identified as Iterable[int] at runtime, even if it is not
guaranteed that all elements are really integers.

On the other hand, an object returned by

def get_ints() -> Iterable[int]:
    some code

does not know its own annotations, so the check is difficult to do at
runtime. And of course, there may not be annotations available.

5) Something else?


And what about PEP544 (protocols), which is being drafted? The PEP seems to
aim for having type objects that represent duck-typing
protocols/interfaces. Checking whether a protocol is implemented by an
object or type is clearly a useful thing to do at runtime, but it is not
really clear if isinstance would be a guaranteed feature for PEP544
Protocols.

So one question is, is it possible to draw the lines between what works
with isinstance and what doesn't, and between what details are checked by
isinstance and what aren't? -- Or should insinstance be reserved for a more
limited purpose, and add another check function, say `implements(...)`,
which would perhaps guarantee some answer for all combinations of object
and type?

I'll stop here---this email is probably already much longer than a single
email should be ;)

-- Koos


-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20170624/26256b6e/attachment.html>