Mailman 3 July 2017 - Python-ideas

PEP: Hide implementation details in the C API
by Victor Stinner July 13, 2017

July 13, 2017

Hi, This is the first draft of a big (?) project to prepare CPython to be able to "modernize" its implementation. Proposed changes should allow to make CPython more efficient in the future. The optimizations themself are out of the scope of the PEP, but some examples are listed to explain why these changes are needed. For the background, see also my talk at the previous Python Language Summit at Pycon US, Portland OR: "Keeping Python competitive" https://lwn.net/Articles/723752/#723949 "… [View More]Python performance", slides (PDF): https://github.com/haypo/conf/raw/master/2017-PyconUS/summit.pdf Since this is really the first draft, I didn't assign a PEP number to it yet. I prefer to wait for a first feedback round. Victor PEP: xxx Title: Hide implementation details in the C API Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner <victor.stinner(a)gmail.com>, Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 31-May-2017 Abstract ======== Modify the C API to remove implementation details. Add an opt-in option to compile C extensions to get the old full API with implementation details. The modified C API allows to more easily experiment new optimizations: * Indirect Reference Counting * Remove Reference Counting, New Garbage Collector * Remove the GIL * Tagged pointers Reference counting may be emulated in a future implementation for backward compatibility. Rationale ========= History of CPython forks ------------------------ Last 10 years, CPython was forked multiple times to attempt different CPython enhancements: * Unladen Swallow: add a JIT compiler based on LLVM * Pyston: add a JIT compiler based on LLVM (CPython 2.7 fork) * Pyjion: add a JIT compiler based on Microsoft CLR * Gilectomy: remove the Global Interpreter Lock nicknamed "GIL" * etc. Sadly, none is this project has been merged back into CPython. Unladen Swallow looses its funding from Google, Pyston looses its funding from Dropbox, Pyjion is developed in the limited spare time of two Microsoft employees. One hard technically issue which blocked these projects to really unleash their power is the C API of CPython. Many old technical choices of CPython are hardcoded in this API: * reference counting * garbage collector * C structures like PyObject which contains headers for reference counting and the garbage collector * specific memory allocators * etc. PyPy ---- PyPy uses more efficient structures and use a more efficient garbage collector without reference counting. Thanks to that (but also many other optimizations), PyPy succeeded to run Python code up to 5x faster than CPython. Plan made of multiple small steps ================================= Step 1: split Include/ into subdirectories ------------------------------------------ Split the ``Include/`` directory of CPython: * ``python`` API: ``Include/Python.h`` remains the default C API * ``core`` API: ``Include/core/Python.h`` is a new C API designed for building Python * ``stable`` API: ``Include/stable/Python.h`` is the stable ABI Expect declarations to be duplicated on purpose: ``#include`` should be not used to include files from a different API to prevent mistakes. In the past, too many functions were exposed *by mistake*, especially symbols exported to the stable ABI by mistake. At this point, ``Include/Python.h`` is not changed at all: zero risk of backward incompatibility. The ``core`` API is the most complete API exposing *all* implementation details and use macros for best performances. XXX should we abandon the stable ABI? Never really used by anyone. Step 2: Add an opt-in API option to tools building packages ----------------------------------------------------------- Modify Python packaging tools (distutils, setuptools, flit, pip, etc.) to add an opt-in option to choose the API: ``python``, ``core`` or ``stable``. For example, debuggers like ``vmprof`` need need the ``core`` API to get a full access to implementation details. XXX handle backward compatibility for packaging tools. Step 3: first pass of implementation detail removal --------------------------------------------------- Modify the ``python`` API: * Add a new ``API`` subdirectory in the Python source code which will "implement" the Python C API * Replace macros with functions. The implementation of new functions will be written in the ``API/`` directory. For example, Py_INCREF() becomes the function ``void Py_INCREF(PyObject *op)`` and its implementation will be written in the ``API`` directory. * Slowly remove more and more implementation details from this API. Modifications of these API should be driven by tests of popular third party packages like: * Django with database drivers * numpy * scipy * Pillow * lxml * etc. Compilation errors on these extensions are expected. This step should help to draw a line for the backward incompatible change. Goal: remove a few implementation details but don't break numpy and lxml. Step 4 ------ Switch the default API to the new restricted ``python`` API. Help third party projects to patch their code: don't break the "Python world". Step 5 ------ Continue Step 3: remove even more implementation details. Long-term goal to complete this PEP: Remove *all* implementation details, remove all structures and macros. Alternative: keep core as the default API ========================================= A smoother transition would be to not touch the existing API but work on a new API which would only be used as an opt-in option. Similar plan used by Gilectomy: opt-in option to get best performances. It would be at least two Python binaries per Python version: default compatible version, and a new faster but incompatible version. Idea: implementation of the C API supporting old Python versions? ================================================================= Open questions. Q: Would it be possible to design an external library which would work on Python 2.7, Python 3.4-3.6, and the future Python 3.7? Q: Should such library be linked to libpythonX.Y? Or even to a pythonX.Y binary which wasn't built with shared library? Q: Would it be easy to use it? How would it be downloaded and installed to build extensions? Collaboration with PyPy, IronPython, Jython and MicroPython =========================================================== XXX to be done Enhancements becoming possible thanks to a new C API ==================================================== Indirect Reference Counting --------------------------- * Replace ``Py_ssize_t ob_refcnt;`` (integer) with ``Py_ssize_t *ob_refcnt;`` (pointer to an integer). * Same change for GC headers? * Store all reference counters in a separated memory block (or maybe multiple memory blocks) Expected advantage: smaller memory footprint when using fork() on UNIX which is implemented with Copy-On-Write on physical memory pages. See also `Dismissing Python Garbage Collection at Instagram <https://engineering.instagram.com/dismissing-python-garbage-collection-at-i…>`_. Remove Reference Counting, New Garbage Collector ------------------------------------------------ If the new C API hides well all implementation details, it becomes possible to change fundamental features like how CPython tracks the lifetime of an object. * Remove ``Py_ssize_t ob_refcnt;`` from the PyObject structure * Replace the current XXX garbage collector with a new tracing garbage collector * Use new macros to define a variable storing an object and to set the value of an object * Reimplement Py_INCREF() and Py_DECREF() on top of that using a hash table: object => reference counter. XXX PyPy is only partially successful on that project, cpyext remains very slow. XXX Would it require an opt-in option to really limit backward compatibility? Remove the GIL -------------- * Don't remove the GIL, but replace the GIL with smaller locks * Builtin mutable types: list, set, dict * Modules * Classes * etc. Backward compatibility: * Keep the GIL Tagged pointers --------------- https://en.wikipedia.org/wiki/Tagged_pointer Common optimization, especially used for "small integers". Current C API doesn't allow to implement tagged pointers. Tagged pointers are used in MicroPython to reduce the memory footprint. Note: ARM64 was recently extended its address space to 48 bits, causing issue in LuaJIT: `47 bit address space restriction on ARM64 <https://github.com/LuaJIT/LuaJIT/issues/49>`_. Misc ideas ---------- * Software Transactional Memory? See `PyPy STM <http://doc.pypy.org/en/latest/stm.html>`_ Idea: Multiple Python binaries ============================== Instead of a single ``python3.7``, providing two or more binaries, as PyPy does, would allow to experiment more easily changes without breaking the backward compatibility. For example, ``python3.7`` would remain the default binary with reference counting and the current garbage collector, whereas ``fastpython3.7`` would not use reference counting and a new garbage collector. It would allow to more quickly "break the backward compatibility" and make it even more explicit than only prepared C extensions will be compatible with the new ``fastpython3.7``. cffi ==== XXX Long term goal: "more cffi, less libpython". Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: [View Less]

9 20

JavaScript-Style Object Creation in Python (using a constructor function instead of a class to create objects)
by Simon Ramstedt July 12, 2017

July 12, 2017

Hi, do you have an opinion on the following? Wouldn't it be nice to define classes via a simple constructor function (as below) instead of a conventional class definition? *conventional*: class MyClass(ParentClass): def __init__(x): self._x = x def my_method(y): z = self._x + y return z *proposed*: def MyClass(x): self = ParentClass() def my_method(y): z = x + y return z self.my_method = my_method # that's … [View More]cumbersome (see comments below) return self Here are the pros and cons I could come up with for the proposed method: (+) Simpler and more explicit. (+) No need to create attributes (like `self._x`) just to pass something from `__init__` to another method. (+) Default arguments / annotations for methods could be different for each class instance. Adaptive defaults wouldn't have to simulated with a None. (+) Class/instance level imports would work. (-/+) Speed: The `def`-based objects take 0.6 μs to create while the `class`-based objects take only 0.4 μs. For method execution however the closure takes only 0.15 μs while the proper method takes 0.22 μs (script <https://gist.github.com/rmst/78b2b0f56a3d9ec13b1ec6f3bd50aa9c>). (-/+) Checking types: In the proposed example above the returned object wouldn't know that it has been created by `MyClass`. There are a couple of solutions to that, though. The easiest to implement would be to change the first line to `self = subclass(ParentClass())` where the subclass function looks at the next item in the call stack (i.e. `MyClass`) and makes it the type of the object. Another solution would be to have a special rule for functions with capital first letter returning a single object to append itself to the list of types of the returned object. Alternatively there could be a special keyword e.g. `classdef` that would be used instead of `def` if we wouldn't want to rely on the name. (-) The current syntax for adding a function to an object is cumbersome. That's what is preventing me from actually using the proposed pattern. But is this really the only reason for not using it? And if so, wouldn't that be a good argument for enabling something like below? *attribute function definitions*: def MyClass(x): self = ParentClass() def self.my_method(y): z = x + y return z return self or alternatively *multiline lambdas*: def MyClass(x): self = ParentClass() self.my_method = (y): z = x + y return z return self Cheers, Simon [View Less]

25 80

Improving Catching Exceptions
by Sven R. Kunze July 8, 2017

July 8, 2017

Hi folks, just one note I'd like to dump here. We usually teach our newbies to catch exceptions as narrowly as possible, i.e. MyModel.DoesNotExist instead of a plain Exception. This works out quite well for now but the number of examples continue to grow where it's not enough. There are at least three examples I can name off the top of my head: 1) nested StopIteration - PEP 479 2) nested ImportError 3) nested AttributeError 1) is clear. 2) usually can be dealt with by applying the … [View More]

13 41

Consider allowing the use of abstractmethod without metaclasses
by Neil Girdhar July 7, 2017

July 7, 2017

I want to use abstractmethod, but I have my own metaclasses and I don't want to build composite metaclasses using abc.ABCMeta. Thanks to PEP 487, one approach is to facator out the abstractmethod checks from ABCMeta into a regular (non-meta) class. So, my first suggestion is to split abc.ABC into two pieces, a parent regular class with metaclass "type": class AbstractBaseClass: def __init_subclass__(cls): # Compute set of abstract method names abstracts = {name … [View More]

1 0

Re: [Python-ideas] Arguments to exceptions
by Jeff Walker July 6, 2017

July 6, 2017

Paul, I think you are fixating too much on Ken's example. I think I understand what he is saying and I agree with him. It is a problem I struggle with routinely. It occurs in the following situations: 1. You are handling an exception that you are not raising. This could be because Python itself is raising the exception, as in Ken's example, or it could be raised by some package you did not write. 2. You need to process or transform the message in some way. Consider this example: … [View More] import json >>> s = '{"abc": 0, "cdf: 1}' >>> try: ... d = json.loads(s) ... except Exception as e: ... print(e) ... print(e.args) Unterminated string starting at: line 1 column 12 (char 11) ('Unterminated string starting at: line 1 column 12 (char 11)',) Okay, I have caught an exception for which I have no control over how the exception was raised. Now, imagine that I am writing an application that highlights json errors in place. To do so, I would need the line and column numbers to highlight the location of the error, and ideally I'd like to strip them from the base message and just show that. You can see from my second print statement that the line and column numbers were not passed as separate arguments. Thus I need to parse the error message to extract them. Not a difficult job, but fragile. Any change to the error message could break my code. I don't know what this code smell is that people keep referring to, but to me, that code would smell. Jeff > On 3 July 2017 at 09:59, Ken Kundert <python-ideas(a)shalmirane.com> wrote: > > I think in trying to illustrate the existing behavior I made things more > > confusing than they needed to be. Let me try again. > > > > Consider this code. > > > > >>> import Food > > >>> try: > > ... import meals > > ... except NameError as e: > > ... name = str(e).split("'")[1] # <-- fragile code > > ... from difflib import get_close_matches > > ... candidates = ', '.join(get_close_matches(name, Food.foods, 1, 0.6)) > > ... print(f'{name}: not found. Did you mean {candidates}?') > > > > In this case *meals* instantiates a collection of foods. It is a Python file, > > but it is also a data file (in this case the user knows Python, so Python is > > a convenient data format). In that file thousands of foods may be instantiated. > > If the user misspells a food, I would like to present the available > > alternatives. To do so, I need the misspelled name. The only way I can get it > > is by parsing the error message. > > As Steven pointed out, this is a pretty good example of a code smell. > My feeling is that you may have just proved that Python isn't quite as > good a fit for your data file format as you thought - or that your > design has flaws. Suppose your user had a breakfast menu, and did > something like: > > if now < lunchtim: # Should have been "lunchtime" > > Your error handling will be fairly confusing in that case. > > > That is the problem. To write the error handler, I need the misspelled name. > > The only way to get it is to extract it from the error message. The need to > > unpack information that was just packed suggests that the packing was done too > > early. That is my point. > > I don't have any problem with *having* the misspelled name as an > attribute to the error, I just don't think it's going to be as useful > as you hope, and it may indeed (as above) encourage people to use it > without thinking about whether there might be problems with using > error handling that way. > > > Fundamentally, pulling the name out of an error message is a really bad coding > > practice because it is fragile. The code will likely break if the formatting or > > the wording of the message changes. But given the way the exception was > > implemented, I am forced to choose between two unpleasant choices: pulling the > > name from the error message or not giving the enhanced message at all. > > Or using a different approach. ("Among our different approaches...!" > :-)) Agreed that's also an unpleasant choice at this point. > > > What I am hoping to do with this proposal is to get the Python developer > > community to see that: > > 1. The code that handles the exception benefits from having access to the > > components of the error message. In the least it can present the message to > > the user is the best possible way. Perhaps that means enforcing a particular > > style, or presenting it in the user's native language, or perhaps it means > > providing additional related information as in the example above. > > I see it as a minor bug magnet, but not really a problem in principle. > > > 2. The current approach to exceptions follows the opposite philosophy, > > suggesting that the best place to construct the error message is at the > > source of the error. It inadvertently puts obstacles in place that make it > > difficult to customize the message in the handler. > > It's more about implicitly enforcing the policy of "catch errors over > as small a section of code as practical". In your example, you're > trapping NameError from anywhere in a "many thousands" of line file. > That's about as far from the typical use of one or two lines in a try > block as you can get. > > > 3. Changing the approach in the BaseException class to provide the best of both > > approaches provides considerable value and is both trivial and backward > > compatible. > > A small amount of value in a case we don't particularly want to encourage. > Whether it's trivial comes down to implementation - I'll leave that to > whoever writes the PR to demonstrate. (Although if it *is* trivial, is > it something you could write a PR for?) > > Also, given that this would be Python 3.7 only, would people needing > this functionality (only you have expressed a need so far) be OK with > either insisting their users go straight to Python 3.7, or including > backward compatible code for older versions? > > Overall, I'm -0 on this request (assuming it is trivial to implement - > I certainly don't feel it's worth significant implementation effort). > > Paul [View Less]

8 11

Re: [Python-ideas] [off topic] Allow function to return multiple values
by Case Van Horsen July 6, 2017

July 6, 2017

On Wed, Jul 5, 2017 at 11:34 AM, Chris Barker <chris.barker(a)noaa.gov> wrote: > On Sun, Jun 25, 2017 at 6:20 PM, Mikhail V <mikhailwas(a)gmail.com> wrote: >> >> And it reminded me times starting with Python and wondering >> why I can't simply write something like: >> >> def move(x,y): >> x = x + 10 >> y = y + 20 >> move(x,y) >> >> Instead of this: >> >> def move(x,y): >> x1 = x + 10 >&… [View More]

2 1

socket module: plain stuples vs named tuples
by Thomas Güttler July 5, 2017

July 5, 2017

AFAIK the socket module returns plain tuples in Python3: https://docs.python.org/3/library/socket.html Why not use named tuples? Regards, Thomas Güttler -- I am looking for feedback for my personal programming guidelines: https://github.com/guettli/programming-guidelines

11 17

Runtime types vs static types
by Koos Zevenhoven July 5, 2017

July 5, 2017

There has been some discussion here and there concerning the differences between runtime types and static types (mypy etc.). What I write below is not really an idea or proposal---just a perspective, or a topic that people may want to discuss. Since the discussion on this is currently very fuzzy and scattered and not really happening either AFAICT (I've probably missed many discussions, though). Anyway, I thought I'd give it a shot: Clearly, there needs to be some sort of distinction between … [View More]runtime classes/types and static types, because static types can be more precise than Python's dynamic runtime semantics. For example, Iterable[int] is an iterable that contains integers. For a static type checker, it is clear what this means. But at runtime, it may be impossible to figure out whether an iterable is really of this type without consuming the whole iterable and checking whether each yielded element is an integer. Even that is not possible if the iterable is infinite. Even Sequence[int] is problematic, because checking the types of all elements of the sequence could take a long time. Since things like isinstance(it, Iterable[int]) cannot guarantee a proper answer, one easily arrives at the conclusion that static types and runtime classes are just two separate things and that one cannot require that all types support something like isinstance at runtime. On the other hand, there are many runtime things that can or could be done using (type) annotations, for example: Multidispatch (example with hypothetical syntax below): @overload def concatenate(parts: Iterable[str]) -> str: return "".join(parts) @overload def concatenate(parts: Iterable[bytes]) -> bytes: return b"".join(parts) @overload def concatenate(parts: Iterable[Iterable]) -> Iterable: return itertools.chain(*parts) or runtime type checking: @check_types def load_from_file(filename: Union[os.PathLike, str, bytes]): with open(filename) as f: return do_stuff_with(f.read()) which would automatically give a nice error message if, say, a file object is given as argument instead of a path to a file. However useful (and efficient) these things might be, the runtime type checks are problematic, as discussed above. Furthermore, other differences between runtime and static typing may emerge (or have emerged), which will complicate the matter further. For instance, the runtime __annotations__ of classes, modules and functions may in some cases contain something completely different from what a type checker thinks the type should be. These and other incompatibilities between runtime and static typing will create two (or more) different kinds of type-annotated Python: runtime-oriented Python and Python with static type checking. These may be incompatible in both directions: a static type checker may complain about code that is perfectly valid for the runtime folks, and code written for static type checking may not be able to use new Python techniques that make use of type hints at runtime. There may not even be a fully functional subset of the two "languages". Different libraries will adhere to different standards and will not be compatible with each other. The split will be much worse and more difficult to understand than Python 2 vs 3, peoples around the world will suffer like never before, and programming in Python will become a very complicated mess. One way of solving the problem would be that type annotations are only a static concept, like with stubs or comment-based type annotations. This would also be nice from a memory and performance perspective, as evaluating and storing the annotations would not occupy memory (although both issues and some more might be nicely solved by making the annotations lazily ealuated). However, leaving out runtime effects of type annotations is not the approach taken, and runtime introspection of annotations seems to have some promising applications as well. And for many cases, the traditional Python class actually acts very nicely as both the runtime and static type. So if type annotations will be both for runtime and for static checking, how to make everything work for both static and runtime typing? Since a writer of a library does not know what the type hints will be used for by the library users, it is very important that there is only one way of making type annotations which will work regardless of what the annotations are used for in the end. This will also make it much easier to learn Python typing. Regarding runtime types and isinstance, let's look at the Iterable[int] example. For this case, there are a few options: 1) Don't implement isinstance This is problematic for runtime uses of annotations. 2) isinstance([1, '2', 'three'], Iterable[int]) returns True This is in fact now the case. This is ok for many runtime situations, but lacks precision compared to the static version. One may want to distinguish between Iterable[int] and Iterable[str] at runtime (e.g. the multidispatch example above). 3) Check as much as you can at runtime There could be something like Reiterable, which means the object is not consumed by iterating over it, so one could actually check if all elements are instances of int. This would be useful in some situations, but not available for every object. Furthermore, the check could take an arbitrary amount of time so it is not really suitable for things like multidispatch or some matching constructs etc., where the performance overhead of the type check is really important. 4) Do a deeper check than in (2) but trust the annotations For example, an instance of a class that has a method like def __iter__(self) -> Iterator[int]: some code could be identified as Iterable[int] at runtime, even if it is not guaranteed that all elements are really integers. On the other hand, an object returned by def get_ints() -> Iterable[int]: some code does not know its own annotations, so the check is difficult to do at runtime. And of course, there may not be annotations available. 5) Something else? And what about PEP544 (protocols), which is being drafted? The PEP seems to aim for having type objects that represent duck-typing protocols/interfaces. Checking whether a protocol is implemented by an object or type is clearly a useful thing to do at runtime, but it is not really clear if isinstance would be a guaranteed feature for PEP544 Protocols. So one question is, is it possible to draw the lines between what works with isinstance and what doesn't, and between what details are checked by isinstance and what aren't? -- Or should insinstance be reserved for a more limited purpose, and add another check function, say `implements(...)`, which would perhaps guarantee some answer for all combinations of object and type? I'll stop here---this email is probably already much longer than a single email should be ;) -- Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven + [View Less]

6 9

Re: [Python-ideas] Allow function to return multiple values
by rymg19＠gmail.com July 5, 2017

July 5, 2017

IIRC I'm pretty sure the OP just didn't know about the existence of tuple unpacking and the ability to use that to return multiple values. -- Ryan (ライアン) Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone elsehttp://refi64.com On Jun 25, 2017 at 6:09 PM, <Mikhail V <mikhailwas(a)gmail.com>> wrote: joannah nanjekye wrote: > [...] > >Today I was writing an example snippet for the book and needed to write a >function that returns two values … [View More]

3 2

+ operator on generators
by lucas July 5, 2017

July 5, 2017

Hello ! I often use generators, and itertools.chain on them. What about providing something like the following: a = (n for n in range(2)) b = (n for n in range(2, 4)) tuple(a + b) # -> 0 1 2 3 This, from user point of view, is just as how the __add__ operator works on lists and tuples. Making generators works the same way could be a great way to avoid calls to itertools.chain everywhere, and to limits the differences between generators and other "linear" collections. I do … [View More]

21 63

Bytecode JIT
by Soni L. July 3, 2017

July 3, 2017

CPython should get a tracing JIT that turns slow bytecode into fast bytecode. A JIT doesn't have to produce machine code. bytecode-to-bytecode compilation is still compilation. bytecode-to-bytecode compilation works on iOS, and doesn't require deviating from C. (This "internal bytecode" should do things like know that 2 variables necessarily hold integers, doing just "x = y + z" in C in an IADD instruction as opposed to all those middle-of-function typechecks and overhead. You can … [View More]

11 20

Should iscoroutine and iscoroutinefunction be in builtins?
by Alex Walters July 2, 2017

July 2, 2017

Before async/await it made sense that iscoroutine and iscoroutinefunction live in asyncio. But now that coroutines are a built in type, supported by its own syntax, wouldn't it make sense to make those functions builtins?

3 2

Re: [Python-ideas] Runtime types vs static types
by rymg19＠gmail.com July 2, 2017

July 2, 2017

For some background on the removal of __instancecheck__, check the linked issues here: https://github.com/python/typing/issues/135 -- Ryan (ライアン) Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone elsehttp://refi64.com On Jun 25, 2017 at 8:11 AM, <Koos Zevenhoven <k7hoven(a)gmail.com>> wrote: On Sat, Jun 24, 2017 at 11:30 PM, Lucas Wiman <lucas.wiman(a)gmail.com> wrote: > > On Sat, Jun 24, 2017 at 12:42 PM, Koos Zevenhoven <k7hoven(a)… [View More]gmail.com> > wrote: > >> There has been some discussion here and there concerning the differences >> between runtime types and static types (mypy etc.). What I write below is >> not really an idea or proposal---just a perspective, or a topic that people >> may want to discuss. Since the discussion on this is currently very fuzzy >> and scattered and not really happening either AFAICT (I've probably missed >> many discussions, though). Anyway, I thought I'd give it a shot: >> >> [...] > Regarding runtime types and isinstance, let's look at the Iterable[int] >> example. For this case, there are a few options: >> >> 1) Don't implement isinstance >> >> This is problematic for runtime uses of annotations. >> >> 2) isinstance([1, '2', 'three'], Iterable[int]) returns True >> >> This is in fact now the case. This is ok for many runtime situations, but >> lacks precision compared to the static version. One may want to distinguish >> between Iterable[int] and Iterable[str] at runtime (e.g. the multidispatch >> example above). >> >> 3) Check as much as you can at runtime >> >> There could be something like Reiterable, which means the object is not >> consumed by iterating over it, so one could actually check if all elements >> are instances of int. This would be useful in some situations, but not >> available for every object. Furthermore, the check could take an arbitrary >> amount of time so it is not really suitable for things like multidispatch >> or some matching constructs etc., where the performance overhead of the >> type check is really important. >> >> 4) Do a deeper check than in (2) but trust the annotations >> >> For example, an instance of a class that has a method like >> >> def __iter__(self) -> Iterator[int]: >> some code >> >> could be identified as Iterable[int] at runtime, even if it is not >> guaranteed that all elements are really integers. >> >> On the other hand, an object returned by >> >> def get_ints() -> Iterable[int]: >> some code >> >> does not know its own annotations, so the check is difficult to do at >> runtime. And of course, there may not be annotations available. >> >> 5) Something else? >> >> >> And what about PEP544 (protocols), which is being drafted? The PEP seems >> to aim for having type objects that represent duck-typing >> protocols/interfaces. Checking whether a protocol is implemented by an >> object or type is clearly a useful thing to do at runtime, but it is not >> really clear if isinstance would be a guaranteed feature for PEP544 >> Protocols. >> >> So one question is, is it possible to draw the lines between what works >> with isinstance and what doesn't, and between what details are checked by >> isinstance and what aren't? -- Or should insinstance be reserved for a more >> limited purpose, and add another check function, say `implements(...)`, >> which would perhaps guarantee some answer for all combinations of object >> and type? >> > >> > I'm guessing to implement PEP 544, many of the `__instancecheck__` and > `__subclasscheck__` methods in `typing.py` would need to be updated to > check the `__annotations__` of the class of the object it's passed against > its own definition, (covered in this section > <https://www.python.org/dev/peps/pep-0544/#runtime-decorator-and-narrowing-t…> > of the PEP). > > I may have missed something, but I believe PEP544 is not suggesting that annotations would have any effect on isinstance. Instead, isinstance would by default not work. > I've been somewhat surprised that many of the `__instancecheck__` > implementations do not work at runtime, even when the implementation would > be trivial (e.g. for `Union`), or would not have subtle edge cases due to > immutability (e.g. for `Tuple`, which cannot be used for checking > parameterized instances). This seems like counterintuitive behavior that > would be straightforward to fix, unless there are subtleties & edge cases > I'm missing. > > Tuple is an interesting case, because for small tuples (say 2- or 3-tuples), it makes perfect sense to check the types of all elements for some runtime purposes. Regarding Union, I believe the current situation has a lot to do with the fact that the relation between type annotations and runtime behavior hasn't really settled yet. If people are amenable to updating those cases, I'd be interested in > submitting a patch to that effect. > > Thanks for letting us know. (There may not be an instant decision on this particular case, though, but who knows :) -- Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven + _______________________________________________ Python-ideas mailing list Python-ideas(a)python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ [View Less]

6 11

Re: [Python-ideas] Bytecode JIT
by rymg19＠gmail.com July 1, 2017

July 1, 2017

This is literally PyPy. There's little reason for something like this to end up in official CPython, at least for now. -- Ryan (ライアン) Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone elsehttp://refi64.com On Jul 1, 2017 at 5:53 PM, <Soni L. <fakedme+py(a)gmail.com>> wrote: On 2017-07-01 07:34 PM, Victor Stinner wrote: > Let's say that you have a function "def mysum (x; y): return x+y", do > you always want to use your new IADD instruction here? … [View More]

2 1