Mailman 3 July 2017 - Python-ideas

Re: [Python-ideas] [off topic] Allow function to return multiple values
by Case Van Horsen July 6, 2017

July 6, 2017

On Wed, Jul 5, 2017 at 11:34 AM, Chris Barker <chris.barker(a)noaa.gov> wrote: > On Sun, Jun 25, 2017 at 6:20 PM, Mikhail V <mikhailwas(a)gmail.com> wrote: >> >> And it reminded me times starting with Python and wondering >> why I can't simply write something like: >> >> def move(x,y): >> x = x + 10 >> y = y + 20 >> move(x,y) >> >> Instead of this: >> >> def move(x,y): >> x1 = x + 10 >> y1 = y + 20 >> return x1,y1 >> x,y = move(x,y) > > > you CAN do that, if x and y are mutable types. > > I've found that when folk want this behavior (often called "pass by > reference" or something), what they really want in a mutable number. And you > can make one of those if you like -- here's a minimal one that can be used > as a counter: [veering off-topic] I've implemented mutable integers as part of the gmpy2 library. The eXperimental MPZ (xmpz) type breaks many of the normal rules. Mutable >>> a=gmpy2.xmpz(1) >>> b=a >>> a+=1 >>> a xmpz(2) >>> b xmpz(2) Direct access to individual bits >>> a=gmpy2.xmpz(123) >>> a[0] 1 >>> a[1] 1 >>> a[0]=0 >>> a xmpz(122) Iterating over bits >>> a=gmpy2.xmpz(104) >>> bin(a) '0b1101000' >>> list(a.iter_bits()) [False, False, False, True, False, True, True] >>> list(a.iter_clear()) [0, 1, 2, 4] >>> list(a.iter_set()) [3, 5, 6] I'm not sure how useful it really is but it was fun to write. :) casevh

2 1

socket module: plain stuples vs named tuples
by Thomas Güttler July 5, 2017

July 5, 2017

AFAIK the socket module returns plain tuples in Python3: https://docs.python.org/3/library/socket.html Why not use named tuples? Regards, Thomas Güttler -- I am looking for feedback for my personal programming guidelines: https://github.com/guettli/programming-guidelines

11 17

Runtime types vs static types
by Koos Zevenhoven July 5, 2017

July 5, 2017

There has been some discussion here and there concerning the differences between runtime types and static types (mypy etc.). What I write below is not really an idea or proposal---just a perspective, or a topic that people may want to discuss. Since the discussion on this is currently very fuzzy and scattered and not really happening either AFAICT (I've probably missed many discussions, though). Anyway, I thought I'd give it a shot: Clearly, there needs to be some sort of distinction between runtime classes/types and static types, because static types can be more precise than Python's dynamic runtime semantics. For example, Iterable[int] is an iterable that contains integers. For a static type checker, it is clear what this means. But at runtime, it may be impossible to figure out whether an iterable is really of this type without consuming the whole iterable and checking whether each yielded element is an integer. Even that is not possible if the iterable is infinite. Even Sequence[int] is problematic, because checking the types of all elements of the sequence could take a long time. Since things like isinstance(it, Iterable[int]) cannot guarantee a proper answer, one easily arrives at the conclusion that static types and runtime classes are just two separate things and that one cannot require that all types support something like isinstance at runtime. On the other hand, there are many runtime things that can or could be done using (type) annotations, for example: Multidispatch (example with hypothetical syntax below): @overload def concatenate(parts: Iterable[str]) -> str: return "".join(parts) @overload def concatenate(parts: Iterable[bytes]) -> bytes: return b"".join(parts) @overload def concatenate(parts: Iterable[Iterable]) -> Iterable: return itertools.chain(*parts) or runtime type checking: @check_types def load_from_file(filename: Union[os.PathLike, str, bytes]): with open(filename) as f: return do_stuff_with(f.read()) which would automatically give a nice error message if, say, a file object is given as argument instead of a path to a file. However useful (and efficient) these things might be, the runtime type checks are problematic, as discussed above. Furthermore, other differences between runtime and static typing may emerge (or have emerged), which will complicate the matter further. For instance, the runtime __annotations__ of classes, modules and functions may in some cases contain something completely different from what a type checker thinks the type should be. These and other incompatibilities between runtime and static typing will create two (or more) different kinds of type-annotated Python: runtime-oriented Python and Python with static type checking. These may be incompatible in both directions: a static type checker may complain about code that is perfectly valid for the runtime folks, and code written for static type checking may not be able to use new Python techniques that make use of type hints at runtime. There may not even be a fully functional subset of the two "languages". Different libraries will adhere to different standards and will not be compatible with each other. The split will be much worse and more difficult to understand than Python 2 vs 3, peoples around the world will suffer like never before, and programming in Python will become a very complicated mess. One way of solving the problem would be that type annotations are only a static concept, like with stubs or comment-based type annotations. This would also be nice from a memory and performance perspective, as evaluating and storing the annotations would not occupy memory (although both issues and some more might be nicely solved by making the annotations lazily ealuated). However, leaving out runtime effects of type annotations is not the approach taken, and runtime introspection of annotations seems to have some promising applications as well. And for many cases, the traditional Python class actually acts very nicely as both the runtime and static type. So if type annotations will be both for runtime and for static checking, how to make everything work for both static and runtime typing? Since a writer of a library does not know what the type hints will be used for by the library users, it is very important that there is only one way of making type annotations which will work regardless of what the annotations are used for in the end. This will also make it much easier to learn Python typing. Regarding runtime types and isinstance, let's look at the Iterable[int] example. For this case, there are a few options: 1) Don't implement isinstance This is problematic for runtime uses of annotations. 2) isinstance([1, '2', 'three'], Iterable[int]) returns True This is in fact now the case. This is ok for many runtime situations, but lacks precision compared to the static version. One may want to distinguish between Iterable[int] and Iterable[str] at runtime (e.g. the multidispatch example above). 3) Check as much as you can at runtime There could be something like Reiterable, which means the object is not consumed by iterating over it, so one could actually check if all elements are instances of int. This would be useful in some situations, but not available for every object. Furthermore, the check could take an arbitrary amount of time so it is not really suitable for things like multidispatch or some matching constructs etc., where the performance overhead of the type check is really important. 4) Do a deeper check than in (2) but trust the annotations For example, an instance of a class that has a method like def __iter__(self) -> Iterator[int]: some code could be identified as Iterable[int] at runtime, even if it is not guaranteed that all elements are really integers. On the other hand, an object returned by def get_ints() -> Iterable[int]: some code does not know its own annotations, so the check is difficult to do at runtime. And of course, there may not be annotations available. 5) Something else? And what about PEP544 (protocols), which is being drafted? The PEP seems to aim for having type objects that represent duck-typing protocols/interfaces. Checking whether a protocol is implemented by an object or type is clearly a useful thing to do at runtime, but it is not really clear if isinstance would be a guaranteed feature for PEP544 Protocols. So one question is, is it possible to draw the lines between what works with isinstance and what doesn't, and between what details are checked by isinstance and what aren't? -- Or should insinstance be reserved for a more limited purpose, and add another check function, say `implements(...)`, which would perhaps guarantee some answer for all combinations of object and type? I'll stop here---this email is probably already much longer than a single email should be ;) -- Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven +

6 9

Re: [Python-ideas] Allow function to return multiple values
by rymg19＠gmail.com July 5, 2017

July 5, 2017

IIRC I'm pretty sure the OP just didn't know about the existence of tuple unpacking and the ability to use that to return multiple values. -- Ryan (ライアン) Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone elsehttp://refi64.com On Jun 25, 2017 at 6:09 PM, <Mikhail V <mikhailwas(a)gmail.com>> wrote: joannah nanjekye wrote: > [...] > >Today I was writing an example snippet for the book and needed to write a >function that returns two values something like this: > >def return_multiplevalues(num1, num2): > return num1, num2 > > I noticed that this actually returns a tuple of the values which I did not >want in the first place.I wanted python to return two values in their own >types so I can work with them as they are but here I was stuck with working >around a tuple. It was quite puzzling at first what was the actual idea but probably I can guess why this question came up by you. It seems to me (I am just intuitively guessing that) that you were about to write a procedure which operates on global variables. If so you should use the keyword "global" for that. E.g. if you want to work with the variables defined in other part of the code you can simply do it: x = 0 y = 0 def move(): global x, y x = x + 10 y = y + 20 move() print (x,y) This function will change x and y (global variables in this case). Note that without the line with the "global" statement this will not work. Another typical usage is initialising variables inside a procedure: def init_coordinates(): global x,y x=0 y=0 init_coordinates() print (x,y) So for some reason it seemed to me that you are trying to do something like that. >My proposal is we provide a way of functions returning multiple values. >This has been implemented in languages like Go and I have found many cases >where I needed and used such a functionality. I wish for this convenience >in python so that I don't have to suffer going around a tuple. So if using globals as in the above examples you certinly don't have to suffer going around a tuple. Mikhail _______________________________________________ Python-ideas mailing list Python-ideas(a)python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

3 2

+ operator on generators
by lucas July 5, 2017

July 5, 2017

Hello ! I often use generators, and itertools.chain on them. What about providing something like the following: a = (n for n in range(2)) b = (n for n in range(2, 4)) tuple(a + b) # -> 0 1 2 3 This, from user point of view, is just as how the __add__ operator works on lists and tuples. Making generators works the same way could be a great way to avoid calls to itertools.chain everywhere, and to limits the differences between generators and other "linear" collections. I do not know exactly how to implement that (i'm not that good at C, nor CPython source itself), but by seeing the sources, i imagine that i could do something like the list_concat function at Objects/listobject.c:473, but in the Objects/genobject.c file, where instead of copying elements i'm creating and initializing a new chainobject as described at Modules/itertoolsmodule.c:1792. (In pure python, the implementation would be something like `def __add__(self, othr): return itertools.chain(self, othr)`) Best regards, --lucas

21 63

Bytecode JIT
by Soni L. July 3, 2017

July 3, 2017

CPython should get a tracing JIT that turns slow bytecode into fast bytecode. A JIT doesn't have to produce machine code. bytecode-to-bytecode compilation is still compilation. bytecode-to-bytecode compilation works on iOS, and doesn't require deviating from C. (This "internal bytecode" should do things like know that 2 variables necessarily hold integers, doing just "x = y + z" in C in an IADD instruction as opposed to all those middle-of-function typechecks and overhead. You can typecheck once at the start of the function and run separate traces on that. Since this "internal bytecode" is extremely unsafe, it should be considered an implementation detail and never exposed to external code.)

11 20

Should iscoroutine and iscoroutinefunction be in builtins?
by Alex Walters July 3, 2017

July 3, 2017

Before async/await it made sense that iscoroutine and iscoroutinefunction live in asyncio. But now that coroutines are a built in type, supported by its own syntax, wouldn't it make sense to make those functions builtins?

3 2

Re: [Python-ideas] Runtime types vs static types
by rymg19＠gmail.com July 2, 2017

July 2, 2017

For some background on the removal of __instancecheck__, check the linked issues here: https://github.com/python/typing/issues/135 -- Ryan (ライアン) Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone elsehttp://refi64.com On Jun 25, 2017 at 8:11 AM, <Koos Zevenhoven <k7hoven(a)gmail.com>> wrote: On Sat, Jun 24, 2017 at 11:30 PM, Lucas Wiman <lucas.wiman(a)gmail.com> wrote: > > On Sat, Jun 24, 2017 at 12:42 PM, Koos Zevenhoven <k7hoven(a)gmail.com> > wrote: > >> There has been some discussion here and there concerning the differences >> between runtime types and static types (mypy etc.). What I write below is >> not really an idea or proposal---just a perspective, or a topic that people >> may want to discuss. Since the discussion on this is currently very fuzzy >> and scattered and not really happening either AFAICT (I've probably missed >> many discussions, though). Anyway, I thought I'd give it a shot: >> >> [...] > Regarding runtime types and isinstance, let's look at the Iterable[int] >> example. For this case, there are a few options: >> >> 1) Don't implement isinstance >> >> This is problematic for runtime uses of annotations. >> >> 2) isinstance([1, '2', 'three'], Iterable[int]) returns True >> >> This is in fact now the case. This is ok for many runtime situations, but >> lacks precision compared to the static version. One may want to distinguish >> between Iterable[int] and Iterable[str] at runtime (e.g. the multidispatch >> example above). >> >> 3) Check as much as you can at runtime >> >> There could be something like Reiterable, which means the object is not >> consumed by iterating over it, so one could actually check if all elements >> are instances of int. This would be useful in some situations, but not >> available for every object. Furthermore, the check could take an arbitrary >> amount of time so it is not really suitable for things like multidispatch >> or some matching constructs etc., where the performance overhead of the >> type check is really important. >> >> 4) Do a deeper check than in (2) but trust the annotations >> >> For example, an instance of a class that has a method like >> >> def __iter__(self) -> Iterator[int]: >> some code >> >> could be identified as Iterable[int] at runtime, even if it is not >> guaranteed that all elements are really integers. >> >> On the other hand, an object returned by >> >> def get_ints() -> Iterable[int]: >> some code >> >> does not know its own annotations, so the check is difficult to do at >> runtime. And of course, there may not be annotations available. >> >> 5) Something else? >> >> >> And what about PEP544 (protocols), which is being drafted? The PEP seems >> to aim for having type objects that represent duck-typing >> protocols/interfaces. Checking whether a protocol is implemented by an >> object or type is clearly a useful thing to do at runtime, but it is not >> really clear if isinstance would be a guaranteed feature for PEP544 >> Protocols. >> >> So one question is, is it possible to draw the lines between what works >> with isinstance and what doesn't, and between what details are checked by >> isinstance and what aren't? -- Or should insinstance be reserved for a more >> limited purpose, and add another check function, say `implements(...)`, >> which would perhaps guarantee some answer for all combinations of object >> and type? >> > >> > I'm guessing to implement PEP 544, many of the `__instancecheck__` and > `__subclasscheck__` methods in `typing.py` would need to be updated to > check the `__annotations__` of the class of the object it's passed against > its own definition, (covered in this section > <https://www.python.org/dev/peps/pep-0544/#runtime-decorator-and-narrowing-t…> > of the PEP). > > I may have missed something, but I believe PEP544 is not suggesting that annotations would have any effect on isinstance. Instead, isinstance would by default not work. > I've been somewhat surprised that many of the `__instancecheck__` > implementations do not work at runtime, even when the implementation would > be trivial (e.g. for `Union`), or would not have subtle edge cases due to > immutability (e.g. for `Tuple`, which cannot be used for checking > parameterized instances). This seems like counterintuitive behavior that > would be straightforward to fix, unless there are subtleties & edge cases > I'm missing. > > Tuple is an interesting case, because for small tuples (say 2- or 3-tuples), it makes perfect sense to check the types of all elements for some runtime purposes. Regarding Union, I believe the current situation has a lot to do with the fact that the relation between type annotations and runtime behavior hasn't really settled yet. If people are amenable to updating those cases, I'd be interested in > submitting a patch to that effect. > > Thanks for letting us know. (There may not be an instant decision on this particular case, though, but who knows :) -- Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven + _______________________________________________ Python-ideas mailing list Python-ideas(a)python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

6 11

Re: [Python-ideas] Bytecode JIT
by rymg19＠gmail.com July 2, 2017

July 2, 2017

This is literally PyPy. There's little reason for something like this to end up in official CPython, at least for now. -- Ryan (ライアン) Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone elsehttp://refi64.com On Jul 1, 2017 at 5:53 PM, <Soni L. <fakedme+py(a)gmail.com>> wrote: On 2017-07-01 07:34 PM, Victor Stinner wrote: > Let's say that you have a function "def mysum (x; y): return x+y", do > you always want to use your new IADD instruction here? What if I call > mysum ("a", "b")? > > Victor Let's say that you do. Given how short it is, it would just get inlined. Your call of mysum ("a", "b") would indeed not use IADD, nor would it be a call. It would potentially not invoke any operators, but instead get replaced with "ab". When you have a tracing JIT, you can do away with a lot of overhead. You can inline functions, variables, do away with typechecks, and many other things. This holds true even if that JIT never emits a single byte of machine code. _______________________________________________ Python-ideas mailing list Python-ideas(a)python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

2 1