Mailman 3 June 2017 - Python-ideas

@classproperty, @abc.abstractclasspropery, etc.
by K. Richard Pixley Dec. 16, 2020

Dec. 16, 2020

There's a whole matrix of these and I'm wondering why the matrix is currently sparse rather than implementing them all. Or rather, why we can't stack them as: class foo(object): @classmethod @property def bar(cls, ...): ... Essentially the permutation are, I think: {'unadorned'|abc.abstract}{'normal'|static|class}{method|property|non-callable attribute}. concreteness implicit first arg type name comments {unadorned} {unadorned} method def foo(): exists … [View More]now {unadorned} {unadorned} property @property exists now {unadorned} {unadorned} non-callable attribute x = 2 exists now {unadorned} static method @staticmethod exists now {unadorned} static property @staticproperty proposing {unadorned} static non-callable attribute {degenerate case - variables don't have arguments} unnecessary {unadorned} class method @classmethod exists now {unadorned} class property @classproperty or @classmethod;@property proposing {unadorned} class non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract {unadorned} method @abc.abstractmethod exists now abc.abstract {unadorned} property @abc.abstractproperty exists now abc.abstract {unadorned} non-callable attribute @abc.abstractattribute or @abc.abstract;@attribute proposing abc.abstract static method @abc.abstractstaticmethod exists now abc.abstract static property @abc.staticproperty proposing abc.abstract static non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract class method @abc.abstractclassmethod exists now abc.abstract class property @abc.abstractclassproperty proposing abc.abstract class non-callable attribute {degenerate case - variables don't have arguments} unnecessary I think the meanings of the new ones are pretty straightforward, but in case they are not... @staticproperty - like @property only without an implicit first argument. Allows the property to be called directly from the class without requiring a throw-away instance. @classproperty - like @property, only the implicit first argument to the method is the class. Allows the property to be called directly from the class without requiring a throw-away instance. @abc.abstractattribute - a simple, non-callable variable that must be overridden in subclasses @abc.abstractstaticproperty - like @abc.abstractproperty only for @staticproperty @abc.abstractclassproperty - like @abc.abstractproperty only for @classproperty --rich [View Less]

10 15

Specify number of items to allocate for array.array() constructor
by Sven Rahmann Feb. 22, 2020

Feb. 22, 2020

At the moment, the array module of the standard library allows to create arrays of different numeric types and to initialize them from an iterable (eg, another array). What's missing is the possiblity to specify the final size of the array (number of items), especially for large arrays. I'm thinking of suffix arrays (a text indexing data structure) for large texts, eg the human genome and its reverse complement (about 6 billion characters from the alphabet ACGT). The suffix array is a long int … [View More]

14 20

Asynchronous exception handling around with/try statement borders
by Erik Bray Sept. 24, 2018

Sept. 24, 2018

Hi folks, I normally wouldn't bring something like this up here, except I think that there is possibility of something to be done--a language documentation clarification if nothing else, though possibly an actual code change as well. I've been having an argument with a colleague over the last couple days over the proper way order of statements when setting up a try/finally to perform cleanup of some action. On some level we're both being stubborn I think, and I'm not looking for resolution … [View More]as to who's right/wrong or I wouldn't bring it to this list in the first place. The original argument was over setting and later restoring os.environ, but we ended up arguing over threading.Lock.acquire/release which I think is a more interesting example of the problem, and he did raise a good point that I do want to bring up. </prologue> My colleague's contention is that given lock = threading.Lock() this is simply *wrong*: lock.acquire() try: do_something() finally: lock.release() whereas this is okay: with lock: do_something() Ignoring other details of how threading.Lock is actually implemented, assuming that Lock.__enter__ calls acquire() and Lock.__exit__ calls release() then as far as I've known ever since Python 2.5 first came out these two examples are semantically *equivalent*, and I can't find any way of reading PEP 343 or the Python language reference that would suggest otherwise. However, there *is* a difference, and has to do with how signals are handled, particularly w.r.t. context managers implemented in C (hence we are talking CPython specifically): If Lock.__enter__ is a pure Python method (even if it maybe calls some C methods), and a SIGINT is handled during execution of that method, then in almost all cases a KeyboardInterrupt exception will be raised from within Lock.__enter__--this means the suite under the with: statement is never evaluated, and Lock.__exit__ is never called. You can be fairly sure the KeyboardInterrupt will be raised from somewhere within a pure Python Lock.__enter__ because there will usually be at least one remaining opcode to be evaluated, such as RETURN_VALUE. Because of how delayed execution of signal handlers is implemented in the pyeval main loop, this means the signal handler for SIGINT will be called *before* RETURN_VALUE, resulting in the KeyboardInterrupt exception being raised. Standard stuff. However, if Lock.__enter__ is a PyCFunction things are quite different. If you look at how the SETUP_WITH opcode is implemented, it first calls the __enter__ method with _PyObjet_CallNoArg. If this returns NULL (i.e. an exception occurred in __enter__) then "goto error" is executed and the exception is raised. However if it returns non-NULL the finally block is set up with PyFrame_BlockSetup and execution proceeds to the next opcode. At this point a potentially waiting SIGINT is handled, resulting in KeyboardInterrupt being raised while inside the with statement's suite, and finally block, and hence Lock.__exit__ are entered. Long story short, because Lock.__enter__ is a C function, assuming that it succeeds normally then with lock: do_something() always guarantees that Lock.__exit__ will be called if a SIGINT was handled inside Lock.__enter__, whereas with lock.acquire() try: ... finally: lock.release() there is at last a small possibility that the SIGINT handler is called after the CALL_FUNCTION op but before the try/finally block is entered (e.g. before executing POP_TOP or SETUP_FINALLY). So the end result is that the lock is held and never released after the KeyboardInterrupt (whether or not it's handled somehow). Whereas, again, if Lock.__enter__ is a pure Python function there's less likely to be any difference (though I don't think the possibility can be ruled out entirely). At the very least I think this quirk of CPython should be mentioned somewhere (since in all other cases the semantic meaning of the "with:" statement is clear). However, I think it might be possible to gain more consistency between these cases if pending signals are checked/handled after any direct call to PyCFunction from within the ceval loop. Sorry for the tl;dr; any thoughts? [View Less]

7 15

Positional-only parameters
by Victor Stinner Sept. 11, 2018

Sept. 11, 2018

Hi, For technical reasons, many functions of the Python standard libraries implemented in C have positional-only parameters. Example: ------- $ ./python Python 3.7.0a0 (default, Feb 25 2017, 04:30:32) >>> help(str.replace) replace(self, old, new, count=-1, /) # <== notice "/" at the end ... >>> "a".replace("x", "y") # ok 'a' >>> "a".replace(old="x", new="y") # ERR! TypeError: replace() takes at least 2 arguments (0 given) ------- When converting the … [View More]

19 27

Implicit string literal concatenation considered harmful?
by Guido van Rossum March 14, 2018

March 14, 2018

I just spent a few minutes staring at a bug caused by a missing comma -- I got a mysterious argument count error because instead of foo('a', 'b') I had written foo('a' 'b'). This is a fairly common mistake, and IIRC at Google we even had a lint rule against this (there was also a Python dialect used for some specific purpose where this was explicitly forbidden). Now, with modern compiler technology, we can (and in fact do) evaluate compile-time string literal concatenation with the '+' … [View More]

51 165

JavaScript-Style Object Creation in Python (using a constructor function instead of a class to create objects)
by Simon Ramstedt July 13, 2017

July 13, 2017

Hi, do you have an opinion on the following? Wouldn't it be nice to define classes via a simple constructor function (as below) instead of a conventional class definition? *conventional*: class MyClass(ParentClass): def __init__(x): self._x = x def my_method(y): z = self._x + y return z *proposed*: def MyClass(x): self = ParentClass() def my_method(y): z = x + y return z self.my_method = my_method # that's … [View More]cumbersome (see comments below) return self Here are the pros and cons I could come up with for the proposed method: (+) Simpler and more explicit. (+) No need to create attributes (like `self._x`) just to pass something from `__init__` to another method. (+) Default arguments / annotations for methods could be different for each class instance. Adaptive defaults wouldn't have to simulated with a None. (+) Class/instance level imports would work. (-/+) Speed: The `def`-based objects take 0.6 μs to create while the `class`-based objects take only 0.4 μs. For method execution however the closure takes only 0.15 μs while the proper method takes 0.22 μs (script <https://gist.github.com/rmst/78b2b0f56a3d9ec13b1ec6f3bd50aa9c>). (-/+) Checking types: In the proposed example above the returned object wouldn't know that it has been created by `MyClass`. There are a couple of solutions to that, though. The easiest to implement would be to change the first line to `self = subclass(ParentClass())` where the subclass function looks at the next item in the call stack (i.e. `MyClass`) and makes it the type of the object. Another solution would be to have a special rule for functions with capital first letter returning a single object to append itself to the list of types of the returned object. Alternatively there could be a special keyword e.g. `classdef` that would be used instead of `def` if we wouldn't want to rely on the name. (-) The current syntax for adding a function to an object is cumbersome. That's what is preventing me from actually using the proposed pattern. But is this really the only reason for not using it? And if so, wouldn't that be a good argument for enabling something like below? *attribute function definitions*: def MyClass(x): self = ParentClass() def self.my_method(y): z = x + y return z return self or alternatively *multiline lambdas*: def MyClass(x): self = ParentClass() self.my_method = (y): z = x + y return z return self Cheers, Simon [View Less]

25 80

Improving Catching Exceptions
by Sven R. Kunze July 8, 2017

July 8, 2017

Hi folks, just one note I'd like to dump here. We usually teach our newbies to catch exceptions as narrowly as possible, i.e. MyModel.DoesNotExist instead of a plain Exception. This works out quite well for now but the number of examples continue to grow where it's not enough. There are at least three examples I can name off the top of my head: 1) nested StopIteration - PEP 479 2) nested ImportError 3) nested AttributeError 1) is clear. 2) usually can be dealt with by applying the … [View More]

13 41

socket module: plain stuples vs named tuples
by Thomas Güttler July 5, 2017

July 5, 2017

AFAIK the socket module returns plain tuples in Python3: https://docs.python.org/3/library/socket.html Why not use named tuples? Regards, Thomas Güttler -- I am looking for feedback for my personal programming guidelines: https://github.com/guettli/programming-guidelines

11 17

Runtime types vs static types
by Koos Zevenhoven July 5, 2017

July 5, 2017

There has been some discussion here and there concerning the differences between runtime types and static types (mypy etc.). What I write below is not really an idea or proposal---just a perspective, or a topic that people may want to discuss. Since the discussion on this is currently very fuzzy and scattered and not really happening either AFAICT (I've probably missed many discussions, though). Anyway, I thought I'd give it a shot: Clearly, there needs to be some sort of distinction between … [View More]runtime classes/types and static types, because static types can be more precise than Python's dynamic runtime semantics. For example, Iterable[int] is an iterable that contains integers. For a static type checker, it is clear what this means. But at runtime, it may be impossible to figure out whether an iterable is really of this type without consuming the whole iterable and checking whether each yielded element is an integer. Even that is not possible if the iterable is infinite. Even Sequence[int] is problematic, because checking the types of all elements of the sequence could take a long time. Since things like isinstance(it, Iterable[int]) cannot guarantee a proper answer, one easily arrives at the conclusion that static types and runtime classes are just two separate things and that one cannot require that all types support something like isinstance at runtime. On the other hand, there are many runtime things that can or could be done using (type) annotations, for example: Multidispatch (example with hypothetical syntax below): @overload def concatenate(parts: Iterable[str]) -> str: return "".join(parts) @overload def concatenate(parts: Iterable[bytes]) -> bytes: return b"".join(parts) @overload def concatenate(parts: Iterable[Iterable]) -> Iterable: return itertools.chain(*parts) or runtime type checking: @check_types def load_from_file(filename: Union[os.PathLike, str, bytes]): with open(filename) as f: return do_stuff_with(f.read()) which would automatically give a nice error message if, say, a file object is given as argument instead of a path to a file. However useful (and efficient) these things might be, the runtime type checks are problematic, as discussed above. Furthermore, other differences between runtime and static typing may emerge (or have emerged), which will complicate the matter further. For instance, the runtime __annotations__ of classes, modules and functions may in some cases contain something completely different from what a type checker thinks the type should be. These and other incompatibilities between runtime and static typing will create two (or more) different kinds of type-annotated Python: runtime-oriented Python and Python with static type checking. These may be incompatible in both directions: a static type checker may complain about code that is perfectly valid for the runtime folks, and code written for static type checking may not be able to use new Python techniques that make use of type hints at runtime. There may not even be a fully functional subset of the two "languages". Different libraries will adhere to different standards and will not be compatible with each other. The split will be much worse and more difficult to understand than Python 2 vs 3, peoples around the world will suffer like never before, and programming in Python will become a very complicated mess. One way of solving the problem would be that type annotations are only a static concept, like with stubs or comment-based type annotations. This would also be nice from a memory and performance perspective, as evaluating and storing the annotations would not occupy memory (although both issues and some more might be nicely solved by making the annotations lazily ealuated). However, leaving out runtime effects of type annotations is not the approach taken, and runtime introspection of annotations seems to have some promising applications as well. And for many cases, the traditional Python class actually acts very nicely as both the runtime and static type. So if type annotations will be both for runtime and for static checking, how to make everything work for both static and runtime typing? Since a writer of a library does not know what the type hints will be used for by the library users, it is very important that there is only one way of making type annotations which will work regardless of what the annotations are used for in the end. This will also make it much easier to learn Python typing. Regarding runtime types and isinstance, let's look at the Iterable[int] example. For this case, there are a few options: 1) Don't implement isinstance This is problematic for runtime uses of annotations. 2) isinstance([1, '2', 'three'], Iterable[int]) returns True This is in fact now the case. This is ok for many runtime situations, but lacks precision compared to the static version. One may want to distinguish between Iterable[int] and Iterable[str] at runtime (e.g. the multidispatch example above). 3) Check as much as you can at runtime There could be something like Reiterable, which means the object is not consumed by iterating over it, so one could actually check if all elements are instances of int. This would be useful in some situations, but not available for every object. Furthermore, the check could take an arbitrary amount of time so it is not really suitable for things like multidispatch or some matching constructs etc., where the performance overhead of the type check is really important. 4) Do a deeper check than in (2) but trust the annotations For example, an instance of a class that has a method like def __iter__(self) -> Iterator[int]: some code could be identified as Iterable[int] at runtime, even if it is not guaranteed that all elements are really integers. On the other hand, an object returned by def get_ints() -> Iterable[int]: some code does not know its own annotations, so the check is difficult to do at runtime. And of course, there may not be annotations available. 5) Something else? And what about PEP544 (protocols), which is being drafted? The PEP seems to aim for having type objects that represent duck-typing protocols/interfaces. Checking whether a protocol is implemented by an object or type is clearly a useful thing to do at runtime, but it is not really clear if isinstance would be a guaranteed feature for PEP544 Protocols. So one question is, is it possible to draw the lines between what works with isinstance and what doesn't, and between what details are checked by isinstance and what aren't? -- Or should insinstance be reserved for a more limited purpose, and add another check function, say `implements(...)`, which would perhaps guarantee some answer for all combinations of object and type? I'll stop here---this email is probably already much longer than a single email should be ;) -- Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven + [View Less]

6 9

Re: [Python-ideas] Allow function to return multiple values
by rymg19＠gmail.com July 5, 2017

July 5, 2017

IIRC I'm pretty sure the OP just didn't know about the existence of tuple unpacking and the ability to use that to return multiple values. -- Ryan (ライアン) Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone elsehttp://refi64.com On Jun 25, 2017 at 6:09 PM, <Mikhail V <mikhailwas(a)gmail.com>> wrote: joannah nanjekye wrote: > [...] > >Today I was writing an example snippet for the book and needed to write a >function that returns two values … [View More]

3 2

+ operator on generators
by lucas July 5, 2017

July 5, 2017

Hello ! I often use generators, and itertools.chain on them. What about providing something like the following: a = (n for n in range(2)) b = (n for n in range(2, 4)) tuple(a + b) # -> 0 1 2 3 This, from user point of view, is just as how the __add__ operator works on lists and tuples. Making generators works the same way could be a great way to avoid calls to itertools.chain everywhere, and to limits the differences between generators and other "linear" collections. I do … [View More]

21 63

Bytecode JIT
by Soni L. July 3, 2017

July 3, 2017

CPython should get a tracing JIT that turns slow bytecode into fast bytecode. A JIT doesn't have to produce machine code. bytecode-to-bytecode compilation is still compilation. bytecode-to-bytecode compilation works on iOS, and doesn't require deviating from C. (This "internal bytecode" should do things like know that 2 variables necessarily hold integers, doing just "x = y + z" in C in an IADD instruction as opposed to all those middle-of-function typechecks and overhead. You can … [View More]

11 20

Re: [Python-ideas] Runtime types vs static types
by rymg19＠gmail.com July 2, 2017

July 2, 2017

For some background on the removal of __instancecheck__, check the linked issues here: https://github.com/python/typing/issues/135 -- Ryan (ライアン) Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone elsehttp://refi64.com On Jun 25, 2017 at 8:11 AM, <Koos Zevenhoven <k7hoven(a)gmail.com>> wrote: On Sat, Jun 24, 2017 at 11:30 PM, Lucas Wiman <lucas.wiman(a)gmail.com> wrote: > > On Sat, Jun 24, 2017 at 12:42 PM, Koos Zevenhoven <k7hoven(a)… [View More]gmail.com> > wrote: > >> There has been some discussion here and there concerning the differences >> between runtime types and static types (mypy etc.). What I write below is >> not really an idea or proposal---just a perspective, or a topic that people >> may want to discuss. Since the discussion on this is currently very fuzzy >> and scattered and not really happening either AFAICT (I've probably missed >> many discussions, though). Anyway, I thought I'd give it a shot: >> >> [...] > Regarding runtime types and isinstance, let's look at the Iterable[int] >> example. For this case, there are a few options: >> >> 1) Don't implement isinstance >> >> This is problematic for runtime uses of annotations. >> >> 2) isinstance([1, '2', 'three'], Iterable[int]) returns True >> >> This is in fact now the case. This is ok for many runtime situations, but >> lacks precision compared to the static version. One may want to distinguish >> between Iterable[int] and Iterable[str] at runtime (e.g. the multidispatch >> example above). >> >> 3) Check as much as you can at runtime >> >> There could be something like Reiterable, which means the object is not >> consumed by iterating over it, so one could actually check if all elements >> are instances of int. This would be useful in some situations, but not >> available for every object. Furthermore, the check could take an arbitrary >> amount of time so it is not really suitable for things like multidispatch >> or some matching constructs etc., where the performance overhead of the >> type check is really important. >> >> 4) Do a deeper check than in (2) but trust the annotations >> >> For example, an instance of a class that has a method like >> >> def __iter__(self) -> Iterator[int]: >> some code >> >> could be identified as Iterable[int] at runtime, even if it is not >> guaranteed that all elements are really integers. >> >> On the other hand, an object returned by >> >> def get_ints() -> Iterable[int]: >> some code >> >> does not know its own annotations, so the check is difficult to do at >> runtime. And of course, there may not be annotations available. >> >> 5) Something else? >> >> >> And what about PEP544 (protocols), which is being drafted? The PEP seems >> to aim for having type objects that represent duck-typing >> protocols/interfaces. Checking whether a protocol is implemented by an >> object or type is clearly a useful thing to do at runtime, but it is not >> really clear if isinstance would be a guaranteed feature for PEP544 >> Protocols. >> >> So one question is, is it possible to draw the lines between what works >> with isinstance and what doesn't, and between what details are checked by >> isinstance and what aren't? -- Or should insinstance be reserved for a more >> limited purpose, and add another check function, say `implements(...)`, >> which would perhaps guarantee some answer for all combinations of object >> and type? >> > >> > I'm guessing to implement PEP 544, many of the `__instancecheck__` and > `__subclasscheck__` methods in `typing.py` would need to be updated to > check the `__annotations__` of the class of the object it's passed against > its own definition, (covered in this section > <https://www.python.org/dev/peps/pep-0544/#runtime-decorator-and-narrowing-t…> > of the PEP). > > I may have missed something, but I believe PEP544 is not suggesting that annotations would have any effect on isinstance. Instead, isinstance would by default not work. > I've been somewhat surprised that many of the `__instancecheck__` > implementations do not work at runtime, even when the implementation would > be trivial (e.g. for `Union`), or would not have subtle edge cases due to > immutability (e.g. for `Tuple`, which cannot be used for checking > parameterized instances). This seems like counterintuitive behavior that > would be straightforward to fix, unless there are subtleties & edge cases > I'm missing. > > Tuple is an interesting case, because for small tuples (say 2- or 3-tuples), it makes perfect sense to check the types of all elements for some runtime purposes. Regarding Union, I believe the current situation has a lot to do with the fact that the relation between type annotations and runtime behavior hasn't really settled yet. If people are amenable to updating those cases, I'd be interested in > submitting a patch to that effect. > > Thanks for letting us know. (There may not be an instant decision on this particular case, though, but who knows :) -- Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven + _______________________________________________ Python-ideas mailing list Python-ideas(a)python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ [View Less]

6 11

Python 4: Concatenation
by Soni L. June 30, 2017

June 30, 2017

Step 1. get rid of + for strings, lists, etc. (string/list concatenation is not addition) Step 2. add concatenation operator for strings, lists, and basically anything that can be iterated. effectively an overloadable itertools.chain. (list cat list = new list, not iterator, but effectively makes itertools.chain useless.) Step 3. add decimal concatenation operator for numbers: 2 cat 3 == 23, 22 cat 33 = 2233, etc. if you need bitwise concatenation, you're already in bitwise "hack" land … [View More]

11 16

Python 4: Concatenation
by electron June 30, 2017

June 30, 2017

---------- Forwarded message ---------- From: Chris Angelico <rosuav(a)gmail.com> Date: Fri, Jun 30, 2017 at 9:12 AM Subject: Re: [Python-ideas] Python 4: Concatenation To: electron <electro.nnn(a)gmail.com> On Fri, Jun 30, 2017 at 2:38 PM, electron <electro.nnn(a)gmail.com> wrote: > On Fri, Jun 30, 2017 at 8:30 AM, Chris Angelico <rosuav(a)gmail.com> wrote: >> >> Your challenge: Find a suitable operator to use. It wants to be ASCII, >> and it has … [View More]

1 0

Re: [Python-ideas] Python 4: Concatenation
by rymg19＠gmail.com June 30, 2017

June 30, 2017

I feel like this would literally break the world for almost no real benefit... -- Ryan (ライアン) Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone elsehttp://refi64.com On Jun 29, 2017 at 6:33 PM, <Soni L. <fakedme+py(a)gmail.com>> wrote: Step 1. get rid of + for strings, lists, etc. (string/list concatenation is not addition) Step 2. add concatenation operator for strings, lists, and basically anything that can be iterated. effectively an overloadable … [View More]

1 0

Re: [Python-ideas] + operator on generators
by Steven D'Aprano June 28, 2017

June 28, 2017

On Tue, Jun 27, 2017 at 01:53:37PM -0700, Mike Miller wrote: > > On 2017-06-25 20:23, Steven D'Aprano wrote: > >I have a counter-proposal: introduce the iterator chaining operator "&": > > > > iterable & iterable --> itertools.chain(iterable, iterable) > > > > I like this suggestion. Here's another color that might be less > controversial: > > iterable3 = iterable1.chain(iterable2) That requires every iterable class to add … [View More]

2 1

Dictionary destructing and unpacking.
by Nick Humrich June 28, 2017

June 28, 2017

In python, we have beautiful unpacking: a, b, c = [1,2,3] and even a, b, *c = [1,2,3,4,5] We also have dictionary destructing for purposes of keywords: myfunc(**mydict) You can currently unpack a dictionary, but its almost certainly not what you would intend. a, b, c = {'a': 1, 'c': 3, 'b': 2}.values() In python 3.6+ this is better since the dictionary is insertion-ordered, but is still not really what one would probably want. It would be cool to have a syntax that would unpack the … [View More]

21 26

Reducing collisions in small dicts/sets
by Tim Peters June 27, 2017

June 27, 2017

Short course: the average number of probes needed when searching small dicts/sets can be reduced, in both successful ("found") and failing ("not found") cases. But I'm not going to pursue this. This is a brain dump for someone who's willing to endure the interminable pain of arguing about benchmarks ;-) Background: http://bugs.python.org/issue30671 raised some questions about how dict collisions are handled. While the analysis there didn't make sense to me, I wrote enough code to dig … [View More]into it. As detailed in that bug report, the current implementation appeared to meet the theoretical performance of "uniform hashing", meaning there was no room left for improvement. However, that missed something: the simple expressions for expected probes under uniform hashing are upper bounds, and while they're excellent approximations for modest load factors in sizable tables, for small tables they're significantly overstated. For example, for a table with 5 items in 8 slots, the load factor is a = 5/8 = 0.625, and avg probes when found = log(1/(1-a))/a = 1.57 when not found = 1/(1-a) = 2.67 However, exact analysis gives 1.34 and 2.25 instead. The current code achieves the upper bounds, but not the exact values. As a sanity check, a painfully slow implementation of uniform hashing does achieve the exact values. Code for all this is attached, in a framework that allows you to easily plug in any probe sequence strategy. The current strategy is implemented by generator "current". There are also implementations of "linear" probing, "quadratic" probing, "pre28201" probing (the current strategy before bug 28201 repaired an error in its coding), "uniform" probing, and ... "double". The last is one form of "double hashing" that gets very close to "uniform". Its loop guts are significantly cheaper than the current scheme, just 1 addition and 1 mask. However, it requires a non-trivial modulus to get started, and that's expensive. Is there a cheaper way to get close to "uniform"? I don't know - this was just the best I came up with so far. Does it matter? See above ;-) If you want to pursue this, take these as given: 1. The first probe must be simply the last `nbits` bits of the hash code. The speed of the first probe is supremely important, that's the fastest possible first probe, and it guarantees no collisions at all for a dict indexed by a contiguous range of integers (an important use case). 2. The probe sequence must (at least eventually) depend on every bit in the hash code. Else it's waaay too easy to stumble into quadratic-time behavior for "bad" sets of keys, even by accident. Have fun :-) [View Less]

2 5

A suggestion for a do...while loop
by Mikhail V June 26, 2017

June 26, 2017

>All these situations could be handled by making a "while:" with no >condition act as "while True:" >But they could also be handled by updating pep8 to make "while True:" the >recommended infinite loop syntax and make linters smarter about this (if >they aren't already). There was a big related discussion on Python-list in April (subject "Looping" ). IMHO the cleanest way to denote an infinite loop would be the statement "loop:" Without introducing new keyword I think the … [View More]

3 2

Re: [Python-ideas] [Python-Dev] Language proposal: variable assignment in functional context
by Steven D'Aprano June 26, 2017

June 26, 2017

Welcome Robert. My response below. Follow-ups to Python-Ideas, thanks. You'll need to subscribe to see any further discussion. On Fri, Jun 16, 2017 at 11:32:19AM +0000, Robert Vanden Eynde wrote: > In a nutshell, I would like to be able to write: > y = (b+2 for b = a + 1) I think this is somewhat similar to a suggestion of Nick Coghlan's. One possible syntax as a statement might be: y = b + 2 given: b = a + 1 https://www.python.org/dev/peps/pep-3150/ In mathematics, I … [View More]

7 9

A suggestion for a do...while loop
by Markus Meskanen June 26, 2017

June 26, 2017

I'm a huge fan of the do...while loop in other languages, and it would often be useful in Python too, when doing stuff like: while True: password = input() if password == ...: break I've seen the pep 315 which got rejected, and I believe these two suggestions were mostly focused on: 1. do ... while <condition>: <body> 2. do: <body> while <condition> But both were rejected for valid reasons: 1. It makes little sense to have the while at the … [View More]

9 11

Re: [Python-ideas] Allow function to return multiple values
by Mikhail V June 25, 2017

June 25, 2017

joannah nanjekye wrote: > [...] > >Today I was writing an example snippet for the book and needed to write a >function that returns two values something like this: > >def return_multiplevalues(num1, num2): > return num1, num2 > > I noticed that this actually returns a tuple of the values which I did not >want in the first place.I wanted python to return two values in their own >types so I can work with them as they are but here I was stuck with working … [View More]

1 0

Restore the __members__ behavior to python3 for C extension writers
by Barry Scott June 20, 2017

June 20, 2017

I have been trying to get dir(c_ext_obj) to work for PyCXX as the method used with python2 was removed in python3, namely use the list of names returned from the __members__ attribute. I have failed to find a simple replacement for the python3 API. It seems that I have implement __dir__. But to do that I need to know what dir() will return and add the member variables to the answer. I have been able to figure out what is necessary to write such code. No one on python users or python dev … [View More]

8 16

the error that raises an AttributeError should be passed to __getattr__
by Jason Maldonis June 20, 2017

June 20, 2017

Hi everyone, A while back I had a conversation with some folks over on python-list. I was having issues implementing error handling of `AttributeError`s using `__getattr__`. My problem is that it is currently impossible for a `__getattr__` in Python to know which method raised the `AttributeError` that was caught by `__getattr__` if there are nested methods. For example, we cannot tell the difference between `A.x` not existing (which would raise an AttributeError) and some attribute inside `… [View More]

4 10