Mailman 3 November 2017 - Python-ideas

Adding a thin wrapper class around the functions in stdlib.heapq
by bunslow Feb. 18, 2022

Feb. 18, 2022

Nothing so bombastic this time. The heapq functions are basically all named "heapsomething", and basically all take a "heap" for their first argument, with supplementary args coming after. It's a textbook example of the (hypothetical) Object Oriented Manifesto™ where defining a class increases type safety and programmers' conceptual clarity. There're practically no drawbacks, and the code to be added would be very simple. Updating the tests and docs would probably be harder. In pure Python, such a class would look like this: class Heap(list): def __init__(self, iterable=None): if iterable: super().__init__(iterable) else: super().__init__() self.heapify() push = heapq.heappush pop = heapq.heappop pushpop = heapq.heappushpop replace = heapq.heapreplace heapify = heapq.heapify # This could be a simple wrapper as well, but I had the following thoughts anyways, so here they are def nsmallest(self, n, key=None): # heapq.nsmallest makes a *max* heap of the first n elements, # while we know that self is already a min heap, so we can # make the max heap construction faster self[:n] = reversed(self[:n]) return heapq.nsmallest(n, self, key) # do we define nlargest on a minheap?? Wrapping around the C builtin functions (which aren't descriptors) would be a bit harder, but not much so: from functools import partial class Heap(list): def __init__(self, iterable=None): if iterable: super().__init__(iterable) else: super().__init__() self.heapify = partial(heapq.heapify, self) self.push = partial(heapq.heappush, self) ... self.heapify() Thoughts?

17 32

@classproperty, @abc.abstractclasspropery, etc.
by K. Richard Pixley Dec. 16, 2020

Dec. 16, 2020

There's a whole matrix of these and I'm wondering why the matrix is currently sparse rather than implementing them all. Or rather, why we can't stack them as: class foo(object): @classmethod @property def bar(cls, ...): ... Essentially the permutation are, I think: {'unadorned'|abc.abstract}{'normal'|static|class}{method|property|non-callable attribute}. concreteness implicit first arg type name comments {unadorned} {unadorned} method def foo(): exists now {unadorned} {unadorned} property @property exists now {unadorned} {unadorned} non-callable attribute x = 2 exists now {unadorned} static method @staticmethod exists now {unadorned} static property @staticproperty proposing {unadorned} static non-callable attribute {degenerate case - variables don't have arguments} unnecessary {unadorned} class method @classmethod exists now {unadorned} class property @classproperty or @classmethod;@property proposing {unadorned} class non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract {unadorned} method @abc.abstractmethod exists now abc.abstract {unadorned} property @abc.abstractproperty exists now abc.abstract {unadorned} non-callable attribute @abc.abstractattribute or @abc.abstract;@attribute proposing abc.abstract static method @abc.abstractstaticmethod exists now abc.abstract static property @abc.staticproperty proposing abc.abstract static non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract class method @abc.abstractclassmethod exists now abc.abstract class property @abc.abstractclassproperty proposing abc.abstract class non-callable attribute {degenerate case - variables don't have arguments} unnecessary I think the meanings of the new ones are pretty straightforward, but in case they are not... @staticproperty - like @property only without an implicit first argument. Allows the property to be called directly from the class without requiring a throw-away instance. @classproperty - like @property, only the implicit first argument to the method is the class. Allows the property to be called directly from the class without requiring a throw-away instance. @abc.abstractattribute - a simple, non-callable variable that must be overridden in subclasses @abc.abstractstaticproperty - like @abc.abstractproperty only for @staticproperty @abc.abstractclassproperty - like @abc.abstractproperty only for @classproperty --rich

10 15

Specify number of items to allocate for array.array() constructor
by Sven Rahmann Feb. 22, 2020

Feb. 22, 2020

At the moment, the array module of the standard library allows to create arrays of different numeric types and to initialize them from an iterable (eg, another array). What's missing is the possiblity to specify the final size of the array (number of items), especially for large arrays. I'm thinking of suffix arrays (a text indexing data structure) for large texts, eg the human genome and its reverse complement (about 6 billion characters from the alphabet ACGT). The suffix array is a long int array of the same size (8 bytes per number, so it occupies about 48 GB memory). At the moment I am extending an array in chunks of several million items at a time at a time, which is slow and not elegant. The function below also initializes each item in the array to a given value (0 by default). Is there a reason why there the array.array constructor does not allow to simply specify the number of items that should be allocated? (I do not really care about the contents.) Would this be a worthwhile addition to / modification of the array module? My suggestions is to modify array generation in such a way that you could pass an iterator (as now) as second argument, but if you pass a single integer value, it should be treated as the number of items to allocate. Here is my current workaround (which is slow): def filled_array(typecode, n, value=0, bsize=(1<<22)): """returns a new array with given typecode (eg, "l" for long int, as in the array module) with n entries, initialized to the given value (default 0) """ a = array.array(typecode, [value]*bsize) x = array.array(typecode) r = n while r >= bsize: x.extend(a) r -= bsize x.extend([value]*r) return x

14 20

Asynchronous exception handling around with/try statement borders
by Erik Bray Sept. 24, 2018

Sept. 24, 2018

Hi folks, I normally wouldn't bring something like this up here, except I think that there is possibility of something to be done--a language documentation clarification if nothing else, though possibly an actual code change as well. I've been having an argument with a colleague over the last couple days over the proper way order of statements when setting up a try/finally to perform cleanup of some action. On some level we're both being stubborn I think, and I'm not looking for resolution as to who's right/wrong or I wouldn't bring it to this list in the first place. The original argument was over setting and later restoring os.environ, but we ended up arguing over threading.Lock.acquire/release which I think is a more interesting example of the problem, and he did raise a good point that I do want to bring up. </prologue> My colleague's contention is that given lock = threading.Lock() this is simply *wrong*: lock.acquire() try: do_something() finally: lock.release() whereas this is okay: with lock: do_something() Ignoring other details of how threading.Lock is actually implemented, assuming that Lock.__enter__ calls acquire() and Lock.__exit__ calls release() then as far as I've known ever since Python 2.5 first came out these two examples are semantically *equivalent*, and I can't find any way of reading PEP 343 or the Python language reference that would suggest otherwise. However, there *is* a difference, and has to do with how signals are handled, particularly w.r.t. context managers implemented in C (hence we are talking CPython specifically): If Lock.__enter__ is a pure Python method (even if it maybe calls some C methods), and a SIGINT is handled during execution of that method, then in almost all cases a KeyboardInterrupt exception will be raised from within Lock.__enter__--this means the suite under the with: statement is never evaluated, and Lock.__exit__ is never called. You can be fairly sure the KeyboardInterrupt will be raised from somewhere within a pure Python Lock.__enter__ because there will usually be at least one remaining opcode to be evaluated, such as RETURN_VALUE. Because of how delayed execution of signal handlers is implemented in the pyeval main loop, this means the signal handler for SIGINT will be called *before* RETURN_VALUE, resulting in the KeyboardInterrupt exception being raised. Standard stuff. However, if Lock.__enter__ is a PyCFunction things are quite different. If you look at how the SETUP_WITH opcode is implemented, it first calls the __enter__ method with _PyObjet_CallNoArg. If this returns NULL (i.e. an exception occurred in __enter__) then "goto error" is executed and the exception is raised. However if it returns non-NULL the finally block is set up with PyFrame_BlockSetup and execution proceeds to the next opcode. At this point a potentially waiting SIGINT is handled, resulting in KeyboardInterrupt being raised while inside the with statement's suite, and finally block, and hence Lock.__exit__ are entered. Long story short, because Lock.__enter__ is a C function, assuming that it succeeds normally then with lock: do_something() always guarantees that Lock.__exit__ will be called if a SIGINT was handled inside Lock.__enter__, whereas with lock.acquire() try: ... finally: lock.release() there is at last a small possibility that the SIGINT handler is called after the CALL_FUNCTION op but before the try/finally block is entered (e.g. before executing POP_TOP or SETUP_FINALLY). So the end result is that the lock is held and never released after the KeyboardInterrupt (whether or not it's handled somehow). Whereas, again, if Lock.__enter__ is a pure Python function there's less likely to be any difference (though I don't think the possibility can be ruled out entirely). At the very least I think this quirk of CPython should be mentioned somewhere (since in all other cases the semantic meaning of the "with:" statement is clear). However, I think it might be possible to gain more consistency between these cases if pending signals are checked/handled after any direct call to PyCFunction from within the ceval loop. Sorry for the tl;dr; any thoughts?

7 15

Positional-only parameters
by Victor Stinner Sept. 11, 2018

Sept. 11, 2018

Hi, For technical reasons, many functions of the Python standard libraries implemented in C have positional-only parameters. Example: ------- $ ./python Python 3.7.0a0 (default, Feb 25 2017, 04:30:32) >>> help(str.replace) replace(self, old, new, count=-1, /) # <== notice "/" at the end ... >>> "a".replace("x", "y") # ok 'a' >>> "a".replace(old="x", new="y") # ERR! TypeError: replace() takes at least 2 arguments (0 given) ------- When converting the methods of the builtin str type to the internal "Argument Clinic" tool (tool to generate the function signature, function docstring and the code to parse arguments in C), I asked if we should add support for keyword arguments in str.replace(). The answer was quick: no! It's a deliberate design choice. Quote of Yury Selivanov's message: """ I think Guido explicitly stated that he doesn't like the idea to always allow keyword arguments for all methods. I.e. `str.find('aaa')` just reads better than `str.find(needle='aaa')`. Essentially, the idea is that for most of the builtins that accept one or two arguments, positional-only parameters are better. """ http://bugs.python.org/issue29286#msg285578 I just noticed a module on PyPI to implement this behaviour on Python functions: https://pypi.python.org/pypi/positional My question is: would it make sense to implement this feature in Python directly? If yes, what should be the syntax? Use "/" marker? Use the @positional() decorator? Do you see concrete cases where it's a deliberate choice to deny passing arguments as keywords? Don't you like writing int(x="123") instead of int("123")? :-) (I know that Serhiy Storshake hates the name of the "x" parameter of the int constructor ;-)) By the way, I read that "/" marker is unknown by almost all Python developers, and [...] syntax should be preferred, but inspect.signature() doesn't support this syntax. Maybe we should fix signature() and use [...] format instead? Replace "replace(self, old, new, count=-1, /)" with "replace(self, old, new[, count=-1])" (or maybe even not document the default value?). Python 3.5 help (docstring) uses "S.replace(old, new[, count])". Victor

19 27

Implicit string literal concatenation considered harmful?
by Guido van Rossum March 14, 2018

March 14, 2018

I just spent a few minutes staring at a bug caused by a missing comma -- I got a mysterious argument count error because instead of foo('a', 'b') I had written foo('a' 'b'). This is a fairly common mistake, and IIRC at Google we even had a lint rule against this (there was also a Python dialect used for some specific purpose where this was explicitly forbidden). Now, with modern compiler technology, we can (and in fact do) evaluate compile-time string literal concatenation with the '+' operator, so there's really no reason to support 'a' 'b' any more. (The reason was always rather flimsy; I copied it from C but the reason why it's needed there doesn't really apply to Python, as it is mostly useful inside macros.) Would it be reasonable to start deprecating this and eventually remove it from the language? -- --Guido van Rossum (python.org/~guido)

51 165

Repurpose `assert' into a general-purpose check
by Ivan Pozdeev Jan. 17, 2018

Jan. 17, 2018

The `assert' statment was created the same as in previous languages like C/C++: a check to only do in debug mode, when you can't yet trust your code to manage and pass around internal data correctly. Examples are array bounds and object state integrity constraints. Unlike C, Python does the aforementioned checks all the time, i.e. it's effectively always in "debug mode". Furthermore, validation checks are an extremily common use case. I have been using assert in my in-house code for input validation for a long time, the only thing preventing me from doing the same in public code is the fact that it only raises AssertionError's while library code is expected to raise TypeError or ValueError on invalid input. The last drop that incited me to do this proposition was https://stackoverflow.com/questions/2142202/what-are-acceptable-use-cases-f… where none of the _seven_ answer authors (beside me) managed to make a _single realistic use case_ for `assert' as a debug-only check. --- So, I'm hereby proposing to: * make `assert' stay in optimized mode * change its syntax to raise other types of exceptions E.g.: "assert condition, type, value" or "assert condition, type, exception_constructor_argument(s)" to maintain backward compatibility. -- Regards, Ivan

17 34

PEP 447: Adding type.__getdescriptor__
by Ronald Oussoren Dec. 3, 2017

Dec. 3, 2017

Hi, I’m trying to get up to speed again w.r.t. PEP 447 (and to be honest, with CPython’s development workflow as its been too long since I’ve actually done work on the CPython codebase…). Let me start by recapping the PEP (<https://www.python.org/dev/peps/pep-0447/ <https://www.python.org/dev/peps/pep-0447/>>) is about: The problem I’m trying to solve is that super.__getattribute__ basicy assumes looking at the __dict__ of classes on the MRO is all that’s needed to check if those classes provide an attribute. The core of both object.__getattribute__ and super.__getattribute__ for finding a descriptor on the class is basically (from the PEP): def _PyType_Lookup(tp, name): mro = tp.mro() assert isinstance(mro, tuple) for base in mro: assert isinstance(base, type) try: return base.__dict__[name] except KeyError: pass return None Note that the real implementation in in C, and accesses base.__dict__ directly instead of through Python’s attribute lookup (which would lead to infinite recursion). This is problematic when the class is dynamically populated, as the descriptor may not yet be present in the class __dict__. This is not a problem for normal attribute lookup, as you can replace the __getattribute__ method of the class to do the additional work (at the cost of having to reimplement all of __getattribute__). This is a problem for super() calls though, as super will unconditionally use the lookup code above. The PEP proposed to replace “base.__dict__[name]” by “base.__class__.__getdescriptor__(name)” (once again, in C with direct access to the two slots instead of accessing them through normal attribute lookup). I have two open questions about this PEP: 1) Last time around Mark Shannon worried that this introduces infinite recursion in the language itself (in my crummy summary, please read this message to get the real concern <https://mail.python.org/pipermail/python-dev/2015-July/140938.html <https://mail.python.org/pipermail/python-dev/2015-July/140938.html>>). Is this truly a problem? I don’t think there is a problem, but I’m worried that I don’t fully understand Mark’s concerns. 2) PEP 487 introduced __init_subclass__ as a class method to avoid having to write a metaclass for a number of use cases. My PEP currently does require a metaclass, but it might be nicer to switch to a regular class method instead (like __init_subclass__). My primary usecase for this PEP is PyObjC, which does populate the class dictionary on demand for performance and correctness reasons and PyObC therefore currently includes a replacement for super. For PyObjC’s implementation making __getdescriptor__ a classmethod would be a win as this would avoid creating yet another level of metaclasses in C code (PyObjC already has a Python class and metaclass for every Objective-C class, PEP 447 would require the introduction of a metaclass for those metaclasses). BTW. Two other open issues w.r.t. this PEP are forward porting the patch (in issue 18181) and performing benchmarks. The patch doesn’t apply cleanly to the current tip of the tree, I’ll try to update it tomorrow. And with some luck will manage to update PyObjC’s implementation as well to validate the design. Ronald

2 9

Re: [Python-ideas] [Python-Dev] What's the status of PEP 505: None-aware operators?
by Steven D'Aprano Dec. 3, 2017

Dec. 3, 2017

On Tue, Nov 28, 2017 at 12:31:06PM -0800, Raymond Hettinger wrote: > > > I also cc python-dev to see if anybody here is strongly in favor or against this inclusion. > > Put me down for a strong -1. The proposal would occasionally save a > few keystokes but comes at the expense of giving Python a more Perlish > look and a more arcane feel. I think that's an unfair characterisation of the benefits of the PEP. It's not just "a few keystrokes". Ironically, the equivalent in Perl is // which Python has used for truncating division since version 2.4 or so. So if we're in danger of looking "Perlish", that ship has sailed a long time ago. Perl is hardly the only language with null-coalescing operators -- we might better describe ?? as being familiar to C#, PHP, Swift and Dart. That's two mature, well-known languages and two up-and-coming languages. [...] > timeout ?? local_timeout ?? global_timeout As opposed to the status quo: timeout if timeout is not None else (local_timeout if local_timeout is not None else global_timeout) Or shorter, but even harder to understand: (global_timeout if local_timeout is None else local_timeout) if timeout is None else timeout I'd much prefer to teach the version with ?? -- it has a simple explanation: "the first of the three given values which isn't None". The ?? itself needs to be memorized, but that's no different from any other operator. The first time I saw ** I was perplexed and couldn't imagine what it meaned. Here ?? doesn't merely save a few keystrokes, it significantly reduces the length and complexity of the expression and entirely cuts out the duplication of names. If you can teach timeout or local_timeout or global_timeout then you ought to be able to teach ??, as it is simpler: it only compares to None, and avoids needing to explain or justify Python's truthiness model. > 'foo' in (None ?? ['foo', 'bar']) If you can understand 'foo' in (False or ['foo', 'bar']) then surely you can understand the version with ??. > requested_quantity ?? default_quantity * price Again: (default_quantity if requested_quantity is None else requested_quantity) * price I'd much prefer to read, write and teach the version with ?? over the status quo. -- Steve

11 21

How assignment should work with generators?
by Kirill Balunov Dec. 2, 2017

Dec. 2, 2017

Currently during assignment, when target list is a comma-separated list of targets (*without "starred" target*) the rule is that the object (rhs) must be an iterable with the same number of items as there are targets in the target list. That is, no check is performed on the number of targets present, and if something goes wrong the ValueError is raised. To show this on simple example: >>> from itertools import count, islice >>> it = count() >>> x, y = it >>> it count(3) Here the count was advanced two times but assignment did not happen. I found that in some cases it is too much restricting that rhs must have the same number of items as targets. It is proposed that if the rhs is a generator or an iterator (better some object that yields values on demand), the assignmenet should be lazy and dependent on the number of targets. I find this feature to be very convenient for interactive use, while it remains readable, expected, and expressed in a more compact code. There are some Pros: 1. No overhead 2. Readable and not so verbose code 3. Optimized case for x,y,*z = iterator 4. Clear way to assign values partially from infinite generators. Cons: 1. A special case of how assignment works 2. As with any implicit behavior, hard-to-find bugs There several cases with "undefined" behavior: 1. Because the items are assigned, from left to right to the corresponding targets, should rhs see side effects during assignment or not? 2. Should this work only for generators or for any iterators? 3. Is it Pythonic to distinguish what is on the rhs during assignment, or it contradicts with duck typing (goose typing)? In many cases it is possible to do this right now, but in too verbose way: >>> x, y = islice(gen(), 2) But it seems for me that: >>> x, y = gen() Looks the same, easy to follow and not so verbose. Another case, which is a pitfall, but will be possible: >>> x, y = iter([0,1,2,3,4]) # Now it is Ok >>> x 1 >>> y 2 But: >>> x, y = [0,1,2,3,4] # raises ValueError Any thoughts? With kind regards, -gdg

14 61

Add a dict with the attribute access capability
by Serhiy Storchaka Dec. 1, 2017

Dec. 1, 2017

In 3.7 I have removed an old-deprecated plistlib.Dict. [1] Actually it already was deprecated when the plistlib module was added to the regular stdlib in Python 2.6. This is a dict subclass which allows to access items as attributes. d = plistlib.Dict() d['a'] = 1 assert d.a == 1 d.b = 2 assert d['b'] == 2 Raymond noticed that that capability seemed nice to have. What do you think about reviving this type as general purpose type in collections or types? Perhaps it can be convenient for working with JSON, plists, configuration files, databases and in other cases that need a dict with string keys. If reintroduce it, there are open questions. 1. The name of the type. 2. The location of the type. collections or types? Or other variants? 3. How it will collaborate with OrderedDict, defaultdict, etc? 4. Should it be a dict subclass, or a mixin, or a proxy? Or add several types? [1] https://bugs.python.org/issue29196

12 12

Re: [Python-ideas] How assignment should work with generators?
by Alon Snir Dec. 1, 2017

Dec. 1, 2017

On Wed, Nov 29, 2017 at 5:46 AM, Alon Snir <AlonSnir(a)hotmail.com> wrote: > I would like to mention that the issue of assignment to a target list, is also relevant to the case of elementwise assignment to a mutable sequence (e.g. lists and arrays). Here as well, the rhs of the assignment statement states the number of elements to be assigned. Consequently, the following example will get into infinite loop: > >>>> from itertools import count >>>> A = [] >>>> A[:] = count() > > Writing "A[:2] = count()" will cause the same result. > Here as well, it is currently safer to use islice if the rhs is a generator or an iterator. e.g.: > >>>> it = count() >>>> A[:] = islice(it,2) > > In my opinion, it is be better to devise a solution that could be applied in both cases. Maybe a new kind of assignment operator that will be dedicated to this kind of assignment. i.e. elementwise assignment with restriction on the number of elements to be assigned, based on the length of the lhs object (or the number of targets in the target list). > ChrisA wrote: Hmm. The trouble is that slice assignment doesn't have a fixed number of targets. If you say "x, y = spam", there's a clear indication that 'spam' needs to provide exactly two values; but "A[:] = spam" could have any number of values, and it'll expand or shrink the list accordingly. Rhodri James wrote: Flatly, no. It is better not to ask for things you don't want in the first place, in this case the infinite sequence. Still, don't let me discourage you from working on this. If you can define how such an assignment would work, or even the length of A[:] as an assignment target, I'm not going to dismiss it out of hand. My answer: The idea is to define an alternative assignment rule, that is to assign exactly as many elements as the current length of the lhs object (without expanding or shrinking it). Suppose "?=" is the operator for the alternative assignment rule; A=[None]*2; and "iterator" is any iterator (finite or infinite). In this case, the following code: >>> A ?= iterator would behave like this: >>> A[:] = islice(iterator, 2) # where 2 is the length of A And as suggested earlier for the case of assignment to a target list, the following code: >>> x, y ?= iterator would behave like this: >>> x, y = islice(iterator, 2) # where 2 is the number of targets Regarding the length issue: Is there any difficulty in finding the length of a sliced sequence? After all, the range object has a len method. Therefore, the length of A[s] (where s is a slice object) could be evaluated as follows: >>> len(range(*s.indices(len(A))))

7 10

Stub class for Generic to further improve PEP 560
by Ilya Kulakov Dec. 1, 2017

Dec. 1, 2017

As explained in PEP 560, creation of Generic subclasses is much slower (~7x on my machine). My guess is that a limited number of applications actually need Generic classes at "production" runtime (i.e. when not under analysis of mypy or IDE). I propose a special class that will alias typing.Generic whenever typing.TYPE_CHECKING == True and a no-op stub otherwise. Best Regards, Ilya Kulakov

2 7

Re: [Python-ideas] [Python-Dev] What's the status of PEP 505: None-aware operators?
by Stephan Houben Nov. 30, 2017

Nov. 30, 2017

What about more English-like syntax: X or else Y E.g. cache.get(foo) or else expensive_call(foo) Stephan Op 29 nov. 2017 12:54 schreef "Serhiy Storchaka" <storchaka(a)gmail.com>: 29.11.17 11:45, Steven D'Aprano пише: On Wed, Nov 29, 2017 at 09:14:12AM +0200, Serhiy Storchaka wrote: > >> 29.11.17 08:08, Steven D'Aprano пише: >> >>> Perl is hardly the only language with null-coalescing operators -- we >>> might better describe ?? as being familiar to C#, PHP, Swift and Dart. >>> That's two mature, well-known languages and two up-and-coming languages. >>> >> >> What is the syntax of the ternary operator in these languages? >> > > All four use: > > condition ? first : second > > for the ternary if operator. > If all four use ?, it is natural that in operators which are shortcuts of the ternary operator they use ?. But in Python the bar of introducing ? is higher. _______________________________________________ Python-ideas mailing list Python-ideas(a)python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

11 14

Re: [Python-ideas] [Python-Dev] What's the status of PEP 505: None-aware operators?
by David Mertz Nov. 30, 2017

Nov. 30, 2017

I like much of the thinking in Random's approach. But I still think None isn't quite special enough to warrant it's own syntax. However, his '(or None: name.strip()[4:].upper())' makes me realize that what is being asked in all the '?(', '?.', '?[' syntax ideas is a kind of ternary expression. Except the ternary isn't based on whether a predicate holds, but rather on whether an exception occurs (AttributeError, KeyError, TypeError). And the fallback in the ternary is always None rather than being general. I think we could generalize this to get something both more Pythonic and more flexible. E.g.: val = name.strip()[4:].upper() except None This would just be catching all errors, which is perhaps too broad. But it *would* allow a fallback other than None: val = name.strip()[4:].upper() except -1 I think some syntax could be possible to only "catch" some exceptions and let others propagate. Maybe: val = name.strip()[4:].upper() except (AttributeError, KeyError): -1 I don't really like throwing a colon in an expression though. Perhaps some other word or symbol could work instead. How does this read: val = name.strip()[4:].upper() except -1 in (AttributeError, KeyError) Where the 'in' clause at the end would be optional, and default to 'Exception'. I'll note that what this idea DOES NOT get us is: val = timeout ?? local_timeout ?? global_timeout Those values that are "possibly None" don't raise exceptions, so they wouldn't apply to this syntax. Yours, David... On Wed, Nov 29, 2017 at 9:03 AM, Random832 <random832(a)fastmail.com> wrote: > On Tue, Nov 28, 2017, at 15:31, Raymond Hettinger wrote: > > > > > I also cc python-dev to see if anybody here is strongly in favor or > against this inclusion. > > > > Put me down for a strong -1. The proposal would occasionally save a few > > keystokes but comes at the expense of giving Python a more Perlish look > > and a more arcane feel. > > > > One of the things I like about Python is that I can walk non-programmers > > through the code and explain what it does. The examples in PEP 505 look > > like a step in the wrong direction. They don't "look like Python" and > > make me feel like I have to decrypt the code to figure-out what it does. > > > > timeout ?? local_timeout ?? global_timeout > > 'foo' in (None ?? ['foo', 'bar']) > > requested_quantity ?? default_quantity * price > > name?.strip()[4:].upper() > > user?.first_name.upper() > > Since we're looking at different syntax for the ?? operator, I have a > suggestion for the ?. operator - and related ?[] and ?() that appeared > in some of the proposals. How about this approach? > > Something like (or None: ...) as a syntax block in which any operation > [lexically within the expression, not within e.g. called functions, so > it's different from simply catching AttributeError etc, even if that > could be limited to only catching when the operand is None] on None that > is not valid for None will yield None instead. > > This isn't *entirely* equivalent, but offers finer control. > > v = name?.strip()[4:].upper() under the old proposal would be more or > less equivalent to: > > v = name.strip()[4:].upper() if name is not None else None > > Whereas, you could get the same result with: > (or None: name.strip()[4:].upper()) > > Though that would technically be equivalent to these steps: > v = name.strip if name is not None else None > v = v() if v """"" > v = v[4:] """"""" > v = v.upper """"""" > v = v() """"""" > > The compiler could optimize this case since it knows none of the > operations are valid on None. This has the advantage of being explicit > about what scope the modified rules apply to, rather than simply > implicitly being "to the end of the chain of dot/bracket/call operators" > > It could also be extended to apply, without any additional syntax, to > binary operators (result is None if either operand is None) (or None: a > + b), for example, could return None if either a or b is none. > > [I think I proposed this before with the syntax ?(...), the (or None: > ...) is just an idea to make it look more like Python.] > _______________________________________________ > Python-Dev mailing list > Python-Dev(a)python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > mertz%40gnosis.cx > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.

6 5

Immutable dictionaries
by Asen Bozhilov Nov. 30, 2017

Nov. 30, 2017

This is my first post here. I have strong experience with JavaScript and I'm lucky that I could move forward to Python. What I miss in Python are immutable dictionaries. They are especially useful for configurations and call functions which expect dictionary as argument. In my opinion they would let a place for underlying optimizations. I'd like to propose also literaling syntax for immutable dictionaries. immutable_dict = ( 'key1' : 'value1', 'key2' : 'value2' ) This syntax is not ambiguous with expression statements and tuple literals, but it requires a bit lookahed during the parsing. I would appreciate your opinions on the topic. Most interesting for me is why they are not already part of the language? Kind regards, Asen Bozhilov

7 6

Re: [Python-ideas] [Python-Dev] What's the status of PEP 505: None-aware operators?
by David Mertz Nov. 29, 2017

Nov. 29, 2017

True enough. I remember the prior discussion, but didn't look up the PEP number. I know it's hard to revisit these ideas, but occasionally it works, especially if a nicer spelling is found (Barry proposed some a bit different than my ideas) On Nov 29, 2017 9:55 AM, "Eric V. Smith" <eric(a)trueblade.com> wrote: On Nov 29, 2017, at 12:40 PM, David Mertz <mertz(a)gnosis.cx> wrote: I like much of the thinking in Random's approach. But I still think None isn't quite special enough to warrant it's own syntax. However, his '(or None: name.strip()[4:].upper())' makes me realize that what is being asked in all the '?(', '?.', '?[' syntax ideas is a kind of ternary expression. Except the ternary isn't based on whether a predicate holds, but rather on whether an exception occurs (AttributeError, KeyError, TypeError). And the fallback in the ternary is always None rather than being general. I think we could generalize this to get something both more Pythonic and more flexible. E.g.: val = name.strip()[4:].upper() except None This would just be catching all errors, which is perhaps too broad. But it *would* allow a fallback other than None: val = name.strip()[4:].upper() except -1 I think some syntax could be possible to only "catch" some exceptions and let others propagate. Maybe: val = name.strip()[4:].upper() except (AttributeError, KeyError): -1 I don't really like throwing a colon in an expression though. Perhaps some other word or symbol could work instead. How does this read: val = name.strip()[4:].upper() except -1 in (AttributeError, KeyError) Where the 'in' clause at the end would be optional, and default to 'Exception'. I'll note that what this idea DOES NOT get us is: val = timeout ?? local_timeout ?? global_timeout Those values that are "possibly None" don't raise exceptions, so they wouldn't apply to this syntax. See the rejected PEP 463 for Exception catching expressions. Eric. Yours, David... On Wed, Nov 29, 2017 at 9:03 AM, Random832 <random832(a)fastmail.com> wrote: > On Tue, Nov 28, 2017, at 15:31, Raymond Hettinger wrote: > > > > > I also cc python-dev to see if anybody here is strongly in favor or > against this inclusion. > > > > Put me down for a strong -1. The proposal would occasionally save a few > > keystokes but comes at the expense of giving Python a more Perlish look > > and a more arcane feel. > > > > One of the things I like about Python is that I can walk non-programmers > > through the code and explain what it does. The examples in PEP 505 look > > like a step in the wrong direction. They don't "look like Python" and > > make me feel like I have to decrypt the code to figure-out what it does. > > > > timeout ?? local_timeout ?? global_timeout > > 'foo' in (None ?? ['foo', 'bar']) > > requested_quantity ?? default_quantity * price > > name?.strip()[4:].upper() > > user?.first_name.upper() > > Since we're looking at different syntax for the ?? operator, I have a > suggestion for the ?. operator - and related ?[] and ?() that appeared > in some of the proposals. How about this approach? > > Something like (or None: ...) as a syntax block in which any operation > [lexically within the expression, not within e.g. called functions, so > it's different from simply catching AttributeError etc, even if that > could be limited to only catching when the operand is None] on None that > is not valid for None will yield None instead. > > This isn't *entirely* equivalent, but offers finer control. > > v = name?.strip()[4:].upper() under the old proposal would be more or > less equivalent to: > > v = name.strip()[4:].upper() if name is not None else None > > Whereas, you could get the same result with: > (or None: name.strip()[4:].upper()) > > Though that would technically be equivalent to these steps: > v = name.strip if name is not None else None > v = v() if v """"" > v = v[4:] """"""" > v = v.upper """"""" > v = v() """"""" > > The compiler could optimize this case since it knows none of the > operations are valid on None. This has the advantage of being explicit > about what scope the modified rules apply to, rather than simply > implicitly being "to the end of the chain of dot/bracket/call operators" > > It could also be extended to apply, without any additional syntax, to > binary operators (result is None if either operand is None) (or None: a > + b), for example, could return None if either a or b is none. > > [I think I proposed this before with the syntax ?(...), the (or None: > ...) is just an idea to make it look more like Python.] > _______________________________________________ > Python-Dev mailing list > Python-Dev(a)python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/mertz% > 40gnosis.cx > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. _______________________________________________ Python-Dev mailing list Python-Dev(a)python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ eric%2Ba-python-dev%40trueblade.com

1 0

generator vs iterator etc. (was: How assignment should work with generators?)
by Koos Zevenhoven Nov. 29, 2017

Nov. 29, 2017

On Mon, Nov 27, 2017 at 3:55 PM, Steven D'Aprano <steve(a)pearwood.info> wrote: > On Mon, Nov 27, 2017 at 12:17:31PM +0300, Kirill Balunov wrote: > > > > 2. Should this work only for generators or for any iterators? > > I don't understand why you are even considering singling out *only* > generators. A generator is a particular implementation of an iterator. I > can write: > > def gen(): > yield 1; yield 2; yield 3 > > it = gen() > > or I can write: > > it = iter([1, 2, 3]) > > and the behaviour of `it` should be identical. > > > I can see where this is coming from. The thing is that "iterator" and "generator" are mostly synonymous, except two things: (1) Generators are iterators that are produced by a generator function (2) Generator functions are sometimes referred to as just "generators" The concept of "generator" thus overlaps with both "iterator" and "generator function". Then there's also "iterator" and "iterable", which are two different things: (3) If `obj` is an *iterable*, then `it = iter(obj)` is an *iterator* (over the contents of `obj`) ( 4) Iterators yield values, for example on explicit calls to next(it). Personally I have leaned towards keeping a clear distinction between "generator function" and "generator", which leads to the situation that "generator" and "iterator" are mostly synonymous for me. Sometimes, for convenience, I use the term "generator" to refer to "iterators" more generally. This further seems to have a minor benefit that "generators" and "iterables" are less easily confused with each other than "iterators" and "iterables". I thought about this issue some time ago for the `views` package, which has a separation between sequences (seq) and other iterables (gen): https://github.com/k7hoven/views The functionality provided by `views.gen` is not that interesting—it's essentially a subset of itertools functionality, but with an API that parallels `views.seq` which works with sequences (iterable, sliceable, chainable, etc.). I used the name `gen`, because iterator/iterable variants of the functionality can be implemented with generator functions (although also with other kinds of iterators/iterables). Calling the thing `iter` would have conflicted with the builtin `iter`. HOWEVER, this naming can be confusing for those that lean more towards using "generator" to also mean "generator function", and for those that are comfortable with the term "iterator" despite its resemblance to "iterable". Now I'm actually seriously considering to consider renaming `views.gen` to ` views.iter` when I have time. After all, there's already `views.range` which "conflicts" with the builtin range. Anyway, the point is that the naming is suboptimal. SOLUTION: Maybe (a) all iterators should be called iterators or (b) all iterators should be called generators, regardless of whether they are somehow a result of a generator function having been called in the past. (I'm not going into the distinction between things that can receive values via `send` or any other possible distinctions between different types of iterators and iterables.) —Koos (discussion originated from python-ideas, but cross-posted to python-dev in case there's more interest there) -- + Koos Zevenhoven + http://twitter.com/k7hoven +

5 7

Logging Levels
by Mike Miller Nov. 29, 2017

Nov. 29, 2017

Hi, I use the logging module extensively for even the simplest scripts, one of the reasons there's been less difficulty moving to Python 3 at work. One of the "nano features" I've often added to its config is the addition of a custom log level. Never mentioned it before because of its triviality and minor use cases, but just realized that I've been doing it five years now and happy with it, so why not? NOTE (~35) It is called the "note" level and used when one needs to express something important, yet positive, and have it be output by default. The standard levels don't support this currently, you often have to potentially scare the end user with a warning or higher to have a message emitted. Typically I use it to return important information that was asked for specifically and retrieved successfully, e.g.: log.note('Your token is: %s', token) log.note(f'⏵ {item.id} {item.name}') There are other examples. Sphinx, has the concept of note admonitions for docs in addition to warning and danger. Bootstrap has note banners for web/apps. There is something important to express or highlight, but nothing to worry about. FATAL (alias of CRITICAL) Can't find it now, but believe the docs in the past stated that CRITICAL was meant for *unrecoverable* errors. I've never had a project where I didn't shut down immediately after such an occurrence. Therefore I find "FATAL" a more accurate description of what happened. Log4j and other industry loggers use this level name as well. There is also an aesthetic reason to prefer "fatal". It is shorter and so aligns better with other level names for readability, e.g.: console_format = ' %(levelname)-7.7s %(message)s' Tried but never found a good abbreviation for critical, unfortunately. The other option is to add length to align the field. Most messages use the shorter level names (debug, info) so extra length results in wasted space that is very rarely needed. Hopefully someone else finds these useful. Neither depends on the other. -Mike

5 14

Proposal: allow length_hint to specify infinite iterators
by Steven D'Aprano Nov. 29, 2017

Nov. 29, 2017

PEP 424 allows iterators to optionally offer a hint as to how long they will be: https://www.python.org/dev/peps/pep-0424/ Unfortunately, there's no good way for an iterator to report that it is infinitely long. Consequently, even those which are known to be infinite report finite lengths: py> from itertools import count py> from operator import length_hint py> infinite = count() py> length_hint(infinite) 0 This wastes the opportunity to fail fast on operations which cannot possibly succeed, e.g. list(count()) must eventually fail with MemoryError. Or worse: if the OS starts thrashing trying to meet the memory requests, you can lock up the computer. I propose that we: (1) extend the __length_hint__ protocol to allow iterators to report that they are infinite; (2) and recommend that consumers of iterators (such as list) that require finite input should fail fast in the event of an infinite iterator. Four possible ways that __length_hint__ and operator.length_hint might signal an infinite iterator: (a) return a negative value such as -1 (this is currently an error); (b) return some special sentinel value; (c) take the convention that returning sys.maxint means infinity; (d) raise an exception. The advantage of (d) is that consumers of check __length_hint__ don't need to do anything special to fail fast on infinite iterators: py> class Thing: ... def __length_hint__(self): ... raise ValueError('infinite') ... def __iter__(self): ... return count() ... py> x = Thing() py> list(x) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 3, in __length_hint__ ValueError: infinite but if they can cope with such, they can explicitly catch the exception. Thoughts? -- Steve

3 2

Looking for input to help with the pip situation
by Michel Desmoulin Nov. 29, 2017

Nov. 29, 2017

Hello, Today I'm going to give a training in Python again. And again it will go the same way. On Mac I will have to make people install python, then tell them to use pip3. On Windows, I will have to warn them about checking the "add python executable to system path" (that one of them will ALWAYS miss anyway). Then tell them to use py -3.x -m pip because some of them will have several versions of Python installed. Then on linux, I will tell them to install python-pip and python-venv and use python3 -m pip. I'll talk about --user, but commands won't be usable on some machine where the Scripts or bin dir is not in the system path. Then I will make them create a virtualenv so that they can avoid messing with their system python and finally can just use "pip install" like in most tutorials on the Web. And eventually I'll talk about pipenv and conda. The first one so they don't have to think about activating the virtualenv everytime, or pip freeze, or create the venv, or add it to gitignore, etc. The second because anaconda is very popular on windows. There is no way a beginner is going to get any that by themselves without a lot of time and pain. They will read some tutorial on the web and struggle to make sens of what pip is and why "pip install" doesn't work and why "python sucks". I think Python is offering an incredible experience for first timer. However, the whole "where is walpip" shenanigans is not one of them. I really want some people from this list to discuss here so we can find a way to either unify a bit the way we install and use pip, or find a way to express a tutorial that always works for people on the most popular platforms and spread the word so that any doc uses it. Michel

27 119

Re: [Python-ideas] How assignment should work with generators?
by Alon Snir Nov. 29, 2017

Nov. 29, 2017

I would like to mention that the issue of assignment to a target list, is also relevant to the case of elementwise assignment to a mutable sequence (e.g. lists and arrays). Here as well, the rhs of the assignment statement states the number of elements to be assigned. Consequently, the following example will get into infinite loop: >>> from itertools import count >>> A = [] >>> A[:] = count() Writing "A[:2] = count()" will cause the same result. Here as well, it is currently safer to use islice if the rhs is a generator or an iterator. e.g.: >>> it = count() >>> A[:] = islice(it,2) In my opinion, it is be better to devise a solution that could be applied in both cases. Maybe a new kind of assignment operator that will be dedicated to this kind of assignment. i.e. elementwise assignment with restriction on the number of elements to be assigned, based on the length of the lhs object (or the number of targets in the target list).

4 3

raise in lambda
by Soni L. Nov. 28, 2017

Nov. 28, 2017

Would be useful to pass exception-raising lambdas around. ensure(cond, lambda: raise TypeError()) I guess one could instead use (and raise in ensure()) ensure(cond, lambda: TypeError()) But I think exception-raising lambdas would be nicer.

2 1

Re: [Python-ideas] Repurpose `assert' into a general-purpose check
by David Mertz Nov. 28, 2017

Nov. 28, 2017

I've used Python for 20 years, and programming in general for about 40 now. I think almost EVERY large code base I've seen used asserts in their intended way. I frequently use them myself in a way differentiated (subtly) from 'if cond: raise SomeException'. I honestly don't know how Ivan has avoided seeing these. Giving up this useful construct would feel about the same as depreciating 'finally'. Yes, Python would be Turing complete without it, and you could work around it being missing... But why?! On Nov 27, 2017 11:12 PM, "Ivan Pozdeev via Python-ideas" < python-ideas(a)python.org> wrote: On 28.11.2017 8:59, Steven D'Aprano wrote: > On Tue, Nov 28, 2017 at 07:35:45AM +0300, Ivan Pozdeev via Python-ideas > wrote: > > Actually, the way I'm using them, >> >> assert condition, "error message", type >> >> would probably be the most expressive way. >> > I disagree that is expressive -- I call it *misleading*. I see something > which looks like an assertion (that is, a checked comment, a contract, a > check on an internal piece of logic etc) but it is actually being used > as a test. > > > I can do anything in any Turing-complete language without any changes to >> the language. That's no reason to never change anything, is it. >> > "We can change this" is not a reason to change this. There needs to be a > *good* reason to change, and you have given no good reasons for this > change. > > > The rationale basically is: >> * As it was intended, the statement has no practical use -- basically a >> rudiment, due to disappear eventually >> > Nonsense. I make extensive use of assert as a way of checking > assertions, and I will fight tooth and nail against any proposal to > either remove it or to misuse it for public input tests instead of > assertions. > I invite you to show me a single use case for those "assertions" because after ~20 years of experience in coding (that included fairly large projects), I've yet to see one. Any, every check that you make at debug time either * belongs in production as well (all the more because it's harder to diagnose there), or * belongs in a test -- something coded independently from the program (if your code as a whole cannot be trusted, how any specific part of it can?), or * isn't needed at all because a fault will inevitably surface somewhere down the line (as some exception or an incorrect result that a test will catch). Finally, I've got much experience using existing code outside its original use cases, where the original author's assumptions may no longer hold but the specific logic can be gauded to produce the desired result. Coding these assumptions in would undermine that goal. So, I see "debug assertions" as either intentionally compromizing correctness for performance (a direct opposite of Python's design principles), or as an inferiour, faulty, half-measure rudiment from times when CI wasn't a thing (thus not something that should be taught and promoted as a best practice any longer). > * It can instead be reused as syntax sugar to cover a very common use case >> > There is no need for such syntactic sugar. It would be harmful > to use assert for something which is not an assertion. > > > > -- Regards, Ivan _______________________________________________ Python-ideas mailing list Python-ideas(a)python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

1 0

Re: [Python-ideas] How assignment should work with generators?
by Kirill Balunov Nov. 27, 2017

Nov. 27, 2017

> > > To show this on simple example: > > > > >>> from itertools import count, islice > > >>> it = count() > > >>> x, y = it > > >>> it > > count(3) > > For everyone else who was confused by this, as I was, that's not > actually a copy and paste from the REPL. There should be a ValueError > raised after the x, y assignment. As given, it is confusing because it > looks like the assignment succeeded, when in fact it didn't. > > > > Here the count was advanced two times but assignment did not happen. > > Correct, because there was an exception raised. > Sorry for that, I did not want to embarrass anyone, so I wrote below that the assignment did not happen. But probably the code should speak for itself, especially if it looks like a copy from REPL > if isinstance(it, collections.abc.Iterator): > # special case for iterators > x, y = it > else: > # sequences keep the old behaviour > x, y = it[:2] > No, it can be simply x, y = iter(it) > > Cons: > > 1. A special case of how assignment works > > 2. As with any implicit behavior, hard-to-find bugs > > Right. Hard-to-find bugs beats any amount of convenience in the > interactive interpreter. To use an analogy: > > "Sure, sometimes my car suddenly shifts into reverse while I'm driving > at 60 kph, sometimes the engine falls out when I go around the corner, > and occasionally the brakes catch fire, but gosh the cup holder makes it > really convenient to drink coffee while I'm stopped at traffic lights!" > :-) Perhaps a better idea might be special syntax to tell the interpreter > you don't want to run the right-hand side to completion. "Explicit is > better than implicit" -- maybe something special like: > > x, y, * = iterable > I wrote somewhere above, that "may be I like this form". But for me * '"starred" target implies -> collect something from iterable. So now I'm towards: x, y, ... = iterable But I have not summed up yet what pitfalls can be on this path. Thank you your remarks were very extensive! With kind regards, -gdg

1 0

Allow additional separator character in variables
by Mikhail V Nov. 27, 2017

Nov. 27, 2017

Python allows underscore character as a separator in variables. This is better than nothing, still it does not make the look much better. **Proposal**: allow additional separator, namely hyphen character. **Benefits**: this should provide significant readability improvement. Compared to most syntax change proposals that I remember till now, this seems to have really tangible benefits. So all in all I see it as a significant language improvement. Besides its direct benefit as a good looking separator, it also gives an opprtunity to reduce "camelCase" or similar ugly inclusions in code. So one can easily compose human-readable variable names e.g.: my-variable figure-shape---width etc. **Problem**: currently hyphen character is used as a minus operator! The problem is as old as the history of most programming languages, and inherited from initial poorness of character sets. Therefore I don't see a single optimal solution to this. Possible solutions: Solution 1: Use another similar looking character from unicode, for example: U+02D7 (modifier letter minus sign). At the same time IMO it is needed to allow the minus character for the minus operator, namely U+2212 Minus sign. This will allow proper typography of source code. Main benefit of such approach: no breakage of current code base, since new chars are additional to existing ones. Solution 2: (radical) Disallow hyphen as minus operator, and use only U+2212 Minus sign. I.e. "a-b" would be a variable and "a − b" a minus operation. Advantage: opportunity to correct the improper character usage once and for all. Disadvantage: breakage of current code base + force UTF-8 storage use (consider e.g. editors without unicode support). Thus most probably such solution will cause howl reaching to the sky among users, despite many modern editors allow unicode and custom operator styling, for example to distinguish dash from hyphen in a monospaced editor. So is my proposal, and as usual urging for constructive conversation. (i.e. proposing to write own language/translator is not constructive conversation) Cheers, Mikhail

18 50

Possible Enhancement to py.exe Python Launcher
by Steve Barnes Nov. 24, 2017

Nov. 24, 2017

Following on from the discussions on pip I would like to suggest, (and possibly implement), a minor change to the current py.exe python launcher. Currently the launcher, when called without a version specifier, defaults to the highest version on the basis of 3>2, x.11 > x.9, -64 > -32 however this may not always be the most desirable behaviour. I would like to suggest that it take a configuration value for the default version to use, (and possibly a separate ones for pyw, & file associations), honouring the following with the later overriding: default - as current system setting - from registry user setting - from registry user setting - from config file maybe %appdata%\pylauncher\defaults.ini environment setting - from getenv local setting - from .pyconfig file in the current directory. Options would be the same format as the -X[.Y][-BB] format currently accepted on the command line plus a --No-Py-Default option which would always error out if the version to invoke was not specified. I see this as potentially adding quite a lot of value for people with multiple python versions installed and it could tie in quite well with the possible use of py.exe as an entry point for tools such as pip. It might also increase the awareness of the launcher as those who have to stick with python 2 for the moment or in a specific context could set the default to what they need but can always override. -- Steve (Gadget) Barnes Any opinions in this message are my personal opinions and do not reflect those of my employer. --- This email has been checked for viruses by AVG. http://www.avg.com

2 2

Re: [Python-ideas] Rewriting the "roundrobin" recipe in the itertools documentation
by Alon Snir Nov. 23, 2017

Nov. 23, 2017

Sorry. Unfortunately I had a mistake. Alon Snir Date: Thu, 23 Nov 2017 12:29:47 +1100 From: Steven D'Aprano <steve(a)pearwood.info> To: python-ideas(a)python.org Subject: Re: [Python-ideas] Rewriting the "roundrobin" recipe in the itertools documentation Message-ID: <20171123012947.GT19802(a)ando.pearwood.info> Content-Type: text/plain; charset=utf-8 On Wed, Nov 22, 2017 at 11:28:20AM +0000, Alon Snir wrote: > It would be faster with ?deque?: It isn't. According to my testing, your version with deque is approximately two times slower than the version from toolz.itertoolz that Wes quotes. -- Steve

1 0

Re: [Python-ideas] Adding a thin wrapper class around the functions in stdlib.heapq
by bunslow Nov. 23, 2017

Nov. 23, 2017

On Wed, Nov 22, 2017 at 10:52 PM, bunslow <bunslow(a)gmail.com> wrote: > I'll just note the original proposal I made was specifically designed to > be the minimum possible improvement, to avoid controversy (and e.g. a PEP). > > I agree that there are significant flaws and room for improvement in the > current heapq module (which is why things like xheap exist), but even so > it's still very useful as is, and the minimal OOP wrapper would go a long > way towards usability of the current functionality (and of course "just use > pip" is often not a viable course of action for certain developers). It > seems that such a wrapper is widely (though not unanimously) approved. > > Next step I suppose is creating a bpo issue? > > On Wed, Nov 22, 2017 at 10:05 PM, Nick Coghlan <ncoghlan(a)gmail.com> wrote: > >> On 22 November 2017 at 11:00, Chris Angelico <rosuav(a)gmail.com> wrote: >> >>> So the question is more: why, with Python being the way it is, do the >>> heap functions operate on a list? I think heapq.heapify is the answer: >>> in linear time, it heapifies a list *in place*. >>> >>> I don't think there's any reason to have *both* interfaces to the heap >>> functionality, but it certainly isn't illogical to try to treat a heap >>> as a thing, and therefore have a Heap type. >>> >> >> Right, the parallel here is that we have a "heapq" module for the same >> reason that we have list.sort(), sorted(), and the bisect module, rather >> than a single "SortedList" type. https://code.activestate.com/r >> ecipes/577197-sortedcollection/ then provides an example of how to >> combine those into a "SortedCollection" type. >> >> That said, I'm still in favour of adding an object-oriented wrapper to >> either `collections` or the `heapq` module for all the classic OO reasons: >> >> - it makes it easier to reliably maintain the heap invariant (just drop >> your list reference after passing it to the heap wrapper) >> - it gives both human readers and static code analysers more information >> to work with ("this is a heap" rather than "this is a list") >> - it provides a hook for improved interactive help on heap instances >> >> I don't have any great concerns about potential confusion - the OO >> wrapper will be easy enough to use that anyone that's unsure will likely >> gravitate towards that, while the lower level `heapq` functions will remain >> available for folks writing their own heap implementations. >> >> This effect would likely be even more pronounced if the OO wrapper were >> made available as `collections.Heap` (`collections` already imports the >> `heapq` module for use in the Counter implementation). >> >> Cheers, >> Nick. >> >> -- >> Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas(a)python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> >

1 0

Re: [Python-ideas] Rewriting the "roundrobin" recipe in the itertools documentation
by Alon Snir Nov. 23, 2017

Nov. 23, 2017

def roundrobin(*iterables): "roundrobin('ABC', 'D', 'EF') --> A D E B F C" nexts = [ iter(it).__next__ for it in iterables ] i = 0 while nexts: i %= len(nexts) try: yield nexts[i]() except StopIteration: del nexts[i] else: i += 1 Regards Alon Snir

3 3

Re: [Python-ideas] Should Python have user-defined constants?
by Saeed Baig Nov. 22, 2017

Nov. 22, 2017

Hi. Just to respond to some peoples’ points… Bernardo asked 'This would be "constant" as in Java constants right? Referential constants, so that if an object were mutable you would still be able to change its internal state.’ Um, I’m not entirely sure what you mean, since I’ve never used Java. But if it means what I think it means, I would be INCLINED to say no. If, for example, I saw code like this: x = [9, 5, 7, 1] let y = x y[0] = 42 print(y) … I would expect that code to raise an error, not print “[42, 5, 7, 1]”, since you’re trying to modify the “constant” y in place, which… well, would kind of go against the idea of y being a constant, I thought (unless I’m misunderstanding what you meant by “object", since everything in Python’s an object). However, I might change my opinion if I heard a good reason why Java-like constants would be better. Serhiy asked, in relation to constants, “To do what? What problem do you need to solve?”. No problem in particular, to be honest. I just thought they’d be nice since they’d increase confidence that my variables-intended-to-be-constants wouldn’t get reassigned, and just to increase readability/explicitness to other programmers. > On 21 Nov 2017, at 20:08, python-ideas-request(a)python.org wrote: > > Send Python-ideas mailing list submissions to > python-ideas(a)python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/python-ideas > or, via email, send a message with subject or body 'help' to > python-ideas-request(a)python.org > > You can reach the person managing the list at > python-ideas-owner(a)python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Python-ideas digest..." > > > Today's Topics: > > 1. Re: Should Python have user-defined constants? (Steven D'Aprano) > 2. Re: Should Python have user-defined constants? (St?fane Fermigier) > 3. Re: Should Python have user-defined constants? (Bernardo Sulzbach) > 4. Star assignment in iterator way? (Kirill Balunov) > 5. Re: Star assignment in iterator way? (Serhiy Storchaka) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 21 Nov 2017 19:13:42 +1100 > From: Steven D'Aprano <steve(a)pearwood.info> > To: python-ideas(a)python.org > Subject: Re: [Python-ideas] Should Python have user-defined constants? > Message-ID: <20171121081342.GK19802(a)ando.pearwood.info> > Content-Type: text/plain; charset=us-ascii > > On Tue, Nov 21, 2017 at 02:38:45AM -0500, Joseph Jevnik wrote: > >> How is that different from "pi = 3.14"? > > pi = 3.14 > pi = 5 > print(pi) > # prints 5 > > let pi = 3.14 > pi = 5 > # raises an exception > > > > -- > Steve > > > ------------------------------ > > Message: 2 > Date: Tue, 21 Nov 2017 09:26:17 +0100 > From: St?fane Fermigier <sf(a)fermigier.com> > To: Python-Ideas <python-ideas(a)python.org> > Subject: Re: [Python-ideas] Should Python have user-defined constants? > Message-ID: > <CABuJJj5-ZhuWTBns8rm8jVWPaOGq08MM+X0ryQp1z7ogQFsHvw(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Javascript (ES6) has 'let' and 'const' for mutable and constant variables, > respectively. > > When programming in JS, I find myself aggressively aiming for as little > 'let' and as much 'const' as reasonably possible, since reasoning about > constants is much easier than about variables. In this context, 'const' is > used as a marker, a runtime checker, and, with proper tooling (IDE or > linter), a coding-time reminder to differentiate between these two > fundamental behaviours. > > I'm not +1 on introducing this idea to Python yet, but not -1 either - this > deserves some discussion, if this has not been discussed already. > > Cheers, > > S. > > On Tue, Nov 21, 2017 at 9:13 AM, Steven D'Aprano <steve(a)pearwood.info> > wrote: > >> On Tue, Nov 21, 2017 at 02:38:45AM -0500, Joseph Jevnik wrote: >> >>> How is that different from "pi = 3.14"? >> >> pi = 3.14 >> pi = 5 >> print(pi) >> # prints 5 >> >> let pi = 3.14 >> pi = 5 >> # raises an exception >> >> >> >> -- >> Steve >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas(a)python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > > -- > Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier - > http://linkedin.com/in/sfermigier > Founder & CEO, Abilian - Enterprise Social Software - > http://www.abilian.com/ > Chairman, Free&OSS Group / Systematic Cluster - > http://www.gt-logiciel-libre.org/ > Co-Chairman, National Council for Free & Open Source Software (CNLL) - > http://cnll.fr/ > Founder & Organiser, PyData Paris - http://pydata.fr/ > --- > ?You never change things by ?ghting the existing reality. To change > something, build a new model that makes the existing model obsolete.? ? R. > Buckminster Fuller >

11 16

Make bytes __repr__ and __str__ representation different?
by Kirill Balunov Nov. 22, 2017

Nov. 22, 2017

Currently, __repr__ and __str__ representation of bytes is the same. Perhaps it is worth making them different, this will make it easier to visually perceive them as a container of integers from 0 to 255, instead of a mixture of printable and non-printable ascii characters. It is proposed: a) __str__ - leave unchanged b) __repr__ - represent as sequence of escaped hex >>> a = bytes([42,43,44,45,46]) >>> a # Current b'*+-./' >>> a # Proposed b'\x2a\x2b\x2d\x2e\x2f' As you can see, the second example is more easily perceived as a sequence, in which '\' is also perceived as ',' in list or tuple. In addition, 2020 is close, it allows the new Pythonistas not to take them as an ascii mixture strings. With kind regards, -gdg

6 6

Re: [Python-ideas] Make bytes __repr__ and __str__ representation different?
by Steven D'Aprano Nov. 21, 2017

Nov. 21, 2017

On Tue, Nov 21, 2017 at 01:35:41PM -0200, Joao S. O. Bueno wrote: > On 21 November 2017 at 13:16, Steven D'Aprano <steve(a)pearwood.info> wrote: > > I'd rather leave __str__ and __repr__ alone. Changing them will have > > huge backwards compatibility implications. I'd rather give bytes a > > hexdump() method that returns a string: > > > > '2a 2b 2d 2e 2f' > > Do you mean, like bytes.hex? Guido's Time Machine strikes again! -- Steve

1 0

Re: [Python-ideas] Should Python have user-defined constants?
by Hugh Fisher Nov. 21, 2017

Nov. 21, 2017

> Message: 4 > Date: Tue, 21 Nov 2017 18:05:17 +0000 > From: ????? <elazarg(a)gmail.com> > To: Chris Barker <chris.barker(a)noaa.gov> > Cc: Python-Ideas <python-ideas(a)python.org> > Subject: Re: [Python-ideas] Should Python have user-defined constants? [munch] > > It was mentioned that there are related conventions in Python. Good point. > But they are not as strong as the "convention" that tuples and strings > cannot change, and this is useful in reasoning about code - in > understanding the program, essentially. > > When I read "final int x = 5;" in Java, I don't have to think about it > anymore - it's 5. When I read "X = 5" in Python, it might be a constant, > but it might also be a misnomer, or something that used to be a constant, > or a class reassigned. I still have to watch whether it will be changed, > intentionally or accidentally, by this or other module. It does not even > communicate intention in an unambiguous way. Here I strongly disagree. The uppercase convention for things you really should not be assigning to has been in Python for ages and well understood. math.pi is the usual example brought up by people who want const/final, perhaps as a glaring exception to the rule. (Can math.PI be added to the standard library?) I think that adding const/let/final to Python would decrease readability and the ability to reason about the code for me, because now I would be reading foo += 1 and I could not easily decide whether this was wrong or not, because it might have been declared as immutable. Textual conventions are popular in many programming languages, not just Python, because they are easier to remember than individual type/attribute declarations. > In my opinion, the inability to control mutation in Python is the biggest > readability issue of the language (thank god monkey-patching is > discouraged). I think this is a separate issue to const/let/final. (And have you read PEP 351?) I don't miss simple scalar const/#define values in Python. In C/Java/Ada/... they are necessary to define array bounds, but everything in Python is dynamic and you can ask an array or whatever how big it is. In C/Ada/C#/... const protects again accidentally passing a value by reference/address and modifying it, but Python doesn't have pass by reference. No pointer aliasing in Python... As for monkey patching, or even just math.pi = 3.0 that is being unable to make dictionaries read-only. I agree, I get frustrated when I've accidentally misspelled an attribute name and Python just creates a new instance value instead of reporting an error. But this kind of dynamic object creation is so fundamental to how Python works, and also the source of so much power and flexibility, that I think it is worth living with the occasional glitch. I would not want to see Python allow module or class writers to declare "my code is perfect and not to be changed". But I would not mind if I could write code "I should not be changing this module/class/object , raise an exception if I do". -- cheers, Hugh Fisher

1 0

Star assignment in iterator way?
by Kirill Balunov Nov. 21, 2017

Nov. 21, 2017

Hello. Currently during star assignement the new list is created. What was the idea to produce new list instead of returning an iterator? It seems to me that returning an iterator more suited to the spirit of Python 3. There are three cases: 1. a,b,c,*d = something_iterable 2. *a,b,c,d = something_iterable 3. a,*b,c,d = something_iterable The first one is obvious. For the rest two we always need to iterate through entire iterable to achieve values for b,c,d (or c,d) binding. But this can be done more memory effiecient than currently (may be I'm wrong). And we can iterate in space of last three (or two) variables. Some rough (simplified) Python code: from itertools import islice, chain from collections import deque def good_star_exp(signature, seq): if signature.count('*') > 1: raise SyntaxError('two starred expressions in assignment') vrs = signature.split(',') idx_max = len(vrs) - 1 star_pos, = (i for i,v in enumerate(vrs) if '*' in v) #First case if star_pos == idx_max: head = islice(seq, idx_max) tail = islice(seq, idx_max, None) return chain(head, (tail,)) #Second case elif star_pos == 0: tail = deque(maxlen=idx_max) for seq_idx_max, v in enumerate(seq): tail.append(v) head = islice(seq, 0, seq_idx_max-(idx_max-1)) return chain([head], tail) #Third case else: head = islice(seq, star_pos) tail = deque(maxlen=(idx_max-star_pos)) for seq_idx_max, v in enumerate(seq): tail.append(v) mid = islice(seq, star_pos, seq_idx_max-(idx_max-2)) return chain(head, [mid], tail) ls = range(100000) a,b,c,d = good_star_exp('a,b,c,*d', ls) a,b,c,d = good_star_exp('*a,b,c,d', ls) a,b,c,d = good_star_exp('a,*b,c,d', ls) Of course this version has drawbacks (the first that come to mind): 1. Will *b see change if rhs is some muttable sequence? 2. Will *b one way iterator or somethong like range? But still it seems to me that the "iterator way" has more useful applications. With best regards, -gdg

5 8

Should Python have user-defined constants?
by Saeed Baig Nov. 21, 2017

Nov. 21, 2017

Hey guys I am thinking of perhaps writing a PEP to introduce user-defined constants to Python. Something along the lines of Swift’s “let” syntax (e.g. “let pi = 3.14”). Do you guys think it would be a good idea? Why or why not? Do you think there’s a better way to do it? I’d like to know what others think about this idea before making any formal submission (I’ve already posted this same question on python-list, but I just wanted to gauge opinion here too).

13 16

Rewriting the "roundrobin" recipe in the itertools documentation
by bunslow Nov. 21, 2017

Nov. 21, 2017

For taking values alternately from a series of iterables, there's two primary functions: builtin.zip itertools.zip_longest zip of course stops when the shortest iterable ends. zip_longest is generally a useful substitute for when you don't want the zip behavior, but it fills extra values in the blanks rather than just ignoring a finished iterator and moving on with the rest. This latter most use case is at least somewhat common, according to this[1] StackOverflow question (and other duplicates), in addition to the existence of the `roundrobin` recipe[2] in the itertools docs. The recipe satisfies this use case, and its code is repeated in the StackOverflow answer. However, it is remarkably unpythonic, in my opinion, which is one thing when such is necessary to achieve a goal, but for this functionality, such is most definitely *not* necessary. I'll paste the code here for quick reference: def roundrobin(*iterables): "roundrobin('ABC', 'D', 'EF') --> A D E B F C" pending = len(iterables) nexts = cycle(iter(it).__next__ for it in iterables) while pending: try: for next in nexts: yield next() except StopIteration: pending -= 1 nexts = cycle(islice(nexts, pending)) Things that strike me as unpythonic: 1) requiring the total number of input iterables 2) making gratuitous use of `next`, 3) using a while loop in code dealing with iterables, 4) combining loops, exceptions, and composed itertools functions in non-obvious ways that make control flow difficult to determine Now, I get it, looking at the "roughly equivalent to" code for zip_longest in the docs, there doesn't seem to be much way around it for generally similar goals, and as I said above, unpythonic is fine when necessary (practicality beats purity), but in this case, for being a "recipe" in the itertools docs, it should *make use* of the zip_longest which already does all the unpythonic stuff for you (though honestly I'm not convinced either that the zip_longest code in the docs is the most possible pythonic-ness). Instead, the following recipe (which I also submitted to the StackOverflow question, and which is generally similar to several other later answers, all remarking that they believe it's more pythonic) is much cleaner and more suited to demonstrating the power of itertools to new developers than the mess of a "recipe" pasted above. def roundrobin(*iters): "roundrobin('ABC', 'D', 'EF') --> A D E B F C" # Perhaps "flat_zip_nofill" is a better name, or something similar sentinel = object() for tup in it.zip_longest(*iters, fillvalue=sentinel): yield from (x for x in tup if x is not sentinel) In particular, this is just an extremely thin wrapper around zip_longest, whose primary purpose is to eliminate the otherwise-mandatory "fillvalues" that zip_longest requires to produce uniform-length tuples. It's also an excellent example of how to make best pythonic use of iterables in general, and itertools in particular, and as such a much better implementation to be demonstrated in documentation. I would thus advocate that the former recipe is replaced with the latter recipe, being much more pythonic, understandable, and useful for helping new developers acquire the style of python. (Using the common linguistics analogy: a dictionary and grammar for a spoken language may be enough to communicate, but we rely on a large body of literature -- fiction, research, poetry, etc -- as children to get that special flavor and most expressive taste to the language. The stdlib is no Shakespeare, but it and its docs still form an important part of the formative literature of the Python language.) I realize at the end of the day this is a pretty trivial and ultimately meaningless nit to pick, but I've never contributed before and have a variety of similar minor pain points in the docs/stdlib, and I'm trying to gauge 1) how well this sort of minor QoL improvement is wanted, and 2) even if it is wanted, am I going about it the right way. If the answers to both of these questions are positive regarding this particular case, then I'll look into making a BPO issue and pull request on GitHub, which IIUC is the standard path for contributions. Thank you for your consideration. ~~~~ [1]: https://stackoverflow.com/questions/3678869/ pythonic-way-to-combine-two-lists-in-an-alternating-fashion/ [2]: https://docs.python.org/3/library/itertools.html#itertools-recipes

9 13

Ignorable whitespaces in the re.VERBOSE mode
by Serhiy Storchaka Nov. 21, 2017

Nov. 21, 2017

Currently the re module ignores only 6 ASCII whitespaces in the re.VERBOSE mode: U+0009 CHARACTER TABULATION U+000A LINE FEED U+000B LINE TABULATION U+000C FORM FEED U+000D CARRIAGE RETURN U+0020 SPACE Perl ignores characters that Unicode calls "Pattern White Space" in the /x mode. It ignores additional 5 non-ASCII characters. U+0085 NEXT LINE U+200E LEFT-TO-RIGHT MARK U+200F RIGHT-TO-LEFT MARK U+2028 LINE SEPARATOR U+2029 PARAGRAPH SEPARATOR The regex module just ignores characters for which str.isspace() returns True. It ignores additional 20 non-ASCII whitespace characters, including characters U+001C..001F whose classification as whitespaces is questionable, but doesn't ignore LEFT-TO-RIGHT MARK and RIGHT-TO-LEFT MARK. U+001C [FILE SEPARATOR] U+001D [GROUP SEPARATOR] U+001E [RECORD SEPARATOR] U+001F [UNIT SEPARATOR] U+00A0 NO-BREAK SPACE U+1680 OGHAM SPACE MARK U+2000 EN QUAD U+2001 EM QUAD U+2002 EN SPACE U+2003 EM SPACE U+2004 THREE-PER-EM SPACE U+2005 FOUR-PER-EM SPACE U+2006 SIX-PER-EM SPACE U+2007 FIGURE SPACE U+2008 PUNCTUATION SPACE U+2009 THIN SPACE U+200A HAIR SPACE U+202F NARROW NO-BREAK SPACE U+205F MEDIUM MATHEMATICAL SPACE U+3000 IDEOGRAPHIC SPACE Is it worth to extend the set of ignored whitespaces to "Pattern Whitespaces"? Would it add any benefit? Or add confusion? Should this depend on the re.ASCII mode? Should the byte b'\x85' be ignorable in verbose bytes patterns? And there is a similar question about the Python parser. If Python uses Unicode definition for identifier, shouldn't it accept non-ASCII "Pattern Whitespaces" as whitespaces? There will be technical problems with supporting this, but are there any benefits? https://perldoc.perl.org/perlre.html https://www.unicode.org/reports/tr31/tr31-4.html#Pattern_Syntax https://unicode.org/L2/L2005/05012r-pattern.html

8 13

Adding a new function "zip_flat" to itertools (Re: Rewriting the "roundrobin" recipe in the itertools documentation)
by bunslow Nov. 21, 2017

Nov. 21, 2017

Dear all: thank you for your replies and thoughts, most especially Steve and Terry. I am more-or-less new to contributing to Python, so I wasn't sure that the bug tracker was the best way to start -- I was looking for a sanity check and received exactly what I wanted :) Thanks to the feedback here, the bug tracker issue will be much cleaner from the start. Regarding the meta discussion about my OP: yes it was long winded, detailed, pedantic and (to a certain extent) bombastic, but I was imaging the many possible responses to the suggestion (and suggested replacement) the I felt I should explain where I was coming from. Even though there was a lot of disagreement about how I described the current recipe as not very Pythonic, I'm very glad for all the perspectives and lessons I got from reading the ensuing discussions. Now, having had some time to think, I've come up with further thoughts on the topic. In particular, I was going to create a new bug, and wrote several paragraphs on the topic summarizing this thread, and my thoughts. I'll paste those paragraphs here: """ To summarize, I found the current implementation of the "roundrobin" function difficult to understand, and proposed a much simpler solution that, while producing correct results, isn't quite "correct" in the sense that it glosses over a detail before "manually" correcting the detail at the end, and as such is prone to severe inefficiency in extreme cases. There were a smattering of comments from several people indicating that they liked the simpler recipe better, despite its performance drawbacks. Terry Reedy proposed a slightly rewritten version of the current recipe, which implements the correct algorithm (without glossing over and manually correcting the details). Although I have since changed my perceptions of the original, now understanding how it works, the rewritten version from Terry was clearer enough that I was able to understand *it* where I could not previously understand the original. (My newfound understanding of the original is largely derived from making sense of the rewritten version, which properly clued me in to what cycle and islice actually do. Either way, the current recipe can certainly be improved. Although I now find the original and its rewrite to be algorithmically clean and correct (even if the code and inline comments can be improved), the StackOverflow question ( https://stackoverflow.com/questions/3678869/pythonic-way-to-combine-two-lis…) that originally got me thinking about the problem demonstrates that the algorithmically clean way is *not* obvious at all to people who aren't very familiar with the itertools module -- which is the large majority of people who use Python (even if that's a very small fraction of the people reading this bug). The second from top answer is the answer which references the recipe in the docs, but as my own first post to python-ideas demonstrates, the (large?) majority of python users aren't familiar enough with the itertools module to be able to understand that recipe, and I also believe many people were looking for one or two liners to use in their own code without a function call. Further confusion on the overall topic is the lack of a clear name -- "roundrobin", "alternate", "interleave", "merge", and variations and others. """ Having completed those, I found a roughly duplicate StackOverflow question to the one from my OP: https://stackoverflow.com/questions/243865/how-do-i-merge-two-python-iterat… Besides emphasizing my points about not having even a clear name for the topic, a desire for one liners, mass confusion around the issue (especially regarding flattening zip [which terminates on the shortest input, a hidden gotcha], zip_longest [someone else found the same solution as me and others in this op), and all around failure to generate anything even resembling a consensus on the topic, I also found this answer: https://stackoverflow.com/questions/243865/how-do-i-merge-two-python-iterat… which proposes a solution that is both more correct and efficient than the zip_longest-with-sentinels, and also noticeably more readable than either the original doc recipe or even Terry's cleaned up replacement of it. Given this variety of problems with the issue, I now think that -- while updating the itertools recipe is certainly better than nothing -- the better thing to do might be to just add a new function to itertools called "zip_flat" which solves this problem. In addition to answering the stack overflow questions with ongoing debate about efficiency, correctness, and pythonicity (pythonicness?), it would also help to greatly clarify the naming issue as well. (Sidenote: whoever came up with "zip" as the name for the builtin was quite creative. It's a remarkably short and descriptive.) What are the sentiments of readers here? If positive, I'll create an issue on BPO about zip_flat (rather than just improving the docs recipe). (Sorry Steve for bringing this back to -ideas! At least this time I'm proposing an addition to the language itself! :) Thanks for your consideration, Bill On Thu, Nov 16, 2017 at 5:06 PM, Steven D'Aprano <steve(a)pearwood.info> wrote: > On Thu, Nov 16, 2017 at 02:56:29PM -0500, Terry Reedy wrote: > > > >3) using a while loop in code dealing with iterables, > > > > I agree that this is not necessary, and give a replacement below. > > The OP isn't just saying that its unnecessary in this case, but that its > unPythonic to ever use a while loop in code dealing with iterables. I > disagree with that stronger statement. > > > > >def roundrobin(*iters): > > > "roundrobin('ABC', 'D', 'EF') --> A D E B F C" > > > # Perhaps "flat_zip_nofill" is a better name, or something similar > > > sentinel = object() > > > for tup in it.zip_longest(*iters, fillvalue=sentinel): > > > yield from (x for x in tup if x is not sentinel) > > > > This adds and then deletes grouping and fill values that are not wanted. > > To me, this is an 'algorithm smell'. One of the principles of > > algorithm design is to avoid unnecessary calculations. For an edge case > > such as roundrobin(1000000*'a', ''), the above mostly does unnecessary > work. > > Its a recipe, not a tuned and optimized piece of production code. > > And if you're going to criticise code on the basis of efficiency, then I > would hope you've actually profiled the code first. Because it isn't > clear to me at all that what you call "unnecessary work" is more > expensive than re-writing the recipe using a more complex algorithm with > calls to cycle and islice. > > But I'm not here to nit-pick your recipe over the earlier ones. > > > [...] > > Since I have a competing 'improvement', I would hold off on a PR until > > Raymond Hettinger, the itertools author, comments. > > Raise a doc issue on the tracker, and take the discussion there. I think > that this is too minor an issue to need long argument on the list. > Besides, it's not really on-topic as such -- it isn't about a change to > the language. Its about an implementation change to a recipe in the > docs. > > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas(a)python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ >

5 8

Consider (one day) adding an inheritance order class precedence mechanism
by Neil Girdhar Nov. 21, 2017

Nov. 21, 2017

Sometimes I get MRO failures when defining classes. For example, if R < E, C B < S, E S < C Z < B, R Then Z cannot be instantiated because C precedes E in B and E precedes C in R. The problem is that when the inheritance graph was topologically-sorted in B, the order was S, C, E. It could just as easily have been S, E, C. So, one solution is to add C explicitly to B's base class list, but this is annoying because B might not care about C (it's an implementation detail of S). It also means that if the hierarchy changes, a lot of these added extra base classes need to be fixed. I propose adding a magic member to classes: __precedes__ that is a list of classes. So, the fix for the above problem would be to define E as follows: class E: from whatever import C __precedes__ = [C] Then, when topologically-sorting (so-called linearizing) the ancestor classes, Python can try to ensure that E precedes C when defining B. Best, Neil

7 21

A proliferation of (un-)Pythonically programmatic pragmas
by Barry Warsaw Nov. 20, 2017

Nov. 20, 2017

I love many of the ancillary tools that help improve the quality of my Python code, including flake8, coverage, and mypy. Each of these usually produces some great diagnostics, but none of them are perfect, so they also produce false positives that have to be suppressed. Each of these tools supports "pragmas", i.e. comments you can add to a line of code to suppress a warning. E.g. with flake8 I can say: import readline, rlcompleter # noqa: F401 to suppress the flake8 warnings about unused imports. These modules work by side-effect, which of course a static analyzer can't know. Similarly, when mypy complains about a line it's confused on, I can add a comment to suppress the useless warning: try: from pathlib import Path, PurePath except ImportError: from pathlib2 import Path, PurePath # type: ignore And when I want to boost coverage, but I know that some lines aren't covered under some versions of Python, I can do something like: self.send_error(HTTPStatus.NOT_FOUND) # pragma: nocover These are all well and good until I have to *combine* suppressions. E.g. in the pathlib2 pragma to mypy, I also get a flake8 warning and I've tried just about every combination of pragma comment I can think of, but I've not been able to make both tools happy. I've resorted to refactoring the code into an entire module for flake8 to ignore and added: # flake8: noqa to suppress all warnings in the file. I actually have hit on a few places where I need to suppress warnings from all three tools on the same line, and I basically can't do it. The specifics aren't as important as the general use case: multiple tools competing for the same valuable real-estate. I have no ideas how to improve the situation, and of course any solution would involve some coordination between all of these tools, but it's beginning to feel like a losing battle. Is there a better way? Cheers, -Barry

13 21

Terminology of types / typing [was: PEP 560 (second post)]
by Koos Zevenhoven Nov. 15, 2017

Nov. 15, 2017

Here are some thoughts––maybe even a proposal––for type-related terminology, because clear terminology makes discussion and reasoning easier, and helps avoid errors. (And related to the PEP 560 thread, the question of what should go into class attributes like __bases__). Terminology regarding types is confusing enough, and of all the terminology I've seen, I like these most: * concrete type: something that concretely implements data storage or functionality. Usually this is a normal class. * abstract type: an assumption or set of assumptions made about an instance. While both kinds can (and probably should) have a corresponding runtime representation, there's not that much that we can currently assume about the second kind––abstract types. They could be almost anything. Anyway, regarding PEP 560, in my speculated naming, the __bases__ attribute of a class would contain both concrete and abstract bases. Those with any concrete method implementations should go in the mro. But then there's the problem that "abstract base classes" may contain both concrete and abstract methods. What do we call such a "type"? Maybe we have both "concrete" and "strictly concrete" types. Perhaps we also have both "abstract" and "strictly abstract" types. An ABC with some concrete default implementations might then be both a concrete type and an abstract type. Note that in the above bullet point "definition" of concrete type, I intentionally left out the requirement that the type can be instantiated. The other two bullet points are: * strictly concrete type: a concrete type that is not abstract––it concretely implements everything that it represents / describes. This is almost always a normal class, so it might be also known as "class". * strictly abstract type: an abstract type that is not concrete––it does not implement any functionality or storage. There might be a way to improve terminology from this, but I feel that what I sketched here is usable but still not very ambiguous. ––Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven +

2 3

Greetings from a lurker
by Robert Nov. 15, 2017

Nov. 15, 2017

Hi, I've been lurking for quite a time and just wanted to let you know I'm out here. Thanks for making Python the best! Robert Kaplan

1 0

Modules as global namespaces rather than dicts
by Neil Schemenauer Nov. 15, 2017

Nov. 15, 2017

This is an idea I have been playing with and seems to hold some promise. I think we should use a module instance as the standard global namespace rather than directly using its dict. I have a prototype version of CPython that does this, not working 100% yet though. Major changes: - In the frameobject, remove f_builtins and f_globals, add f_namespace that refers to the module. Make f_globals and f_builtins into properties. - Change ceval to use f_namespace, rather than carrying around globals and builtins. Change functions that take globals as a dict so they also accept a module object. - Change the module object to keep track of the builtins for it. - Change funcobject (e.g. PyFunction_NewWithQualName) to accept 'globals' as a module object - When given a dict and we now expect a module object, create an anonymous module to wrap it. This part is a bit tricky due to reference cycles and speed requirements. However, my hope is that if the internals of CPython can be made to pass modules instead of dicts around, these anonymous modules will become a rare case and so won't matter if they are a little slower. So, what is the purpose of all this trouble? - I believe quite a lot of Python internals can be simpler. For example, importlib is complicated by the fact that a dict is passed around when most of the logic would prefer to have the module. Grubbing in sys.modules to lookup the module object is ugly. The expression "exec(code, module)" is elegant to me. - I believe it will be possible to make global variable access work similar to fast locals. I.e. have each module contain a indexed list of global names, and use an array lookup rather than a dict lookup in the normal case. For backwards compatibility, my idea is to keep a flag on the module object noting if fast global behavior is okay. If someone grabs the module dict, e.g. using module.__dict__ , vars() or frame.f_globals then we clear the flag and revert to the slow dict behavior. Also, if a piece of code is executed in a different namespace, we have to revert to the slow case. That does not seem so hard to do. - I want to have properties for module globals and have them work from within the module. I.e. changing something from a global variable to a property should not require changes to other module code. I.e. LOAD_GLOBAL should trigger a property get. Regards, Neil

4 4

PEP 560 (second post)
by Ivan Levkivskyi Nov. 15, 2017

Nov. 15, 2017

Previously I posted PEP 560 two weeks ago, while several other PEPs were also posted, so it didn't get much of attention. Here I post the PEP 560 again, now including the full text for convenience of commenting. -- Ivan ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ PEP: 560 Title: Core support for generic types Author: Ivan Levkivskyi <levkivskyi(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 03-Sep-2017 Python-Version: 3.7 Post-History: 09-Sep-2017 Abstract ======== Initially PEP 484 was designed in such way that it would not introduce *any* changes to the core CPython interpreter. Now type hints and the ``typing`` module are extensively used by the community, e.g. PEP 526 and PEP 557 extend the usage of type hints, and the backport of ``typing`` on PyPI has 1M downloads/month. Therefore, this restriction can be removed. It is proposed to add two special methods ``__class_getitem__`` and ``__subclass_base__`` to the core CPython for better support of generic types. Rationale ========= The restriction to not modify the core CPython interpreter lead to some design decisions that became questionable when the ``typing`` module started to be widely used. There are three main points of concerns: performance of the ``typing`` module, metaclass conflicts, and the large number of hacks currently used in ``typing``. Performance: ------------ The ``typing`` module is one of the heaviest and slowest modules in the standard library even with all the optimizations made. Mainly this is because subscripted generic types (see PEP 484 for definition of terms used in this PEP) are class objects (see also [1]_). The three main ways how the performance can be improved with the help of the proposed special methods: - Creation of generic classes is slow since the ``GenericMeta.__new__`` is very slow; we will not need it anymore. - Very long MROs for generic classes will be twice shorter; they are present because we duplicate the ``collections.abc`` inheritance chain in ``typing``. - Time of instantiation of generic classes will be improved (this is minor however). Metaclass conflicts: -------------------- All generic types are instances of ``GenericMeta``, so if a user uses a custom metaclass, then it is hard to make a corresponding class generic. This is particularly hard for library classes that a user doesn't control. A workaround is to always mix-in ``GenericMeta``:: class AdHocMeta(GenericMeta, LibraryMeta): pass class UserClass(LibraryBase, Generic[T], metaclass=AdHocMeta): ... but this is not always practical or even possible. With the help of the proposed special attributes the ``GenericMeta`` metaclass will not be needed. Hacks and bugs that will be removed by this proposal: ----------------------------------------------------- - ``_generic_new`` hack that exists since ``__init__`` is not called on instances with a type differing form the type whose ``__new__`` was called, ``C[int]().__class__ is C``. - ``_next_in_mro`` speed hack will be not necessary since subscription will not create new classes. - Ugly ``sys._getframe`` hack, this one is particularly nasty, since it looks like we can't remove it without changes outside ``typing``. - Currently generics do dangerous things with private ABC caches to fix large memory consumption that grows at least as O(N\ :sup:`2`), see [2]_. This point is also important because it was recently proposed to re-implement ``ABCMeta`` in C. - Problems with sharing attributes between subscripted generics, see [3]_. Current solution already uses ``__getattr__`` and ``__setattr__``, but it is still incomplete, and solving this without the current proposal will be hard and will need ``__getattribute__``. - ``_no_slots_copy`` hack, where we clean-up the class dictionary on every subscription thus allowing generics with ``__slots__``. - General complexity of the ``typing`` module, the new proposal will not only allow to remove the above mentioned hacks/bugs, but also simplify the implementation, so that it will be easier to maintain. Specification ============= The idea of ``__class_getitem__`` is simple: it is an exact analog of ``__getitem__`` with an exception that it is called on a class that defines it, not on its instances, this allows us to avoid ``GenericMeta.__getitem__`` for things like ``Iterable[int]``. The ``__class_getitem__`` is automatically a class method and does not require ``@classmethod`` decorator (similar to ``__init_subclass__``) and is inherited like normal attributes. For example:: class MyList: def __getitem__(self, index): return index + 1 def __class_getitem__(cls, item): return f"{cls.__name__}[{item.__name__}]" class MyOtherList(MyList): pass assert MyList()[0] == 1 assert MyList[int] == "MyList[int]" assert MyOtherList()[0] == 1 assert MyOtherList[int] == "MyOtherList[int]" Note that this method is used as a fallback, so if a metaclass defines ``__getitem__``, then that will have the priority. If an object that is not a class object appears in the bases of a class definition, the ``__subclass_base__`` is searched on it. If found, it is called with the original tuple of bases as an argument. If the result of the call is not ``None``, then it is substituted instead of this object. Otherwise (if the result is ``None``), the base is just removed. This is necessary to avoid inconsistent MRO errors, that are currently prevented by manipulations in ``GenericMeta.__new__``. After creating the class, the original bases are saved in ``__orig_bases__`` (currently this is also done by the metaclass). NOTE: These two method names are reserved for exclusive use by the ``typing`` module and the generic types machinery, and any other use is strongly discouraged. The reference implementation (with tests) can be found in [4]_, the proposal was originally posted and discussed on the ``typing`` tracker, see [5]_. Backwards compatibility and impact on users who don't use ``typing``: ===================================================================== This proposal may break code that currently uses the names ``__class_getitem__`` and ``__subclass_base__``. This proposal will support almost complete backwards compatibility with the current public generic types API; moreover the ``typing`` module is still provisional. The only two exceptions are that currently ``issubclass(List[int], List)`` returns True, with this proposal it will raise ``TypeError``. Also ``issubclass(collections.abc.Iterable, typing.Iterable)`` will return ``False``, which is probably desirable, since currently we have a (virtual) inheritance cycle between these two classes. With the reference implementation I measured negligible performance effects (under 1% on a micro-benchmark) for regular (non-generic) classes. References ========== .. [1] Discussion following Mark Shannon's presentation at Language Summit (https://github.com/python/typing/issues/432) .. [2] Pull Request to implement shared generic ABC caches (https://github.com/python/typing/pull/383) .. [3] An old bug with setting/accessing attributes on generic types (https://github.com/python/typing/issues/392) .. [4] The reference implementation (https://github.com/ilevkivskyi/cpython/pull/2/files) .. [5] Original proposal (https://github.com/python/typing/issues/468) Copyright ========= This document has been placed in the public domain.

6 18

Re: [Python-ideas] Looking for input to help with the pip situation
by Michel Desmoulin Nov. 15, 2017

Nov. 15, 2017

General summary so far ####################### This debate has been very civil so far and quite productive, so thanks to everyone involved. Several issues and solution proposals have been mentioned, so I will make a summary here. Issue 1: running Python is a different command different OS when several versions are installed ================================================================================================== Proposal A: ----------- Suffix Python executable on Windows like on Unix, so that people will type pythonX.X if they want a specify version. Pros: easy and discoverable. Cons: you need a lot of stuff in the system path. Proposal B: ------------ Provide the "py" command on Unix, so that people will type "py -x.x" Pros: no more issues with system path. Just add this command, and register python installations to be listed by "py". Cons: very few people know about "py", even on windows. This would need lots of communication. Harder to change Mac and majors Linux distros setup than just the next windows installer. Issue 2: there is no one and only one obvious way to do venv / pip =================================================================== Because pip and venv are tied to a Python version, there are numerous ways to call them. Just for pip, you have "pip", "py -x.x -m pip", "pythonx.x -m pip" and "pipx". Proposal A: ----------- Decide on one form, and promote it everywhere. Pros: simple. Cons: long and tedious process, with a huge inertia before all 3rd party to it. Would need a "PEP 8 for Python command". Requires "issue 1" to be solved before. Proposal B: ------------ Provide a pipenv-like behavior pipenv is a tool created by the author of "requests" that automatically "do the right thing" by default. The first time you install a package, it creates a virtualenv for the current directory (but don't store it in the current directory). It will then install all packages in there. "pipenv shell" starts a shell in the virtualenv. Dependencies are automatically dumped in a requirement files separating dev and prod. Pros: bypass the multiple installers issue by prompting you what version you wish the first time it creates a virtualenv. Solve the problem of admin rights and system path. Cons: lots of work. Either a new tool or breaking compat with pip. If debian didn't install pip by default, this may not help. Issue 3: pip and virtualenv are not available everywhere =================================================================== Proposal A: ------------ Make a mock "pip" commands for when it's not installed. venv already does that in some cases (see attachements), alhought it does give you the wrong command so it should be fixed. The gist of it: if the users try to run pip install while it's missing, it prints a warning explaining it's not installed and the command you can run to solve the problem plus a link to more information in the documentation. Pros: low friction to implement. Cons: requires to put up a hook somewhere so that each linux distro can input "the proper commands to use" for their system when they package python. More a workaround than a solution. Proposal B: ------------ Make pip and venv part of the standard and request debian that they provide it. Pros: straight forward. Cons: holy war in the making. ................. Can I have +1 and -1 from each of you on each point ? Should I start a new thread for each of them ? Favorite personnal combination ================================ 1 - So far, I like "providing py on unix" better, but I think it's too hard to do. So provide suffix on windows seems more achievable. So +1 for having pythonX.X on windows. 2 - Decide on one form, and promote it everywhere is again the simplest IMO. I love pipenv but the strings to pull to get something like this means it probably would not happen. If we stick to suffix for issue 1, this mean we can promote: pythonX.X -m pip pythonX.X -m venv Everywhere. In the docs. In the tutorials. In teaching classes. This will help a lot to remove all confusions and make the Python start up experience way easier. 3 - getting angry debian devs is probably not wise. The "mock" pip/venv command also plays very well with the 2 other solutions. It means we can promote everywhere: pythonX.X -m cmd It will work most of the time. And when it doesn't, give clear contextual steps on

5 6

PEP 562
by Ivan Levkivskyi Nov. 14, 2017

Nov. 14, 2017

I have written a short PEP as a complement/alternative to PEP 549. I will be grateful for comments and suggestions. The PEP should appear online soon. -- Ivan *********************************************************** PEP: 562 Title: Module __getattr__ Author: Ivan Levkivskyi <levkivskyi(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 09-Sep-2017 Python-Version: 3.7 Post-History: 09-Sep-2017 Abstract ======== It is proposed to support ``__getattr__`` function defined on modules to provide basic customization of module attribute access. Rationale ========= It is sometimes convenient to customize or otherwise have control over access to module attributes. A typical example is managing deprecation warnings. Typical workarounds are assigning ``__class__`` of a module object to a custom subclass of ``types.ModuleType`` or substituting ``sys.modules`` item with a custom wrapper instance. It would be convenient to simplify this procedure by recognizing ``__getattr__`` defined directly in a module that would act like a normal ``__getattr__`` method, except that it will be defined on module *instances*. For example:: # lib.py from warnings import warn deprecated_names = ["old_function", ...] def _deprecated_old_function(arg, other): ... def __getattr__(name): if name in deprecated_names: warn(f"{name} is deprecated", DeprecationWarning) return globals()[f"_deprecated_{name}"] raise AttributeError(f"module {__name__} has no attribute {name}") # main.py from lib import old_function # Works, but emits the warning There is a related proposal PEP 549 that proposes to support instance properties for a similar functionality. The difference is this PEP proposes a faster and simpler mechanism, but provides more basic customization. An additional motivation for this proposal is that PEP 484 already defines the use of module ``__getattr__`` for this purpose in Python stub files, see [1]_. Specification ============= The ``__getattr__`` function at the module level should accept one argument which is a name of an attribute and return the computed value or raise an ``AttributeError``:: def __getattr__(name: str) -> Any: ... This function will be called only if ``name`` is not found in the module through the normal attribute lookup. The reference implementation for this PEP can be found in [2]_. Backwards compatibility and impact on performance ================================================= This PEP may break code that uses module level (global) name ``__getattr__``. The performance implications of this PEP are minimal, since ``__getattr__`` is called only for missing attributes. References ========== .. [1] PEP 484 section about ``__getattr__`` in stub files (https://www.python.org/dev/peps/pep-0484/#stub-files) .. [2] The reference implementation (https://github.com/ilevkivskyi/cpython/pull/3/files) Copyright ========= This document has been placed in the public domain.

10 17

Complete typing.Union with the rest of set-like functions
by Ilya Kulakov Nov. 13, 2017

Nov. 13, 2017

I needed to declare a type that would mean "Any but None" and didn't find how. Current implementation allows to expand set of allowed types, not narrow it. Perhaps typing needs the rest of set operators in addition to Union? Best Regards, Ilya Kulakov

2 1

Any chance on (slowly) deprecating `eval` and `exec` as builtins?
by אלעזר Nov. 9, 2017

Nov. 9, 2017

Hi, The dangers of eval and exec are obvious and well known to advanced users, but the availability as built-in functions makes it too tempting for beginners or even medium-level programmers. You can see questions about these function pretty often in stackoverflow (roughly once a day <https://stackoverflow.com/search?tab=newest&q=eval%20python>, though sometimes the uses are legitimate). Maybe we could start a ten-year process of deprecating the use of `builtins.eval` (in the docs, and then with warnings)? `builtins.eval` will be a wrapper to the real evaluation function, moved to `unsafe.eval` or something obvious like that, so all you need to do to port your code is to add `from unsafe import unsafe_eval as eval, unsafe_exec as exec` at the top of the file; it will be a nice warning to the reader. The fact that it is a wrapper will slightly slow it down and make the stack traces noisier - both are good things, IMO. Also, it is unfortunate that `ast.literal_eval` is less accessible than `builtins.eval`. Giving it an alias in builtins might make it easier for programmers (and less scary - "ast" might sound like I need a PhD to use it). What do you think? Elazar

10 15

Possible enhancement to typing
by Steve Barnes Nov. 8, 2017

Nov. 8, 2017

The discussions on moving typing to a standalone library prompted me to take a look at the documentation for typing which in turn lead to an idea that should be reasonably simple to implement but greatly add value to typing for some applications. If a group of iterators were to be added to the typing module it would be reasonably simple to automatically add and assert to any decorated modules to ensure that such modules were always called with the documented types. I am thinking of decorators such as: - @typing.assert_params_mismatch - this would provide a wrapper that had auto-generated asserts that all the parameters were of designated types. - @typing.debug_assert_params_mismatch - this would provide a wrapper that had auto-generated asserts that all the parameters were of designated types only if a DEBUG environmental variable was set or similar. - @typing.warn_params_mismatch - this would provide a wrapper that had auto-generated warnings issued to stderr if any of the parameters were of not of the designated types. - @typing.coerce_params - this would add a wrapper to attempt to convert any mismatched parameters to the specified type(s). I also think that this might increase the uptake of typing by giving some clear benefits outside of documentation and static type checking. Pushing this out for comment before considering a more formal, (PEP), proposal. -- Steve (Gadget) Barnes Any opinions in this message are my personal opinions and do not reflect those of my employer. --- This email has been checked for viruses by AVG. http://www.avg.com

4 8

Fwd: Any chance on (slowly) deprecating `eval` and `exec` as builtins?
by Zero Piraeus Nov. 8, 2017

Nov. 8, 2017

: On 7 November 2017 at 18:53, Guido van Rossum <guido(a)python.org> wrote: > I find it dubious to claim that these functions are dangerous to beginners. > The dangers are related to attacks on servers that are exposed to the > internet and beginners have no business running servers. I can't be the only one whose experience is that it's very often those with the least business running servers that actually do run them, surely? +1 -[]z.

1 0

Moving typing out of the stdlib in Python 3.7?
by Guido van Rossum Nov. 7, 2017

Nov. 7, 2017

[A copy from https://github.com/python/typing/issues/495 to get more people's attention to this issue.] I'm wondering if we should remove typing from the stdlib. Now's the time to think about this, as the feature freeze for 3.7 is about 12 weeks away. Cons: - People have to depend on a PyPI package to use typing (but they do anyway for typing_extensions) - It's a backward incompatibility for users of Python 3.5 and 3.6 (but the typing module was always provisional) Pros: - The typing module can evolve much faster outside the stdlib - We could get rid of typing_extensions (and maybe even mypy_extensions) If we don't do this I worry that we're entering a period where many new typesystem features end up in typing_extensions and users will be confused about which items are in typing and which in typing_extensions (not to mention mypy_extensions). Anything new to be added to typing (e.g. Const, Final, Literal, or changing ABCs to Protocols) would have to be added to typing_extensions instead, and users would be confused about which features exist in which module. Moving typing out of the stdlib can make things potentially simpler, at the cost of an extra pip install (but they'll need one anyway for mypy). Thoughts? -- --Guido van Rossum (python.org/~guido)

14 33

Proposal to change Python version release cycle
by tds333＠mailbox.org Nov. 7, 2017

Nov. 7, 2017

Hello, one of my long standing ideas to improve Python is to adjust the release cycle and version number handling. In short, to simplify it. This is the first draft of the idea: Proposal to change Python version release cycle =============================================== Goal ---- Increment the major version number more frequently for every new language feature release. Change Python feature version counting from "major.minor" to "major". This is simpler, better to handle compatibility and allows adding new library features more frequently. The current Python version scheme is Major.Minor.Patch this will be kept but the major version number will be incremented faster and for every language feature release as it is now for the minor number. For example today the minor version number is increased for every big release done in 1,5 year cycle. In this new language features with new keywords can appear, new libraries can be included and other C level API changes can happen. There backward compatible feature improvements and new features also backward incompatible can happen. I suggest to change this to increment the major version for every new release of the 1,5 year cycle. And allow new Python standard library backward compatible changes for every minor release cycle every 6 months. Start with this from Python version 4 on. Benefit ------- Clear separation of new language features and C API changes and new backward compatible feature additions. So for a major version on Python level new keywords can be added, build ins can be added or removed and the C API can be changed (new features or other incompatible changes). For a minor version new standard libraries can be added and API can be improved in a backward compatible way. This can be done more frequently for example in a 0,5 year cycle. Allowing the Python library part to improve in a higher rate. For a patch version only none breaking bug and security fixes are allowed. (Same as it is now, or even a little bit stricter) With this change there is a clear separation of new language Features which are also tracked by other Python implementations and new library features. This can be seen very easy by only looking at the Python major version. The C API level is included in this standard and is not allowed to change in a minor version as it is now. This is because booth the Python language and the C API of CPython implementation is a standard and tracked by other implementations. For them it is easy to say I am compatible to Version 4. For everyone in computer business it is clear what is meant. The Python standard library can be improved in a higher rate because backward compatible feature changes are allowed in a higher rate for minor versions. Also provisional libraries can be improved in a faster rate for every minor version. (for example every 6 months instead of 1,5 years) For Unix in the Python executable the major version can be appended and this specifies the Python language standard used. This is done already but most of the time only EXECUTABLEmajor.minor is useful. Also more complicated to handle as a simple Number. Can be problematic on some platforms because of the dot ".". Python major version every 1,5 years (same as it is now for major.minor). Python minor version every 6 months (to allow faster standard library changes). Python patch version as needed (to fix only bugs and security related stuff) This can solve the problem of adding something as a standard library and then this is the dead of the library because it cannot be improved any further in a higher update rate. The Python standard library should not be a dead end for a library after inclusion. This allows Python code to improve faster than the C code. This is needed to include new features faster and because we have more people capable in Python programming than for C in the Python community. On C level the compatibility distinction is simple only track the major version and no longer a mixture between major and major.minor compatibility. Risk ---- Minimal because to track features the major.minor version already has to be tracked. It is a change some assumptions about Python versions are at this point no longer valid. On C level external libraries and wrappers must adopt it and changes are needed. Open questions -------------- The exact release cycle for minor, can also be every 4 months if needed. Clear separation what is part of the language standard and what is part of the C API. Some modules, for example "sys" can also be part of the language Standard and should not change between minor versions. The difference between the stable C API and that for every minor version can be removed. There is only one C API guarantee for a major version and it does not change for a minor version. None goals ---------- Change the release cycle of 1,5 years for new big releases. Change the versioning scheme to something other than "major.minor.patch". New big ground breaking changes to everything. To complicate something. Regards, Wolfgang

16 32

install pip packages from Python prompt
by Stephan Houben Nov. 6, 2017

Nov. 6, 2017

Hi all, Here is in somewhat more detail my earlier proposal for having in the interactive Python interpreter a `pip` function to install packages from Pypi. Motivation: it appears to me that there is a category of newbies for which "open a shell and do `pip whatever`" is a bit too much. It would, in my opinion, simplify things a bit if they could just copy-and-paste some text into the Python interpreter and have some packages from pip installed. That would simplify instructions on how to install package xyz, without going into the vagaries of how to open a shell on various platforms, and how to get to the right pip executable. I think this could be as simple as: def pip(args): import sys import subprocess subprocess.check_call([sys.executable, "-m", "pip"] + args.split()) print("Please re-start Python now to use installed or upgraded packages.") Note that I added the final message about restarting the interpreter as a low-tech solution to the problem of packages being already imported in the current Python session. I would imagine that the author of package xyz would then put on their webpage something like: To use, enter in your Python interpreter: pip("install xyz --user") As another example, consider prof. Baldwin from Woolamaloo university who teaches a course "Introductory Python programming for Sheep Shavers". In his course material, he instructs his students to execute the following line in their Python interpreter. pip("install woolamaloo-sheepshavers-goodies --user") which will install a package which will in turn, as dependencies, pull in a number of packages which are relevant for sheep shaving but which have nevertheless irresponsibly been left outside the stdlib. Stephan

18 52

array.array.index() and Sequence ABC
by Niki Spahiev Nov. 6, 2017

Nov. 6, 2017

Hello, I noticed that array.index() method takes single argument in python 3.6, while Sequence protocol specifies 2 optional arguments after the first. example from list: L.index(value, [start, [stop]]) -> integer -- return first index of value. I propose adding start and stop arguments to array.index() Regards, Niki

2 1

About the efficiency of range()
by Juancarlo Añez Nov. 5, 2017

Nov. 5, 2017

I found this interesting: https://stackoverflow.com/a/46996392 -- Juancarlo *Añez*

3 2

Membership of infinite iterators
by Jason Campbell Nov. 3, 2017

Nov. 3, 2017

I recently came across a bug where checking negative membership (__contains__ returns False) of an infinite iterator will freeze the program. It may seem a bit obvious, but you can check membership in range for example without iterating over the entire range. `int(1e100) in range(int(1e101))` on my machine takes about 1.5us `int(1e7) in itertools.count()` on my machine takes about 250ms (and gets worse quite quickly). Any membership test on the infinite iterators that is negative will freeze the program as stated earlier, which is odd to me. itertools.count can use the same math as range itertools.cycle could use membership from the underlying iterable itertools.repeat is even easier, just compare to the repeatable element Does anyone else think this would be useful?

17 63

Re: [Python-ideas] Add single() to itertools
by Ivan Pozdeev Nov. 2, 2017

Nov. 2, 2017

On 30.10.2017 9:29, python-ideas-request(a)python.org wrote: > If I have understood your use-case, you have a function that returns a > list of results (or possibly an iterator, or a tuple, or some other > sequence): > > print(search(haystack, needle)) > # prints ['bronze needle', 'gold needle', 'silver needle'] > > There are times you expect there to be a single result, and if there are > multiple results, that is considered an error. Am I correct so far? Correct. > If so, then sequence unpacking is your friend: > > result, = search(haystack, needle) > > <...> > > I *think* this will solve your problem. > > If not, can you please explain what "single()" is supposed to do, why it > belongs in itertools, and show an example of how it will work. That works. Too arcane in my book though (and others' too according to https://stackoverflow.com/a/473337/648265), and the error messages are cryptic in this use case. It also cannot be a part of an expression, unlike next(). The initial post on the above link summarizes the suggested implementation pretty well. -- Regards, Ivan

9 11

Defining an easily installable "Recommended baseline package set"
by Nick Coghlan Nov. 2, 2017

Nov. 2, 2017

Over on python-dev, the question of recommending MRAB's "regex" module over the standard library's "re" module for more advanced regular expressions recently came up again. Because of various logistical issues and backwards compatibility risks, it's highly unlikely that we'll ever be able to swap out the current _sre based re module implementation in the standard library for an implementation based on the regex module. At the same time, it would be beneficial to have a way to offer an even stronger recommendation to redistributors that we think full-featured general purpose Python scripting environments should offer the regex module as an opt-in alternative to the baseline re feature set, since that would also help with other currently difficult cases like the requests module. What I'm thinking is that we could make some relatively simple additions to the `ensurepip` and `venv` modules to help with this: 1. Add a ensurepip.RECOMMENDED_PACKAGES mapping keyed by standard library module names containing dependency specifiers for recommended third party packages for particular tasks (e.g. "regex" as an enhanced alternative to "re", "requests" as an enhanced HTTPS-centric alternative to "urllib.request") 2. Add a new `install_recommended` boolean flag to ensurepip.bootstrap 3. Add a corresponding `--install-recommended flag to the `python -m ensurepip` CLI 4. Add a corresponding `--install-recommended flag to the `python -m venv` CLI (when combined with `--without-pip`, this would run pip directly from the bundled wheel file to do the installations) We'd also need either a new informational PEP or else a section in the developer guide to say that the contents of `ensurepip.RECOMMENDED_PACKAGES` are up to the individual module maintainers (hence keying the mapping by standard library module name, rather than having a single flat list for the entire standard library). For redistributors with weak dependency support, these reference interpreter level recommendations could become redistributor level recommendations. Redistributors without weak dependency support could still make a distinction between "default" installations (which would include them) and "minimal" installations (which would exclude them). Folks writing scripts and example code for independent distribution (i.e. no explicitly declared dependencies) could then choose between relying on just the standard library (as now), or on the standard library plus independently versioned recommended packages. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

13 39

Hello from a new lurker
by brent bejot Nov. 1, 2017

Nov. 1, 2017

Hello everyone, The python-dev website said I should introduce myself before lurking around for a while. So hi! I've been using python for 7 or so years now, live on the east coast of the U.S., grew up in Nebraska, and occasionally play the tuba and table-top games (through rarely together). I'm looking forward to hearing where the language is going and hope I can be of use after I lurk in the shadows for a while. Cheers, Brent

2 1