Mailman 3 November 2014 - Python-ideas

@classproperty, @abc.abstractclasspropery, etc.
by K. Richard Pixley Dec. 16, 2020

Dec. 16, 2020

There's a whole matrix of these and I'm wondering why the matrix is currently sparse rather than implementing them all. Or rather, why we can't stack them as: class foo(object): @classmethod @property def bar(cls, ...): ... Essentially the permutation are, I think: {'unadorned'|abc.abstract}{'normal'|static|class}{method|property|non-callable attribute}. concreteness implicit first arg type name comments {unadorned} {unadorned} method def foo(): exists … [View More]now {unadorned} {unadorned} property @property exists now {unadorned} {unadorned} non-callable attribute x = 2 exists now {unadorned} static method @staticmethod exists now {unadorned} static property @staticproperty proposing {unadorned} static non-callable attribute {degenerate case - variables don't have arguments} unnecessary {unadorned} class method @classmethod exists now {unadorned} class property @classproperty or @classmethod;@property proposing {unadorned} class non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract {unadorned} method @abc.abstractmethod exists now abc.abstract {unadorned} property @abc.abstractproperty exists now abc.abstract {unadorned} non-callable attribute @abc.abstractattribute or @abc.abstract;@attribute proposing abc.abstract static method @abc.abstractstaticmethod exists now abc.abstract static property @abc.staticproperty proposing abc.abstract static non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract class method @abc.abstractclassmethod exists now abc.abstract class property @abc.abstractclassproperty proposing abc.abstract class non-callable attribute {degenerate case - variables don't have arguments} unnecessary I think the meanings of the new ones are pretty straightforward, but in case they are not... @staticproperty - like @property only without an implicit first argument. Allows the property to be called directly from the class without requiring a throw-away instance. @classproperty - like @property, only the implicit first argument to the method is the class. Allows the property to be called directly from the class without requiring a throw-away instance. @abc.abstractattribute - a simple, non-callable variable that must be overridden in subclasses @abc.abstractstaticproperty - like @abc.abstractproperty only for @staticproperty @abc.abstractclassproperty - like @abc.abstractproperty only for @classproperty --rich [View Less]

10 15

Specify number of items to allocate for array.array() constructor
by Sven Rahmann Feb. 22, 2020

Feb. 22, 2020

At the moment, the array module of the standard library allows to create arrays of different numeric types and to initialize them from an iterable (eg, another array). What's missing is the possiblity to specify the final size of the array (number of items), especially for large arrays. I'm thinking of suffix arrays (a text indexing data structure) for large texts, eg the human genome and its reverse complement (about 6 billion characters from the alphabet ACGT). The suffix array is a long int … [View More]

14 20

Implicit string literal concatenation considered harmful?
by Guido van Rossum March 14, 2018

March 14, 2018

I just spent a few minutes staring at a bug caused by a missing comma -- I got a mysterious argument count error because instead of foo('a', 'b') I had written foo('a' 'b'). This is a fairly common mistake, and IIRC at Google we even had a lint rule against this (there was also a Python dialect used for some specific purpose where this was explicitly forbidden). Now, with modern compiler technology, we can (and in fact do) evaluate compile-time string literal concatenation with the '+' … [View More]

51 165

Extending traceback to (optionally) format and show locals.
by Robert Collins Jan. 18, 2015

Jan. 18, 2015

For context please see http://bugs.python.org/issue22937 and http://bugs.python.org/issue22936. I have two questions I'm hoping to get answered through this thread: - does the change in question need a PEP? Antoine seemed to think it didn't, as long as it was opt-in (via the unittest CLI etc) - Is my implementation approach sound (for traceback, unittest I think I have covered :))? Implementation wise, I think its useful to work in the current traceback module layout - that is to alter … [View More]

4 7

Asynchronous IO ideas for Python
by Paul Colomiets Dec. 6, 2014

Dec. 6, 2014

Hi, I've written an article about how I perceive the future of asynchronous I/O in Python. It's not something that should directly be incorporated into python now, but I believe it's useful for python-ideas list. https://medium.com/@paulcolomiets/the-future-of-asynchronous-io-in-python-c… And a place for comments at Hacker News: https://news.ycombinator.com/item?id=8662782 I hope being helpful with this writeup :) -- Paul

7 7

Fast context creation
by Stefano Borini Dec. 1, 2014

Dec. 1, 2014

I have a situation where, no matter how the routine ends, I need to return the focus to a widget. It would be rather clever to do this with a context, but I would expect something like this to be a possible strategy with contextlib.context(enter=None, exit=lambda *args: my_widget.setFocus()): do what I need to do as far as I know, at the moment it's not possible. Am I right? I think it would be an easy and practical addition to the contextlib module to quickly register two … [View More]

8 10

improve compatibility
by Liam Marsh Nov. 29, 2014

Nov. 29, 2014

hello, the problem is that even with extreme precaution, it is impossible to keep ALL modules compatible from a version to another. what I want to ask is this: -some "packs" which can, like py-compile, generate .pyc files, but using "old" versions of the default library, and of __builtins__. -One will go with every minor version and will be optionnal in the installation -any imported py file will be able to choose which version it wants with the "#! py recommended version X.X" or "#! py … [View More]

3 2

Re: [Python-ideas] improve compatibility
by Liam Marsh Nov. 29, 2014

Nov. 29, 2014

oops.. I sent an answer to andrew barnert, who have sent the first answer to my question, and not to the list... however , I do have an answer to my question. thank you! 2014-11-29 11:51 GMT+01:00 Andrew Barnert <abarnert(a)yahoo.com>: > On Nov 29, 2014, at 0:46, Liam Marsh <liam.marsh.home(a)gmail.com> wrote: > > hello, > > yes, the bug which gave me this idea is a py3.2 to py3.3 bug with Vpython, > a 3D graphs library. > > > And is the bug a stdlib … [View More]bug, or a language bug? > > I also thought this will be the reconciliation between py2 and py3. (this > is why I thought the packs will include a version of the stdlibs) > > > The language differences between 2.x and 3.x are huge, and most of the > stdlib differences between 2.6 and 3.1 or 2.7 and 3.2 are related to those > language differences. Porting code to 3.x is primarily about fixing Unicode > stuff, or code that was already deprecated in 2.5 or 2.6 but wasn't broken > until 3.x. Having the 2.7 stdlib in 3.x would be a huge amount of work for > almost no benefit. > > in fact, how do the .pyc work? were them modified by the "language and > implementation changes"? how do them import other modules? > > > .pyc files are just compiled bytecode, with a version-specific header. > If you fixed the 2.7 stdlib to work in 3.x (which, again, would be a huge > amount of work), you could compile it with 3.4. > > But you're missing the fact that large chunks of the stdlib are written in > C, and compiler against the Python C API. And parts of the stdlib > (especially builtins and the sys module) are exposing types and functions > written in C that are part of the core implementation, so the 2.x version > of the sys module wouldn't be compatible with 2.x code anyway. > > Also, did you mean to write just to me instead of to the list? > > > thank you! > > > > 2014-11-29 6:07 GMT+01:00 Andrew Barnert <abarnert(a)yahoo.com>: > >> On Nov 28, 2014, at 8:29, Liam Marsh <liam.marsh.home(a)gmail.com> wrote: >> >> > hello, >> > the problem is that even with extreme precaution, it is impossible to >> keep ALL modules compatible from a version to another. >> >> Do you have a specific library or app that you've had a problem with? >> There were a handful of modules that had a problem with the 3.2 to 3.3 >> conversion, but every one I saw was caused by language and implementation >> changes, not stdlib changes. I don't think I've seen anything that works >> with 3.3 but not 3.4. I'm sure it's not impossible for such a thing to >> happen, but it would be helpful to have at least one real-life example. >> >> > what I want to ask is this: >> > -some "packs" which can, like py-compile, generate .pyc files, but >> using "old" versions of the default library, and of __builtins__. >> >> But how would this work? The same changes that broke a handful of >> third-party modules between 3.2 and 3.3 probably also mean that the 3.2 >> stdlib wouldn't work in 3.3 without minor changes. And as for builtins, >> most of those are exposing internals of the implementation, so trying to >> make the 3.2 builtins work with 3.3 would take a lot more work than just >> building the 3.2 code against 3.3. >> >> > -One will go with every minor version and will be optionnal in the >> installation >> > -any imported py file will be able to choose which version it wants >> with the >> > "#! py recommended version X.X" or >> > "#! py mandatory version X.X" commentaries at the begining of the >> file. >> > >> > thank you and have a nice day/evening/night. >> > _______________________________________________ >> > Python-ideas mailing list >> > Python-ideas(a)python.org >> > https://mail.python.org/mailman/listinfo/python-ideas >> > Code of Conduct: http://python.org/psf/codeofconduct/ >> > > [View Less]

1 0

Re: [Python-ideas] Asynchronous IO ideas for Python]
by Trent Nelson Nov. 27, 2014

Nov. 27, 2014

On Thu, Nov 27, 2014 at 09:59:45PM +0000, Charles-François Natali wrote: > 2014-11-27 1:18 GMT+00:00 Trent Nelson <trent(a)snakebite.org>: > > > > Everything else is just normal Python, nothing special -- it just conforms > > to the current constraints of PyParallel. Basically, the > > HttpServer.data_received() method will be invoked from parallel threads, not > > the main interpreter thread. > > So, still no garbage collection from the threads? … [View More]

1 0

Re: [Python-ideas] Asynchronous IO ideas for Python
by Paul Colomiets Nov. 27, 2014

Nov. 27, 2014

Hi Trent, On Wed, Nov 26, 2014 at 10:36 PM, Trent Nelson <trent(a)trent.me> wrote: > Have you seen this?: > > https://speakerdeck.com/trent/pyparallel-how-we-removed-the-gil-and-exploit… > > I spend the first 80-ish slides on async I/O. > > (That was a year ago. I've done 2-3 sprints on it since then and have gotten it to a point where I can back up the claims with hard numbers on load testing benchmarks, demonstrated in the most recent video: https://www.… [View More]

1 0

PEP 479: Change StopIteration handling inside generators
by Chris Angelico Nov. 26, 2014

Nov. 26, 2014

PEP: 479 Title: Change StopIteration handling inside generators Version: $Revision$ Last-Modified: $Date$ Author: Chris Angelico <rosuav(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 15-Nov-2014 Python-Version: 3.5 Post-History: 15-Nov-2014 Abstract ======== This PEP proposes a semantic change to ``StopIteration`` when raised inside a generator, unifying the behaviour of list comprehensions and generator expressions somewhat. Rationale ========= … [View More]The interaction of generators and ``StopIteration`` is currently somewhat surprising, and can conceal obscure bugs. An unexpected exception should not result in subtly altered behaviour, but should cause a noisy and easily-debugged traceback. Currently, ``StopIteration`` can be absorbed by the generator construct. Proposal ======== If a ``StopIteration`` is about to bubble out of a generator frame, it is replaced with some other exception (maybe ``RuntimeError``, maybe a new custom ``Exception`` subclass, but *not* deriving from ``StopIteration``) which causes the ``next()`` call (which invoked the generator) to fail, passing that exception out. From then on it's just like any old exception. [3]_ Consequences to existing code ============================= This change will affect existing code that depends on ``StopIteration`` bubbling up. The pure Python reference implementation of ``groupby`` [1]_ currently has comments "Exit on ``StopIteration``" where it is expected that the exception will propagate and then be handled. This will be unusual, but not unknown, and such constructs will fail. (Nick Coghlan comments: """If you wanted to factor out a helper function that terminated the generator you'd have to do "return yield from helper()" rather than just "helper()".""") As this can break code, it is proposed to utilize the ``__future__`` mechanism to introduce this, finally making it standard in Python 3.6 or 3.7. Alternate proposals =================== Supplying a specific exception to raise on return ------------------------------------------------- Nick Coghlan suggested a means of providing a specific ``StopIteration`` instance to the generator; if any other instance of ``StopIteration`` is raised, it is an error, but if that particular one is raised, the generator has properly completed. Making return-triggered StopIterations obvious ---------------------------------------------- For certain situations, a simpler and fully backward-compatible solution may be sufficient: when a generator returns, instead of raising ``StopIteration``, it raises a specific subclass of ``StopIteration`` which can then be detected. If it is not that subclass, it is an escaping exception rather than a return statement. Criticism ========= Unofficial and apocryphal statistics suggest that this is seldom, if ever, a problem. [4]_ Code does exist which relies on the current behaviour, and there is the concern that this would be unnecessary code churn to achieve little or no gain. References ========== .. [1] Initial mailing list comment (https://mail.python.org/pipermail/python-ideas/2014-November/029906.html) .. [2] Pure Python implementation of groupby (https://docs.python.org/3/library/itertools.html#itertools.groupby) .. [3] Proposal by GvR (https://mail.python.org/pipermail/python-ideas/2014-November/029953.html) .. [4] Response by Steven D'Aprano (https://mail.python.org/pipermail/python-ideas/2014-November/029994.html) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: [View Less]

23 132

Counter comparison
by Antony Lee Nov. 24, 2014

Nov. 24, 2014

As Counter objects effectively behave like multi-sets, it seems reasonable to overload <, <=, >, >= as for set objects to check whether a Counter is a sub/super-set of another Counter: c < d <===> all(c[k] < d[k] for k in c) Thoughts? Antony

11 19

Re: [Python-ideas] Counter comparison
by MRAB Nov. 22, 2014

Nov. 22, 2014

On 2014-11-22 01:36, Antony Lee wrote: > I guess it depends on whether you think {"a": 1} < {"a": 2, "b": -1} > or not. I actually thought about it and initially decided that the > inequality should hold, but given that most_common works fine with > negative counts, perhaps it shouldn't, indeed. > The other point is that if the 'b' entry was in the first, then it would be checked, and the negative count would matter, but as it wasn't, it won't, and it won't. > 2014-… [View More]

1 0

Re: [Python-ideas] PEP 479: Change StopIteration handling inside generators
by Chris Barker Nov. 21, 2014

Nov. 21, 2014

I htink I'm getting closer to clarity here, but: let's say I have this silly helper function (similar to the one in the PEP): ## (sorry -- python2) def helper(x): if x > 2: raise StopIteration else: return x # I call that helper from both an iterator class and a generator function. class Iter(object): def __init__(self): self.x = 0 def __iter__(self): return self def next(self): if self.x == 3: raise StopIteration self.x += 1 … [View More]

2 1

Return expressions
by Nick Coghlan Nov. 21, 2014

Nov. 21, 2014

This idea occurred to me in context of PEP 479 and the "or stop()" hack for generator expressions. I'm not a big enough fan of the idea to pursue it myself, but if anyone is bothered by the prospect of PEP 479 taking the "or stop()" technique away, this may be worth pursuing further. As a reminder of how the hack works, the following generator: >>> def takewhile(iterable, pred): ... for x in iter(iterable): ... if not pred(x): ... return ... yield x ...… [View More]

13 23

Re: [Python-ideas] Implement `itertools.permutations.__getitem__` and `itertools.permutations.index`
by Ram Rachum Nov. 17, 2014

Nov. 17, 2014

Hi guys, I just wanted to give an update on this: I just released my own code that does this to PyPI: https://pypi.python.org/pypi/combi Thanks, Ram. On Tue, Jun 10, 2014 at 9:24 AM, Ram Rachum <ram(a)rachum.com> wrote: > I'll email you if/when it's released :) > > > On Tue, Jun 10, 2014 at 9:04 AM, Neil Girdhar <mistersheik(a)gmail.com> > wrote: > >> I really like this and hope that it eventually makes it into the stdlib. >> It's also a good … [View More]

3 9

Re: [Python-ideas] Change how Generator Expressions handle StopIteration
by Chris Angelico Nov. 15, 2014

Nov. 15, 2014

On Sat, Nov 15, 2014 at 11:36 PM, Skip Montanaro <skip.montanaro(a)gmail.com> wrote: > On Sat, Nov 15, 2014 at 1:06 AM, Chris Angelico <rosuav(a)gmail.com> wrote: >> >> Draft PEP exists, and now has your concerns incorporated. >> >> https://raw.githubusercontent.com/Rosuav/GenStopIter/master/pep-xxx.txt > > > Chris, > > 404 for me... (replying on-list as this will affect everyone now) It's now been allocated an actual number: https://raw.githubusercontent.com/Rosuav/GenStopIter/master/pep-0479.txt ChrisA

1 0

Change how Generator Expressions handle StopIteration
by yotam vaknin Nov. 15, 2014

Nov. 15, 2014

Hi, I would like to purpose that generator expressions will not catch StopIteration exception, if this exception did not come from the iterated object's __next__ function specifically. So generator expressions will be able to raise StopIteration by calculating the current value of the Generator. Here is an example of a use-case : def izip(*args): iters = [iter(obj) for obj in args] while True: yield tuple(next(it) for it in iters) a = izip([1,2],[3,4]) print(next(a),next(a),… [View More]

17 56

Python IDE (New IDE for Python scripting)
by Danilo Peixoto Nov. 15, 2014

Nov. 15, 2014

An alternative IDE with a new dark and modern interface. http://s27.postimg.org/r4cjxf00z/Python_IDE.png (Concept) Concept shows the Python IDE functionality to organize projects and files in own software and quick information on the status bar. Some additional features: AutocompleteFind and replaceIncrease and decrease line indentand more...

7 7

A way to tell the gc it's OK to delete an object with a __del__
by Chris Barker Nov. 14, 2014

Nov. 14, 2014

I've been poking into reference counting, circular references, etc, trying to understand how the garbage collector works. In practice, I'm trying to figure out a way to get a couple modules I work with not to leak memory: one is an extension to wxPython that I wrote, that creates a circular reference involving wx objects -- these, as is often the case with wrappers, have a custom __del__ method. The other is the python netcdf4 lib -- which has a circular reference built in, so that Datasets (… [View More]

6 10

including psutil in the standard library?
by Stefano Borini Nov. 11, 2014

Nov. 11, 2014

I am wondering if it would be possible to include psutil (https://pypi.python.org/pypi/psutil ) in the standard library, and if not, what would be needed. I am not a developer of it, but I am using psutil at work with good success. it provides a good deal of services for querying and managing processes in a cross platform way. Any thoughts?

12 15

Store shared/locked state inside of the lock object
by Masklinn Nov. 11, 2014

Nov. 11, 2014

Since the introduction of context managers, using the lock object itself has become easier and safer (in that there's very little chance of forgetting to release a lock anymore). A big annoyance remains though: relation between lock and lockee remains informal and there is no structured way to indicate: 1. whether a state-set is protected by a lock 2. which lock protects the state-set 3. which state-set is protected by a lock Some languages (e.g. Java) have tried to solve 2 and 3 using … [View More]intrinsic locks, however * that does not solve 1 (and it becomes impossible to informally look for lock objects lying around and try to find corresponding state-sets) * it does not help much when state isn't coalesced in a single object, and for state hierarchies there is no way to express whether the whole hierarchy should be protected under the same lock (the root's) or each leaf should be locked individually. AFAIK intrinsic locks are not hierarchical themselves * things get very awkward when using alternate concurrency-management strategies such as explicit locks for security reason[0], the non-use of intrinsic locks has to be again documented informally A fairly small technical change I've been considering to improve this situation is to store the state-set inside the lock object, and only yield it through the context manager: that the state-set is protected by a lock is made obvious, and so is the relation between lock and state-set. I was delighted to discover that Rust's sync::Mutex[1] and sync::RWLock[2] follow a very similar strategy of owning the state-set It's not a panacea, it doesn't fix issues of lock acquisition ordering for instance, but I think it would go a fairly long way towards making correct use of locks easier in Python. The basic changes would be: * threading.Lock and threading.RLock would now take an optional `data` parameter * the parameter would be stored internally and not directly accessible by users of the lock * that parameter would be returned by __enter__ and provided to the current "owner" of the lock These should cause no forward-compatibility issues, Lock() currently takes no arguments, and its __enter__ returns no value. Possible improvements/questions/issues I can see: * with Lock, the locked state would not be available unless using as a context manager. RLock could allow getting the protected state only while locked by the current thread * as-is, the scheme requires mutable state as it's not possible to swap the internal state entirely. RLock could allow state-replacement when locked * because Python has no ownership concept, it would be possible for a consumer to keep a reference to the locked state and manipulate it without locking I don't consider the third issue to be huge, it could be mitigated by yielding a proxy to the internal state only valid for the current lock span. However I do not know if it's possible to create completely transparent proxies in Python. The first two issues are slightly more troubling and could be mitigated by yielding not the state-set alone but a proxy object living only for the current lock span (or both the state-set and a proxy object) that proxy would allow getting and setting the state-set, and would error-out after unlocking. Lock.acquire() could be altered to return the same proxy (or (state-set, proxy) pair) however it's currently defined as returning either True or False so that'd be a backwards- incompatible change. An alternative would be to add a new acquisition method or a new flag parameter changing the return value from True to these on acquisition. A drawback of this additional change is that it would require the lock object to keep track of the current live proxy(/proxies for rlock?), and invalidate it(/them) on unlocking, increasing its complexity much more than just adding a new attribute. Thoughts? [0] https://www.securecoding.cert.org/confluence/display/java/LCK00-J.+Use+priv… [1] http://doc.rust-lang.org/sync/struct.Mutex.html [2] http://doc.rust-lang.org/sync/struct.RWLock.html [View Less]

5 6

Deprecate dunder functions from operator module
by Steven D'Aprano Nov. 4, 2014

Nov. 4, 2014

(For the avoidance of all doubt, the following has nothing to do with dunder methods like int.__add__ etc., but only the functions in the operator module.) The operator module contains various function versions of operators, e.g. operator.add for + , operator.sub for - , etc. There are also dunder versions of each function, and according to the documentation: The function names are those used for special class methods; variants without leading and trailing __ are also provided … [View More]

4 7

"or raise" syntax
by Javier Dehesa Nov. 3, 2014

Nov. 3, 2014

This happens to me with some frequency: result = f(args) if not result: # of "if result is None:" raise Exception(...) What if I could just say? result = f(args) or raise Exception(...) I think that's pretty idiomatic. What's more, this would give you a shorter syntax for things like: if x <= y: raise Exception(...) Which you could write simply as: x > y or raise Exception(...) Effectively covering the use case of this proposal http://article.… [View More]

4 3