Mailman 3 November 2015 - Python-ideas

@classproperty, @abc.abstractclasspropery, etc.
by K. Richard Pixley Dec. 16, 2020

Dec. 16, 2020

There's a whole matrix of these and I'm wondering why the matrix is currently sparse rather than implementing them all. Or rather, why we can't stack them as: class foo(object): @classmethod @property def bar(cls, ...): ... Essentially the permutation are, I think: {'unadorned'|abc.abstract}{'normal'|static|class}{method|property|non-callable attribute}. concreteness implicit first arg type name comments {unadorned} {unadorned} method def foo(): exists … [View More]now {unadorned} {unadorned} property @property exists now {unadorned} {unadorned} non-callable attribute x = 2 exists now {unadorned} static method @staticmethod exists now {unadorned} static property @staticproperty proposing {unadorned} static non-callable attribute {degenerate case - variables don't have arguments} unnecessary {unadorned} class method @classmethod exists now {unadorned} class property @classproperty or @classmethod;@property proposing {unadorned} class non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract {unadorned} method @abc.abstractmethod exists now abc.abstract {unadorned} property @abc.abstractproperty exists now abc.abstract {unadorned} non-callable attribute @abc.abstractattribute or @abc.abstract;@attribute proposing abc.abstract static method @abc.abstractstaticmethod exists now abc.abstract static property @abc.staticproperty proposing abc.abstract static non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract class method @abc.abstractclassmethod exists now abc.abstract class property @abc.abstractclassproperty proposing abc.abstract class non-callable attribute {degenerate case - variables don't have arguments} unnecessary I think the meanings of the new ones are pretty straightforward, but in case they are not... @staticproperty - like @property only without an implicit first argument. Allows the property to be called directly from the class without requiring a throw-away instance. @classproperty - like @property, only the implicit first argument to the method is the class. Allows the property to be called directly from the class without requiring a throw-away instance. @abc.abstractattribute - a simple, non-callable variable that must be overridden in subclasses @abc.abstractstaticproperty - like @abc.abstractproperty only for @staticproperty @abc.abstractclassproperty - like @abc.abstractproperty only for @classproperty --rich [View Less]

10 15

Specify number of items to allocate for array.array() constructor
by Sven Rahmann Feb. 22, 2020

Feb. 22, 2020

At the moment, the array module of the standard library allows to create arrays of different numeric types and to initialize them from an iterable (eg, another array). What's missing is the possiblity to specify the final size of the array (number of items), especially for large arrays. I'm thinking of suffix arrays (a text indexing data structure) for large texts, eg the human genome and its reverse complement (about 6 billion characters from the alphabet ACGT). The suffix array is a long int … [View More]

14 20

Implicit string literal concatenation considered harmful?
by Guido van Rossum March 14, 2018

March 14, 2018

I just spent a few minutes staring at a bug caused by a missing comma -- I got a mysterious argument count error because instead of foo('a', 'b') I had written foo('a' 'b'). This is a fairly common mistake, and IIRC at Google we even had a lint rule against this (there was also a Python dialect used for some specific purpose where this was explicitly forbidden). Now, with modern compiler technology, we can (and in fact do) evaluate compile-time string literal concatenation with the '+' … [View More]

51 165

proposal: "python -m foo" should bind sys.modules['foo']
by Cameron Simpson Jan. 18, 2017

Jan. 18, 2017

Hello all, This is a writeup of a proposal I floated here: https://mail.python.org/pipermail/python-list/2015-August/694905.html last Sunday. If the response is positive I wish to write a PEP. Briefly, it is a natural expectation in users that the command: python -m module_name ... used to invoke modules in "main program" mode on the command line imported the module as "module_name". It does not, it imports it as "__main__". An import within the program of "module_name" makes a new … [View More]instance of the module, which causes cognitive dissonance and has the side effect that now the program has two instances of the module. What I propose is that the above command line _should_ bind sys.modules['module_name'] as well as binding '__main__' as it does currently. I'm proposing that the python -m option have this effect (python pseudocode): % python -m module.name ... runs: # pseudocode, with values hardwired for clarity import sys M = new_empty_module(name='__main__', qualname='module.name') sys.modules['__main__'] = M sys.modules['module.name'] = M # load the module code from wherever (not necessarily a file - CPython # already must do this phase) M.execfile('/path/to/module/name.py') Specificly, this would have the following two changes to current practice: 1) the module is imported _once_, and bound to both its canonical name and also to __main__. 2) imported modules acquire a new attribute __qualname__ (analogous to the recent __qualname__ on functions). This is always the conanoical name of the module as resolved by the importer. For most modules __name__ will be the same as __qualname__, but for the "main" module __name__ will be '__main__'. This change has the following advantages: The current standard boilerplate: if __name__ == '__main__': ... invoke "main program" here ... continues to work unchanged. Importantly, if the program then issues "import module_name", it is already there and the existing instance is found and used. The thread referenced above outlines my most recent encounter with this and the trouble it caused me. Followup messages include some support for this proposed change, and some criticism. The critiquing article included some workarounds for this multiple module situation, but they were (1) somewhat dependent on modules coming from a file pathname and (2) cumbersome and require every end user to adopt these changes if affected by the situation. I'd like to avoid that. Cheers, Cameron Simpson <cs(a)zip.com.au> The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man. - George Bernard Shaw [View Less]

5 13

solving multi-core Python
by Eric Snow June 3, 2016

June 3, 2016

tl;dr Let's exploit multiple cores by fixing up subinterpreters, exposing them in Python, and adding a mechanism to safely share objects between them. This proposal is meant to be a shot over the bow, so to speak. I plan on putting together a more complete PEP some time in the future, with content that is more refined along with references to the appropriate online resources. Feedback appreciated! Offers to help even more so! :) -eric -------- Python's multi-core story is murky at best. … [View More]Not only can we be more clear on the matter, we can improve Python's support. The result of any effort must make multi-core (i.e. parallelism) support in Python obvious, unmistakable, and undeniable (and keep it Pythonic). Currently we have several concurrency models represented via threading, multiprocessing, asyncio, concurrent.futures (plus others in the cheeseshop). However, in CPython the GIL means that we don't have parallelism, except through multiprocessing which requires trade-offs. (See Dave Beazley's talk at PyCon US 2015.) This is a situation I'd like us to solve once and for all for a couple of reasons. Firstly, it is a technical roadblock for some Python developers, though I don't see that as a huge factor. Regardless, secondly, it is especially a turnoff to folks looking into Python and ultimately a PR issue. The solution boils down to natively supporting multiple cores in Python code. This is not a new topic. For a long time many have clamored for death to the GIL. Several attempts have been made over the years and failed to do it without sacrificing single-threaded performance. Furthermore, removing the GIL is perhaps an obvious solution but not the only one. Others include Trent Nelson's PyParallels, STM, and other Python implementations.. Proposal ======= In some personal correspondence Nick Coghlan, he summarized my preferred approach as "the data storage separation of multiprocessing, with the low message passing overhead of threading". For Python 3.6: * expose subinterpreters to Python in a new stdlib module: "subinterpreters" * add a new SubinterpreterExecutor to concurrent.futures * add a queue.Queue-like type that will be used to explicitly share objects between subinterpreters This is less simple than it might sound, but presents what I consider the best option for getting a meaningful improvement into Python 3.6. Also, I'm not convinced that the word "subinterpreter" properly conveys the intent, for which subinterpreters is only part of the picture. So I'm open to a better name. Influences ======== Note that I'm drawing quite a bit of inspiration from elsewhere. The idea of using subinterpreters to get this (more) efficient isolated execution is not my own (I heard it from Nick). I have also spent quite a bit of time and effort researching for this proposal. As part of that, a number of people have provided invaluable insight and encouragement as I've prepared, including Guido, Nick, Brett Cannon, Barry Warsaw, and Larry Hastings. Additionally, Hoare's "Communicating Sequential Processes" (CSP) has been a big influence on this proposal. FYI, CSP is also the inspiration for Go's concurrency model (e.g. goroutines, channels, select). Dr. Sarah Mount, who has expertise in this area, has been kind enough to agree to collaborate and even co-author the PEP that I hope comes out of this proposal. My interest in this improvement has been building for several years. Recent events, including this year's language summit, have driven me to push for something concrete in Python 3.6. The subinterpreter Module ===================== The subinterpreters module would look something like this (a la threading/multiprocessing): settrace() setprofile() stack_size() active_count() enumerate() get_ident() current_subinterpreter() Subinterpreter(...) id is_alive() running() -> Task or None run(...) -> Task # wrapper around PyRun_*, auto-calls Task.start() destroy() Task(...) # analogous to a CSP process id exception() # other stuff? # for compatibility with threading.Thread: name ident is_alive() start() run() join() Channel(...) # shared by passing as an arg to the subinterpreter-running func # this API is a bit uncooked still... pop() push() poison() # maybe select() # maybe Note that Channel objects will necessarily be shared in common between subinterpreters (where bound). This sharing will happen when the one or more of the parameters to the function passed to Task() is a Channel. Thus the channel would be open to the (sub)interpreter calling Task() (or Subinterpreter.run()) and to the new subinterpreter. Also, other channels could be fed into such a shared channel, whereby those channels would then likewise be shared between the interpreters. I don't know yet if this module should include *all* the essential pieces to implement a complete CSP library. Given the inspiration that CSP is providing, it may make sense to support it fully. It would be interesting then if the implementation here allowed the (complete?) formalisms provided by CSP (thus, e.g. rigorous proofs of concurrent system models). I expect there will also be a _subinterpreters module with low-level implementation-specific details. Related Ideas and Details Under Consideration ==================================== Some of these are details that need to be sorted out. Some are secondary ideas that may be appropriate to address in this proposal or may need to be tabled. I have some others but these should be sufficient to demonstrate the range of points to consider. * further coalesce the (concurrency/parallelism) abstractions between threading, multiprocessing, asyncio, and this proposal * only allow one running Task at a time per subinterpreter * disallow threading within subinterpreters (with legacy support in C) + ignore/remove the GIL within subinterpreters (since they would be single-threaded) * use the GIL only in the main interpreter and for interaction between subinterpreters (and a "Local Interpreter Lock" for within a subinterpreter) * disallow forking within subinterpreters * only allow passing plain functions to Task() and Subinterpreter.run() (exclude closures, other callables) * object ownership model + read-only in all but 1 subinterpreter + RW in all subinterpreters + only allow 1 subinterpreter to have any refcounts to an object (except for channels) * only allow immutable objects to be shared between subinterpreters * for better immutability, move object ref counts into a separate table * freeze (new machinery or memcopy or something) objects to make them (at least temporarily) immutable * expose a more complete CSP implementation in the stdlib (or make the subinterpreters module more compliant) * treat the main interpreter differently than subinterpreters (or treat it exactly the same) * add subinterpreter support to asyncio (the interplay between them could be interesting) Key Dependencies ================ There are a few related tasks/projects that will likely need to be resolved before subinterpreters in CPython can be used in the proposed manner. The proposal could implemented either way, but it will help the multi-core effort if these are addressed first. * fixes to subinterpreter support (there are a couple individuals who should be able to provide the necessary insight) * PEP 432 (will simplify several key implementation details) * improvements to isolation between subinterpreters (file descriptors, env vars, others) Beyond those, the scale and technical scope of this project means that I am unlikely to be able to do all the work myself to land this in Python 3.6 (though I'd still give it my best shot). That will require the involvement of various experts. I expect that the project is divisible into multiple mostly independent pieces, so that will help. Python Implementations =================== They can correct me if I'm wrong, but from what I understand both Jython and IronPython already have subinterpreter support. I'll be soliciting feedback from the different Python implementors about subinterpreter support. C Extension Modules ================= Subinterpreters already isolate extension modules (and built-in modules, including sys). PEP 384 provides some help too. However, global state in C can easily leak data between subinterpreters, breaking the desired data isolation. This is something that will need to be addressed as part of the effort. [View Less]

26 130

OrderedCounter and OrderedDefaultDict
by Ian Foote Feb. 12, 2016

Feb. 12, 2016

I recently wanted to use an OrderedCounter and an OrderedDefaultDict. After a bit of googling I discovered that OrderedCounter is quite easy to implement: from collections import Counter, OrderedDict class OrderedCounter(Counter, OrderedDict): 'Counter that remembers the order elements are first seen' def __repr__(self): return '%s(%r)' % (self.__class__.__name__, OrderedDict(self)) def __reduce__(self): return self.… [View More]

14 23

Adding collections.abc.Ordered
by Amir Rachum Dec. 27, 2015

Dec. 27, 2015

I am suggesting the addition of a collections abstract base class called "Ordered". Its meaning is that a collection's iteration order is part of its API. The bulk of this mail describes a use case for this. The reason I believe that such abstract base class is required is that there is no way to test this behavior in a given class. An ordered collection has the exact same interface as an unordered collection (e.g, dict and OrderedDict), other than a _promise_ of the API that the order in which … [View More]this collection will be iterated has some sort of meaning (In OrderedDict, it is the order in which keys were added to it.) As examples, set, frozenset, dict and defaultdict should *not* be considered as ordered. list, OrderedDict, deque and tuple should be considered ordered. Before I dive into the use case I am presenting, I would like to point out that a similar discussion was already done on the subject (suggesting at first that OrderedDict would abstract-subclass from Sequence) at the following thread* - http://code.activestate.com/lists/python-ideas/29532/. That thread was abandoned largely from a lack of a clear use case, which I hope will be more clear in this suggestion. I am working on package called basicstruct ( https://pypi.python.org/pypi/basicstruct). The way it works is that you define an object that inherits from basicstruct.BasicStruct, define __slots__ , and automatically get behaviors such as a nice __init__, __str__, comparison functions, attribute access, dict-like access, pickleability, etc. It is similar to namedtuple, except that it is mutable. In the Python documentation, the following is said regarding __slots__: Any non-string iterable may be assigned to __slots__. Mappings may also be used; however, in the future, special meaning may be assigned to the values corresponding to each key. Here's how the current__init__ method of BasicStruct looks like: class BasicStruct(object): """Class for holding struct-like objects.""" __slots__ = () # should be extended by deriving classes def __init__(self, *args, **kwargs): arg_pairs = zip(self.__slots__, args) for key, value in chain(arg_pairs, six.iteritems(kwargs)): setattr(self, key, value) for key in self.__slots__: if not hasattr(self, key): setattr(self, key, None) Notice the following line: arg_pairs = zip(self.__slots__, args) It assumes that __slots__ defines attributes in a certain order. So as a use I would expect that the following code class MyStruct(BasicStruct): __slots__ = ('x', 'y') MyStruct(0, 1) ... will create a struct in which x is 0 and y is 1. However, if I define __slots__ as a set, or dict, the result is undefined. class MyStruct(BasicStruct): __slots__ = {'x', 'y'} # No order is defined here MyStruct(0, 1) # Which is which? So, In BasicStruct's __init__ method it is required to test whether __slots__ was defined with a collection that has a meaningful order. If it was _not_, using non-keyword arguments in __init__ should be forbidden, since the order of __slots__ is arbitrary. Here is how I wish the code would look like: class BasicStruct(object): """Class for holding struct-like objects.""" __slots__ = () # should be extended by deriving classes def __init__(self, *args, **kwargs): ordered = isinstance(self.__slots__, Ordered) if args and not ordered: raise ValueError("Can't pass non-keyword arguments to {}, since " "__slots__ was declared with an unordered " "iterable.".format(self.__class__.__name__)) arg_pairs = zip(self.__slots__, args) for key, value in chain(arg_pairs, six.iteritems(kwargs)): setattr(self, key, value) for key in self.__slots__: if not hasattr(self, key): setattr(self, key, None) Thanks, Amir Rachum * The author of the original thread is Ram Rachum. To avoid some confusion - yes, we are related - Ram is my big brother. It's just so happens that I was looking around for this issue and found that he did so in the past as well. [View Less]

14 48

Re: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators
by Henry Schreiner Dec. 1, 2015

Dec. 1, 2015

Sorry, the order was incorrect in that last message. It should be: When x and y is written, if x has __booland__, then x.__booland__(y) is returned. If not, then return x if x==False else if x==True, then if y has y.__rbooland__(x), that is returned. If not, then return y. On Monday, November 30, 2015 at 10:11:03 AM UTC-6, Henry Schreiner wrote: > > It seems to me there are two ideas being discussed. One is adding 2 new > operators, something like && and ||, and providing … [View More]them for some sort of > "element wise" or other boolean context, and giving them a better order of > evaluations than bitwise ops. This would have the advantage of being usable > on the built in python lists/tuples if needed, but otherwise doesn't seem > to solve all the issues above, including expressions like -1 < mat < 1, > which work for python variables but not for things like Numpy arrays. (for > example, selecting values from an array works like this: mat[(-1<mat) & > (mat<1)] but not like this: mat[-1 < mat < 1] or this: mat[-1<mat & > mat<1] ). > > The common "general purpose infix notation" pops up here, of course, but > I'd consider it a variant of the first idea. > > The other idea is that overloads could be added to the existing and and or > operators. This, I believe, would solve all the main problems, and would > not add more to the language for a newcomer to learn. (In fact, most people > could forget about the current bitwise and and or, since that's not what > most people need, at least those using databases and Numpy). The downside > is that the current and and or operators are short-circuiting, and adding > a way to overload a short-circuiting operator is tricky/ugly/expensive > (though the last one might not be that bad anymore). But, as I think Andrew > was mentioning, what about a mixed solution? One that provides a > non-short-circuiting overload, but the original short-circuiting behavior > is default? It would look like this: > > Two new magic methods would be added, for now called __booland__ and > __boolor__, along with __rbooland__ and __rboolor__. > > > > Evaluation would work like this, using and for an example: > When x and y is written, if x has __booland__, then y.__booland__(y) is > called. If it does not, then if y exists and has y.__rbooland__(x), that > is called. If neither, then x and y is performed in a normal, > short circuiting manner, calling bool(x) then calling bool(y) only if x > == True. (The second line here might be the hard part due to the way > short circuiting works) > > > > This would allow object that want custom and/or behavior to implement > these magic functions. They would lose (explicit) short-circuiting, but > they would gain custom behavior. Most of these special objects do not need > or expect the short circuit behavior (Only if an array was all True or > False would it even make sense in numpy , and this would allow > expressions like -1 < mat < 1 to work as expected in Python. This would > be easy to teach to newcomers, too, since only objects that have a definite > True/False should short circuit, others have the special methods. The > old behavior could be forced on overriding objects by wrapping them in > bool(). > > > Would something like that be possible and better than PEP 335? > > PS: > Either type of solution would also be useful for the Plumbum library, by > the way, as it would be useful to add and-ing and or-ing to pipelines, but > the bitwise operators have a different meaning for Plumbum. Due to the lazy > evaluation of Plumbum, even the above solution (or any solution) would > still allow short-circuiting in the pipeline. > > > [View Less]

3 4

Re: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators
by Henry Schreiner Nov. 30, 2015

Nov. 30, 2015

My proposal, I believe, is exactly the same as the “Reduced Special Methods set” of PEP 335; that’s the phase two only part. I believe the __r*__ methods there are the same as my proposal. A portion of the complexity of PEP 335 is used in the “phase 1” special methods, which are only helpful if you want to overload the shortcutting behavior itself on a per object basis, which I don’t think there is much need for. Assuming your methods are part of the class, as Python usually requires anyway, … [View More]

1 0

Re: [Python-ideas] The AttributeError/__getattr__ mechanism
by 王珺 Nov. 30, 2015

Nov. 30, 2015

> A dedicated subclass of *Attr*ibuteError to trigger __get*attr*__ is more natural. I mean it's unnatural to convert AttributeError to AttributeDynamicError by property or descriptor. Why not use RuntimeError as that of PEP479? And there's no need to write custom property-like descriptor factory when you want to trigger __getattr__, although it may be only a few lines of code.

4 6

Re: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators
by Henry Schreiner Nov. 30, 2015

Nov. 30, 2015

It seems to me there are two ideas being discussed. One is adding 2 new operators, something like && and ||, and providing them for some sort of "element wise" or other boolean context, and giving them a better order of evaluations than bitwise ops. This would have the advantage of being usable on the built in python lists/tuples if needed, but otherwise doesn't seem to solve all the issues above, including expressions like -1 < mat < 1, which work for python variables but not for … [View More]things like Numpy arrays. (for example, selecting values from an array works like this: mat[(-1<mat) & (mat<1)] but not like this: mat[-1 < mat < 1] or this: mat[-1<mat & mat<1] ). The common "general purpose infix notation" pops up here, of course, but I'd consider it a variant of the first idea. The other idea is that overloads could be added to the existing and and or operators. This, I believe, would solve all the main problems, and would not add more to the language for a newcomer to learn. (In fact, most people could forget about the current bitwise and and or, since that's not what most people need, at least those using databases and Numpy). The downside is that the current and and or operators are short-circuiting, and adding a way to overload a short-circuiting operator is tricky/ugly/expensive (though the last one might not be that bad anymore). But, as I think Andrew was mentioning, what about a mixed solution? One that provides a non-short-circuiting overload, but the original short-circuiting behavior is default? It would look like this: Two new magic methods would be added, for now called __booland__ and __boolor__, along with __rbooland__ and __rboolor__. Evaluation would work like this, using and for an example: When x and y is written, if x has __booland__, then x.__booland__(y) is returned. If not, then return x if x==False else if x==True, then if y has y.__rbooland__(x), that is returned. If not, then return y. This would allow object that want custom and/or behavior to implement these magic functions. They would lose (explicit) short-circuiting, but they would gain custom behavior. Most of these special objects do not need or expect the short circuit behavior (Only if an array was all True or False would it even make sense in numpy , and this would allow expressions like -1 < mat < 1 to work as expected in Python. This would be easy to teach to newcomers, too, since only objects that have a definite True/False should short circuit, others have the special methods. The old behavior could be forced on overriding objects by wrapping them in bool(). Would something like that be possible and better than PEP 335? PS: Either type of solution would also be useful for the Plumbum library, by the way, as it would be useful to add and-ing and or-ing to pipelines, but the bitwise operators have a different meaning for Plumbum. Due to the lazy evaluation of Plumbum, even the above solution (or any solution) would still allow short-circuiting in the pipeline. PPS: Hopefully I’m not double posting, apparently the google group and the mailing list are separate subscriptions. Henry Schreiner [View Less]

1 0

PEP471 - (os.scandir())
by Erik Nov. 30, 2015

Nov. 30, 2015

PEP471 introduces a faster way of doing low-level directory traversal which is then used to implement and speed up the higher-level API os.walk() - which for me at least is the "go to API" for most directory scanning code I write. However, when using os.walk() the first thing that one tends to do with the results is to analyse them in some way (look at file sizes, datestamps and other things that stat() returns) which is exactly the information that os.scandir() is caching and speeding … [View More]

6 9

The AttributeError/__getattr__ mechanism
by 王珺 Nov. 29, 2015

Nov. 29, 2015

Hello everyone: I'm suggesting a modification to the AttributeError/__getattr__ mechanism, see issue25634: http://bugs.python.org/issue25634 I used __getattr__ sometimes, and descriptor especially property is so widely used. I wonder whether someone had encountered the same problem with me. However, this is a complicated problem including performance issues, and backward compatibility. Thanks for your attention, Jun Wang

5 14

add a single __future__ for py3?
by Michael Scott Cuthbert Nov. 28, 2015

Nov. 28, 2015

(an old list topic that I accidentally sent to the Google Groups list) One smaller request is to remove 'barry_as_FLUFL' from "__future__.all_feature_names". It's a great practical joke, but it bit me hard on a project where I check that, for Py2/Py3 consistency, each module imports all relevant __future__ names, and ensures that doctests on Py2 are run with them on. I figured that I could iterate through .all_feature_names and turn them all on except unicode_literals and be safe, but … [View More]

3 2

[Python-ideos] Dedicated overloadable boolean operators
by Jelte Fennema Nov. 27, 2015

Nov. 27, 2015

Hi, After reading PEP0465 <https://www.python.org/dev/peps/pep-0465/> about the dedicated matrix multiplication I started wondering if the same solution couldn't be applied to boolean operators as well. There currently are a lot of high profile libraries that have their own functions for boolean operators, like Numpy, Pandas or SQLAlchemy. They do this because the current boolean operators can't be overloaded. PEP0335 <https://www.python.org/dev/peps/pep-0335/> was created to solve … [View More]

19 56

Re: [Python-ideas] The prohibition overrides the class method
by Иван Спиненко Nov. 18, 2015

Nov. 18, 2015

Thank you all very much. Perhaps I'll leave that idea, Python too flexible language for such things.

3 3

Lessons from typing hinting Whoosh (PEP484)
by Matt Chaput Nov. 17, 2015

Nov. 17, 2015

Since PyCharm added support for pep484 I've been working on adding type hints to the Whoosh code base. It makes it a lot more enjoyable to work with the code in the IDE. As I add more and more hints, it's already revealed bugs where arguments were accidentally swapped, or interfaces weren't updated everywhere. I feel much more confident working in the (quite large) codebase with typing. Basically, I'm loving the idea of PEP484. For me, it's a huge improvement in the development experience. … [View More]However, I've already run into a few issues. I'm not sure if these were discussed during the development of PEP484. 1. The biggest problem, most perplexing problem for me is circular imports. That type hints must actually be imported at the top level is a MASSIVE limitation, at least in a big codebase. I started off importing the types I needed, then switched to the "import the module, specify the type in a string" trick, but even that has failed. In a large/complex codebase, the more things you have to import at the top level (to use for typing), the more potential import circles you create. Now I've hit an impasse where every step in the circle is using the module/string trick, but it doesn't help because I need to subclass something in the circle. For now, I'm going to have to workaround this, either by trying to move "interface" base classes out to their own side package, or just not type hinting some things. Neither is appetizing at all. I think a good solution might be if the typing system could have its own special "imports" that are only used by the typing system, e.g. something like this at the top of a file: # "real" import from foo import bar # imports just for type checking __typeimports__ = """ from baz import Qux """ 2. It would be really nice if we could have "type aliasing" (or whatever it's called). I have a lot of types that are just something like "Tuple[int, int]", so type checking doesn't help much. It would be much more useful if I have a value that Python sees as (e.g.) an int, but have the type system track it as something more specific. Something like this: DocId = typing.TypeAlias(int) DocLength = typing.TypeAlias(int) def id_and_length() -> Tuple[DocId, Length]: docid = 5 # type: DocId length = 10 # type: Length return docid, length 3. I can't tell from the PEP484 documentation... If I have a fully hinted base class, and subclass it with the same signature for every method, am I still supposed to hint all the arguments and returns in the subclass? That's what I've been doing, but it's pretty tedious. Cheers, [View Less]

8 18

The prohibition overrides the class method
by Иван Спиненко Nov. 17, 2015

Nov. 17, 2015

Hello, I want to apologize for my English. I propose to make an analogue of the final specifier in C ++. In class abc.ABCMeta has decorator abstractmethod, you can add decorator finalmethod. I have a sketch of implementation: def finalmethod(funcobj): funcobj.__isfinalmethod__ = True return funcobj class Override(ABCMeta): def __init__(cls, name, bases, namespace): super(Override, cls).__init__(name, bases, namespace) finals = {name for name,… [View More]

7 6

Re: [Python-ideas] The prohibition overrides the class method
by Иван Спиненко Nov. 17, 2015

Nov. 17, 2015

Thank you for the constructive responses, perhaps, I agree with you that this is not in the spirit of Python. As for the real example of the use, sometimes there to make the implementation of the method in the base class, but that none of the rest of the development team could not accidentally or unknowingly override this method (perhaps due to the need to be unsuccessful architecture).

7 11

Use `isinstance` check during exception handling
by sjoerdjob＠sjec.nl Nov. 10, 2015

Nov. 10, 2015

Hi all, TL;DR: Change exception checking logic from err.__class__ in exc.__mro__ to isinstance(err, exc) Because it is more powerful, more consistent and allows for cleaner code. The reason I am proposing this change is that quite often, one has to deal with libraries behaving 'badly' with regard to exception granularity. I will take the Python base library before PEP3151 [1] as an example. It is not hard to imagine there are a lot of other cases of libraries that behave with little … [View More]granularity in the exception classes. try: some(operation) except OSError as err: if err.errno == errno.ENOENT: # handle file-not-exists elif err.errno == errno.EACCES: # handle permission stuff. else: raise After PEP3151 one can now write: try: some(operation) except FileNotFoundError: # handle file-not-exists except PermissionError: # handle permission stuff. Now, PEP3151 took over a year to be resolved, judging by the dates. If we have to go through changes like this more often, one will have to wait quite a while. Also, consider backwards compatibility issues. Code using FileNotFoundError will only run on Python3.3+. Thus I propose to have the exception handling code use the normal `isinstance` machinery, as it is more powerful, and allows one to use virtual base classes to check if the instance of the exception is relevant to catch. An example: import errno # Fake `FileNotFoundError` in Python3.2 class ENOENTCheck(type): def __instancecheck__(cls, inst): return isinstance(inst, EnvironmentError) and \ inst.errno == errno.ENOENT class FileNotFoundError(EnvironmentError, metaclass=ENOENTCheck): pass try: open("no-such-file") except Exception as e: print(isinstance(e, FileNotFoundError)) This would print `True`. But, using `FileNotFoundError` as a class to catch will not work, because the `__mro__` is checked instead. I proposed the change on the Python issue tracker [2] already, but it was (rightfully, I think) decided that it needs some more discussion because it will bring a (potentially huge) semantic change. As an opening, here are some pros and cons I can think of: Cons: - *Code already written might break*: Highly unlikely. If you have written code that overrides `isinstance` logic, you'd more than likely not write code that depends on there being a difference in matching logic between the exception handling and normal `isinstance` calls. - *Performance will go down*: Yes, instance-checking is probably a lot more expensive then subclass-checking. I haven't made the change and benchmarked it yet, but based on Martin Panter's comment [3], it might only cost a lot when actually checking against a virtual base class. If cost is limited to those cases, I think the cost is negligible. - *Exception handling like this is just magic*: Yes, anything regarding the metaclasses might be considered as magic. But metaclasses also allow you to build great abstractions, like for instance the ORM included with Django. I think this is one of the cases where the benefits outweigh the cost. Pros: - *Cleaner code*: The same benefit PEP 3151 brought for the base library and OS errors can be extended to all the Python libraries that lack granularity. - *Consistency*: Who would ever expect that you can not always catch an exception `foo` using class `Bar` when `isinstance(foo, Bar)`. Even though I can currently think of more downsides than upsides, I do think that the downsides are really minor, while the upsides are huge: clean code and consistency. Also, I set up a proof-of-concept PEP3151 compatibility layer for Python2.7 that sort-of fakes this feature. It fakes it by checking the instance `sys.exc_info()[1]` in the `__subclasscheck__`. For the proof of concept see [4]. I only found out during porting to Python3 that it does not actually work, because of the direct checking of the `__mro__`, but that's hopefully going to be fixed soon-ish [5]. I'd be interested to hear your thoughts about this. Any more pros and cons would be welcome, as well as your judgement as to whether the breaking of backwards compatibility is potentially a bad idea. Kind regards, Sjoerd Job Postmus [1] PEP 3151 -- Reworking the OS and IO exception hierarchy https://www.python.org/dev/peps/pep-3151/ [2] Call `isinstance` instead of `issubclass` during exception handling http://bugs.python.org/issue25537 [3] http://bugs.python.org/issue25537#msg253963 [4] https://github.com/sjoerdjob/exhacktion/ [5] Catching virtual subclasses in except clauses http://bugs.python.org/issue12029 [View Less]

11 29

Re: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence
by Michael Selik Nov. 5, 2015

Nov. 5, 2015

On 02.10.2015 05:49, Steven D'Aprano wrote: > At most, some have suggested that we don't have a good > word for those iterables which are not iterators. Isn't that a "sequence"? The collections module defines a Sequence as having a __getitem__ and __len__. Is a "non-iterator iterable" something different?

2 2

[Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen
by Michael Selik Nov. 5, 2015

Nov. 5, 2015

Placing current working directory at the beginning of sys.path or the end of sys.path is a tradeoff between two debugging tasks given the error message, `AttributeError: 'module' object has no attribute 'blah'`. The traceback tells you the name of the module. Which would you prefer: (1) needing to notice that you made a module of the same name, or (2) needing to notice that somewhere in the cryptic locations on sys.path there's a module of the same name? I'll pick option 1. Sure, it's tricky … [View More]

1 0

Include partitioning in itertools
by Albert ten Oever Nov. 4, 2015

Nov. 4, 2015

Hi all, I really hope this isn't proposed before - I couldn't find anything in the archives. I want to propose to include a 'partition' (mathematically more correct: a set partition) function in itertools. To my knowledge, partitioning of a set (iterable) is quite a common thing to do and a logic extension of the combinatoric generators in itertools. It is implemented as a Python recipe (http://code.activestate.com/recipes/576795-partitioning-a-sequence/). I found that implementing a partition … [View More]

11 13

Re: [Python-ideas] Include partitioning in itertools
by Albert ten Oever Nov. 3, 2015

Nov. 3, 2015

> [Sven R. Kunze] > > ... > > Not quite sure if I understand the implementation. Reading this > > http://mathworld.wolfram.com/BellNumber.html for the given example of 6, it > > should have 203 possible partitions. What am I missing? > > [Tim Peters] > The ActiveState recipe has nothing to do with set partitions. Instead > it returns all ways of breaking a sequence into a sequence of > subsequences whose sum (catenation) is the original sequence. If &… [View More]

1 0

Re: [Python-ideas] [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen
by Steven D'Aprano Nov. 3, 2015

Nov. 3, 2015

CC'ing Python-Ideas. Follow-ups to Python-Ideas please. On Thu, Oct 29, 2015 at 09:22:15PM -0400, Terry Reedy wrote: > Leaving IDLE aside, the reason '' is added to sys.path is so that people > can import their own modules. This is very useful. Shadowing is the > result of putting it at the front. I have long thought this a dubious > choice. If '' were instead appended, people could still import modules > that did not duplicate stdlib names. Anyone who wanted shadowing … [View More]could > move '' to the front. But then shadowing would be intentional, not an > accident. Terry is right. Shadowing should be possible, and it should require a deliberate decision on the part of the programmer. Consider the shell, say, bash or similar. My understanding is that the shell PATH deliberately excludes the current directory because of the possibility of malicious software shadowing usual commands in /bin etc. If you want to run an executable in the current directory, you have to explicitly provide the path to it: ./myscript rather than just myscript. Now Python isn't exactly the shell, and so I'm not proposing that Python does the same thing. But surely we can agree on the following? - Shadowing explicitly installed packages, including the stdlib, is *occasionally* useful. - But when shadowing occurs, it is *nearly always* accidental. - Such accidental shadowing often causes problems. - And further more, debugging shadowing problems is sometimes tricky even for experienced coders, and almost impossible for beginners. (It's not until you've been burned once or thrice by shadowing that you recognise the symptoms, at which point it is then usually easy to debug.) - Hence, we should put the onus on those who want to shadow installed packages) to do so *explicitly*, or at least make it easier to avoid accidental shadowing. I propose the following two changes: (1) Beginning with Python 3.6, the default is that the current directory is put at the end of sys.path rather than the beginning. Instead of: >>> print(sys.path) ['', '/this', '/that', '/another'] we will have this instead: >>> print(sys.path) ['/this', '/that', '/another', ''] Those who don't shadow installed packages won't notice any difference. Scripts which deliberately or unintentionally shadow installed packages will break from this change. I don't have a problem with this. You can't fix harmful behaviour without breaking code that depends on that harmful behaviour. Additionally, I expect that those who rely on the current behaviour will be in a small minority, much fewer than those who will be bitten by accidental shadowing into the indefinite future. And if you want the old behaviour back, it is easy to do so, by changing the path before doing your imports: import sys if sys.path[-1] == "": sys.path = [""] + sys.path[:-1] or equivalent. I do not belive that it is onerous for those who want shadowing to have to take steps to do so explicitly. That can be added to your scripts on a case-by-case basis, or your PYTHONSTARTUP file, by modifying your site.py, or (I think) by putting the code into the sitecustomize or usercustomize modules. (2) IDLE doesn't need to wait for Python 3.6 to make this change. I believe that IDLE is permitted to make backwards incompatible changes in minor releases, so there is no reason why it can't change the path effective immediately. That's a simpler fix than scanning the entire path, raising warnings (which beginners won't understand and will either ignore or panic over) or other complex solutions. It may not prevent *every* shadowing incident, but it will improve the situation immeasurably. Thoughts? -- Steve [View Less]

11 27