Mailman 3 November 2015 - Python-ideas

Re: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators
by Henry Schreiner 30 Nov '15

30 Nov '15

It seems to me there are two ideas being discussed. One is adding 2 new operators, something like && and ||, and providing them for some sort of "element wise" or other boolean context, and giving them a better order of evaluations than bitwise ops. This would have the advantage of being usable on the built in python lists/tuples if needed, but otherwise doesn't seem to solve all the issues above, including expressions like -1 < mat < 1, which work for python variables but not for things like Numpy arrays. (for example, selecting values from an array works like this: mat[(-1<mat) & (mat<1)] but not like this: mat[-1 < mat < 1] or this: mat[-1<mat & mat<1] ). The common "general purpose infix notation" pops up here, of course, but I'd consider it a variant of the first idea. The other idea is that overloads could be added to the existing and and or operators. This, I believe, would solve all the main problems, and would not add more to the language for a newcomer to learn. (In fact, most people could forget about the current bitwise and and or, since that's not what most people need, at least those using databases and Numpy). The downside is that the current and and or operators are short-circuiting, and adding a way to overload a short-circuiting operator is tricky/ugly/expensive (though the last one might not be that bad anymore). But, as I think Andrew was mentioning, what about a mixed solution? One that provides a non-short-circuiting overload, but the original short-circuiting behavior is default? It would look like this: Two new magic methods would be added, for now called __booland__ and __boolor__, along with __rbooland__ and __rboolor__. Evaluation would work like this, using and for an example: When x and y is written, if x has __booland__, then x.__booland__(y) is returned. If not, then return x if x==False else if x==True, then if y has y.__rbooland__(x), that is returned. If not, then return y. This would allow object that want custom and/or behavior to implement these magic functions. They would lose (explicit) short-circuiting, but they would gain custom behavior. Most of these special objects do not need or expect the short circuit behavior (Only if an array was all True or False would it even make sense in numpy , and this would allow expressions like -1 < mat < 1 to work as expected in Python. This would be easy to teach to newcomers, too, since only objects that have a definite True/False should short circuit, others have the special methods. The old behavior could be forced on overriding objects by wrapping them in bool(). Would something like that be possible and better than PEP 335? PS: Either type of solution would also be useful for the Plumbum library, by the way, as it would be useful to add and-ing and or-ing to pipelines, but the bitwise operators have a different meaning for Plumbum. Due to the lazy evaluation of Plumbum, even the above solution (or any solution) would still allow short-circuiting in the pipeline. PPS: Hopefully I’m not double posting, apparently the google group and the mailing list are separate subscriptions. Henry Schreiner

1 0

PEP471 - (os.scandir())
by Erik 30 Nov '15

30 Nov '15

PEP471 introduces a faster way of doing low-level directory traversal which is then used to implement and speed up the higher-level API os.walk() - which for me at least is the "go to API" for most directory scanning code I write. However, when using os.walk() the first thing that one tends to do with the results is to analyse them in some way (look at file sizes, datestamps and other things that stat() returns) which is exactly the information that os.scandir() is caching and speeding up but which is then thrown away in order to emulate os.walk()'s original name-based API (well, name and type as the directory/file distinction is also there). So, I'd like to suggest an os.walk()-like API that returns the os.scandir() DirEntry structures rather than names (*). I have my own local version that's just a copy of os.walk() that appends "entry" rather than "entry.name" to the returned lists, but that's a nasty way of achieving this. How to do it - os.walk() "direntries=True" keyword? os.walkentries() function? Something else better than those? Regards, E. (*) I have studied the PEP, followed a lot of the references and looked at the 3.5.0 implementation. I can't see that I've missed such a thing already existing, but it's possible. If so, perhaps this is instead a request to make that thing more obvious somehow!

6 9

The AttributeError/__getattr__ mechanism
by 王珺 29 Nov '15

29 Nov '15

Hello everyone: I'm suggesting a modification to the AttributeError/__getattr__ mechanism, see issue25634: http://bugs.python.org/issue25634 I used __getattr__ sometimes, and descriptor especially property is so widely used. I wonder whether someone had encountered the same problem with me. However, this is a complicated problem including performance issues, and backward compatibility. Thanks for your attention, Jun Wang

5 14

add a single __future__ for py3?
by Michael Scott Cuthbert 28 Nov '15

28 Nov '15

(an old list topic that I accidentally sent to the Google Groups list) One smaller request is to remove 'barry_as_FLUFL' from "__future__.all_feature_names". It's a great practical joke, but it bit me hard on a project where I check that, for Py2/Py3 consistency, each module imports all relevant __future__ names, and ensures that doctests on Py2 are run with them on. I figured that I could iterate through .all_feature_names and turn them all on except unicode_literals and be safe, but suddenly I was getting Syntax errors on all my != tests, and there was not anywhere to turn to for an explanation. Barry might be pleased. In any case it seems better to treat it like "braces" which isn’t listed in "all_feature_names" Best, Myke Cuthbert (long-time-reader-first-time-writer intro: switched from Perl to Python in 2006, wrote music21 (music analysis toolkit) in Python, got tenure at MIT for it. Hugely thankful to the amazing community)

3 2

[Python-ideos] Dedicated overloadable boolean operators
by Jelte Fennema 27 Nov '15

27 Nov '15

Hi, After reading PEP0465 <https://www.python.org/dev/peps/pep-0465/> about the dedicated matrix multiplication I started wondering if the same solution couldn't be applied to boolean operators as well. There currently are a lot of high profile libraries that have their own functions for boolean operators, like Numpy, Pandas or SQLAlchemy. They do this because the current boolean operators can't be overloaded. PEP0335 <https://www.python.org/dev/peps/pep-0335/> was created to solve this problem (and makes the problem more clear), but was rejected because it needed changes to the byte code for the boolean operators, which would make them slower. Currently some of these libraries resort to the bitwise operators (at least Pandas), but those don't bind as strong as comparison operators, which means you have to do comparisons like this: (series1 == 2) & (series2 == 3) That is why I propose to create new operators just like for matrix multiplication which can be used in libraries that need one. I'm not sure what the operators should look like, but my first guess would be &&, || and ! for and, or and not respectively. Is this an idea that sounds reasonable? Jelte

19 56

Re: [Python-ideas] The prohibition overrides the class method
by Иван Спиненко 18 Nov '15

18 Nov '15

Thank you all very much. Perhaps I'll leave that idea, Python too flexible language for such things.

3 3

Lessons from typing hinting Whoosh (PEP484)
by Matt Chaput 17 Nov '15

17 Nov '15

Since PyCharm added support for pep484 I've been working on adding type hints to the Whoosh code base. It makes it a lot more enjoyable to work with the code in the IDE. As I add more and more hints, it's already revealed bugs where arguments were accidentally swapped, or interfaces weren't updated everywhere. I feel much more confident working in the (quite large) codebase with typing. Basically, I'm loving the idea of PEP484. For me, it's a huge improvement in the development experience. However, I've already run into a few issues. I'm not sure if these were discussed during the development of PEP484. 1. The biggest problem, most perplexing problem for me is circular imports. That type hints must actually be imported at the top level is a MASSIVE limitation, at least in a big codebase. I started off importing the types I needed, then switched to the "import the module, specify the type in a string" trick, but even that has failed. In a large/complex codebase, the more things you have to import at the top level (to use for typing), the more potential import circles you create. Now I've hit an impasse where every step in the circle is using the module/string trick, but it doesn't help because I need to subclass something in the circle. For now, I'm going to have to workaround this, either by trying to move "interface" base classes out to their own side package, or just not type hinting some things. Neither is appetizing at all. I think a good solution might be if the typing system could have its own special "imports" that are only used by the typing system, e.g. something like this at the top of a file: # "real" import from foo import bar # imports just for type checking __typeimports__ = """ from baz import Qux """ 2. It would be really nice if we could have "type aliasing" (or whatever it's called). I have a lot of types that are just something like "Tuple[int, int]", so type checking doesn't help much. It would be much more useful if I have a value that Python sees as (e.g.) an int, but have the type system track it as something more specific. Something like this: DocId = typing.TypeAlias(int) DocLength = typing.TypeAlias(int) def id_and_length() -> Tuple[DocId, Length]: docid = 5 # type: DocId length = 10 # type: Length return docid, length 3. I can't tell from the PEP484 documentation... If I have a fully hinted base class, and subclass it with the same signature for every method, am I still supposed to hint all the arguments and returns in the subclass? That's what I've been doing, but it's pretty tedious. Cheers,

8 18

The prohibition overrides the class method
by Иван Спиненко 17 Nov '15

17 Nov '15

Hello, I want to apologize for my English. I propose to make an analogue of the final specifier in C ++. In class abc.ABCMeta has decorator abstractmethod, you can add decorator finalmethod. I have a sketch of implementation: def finalmethod(funcobj): funcobj.__isfinalmethod__ = True return funcobj class Override(ABCMeta): def __init__(cls, name, bases, namespace): super(Override, cls).__init__(name, bases, namespace) finals = {name for name, value in namespace.items() if getattr(value, "__isfinalmethod__", False) } for base in bases: for name in getattr(base, "__finalmethods__", set()): value = getattr(cls, name, None) if getattr(value, "__isfinalmethod__", False): finals.add(name) else: raise TypeError("function '" + str(value.__name__) + "' in class '" + str(cls.__name__) + "' is final") cls.__finalmethods__ = frozenset(finals)

7 6

Re: [Python-ideas] The prohibition overrides the class method
by Иван Спиненко 17 Nov '15

17 Nov '15

Thank you for the constructive responses, perhaps, I agree with you that this is not in the spirit of Python. As for the real example of the use, sometimes there to make the implementation of the method in the base class, but that none of the rest of the development team could not accidentally or unknowingly override this method (perhaps due to the need to be unsuccessful architecture).

7 11

Use `isinstance` check during exception handling
by sjoerdjob＠sjec.nl 10 Nov '15

10 Nov '15

Hi all, TL;DR: Change exception checking logic from err.__class__ in exc.__mro__ to isinstance(err, exc) Because it is more powerful, more consistent and allows for cleaner code. The reason I am proposing this change is that quite often, one has to deal with libraries behaving 'badly' with regard to exception granularity. I will take the Python base library before PEP3151 [1] as an example. It is not hard to imagine there are a lot of other cases of libraries that behave with little granularity in the exception classes. try: some(operation) except OSError as err: if err.errno == errno.ENOENT: # handle file-not-exists elif err.errno == errno.EACCES: # handle permission stuff. else: raise After PEP3151 one can now write: try: some(operation) except FileNotFoundError: # handle file-not-exists except PermissionError: # handle permission stuff. Now, PEP3151 took over a year to be resolved, judging by the dates. If we have to go through changes like this more often, one will have to wait quite a while. Also, consider backwards compatibility issues. Code using FileNotFoundError will only run on Python3.3+. Thus I propose to have the exception handling code use the normal `isinstance` machinery, as it is more powerful, and allows one to use virtual base classes to check if the instance of the exception is relevant to catch. An example: import errno # Fake `FileNotFoundError` in Python3.2 class ENOENTCheck(type): def __instancecheck__(cls, inst): return isinstance(inst, EnvironmentError) and \ inst.errno == errno.ENOENT class FileNotFoundError(EnvironmentError, metaclass=ENOENTCheck): pass try: open("no-such-file") except Exception as e: print(isinstance(e, FileNotFoundError)) This would print `True`. But, using `FileNotFoundError` as a class to catch will not work, because the `__mro__` is checked instead. I proposed the change on the Python issue tracker [2] already, but it was (rightfully, I think) decided that it needs some more discussion because it will bring a (potentially huge) semantic change. As an opening, here are some pros and cons I can think of: Cons: - *Code already written might break*: Highly unlikely. If you have written code that overrides `isinstance` logic, you'd more than likely not write code that depends on there being a difference in matching logic between the exception handling and normal `isinstance` calls. - *Performance will go down*: Yes, instance-checking is probably a lot more expensive then subclass-checking. I haven't made the change and benchmarked it yet, but based on Martin Panter's comment [3], it might only cost a lot when actually checking against a virtual base class. If cost is limited to those cases, I think the cost is negligible. - *Exception handling like this is just magic*: Yes, anything regarding the metaclasses might be considered as magic. But metaclasses also allow you to build great abstractions, like for instance the ORM included with Django. I think this is one of the cases where the benefits outweigh the cost. Pros: - *Cleaner code*: The same benefit PEP 3151 brought for the base library and OS errors can be extended to all the Python libraries that lack granularity. - *Consistency*: Who would ever expect that you can not always catch an exception `foo` using class `Bar` when `isinstance(foo, Bar)`. Even though I can currently think of more downsides than upsides, I do think that the downsides are really minor, while the upsides are huge: clean code and consistency. Also, I set up a proof-of-concept PEP3151 compatibility layer for Python2.7 that sort-of fakes this feature. It fakes it by checking the instance `sys.exc_info()[1]` in the `__subclasscheck__`. For the proof of concept see [4]. I only found out during porting to Python3 that it does not actually work, because of the direct checking of the `__mro__`, but that's hopefully going to be fixed soon-ish [5]. I'd be interested to hear your thoughts about this. Any more pros and cons would be welcome, as well as your judgement as to whether the breaking of backwards compatibility is potentially a bad idea. Kind regards, Sjoerd Job Postmus [1] PEP 3151 -- Reworking the OS and IO exception hierarchy https://www.python.org/dev/peps/pep-3151/ [2] Call `isinstance` instead of `issubclass` during exception handling http://bugs.python.org/issue25537 [3] http://bugs.python.org/issue25537#msg253963 [4] https://github.com/sjoerdjob/exhacktion/ [5] Catching virtual subclasses in except clauses http://bugs.python.org/issue12029

11 29