Mailman 3 October 2014 - Python-ideas

@classproperty, @abc.abstractclasspropery, etc.
by K. Richard Pixley 16 Dec '20

16 Dec '20

There's a whole matrix of these and I'm wondering why the matrix is currently sparse rather than implementing them all. Or rather, why we can't stack them as: class foo(object): @classmethod @property def bar(cls, ...): ... Essentially the permutation are, I think: {'unadorned'|abc.abstract}{'normal'|static|class}{method|property|non-callable attribute}. concreteness implicit first arg type name comments {unadorned} {unadorned} method def foo(): exists now {unadorned} {unadorned} property @property exists now {unadorned} {unadorned} non-callable attribute x = 2 exists now {unadorned} static method @staticmethod exists now {unadorned} static property @staticproperty proposing {unadorned} static non-callable attribute {degenerate case - variables don't have arguments} unnecessary {unadorned} class method @classmethod exists now {unadorned} class property @classproperty or @classmethod;@property proposing {unadorned} class non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract {unadorned} method @abc.abstractmethod exists now abc.abstract {unadorned} property @abc.abstractproperty exists now abc.abstract {unadorned} non-callable attribute @abc.abstractattribute or @abc.abstract;@attribute proposing abc.abstract static method @abc.abstractstaticmethod exists now abc.abstract static property @abc.staticproperty proposing abc.abstract static non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract class method @abc.abstractclassmethod exists now abc.abstract class property @abc.abstractclassproperty proposing abc.abstract class non-callable attribute {degenerate case - variables don't have arguments} unnecessary I think the meanings of the new ones are pretty straightforward, but in case they are not... @staticproperty - like @property only without an implicit first argument. Allows the property to be called directly from the class without requiring a throw-away instance. @classproperty - like @property, only the implicit first argument to the method is the class. Allows the property to be called directly from the class without requiring a throw-away instance. @abc.abstractattribute - a simple, non-callable variable that must be overridden in subclasses @abc.abstractstaticproperty - like @abc.abstractproperty only for @staticproperty @abc.abstractclassproperty - like @abc.abstractproperty only for @classproperty --rich

10 15

Specify number of items to allocate for array.array() constructor
by Sven Rahmann 22 Feb '20

22 Feb '20

At the moment, the array module of the standard library allows to create arrays of different numeric types and to initialize them from an iterable (eg, another array). What's missing is the possiblity to specify the final size of the array (number of items), especially for large arrays. I'm thinking of suffix arrays (a text indexing data structure) for large texts, eg the human genome and its reverse complement (about 6 billion characters from the alphabet ACGT). The suffix array is a long int array of the same size (8 bytes per number, so it occupies about 48 GB memory). At the moment I am extending an array in chunks of several million items at a time at a time, which is slow and not elegant. The function below also initializes each item in the array to a given value (0 by default). Is there a reason why there the array.array constructor does not allow to simply specify the number of items that should be allocated? (I do not really care about the contents.) Would this be a worthwhile addition to / modification of the array module? My suggestions is to modify array generation in such a way that you could pass an iterator (as now) as second argument, but if you pass a single integer value, it should be treated as the number of items to allocate. Here is my current workaround (which is slow): def filled_array(typecode, n, value=0, bsize=(1<<22)): """returns a new array with given typecode (eg, "l" for long int, as in the array module) with n entries, initialized to the given value (default 0) """ a = array.array(typecode, [value]*bsize) x = array.array(typecode) r = n while r >= bsize: x.extend(a) r -= bsize x.extend([value]*r) return x

14 20

Implicit string literal concatenation considered harmful?
by Guido van Rossum 14 Mar '18

14 Mar '18

I just spent a few minutes staring at a bug caused by a missing comma -- I got a mysterious argument count error because instead of foo('a', 'b') I had written foo('a' 'b'). This is a fairly common mistake, and IIRC at Google we even had a lint rule against this (there was also a Python dialect used for some specific purpose where this was explicitly forbidden). Now, with modern compiler technology, we can (and in fact do) evaluate compile-time string literal concatenation with the '+' operator, so there's really no reason to support 'a' 'b' any more. (The reason was always rather flimsy; I copied it from C but the reason why it's needed there doesn't really apply to Python, as it is mostly useful inside macros.) Would it be reasonable to start deprecating this and eventually remove it from the language? -- --Guido van Rossum (python.org/~guido)

51 165

including psutil in the standard library?
by Stefano Borini 11 Nov '14

11 Nov '14

I am wondering if it would be possible to include psutil (https://pypi.python.org/pypi/psutil ) in the standard library, and if not, what would be needed. I am not a developer of it, but I am using psutil at work with good success. it provides a good deal of services for querying and managing processes in a cross platform way. Any thoughts?

12 15

Deprecate dunder functions from operator module
by Steven D'Aprano 04 Nov '14

04 Nov '14

(For the avoidance of all doubt, the following has nothing to do with dunder methods like int.__add__ etc., but only the functions in the operator module.) The operator module contains various function versions of operators, e.g. operator.add for + , operator.sub for - , etc. There are also dunder versions of each function, and according to the documentation: The function names are those used for special class methods; variants without leading and trailing __ are also provided for convenience. https://docs.python.org/3/library/operator.html (except that's not quite right, because there are no functions __radd__ etc. matching the special class methods). The existence of redundant dunder functions came as a complete surprise to me when I recently discovered it, and I daresay it will come as a surprise to many people. Having two versions in the operator module has been known to lead to confusion, e.g.: https://mail.python.org/pipermail/tutor/2014-October/102877.html All the third party documentation I've seen for the operator module refers to the dunderless versions. I doubt that there is anyone who prefers to write operator.__add__ instead of operator.add, and Terry Reedy has kindly determined that (aside from tests) the standard library doesn't use any of the dunder versions. https://mail.python.org/pipermail/python-list/2014-October/679226.html I propose a few things: * institute a policy that, in the event of a new function being added to the operator module, only the dunderless version will be added; * change the documentation to make it clear that the dunderless versions should be used, rather than merely being "convenience" functions; * add a prominent note that the dunder versions exist for backwards compatibility only and should not be used in new code. At this point, I do NOT propose having the dunder versions of the functions raise a depreciation warning. I would be content for them to merely be documented as discouraged, and defer actual depreciation until Python 5000 or so. (But if there is a strong preference for actual deprecation, I won't argue against it.) Thoughts? -- Steven

4 7

"or raise" syntax
by Javier Dehesa 03 Nov '14

03 Nov '14

This happens to me with some frequency: result = f(args) if not result: # of "if result is None:" raise Exception(...) What if I could just say? result = f(args) or raise Exception(...) I think that's pretty idiomatic. What's more, this would give you a shorter syntax for things like: if x <= y: raise Exception(...) Which you could write simply as: x > y or raise Exception(...) Effectively covering the use case of this proposal http://article.gmane.org/gmane.comp.python.ideas/26005/ Has this ever been considered? Javier

4 3

Re: [Python-ideas] Improving memory usage in shared-key attribute dicts
by Hill, Bruce 31 Oct '14

31 Oct '14

On Thu Oct 30 23:42:21 CET 2014, Steven D'Aprano wrote: > How does this compare to the "Alternate Implementation" described in PEP 412? > > http://python.org/dev/peps/pep-0412/#id17 I feel pretty dumb--I didn't read the alternative implementation section. It would be identical. Thanks for catching that. I'm going to try implementing it to see how it pans out.

1 0

Improving memory usage in shared-key attribute dicts
by Hill, Bruce 30 Oct '14

30 Oct '14

Thanks to PEP 412: "Key-Sharing Dictionary", CPython attribute dictionaries can share keys between multiple instances, so the memory cost of new attribute dicts comes primarily from the values array. In the current implementation, the keys array and the values array are always kept to be the same size. This is done so that once the key's location in the key array has been tracked down, the same array offset can be used on the value array to find the value. Rather than storing values in a sparse array of the same size as the keys array, it would make more sense to store values in a compact array. When a dict uses key sharing, there is an unused field in the PyDictKeyEntry struct ("me_value"), which could be repurposed to hold an index into the value array (perhaps by converting "me_value" into a payload union in PyDictKeyEntry). Since the sparse arrays in dictobject.c never use more than (2n+1)/3 of their entries, this change would reduce the memory footprint of each shared-key dict by roughly 1/3 (or more) and also improve data locality. With the current code, the memory usage looks something like: On the class: keys = {NULL, PyDictKeyEntry(key="keyA", value=<unused>), NULL, PyDictKeyEntry(key="keyB", value=<unused>)} On every instance: values = {NULL, "value A", NULL, "value B"} In this example, 50% of the values array is holding NULL pointers. With the proposed changes, it would look more like: On the class: keys = {NULL, PyDictKeyEntry(key="keyA", value_index=0), NULL, PyDictKeyEntry(key="keyB", value_index=1)} On every instance: values = {"value A", "value B"} Here, the memory footprint of instance dicts is cut in half because none of the array is wasted. There's a few implementation details to work out (e.g. uncommon cases like deleting attributes in __init__), but does this seem like a good idea to anyone else?

2 1

operator.call / operator.__call__
by Antony Lee 30 Oct '14

30 Oct '14

A simple suggestion: add "operator.call" and "operator.__call__", which would provide a function like Python2's "apply" (but it'd be called with already unpacked arguments, i.e. operator.call(f, *args, **kwargs)). Why? To be able to pass it as an argument to other function, just like the other functions defined in the "operator" module. Antony

6 7

Re: [Python-ideas] Adding iOS/Android support to Python
by Todd 28 Oct '14

28 Oct '14

On Oct 25, 2014 10:13 AM, "Russell Keith-Magee" <russell(a)keith-magee.com> wrote: > > > On Sat, Oct 25, 2014 at 3:36 PM, Todd <toddrjen(a)gmail.com> wrote: >> >> On Oct 25, 2014 4:22 AM, "Russell Keith-Magee" <russell(a)keith-magee.com> wrote: >> > 3) Disabling certain modules on mobile platforms. Supporting modules like linuxaudiodev, ossaudiodev, readline, curses, idle and tkinter on mobile platforms doesn't make much sense; modules likes bsddb and bz2 are difficult to support due to library dependencies; and the need for modules like multiprocessing is arguable (and difficult to support on mobile). Even providing a Python executable/shell is arguable for these platforms. >> >> I would definitely be extremely interested in a python shell in android. One thing I feel are lacking on android are good advanced mathematical tools and and python shell with appropriate libraries could make a very powerful open-source tool for that. There have been some attempts at that already. > > Yes - and (to the best of my knowledge) none of them provide the default Python shell. They're custom user interfaces, using native system controls, that provide a shell-like UI. What I'm talking about here is the literal "python.exe" build target - the one that is an executable that starts and expects to attach to stdin/stdout. *That* executable isn't practical on Android *or* iOS, because neither platform has the concept of a "console" in the traditional Unix sense of the word. Perhaps no console by default, but it is possible to have a traditional console on android. I have one and many ROMs install one by default. So although it may not be part of the default configuration I think it would be good to support it for the people who do have a console. Further, with rooted users, python could be set up to be used with the built-in adb shell. It is unclear from the discussion where things ultimately came out on this issue. If there still a possibility it might removed, although I understand that consoles are not the primary use-case, I think is still a valid use-case that should supported. >> I would also differentiate android and iOs more. Android seems to be betting on multi-core performance while iOs seems to be betting on single-chore performance. So while multiprocessing may not make much sense on iOs, I think it may be more sense on Android, especially if they move from 4 to 8 cores. > > Firstly - I don't know what gave you the impression Apple devices aren't multicore - every Apple processor since the A5 (introduced in the iPhone 4S and iPad 2) has been at least dual core, and the A8X in the newest iPads is triple core. I was referring to the benchmarks where corresponding iOs and android devices generally have better single and multi-core performance, respectively, but you right that isn't that important. > Secondly, if you're assuming "multicore" automatically means "mathematical powerhouse", you're mistaken. If you're planning on doing serious mathematical heavy lifting on a phone... well, you've already made your first mistake :-) No, on the contrary, I was thinking that on devices with limited performance, being able to divide components between processes, such as UI and logic, is all the more important. It probably would not be any use for the sort of calculator I am thinking about.

4 4

Python-ideas October 2014