Python-ideas April 2017

python-ideas@python.org

50 participants
24 discussions

Specify number of items to allocate for array.array() constructor

by Sven Rahmann

At the moment, the array module of the standard library allows to create arrays of different numeric types and to initialize them from an iterable (eg, another array). What's missing is the possiblity to specify the final size of the array (number of items), especially for large arrays. I'm thinking of suffix arrays (a text indexing data structure) for large texts, eg the human genome and its reverse complement (about 6 billion characters from the alphabet ACGT). The suffix array is a long int array of the same size (8 bytes per number, so it occupies about 48 GB memory). At the moment I am extending an array in chunks of several million items at a time at a time, which is slow and not elegant. The function below also initializes each item in the array to a given value (0 by default). Is there a reason why there the array.array constructor does not allow to simply specify the number of items that should be allocated? (I do not really care about the contents.) Would this be a worthwhile addition to / modification of the array module? My suggestions is to modify array generation in such a way that you could pass an iterator (as now) as second argument, but if you pass a single integer value, it should be treated as the number of items to allocate. Here is my current workaround (which is slow): def filled_array(typecode, n, value=0, bsize=(1<<22)): """returns a new array with given typecode (eg, "l" for long int, as in the array module) with n entries, initialized to the given value (default 0) """ a = array.array(typecode, [value]*bsize) x = array.array(typecode) r = n while r >= bsize: x.extend(a) r -= bsize x.extend([value]*r) return x

8 months, 1 week

Positional-only parameters

by Victor Stinner

Hi, For technical reasons, many functions of the Python standard libraries implemented in C have positional-only parameters. Example: ------- $ ./python Python 3.7.0a0 (default, Feb 25 2017, 04:30:32) >>> help(str.replace) replace(self, old, new, count=-1, /) # <== notice "/" at the end ... >>> "a".replace("x", "y") # ok 'a' >>> "a".replace(old="x", new="y") # ERR! TypeError: replace() takes at least 2 arguments (0 given) ------- When converting the methods of the builtin str type to the internal "Argument Clinic" tool (tool to generate the function signature, function docstring and the code to parse arguments in C), I asked if we should add support for keyword arguments in str.replace(). The answer was quick: no! It's a deliberate design choice. Quote of Yury Selivanov's message: """ I think Guido explicitly stated that he doesn't like the idea to always allow keyword arguments for all methods. I.e. `str.find('aaa')` just reads better than `str.find(needle='aaa')`. Essentially, the idea is that for most of the builtins that accept one or two arguments, positional-only parameters are better. """ http://bugs.python.org/issue29286#msg285578 I just noticed a module on PyPI to implement this behaviour on Python functions: https://pypi.python.org/pypi/positional My question is: would it make sense to implement this feature in Python directly? If yes, what should be the syntax? Use "/" marker? Use the @positional() decorator? Do you see concrete cases where it's a deliberate choice to deny passing arguments as keywords? Don't you like writing int(x="123") instead of int("123")? :-) (I know that Serhiy Storshake hates the name of the "x" parameter of the int constructor ;-)) By the way, I read that "/" marker is unknown by almost all Python developers, and [...] syntax should be preferred, but inspect.signature() doesn't support this syntax. Maybe we should fix signature() and use [...] format instead? Replace "replace(self, old, new, count=-1, /)" with "replace(self, old, new[, count=-1])" (or maybe even not document the default value?). Python 3.5 help (docstring) uses "S.replace(old, new[, count])". Victor

2 years, 1 month

Implicit string literal concatenation considered harmful?

by Guido van Rossum

I just spent a few minutes staring at a bug caused by a missing comma -- I got a mysterious argument count error because instead of foo('a', 'b') I had written foo('a' 'b'). This is a fairly common mistake, and IIRC at Google we even had a lint rule against this (there was also a Python dialect used for some specific purpose where this was explicitly forbidden). Now, with modern compiler technology, we can (and in fact do) evaluate compile-time string literal concatenation with the '+' operator, so there's really no reason to support 'a' 'b' any more. (The reason was always rather flimsy; I copied it from C but the reason why it's needed there doesn't really apply to Python, as it is mostly useful inside macros.) Would it be reasonable to start deprecating this and eventually remove it from the language? -- --Guido van Rossum (python.org/~guido)

2 years, 7 months

dict(default=int)

by Steven Piantadosi

Hi All, I find importing defaultdict from collections to be clunky and it seems like having a default should just be an optional keyword to dict. Thus, something like, d = dict(default=int) would be the same as from collections import defaultdict d = defaultdict(int) Any thoughts? Thanks, ++Steve

3 years, 5 months

Augmented assignment syntax for objects.

by Erik

Hi. I suspect that this may have been discussed to death at some point in the past, but I've done some searching and I didn't come up with much. Apologies if I'm rehashing an old argument ;) I often find myself writing __init__ methods of the form: def __init__(self, foo, bar, baz, spam, ham): self.foo = foo self.bar = bar self.baz = baz self.spam = spam self.ham = ham This seems a little wordy and uses a lot of vertical space on the screen. Occasionally, I have considered something like: def __init__(self, foo, bar, baz, spam, ham): self.foo, self.bar, self.baz, self.spam, self.ham = \ foo, bar, baz, spam, ham ... just to make it a bit more compact - though in practice, I'd probably not do that with a list quite that long ... two or three items at most: def __init__(self, foo, bar, baz): self.foo, self.bar, self.baz = foo, bar, baz When I do that I'm torn because I know it has a runtime impact to create and unpack the implicit tuples and I'm also introducing a style asymmetry in my code just because of the number of parameters a method happens to have. So why not have an augmented assignment operator for object attributes? It addresses one of the same broad issues that the other augmented assignment operators were introduced for (that of repeatedly spelling names). The suggestion therefore is: def __init__(self, foo, bar, baz, spam, ham): self .= foo, bar, baz, spam, ham This is purely syntactic sugar for the original example: def __init__(self, foo, bar, baz, spam, ham): self.foo = foo self.bar = bar self.baz = baz self.spam = spam self.ham = ham ... so if any of the attributes have setters, then they are called as usual. It's purely a syntactic shorthand. Any token which is not suitable on the RHS of the dot in a standard "obj.attr =" assignment is a syntax error (no "self .= 1"). The comma-separators in the example are not creating a tuple object, they would work at the same level in the parser as the import statement's comma-separated lists - in the same way that "from pkg import a, b, c" is the same as saying: import pkg a = pkg.a b = pkg.b c = pkg.c ... "self .= a, b, c" is the same as writing: self.a = a self.b = b self.c = c E.

3 years, 6 months

Proposed PEP 484 addition: describe a way of annotating decorated declarations

by Guido van Rossum

There's a PR to the peps proposal here: https://github.com/python/peps/pull/242 The full text of the current proposal is below. The motivation for this is that for complex decorators, even if the type checker can figure out what's going on (by taking the signature of the decorator into account), it's sometimes helpful to the human reader of the code to be reminded of the type after applying the decorators (or a stack thereof). Much discussion can be found in the PR. Note that we ended up having `Callable` in the type because there's no rule that says a decorator returns a function type (e.g. `property` doesn't). This is a small thing but I'd like to run it by a larger audience than the core mypy devs who have commented so far. Here's the proposed text (wordsmithing suggestions in the PR please): +Decorators +---------- + +Decorators can modify the types of the functions or classes they +decorate. Use the ``decorated_type`` decorator to declare the type of +the resulting item after all other decorators have been applied:: + + from typing import ContextManager, Iterator, decorated_type + from contextlib import contextmanager + + class DatabaseSession: ... + + @decorated_type(Callable[[str], ContextManager[DatabaseSession]]) + @contextmanager + def session(url: str) -> Iterator[DatabaseSession]: + s = DatabaseSession(url) + try: + yield s + finally: + s.close() + +The argument of ``decorated_type`` is a type annotation on the name +being declared (``session``, in the example above). If you have +multiple decorators, ``decorated_type`` must be topmost. The +``decorated_type`` decorator is invalid on a function declaration that +is also decorated with ``overload``, but you can annotate the +implementation of the overload series with ``decorated_type``. + -- --Guido van Rossum (python.org/~guido <http://python.org/%7Eguido>)

3 years, 6 months

Re: [Python-ideas] Augmented assignment syntax for objects.

by Tin Tvrtković

I'm gonna take a shot at elaborating this point. We Python programmers often tout Python as a high-level, high-productivity language where complex and useful things are easy to do. However, vanilla Python classes are anything but high level; they're basically glorified dictionaries that require an awful lot of boilerplate for basic features. __init__ verbosity is just one of the symptoms of this. I'm going to posit we need declarative classes. (This is what a library like attrs provides, basically.) For a class to be declarative, it needs to be possible to inspect the class for its attributes and more. I'm putting forward three examples. These examples are based on attrs since that's what I consider to be the best way of having declarative classes in Python today. First example. Here's a slide from a recent talk I've given to a local Python meetup: http://tinche.github.io/talks/attrs/#/8/5 This is a class decorator that, when applied to any attrs class, will generate a simple shallow copy() method for that class. This copy method will be faster than copy.copy() and IMO less surprising since it will actually invoke __init__ (and all logic inside). This isn't production code, just a proof of concept for the talk. Second example: I'm working on a library that can turn dictionaries into attrs classes (i.e. turn unstructured data into structured data). Example: http://cattrs.readthedocs.io/en/latest/structuring.html#attrs-classes. When dealing with nested classes the library not only needs to know exactly which fields a class has but also the types of those fields, recursively (for which I'm planning to use attrs metadata). This functionality is useful when preparing data to be stored or sent elsewhere, since serialization libraries and clients in other languages expect json/msgpack etc. Third example: I work at a mobile games company. The backends are Python, the games are written in other languages. The backends and the games share data structures - requests, responses, the save game, game data. I want to have a single source of truth for these structures, so I want to generate classes/structs in other languages from the backend Python classes. All of these examples require declarative classes to do properly. Other approaches are, I suppose, possible in some cases - like inspecting __init__ and whatnot. But I claim the underlying issue is that we should be declaring classes differently, both to enable these new, high-level use cases and because declarative classes are much more readable and contain less boilerplate than what we have now, and this is (for me) Python is about. Message: 1 > Date: Fri, 28 Apr 2017 09:28:55 +1000 > From: Steven D'Aprano <steve(a)pearwood.info> > To: python-ideas(a)python.org > Subject: Re: [Python-ideas] Augmented assignment syntax for objects. > Message-ID: <20170427232853.GC22525(a)ando.pearwood.info> > Content-Type: text/plain; charset=iso-8859-1 > > On Wed, Apr 26, 2017 at 08:52:46PM -0400, Juancarlo A?ez wrote: > > > In my experience, what Python is lacking is a way to declare attributes > > outside of the constructor. Take a look at how it's done in C#, Swisft, > or > > Go. > > Since you apparently already know how they do it, how about telling us > (1) how they do it, and (2) *why* they do it? > > > > Object attributes outside of the constructor would solve things more > > relevant than the vertical space used when assigning constructor > parameters > > to attributes. > > Solve which things? > > > > For example, multiple inheritance is well designed in > > Python, except that it often requires constructors with no parameters, > > which leads to objects with no default attributes. > > Can you elaborate? > > > > -- > Steve > >

3 years, 6 months

Discourage operator.__dunder__ functions

by Steven D'Aprano

Notice that I said *discourage* rather than *deprecate*. Quoting the documentation: The operator module exports a set of efficient functions corresponding to the intrinsic operators of Python. For example, operator.add(x, y) is equivalent to the expression x+y. The function names are those used for special class methods; variants without leading and trailing __ are also provided for convenience. https://docs.python.org/3/library/operator.html I propose four simple documentation changes, and no code change: (1) The quoted paragraph is factually wrong to say: "The function names are those used for special class methods" We can fix that by changing it to: "Some function names are those used for special class methods". (2) Replace the word "convenience" in the quoted paragraph by "backward compatibility"; (3) Add a note close to the top of the page that the non-dunder names are preferred for new code. (4) And a note stating that existing dunder functions will remain, but no new ones will be added. The existing dunder names will remain aliases to the non-dunder names; they will not be deprecated (maybe in Python 5000 *wink*). Those who really want to use them can continue to do so. Regarding point (1) above: - Not all operator functions have a dunder alias. - The dunder functions don't always correspond to a dunder method. For example, there is operator.__concat__ for sequence concatenation, but no str.__concat__ or list.__concat__ methods. - Even when the dunder function corresponds by name to the dunder method, they don't mean the same thing. For example, operator.__add__ is *not* the same as just calling the first argument's __add__ method. - And finally, I fail to see how having to type an extra four characters is a "convenience". Long ago, when the operator module was first introduced, there was a much stronger correspondence between the operator.__dunder__ functions and dunder methods. But I think that correspondence is now so weak that we should simply treat it as a historical artifact. -- Steve

3 years, 6 months

Binary arithmetic does not always call subclasses first

by Stephan Hoyer

I recently filed this as a bug, and was asked to repost to python-dev or python-ideas for greater visibility: http://bugs.python.org/issue30140 Without further ado, here is my original report: --------------------------- We are writing a system for overloading NumPy operations (see PR [1] and design doc [2]) that is designed to copy and extend Python's system for overloading binary operators. The reference documentation on binary arithmetic [3] states: > Note: If the right operand's type is a subclass of the left operand’s type and that subclass provides the reflected method for the operation, this method will be called before the left operand’s non-reflected method. This behavior allows subclasses to override their ancestors’ operations. However, this isn't actually done if the right operand merely inherits from the left operand's type. In practice, CPython requires that the right operand defines a different method before it defers to it. Note that the behavior is different for comparisons, which defer to subclasses regardless of whether they implement a new method [4]. I think this behavior is a mistake and should be corrected. It is just as useful to write generic binary arithmetic methods that are well defined on subclasses as generic comparison operations. In fact, this is exactly the design pattern we propose for objects implementing special operators like NumPy arrays (see NDArrayOperatorsMixin in [1] and [2]). Here is a simple example, of a well-behaved type that implements addition by wrapping its value and that returns NotImplemented when the other operand has the wrong type: class A: def __init__(self, value): self.value = value def __add__(self, other): if not isinstance(other, A): return NotImplemented return type(self)(self.value + other.value) __radd__ = __add__ def __repr__(self): return f'{type(self).__name__}({self.value!r})' class B(A): pass class C(A): def __add__(self, other): if not isinstance(other, A): return NotImplemented return type(self)(self.value + other.value) __radd__ = __add__ A does not defer to subclass B: >>> A(1) + B(1) A(2) But it does defer to subclass C, which defines new methods (literally copied/pasted) for __add__/__radd__: >>> A(1) + C(1) C(2) With the current behavior, special operator implementations need to explicitly account for the possibility that they are being called from a subclass by returning NotImplemented. My guess is that this is rarely done, which means that most of these methods are broken when used with subclasses, or subclasses needlessly reimplement these methods. Can we fix this logic for Python 3.7? [1] https://github.com/numpy/numpy/pull/8247 [2] https://github.com/charris/numpy/blob/406bbc652424fff332f49b0d2f2e5aedd8191… [3] https://docs.python.org/3/reference/datamodel.html#object.__ror__ [4] http://bugs.python.org/issue22052 ----------------------------- Mark Dickinson's response: This is probably worth bringing up on the python-dev or python-ideas mailing lists for greater visibility. I can't think of any plausible non-historical reason why it makes sense for comparisons to behave one way and arithmetic operators another. Altering this might be a PEP-level change, though. The "Coercion rules" section[1] of the Python 2.7 docs is a bit more explicit about the intent: """ Exception to the previous item: if the left operand is an instance of a built-in type or a new-style class, and the right operand is an instance of a proper subclass of that type or class and overrides the base’s __rop__() method, the right operand’s __rop__() method is tried before the left operand’s __op__() method. """ so the check for an override was clearly intentional, rather than an implementation convenience or accident. (It's also clearly intentional in the source and comments.) The 3.x docs don't have the "and overrides" language; I haven't figured out why and when that language changed. [1] https://docs.python.org/release/2.7.6/reference/datamodel.html#coercion-rul…

3 years, 6 months

Re: [Python-ideas] Augmented assignment syntax for objects.

by Tin Tvrtković

You might want to check out attrs (http://attrs.readthedocs.io/en/stable/). It can generate the __init__ for you, and much much more. Date: Tue, 25 Apr 2017 14:33:49 +0200 > From: George Fischhof <george(a)fischhof.hu> > To: Paul Moore <p.f.moore(a)gmail.com> > Cc: Python-Ideas <python-ideas(a)python.org>, "Steven D'Aprano" > <steve(a)pearwood.info> > Subject: Re: [Python-ideas] Augmented assignment syntax for objects. > Message-ID: > < > CAFwcP0gjg1BXeQ2pmv637bXG6+vcXiVu1QNEkdetuHjmoT_yCw(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > 2017. ?pr. 25. de. 10:04 ezt ?rta ("Paul Moore" <p.f.moore(a)gmail.com>): > > On 25 April 2017 at 03:53, Steven D'Aprano <steve(a)pearwood.info> wrote: > > On Tue, Apr 25, 2017 at 02:08:05AM +0100, Erik wrote: > > > >> I often find myself writing __init__ methods of the form: > >> > >> def __init__(self, foo, bar, baz, spam, ham): > >> self.foo = foo > >> self.bar = bar > >> self.baz = baz > >> self.spam = spam > >> self.ham = ham > >> > >> This seems a little wordy and uses a lot of vertical space on the > >> screen. > > > > It does, and while both are annoying, in the grand scheme of things > > they're a very minor annoyance. After all, this is typically only an > > issue once per class, and not even every class, and vertical space is > > quite cheap. In general, the barrier for accepting new syntax is quite > > high, and "minor annoyance" generally doesn't reach it. > > I suspect that with a suitably creative use of inspect.signature() you > could write a decorator for this: > > @auto_attrs > def __init__(self, a, b, c): > # Remaining init code, called with self.a, self.b and self.c set > > I don't have time to experiment right now, but will try to find time > later. If nothing else, such a decorator would be a good prototype for > the proposed functionality, and may well be sufficient for the likely > use cases without needing a syntax change. > > Paul > >

3 years, 6 months

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

Python-ideas April 2017