Mailman 3 July 2017 - Python-ideas

@classproperty, @abc.abstractclasspropery, etc.
by K. Richard Pixley 16 Dec '20

16 Dec '20

There's a whole matrix of these and I'm wondering why the matrix is currently sparse rather than implementing them all. Or rather, why we can't stack them as: class foo(object): @classmethod @property def bar(cls, ...): ... Essentially the permutation are, I think: {'unadorned'|abc.abstract}{'normal'|static|class}{method|property|non-callable attribute}. concreteness implicit first arg type name comments {unadorned} {unadorned} method def foo(): exists now {unadorned} {unadorned} property @property exists now {unadorned} {unadorned} non-callable attribute x = 2 exists now {unadorned} static method @staticmethod exists now {unadorned} static property @staticproperty proposing {unadorned} static non-callable attribute {degenerate case - variables don't have arguments} unnecessary {unadorned} class method @classmethod exists now {unadorned} class property @classproperty or @classmethod;@property proposing {unadorned} class non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract {unadorned} method @abc.abstractmethod exists now abc.abstract {unadorned} property @abc.abstractproperty exists now abc.abstract {unadorned} non-callable attribute @abc.abstractattribute or @abc.abstract;@attribute proposing abc.abstract static method @abc.abstractstaticmethod exists now abc.abstract static property @abc.staticproperty proposing abc.abstract static non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract class method @abc.abstractclassmethod exists now abc.abstract class property @abc.abstractclassproperty proposing abc.abstract class non-callable attribute {degenerate case - variables don't have arguments} unnecessary I think the meanings of the new ones are pretty straightforward, but in case they are not... @staticproperty - like @property only without an implicit first argument. Allows the property to be called directly from the class without requiring a throw-away instance. @classproperty - like @property, only the implicit first argument to the method is the class. Allows the property to be called directly from the class without requiring a throw-away instance. @abc.abstractattribute - a simple, non-callable variable that must be overridden in subclasses @abc.abstractstaticproperty - like @abc.abstractproperty only for @staticproperty @abc.abstractclassproperty - like @abc.abstractproperty only for @classproperty --rich

10 15

Specify number of items to allocate for array.array() constructor
by Sven Rahmann 22 Feb '20

22 Feb '20

At the moment, the array module of the standard library allows to create arrays of different numeric types and to initialize them from an iterable (eg, another array). What's missing is the possiblity to specify the final size of the array (number of items), especially for large arrays. I'm thinking of suffix arrays (a text indexing data structure) for large texts, eg the human genome and its reverse complement (about 6 billion characters from the alphabet ACGT). The suffix array is a long int array of the same size (8 bytes per number, so it occupies about 48 GB memory). At the moment I am extending an array in chunks of several million items at a time at a time, which is slow and not elegant. The function below also initializes each item in the array to a given value (0 by default). Is there a reason why there the array.array constructor does not allow to simply specify the number of items that should be allocated? (I do not really care about the contents.) Would this be a worthwhile addition to / modification of the array module? My suggestions is to modify array generation in such a way that you could pass an iterator (as now) as second argument, but if you pass a single integer value, it should be treated as the number of items to allocate. Here is my current workaround (which is slow): def filled_array(typecode, n, value=0, bsize=(1<<22)): """returns a new array with given typecode (eg, "l" for long int, as in the array module) with n entries, initialized to the given value (default 0) """ a = array.array(typecode, [value]*bsize) x = array.array(typecode) r = n while r >= bsize: x.extend(a) r -= bsize x.extend([value]*r) return x

14 20

Asynchronous exception handling around with/try statement borders
by Erik Bray 24 Sep '18

24 Sep '18

Hi folks, I normally wouldn't bring something like this up here, except I think that there is possibility of something to be done--a language documentation clarification if nothing else, though possibly an actual code change as well. I've been having an argument with a colleague over the last couple days over the proper way order of statements when setting up a try/finally to perform cleanup of some action. On some level we're both being stubborn I think, and I'm not looking for resolution as to who's right/wrong or I wouldn't bring it to this list in the first place. The original argument was over setting and later restoring os.environ, but we ended up arguing over threading.Lock.acquire/release which I think is a more interesting example of the problem, and he did raise a good point that I do want to bring up. </prologue> My colleague's contention is that given lock = threading.Lock() this is simply *wrong*: lock.acquire() try: do_something() finally: lock.release() whereas this is okay: with lock: do_something() Ignoring other details of how threading.Lock is actually implemented, assuming that Lock.__enter__ calls acquire() and Lock.__exit__ calls release() then as far as I've known ever since Python 2.5 first came out these two examples are semantically *equivalent*, and I can't find any way of reading PEP 343 or the Python language reference that would suggest otherwise. However, there *is* a difference, and has to do with how signals are handled, particularly w.r.t. context managers implemented in C (hence we are talking CPython specifically): If Lock.__enter__ is a pure Python method (even if it maybe calls some C methods), and a SIGINT is handled during execution of that method, then in almost all cases a KeyboardInterrupt exception will be raised from within Lock.__enter__--this means the suite under the with: statement is never evaluated, and Lock.__exit__ is never called. You can be fairly sure the KeyboardInterrupt will be raised from somewhere within a pure Python Lock.__enter__ because there will usually be at least one remaining opcode to be evaluated, such as RETURN_VALUE. Because of how delayed execution of signal handlers is implemented in the pyeval main loop, this means the signal handler for SIGINT will be called *before* RETURN_VALUE, resulting in the KeyboardInterrupt exception being raised. Standard stuff. However, if Lock.__enter__ is a PyCFunction things are quite different. If you look at how the SETUP_WITH opcode is implemented, it first calls the __enter__ method with _PyObjet_CallNoArg. If this returns NULL (i.e. an exception occurred in __enter__) then "goto error" is executed and the exception is raised. However if it returns non-NULL the finally block is set up with PyFrame_BlockSetup and execution proceeds to the next opcode. At this point a potentially waiting SIGINT is handled, resulting in KeyboardInterrupt being raised while inside the with statement's suite, and finally block, and hence Lock.__exit__ are entered. Long story short, because Lock.__enter__ is a C function, assuming that it succeeds normally then with lock: do_something() always guarantees that Lock.__exit__ will be called if a SIGINT was handled inside Lock.__enter__, whereas with lock.acquire() try: ... finally: lock.release() there is at last a small possibility that the SIGINT handler is called after the CALL_FUNCTION op but before the try/finally block is entered (e.g. before executing POP_TOP or SETUP_FINALLY). So the end result is that the lock is held and never released after the KeyboardInterrupt (whether or not it's handled somehow). Whereas, again, if Lock.__enter__ is a pure Python function there's less likely to be any difference (though I don't think the possibility can be ruled out entirely). At the very least I think this quirk of CPython should be mentioned somewhere (since in all other cases the semantic meaning of the "with:" statement is clear). However, I think it might be possible to gain more consistency between these cases if pending signals are checked/handled after any direct call to PyCFunction from within the ceval loop. Sorry for the tl;dr; any thoughts?

7 15

Positional-only parameters
by Victor Stinner 11 Sep '18

11 Sep '18

Hi, For technical reasons, many functions of the Python standard libraries implemented in C have positional-only parameters. Example: ------- $ ./python Python 3.7.0a0 (default, Feb 25 2017, 04:30:32) >>> help(str.replace) replace(self, old, new, count=-1, /) # <== notice "/" at the end ... >>> "a".replace("x", "y") # ok 'a' >>> "a".replace(old="x", new="y") # ERR! TypeError: replace() takes at least 2 arguments (0 given) ------- When converting the methods of the builtin str type to the internal "Argument Clinic" tool (tool to generate the function signature, function docstring and the code to parse arguments in C), I asked if we should add support for keyword arguments in str.replace(). The answer was quick: no! It's a deliberate design choice. Quote of Yury Selivanov's message: """ I think Guido explicitly stated that he doesn't like the idea to always allow keyword arguments for all methods. I.e. `str.find('aaa')` just reads better than `str.find(needle='aaa')`. Essentially, the idea is that for most of the builtins that accept one or two arguments, positional-only parameters are better. """ http://bugs.python.org/issue29286#msg285578 I just noticed a module on PyPI to implement this behaviour on Python functions: https://pypi.python.org/pypi/positional My question is: would it make sense to implement this feature in Python directly? If yes, what should be the syntax? Use "/" marker? Use the @positional() decorator? Do you see concrete cases where it's a deliberate choice to deny passing arguments as keywords? Don't you like writing int(x="123") instead of int("123")? :-) (I know that Serhiy Storshake hates the name of the "x" parameter of the int constructor ;-)) By the way, I read that "/" marker is unknown by almost all Python developers, and [...] syntax should be preferred, but inspect.signature() doesn't support this syntax. Maybe we should fix signature() and use [...] format instead? Replace "replace(self, old, new, count=-1, /)" with "replace(self, old, new[, count=-1])" (or maybe even not document the default value?). Python 3.5 help (docstring) uses "S.replace(old, new[, count])". Victor

19 27

Implicit string literal concatenation considered harmful?
by Guido van Rossum 14 Mar '18

14 Mar '18

I just spent a few minutes staring at a bug caused by a missing comma -- I got a mysterious argument count error because instead of foo('a', 'b') I had written foo('a' 'b'). This is a fairly common mistake, and IIRC at Google we even had a lint rule against this (there was also a Python dialect used for some specific purpose where this was explicitly forbidden). Now, with modern compiler technology, we can (and in fact do) evaluate compile-time string literal concatenation with the '+' operator, so there's really no reason to support 'a' 'b' any more. (The reason was always rather flimsy; I copied it from C but the reason why it's needed there doesn't really apply to Python, as it is mostly useful inside macros.) Would it be reasonable to start deprecating this and eventually remove it from the language? -- --Guido van Rossum (python.org/~guido)

51 165

HTTP compression support for http.server
by Pierre Quentel 07 Aug '17

07 Aug '17

I have reported an issue in the tracker (https://bugs.python.org/issue30576) and proposed a Pull Request on the Github CPython repository ( https://github.com/python/cpython/pull/2078) to make http.server in the standard library support HTTP compression (gzip). I have been suggested to require feedback from core devs : - should HTTP compression be supported ? - if so, should it be supported by default ? It is the case in the PR, where a number of content types, eg text/html, are compressed if the user agent accepts the gzip "encoding" - if not, should the implementation of http.server be adapted so that subclasses could implement it ? For the moment the only way to add it is to modify method send_head() of SimpleHTTPRequestHandler My opinion is that it should be supported : http.server is not meant to be a full-featured HTTP server, but compression it is a basic and normalized feature of HTTP servers, it is supported by most browsers including on smartphones, it reduces network load, and it's very easy to implement (cf. the Pull Request). For the same reason, I recently added browser cache to http.server (PR #298 <https://github.com/python/cpython/pull/298>). I also think that it should be supported by default for the most common content types (text/html, text/css, application/javascript...) ; the implementation is based on a list of types to compress (SimpleHTTPServer.compressed_types) that can be modified at will, eg set to the empty list to disable compression.

5 5

namedtuple literals [Was: RE a new namedtuple]
by Giampaolo Rodola' 31 Jul '17

31 Jul '17

On Tue, Jul 18, 2017 at 6:31 AM, Guido van Rossum <guido(a)python.org> wrote: > On Mon, Jul 17, 2017 at 6:25 PM, Eric Snow <ericsnowcurrently(a)gmail.com> > wrote: > >> On Mon, Jul 17, 2017 at 6:01 PM, Ethan Furman <ethan(a)stoneleaf.us> wrote: >> > Guido has decreed that namedtuple shall be reimplemented with speed in >> mind. >> >> FWIW, I'm sure that any changes to namedtuple will be kept as minimal >> as possible. Changes would be limited to the underlying >> implementation, and would not include the namedtuple() signature, or >> using metaclasses, etc. However, I don't presume to speak for Guido >> or Raymond. :) >> > > Indeed. I referred people here for discussion of ideas like this: > > >>> a = (x=1, y=0) > Thanks for bringing this up, I'm gonna summarize my idea in form of a PEP-like draft, hoping to collect some feedback. Proposal ======== Introduction of a new syntax and builtin function to create lightweight namedtuples "on the fly" as in: >>> (x=10, y=20) (x=10, y=20) >>> ntuple(x=10, y=20) (x=10, y=20) Motivations =========== Avoid declaration ----------------- Other than the startup time cost: https://mail.python.org/pipermail/python-dev/2017-July/148592.html ...the fact that namedtuples need to be declared upfront implies they mostly end up being used only in public, end-user APIs / functions. For generic functions returning more than 1 argument it would be nice to just do: def get_coordinates(): return (x=10, y=20) ...instead of: from collections import namedtuple Coordinates = namedtuple('coordinates', ['x', 'y']) def get_coordinates(): return Coordinates(10, 20) Declaration also has the drawback of unnecessarily polluting the module API with an object (Coordinates) which is rarely needed. AFAIU namedtuple was designed this way for efficiency of the pure-python implementation currently in place and for serialization purposes (e.g. pickle), but I may be missing something else. Generally namedtuples are declared in a private module, imported from elsewhere and they are never exposed in the main namespace, which is kind of annoying. In case of one module scripts it's not uncommon to add a leading underscore which makes __repr__ uglier. To me, this suggests that the factory function should have been a first-class function instead. Speed ------ Other than the startup declaration overhead, a namedtuple is slower than a tuple or a C structseq in almost any aspect: - Declaration (50x slower than cnamedtuple): $ python3.7 -m timeit -s "from collections import namedtuple" \ "namedtuple('Point', ('x', 'y'))" 1000 loops, best of 5: 264 usec per loop $ python3.7 -m timeit -s "from cnamedtuple import namedtuple" \ "namedtuple('Point', ('x', 'y'))" 50000 loops, best of 5: 5.27 usec per loop - Instantiation (3.5x slower than tuple): $ python3.7 -m timeit -s "import collections; Point = collections.namedtuple('Point', ('x', 'y')); x = [1, 2]" "Point(*x)" 1000000 loops, best of 5: 310 nsec per loop $ python3.7 -m timeit -s "x = [1, 2]" "tuple(x)" 5000000 loops, best of 5: 88 nsec per loop - Unpacking (2.8x slower than tuple): $ python3.7 -m timeit -s "import collections; p = collections.namedtuple( \ 'Point', ('x', 'y'))(5, 11)" "x, y = p" 5000000 loops, best of 5: 41.9 nsec per loop $ python3.7 -m timeit -s "p = (5, 11)" "x, y = p" 20000000 loops, best of 5: 14.8 nsec per loop - Field access by name (1.9x slower than structseq and cnamedtuple): $ python3.7 -m timeit -s "from collections import namedtuple as nt; \ p = nt('Point', ('x', 'y'))(5, 11)" "p.x" 5000000 loops, best of 5: 42.7 nsec per loop $ python3.7 -m timeit -s "from cnamedtuple import namedtuple as nt; \ p = nt('Point', ('x', 'y'))(5, 11)" "p.x" 10000000 loops, best of 5: 22.5 nsec per loop $ python3.7 -m timeit -s "import os; p = os.times()" "p.user" 10000000 loops, best of 5: 22.6 nsec per loop - Field access by index is the same as tuple: $ python3.7 -m timeit -s "from collections import namedtuple as nt; \ p = nt('Point', ('x', 'y'))(5, 11)" "p[0]" 10000000 loops, best of 5: 20.3 nsec per loop $ python3.7 -m timeit -s "p = (5, 11)" "p[0]" 10000000 loops, best of 5: 20.5 nsec per loop It is being suggested that most of these complaints about speed aren't an issue but in certain circumstances such as busy loops, getattr() being 1.9x slower could make a difference, e.g.: https://github.com/python/cpython/blob/3e2ad8ec61a322370a6fbdfb2209cf74546f… Same goes for values unpacking. isinstance() ------------ Probably a minor complaint, I just bring this up because I recently had to do this in psutil's unit tests. Anyway, checking a namedtuple instance isn't exactly straightforward: https://stackoverflow.com/a/2166841 Backward compatibility ====================== This is probably the biggest barrier other than the "a C implementation is less maintainable" argument. In order to avoid duplication of functionality it would be great if collections.namedtuple() could remain a (deprecated) factory function using ntuple() internally. FWIW I tried running stdlib's unittests against https://github.com/llllllllll/cnamedtuple, I removed the ones about "_source", "verbose" and "module" arguments and I get a couple of errors about __doc__. I'm not sure about more advanced use cases (subclassing, others...?) but overall it appears pretty doable. collections.namedtuple() Python wrapper can include the necessary logic to implement "verbose" and "rename" parameters when they're used. I'm not entirely sure about the implications of the "module" parameter though (Raymond?). _make(), _asdict(), _replace() and _fields attribute should also be exposed; as for "_source" it appears it can easily be turned into a property which would also save some memory. The biggest annoyance is probably fields' __doc__ assignment: https://github.com/python/cpython/blob/ced36a993fcfd1c76637119d31c03156a877… ...which would require returning a clever class object slowing down the namedtuple declaration also in case no parameters are passed, but considering that the long-term plan is the replace collections.namedtuple() with ntuple() I consider this acceptable. Thoughts? -- Giampaolo - http://grodola.blogspot.com

32 81

namedtuple nit...
by Chris Barker 29 Jul '17

29 Jul '17

Since we are talking about namedtuple and implementation, I just noticed: In [22]: Point = namedtuple('Point', ['x', 'y']) In [23]: p = Point(2,3) In [24]: p.x = 5 --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) <ipython-input-24-328d3fab3e30> in <module>() ----> 1 p.x = 5 AttributeError: can't set attribute OK -- that makes sense. but then, if you try: In [25]: p.z = 5 --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) <ipython-input-25-625ed954d865> in <module>() ----> 1 p.z = 5 AttributeError: 'Point' object has no attribute 'z' I think this should be a different message -- key here is that you can't set a new attribute, not that one doesn't exist. Maybe: "AttributeError: can't set new attribute" -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

7 11

Idea : for smarter assignment?
by Jason H 25 Jul '17

25 Jul '17

I experimented with Python in college and I've been for close to 20 years now. (Coming and going as needed) I love the language. But there is one annoyance that I continually run into. There are basically two assignment operators, based on context, = and : a = 1 { a: 1 } They cannot be used interchangeably: a: 1 # error {a=1} # error I don't think I should be this way. There are times when I have a bunch of variables that I want to collect into an object or destructure. This involves adding commas, and swapping the :/=. I don't have a good fix for the adding of commas (maybe a newline?) but I think : should at least be accepted as = everywhere except in ifs: a: 1 # same as a = 1 One area where it might help (although the python parser already catches it) is in ifs: if a:1 # always error ? if a=1 # currently error, but might be accepted shorthand for == ? Ideally, I could take a: 1 b: 2 then in 3 edits: 1. first line prepend 'x: {' 2. last line append '}' 3. indent between { } I guess his would imply that { open up an assignment scope, where newlines are commas if the last line did not end with an operator or the next line did not start with an operator: x: { a: x - # - operator f(x) b: # : operator 5487234728394720348988734574357 c: 7 # c is 13, no trailing operator but next line has a preceding operator + 6 } The only issue then is how do we address a? x.a # looks fine to me x['a'] # as a dict, but the conversion of a to string 'a' could be confusing. Additionally, I was also thinking about : as an implied await: a = await f() # await generator a: f() # await generator, or direct assignment if return type is not a generator/async func Thoughts? Please be gentle :-)

9 8

Re: [Python-ideas] namedtuple literals [Was: RE a new namedtuple]
by C Anthony Risinger 25 Jul '17

25 Jul '17

On Jul 23, 2017 1:56 PM, "MRAB" <python(a)mrabarnett.plus.com> wrote: On 2017-07-23 17:08, Todd wrote: > On Jul 20, 2017 1:13 AM, "David Mertz" <mertz(a)gnosis.cx <mailto: > mertz(a)gnosis.cx>> wrote: > > I'm concerned in the proposal about losing access to type > information (i.e. name) in this proposal. For example, I might > write some code like this now: > > >>> from collections import namedtuple > >>> Car = namedtuple("Car", "cost hp weight") > >>> Motorcycle = namedtuple("Motorcycle", "cost hp weight") > >>> smart = Car(18_900, 89, 949) > >>> harley = Motorcyle(18_900, 89, 949) > >>> if smart==harley and type(smart)==type(harley): > ... print("These are identical vehicles") > > The proposal to define this as: > > >>> smart = (cost=18_900, hp=89, weight=949) > >>> harley = (cost=18_900, hp=89, weight=949) > > Doesn't seem to leave any way to distinguish the objects of > different types that happen to have the same fields. Comparing > `smart._fields==harley._fields` doesn't help here, nor does any type > constructed solely from the fields. > > > What about making a syntax to declare a type? The ones that come to mind > are > > name = (x=, y=) > > Or > > name = (x=pass, y=pass) > > They may not be clear enough, though. > > Guido has already declared that he doesn't like those bare forms, so it'll probably be something like ntuple(...). Not exactly a literal in that case. If this is true, why not simply add keyword arguments to tuple(...)? Something like (a=1, b=2, c=3) makes very clear sense to me, or even (1, 2, c=3), where the first two are accessible by index only. Or even (1, 2, c: 3), which reminds me of Elixir's expansion of tuple and list keywords. A tuple is a tuple is a tuple. No types. Just convenient accessors. -- C Anthony

3 4

Python-ideas July 2017