Mailman 3 February 2014 - Python-ideas

@classproperty, @abc.abstractclasspropery, etc.
by K. Richard Pixley 16 Dec '20

16 Dec '20

There's a whole matrix of these and I'm wondering why the matrix is currently sparse rather than implementing them all. Or rather, why we can't stack them as: class foo(object): @classmethod @property def bar(cls, ...): ... Essentially the permutation are, I think: {'unadorned'|abc.abstract}{'normal'|static|class}{method|property|non-callable attribute}. concreteness implicit first arg type name comments {unadorned} {unadorned} method def foo(): exists now {unadorned} {unadorned} property @property exists now {unadorned} {unadorned} non-callable attribute x = 2 exists now {unadorned} static method @staticmethod exists now {unadorned} static property @staticproperty proposing {unadorned} static non-callable attribute {degenerate case - variables don't have arguments} unnecessary {unadorned} class method @classmethod exists now {unadorned} class property @classproperty or @classmethod;@property proposing {unadorned} class non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract {unadorned} method @abc.abstractmethod exists now abc.abstract {unadorned} property @abc.abstractproperty exists now abc.abstract {unadorned} non-callable attribute @abc.abstractattribute or @abc.abstract;@attribute proposing abc.abstract static method @abc.abstractstaticmethod exists now abc.abstract static property @abc.staticproperty proposing abc.abstract static non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract class method @abc.abstractclassmethod exists now abc.abstract class property @abc.abstractclassproperty proposing abc.abstract class non-callable attribute {degenerate case - variables don't have arguments} unnecessary I think the meanings of the new ones are pretty straightforward, but in case they are not... @staticproperty - like @property only without an implicit first argument. Allows the property to be called directly from the class without requiring a throw-away instance. @classproperty - like @property, only the implicit first argument to the method is the class. Allows the property to be called directly from the class without requiring a throw-away instance. @abc.abstractattribute - a simple, non-callable variable that must be overridden in subclasses @abc.abstractstaticproperty - like @abc.abstractproperty only for @staticproperty @abc.abstractclassproperty - like @abc.abstractproperty only for @classproperty --rich

10 15

Specify number of items to allocate for array.array() constructor
by Sven Rahmann 21 Feb '20

21 Feb '20

At the moment, the array module of the standard library allows to create arrays of different numeric types and to initialize them from an iterable (eg, another array). What's missing is the possiblity to specify the final size of the array (number of items), especially for large arrays. I'm thinking of suffix arrays (a text indexing data structure) for large texts, eg the human genome and its reverse complement (about 6 billion characters from the alphabet ACGT). The suffix array is a long int array of the same size (8 bytes per number, so it occupies about 48 GB memory). At the moment I am extending an array in chunks of several million items at a time at a time, which is slow and not elegant. The function below also initializes each item in the array to a given value (0 by default). Is there a reason why there the array.array constructor does not allow to simply specify the number of items that should be allocated? (I do not really care about the contents.) Would this be a worthwhile addition to / modification of the array module? My suggestions is to modify array generation in such a way that you could pass an iterator (as now) as second argument, but if you pass a single integer value, it should be treated as the number of items to allocate. Here is my current workaround (which is slow): def filled_array(typecode, n, value=0, bsize=(1<<22)): """returns a new array with given typecode (eg, "l" for long int, as in the array module) with n entries, initialized to the given value (default 0) """ a = array.array(typecode, [value]*bsize) x = array.array(typecode) r = n while r >= bsize: x.extend(a) r -= bsize x.extend([value]*r) return x

14 20

Implicit string literal concatenation considered harmful?
by Guido van Rossum 14 Mar '18

14 Mar '18

I just spent a few minutes staring at a bug caused by a missing comma -- I got a mysterious argument count error because instead of foo('a', 'b') I had written foo('a' 'b'). This is a fairly common mistake, and IIRC at Google we even had a lint rule against this (there was also a Python dialect used for some specific purpose where this was explicitly forbidden). Now, with modern compiler technology, we can (and in fact do) evaluate compile-time string literal concatenation with the '+' operator, so there's really no reason to support 'a' 'b' any more. (The reason was always rather flimsy; I copied it from C but the reason why it's needed there doesn't really apply to Python, as it is mostly useful inside macros.) Would it be reasonable to start deprecating this and eventually remove it from the language? -- --Guido van Rossum (python.org/~guido)

51 165

Making sys.std* thread-local
by Antony Lee 31 Mar '14

31 Mar '14

The recent discussion about lexical vs. dynamic scoping reminded me of the following issue, that plagues e.g. the new contextlib.redirect_stdout context manager: the standard file-like objects sys.std* are global instead of thread-local, making their manipulation unsafe in multithreaded programs (this issue is related to lexical vs dynamic scoping because in a language with (optional) dynamic scoping, these objects can be made dynamically scoped, thus solving the issue). Of course, changing sys.std* to being thread-local would be backwards incompatible, but perhaps some new objects, e.g. sys.thread_local_std*, could be added to sys, with print() and related functions using sys.thread_local_std* if it is set and sys.std* otherwise. Antony

3 3

Enhance exceptions by attaching some more information to them
by Sebastian Kreft 08 Mar '14

08 Mar '14

More than once I've been in a situation where I wish that some of the stdlib exceptions had a better message or some more information to help me diagnose the problem. For example: a = [1, 2, 3, 4, 5] a[5] IndexError: list index out of range In this case there's no reference to neither the length of the array nor to the offending index. I'm of the idea that we could extend the exceptions by adding some more information to them, so 3rd party libraries could use them for debugging/logging. For example, one such use case would be to have a modified test runner, that in case of exceptions automatically prints some debug information. Another would be a logger system that in case of an exception also logs some debug info that could be relevant to understand and solve the issue. I propose extending (at least) the following exceptions with the following attributes: KeyError: key, object IndexError: index, object AttributeError: attribute, object NameError: name Of course that populating these attributes could be controlled by a flag. I know that some of this information is already present in some exceptions, depending on the context. However, I propose adding these attributes, as in this way a tool could easily and reliably extract the information and work with it, as opposed to have to parse the huge variety of different messages there are. For the first use case mentioned above I have a working prototype, although it does not use this proposal, but a series of hacks (I'm modifying the bytecode to save a reference to the key and object :() and parsing of the exception messages. But I want to share what the output of such a tool could be. ====================================================================== ERROR: test_attribute (example.ExampleTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/skreft/test/debug_exception/example.py.py", line 18, in test_attribute AttributeError: 'str' object has no attribute 'Lower'. Did you mean 'islower', 'lower'? Debug info: Object: '' Type: <type 'str'> ====================================================================== ERROR: test_index (example.ExampleTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/skreft/test/debug_exception/example.py.py", line 6, in test_index IndexError: list index out of range Debug info: Object: [1, 2] Object len: 2 Index: 2 ====================================================================== ERROR: test_key (example.ExampleTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/skreft/test/debug_exception/example.py.py", line 10, in test_key KeyError_: 'fooo', did you mean 'foo'? Debug info: Object: {'foo': 1} Key: 'fooo' ====================================================================== ERROR: test_name (example.ExampleTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/skreft/test/debug_exception/example.py.py", line 14, in test_name NameError: global name 'fooo' is not defined. Did you mean 'foo'? ---------------------------------------------------------------------- Ran 4 tests in 0.005s -- Sebastian Kreft

4 4

One more time... lambda function <--- from *** signature def.
by Ron Adam 06 Mar '14

06 Mar '14

Starting new thread because this bike has a different shape and color. Yesterday I was thinking that just making the keyword lambda assignable like True, False, and None, would be enough. But the issue with that is lambda isn't a name to an actual object or type. That was the seed for this idea. How to get lambda like functionality into some sort of object that would be easy to use and explain. This morning I thought we could have in a functions definition something, like "*", and "**", to take an expression. Similar to Nicks idea with =:, but more general. The idea is to have "***" used in def mean to take "any" call expression and not evaluate it until *** is used on it. ie... the same rules as *. When used in a def to pack a tuple, and when used outside def, to unpack it. So, "***" used in a def, stores the call expression, at call time, and when used later, expresses it. A function call that captures an expression may be tricky to do. Here's one approach that requires sugar when a function defined with "***" is called. class TriStar: def __init__(self, expr): """ expr is a callable that takes no arguments. """ self.expr = expr def __tristar__(self): """ ***obj --> result """ return self.expr() def fn(***expr):... (Any other suggestions for how to do this would be good.) And at call time.... fn(...) --> fn(TriStar(expr=lambda:...)) So presuming we can do something like the above, the first case is ... def star_fn(***expr) return ***expr ... = star_fn(...) Which is a function that just returns whatever it's input is, and is even more general than using *args, **kwds. The call signature stored in expr isn't evaluated until it's returned with ***expr. So the evaluation is delayed, or lazy, but it's still explicit and very easy to read. This returns a lambda-like function. def star_lambda(***expr): return expr And is used this way... result = star_lambda(a * b + c) # captures expression. actual_result = ***result # *** resolves "result" here! The resolution is done with ***name, rather than name(). That's actually very good because it can pass through callable tests. So you can safely pass callable objects around without them getting called at the wrong time or place. We can shorten the name because star_lambda is just a function. L = star_lambda To me this is an exceptionally clean solution. Easy to use, and not to hard to explain. Seems a lot more like a python solution to me as well. Hoping it doesn't get shot down too quickly, Ron ;-)

11 40

PEP-257: drop recommendation for extra line at end of multi-line docstring
by Ziad Sawalha 03 Mar '14

03 Mar '14

PEP-257 includes this recommendation: “The BDFL [3] recommends inserting a blank line between the last paragraph in a multi-line docstring and its closing quotes, placing the closing quotes on a line by themselves. This way, Emacs' fill-paragraph command can be used on it.” I believe emacs no longer has this limitation. "If you do fill-paragraph in emacs in Python mode within a docstring, emacs already ignores the closing triple-quote. In fact, the most recent version of emacs supports several different docstring formatting styles and gives you the ability to switch between them.” - quoting Kevin L. Mitchell who is more familiar with emacs than I am. I’m considering removing that recommendation and updating some of the examples in PEP-257, but I’d like some thoughts from this group before I submit the patch. Any thoughts or references to conversations that may have already been had on this topic? Regards, Ziad

6 13

A python bridge between versions
by ian o 02 Mar '14

02 Mar '14

The most quoted reason for staying with python 2 is that some required library is not available to support python 3. All it takes is one legacy library module in python 2 to keep an entire project in python 2. Many significant projects (e.g web2py) continue in python 2 for this reason. Even projects that produce code that can work under either python 2 or python 3, are themselves trapped into only using python 2 features. This is really holding back python! You cannot just import python2 code into the python 3 interpreter as it is not compatible. But could this be solved by a new approach which is to treat python 2 as another language and not directly import the code but handle python 2 code with an an extension module, which acts as bridge between versions. I mean, python three program can access C/C++ code (which is not python3), why can't the access python 2 code by treating it like C/C++? This would require an extension C module called from python 3. How to do this is well known. This extension C module would itself call the required module library(s) from python2. Calling python from C is also a well known technique. Ptyhon can call C. C can call python. So python3 call C which calls python2. So in python3 imput <python2-module> # this usually won't work because of language changes becomes import python2 <python2-module> = python2.import('<python2-module>') The resultant imported module would appear as a c++ extension to python3 and in effect be a wrapper for the python2 module using the python2 extension to execute the code This would mean programs using python2 imports have the overhead of both interpreters running in memory and calls across the python3/python2 boundary are 'wrapped' as c++ calls that then activate the required code within python 2. In reality quite a small overhead on modern computers. python2.sys.path (within the 'python2' module) or similar would be needed for the separate import space running inside the python2 boundary. Callbacks back across the python3/python2 boundary back into python3 are a further complication but several possible solutions exist to deliver this. Not many library modules require callback functions to the library module itself so this not the central issue. There is some work for the person coding the python 3 app in terms of a different syntax for legacy module imports and potentially setting python 2 environment values (like python2.sys.path), but not a lot and the python3 is the new program being worked on as opposed to the legacy library which can remain untouched. I think this type of approach could change the recommendation on 'python 2 or python 3' to a simple 'get the latest available to you and here is how to use legacy libraries if you have the need'. Thoughts or criticisms?

14 29

Re: [Python-ideas] Update the required C compiler for Windows to a supported version.
by Steve Dower 02 Mar '14

02 Mar '14

Stephen J. Turnbull wrote: > Vernon D. Cole writes: > >> I cannot compile a Python extension module with any Microsoft compiler >> I can obtain. > > Your pain is understood, but it's not simple to address it. FWIW, I'm working on making the compiler easily obtainable. The VS 2008 link that was posted is unofficial, and could theoretically disappear at any time (I'm not in control of that), but the Windows SDK for Windows 7 and .NET 3.5 SP1 (http://www.microsoft.com/en-us/download/details.aspx?id=3138) should be around for as long as Windows 7 is supported. The correct compiler (VC9) is included in this SDK, but unfortunately does not install the vcvarsall.bat file that distutils expects. (Though it's pretty simple to add one that will switch on %1 and call the correct vcvars(86|64|...).bat.) The SDK needed for Python 3.3 and 3.4 (VC10) is even worse - there are many files missing. I'm hoping we'll be able to set up some sort of downloadable package/tool that will fix this. While we'd obviously love to move CPython onto our latest compilers, it's simply not possible (for good reason). Python 3.4 is presumably locked to VC10, but hopefully 3.5 will be able to use whichever version is current when that decision is made. > The basic problem is that the ABI changes. Therefore it's going to require > a complete new set of *all* C extensions for Windows, and the duplication > of download links for all those extensions from quite a few different vendors > is likely to confuse a lot of users. Specifically, the CRT changes. The CRT is an interesting mess of data structures that are exposed in header files, which means while you can have multiple CRTs loaded, they cannot touch each other's data structures at all or things will go bad/crash, and there's no nice way to set it up to avoid this (my colleague who currently owns MSVCRT suggested a not-very-nice way to do it, but I don't think it's going to be reliable enough). Python's stable ABI helps, but does not solve this problem. The file APIs are the worst culprits. The layout of FILE* objects can and do change between CRT versions, and file descriptors are simply indices into an array of these objects that is exposed through macros rather than function calls. As a result, you cannot mix either FILE pointers or file descriptors between CRTs. The only safe option is to build with the matching CRT, and for MSVCRT, this means with the matching compiler. It's unfortunate, and the responsible teams are well aware of the limitation, but it's history at this point, so we have no choice but to work with it. Cheers, Steve

4 3

stdlib process GSoC 2014 ideas
by anatoly techtonik 28 Feb '14

28 Feb '14

Hi, I am looking at empty ideas page at: https://wiki.python.org/moin/SummerOfCode/2014/python-core and I'd like to propose idea for GSoC students. https://bitbucket.org/techtonik/python-stdlib This needs to be extended and integrated into stdlib development process. The idea is to get a focused development gathered around modules. This includes creating a pydotorg pages (Django) that will list feeds of activity per module, including: Iteration one: - bugs - patches - discussions - votes - commits - hiscore based on all the above (shamelessly stolen from https://twistedmatrix.com/highscores/) Iteration two: - sprints info (short term activities and coordination in real-time) - past - planned - current - bug research and development (long term with visualization and reports) - subscribe to the research team - publish research materials - get scores and achievements This is just for the start. -- anatoly t.

8 8

Python-ideas February 2014