Mailman 3 April 2012 - Python-ideas

@classproperty, @abc.abstractclasspropery, etc.
by K. Richard Pixley 16 Dec '20

16 Dec '20

There's a whole matrix of these and I'm wondering why the matrix is currently sparse rather than implementing them all. Or rather, why we can't stack them as: class foo(object): @classmethod @property def bar(cls, ...): ... Essentially the permutation are, I think: {'unadorned'|abc.abstract}{'normal'|static|class}{method|property|non-callable attribute}. concreteness implicit first arg type name comments {unadorned} {unadorned} method def foo(): exists now {unadorned} {unadorned} property @property exists now {unadorned} {unadorned} non-callable attribute x = 2 exists now {unadorned} static method @staticmethod exists now {unadorned} static property @staticproperty proposing {unadorned} static non-callable attribute {degenerate case - variables don't have arguments} unnecessary {unadorned} class method @classmethod exists now {unadorned} class property @classproperty or @classmethod;@property proposing {unadorned} class non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract {unadorned} method @abc.abstractmethod exists now abc.abstract {unadorned} property @abc.abstractproperty exists now abc.abstract {unadorned} non-callable attribute @abc.abstractattribute or @abc.abstract;@attribute proposing abc.abstract static method @abc.abstractstaticmethod exists now abc.abstract static property @abc.staticproperty proposing abc.abstract static non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract class method @abc.abstractclassmethod exists now abc.abstract class property @abc.abstractclassproperty proposing abc.abstract class non-callable attribute {degenerate case - variables don't have arguments} unnecessary I think the meanings of the new ones are pretty straightforward, but in case they are not... @staticproperty - like @property only without an implicit first argument. Allows the property to be called directly from the class without requiring a throw-away instance. @classproperty - like @property, only the implicit first argument to the method is the class. Allows the property to be called directly from the class without requiring a throw-away instance. @abc.abstractattribute - a simple, non-callable variable that must be overridden in subclasses @abc.abstractstaticproperty - like @abc.abstractproperty only for @staticproperty @abc.abstractclassproperty - like @abc.abstractproperty only for @classproperty --rich

10 15

Specify number of items to allocate for array.array() constructor
by Sven Rahmann 22 Feb '20

22 Feb '20

At the moment, the array module of the standard library allows to create arrays of different numeric types and to initialize them from an iterable (eg, another array). What's missing is the possiblity to specify the final size of the array (number of items), especially for large arrays. I'm thinking of suffix arrays (a text indexing data structure) for large texts, eg the human genome and its reverse complement (about 6 billion characters from the alphabet ACGT). The suffix array is a long int array of the same size (8 bytes per number, so it occupies about 48 GB memory). At the moment I am extending an array in chunks of several million items at a time at a time, which is slow and not elegant. The function below also initializes each item in the array to a given value (0 by default). Is there a reason why there the array.array constructor does not allow to simply specify the number of items that should be allocated? (I do not really care about the contents.) Would this be a worthwhile addition to / modification of the array module? My suggestions is to modify array generation in such a way that you could pass an iterator (as now) as second argument, but if you pass a single integer value, it should be treated as the number of items to allocate. Here is my current workaround (which is slow): def filled_array(typecode, n, value=0, bsize=(1<<22)): """returns a new array with given typecode (eg, "l" for long int, as in the array module) with n entries, initialized to the given value (default 0) """ a = array.array(typecode, [value]*bsize) x = array.array(typecode) r = n while r >= bsize: x.extend(a) r -= bsize x.extend([value]*r) return x

14 20

Yielding through context managers
by Joshua Bartlett 09 Jan '13

09 Jan '13

I'd like to propose adding the ability for context managers to catch and handle control passing into and out of them via yield and generator.send() / generator.next(). For instance, class cd(object): def __init__(self, path): self.inner_path = path def __enter__(self): self.outer_path = os.getcwd() os.chdir(self.inner_path) def __exit__(self, exc_type, exc_val, exc_tb): os.chdir(self.outer_path) def __yield__(self): self.inner_path = os.getcwd() os.chdir(self.outer_path) def __send__(self): self.outer_path = os.getcwd() os.chdir(self.inner_path) Here __yield__() would be called when control is yielded through the with block and __send__() would be called when control is returned via .send() or .next(). To maintain compatibility, it would not be an error to leave either __yield__ or __send__ undefined. The rationale for this is that it's sometimes useful for a context manager to set global or thread-global state as in the example above, but when the code is used in a generator, the author of the generator needs to make assumptions about what the calling code is doing. e.g. def my_generator(path): with cd(path): yield do_something() do_something_else() Even if the author of this generator knows what effect do_something() and do_something_else() have on the current working directory, the author needs to assume that the caller of the generator isn't touching the working directory. For instance, if someone were to create two my_generator() generators with different paths and advance them alternately, the resulting behaviour could be most unexpected. With the proposed change, the context manager would be able to handle this so that the author of the generator doesn't need to make these assumptions. Naturally, nested with blocks would be handled by calling __yield__ from innermost to outermost and __send__ from outermost to innermost. I rather suspect that if this change were included, someone could come up with a variant of the contextlib.contextmanager decorator to simplify writing generators for this sort of situation. Cheers, J. D. Bartlett

6 14

i18n and Python tracebacks
by Andre Roberge 28 Oct '12

28 Oct '12

I think it would be a good idea if Python tracebacks could be translated into languages other than English - and it would set a good example. For example, using French as my default local language, instead of >>> 1/0 Traceback (most recent call last): File "<stdin>", line 1, in <module> ZeroDivisionError: integer division or modulo by zero I might get something like >>> 1/0 Suivi d'erreur (appel le plus récent en dernier) : Fichier "<stdin>", à la ligne 1, dans <module> ZeroDivisionError: division entière ou modulo par zéro André

9 15

Re: [Python-ideas] Cofunctions - Back to Basics
by Mark Shannon 21 Oct '12

21 Oct '12

Greg Ewing wrote: > Mark Shannon wrote: > >> Why not have proper co-routines, instead of hacked-up generators? > > What do you mean by a "proper coroutine"? > A parallel, non-concurrent, thread of execution. It should be able to transfer control from arbitrary places in execution, not within generators. Stackless provides coroutines. Greenlets are also coroutines (I think). Lua has them, and is implemented in ANSI C, so it can be done portably. See: http://www.jucs.org/jucs_10_7/coroutines_in_lua/de_moura_a_l.pdf (One of the examples in the paper uses coroutines to implement generators, which is obviously not required in Python :) ) Cheers, Mark.

9 25

Cofunctions PEP - Revision 4
by Greg Ewing 13 Oct '12

13 Oct '12

Here's an updated version of the PEP reflecting my recent suggestions on how to eliminate 'codef'. PEP: XXX Title: Cofunctions Version: $Revision$ Last-Modified: $Date$ Author: Gregory Ewing <greg.ewing(a)canterbury.ac.nz> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 13-Feb-2009 Python-Version: 3.x Post-History: Abstract ======== A syntax is proposed for defining and calling a special type of generator called a 'cofunction'. It is designed to provide a streamlined way of writing generator-based coroutines, and allow the early detection of certain kinds of error that are easily made when writing such code, which otherwise tend to cause hard-to-diagnose symptoms. This proposal builds on the 'yield from' mechanism described in PEP 380, and describes some of the semantics of cofunctions in terms of it. However, it would be possible to define and implement cofunctions independently of PEP 380 if so desired. Specification ============= Cofunction definitions ---------------------- A cofunction is a special kind of generator, distinguished by the presence of the keyword ``cocall`` (defined below) at least once in its body. It may also contain ``yield`` and/or ``yield from`` expressions, which behave as they do in other generators. From the outside, the distinguishing feature of a cofunction is that it cannot be called the same way as an ordinary function. An exception is raised if an ordinary call to a cofunction is attempted. Cocalls ------- Calls from one cofunction to another are made by marking the call with a new keyword ``cocall``. The expression :: cocall f(*args, **kwds) is evaluated by first checking whether the object ``f`` implements a ``__cocall__`` method. If it does, the cocall expression is equivalent to :: yield from f.__cocall__(*args, **kwds) except that the object returned by __cocall__ is expected to be an iterator, so the step of calling iter() on it is skipped. If ``f`` does not have a ``__cocall__`` method, or the ``__cocall__`` method returns ``NotImplemented``, then the cocall expression is treated as an ordinary call, and the ``__call__`` method of ``f`` is invoked. Objects which implement __cocall__ are expected to return an object obeying the iterator protocol. Cofunctions respond to __cocall__ the same way as ordinary generator functions respond to __call__, i.e. by returning a generator-iterator. Certain objects that wrap other callable objects, notably bound methods, will be given __cocall__ implementations that delegate to the underlying object. Grammar ------- The full syntax of a cocall expression is described by the following grammar lines: :: atom: cocall | <existing alternatives for atom> cocall: 'cocall' atom cotrailer* '(' [arglist] ')' cotrailer: '[' subscriptlist ']' | '.' NAME Note that this syntax allows cocalls to methods and elements of sequences or mappings to be expressed naturally. For example, the following are valid: :: y = cocall self.foo(x) y = cocall funcdict[key](x) y = cocall a.b.c[i].d(x) Also note that the final calling parentheses are mandatory, so that for example the following is invalid syntax: :: y = cocall f # INVALID New builtins, attributes and C API functions -------------------------------------------- To facilitate interfacing cofunctions with non-coroutine code, there will be a built-in function ``costart`` whose definition is equivalent to :: def costart(obj, *args, **kwds): try: m = obj.__cocall__ except AttributeError: result = NotImplemented else: result = m(*args, **kwds) if result is NotImplemented: raise TypeError("Object does not support cocall") return result There will also be a corresponding C API function :: PyObject *PyObject_CoCall(PyObject *obj, PyObject *args, PyObject *kwds) It is left unspecified for now whether a cofunction is a distinct type of object or, like a generator function, is simply a specially-marked function instance. If the latter, a read-only boolean attribute ``__iscofunction__`` should be provided to allow testing whether a given function object is a cofunction. Motivation and Rationale ======================== The ``yield from`` syntax is reasonably self-explanatory when used for the purpose of delegating part of the work of a generator to another function. It can also be used to good effect in the implementation of generator-based coroutines, but it reads somewhat awkwardly when used for that purpose, and tends to obscure the true intent of the code. Furthermore, using generators as coroutines is somewhat error-prone. If one forgets to use ``yield from`` when it should have been used, or uses it when it shouldn't have, the symptoms that result can be extremely obscure and confusing. Finally, sometimes there is a need for a function to be a coroutine even though it does not yield anything, and in these cases it is necessary to resort to kludges such as ``if 0: yield`` to force it to be a generator. The ``cocall`` construct address the first issue by making the syntax directly reflect the intent, that is, that the function being called forms part of a coroutine. The second issue is addressed by making it impossible to mix coroutine and non-coroutine code in ways that don't make sense. If the rules are violated, an exception is raised that points out exactly what and where the problem is. Lastly, the need for dummy yields is eliminated by making it possible for a cofunction to call both cofunctions and ordinary functions with the same syntax, so that an ordinary function can be used in place of a cofunction that yields zero times. Record of Discussion ==================== An earlier version of this proposal required a special keyword ``codef`` to be used in place of ``def`` when defining a cofunction, and disallowed calling an ordinary function using ``cocall``. However, it became evident that these features were not necessary, and the ``codef`` keyword was dropped in the interests of minimising the number of new keywords required. The use of a decorator instead of ``codef`` was also suggested, but the current proposal makes this unnecessary as well. It has been questioned whether some combination of decorators and functions could be used instead of a dedicated ``cocall`` syntax. While this might be possible, to achieve equivalent error-detecting power it would be necessary to write cofunction calls as something like :: yield from cocall(f)(args) making them even more verbose and inelegant than an unadorned ``yield from``. It is also not clear whether it is possible to achieve all of the benefits of the cocall syntax using this kind of approach. Prototype Implementation ======================== An implementation of an earlier version of this proposal in the form of patches to Python 3.1.2 can be found here: http://www.cosc.canterbury.ac.nz/greg.ewing/python/generators/cofunctions.h… If this version of the proposal is received favourably, the implementation will be updated to match. Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

11 29

Module __getattr__ [Was: breaking out of module execution]
by Jim Jewett 08 May '12

08 May '12

On Wed, Apr 25, 2012 at 2:35 PM, Matt Joiner <anacrolix(a)gmail.com> wrote: > If this is to be done I'd like to see all special methods supported. One of > particular interest to modules is __getattr__... For What It's Worth, supporting __setattr__ and __getattr__ is one of the few reasons that I have considered subclassing modules. The workarounds of either offering public set_varX and get_varX functions, or moving configuration to a separate singleton, just feel kludgy. Since those module methods would be defined at the far left, I don't think it would mess up understanding any more than they already do on regular classes. (There is always *some* surprise, just because they are unusual.) That said, I personally tend to view modules as a special case of classes, so I wouldn't be shocked if others found it more confusing than I would -- particularly as to whether or not the module's __getattr__ would somehow affect the lookup chain for classes defined within the module. -jJ

5 5

PEP 4XX: Adding sys.implementation
by Eric Snow 04 May '12

04 May '12

I've written up a PEP for the sys.implementation idea. Feedback is welcome! You'll notice some gaps which I'll be working on to fill in over the next couple days. Don't mind the gaps. <wink> They are in less critical (?) portions and I wanted to get this out to you before the weekend. Thanks! -eric -------------------------------------------------------------- PEP: 4XX Title: Adding sys.implementation Version: $Revision$ Last-Modified: $Date$ Author: Eric Snow <ericsnowcurrently(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 26-April-2012 Python-Version: 3.3 Abstract ======== This PEP introduces a new variable for the sys module: ``sys.implementation``. The variable holds consolidated information about the implementation of the running interpreter. Thus ``sys.implementation`` is the source to which the standard library may look for implementation-specific information. The proposal in this PEP is in line with a broader emphasis on making Python friendlier to alternate implementations. It describes the new variable and the constraints on what that variable contains. The PEP also explains some immediate use cases for ``sys.implementation``. Motivation ========== For a number of years now, the distinction between Python-the-language and CPython (the reference implementation) has been growing. Most of this change is due to the emergence of Jython, IronPython, and PyPy as viable alternate implementations of Python. Consider, however, the nearly two decades of CPython-centric Python (i.e. most of its existance). That focus had understandably contributed to quite a few CPython-specific artifacts both in the standard library and exposed in the interpreter. Though the core developers have made an effort in recent years to address this, quite a few of the artifacts remain. Part of the solution is presented in this PEP: a single namespace on which to consolidate implementation specifics. This will help focus efforts to differentiate the implementation specifics from the language. Additionally, it will foster a multiple-implementation mindset. Proposal ======== We will add ``sys.implementation``, in the sys module, as a namespace to contain implementation-specific information. The contents of this namespace will remain fixed during interpreter execution and through the course of an implementation version. This ensures behaviors don't change between versions which depend on variables in ``sys.implementation``. ``sys.implementation`` is a dictionary, as opposed to any form of "named" tuple (a la ``sys.version_info``). This is partly because it doesn't have meaning as a sequence, and partly because it's a potentially more variable data structure. The namespace will contain at least the variables described in the `Required Variables`_ section below. However, implementations are free to add other implementation information there. Some possible extra variables are described in the `Other Possible Variables`_ section. This proposal takes a conservative approach in requiring only two variables. As more become appropriate, they may be added with discretion. Required Variables -------------------- These are variables in ``sys.implementation`` on which the standard library would rely, meaning they would need to be defined: name the name of the implementation (case sensitive). version the version of the implementation, as opposed to the version of the language it implements. This would use a standard format, similar to ``sys.version_info`` (see `Version Format`_). Other Possible Variables ------------------------ These variables could be useful, but don't necessarily have a clear use case presently: cache_tag a string used for the PEP 3147 cache tag (e.g. 'cpython33' for CPython 3.3). The name and version from above could be used to compose this, though an implementation may want something else. However, module caching is not a requirement of implementations, nor is the use of cache tags. repository the implementation's repository URL. repository_revision the revision identifier for the implementation. build_toolchain identifies the tools used to build the interpreter. url (or website) the URL of the implementation's site. site_prefix the preferred site prefix for this implementation. runtime the run-time environment in which the interpreter is running. gc_type the type of garbage collection used. Version Format -------------- XXX same as sys.version_info? Rationale ========= The status quo for implementation-specific information gives us that information in a more fragile, harder to maintain way. It's spread out over different modules or inferred from other information, as we see with ``platform.python_implementation()``. This PEP is the main alternative to that approach. It consolidates the implementation-specific information into a single namespace and makes explicit that which was implicit. With the single-namespace-under-sys so straightforward, no alternatives have been considered for this PEP. Discussion ========== The topic of ``sys.implementation`` came up on the python-ideas list in 2009, where the reception was broadly positive [1]_. I revived the discussion recently while working on a pure-python ``imp.get_tag()`` [2]_. The messages in `issue 14673`_ are also relevant. Use-cases ========= ``platform.python_implementation()`` ------------------------------------ "explicit is better than implicit" The platform module guesses the python implementation by looking for clues in a couple different sys variables [3]_. However, this approach is fragile. Beyond that, it's limited to those implementations that core developers have blessed by special-casing them in the platform module. With ``sys.implementation` the various implementations would *explicitly* set the values in their own version of the sys module. Aside from the guessing, another concern is that the platform module is part of the stdlib, which ideally would minimize implementation details such as would be moved to ``sys.implementation``. Any overlap between ``sys.implementation`` and the platform module would simply defer to ``sys.implementation`` (with the same interface in platform wrapping it). Cache Tag Generation in Frozen Importlib ---------------------------------------- PEP 3147 defined the use of a module cache and cache tags for file names. The importlib bootstrap code, frozen into the Python binary as of 3.3, uses the cache tags during the import process. Part of the project to bootstrap importlib has been to clean out of Lib/import.c any code that did not need to be there. The cache tag defined in Lib/import.c was hard-coded to ``"cpython" MAJOR MINOR`` [4]_. For importlib the options are either hard-coding it in the same way, or guessing the implementation in the same way as does ``platform.python_implementation()``. As long as the hard-coded tag is limited to CPython-specific code, it's livable. However, inasmuch as other Python implementations use the importlib code to work with the module cache, a hard-coded tag would become a problem.. Directly using the platform module in this case is a non-starter. Any module used in the importlib bootstrap must be built-in or frozen, neither of which apply to the platform module. This is the point that led to the recent interest in ``sys.implementation``. Regardless of how the implementation name is gotten, the version to use for the cache tag is more likely to be the implementation version rather than the language version. That implementation version is not readily identified anywhere in the standard library. Implementation-Specific Tests ----------------------------- XXX http://hg.python.org/cpython/file/2f563908ebc5/Lib/test/support.py#l509 http://hg.python.org/cpython/file/2f563908ebc5/Lib/test/support.py#l1246 http://hg.python.org/cpython/file/2f563908ebc5/Lib/test/support.py#l1252 http://hg.python.org/cpython/file/2f563908ebc5/Lib/test/support.py#l1275 Jython's ``os.name`` Hack ------------------------- XXX http://hg.python.org/cpython/file/2f563908ebc5/Lib/test/support.py#l512 Impact on CPython ================= XXX Feedback From Other Python Implementators ========================================= IronPython ---------- XXX Jython ------ XXX PyPy ---- XXX Past Efforts ============ XXX PEP 3139 XXX PEP 399 Open Issues =========== * What are the long-term objectives for sys.implementation? - pull in implementation detail from the main sys namespace and elsewhere (PEP 3137 lite). * Alternatives to the approach dictated by this PEP? * ``sys.implementation`` as a proper namespace rather than a dict. It would be it's own module or an instance of a concrete class. Implementation ============== The implementatation of this PEP is covered in `issue 14673`_. References ========== .. [1] http://mail.python.org/pipermail/python-dev/2009-October/092893.html .. [2] http://mail.python.org/pipermail/python-ideas/2012-April/014878.html .. [3] http://hg.python.org/cpython/file/2f563908ebc5/Lib/platform.py#l1247 .. [4] http://hg.python.org/cpython/file/2f563908ebc5/Python/import.c#l121 .. _issue 14673 http://bugs.python.org/issue14673 Copyright ========= This document has been placed in the public domain. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

8 27

argparse FileType v.s default arguments...
by Mike Meyer 03 May '12

03 May '12

While I really like the argparse module, I've run into a case I think it ought to handle that it doesn't. So I'm asking here to see if 1) I've overlooked something, and it can do this, or 2) there's a good reason for it not to do this or maybe 3) this is a bad idea. The usage I ran into looks like this: parser.add_argument('configfile', default='/my/default/config', type=FileType('r'), nargs='?') If I provide the argument, everything works fine, and it opens the named file for me. If I don't, parser.configfile is set to the string, which doesn't work very well when I try to use it's read method. Unfortunately, setting default to open('/my/default/config') has the side affect of opening the file. Or raising an exception if the file doesn't exist (which is a common reason for wanting to provide an alternative!) Could default handling could be made smarter, and if 1) type is set and 2) the value of default is a string, call pass the value of default to type? Or maybe a flag to make that happen, or even a default_factory argument (incompatible with default) that would accept something like default_factory=lambda: open('/my/default/config')? Thanks, <mike -- Mike Meyer <mwm(a)mired.org> http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org

2 2

Anyone working on a platform-agnostic os.startfile()
by Hobson Lane 28 Apr '12

28 Apr '12

There is significant interest in a cross-platform file-launcher.[1][2][3][4] The ideal implementation would be an operating-system-agnostic interface that launches a file for editing or viewing, similar to the way os.startfile() works for Windows, but generalized to allow caller-specification of view vs. edit preference and support all registered os.name operating systems, not just 'nt'. Mercurial has a mature python implementation for cross-platform launching of an editor (either GUI editor or terminal-based editor like vi).[5][6] The python std lib os.startfile obviously works for Windows. The Mercurial functionality could be rolled into os.startfile() with additional named parameters for edit or view preference and gui or non-gui preference. Perhaps that would enable backporting belwo Python 3.x. Or is there a better place to incorporate this multi-platform file launching capability? [1]: http://stackoverflow.com/questions/1856792/intelligently-launching-the-defa… [2]: http://stackoverflow.com/questions/434597/open-document-with-default-applic… [3]: http://stackoverflow.com/questions/1442841/lauch-default-editor-like-webbro… [4]: http://stackoverflow.com/questions/434597/open-document-with-default-applic… [5]: http://selenic.com/repo/hg-stable/file/2770d03ae49f/mercurial/ui.py [6]: http://selenic.com/repo/hg-stable/file/2770d03ae49f/mercurial/util.py

3 2

Python-ideas April 2012