Mailman 3 June 2011 - Python-ideas

@classproperty, @abc.abstractclasspropery, etc.
by K. Richard Pixley 16 Dec '20

16 Dec '20

There's a whole matrix of these and I'm wondering why the matrix is currently sparse rather than implementing them all. Or rather, why we can't stack them as: class foo(object): @classmethod @property def bar(cls, ...): ... Essentially the permutation are, I think: {'unadorned'|abc.abstract}{'normal'|static|class}{method|property|non-callable attribute}. concreteness implicit first arg type name comments {unadorned} {unadorned} method def foo(): exists now {unadorned} {unadorned} property @property exists now {unadorned} {unadorned} non-callable attribute x = 2 exists now {unadorned} static method @staticmethod exists now {unadorned} static property @staticproperty proposing {unadorned} static non-callable attribute {degenerate case - variables don't have arguments} unnecessary {unadorned} class method @classmethod exists now {unadorned} class property @classproperty or @classmethod;@property proposing {unadorned} class non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract {unadorned} method @abc.abstractmethod exists now abc.abstract {unadorned} property @abc.abstractproperty exists now abc.abstract {unadorned} non-callable attribute @abc.abstractattribute or @abc.abstract;@attribute proposing abc.abstract static method @abc.abstractstaticmethod exists now abc.abstract static property @abc.staticproperty proposing abc.abstract static non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract class method @abc.abstractclassmethod exists now abc.abstract class property @abc.abstractclassproperty proposing abc.abstract class non-callable attribute {degenerate case - variables don't have arguments} unnecessary I think the meanings of the new ones are pretty straightforward, but in case they are not... @staticproperty - like @property only without an implicit first argument. Allows the property to be called directly from the class without requiring a throw-away instance. @classproperty - like @property, only the implicit first argument to the method is the class. Allows the property to be called directly from the class without requiring a throw-away instance. @abc.abstractattribute - a simple, non-callable variable that must be overridden in subclasses @abc.abstractstaticproperty - like @abc.abstractproperty only for @staticproperty @abc.abstractclassproperty - like @abc.abstractproperty only for @classproperty --rich

10 15

i18n and Python tracebacks
by Andre Roberge 28 Oct '12

28 Oct '12

I think it would be a good idea if Python tracebacks could be translated into languages other than English - and it would set a good example. For example, using French as my default local language, instead of >>> 1/0 Traceback (most recent call last): File "<stdin>", line 1, in <module> ZeroDivisionError: integer division or modulo by zero I might get something like >>> 1/0 Suivi d'erreur (appel le plus récent en dernier) : Fichier "<stdin>", à la ligne 1, dans <module> ZeroDivisionError: division entière ou modulo par zéro André

9 15

Cofunctions PEP - Revision 4
by Greg Ewing 13 Oct '12

13 Oct '12

Here's an updated version of the PEP reflecting my recent suggestions on how to eliminate 'codef'. PEP: XXX Title: Cofunctions Version: $Revision$ Last-Modified: $Date$ Author: Gregory Ewing <greg.ewing(a)canterbury.ac.nz> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 13-Feb-2009 Python-Version: 3.x Post-History: Abstract ======== A syntax is proposed for defining and calling a special type of generator called a 'cofunction'. It is designed to provide a streamlined way of writing generator-based coroutines, and allow the early detection of certain kinds of error that are easily made when writing such code, which otherwise tend to cause hard-to-diagnose symptoms. This proposal builds on the 'yield from' mechanism described in PEP 380, and describes some of the semantics of cofunctions in terms of it. However, it would be possible to define and implement cofunctions independently of PEP 380 if so desired. Specification ============= Cofunction definitions ---------------------- A cofunction is a special kind of generator, distinguished by the presence of the keyword ``cocall`` (defined below) at least once in its body. It may also contain ``yield`` and/or ``yield from`` expressions, which behave as they do in other generators. From the outside, the distinguishing feature of a cofunction is that it cannot be called the same way as an ordinary function. An exception is raised if an ordinary call to a cofunction is attempted. Cocalls ------- Calls from one cofunction to another are made by marking the call with a new keyword ``cocall``. The expression :: cocall f(*args, **kwds) is evaluated by first checking whether the object ``f`` implements a ``__cocall__`` method. If it does, the cocall expression is equivalent to :: yield from f.__cocall__(*args, **kwds) except that the object returned by __cocall__ is expected to be an iterator, so the step of calling iter() on it is skipped. If ``f`` does not have a ``__cocall__`` method, or the ``__cocall__`` method returns ``NotImplemented``, then the cocall expression is treated as an ordinary call, and the ``__call__`` method of ``f`` is invoked. Objects which implement __cocall__ are expected to return an object obeying the iterator protocol. Cofunctions respond to __cocall__ the same way as ordinary generator functions respond to __call__, i.e. by returning a generator-iterator. Certain objects that wrap other callable objects, notably bound methods, will be given __cocall__ implementations that delegate to the underlying object. Grammar ------- The full syntax of a cocall expression is described by the following grammar lines: :: atom: cocall | <existing alternatives for atom> cocall: 'cocall' atom cotrailer* '(' [arglist] ')' cotrailer: '[' subscriptlist ']' | '.' NAME Note that this syntax allows cocalls to methods and elements of sequences or mappings to be expressed naturally. For example, the following are valid: :: y = cocall self.foo(x) y = cocall funcdict[key](x) y = cocall a.b.c[i].d(x) Also note that the final calling parentheses are mandatory, so that for example the following is invalid syntax: :: y = cocall f # INVALID New builtins, attributes and C API functions -------------------------------------------- To facilitate interfacing cofunctions with non-coroutine code, there will be a built-in function ``costart`` whose definition is equivalent to :: def costart(obj, *args, **kwds): try: m = obj.__cocall__ except AttributeError: result = NotImplemented else: result = m(*args, **kwds) if result is NotImplemented: raise TypeError("Object does not support cocall") return result There will also be a corresponding C API function :: PyObject *PyObject_CoCall(PyObject *obj, PyObject *args, PyObject *kwds) It is left unspecified for now whether a cofunction is a distinct type of object or, like a generator function, is simply a specially-marked function instance. If the latter, a read-only boolean attribute ``__iscofunction__`` should be provided to allow testing whether a given function object is a cofunction. Motivation and Rationale ======================== The ``yield from`` syntax is reasonably self-explanatory when used for the purpose of delegating part of the work of a generator to another function. It can also be used to good effect in the implementation of generator-based coroutines, but it reads somewhat awkwardly when used for that purpose, and tends to obscure the true intent of the code. Furthermore, using generators as coroutines is somewhat error-prone. If one forgets to use ``yield from`` when it should have been used, or uses it when it shouldn't have, the symptoms that result can be extremely obscure and confusing. Finally, sometimes there is a need for a function to be a coroutine even though it does not yield anything, and in these cases it is necessary to resort to kludges such as ``if 0: yield`` to force it to be a generator. The ``cocall`` construct address the first issue by making the syntax directly reflect the intent, that is, that the function being called forms part of a coroutine. The second issue is addressed by making it impossible to mix coroutine and non-coroutine code in ways that don't make sense. If the rules are violated, an exception is raised that points out exactly what and where the problem is. Lastly, the need for dummy yields is eliminated by making it possible for a cofunction to call both cofunctions and ordinary functions with the same syntax, so that an ordinary function can be used in place of a cofunction that yields zero times. Record of Discussion ==================== An earlier version of this proposal required a special keyword ``codef`` to be used in place of ``def`` when defining a cofunction, and disallowed calling an ordinary function using ``cocall``. However, it became evident that these features were not necessary, and the ``codef`` keyword was dropped in the interests of minimising the number of new keywords required. The use of a decorator instead of ``codef`` was also suggested, but the current proposal makes this unnecessary as well. It has been questioned whether some combination of decorators and functions could be used instead of a dedicated ``cocall`` syntax. While this might be possible, to achieve equivalent error-detecting power it would be necessary to write cofunction calls as something like :: yield from cocall(f)(args) making them even more verbose and inelegant than an unadorned ``yield from``. It is also not clear whether it is possible to achieve all of the benefits of the cocall syntax using this kind of approach. Prototype Implementation ======================== An implementation of an earlier version of this proposal in the form of patches to Python 3.1.2 can be found here: http://www.cosc.canterbury.ac.nz/greg.ewing/python/generators/cofunctions.h… If this version of the proposal is received favourably, the implementation will be updated to match. Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

11 29

dir with a glob?
by Sturla Molden 01 Jul '11

01 Jul '11

Often when exploring an object with the 'dir' function, particularly in large packages like SciPy, I find that I need to filter the outout. Since a dir reminds me of a dos 'dir' or linux 'ls', a glob feels like the most natural to use. For example, none of these would work: >>> dir(sp.fft.i*) # syntax error >>> dir('sp.fft.i*') # returns the attributes a string I believe a new dir functions is needed, or a change in the behviour of the current version. Of course a 'glob aware dir function' can be implemented by monitoring the call stack (cf. sys._getframe().f_back), but I think the problem is general enough to warrant an official sultion. Sturla

13 26

Adding shutil.disk_usage()
by Giampaolo Rodolà 29 Jun '11

29 Jun '11

Hi all, I've just implemented this functionality in psutil for both POSIX and Windows and thought it might be nice to have it in shutil module as well since it's useful when doing system monitoring: http://code.google.com/p/psutil/issues/detail?id=172 The posix implementation is nothing but a wrapper around os.statvfs(): def disk_usage(path): """Return disk usage associated with path.""" st = os.statvfs(path) free = (st.f_bavail * st.f_frsize) total = (st.f_blocks * st.f_frsize) used = (st.f_blocks - st.f_bfree) * st.f_frsize percent = (float(used) / total) * 100 # NB: the percentage is -5% than what shown by df due to # reserved blocks that we are currently not considering: # http://goo.gl/sWGbH return ntuple_diskinfo(total, used, free, round(percent, 1)) ...and reflects what returned by "df /somepath". The Windows implementation requires GetDiskFreeSpaceEx() which is not exposed in python stdlib but can be added as a privade module (Modules/_winutil.c maybe?) or retrieved via ctypes. Thoughts? --- Giampaolo http://code.google.com/p/pyftpdlib http://code.google.com/p/psutil

4 4

A few suggestions for the random module
by Sven Marnach 25 Jun '11

25 Jun '11

I'd like to suggest what I consider a few minor improvements to Python's random module. 1. random.expovariate(lambd) The current implementation [1] is random = self.random u = random() while u <= 1e-7: u = random() return -_log(u)/lambd I'd suggest to simplify this to return -log(1.0 - self.random())/lambd self.random() returns a float in the half-open interval [0.0, 1.0), so 1.0 - self.random() is in the half-open interval (0.0, 1.0], which is exactly what we need here. Even if the random number gets as close to 1.0 as possible within the limits of double precision, taking the logarithm won't be any problem (the lowest occuring value of log(...) being roughly -36.7). The suggested implementation is not only simpler and faster, it is also more correct. The current implementation will never return 0.0, although this is a perfectly valid return value, and it chooses the arbitrary cut-off value 1e7, while the suggested implementation is only limited by floating point precision and the range of random.random(). [1]: http://hg.python.org/cpython/file/54fb77e0762c/Lib/random.py#l392 2. random.sample(population, k) The current implementation [2] chooses one of two algorithms, depending on whether the number of samples k is relatively small compared to the size n of the population. I suggest to add reservoir sampling [3] as a third algorithm, used in either of the following cases: * k is "close" to n (for example k > .75 * n) * population is not a sequence (but of course an iterable) Reservoir sampling would only use O(1) additional memory, compared to O(n) for the current algorithm for relativly large values of k. It would also facilitate to select samples from iterables without storing a complete list of all items in memory at any time. It could also be used for smaller values of k, trading reduced memory usage for increased execution time. Example implementation: it = iter(population) result = list(_islice(it, k)) self.shuffle(result) for i, x in enumerate(it, k + 1): j = randbelow(i) if j < k: result[j] = x We need to call self.shuffle(result) here to ensure that all subslices are valid random samples again. An alternative to adding this algorithm to the current sample() function is to add a new function sample_iterable() or similar. (This new function probably shouldn't call shuffle(), since the caller can always do this if required.) [2]: http://hg.python.org/cpython/file/54fb77e0762c/Lib/random.py#l268 [3]: http://en.wikipedia.org/wiki/Reservoir_sampling 3. random.choice(seq) Similarly to the above explanation, random.choice() could be generalised to arbitrary iterables. Of course the current O(1) alogrithm should be used if a sequence is passed. In case of positive feedback on one or more of those points, I'd be happy to prepare a patch. -- Sven

4 4

Simpler namespace packages
by Mike Graham 21 Jun '11

21 Jun '11

I was reading over PEP 382 and am not sure I really understand how it is a worthwhile step forward. Namespace packages are currently a mess and personally if anyone asks me I recommend not using the currently-available approach at all. Would it be possible to allow a more simple definition, for example putting the dot itself in the filename? Where these would be similar? site-packages/ foo/ __init__.py foo.bar.py foo.baz/ __init__.py qux.py foo.spam.eggs.py and site-packages/ foo/ __init__.py bar.py baz/ __init__.py qux.py spam/ __init__.py #empty eggs.py Obviously this is not fully-defined, but would this kind of approach fill the same need and be nicer to use, create, find, and understand then what is proposed in PEP 382?

2 3

'Injecting' objects as function-local constants
by Jan Kaliszewski 20 Jun '11

20 Jun '11

== Use cases == A quite common practice is 'injecting' objects into a function as its locals, at def-time, using function arguments with default values... Sometimes to keep state using a mutable container: def do_and_remember(val, verbose=False, mem=collections.Counter()): result = do_something(val) mem[val] += 1 if verbose: print('Done {} times for {!r}'.format(mem[val], val)) Sometimes, when creating functions dynamically (making use of nested scopes), e.g. to keep some individual function features (usable within that functions): def make_my_callbacks(callback_params): my_callbacks = [] for params in callback_params: def fun1(*args, _params=params, **kwargs): "...do something with args and params..." def fun2(*args, _params=params, **kwargs): "...do something with args and params..." def fun3(*args, _fun1=fun1, _fun2=fun2, **kwargs): """...do something with args and with functions fun1, fun2, for example pass them as callbacks to other functions..." my_callbacks.append((fun1, fun2, fun3)) return my_callbacks Sometimes simply to make critical parts of code optimised... def do_it_quickly(fields, _len=len, _split=str.split, _sth=something): return [_len(f), _split(f), _sth(f) for f in fields] ...or even for readability -- keeping function-specific constants within the function definition: def check_value(val, VAL_REGEX=re.compile('^...$'), VAL_MAX_LEN=38): return len(val) <= VAL_MAX_LEN and VAL_RE.search(val) is not None In all that cases (and probably some other too) that technique appears to be quite useful. == The problem == ...is that it is not very elegant. We add arguments which: a) mess up function signatures (both in the code and in auto-generated docs); b) can be incidentally overriden (especially when a function has an "open" signature with **kwargs). == Proposed solutions == I see three possibilities: 1. To add a new keyword, e.g. `inject': def do_and_remember(val, verbose=False): inject mem = collections.Counter() ... or maybe: def do_and_remember(val, verbose=False): inject collections.Counter() as mem ... 2. (which personally I would prefer) To add `dummy' (or `hidden') keyword arguments, defined after **kwargs (and after bare ** if kwargs are not needed; we have already have keyword-only arguments after *args or bare *): def do_and_remember(val, verbose=False, **, mem=collections.Counter()): ... do_and_remember(val, False, mem='something') would raise TypeError and `mem' shoudn not appear in help() etc. as a function argument. 3. To provide a special decorator, e.g. functools.within: @functools.within(mem=collections.Counter()) def do_and_remember(val, verbose=False): ... Regards. *j

17 79

PEP 3150 (statement local namespaces) updated based on April discussion
by Nick Coghlan 14 Jun '11

14 Jun '11

The rationale and proposed semantics sections of PEP 3150 have been heavily modified based on the various discussions on the topic back in April. http://www.python.org/dev/peps/pep-3150/ (Note that the PEP is still officially Deferred - this entire idea still intrigues me, but I'm also still not convinced it's worth the additional language complexity). Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

6 9

Re: [Python-ideas] inheriting docstrings
by Eric Snow 11 Jun '11

11 Jun '11

On Fri, Jun 10, 2011 at 7:29 AM, Dj Gilcrease <digitalxero(a)gmail.com> wrote: > Though I guess __basedocs__ mapping to [c.__doc__ for c in C.__mro__ > if c != C] could be handy Or: next(c.__doc__ for c in C.__mro__[1:] if c.__doc__) or None Right now you could do something like this: def get_basedoc(mro): return next(c.__doc__ for c in mro[1:] if c.__doc__) or None class Meta(type): __basedoc__ = property(lambda cls: get_basedoc(cls.__mro__)) But then instances don't get __basedoc__, since the metaclass is not in the MRO. To get it on instances you could do this: class C: __basedoc__ = property(lambda self: get_basedoc(self.__class__.__mro__)) But then getting that attribute on the class will give you the property object and not the docstring. I'm not sure of a way to resolve that. However, inheriting the docstring between classes is only part of the problem. You'll probably want to have the functions on a class also "inherit" their docstring from the first matching attribute of the bases on the MRO. In that case the function will not have enough information (the class isn't available to the function) to do the lookup. Instead, the class would have to do it. Here's an example: class BasedocMethod: def __init__(self, f, cls): self.f = f self.cls = cls def __getattribute__(self, name): if name == "__basedoc__": return object.__getattribute__(self, "__basedoc__") return getattr(f, name) @property def __basedoc__(self): for base in self.cls.__mro__: basefunc = base.__dict__.get(self.f.__name__) if not basefunc: continue return getattr(basefunc, "__doc__", None) return None class Meta(type): def __init__(cls, name, bases, namespace): for attrname, obj in namespace.items(): if isinstance(obj, FunctionType): setattr(cls, attrname, BasedocMethod(obj, cls)) Obviously not a perfect example, but hopefully demonstrates the idea. The whole point is that it would be nice to have a mechanism built in that makes it easy to inherit docstrings to functions, without necessarily implicitly inheriting them. The__basedoc__ idea works for that. My main motivation for this docstring inheritance is primarily for the case of abstract base classes. My implementations of those rarely have docstrings unique from that of the base class, nor do the methods. I can get around this now with metaclasses and class decorators, but it would be nice to have a little more help from the language. It would always be nice. :) -eric > _______________________________________________ > Python-ideas mailing list > Python-ideas(a)python.org > http://mail.python.org/mailman/listinfo/python-ideas >

2 5