Mailman 3 April 2009 - Python-ideas

registery system in Python ?
by Tarek Ziadé 23 Apr '09

23 Apr '09

Hello I am working on a plugin system for Distutils, inspired from what setuptools provides (entry_points) so I am trying to describe how a generic registery could work. But, as discussed with some people at Pycon, this is a general need. What about adding a simple generic registery system in Python stdlib ? The APIs I was thinking about would register plugins under group names for an easy classification: - get_plugin(group, name) : returns an object for (group, name) - register_plugin(group, name, object): register an object, for (group, name) - unregister_plugin(group, name): removes an object for (group, name) - list_plugins(group=None, doc=False): returns a list of all objects for the given group. - list_groups(): return a list of all groups having groups make it simpler to classify plugins. In my use case, group could be : 'distutils:filelist' to list all plugins that knows how to build a file list. (see http://wiki.python.org/moin/Distutils/ManifestPluginSystem) Regards Tarek -- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/

2 1

PYTHONUSERBASES (plural!)
by Larry Hastings 23 Apr '09

23 Apr '09

I was excited to see PYTHONUSERBASE added in 2.6/3.0. But I became disappointed when I learned it only allows you to specify one directory. I have use cases where I'd want more than one. Also, PYTHONUSERBASE doesn't support all the directories that site.addsitepackages does. PYTHONUSERBASE examines exactly one; on non-Windows platforms, it's: {prefix}/lib/pythonX.X/site-packages However, for each "prefix" directory Python knows about, site.addsitepackages examines these two on Linux: {prefix}/lib/pythonX.X/site-packages {prefix}/lib/site-python (The list is even longer on OS X.) I therefore propose a new environment variable called PYTHONUSERBASES. PYTHONUSERBASES is a list of directories separated by whatever directory separator characters PYTHONPATH permits on the local platform. At startup, after processing PYTHONUSERBASE but before calling site.addsitepackages, Python iterates down the list of directories listed in PYTHONUSERBASES and causes them to be processed by site.addsitepackages before the default prefix directories. (Probably by adding them to sys.prefixes.) PYTHONUSERBASES obeys -s, PYTHONNOUSERSITE, and the rules about effective user/group IDs, just as PYTHONUSERBASE does. The default value of PYTHONUSERBASES is the empty string; this would add no new directories. It seems to me that PYTHONUSERBASES would turn "virtualenv" into a twenty-line shell script. Add the appropriate directories to LD_LIBRARY_PATH and PYTHONUSERBASES and you'd be off to the races. You'd have to be careful to manually specify --prefix when installing new software, which I'm guessing virtualenv obviates. But apart from that I think it would work fine. Or am I missing something important? I would be happy--thrilled!, even--to write this up as a patch for Python 3.1 before the feature-freeze, /larry/

4 14

Re: [Python-ideas] [Python-Dev] Issue5434: datetime.monthdelta
by Jess Austin 23 Apr '09

23 Apr '09

On Thu, Apr 16, 2009 at 4:50 PM, Eric Smith <eric(a)trueblade.com> wrote: > Jess Austin wrote: >> >> What other behavior options besides "last-valid-day-of-the-month" >> would you like to see? > > - Add 30 days to the source date. > I'm sure there are others. Python can do this already: dt + timedelta(30) > Followups to python-ideas. This is fine with me, and I have joined -ideas now. I do still hope that an experienced core developer will have the time to comment on the code itself, but it might be best to get discussion of alternate functionality off -dev. cheers, Jess

3 4

Re: [Python-ideas] [Private Note] An idea for a new pickling tool
by Alexandre Vassalotti 23 Apr '09

23 Apr '09

On Wed, Apr 22, 2009 at 5:37 PM, Raymond Hettinger <python(a)rcn.com> wrote: >>> * it is a major security risk for untrusted inputs > >> There are way to fix this without replacing pickle. See the recipe in >> pickle documentation: >> >> http://docs.python.org/3.0/library/pickle.html#restricting-globals > > If you think untrusted pickles can easily be made secure, then you've > missed the last ten years of discussions on the subject. There's a > reason we put the big red warnings in the docs. > Could you elaborate on this, or point me to the specific discussions? And how to you plan to make your alternative secure? > >> But how are you going to handle serialization of class instances in a >> language independent manner? > > The same way RPC works, you need to have similar structures on > each end. Take a look at JSON-RPC to get an idea of how this > works. That makes sense, thanks. > Overall, I don't see what you're getting at. I'm not looking to > eliminate the current pickles. Ah then I have nothing against your proposal. It is the way you presented your idea against pickle that confused me; I actually thought you wanted to replace pickle. In that case, you probably want to take a look at the twisted.jelly module and pysyck. They each share some of the goals you aiming for. Cheers, -- Alexandre

1 0

Re: [Python-ideas] from __future__ import range
by Nick Coghlan 22 Apr '09

22 Apr '09

spir wrote: > Well, if (as intended) "from __future__ import range" means "use > future (py3) semantics for 'range'", then the compiler need to know > it ;-) Nope, the compiler hasn't got a clue what different functions do - it's job is done and dusted long before the bytecode is actually executed. So it doesn't care what the semantics of range actually are - it just cares that it is the name of something. The reason "__future__" is special is because the compiler *does* know about it, so such imports can actually change the behaviour of the parser and code generator, thus making statements and expressions actually mean different things. It does exist as a real module as well, which provides information about the various features that it can be used to enable: >>> for name in __future__.all_feature_names: ... feature = getattr(__future__, name) ... print("{0:18}{1.optional:22}{1.mandatory}".format(name, feature)) ... nested_scopes (2, 1, 0, 'beta', 1) (2, 2, 0, 'alpha', 0) generators (2, 2, 0, 'alpha', 1) (2, 3, 0, 'final', 0) division (2, 2, 0, 'alpha', 2) (3, 0, 0, 'alpha', 0) absolute_import (2, 5, 0, 'alpha', 1) (2, 7, 0, 'alpha', 0) with_statement (2, 5, 0, 'alpha', 1) (2, 6, 0, 'alpha', 0) print_function (2, 6, 0, 'alpha', 2) (3, 0, 0, 'alpha', 0) unicode_literals (2, 6, 0, 'alpha', 2) (3, 0, 0, 'alpha', 0) barry_as_FLUFL (3, 1, 0, 'alpha', 2) (3, 9, 0, 'alpha', 0) All of those are changes which affect the behaviour of the compiler. The nested_scopes, division, absolute_import and unicode_literals features don't add new syntax, but they change the meaning of existing syntax (i.e. the compiler has to generate different bytecode when they are in effect). The generators and with_statement features actually added new keywords for the parser to recognise (yield for generators, with and as for the with statement). The print_function features goes the other way: it tells the parser *not* to treat print as a keyword. All the details about this mechanism and how it works can be found in the documentation ([1], [2]) and in the PEP that added it [3]. That's all tangential to the current point though, since the change to range's semantics *isn't* something the compiler cares about. range() is still a function in Py3k, it just behaves differently from the way it behaves in 2.x. That's where future_builtins comes in: as a location for new builtins where the Py3k semantics are sufficiently different that they can't readily be introduced into the 2.x series, but don't involve any changes to syntax. Adding the Py3k range type to that module would be a good idea, since it actually isn't exactly the same as xrange in 2.x. Unlike xrange, Py3k's range object can handle arbitrary ranges instead of being restricted by the size of a C integer. I suggest making this idea (add Py3k range implementation to 2.x future_builtins) a feature request on the bug tracker. The benefits are consistency with other builtins that are changing in Py3k and also providing an arbitrary length range implementation in the 2.x series. Cheers, Nick. [1] http://docs.python.org/library/__future__ [2] http://docs.python.org/reference/simple_stmts.html#future-statements [3] http://www.python.org/dev/peps/pep-0236/ -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia ---------------------------------------------------------------

1 0

Add setpriority / getpriority to os module.
by Christian Heimes 22 Apr '09

22 Apr '09

Hello, I would like to add straight forward wrapper for the setpriority and getpriority functions to posixmodule.c for the os module. The functions allow to retrieve and modify the niceness of a process. They are in particular useful for multiprocessing to implement low priority processes. The functions are well known as 'nice' or 'renice' commands. The two functions and the constants PRIO_PROCESS, PRIO_PGRP, PRIO_USER, PRIO_MIN and PRIO_MAX should be available on most POSIX operating systems. A patch would come with autoconf tests for the feature. A while ago somebody suggested a more high level wrapper for the scheduling niceness that abstracts the niceness level. Windows (through pywin32) has a different set of values. Such a wrapper is out of scope for my proposal. Once the os module has the POSIX wrappers it's going to be easy to create a pure Python abstraction. I've also some code laying around to modify the IO priority and CPU affinity on Linux. However the code is more complex and won't work on other Unix operating systems. If somebody can implement the feature for *BSD I'm willing to give it a try. Christian

4 4

from __future__ import range
by spir 21 Apr '09

21 Apr '09

Hello, Below a suggestion by Paul MacGuire on the python-tutor. I found it so sensible that I asked him to propose it on python-ideas. His answer was: "Please feel free to post this on python-ideas as my proxy. I'm barely keeping up with my other pythonic duties as it is." So here it is. ==================================================== So I'll propose some usages for those who use range: 1. For Py2-Py3 range-xrange compatibility, add this code to the top of your Python scripts: try: range = xrange except NameError: pass In this way, your code under Py2 will use xrange whenever you call range, and you will adopt the future-compatible range behavior. (Hmm, maybe this would have been a good option to add to the "future" module...) 2. In all cases where you really must use the result returned from range as a list, ALWAYS write this as "list(range(n))", as in: nonnegative_numbers_up_to_but_not_including_10 = list(range(10)) even_numbers_up_to_but_not_including_20 = list(range(0,20,2)) Using the "list(range(n))" form when you actually need a list will prepare you for the change in idiom when you upgrade to Py3, and is fully Py2 compatible (even if you don't first bind xrange to range as in suggestion 1 - it simply copies the original list). It also makes it very explicit that you REALLY WANT THE RANGE AS A LIST, and not just as something for counting up to n. ==================================================== I think that from __future__ import range used together with list(range(...)) would: -1- Be a good programming practice, -2- Help & remove one of the 2to3 issues that cannot be solved automatically. Denis ------ la vita e estrany

2 1

Re: [Python-ideas] Needing help to change the grammar
by Nick Coghlan 21 Apr '09

21 Apr '09

(moving discussion to Python Ideas) (Context for py-ideas: a teacher in Brazil is working on a Python language variant that uses Portuguese rather than English-based keywords. This is intended for use in teaching introductory programming lessons, not as a professional development tool) Glenn Linderman wrote: > import pt_BR > > An implementation along that line, except for things like reversing the > order of "not" and "is", would allow the next national language > customization to be done by just recoding the pt_BR module, renaming to > pt_it or pt_fr or pt_no and translating a bunch of strings, no? > > Probably it would be sufficient to allow for one language at a time, per > module. Making that work would actually require something like the file encoding cookie that is detected at the parsing stage. Otherwise the parser and compiler would choke on the unexpected keywords long before the interpreter reached the stage of attempting to import anything. Adjusting the parser to accept different keyword names would be even more difficult though, since changing the details of the grammar definition is a lot more invasive than just changing the encoding of the file being read. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia ---------------------------------------------------------------

5 4

Revised**11 PEP on Yield-From
by Greg Ewing 19 Apr '09

19 Apr '09

Draft 12 of the PEP. Fixed a bug in the expansion (didn't handle StopIteration raised by throw). Removed paragraph about StopIteration left over from an earlier version. Added some discussion about rejected ideas. -- Greg PEP: XXX Title: Syntax for Delegating to a Subgenerator Version: $Revision$ Last-Modified: $Date$ Author: Gregory Ewing <greg.ewing(a)canterbury.ac.nz> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 13-Feb-2009 Python-Version: 3.x Post-History: Abstract ======== A syntax is proposed for a generator to delegate part of its operations to another generator. This allows a section of code containing 'yield' to be factored out and placed in another generator. Additionally, the subgenerator is allowed to return with a value, and the value is made available to the delegating generator. The new syntax also opens up some opportunities for optimisation when one generator re-yields values produced by another. Motivation ========== A Python generator is a form of coroutine, but has the limitation that it can only yield to its immediate caller. This means that a piece of code containing a ``yield`` cannot be factored out and put into a separate function in the same way as other code. Performing such a factoring causes the called function to itself become a generator, and it is necessary to explicitly iterate over this second generator and re-yield any values that it produces. If yielding of values is the only concern, this can be performed without much difficulty using a loop such as :: for v in g: yield v However, if the subgenerator is to interact properly with the caller in the case of calls to ``send()``, ``throw()`` and ``close()``, things become considerably more difficult. As will be seen later, the necessary code is very complicated, and it is tricky to handle all the corner cases correctly. A new syntax will be proposed to address this issue. In the simplest use cases, it will be equivalent to the above for-loop, but it will also handle the full range of generator behaviour, and allow generator code to be refactored in a simple and straightforward way. Proposal ======== The following new expression syntax will be allowed in the body of a generator: :: yield from <expr> where <expr> is an expression evaluating to an iterable, from which an iterator is extracted. The iterator is run to exhaustion, during which time it yields and receives values directly to or from the caller of the generator containing the ``yield from`` expression (the "delegating generator"). Furthermore, when the iterator is another generator, the subgenerator is allowed to execute a ``return`` statement with a value, and that value becomes the value of the ``yield from`` expression. The full semantics of the ``yield from`` expression can be described in terms of the generator protocol as follows: * Any values that the iterator yields are passed directly to the caller. * Any values sent to the delegating generator using ``send()`` are passed directly to the iterator. If the sent value is None, the iterator's ``next()`` method is called. If the sent value is not None, the iterator's ``send()`` method is called. If the call raises StopIteration, the delegating generator is resumed. Any other exception is propagated to the delegating generator. * Exceptions other than GeneratorExit thrown into the delegating generator are passed to the ``throw()`` method of the iterator. If the call raises StopIteration, the delegating generator is resumed. Any other exception is propagated to the delegating generator. * If a GeneratorExit exception is thrown into the delegating generator, or the ``close()`` method of the delegating generator is called, then the ``close()`` method of the iterator is called if it has one. If this call results in an exception, it is propagated to the delegating generator. Otherwise, GeneratorExit is raised in the delegating generator. * The value of the ``yield from`` expression is the first argument to the ``StopIteration`` exception raised by the iterator when it terminates. * ``return expr`` in a generator causes ``StopIteration(expr)`` to be raised. Enhancements to StopIteration ----------------------------- For convenience, the ``StopIteration`` exception will be given a ``value`` attribute that holds its first argument, or None if there are no arguments. Formal Semantics ---------------- Python 3 syntax is used in this section. 1. The statement :: RESULT = yield from EXPR is semantically equivalent to :: _i = iter(EXPR) try: _y = next(_i) except StopIteration as _e: _r = _e.value else: while 1: try: _s = yield _y except GeneratorExit: _m = getattr(_i, 'close', None) if _m is not None: _m() raise except: _m = getattr(_i, 'throw', None) if _m is not None: try: _y = _m(*sys.exc_info()) except StopIteration as _e: _r = _e.value break else: raise else: try: if _s is None: _y = next(_i) else: _y = _i.send(_s) except StopIteration as _e: _r = _e.value break RESULT = _r 2. In a generator, the statement :: return value is semantically equivalent to :: raise StopIteration(value) except that, as currently, the exception cannot be caught by ``except`` clauses within the returning generator. 3. The StopIteration exception behaves as though defined thusly: :: class StopIteration(Exception): def __init__(self, *args): if len(args) > 0: self.value = args[0] else: self.value = None Exception.__init__(self, *args) Rationale ========= The Refactoring Principle ------------------------- The rationale behind most of the semantics presented above stems from the desire to be able to refactor generator code. It should be possible to take an section of code containing one or more ``yield`` expressions, move it into a separate function (using the usual techniques to deal with references to variables in the surrounding scope, etc.), and call the new function using a ``yield from`` expression. The behaviour of the resulting compound generator should be, as far as reasonably practicable, the same as the original unfactored generator in all situations, including calls to ``next()``, ``send()``, ``throw()`` and ``close()``. The semantics in cases of subiterators other than generators has been chosen as a reasonable generalization of the generator case. The proposed semantics have the following limitations with regard to refactoring: * A block of code that catches GeneratorExit without subsequently re-raising it cannot be factored out while retaining exactly the same behaviour. * Factored code may not behave the same way as unfactored code if a StopIteration exception is thrown into the delegating generator. With use cases for these being rare to non-existent, it was not considered worth the extra complexity required to support them. Finalization ------------ There was some debate as to whether explicitly finalizing the delegating generator by calling its ``close()`` method while it is suspended at a ``yield from`` should also finalize the subiterator. An argument against doing so is that it would result in premature finalization of the subiterator if references to it exist elsewhere. Consideration of non-refcounting Python implementations led to the decision that this explicit finalization should be performed, so that explicitly closing a factored generator has the same effect as doing so to an unfactored one in all Python implementations. The assumption made is that, in the majority of use cases, the subiterator will not be shared. The rare case of a shared subiterator can be accommodated by means of a wrapper that blocks ``throw()`` and ``close()`` calls, or by using a means other than ``yield from`` to call the subiterator. Generators as Threads --------------------- A motivation for generators being able to return values concerns the use of generators to implement lightweight threads. When using generators in that way, it is reasonable to want to spread the computation performed by the lightweight thread over many functions. One would like to be able to call a subgenerator as though it were an ordinary function, passing it parameters and receiving a returned value. Using the proposed syntax, a statement such as :: y = f(x) where f is an ordinary function, can be transformed into a delegation call :: y = yield from g(x) where g is a generator. One can reason about the behaviour of the resulting code by thinking of g as an ordinary function that can be suspended using a ``yield`` statement. When using generators as threads in this way, typically one is not interested in the values being passed in or out of the yields. However, there are use cases for this as well, where the thread is seen as a producer or consumer of items. The ``yield from`` expression allows the logic of the thread to be spread over as many functions as desired, with the production or consumption of items occuring in any subfunction, and the items are automatically routed to or from their ultimate source or destination. Concerning ``throw()`` and ``close()``, it is reasonable to expect that if an exception is thrown into the thread from outside, it should first be raised in the innermost generator where the thread is suspended, and propagate outwards from there; and that if the thread is terminated from outside by calling ``close()``, the chain of active generators should be finalised from the innermost outwards. Syntax ------ The particular syntax proposed has been chosen as suggestive of its meaning, while not introducing any new keywords and clearly standing out as being different from a plain ``yield``. Optimisations ------------- Using a specialised syntax opens up possibilities for optimisation when there is a long chain of generators. Such chains can arise, for instance, when recursively traversing a tree structure. The overhead of passing ``next()`` calls and yielded values down and up the chain can cause what ought to be an O(n) operation to become, in the worst case, O(n\*\*2). A possible strategy is to add a slot to generator objects to hold a generator being delegated to. When a ``next()`` or ``send()`` call is made on the generator, this slot is checked first, and if it is nonempty, the generator that it references is resumed instead. If it raises StopIteration, the slot is cleared and the main generator is resumed. This would reduce the delegation overhead to a chain of C function calls involving no Python code execution. A possible enhancement would be to traverse the whole chain of generators in a loop and directly resume the one at the end, although the handling of StopIteration is more complicated then. Use of StopIteration to return values ------------------------------------- There are a variety of ways that the return value from the generator could be passed back. Some alternatives include storing it as an attribute of the generator-iterator object, or returning it as the value of the ``close()`` call to the subgenerator. However, the proposed mechanism is attractive for a couple of reasons: * Using a generalization of the StopIteration exception makes it easy for other kinds of iterators to participate in the protocol without having to grow an extra attribute or a close() method. * It simplifies the implementation, because the point at which the return value from the subgenerator becomes available is the same point at which the exception is raised. Delaying until any later time would require storing the return value somewhere. Rejected Ideas -------------- Some ideas were discussed but rejected. Suggestion: There should be some way to prevent the initial call to next(), or substitute it with a send() call with a specified value, the intention being to support the use of generators wrapped so that the initial next() is performed automatically. Resolution: Outside the scope of the proposal. Such generators should not be used with ``yield from``. Suggestion: If closing a subiterator raises StopIteration with a value, return that value from the ``close()`` call to the delegating generator. Resolution: Undesirable for a number of reasons. The purpose of closing a generator is to ensure proper cleanup, not to obtain a meaningful return value. Also, it would be unreliable unless the return value were stored so as to be available to subsequent close calls, which would cause it to persist for longer than expected. Suggestion: If ``close()`` is not to return a value, then raise an exception if StopIteration with a non-None value occurs. Resolution: No clear reason to do so. Ignoring a return value is not considered an error anywhere else in Python. Criticisms ========== Under this proposal, the value of a ``yield from`` expression would be derived in a very different way from that of an ordinary ``yield`` expression. This suggests that some other syntax not containing the word ``yield`` might be more appropriate, but no acceptable alternative has so far been proposed. Rejected alternatives include ``call``, ``delegate`` and ``gcall``. It has been suggested that some mechanism other than ``return`` in the subgenerator should be used to establish the value returned by the ``yield from`` expression. However, this would interfere with the goal of being able to think of the subgenerator as a suspendable function, since it would not be able to return values in the same way as other functions. The use of an exception to pass the return value has been criticised as an "abuse of exceptions", without any concrete justification of this claim. In any case, this is only one suggested implementation; another mechanism could be used without losing any essential features of the proposal. Alternative Proposals ===================== Proposals along similar lines have been made before, some using the syntax ``yield *`` instead of ``yield from``. While ``yield *`` is more concise, it could be argued that it looks too similar to an ordinary ``yield`` and the difference might be overlooked when reading code. To the author's knowledge, previous proposals have focused only on yielding values, and thereby suffered from the criticism that the two-line for-loop they replace is not sufficiently tiresome to write to justify a new syntax. By dealing with the full generator protocol, this proposal provides considerably more benefit. Additional Material =================== Some examples of the use of the proposed syntax are available, and also a prototype implementation based on the first optimisation outlined above. `Examples and Implementation`_ .. _Examples and Implementation: http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/ Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

4 8

Addition to I/O stack: send()
by Giovanni Bajo 19 Apr '09

19 Apr '09

Hello, currently, Python has no builtin way to "copy a file object into another", an operation commonly referred to as "send". I think this would be a good addition to the I/O stack. To make things clearer, this is pseudo-code of what I meant: def send(self, out): while 1: data = self.read(4096) if not data: break out.write(data) I'll notice that the implementation can be tweaked in different ways: * Some operating systems do support this operation natively (through syscalls like "sendfile"), which is faster because the data does not roundtrips to user space. This is useful eg. for webservers serving files to network. * In the user-space implementation, the buffer size could match whatever buffer already exists for a buffered file reader (eg: BufferedIOBase and derivates). Because of these different details, I think it would be a good addition to the I/O stack. Though, I don't fully understand it yet as to provide a more concrete proposals as where to put each different possible implementation. Also, I'm unsure how this could couple with different kind of files (pipes, sockets, etc.). As far as I can tell, the new I/O stack is not used by the standard socket library yet, for instance. -- Giovanni Bajo Develer S.r.l. http://www.develer.com

2 2