Python-ideas August 2011

python-ideas@python.org

70 participants
44 discussions

Specify number of items to allocate for array.array() constructor

by Sven Rahmann

At the moment, the array module of the standard library allows to create arrays of different numeric types and to initialize them from an iterable (eg, another array). What's missing is the possiblity to specify the final size of the array (number of items), especially for large arrays. I'm thinking of suffix arrays (a text indexing data structure) for large texts, eg the human genome and its reverse complement (about 6 billion characters from the alphabet ACGT). The suffix array is a long int array of the same size (8 bytes per number, so it occupies about 48 GB memory). At the moment I am extending an array in chunks of several million items at a time at a time, which is slow and not elegant. The function below also initializes each item in the array to a given value (0 by default). Is there a reason why there the array.array constructor does not allow to simply specify the number of items that should be allocated? (I do not really care about the contents.) Would this be a worthwhile addition to / modification of the array module? My suggestions is to modify array generation in such a way that you could pass an iterator (as now) as second argument, but if you pass a single integer value, it should be treated as the number of items to allocate. Here is my current workaround (which is slow): def filled_array(typecode, n, value=0, bsize=(1<<22)): """returns a new array with given typecode (eg, "l" for long int, as in the array module) with n entries, initialized to the given value (default 0) """ a = array.array(typecode, [value]*bsize) x = array.array(typecode) r = n while r >= bsize: x.extend(a) r -= bsize x.extend([value]*r) return x

7 months

i18n and Python tracebacks

by Andre Roberge

I think it would be a good idea if Python tracebacks could be translated into languages other than English - and it would set a good example. For example, using French as my default local language, instead of >>> 1/0 Traceback (most recent call last): File "<stdin>", line 1, in <module> ZeroDivisionError: integer division or modulo by zero I might get something like >>> 1/0 Suivi d'erreur (appel le plus récent en dernier) : Fichier "<stdin>", à la ligne 1, dans <module> ZeroDivisionError: division entière ou modulo par zéro André

7 years, 11 months

Cofunctions PEP - Revision 4

by Greg Ewing

Here's an updated version of the PEP reflecting my recent suggestions on how to eliminate 'codef'. PEP: XXX Title: Cofunctions Version: $Revision$ Last-Modified: $Date$ Author: Gregory Ewing <greg.ewing(a)canterbury.ac.nz> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 13-Feb-2009 Python-Version: 3.x Post-History: Abstract ======== A syntax is proposed for defining and calling a special type of generator called a 'cofunction'. It is designed to provide a streamlined way of writing generator-based coroutines, and allow the early detection of certain kinds of error that are easily made when writing such code, which otherwise tend to cause hard-to-diagnose symptoms. This proposal builds on the 'yield from' mechanism described in PEP 380, and describes some of the semantics of cofunctions in terms of it. However, it would be possible to define and implement cofunctions independently of PEP 380 if so desired. Specification ============= Cofunction definitions ---------------------- A cofunction is a special kind of generator, distinguished by the presence of the keyword ``cocall`` (defined below) at least once in its body. It may also contain ``yield`` and/or ``yield from`` expressions, which behave as they do in other generators. From the outside, the distinguishing feature of a cofunction is that it cannot be called the same way as an ordinary function. An exception is raised if an ordinary call to a cofunction is attempted. Cocalls ------- Calls from one cofunction to another are made by marking the call with a new keyword ``cocall``. The expression :: cocall f(*args, **kwds) is evaluated by first checking whether the object ``f`` implements a ``__cocall__`` method. If it does, the cocall expression is equivalent to :: yield from f.__cocall__(*args, **kwds) except that the object returned by __cocall__ is expected to be an iterator, so the step of calling iter() on it is skipped. If ``f`` does not have a ``__cocall__`` method, or the ``__cocall__`` method returns ``NotImplemented``, then the cocall expression is treated as an ordinary call, and the ``__call__`` method of ``f`` is invoked. Objects which implement __cocall__ are expected to return an object obeying the iterator protocol. Cofunctions respond to __cocall__ the same way as ordinary generator functions respond to __call__, i.e. by returning a generator-iterator. Certain objects that wrap other callable objects, notably bound methods, will be given __cocall__ implementations that delegate to the underlying object. Grammar ------- The full syntax of a cocall expression is described by the following grammar lines: :: atom: cocall | <existing alternatives for atom> cocall: 'cocall' atom cotrailer* '(' [arglist] ')' cotrailer: '[' subscriptlist ']' | '.' NAME Note that this syntax allows cocalls to methods and elements of sequences or mappings to be expressed naturally. For example, the following are valid: :: y = cocall self.foo(x) y = cocall funcdict[key](x) y = cocall a.b.c[i].d(x) Also note that the final calling parentheses are mandatory, so that for example the following is invalid syntax: :: y = cocall f # INVALID New builtins, attributes and C API functions -------------------------------------------- To facilitate interfacing cofunctions with non-coroutine code, there will be a built-in function ``costart`` whose definition is equivalent to :: def costart(obj, *args, **kwds): try: m = obj.__cocall__ except AttributeError: result = NotImplemented else: result = m(*args, **kwds) if result is NotImplemented: raise TypeError("Object does not support cocall") return result There will also be a corresponding C API function :: PyObject *PyObject_CoCall(PyObject *obj, PyObject *args, PyObject *kwds) It is left unspecified for now whether a cofunction is a distinct type of object or, like a generator function, is simply a specially-marked function instance. If the latter, a read-only boolean attribute ``__iscofunction__`` should be provided to allow testing whether a given function object is a cofunction. Motivation and Rationale ======================== The ``yield from`` syntax is reasonably self-explanatory when used for the purpose of delegating part of the work of a generator to another function. It can also be used to good effect in the implementation of generator-based coroutines, but it reads somewhat awkwardly when used for that purpose, and tends to obscure the true intent of the code. Furthermore, using generators as coroutines is somewhat error-prone. If one forgets to use ``yield from`` when it should have been used, or uses it when it shouldn't have, the symptoms that result can be extremely obscure and confusing. Finally, sometimes there is a need for a function to be a coroutine even though it does not yield anything, and in these cases it is necessary to resort to kludges such as ``if 0: yield`` to force it to be a generator. The ``cocall`` construct address the first issue by making the syntax directly reflect the intent, that is, that the function being called forms part of a coroutine. The second issue is addressed by making it impossible to mix coroutine and non-coroutine code in ways that don't make sense. If the rules are violated, an exception is raised that points out exactly what and where the problem is. Lastly, the need for dummy yields is eliminated by making it possible for a cofunction to call both cofunctions and ordinary functions with the same syntax, so that an ordinary function can be used in place of a cofunction that yields zero times. Record of Discussion ==================== An earlier version of this proposal required a special keyword ``codef`` to be used in place of ``def`` when defining a cofunction, and disallowed calling an ordinary function using ``cocall``. However, it became evident that these features were not necessary, and the ``codef`` keyword was dropped in the interests of minimising the number of new keywords required. The use of a decorator instead of ``codef`` was also suggested, but the current proposal makes this unnecessary as well. It has been questioned whether some combination of decorators and functions could be used instead of a dedicated ``cocall`` syntax. While this might be possible, to achieve equivalent error-detecting power it would be necessary to write cofunction calls as something like :: yield from cocall(f)(args) making them even more verbose and inelegant than an unadorned ``yield from``. It is also not clear whether it is possible to achieve all of the benefits of the cocall syntax using this kind of approach. Prototype Implementation ======================== An implementation of an earlier version of this proposal in the form of patches to Python 3.1.2 can be found here: http://www.cosc.canterbury.ac.nz/greg.ewing/python/generators/cofunctions.h… If this version of the proposal is received favourably, the implementation will be updated to match. Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

7 years, 11 months

Columns for pprint.PrettyPrinter

by Julian Berman

For the most part I find pprint to be lovely. Simple and does a useful thing that I don't want to have to write every time, especially if I'm using it just to debug. However, I often find that I'm using it to print out a long-length singly nested structure whose elements' reprs are not really that long, for which its output is cumbersome. Try to pprint.pprint(range(100)). Instead of seeing a long really skinny column, it'd be nice if it could quickly provide a way (combined with the width arg that it already takes) to split up the elements of the structure within the width, so that instead of seeing things like >>> pprint.pprint(range(30)) [0, 1, 2, ... ] it could be coerced into something like >>> pprint.pprint(range(30), columns=5) [0, 1, 2, 3, 4 3, 4, 5, 6, 7, ... ] or for something nested, which I'm less thrilled with, and haven't thought out how to implement unless you have a somewhat balanced structure, but for posterity: {"foo" : {"bar" : 1, {"hello" : 2, {"other" : 1, "baz" : 2, "world" : 1}, "thing" : 2, "foo" : 3}, "here" : 3}, ... } Obviously it's meant to be simple, the comment at the top of the module even says so, and doing something like ^ is easy enough, but for what it's good for (saving me from having to write code that makes my objects easier to debug by displaying them nicely), just making it do a bit more would make life easier.

9 years

Re: [Python-ideas] Create a StringBuilder class and use it everywhere

by k_bx

> This doesn't seem nicer to read and write to me than the list form. I > also do not see any reason to believe it will stop people from doing > it the quadratic way if the ubiquitous make-a-list-then-join idiom > does not. The whole point is that people don't write u''.join idiom just because they don't know it is slow. And when they see StringBuilder -- they can ask themselves "why is he using that". I don't mind using u''.join, but it just doesn't make people think about speed at all. The most popular (as from what I can see) thing right now where people start seeing that += is slow is when they try to do that on PyPy (which doesn't have hack like CPython, who is still slow) and ask "why my pypy code is sooooo slow". With StringBuilder used widely that would not be the case.

9 years

Add from __experimental__ import bla [was: Should we move to replace re with regex?]

by Dj Gilcrease

In the thread about replacing re with regex someone mentioned adding to __future__ which isnt a great idea as future APIs are already solidified, they just live there to give developer time to adapt their code. The idea of a __experimental__ area is good for any pep's or stliib additions that are somewhat controversial (API isnt agreed on, code may take a while to integrate properly, developer wants some time to hash out any edge case bugs or API clarifications that may come up in large scale testing, etc). __experimental__ should emit a warning on import that says anything in here may change or be removed at any time and should not be used in stable code. __experimental__ features should behave the same as __future__ in that they can add new keywords or semantics to the existing language __experimental__ features can move directly to the stlib or builtins if they do not add new keywords and/or are backwards compatible with the feature they are replacing. Otherwise they move into __future__ for how ever many releases are deemed reasonable time for developers to adapt their code.

9 years

aliasing

by Peio Borthelle

Hi, First, thank you to all the development community for this fabulous language (I'm french so my english is a bit...bad and basic). I'm quite new to programming and python is my first language. As beginner I had problems to deal with all aliasing stuff (in-place changes or not...). So my suggestion is perhaps not to change this (altough I find it a bit opposite to the python sense...) but to have a real aliasing fonction: ----------------------------- >>> a = 2 >>> b = alias("a") >>> a = 'foo' >>> b 'foo' ----------------------------- b is always a, it doesn't point to the same data, it points to the pointer a itself ! The arg is a string because otherwise the interpreter would give the value as arg, not the pointer. It could also be more complexe: ----------------------------- >>> a = 3 >>> b = alias("(a*3)+2") >>> b 11 >>> a = 5 >>> b 17 ----------------------------- Here I alias an expression (that's also why the arg must be a string). But I don't know if this last example is something good because a would have to stay a number (or generate an exception by calling b). If a is destroyed, then calling b would generate an exception...I don't have enough experience in python programming to really know how work the garbage collector, so I don't know if a could be destroyed by it. This is the end of my suggestion (I hope it wasn't already proposed, else...). Amicalement, Peio

9 years

Access to function objects

by David Townshend

Has anyone else ever thought that it might be useful to access a function object from within the call? I've come across this situation a few times recently, and thought it would be very useful to be able to do something like, for example: def a_function() as func: print(func.__doc__) It would be useful, for example, to record state between calls, or if the function wants to reuse its own properties (like in the above example). Consider a function which should print the number of times it has been called on every call def counter(add) as func: if not hasattr(func, 'count'): func.count = 0 func.count += 1 print(func.count) This could also be implemented using classes, i.e. class Counter: def __call__(self, add): if not hasattr(func, 'count'): func.count = 0 func.count += 1 print(func.count) counter = Counter() But this is much more clumsy, results in an extra object (the Counter class) and will be quite complicated if counter is a method rather than a function. The reason I've used "as" syntax is that it is consistent with other python statements (e.g. "with" and "except"), wouldn't require a new keyword and is backwardly compatible. Any thoughts? David

9 years

IP addressing library, aka reviving PEP 3144 (Re: Add from __experimental__ import bla)

by Nick Coghlan

On Tue, Aug 30, 2011 at 10:00 AM, Guido van Rossum <guido(a)python.org> wrote: > On Sun, Aug 28, 2011 at 7:53 PM, Nick Coghlan <ncoghlan(a)gmail.com> wrote: > > [Lots of stuff I agree with, and then] > >> Here's another potentially useful litmus test question for Guido: if >> it was spelled "from __experimental__ import ipaddr", would you be >> more inclined to approve PEP 3144 (IP address library) for 3.3? > > IIRC the issue with that PEP is that there is a stand-off between two > authors of competing implementations, and I don't have enough > understanding of the issues to be able to tell who's right. (Or maybe > they're both right and they're just aiming at different audiences -- > but I can't even tell that.) In reviewing the situation, that was actually the objection to the 3.1 incarnation of the ipaddr module (which was in SVN for a while before being reverted). Several changes were made to ipaddr before it was proposed for inclusion in 3.2 that largely resolved those objections (primarily, the 2.x series of ipaddr-py makes a much stronger distinction between networks and hosts: http://mail.python.org/pipermail/python-dev/2009-September/092384.html) The netaddr folks have a nice page (albeit a couple of years old now) listing some of the many incarnations of this particular wheel, so it's clearly base functionality worth providing as an included battery: https://code.google.com/p/netaddr/wiki/YetAnotherPythonIPModule > I think we need to have someone who cares more (but is not either of > those authors) to review the PEP and its criticism and decide on a way > forward. > > (Or am I being too soft? If nobody cares I'd be happy to toss a coin.) This request inspired me to go back and look at those old discussions. This comment from Scott Dial seems to summarise the core of the concern with the PEP 3144 version of the ipaddr API: http://mail.python.org/pipermail/python-dev/2009-September/091713.html RDM elaborated further here: http://mail.python.org/pipermail/python-dev/2009-September/092262.html It also turns out ipaddr is completely lacking in explicit prose documentation, so it would at least need that before it could be included. Such documentation would also be a useful adjunct to the PEP, since it would focus on what the module is like to *use* rather than how it is built. As near as I can tell, the core objection (the fact that IP network objects aren't normalised on creation, making their behaviour thoroughly surprising to anyone that actually understands the differences between IP addresses, IP networks, IP interfaces and IP hosts) remains valid, so I withdraw my suggestion that the current API should be added even as experimental code. I had forgotten about the identified problems with ipaddr until rereading the old thread reminded me of them. I believe the PEP would be significantly more palatable with the following changes/additions: 1. Draft ReStructuredText documentation for inclusion in the stdlib docs 2. Removal of the "ip" attribute of IP network objects (since it makes the nominal "networks" behave like IP interface definitions) 3. "network" property renamed to "netaddr" (since it returns an address object rather than a network object) 4. "strict" parameter removed from class signatures, replaced with class method for non-strict behaviour 5. Factory functions renamed so they don't look like class names (ip_network, ip_address, ip) 6. "strict" parameter on factory functions modified to default to True rather than False 7. Addition of an explicit "IPInterface" class to cover the association of an address with a specific network that is currently handled by storing arbitrary addresses on IP network objects IIRC, this is basically the point we reached last time, but Peter either wasn't interested in contributing/maintaining the module on those terms or else got sidetracked by other things. If he's no longer interested in contributing the module or not willing to implement any changes to address the concerns, it would be good to have an explicit statement to that effect, then we can mark the PEP as rejected and leave the field open to other proposals. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

9 years

Create a StringBuilder class and use it everywhere

by k_bx

Hi! There's a certain problem right now in python that when people need to build string from pieces they really often do something like this:: def main_pure(): b = u"initial value" for i in xrange(30000): b += u"more data" return b The bad thing about it is that new string is created every time you do +=, so it performs bad on CPython (and horrible on PyPy). If people would use, for example, list of strings it would be much better (performance):: def main_list_append(): b = [u"initial value"] for i in xrange(3000000): b.append(u"more data") return u"".join(b) The results are:: kost@kost-laptop:~/tmp$ time python string_bucket_pure.py real 0m7.194s user 0m3.590s sys 0m3.580s kost@kost-laptop:~/tmp$ time python string_bucket_append.py real 0m0.417s user 0m0.330s sys 0m0.080s Fantastic, isn't it? Also, now let's forget about speed and think about semantics a little: your task is: "build a string from it's pieces", or in other words "build a string from list of pieces", so from this point of view you can say that using [] and u"".join is better in semantic way. Java has it's StringBuilder class for a long time (I'm not really into java, I've just been told about that), and what I think is that python should have it's own StringBuilder:: class StringBuilder(object): """Use it instead of doing += for building unicode strings from pieces""" def __init__(self, val=u""): self.val = val self.appended = [] def __iadd__(self, other): self.appended.append(other) return self def __unicode__(self): self.val = u"".join((self.val, u"".join(self.appended))) self.appended = [] return self.val Why StringBuilder class, not just use [] + u''.join ? Well, I have two reasons for that: 1. It has caching 2. You can document it, because when programmer looks at [] + u"" method he doesn't see _WHY_ is it done so, while when he sees StringBuilder class he can go ahead and read it's help(). Performance of StringBuilder is ok compared to [] + u"" (I've increased number of += from 30000 to 30000000): def main_bucket(): b = StringBuilder(u"initial value ") for i in xrange(30000000): b += u"more data" return unicode(b) For CPython:: kost@kost-laptop:~/tmp$ time python string_bucket_bucket.py real 0m12.944s user 0m11.670s sys 0m1.260s kost@kost-laptop:~/tmp$ time python string_bucket_append.py real 0m3.540s user 0m2.830s sys 0m0.690s For PyPy 1.6:: (pypy)kost@kost-laptop:~/tmp$ time python string_bucket_bucket.py real 0m18.593s user 0m12.930s sys 0m5.600s (pypy)kost@kost-laptop:~/tmp$ time python string_bucket_append.py real 0m16.214s user 0m11.750s sys 0m4.280s Of course, C implementation could be done to make things faster for CPython, I guess, but really, in comparision to += method it doesn't matter now. It's done to be explicit. p.s.: also, why not use cStringIO? 1. it's not semantically right to create file-like string just to join multiple string pieces into one. 2. if you talk about using it in your code right away -- you can see that noone still uses it because people want += (while with StringBuilder you give them +=). 3. it's somehow slow on pypy right now :-) Thanks.

9 years

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

Python-ideas August 2011