Mailman 3 June 2010 - Python-ideas

i18n and Python tracebacks
by Andre Roberge Oct. 28, 2012

Oct. 28, 2012

I think it would be a good idea if Python tracebacks could be translated into languages other than English - and it would set a good example. For example, using French as my default local language, instead of >>> 1/0 Traceback (most recent call last): File "<stdin>", line 1, in <module> ZeroDivisionError: integer division or modulo by zero I might get something like >>> 1/0 Suivi d'erreur (appel le plus récent en dernier) : Fichier "<stdin>", à la ligne 1, dans <module> ZeroDivisionError: division entière ou modulo par zéro André

9 15

explicitation lines in python ?
by Daniel DELAY July 12, 2010

July 12, 2010

If we could explicitate a too complex expression in an indented next line, I would use this feature very often : htmltable = ''.join( '<tr>{}</tr>'.format(htmlline) for line in table) : # main line htmlline : ''.join( '<td>{}</td>'.format(cell) for cell in line) # explicitation(s) line(s) (Sorry if this has already been discussed earlier on this list, I have not read all the archives) ******* in details : List comprehension "<expression> for x in mylist" often greatly improve readability of python programs, when <expression> is not too complex When <expression> is too complex (ex: nested lists), this become not readable, so we have to find another solution : a) defining a function expression(x), or an iterator function, wich will only used once in the code b) or droping this beautiful syntax to replace it the very basic list construction : newlist = [] for x in myiterable newlist.append(<expression>) I often choose b), but I dislike both solutions : - in solution a) function def can be far from list comprehension, in fact instructions to build the new list are split in two different places in the code. - solution b) seems a bit better to me, but the fact we build a new list from myiterable is not visible in a glance, unlike list comprehensions. Renouncing to list comprehension occurs rather often when I write python code I think we could greatly improve readability if we could keep list comprehension anywhere in all cases, but when necessary explicit a too complex expression in an indented line : htmltable = ''.join( '<tr>{}</tr>'.format(htmlline) for line in table) : # main line htmlline : ''.join( '<td>{}</td>'.format(cell) for cell in line) # explicitation(s) line(s) In the case the main line is the header of a "classical" indented block (starting with "for", "if", "with"...) , this idented block would simply follow the explicitation(s) line(s). The explicitations lines can be surely identified as the lines than begin with "identifier :" (when we are not in an unclosed dict) with open('data.txt') as f : if line in enumerate(mylist) : # main line mylist : f.read().strip().lower() # explicitation(s) line(s) print line # "classical" indented block Another possible use of "explicitations lines" is a coding style wich would start by "the whole picture" first, and completing with details after, wich is the usual way we mentally solve problems. Let's take an example : we want to write a function wich returns a multiplication table in a simle html document. When I solve this problem, il think a bit like that : - I need to return an html page. For that I need a "header" and a "body". My body will contain an "htmltable", wich be built from a "table" of numbers etc. My code could look like that : def tag(content, *tags): # little convenient function retval = content for t in tags: retval = '<{0}>{1}</{0}>'.format(t, retval) return retval def xhtml_mult_table(a, b): return tag(header + body, 'html') : header : tag('multiplication table', 'title') body : tag(htmltable, 'tbody', 'table', 'body') : htmltable : ''.join(tag(xhtmlline, 'tr') for line in table) : table : headerline + otherlines : headerline : [[''] + range(a)] otherlines : [[y] + [x*y for x in range(a)] for y in range(b)] xhtmlline : ''.join(tag(str(cell), 'td') for cell in line) This example is a "heavy use" of the "explicitation line" feature, to illustrate how it could work. I don't mean this should replace the "classical" syntax everywhere possible, but for me this would be for me a nice way to explicitate complex expressions from time to time, and the ability to use list comprehension everywhere I wan't. Daniel

8 17

Maybe allow br"" or rb"" e.g., for bytes regexes in Py3?
by Mark Summerfield June 30, 2010

June 30, 2010

Hi, Python 3 has two string prefixes r"" for raw strings and b"" for bytes. So if you want to create a regex based on bytes as far as I can tell you have to do something like this: FONTNAME_RE = re.compile(r"/FontName\s+/(\S+)".encode("ascii")) # or FONTNAME_RE = re.compile(b"/FontName\\s+/(\\S+)") I think it would be much nicer if one could write: FONTNAME_RE = re.compile(br"/FontName\s+/(\S+)") # or FONTNAME_RE = re.compile(rb"/FontName\s+/(\S+)") I _slightly_ prefer rb"" to br"" but either would be great:-) Why would you want a bytes regex? In my case I am reading PostScript files and PostScript .pfa font files so that I can embed the latter into the former. But I don't know what encoding these files use beyond the fact that it is ASCII or some ASCII superset like Latin1. So in true Python style I don't assume: instead I read the files as bytes and do all my processing using bytes, at no point decoding since I only ever insert ASCII characters. I don't think this is a rare example: with Python 3's clean separation between strings & bytes (a major advance IMO), I think there will often be cases where all the processing is done using bytes. -- Mark Summerfield, Qtrac Ltd, www.qtrac.eu C++, Python, Qt, PyQt - training and consultancy "Advanced Qt Programming" - ISBN 0321635906 http://www.qtrac.eu/aqpbook.html I ordered a Dell netbook with Ubuntu... I got no OS, no apology, no solution, & no refund (so far) http://www.qtrac.eu/dont-buy-dell.html

7 7

feature to make traceback objects usable without references to frame locals and globals
by ghazel＠gmail.com June 30, 2010

June 30, 2010

Hi, I'm interested in a feature which allows users to discard the locals and globals references from frames held by a traceback object. Currently, traceback objects are used when capturing and re-raising exceptions. However, they hold a reference to all frames, which hold a reference to their locals and globals. These are not needed by the default traceback output, and can cause serious memory bloat if a reference to a traceback object is kept for any significant length of time, and there are even big red warnings in the Python docs about using them in one frame. ( http://docs.python.org/release/3.1/library/sys.html#sys.exc_info ). Example usage would be something like: import sys try: 1/0 except: t, v, tb = sys.exc_info() tb.clean() # ... much later ... raise t, v, tb Which would be basically a function to do this: import sys try: 1/0 except: t, v, tb = sys.exc_info() c = tb while c: c.tb_frame.f_locals = None c.tb_frame.f_globals = None c = c.tb_next # ... much later ... raise t, v, tb Twisted has done a very similar thing with their twisted.python.failure.Failure object, which stringifies the traceback data and discards the reference to the Python traceback entirely ( http://twistedmatrix.com/trac/browser/tags/releases/twisted-10.0.0/twisted/… ) - they also replicate a lot of traceback printing functions to make use of this stringified data. It's worth noting that cgitb and other applications make use of locals and globals in its traceback output. However, I believe the vast majority of traceback usage does not make use of these references, and a significant penalty is paid as a result. Is there any interest in such a feature? -Greg

11 31

Re: [Python-ideas] [Python-Dev] [ANN]: "newthreading" - an approach to simplified thread usage, and a path to getting rid of the GIL
by Guido van Rossum June 29, 2010

June 29, 2010

I'm moving this thread to python-ideas, where it belongs. I've looked at the implementation code (even stepped through it with pdb!), read the sample/test code, and read the two papers on animats.com fairly closely (they have a lot of overlap, and the memory model described below seems copied verbatim from http://www.animats.com/papers/languages/pythonconcurrency.html version 0.8). Some reactions (trying to hide my responses to the details of the code): - First of all, I'm very happy to see radical ideas proposed, even if they are at present unrealistic. We need a big brainstorm to come up with ideas from which an eventual solution to the multicore problem might be chosen. (Jesse Noller's multiprocessing is another; Adam Olsen's work yet another, at a different end of the spectrum.) - The proposed new semantics (frozen objects, memory model, auto-freezing of globals, enforcement of naming conventions) are radically different from Python's current semantics. They will break every 3rd party library in many more ways than Python 3. This is not surprising given the goals of the proposal (and its roots in Adam Olsen's work) but places a huge roadblock for acceptance. I see no choice but to keep trying to come up with a compromise that is more palatable and compatible without throwing away all the advantages. As it now stands, the proposal might as well be a new and different language. - SynchronizedObject looks like a mixture of a Java synchronized class (a non-standard concept in Java but easily understood as a class all whose public methods are synchronized) and a condition variable (which has the same semantics of releasing the lock while waiting but without crawling the stack for other locks to release). It looks like the examples showing off SynchronizedObject could be implemented just as elegantly using a condition variable (and voluntary abstention from using shared mutable objects). - If the goal is to experiment with new control structures, I recommend decoupling them from the memory model and frozen objects, instead relying (as is traditional in Python) on programmer caution to avoid races. This would make it much easier to see how programmers respond to the new control structures. - You could add the freeze() function for voluntary use, and you could even add automatic wrapping of arguments and return values for certain classes using a class decorator or a metaclass, but the performance overhead makes this unlikely to win over many converts. I don't see much use for the "whole program freezing" done by the current prototype -- there are way too many backdoors in Python for the prototype approach to be anywhere near foolproof, and if we want a non-foolproof approach, voluntary constraint (and, in some cases, voluntary, i.e. explicit, wrapping of modules or classes) would work just as well. - For a larger-scale experiment with the new memory model and semantic restrictions (or would it be better to call them syntactic restrictions? -- after all they are about statically detectable properties like naming conventions) I recommend looking at PyPy, which has as one of its explicitly stated project goals easy experimentation with different object models. - I'm sure I've forgotten something, but I wanted to keep my impressions fresh. - Again, John, thanks for taking the time to come up with an implementation of your idea! --Guido On Sat, Jun 26, 2010 at 9:39 AM, John Nagle <nagle(a)animats.com> wrote: > On 6/26/2010 7:44 AM, Jesse Noller wrote: >> >> On Sat, Jun 26, 2010 at 9:29 AM, Michael Foord >> <fuzzyman(a)voidspace.org.uk> wrote: >>> >>> On 26/06/2010 07:11, John Nagle wrote: >>>> >>>> We have just released a proof-of-concept implementation of a new >>>> approach to thread management - "newthreading". > > .... > >>> The import * form is considered bad practise in *general* and >>> should not be recommended unless there is a good reason. > > I agree. I just did that to make the examples cleaner. > >>> however the introduction of free-threading in Python has not been >>> hampered by lack of synchronization primitives but by the >>> difficulty of changing the interpreter without unduly impacting >>> single threaded code. > > That's what I'm trying to address here. > >>> Providing an alternative garbage collection mechanism other than >>> reference counting would be a more interesting first-step as far as >>> I can see, as that removes the locking required around every access >>> to an object (which currently touches the reference count). >>> Introducing free-threading by *changing* the threading semantics >>> (so you can't share non-frozen objects between threads) would not >>> be acceptable. That comment is likely to be based on a >>> misunderstanding of your future intentions though. :-) > > This work comes out of a discussion a few of us had at a restaurant > in Palo Alto after a Stanford talk by the group at Facebook which > is building a JIT compiler for PHP. We were discussing how to > make threading both safe for the average programmer and efficient. > Javascript and PHP don't have threads at all; Python has safe > threading, but it's slow. C/C++/Java all have race condition > problems, of course. The Facebook guy pointed out that you > can't redefine a function dynamically in PHP, and they get > a performance win in their JIT by exploiting this. > > I haven't gone into the memory model in enough detail in the > technical paper. The memory model I envision for this has three > memory zones: > > 1. Shared fully-immutable objects: primarily strings, numbers, > and tuples, all of whose elements are fully immutable. These can > be shared without locking, and reclaimed by a concurrent garbage > collector like Boehm's. They have no destructors, so finalization > is not an issue. > > 2. Local objects. These are managed as at present, and > require no locking. These can either be thread-local, or local > to a synchronized object. There are no links between local > objects under different "ownership". Whether each thread and > object has its own private heap, or whether there's a common heap with > locks at the allocator is an implementation decision. > > 3. Shared mutable objects: mostly synchronized objects, but > also immutable objects like tuples which contain references > to objects that aren't fully immutable. These are the high-overhead > objects, and require locking during reference count updates, or > atomic reference count operations if supported by the hardware. > The general idea is to minimize the number of objects in this > zone. > > The zone of an object is determined when the object is created, > and never changes. This is relatively simple to implement. > Tuples (and frozensets, frozendicts, etc.) are normally zone 2 > objects. Only "freeze" creates collections in zones 1 and 3. > Synchronized objects are always created in zone 3. > There are no difficult handoffs, where an object that was previously > thread-local now has to be shared and has to acquire locks during > the transition. > > Existing interlinked data structures, like parse trees and GUIs, > are by default zone 2 objects, with the same semantics as at > present. They can be placed inside a SynchronizedObject if > desired, which makes them usable from multiple threads. > That's optional; they're thread-local otherwise. > > The rationale behind "freezing" some of the language semantics > when the program goes multi-thread comes from two sources - > Adam Olsen's Safethread work, and the acceptance of the > multiprocessing module. Olsen tried to retain all the dynamism of > the language in a multithreaded environment, but locking all the > underlying dictionaries was a boat-anchor on the whole system, > and slowed things down so much that he abandoned the project. > The Unladen Swallow documentation indicates that early thinking > on the project was that Olsen's approach would allow getting > rid of the GIL, but later notes indicate that no path to a > GIL-free JIT system is currently in development. > > The multiprocessing module provides semantics similar to > threading with "freezing". Data passed between processes is "frozen" > by pickling. Processes can't modify each other's code. Restrictive > though the multiprocessing module is, it appears to be useful. > It is sometimes recommended as the Pythonic approach to multi-core CPUs. > This is an indication that "freezing" is not unacceptable to the > user community. > > Most of the real-world use cases for extreme dynamism > involve events that happen during startup. Configuration files are > read, modules are selectively included, functions are overridden, tables > of references to functions are set up, regular expressions are compiled, > and the code is brought into the appropriately configured state. Then > the worker threads are started and the real work starts. The > "newthreading" approach allows all that. > > After two decades of failed attempts remove the Global > Interpreter Lock without making performance worse, it is perhaps > time to take a harder look at scaleable threading semantics. > > John Nagle > Animats -- --Guido van Rossum (python.org/~guido)

5 5

Re: [Python-ideas] [Python-Dev] [ANN]: "newthreading" - an approach to simplified thread usage, and a path to getting rid of the GIL
by Adam Olsen June 29, 2010

June 29, 2010

On Sat, Jun 26, 2010 at 10:39, John Nagle <nagle(a)animats.com> wrote: > The rationale behind "freezing" some of the language semantics > when the program goes multi-thread comes from two sources - > Adam Olsen's Safethread work, and the acceptance of the > multiprocessing module. Olsen tried to retain all the dynamism of > the language in a multithreaded environment, but locking all the > underlying dictionaries was a boat-anchor on the whole system, > and slowed things down so much that he abandoned the project. > The Unladen Swallow documentation indicates that early thinking > on the project was that Olsen's approach would allow getting > rid of the GIL, but later notes indicate that no path to a > GIL-free JIT system is currently in development. That's not true. Refcounting was the boat-anchor, not dicts. I was unable to come up with a relatively simple replacement that scaled fully. The dicts shared as module globals and class dicts were a design issue, but more of an ideological one: concurrency mentality says you should only share immutable objects. Python prefers ad-hoc design, where you can do what you want so long as it's not particularly nasty. I was unable to find a way to have both, so I declared the python mentality the winner. The shareddict I came up with uses a read write lock, so that it's safe when you do mutate and doesn't bottleneck when you don't mutate. The only thing fancy was my method of checkpointing when doing a readlock->writelock transition, but there's a hundred other ways to accomplish that.

1 0

Rename time module to "posixtime"
by Alexander Belopolsky June 18, 2010

June 18, 2010

One of the common complains about working with time values in Python, is that it some functionality is available in time module, some in datetime module and some in both. I propose a series of steps towards improving this situation. 1. Create posixtime.py initially containing just "from time import *" 2. Add python implementation of time.* functions to posixtime.py. 3. Rename time module to _posixtime and add time.py with a deprecation warning and "from _posixtime import *". Note that #2 may require to move some code from timemodule.c to datetimemodule.c, but at the binary level code compiled from these files is already linked together in datetimemodule. Moving the necessary code to datetime.c will help to eliminate current circular dependency between time and datetime.

10 19

Issue9004 Was:Rename time module to "posixtime"
by Alexander Belopolsky June 17, 2010

June 17, 2010

On Thu, Jun 17, 2010 at 1:01 AM, Bruce Leban <bruce(a)leapyear.org> wrote: .. > When you say "And where in the docs would you explain the following: :-)" > that sounds like you're saying "this is too confusing we shouldn't document > it." To which I can only say :-( I presented what I consider to be a bug. I opened an issue 9004, [1] "datetime.utctimetuple() should not set tm_isdst flag to 0" for that. There is no point in documenting the following as expected behavior: >>> time.strftime('%c %z %Z', datetime.utcnow().utctimetuple()) 'Wed Jun 16 03:26:26 2010 -0500 EST' I believe it is better to fix it so that it produces >>> time.strftime('%c %z %Z', datetime.utcnow().utctimetuple()) 'Wed Jun 16 03:26:26 2010 ' instead. This, however shows limitation of datetime to timetuple conversion: there is currently no mechanism to store daylight saving time info in datetime object. See issue 9013. [2] Rather than fixing that, it would be much better to eliminate need for datetime to timetuple conversion in the first place. [1] http://bugs.python.org/issue9004 [2] http://bugs.python.org/issue9013

1 0

Re: [Python-ideas] Rename time module to "posixtime"
by Bruce Leban June 17, 2010

June 17, 2010

-1 to moving anything The situation is confusing and moving things will add to that confusion for a significant length of time. What I would instead suggest is improving the docs. If I could look in one place to find any time function it would mitigate the fact that they're implemented in multiple places. --- Bruce (via android) On Jun 16, 2010 12:56 AM, "M.-A. Lemburg" <mal(a)egenix.com> wrote: Brett Cannon wrote: > On Tue, Jun 15, 2010 at 16:01, Cameron Simpson <cs(a)zip.com.au> wrote: >> On 15... -1. Please note that the time module provides access to low-level OS provided services which the datetime module does not expose. You cannot seriously expect that an application which happily uses the time module (only) for its limited date/time functionality to have to be rewritten just to stay compatible with Python. Note that not all applications are interested in sub-second accuracy and a computer without properly configured NTP and good internal clock doesn't even provide this accuracy to begin with (even if they happily pretend they do by exposing sub-second floats). You might want to do that for Python4 and then add all those time module functions using struct_time to the datetime module (returning datetime instances), but for Python3, we've had the stdlib reorg already. Renaming time -> posixtime falls into the same category. The only improvement I could see, would be to move calendar.timegm() to the time module, since that's where it belongs (keeping an alias in the calendar module, of course). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 16 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Ad... 2010-07-19: EuroPython 2010, Birmingham, UK 32 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, ... Python-ideas mailing list Python-ideas(a)python.org http://mail.python.org/mailman/listinfo/python-ide...

2 2

Moving calendar.timegm() to time module Was: Rename time module to "posixtime"
by Alexander Belopolsky June 16, 2010

June 16, 2010

On Wed, Jun 16, 2010 at 1:37 PM, Brett Cannon <brett(a)python.org> wrote: .. >> The only improvement I could see, would be to move >> calendar.timegm() to the time module, since that's where >> it belongs (keeping an alias in the calendar module, of >> course). > > That should definitely happen at some point. > This is discussed in Issue 6280 <http://bugs.python.org/issue6280>. There are several issues with this proposal: 1. According to help(time), """ The Epoch is system-defined; on Unix, it is generally January 1st, 1970. The actual value can be retrieved by calling gmtime(0). """ Current calendar.gmtime implementation ignores this. The solution to this, may be to change help(time), though. 2. Current calendar.gmtime supports float values for hours, minutes and seconds in timedelta tuple. This is probably an unintended implementation artifact, but it is relied upon even in stdlib. See http://bugs.python.org/issue6280#msg107808 .

3 2

Globalize lonely augmented assignment
by Fake Name June 14, 2010

June 14, 2010

Mark Dickinson suggested discussion take place here, rather than at bugs.python http://bugs.python.org/issue8977 To note, his counter example would currently raise an UnboundLocalError _________________________________________________________________ Jeux Messenger : mettez vos amis au défi! http://go.microsoft.com/?linkid=9734397

10 23

reiter: decorator to make generators reiterable
by Mathias Panzenböck June 13, 2010

June 13, 2010

I think this decorator sould be included in itertools: from functools import wraps class ReIter(object): __slots__ = 'f', 'args', 'kwargs' def __init__(self,f,args,kwargs): self.f = f self.args = args self.kwargs = kwargs def __iter__(self): return self.f(*self.args, **self.kwargs) def reiter(f): @wraps(f) def _reiter(*args,**kwargs): return ReIter(f,args,kwargs) return _reiter Using this you can iterate over the return value of a generator decorated with reiter as often as you like (and not just once): @reiter def gen(x,y): for i in xrange(x): yield i*y g = gen(5,3) for x in g: sys.stdout.write('%r\n' % x) sys.stdout.write('\n') for x in g: sys.stdout.write('%r\n' % x) The difference to tee is that old values are not remembered but the generator is evaluated when __iter__() is called. This might come in handy when you implement methods like items(), values() and keys() in a custom dict implementation. Or is there already such a thing and I missed it? -panzi PS: Maybe it should be called regen/ReGen instead of reiter/ReIter? PPS: Funny thing, both "Reiter" and "Regen" are german words. Reiter = rider or tab; Regen = rain.

4 3

Re: [Python-ideas] Globalize lonely augmented assignment
by Bruce Leban June 13, 2010

June 13, 2010

Huh? That makes no sense. a[x]=x is not a = a [] x or anything like that. Language decisions shouldn't be made based on wrong understandings of how the language works. As to the idea of turning a guaranteed run time error into a compile time error I'm usually in favor of that. If it doesn't muck up the compiler. --- Bruce (via android) On Jun 13, 2010 5:54 AM, "Demur Rumed" <junkmute(a)hotmail.com> wrote: > > But if the programmer intended a to be global, the *only* > > reason it's a bug is the current s... a=[1,2,3] def f(x):a[x]=x f(0) Some like to think of []= as a form of augmented assignment Currently, []= doesn't align with other augmenteds on this point That doesn't seem very consistent. Add on that augmented assignment is the only globalizing store statement which also dereferences, and consistency doesn't seem to be a strong point against this proposal ------------------------------ Jeux Messenger : mettez vos amis au défi! Jeux Messenger!<http://go.microsoft.com/?linkid=9734391> _______________________________________________ Python-ideas mailing list Python-ideas(a)python.org http://mail.python.org/mailman/listinfo/python-ideas

1 0

Moving development out of the standard library
by Ian Bicking June 10, 2010

June 10, 2010

OK... after a bit of off-list discussion I realize what I am really concerned about with respect to the standard library wasn't well expressed. So here's my real assertion: There is no reason any new library or functionality should be tied to a Python release. Outside of a few exceptions (like ast or importlib) functionality in the standard library seldom relies on anything in a particular Python release; e.g., code might use conditional expressions, but it never *has* to use conditional expressions. The standard library that most people know and love is really the least common denominator of Python's that person has to handle; for someone writing an open source library that's probably 2.5, for someone using Zope 2 that's traditionally been 2.4, and if you have a controlled environment (e.g., internal development) maybe you can do 2.6. I think there is a general consensus that functionality should not be tied to a Python release, but the results are ad hoc. That is, truly useful libraries that are added to the stdlib are backported, or more often were originally maintained as a library with backward compatibility before being integrated into the standard library. I think we should have a more formalized process about how this functionality is maintained, including a process that considers the years of ongoing maintenance and improvement that should happen on these libraries. (Most specifically without serious thought about this development process I am pessimistic about an orderly or positive inclusion of distutils2 in packaging workflows.) Another alternative is to simply not make improvements to the standard library beyond a very well-defined set of appropriate functionality. This would be much closer to the status quo. Defining what categories would be "appropriate" would be contentious, I am sure, but would sharply focus future discussions. -- Ian Bicking | http://blog.ianbicking.org

13 56

stdlib upgrades
by Ian Bicking June 8, 2010

June 8, 2010

Threading will probably break here as I wasn't on the list for the first email... My concern with the standard library is that there's a couple things going on: 1. The standard library represents "accepted" functionality, kind of best practice, kind of just conventional. Everyone (roughly) knows what you are talking about when you use things from the standard library. 2. The standard library has some firm backward compatibility guarantees. It also has some firm stability guarantees, especially within releases (though in practice, nearly for eternity). 3. The standard library is kind of collectively owned; it's not up to the whims of one person, and can't be abandoned. 4. The standard library is one big chunk of functionality, upgraded all under one version number, and specifically works together (though in practice cross-module refactorings are uncommon). There's positive things about these features, but 4 really drives me nuts, and I think is a strong disincentive to putting stuff into the standard library. For packaging I think 4 actively damages maintainability. Packaging is at the intersection of several systems: * Python versions * Forward and backward compatibility with distributed libraries * System policies (e.g., Debian has changed things around a lot in the last few years) * A whole other ecosystem of libraries outside of Python (e.g., binding to C libraries) * Various developer toolkits, some Python specific (e.g., Cython) some not (gcc) I don't think it's practical to think that we can determine some scope of packaging where it will be stable in the long term, all these things are changing and many are changing without any particular concern for how it affects Python (i.e., packaging must be reactive). And frankly we clearly do not have packaging figured out, we're still circling in on something... and I think the circling will be more like a Strange Attractor than a sink drain. The issues exist for other libraries that aren't packaging-related, of course, it's just worse for packaging. argparse for instance is not "done"... it has bugs that won't be fixed before release, and functionality that it should reasonably include. But there's no path for it to get better. Will it have new and better features in Python 3.3? Who seriously wants to write code that is only compatible with Python 3.3+ just because of some feature in argparse? Instead everyone will work around argparse as it currently exists. In the process they'll probably use undocumented APIs, further calcifying the library and making future improvements disruptive. It's not very specific to argparse, I think ElementTree has similar issues. The json library is fairly unique in that it has a scope that can be "done". I don't know what to say about wsgiref... it's completely irrelevant in Python 3 because it was upgraded along the Python schedule despite being unready to be released (this is relatively harmless as I don't think anyone is using wsgiref in Python 3). So, this is the tension I see. I think aspects of the standard library process and its guarantees are useful, but the current process means releasing code that isn't ready or not releasing code that should be released, and neither is good practice and both compromise those guarantees. Lots of moving versions can indeed be difficult to manage... though it can be made a lot easier with good practices. Though even then distutils2 (and pip) does not even fit into that... they both enter into the workflow before you start working with libraries and versions, making them somewhat unique (though also giving them some more flexibility as they are not so strongly tied to the Python runtime, which is where stability requirements are most needed). -- Ian Bicking | http://blog.ianbicking.org

11 14

Callable properties
by George Sakkis June 7, 2010

June 7, 2010

I'm wondering if there is any downside in making properties callable: class callableproperty(property): def __call__(self, obj): return self.fget(obj) class Foo(object): @property def bar(self): return self @callableproperty def baz(self): return self >>> foo = Foo() >>> foo.baz is Foo.baz(foo) True >>> foo.bar is Foo.bar(foo) ... TypeError: 'property' object is not callable As for the motivation, having callable properties would make it easier to stack them with other decorators that typically expect callables. Am I missing something ? George

5 7

@setattr(obj, [name])
by George Sakkis June 6, 2010

June 6, 2010

It would be nice if setattr() was extended to allow usage as a decorator: class Foo(object): pass @setattr(Foo) def bar(self): print 'bar' @setattr(Foo, 'baz') def get_baz(self): print 'baz' >>> Foo().bar() bar >>> Foo().baz() baz Here's a pure Python implementation: _setattr = setattr def setattr(obj, *args): if len(args) >= 2: return _setattr(obj, *args) return lambda f: _setattr(obj, args[0] if args else f.__name__, f) or f Thoughts ? George

5 7

Date/time literals
by Marcos Bonci June 4, 2010

June 4, 2010

Hi! I've been working a lot with date/time variables lately, and maybe it's just me, and maybe I just don't get it yet, but it sure doesn't feel like there's one-and-only-one elegant, obvious way to work with them. Feels like working with strings in C. So I was thinking, have date and time literals ever been seriously discussed anywhere in these discussion lists? (If so, I apologize in advance for reviving this subject.) Is there any chance we could ever see date/time literals in Python? Perhaps as a top-level abstraction itself, or as a subclass of numbers.Number, or even as common float or int (by adding attributes regarding specific properties of a number's date/time interpretation, e.g. "<2010-12-31>.incr_month(2) == <2011-02-28>" -- just an example though, i don't really think this exact method would be practical). If we adopt a standard like ISO 8601, then we automatically get an unambiguous one-to-one relationship between (date+time) and (amount of time elapsed from <epoch>), over a continuous, infinite domain (towards +inf and -inf). Very often the "time elapsed" value is what we really want (or at least the absolute difference between two values, which is of course easier to find in this form), and besides being more compact and flexible, no functionality is lost. I guess the most controversial point here would be the meaning of arithmetic operations between dates (or the very validity of such operations). Another (not so obvious) problem is leap seconds (btw, who the hell invented that?!..). It's a weird and complicated problem, but I think it's not impossible to turn this "complicated" into just "complex", keeping the tricky concepts buried inside the implementation. Anyway, here's a couple of suggestions for a syntax: 2010.05.29 + 20.06.17 2010.05.29d + 20.06.17t (just like complex literals) So, what do you guys think? I'd love to hear others' opinions, even if it's just for me to understand what I got wrong and see the obvious way that's already there :) PS: Just for fun, is there a standard way I could experiment with creating new types of literals? -- Marcos --

14 39

An identity dict
by Benjamin Peterson June 3, 2010

June 3, 2010

In the spirit of collections.OrderedDict and collections.defaultdict, I'd like to propose collections.identitydict. It would function just like a normal dictionary, but ignore hash values and comparison operators and merely lookup keys based on the key's id(). This dict is very useful for keep track of objects that should not be compared by normal comparison methods. For example, I would use an identitydict to hold weak references that would otherwise fallback to their referant object's hash. An advantage of formalizing this in collections would be to enable other Python implementations like PyPy, where id() is expensive, to provide an optimized identitydict. Regards, Benjamin

20 43

stdlib upgrades
by Tarek Ziadé June 3, 2010

June 3, 2010

Hello, That's not a new idea, but I'd like to throw it here again. Some modules/packages in the stdlib are pretty isolated, which means that they could be upgraded with no harm, independently from the rest. For example the unittest package, or the email package. Here's an idea: 1 - add a version number in each package or module of the stdlib that is potentially upgradable 2 - create standalone releases of these modules/packages at PyPI, in a restricted area 'stdlib upgrades' that can be used only by core devs to upload new versions. Each release lists the precise Python versions it's compatible with. 3 - once distutils2 is back in the stdlib, provide a command line interface to list upgradable packages, and make it possible to upgrade them 4 - an upgraded package lands in a new specific site-packages directory and is loaded *before* the one in Lib Regards Tarek -- Tarek Ziadé | http://ziade.org

6 9

keyfuncdict -- the good, the bad, and the ugly
by Raymond Hettinger June 2, 2010

June 2, 2010

Was thinking of all the things that could be done with Antoine's generalization: The Good ======== d = keyfuncdict(key=str.lower) # case insensitive dict def track(obj): logging.info(obj) return obj d = keyfuncdict(key=track) # monitored dict d = keyfuncdict(key=id) # makes benjamin happy The Bad ======= d = keyfuncdict(key=tuple) # lets you use lists as keys d = keyfuncdict(key=repr) # support many kinds of mutable or unhashable keys d = keyfuncdict(key=pickle.loads) # use anything picklable as a key d = keyfuncdict(key=getuser) # track one most recent entry per user The Ugly ======== d = keyfuncdict(key=random.random) # just plain weird d = keyfuncdict(key=itertools.count().next) # all entries are unique and unretrievable ;-) def remove(obj): d.pop(obj) return obj d = keyfuncdict(key=remove) # self deleting dict ;-) Raymond

2 1

Having unbound methods refer to the classes their defined on
by cool-RR June 2, 2010

June 2, 2010

Hello, I would like to raise an issue here that I've been discussing at python-porting. (And I'd like to preface by saying that I'm not intimately familiar with Python's innards, so if I make any mistakes please correct me.) In Python 2.x there was an "unbound method" type. An unbound method would have an attribute `.im_class` that would refer to the class on which the method was defined. This allowed users to use the `copy_reg` module to pickle unbound methods by name. (In a similar way to how functions and classes are pickled by default.) In Python 3.x unbound methods are plain functions. There is no way of knowing on which class they are defined, so therefore it's impossible to pickle them. It is even impossible to tell `copyreg` to use a custom reducer: http://stackoverflow.com/questions/2932742/python-using-copyreg-to-define-r… (To the people who wonder why would anyone want to pickle unbound methods: I know that it sounds like a weird thing to do. Keep in mind that sometimes your objects need to get pickled. For example if you're using the multiprocessing module, and you pass into it an object that somehow refers to an unbound method, then that method has to be picklable.) The idea is: Let's give unbound methods an attribute that will refer to the class on which they were defined. What do you think? Ram.

3 3