Mailman 3 June 2010 - Python-ideas

i18n and Python tracebacks
by Andre Roberge Oct. 28, 2012

Oct. 28, 2012

I think it would be a good idea if Python tracebacks could be translated into languages other than English - and it would set a good example. For example, using French as my default local language, instead of >>> 1/0 Traceback (most recent call last): File "<stdin>", line 1, in <module> ZeroDivisionError: integer division or modulo by zero I might get something like >>> 1/0 Suivi d'erreur (appel le plus récent en dernier) : Fichier "<stdin>", à la … [View More]

9 15

explicitation lines in python ?
by Daniel DELAY July 12, 2010

July 12, 2010

If we could explicitate a too complex expression in an indented next line, I would use this feature very often : htmltable = ''.join( '<tr>{}</tr>'.format(htmlline) for line in table) : # main line htmlline : ''.join( '<td>{}</td>'.format(cell) for cell in line) # explicitation(s) line(s) (Sorry if this has already been discussed earlier on this list, I have not read all the archives) ******* in details : List comprehension "<expression> for … [View More]x in mylist" often greatly improve readability of python programs, when <expression> is not too complex When <expression> is too complex (ex: nested lists), this become not readable, so we have to find another solution : a) defining a function expression(x), or an iterator function, wich will only used once in the code b) or droping this beautiful syntax to replace it the very basic list construction : newlist = [] for x in myiterable newlist.append(<expression>) I often choose b), but I dislike both solutions : - in solution a) function def can be far from list comprehension, in fact instructions to build the new list are split in two different places in the code. - solution b) seems a bit better to me, but the fact we build a new list from myiterable is not visible in a glance, unlike list comprehensions. Renouncing to list comprehension occurs rather often when I write python code I think we could greatly improve readability if we could keep list comprehension anywhere in all cases, but when necessary explicit a too complex expression in an indented line : htmltable = ''.join( '<tr>{}</tr>'.format(htmlline) for line in table) : # main line htmlline : ''.join( '<td>{}</td>'.format(cell) for cell in line) # explicitation(s) line(s) In the case the main line is the header of a "classical" indented block (starting with "for", "if", "with"...) , this idented block would simply follow the explicitation(s) line(s). The explicitations lines can be surely identified as the lines than begin with "identifier :" (when we are not in an unclosed dict) with open('data.txt') as f : if line in enumerate(mylist) : # main line mylist : f.read().strip().lower() # explicitation(s) line(s) print line # "classical" indented block Another possible use of "explicitations lines" is a coding style wich would start by "the whole picture" first, and completing with details after, wich is the usual way we mentally solve problems. Let's take an example : we want to write a function wich returns a multiplication table in a simle html document. When I solve this problem, il think a bit like that : - I need to return an html page. For that I need a "header" and a "body". My body will contain an "htmltable", wich be built from a "table" of numbers etc. My code could look like that : def tag(content, *tags): # little convenient function retval = content for t in tags: retval = '<{0}>{1}</{0}>'.format(t, retval) return retval def xhtml_mult_table(a, b): return tag(header + body, 'html') : header : tag('multiplication table', 'title') body : tag(htmltable, 'tbody', 'table', 'body') : htmltable : ''.join(tag(xhtmlline, 'tr') for line in table) : table : headerline + otherlines : headerline : [[''] + range(a)] otherlines : [[y] + [x*y for x in range(a)] for y in range(b)] xhtmlline : ''.join(tag(str(cell), 'td') for cell in line) This example is a "heavy use" of the "explicitation line" feature, to illustrate how it could work. I don't mean this should replace the "classical" syntax everywhere possible, but for me this would be for me a nice way to explicitate complex expressions from time to time, and the ability to use list comprehension everywhere I wan't. Daniel [View Less]

8 17

Maybe allow br"" or rb"" e.g., for bytes regexes in Py3?
by Mark Summerfield June 30, 2010

June 30, 2010

Hi, Python 3 has two string prefixes r"" for raw strings and b"" for bytes. So if you want to create a regex based on bytes as far as I can tell you have to do something like this: FONTNAME_RE = re.compile(r"/FontName\s+/(\S+)".encode("ascii")) # or FONTNAME_RE = re.compile(b"/FontName\\s+/(\\S+)") I think it would be much nicer if one could write: FONTNAME_RE = re.compile(br"/FontName\s+/(\S+)") # or FONTNAME_RE = re.compile(rb"/FontName\s+/(\S+)") I _slightly_ … [View More]

7 7

feature to make traceback objects usable without references to frame locals and globals
by ghazel＠gmail.com June 30, 2010

June 30, 2010

Hi, I'm interested in a feature which allows users to discard the locals and globals references from frames held by a traceback object. Currently, traceback objects are used when capturing and re-raising exceptions. However, they hold a reference to all frames, which hold a reference to their locals and globals. These are not needed by the default traceback output, and can cause serious memory bloat if a reference to a traceback object is kept for any significant length of time, and there are … [View More]

11 31

Re: [Python-ideas] [Python-Dev] [ANN]: "newthreading" - an approach to simplified thread usage, and a path to getting rid of the GIL
by Guido van Rossum June 29, 2010

June 29, 2010

I'm moving this thread to python-ideas, where it belongs. I've looked at the implementation code (even stepped through it with pdb!), read the sample/test code, and read the two papers on animats.com fairly closely (they have a lot of overlap, and the memory model described below seems copied verbatim from http://www.animats.com/papers/languages/pythonconcurrency.html version 0.8). Some reactions (trying to hide my responses to the details of the code): - First of all, I'm very happy to see … [View More]radical ideas proposed, even if they are at present unrealistic. We need a big brainstorm to come up with ideas from which an eventual solution to the multicore problem might be chosen. (Jesse Noller's multiprocessing is another; Adam Olsen's work yet another, at a different end of the spectrum.) - The proposed new semantics (frozen objects, memory model, auto-freezing of globals, enforcement of naming conventions) are radically different from Python's current semantics. They will break every 3rd party library in many more ways than Python 3. This is not surprising given the goals of the proposal (and its roots in Adam Olsen's work) but places a huge roadblock for acceptance. I see no choice but to keep trying to come up with a compromise that is more palatable and compatible without throwing away all the advantages. As it now stands, the proposal might as well be a new and different language. - SynchronizedObject looks like a mixture of a Java synchronized class (a non-standard concept in Java but easily understood as a class all whose public methods are synchronized) and a condition variable (which has the same semantics of releasing the lock while waiting but without crawling the stack for other locks to release). It looks like the examples showing off SynchronizedObject could be implemented just as elegantly using a condition variable (and voluntary abstention from using shared mutable objects). - If the goal is to experiment with new control structures, I recommend decoupling them from the memory model and frozen objects, instead relying (as is traditional in Python) on programmer caution to avoid races. This would make it much easier to see how programmers respond to the new control structures. - You could add the freeze() function for voluntary use, and you could even add automatic wrapping of arguments and return values for certain classes using a class decorator or a metaclass, but the performance overhead makes this unlikely to win over many converts. I don't see much use for the "whole program freezing" done by the current prototype -- there are way too many backdoors in Python for the prototype approach to be anywhere near foolproof, and if we want a non-foolproof approach, voluntary constraint (and, in some cases, voluntary, i.e. explicit, wrapping of modules or classes) would work just as well. - For a larger-scale experiment with the new memory model and semantic restrictions (or would it be better to call them syntactic restrictions? -- after all they are about statically detectable properties like naming conventions) I recommend looking at PyPy, which has as one of its explicitly stated project goals easy experimentation with different object models. - I'm sure I've forgotten something, but I wanted to keep my impressions fresh. - Again, John, thanks for taking the time to come up with an implementation of your idea! --Guido On Sat, Jun 26, 2010 at 9:39 AM, John Nagle <nagle(a)animats.com> wrote: > On 6/26/2010 7:44 AM, Jesse Noller wrote: >> >> On Sat, Jun 26, 2010 at 9:29 AM, Michael Foord >> <fuzzyman(a)voidspace.org.uk> wrote: >>> >>> On 26/06/2010 07:11, John Nagle wrote: >>>> >>>> We have just released a proof-of-concept implementation of a new >>>> approach to thread management - "newthreading". > > .... > >>> The import * form is considered bad practise in *general* and >>> should not be recommended unless there is a good reason. > > I agree. I just did that to make the examples cleaner. > >>> however the introduction of free-threading in Python has not been >>> hampered by lack of synchronization primitives but by the >>> difficulty of changing the interpreter without unduly impacting >>> single threaded code. > > That's what I'm trying to address here. > >>> Providing an alternative garbage collection mechanism other than >>> reference counting would be a more interesting first-step as far as >>> I can see, as that removes the locking required around every access >>> to an object (which currently touches the reference count). >>> Introducing free-threading by *changing* the threading semantics >>> (so you can't share non-frozen objects between threads) would not >>> be acceptable. That comment is likely to be based on a >>> misunderstanding of your future intentions though. :-) > > This work comes out of a discussion a few of us had at a restaurant > in Palo Alto after a Stanford talk by the group at Facebook which > is building a JIT compiler for PHP. We were discussing how to > make threading both safe for the average programmer and efficient. > Javascript and PHP don't have threads at all; Python has safe > threading, but it's slow. C/C++/Java all have race condition > problems, of course. The Facebook guy pointed out that you > can't redefine a function dynamically in PHP, and they get > a performance win in their JIT by exploiting this. > > I haven't gone into the memory model in enough detail in the > technical paper. The memory model I envision for this has three > memory zones: > > 1. Shared fully-immutable objects: primarily strings, numbers, > and tuples, all of whose elements are fully immutable. These can > be shared without locking, and reclaimed by a concurrent garbage > collector like Boehm's. They have no destructors, so finalization > is not an issue. > > 2. Local objects. These are managed as at present, and > require no locking. These can either be thread-local, or local > to a synchronized object. There are no links between local > objects under different "ownership". Whether each thread and > object has its own private heap, or whether there's a common heap with > locks at the allocator is an implementation decision. > > 3. Shared mutable objects: mostly synchronized objects, but > also immutable objects like tuples which contain references > to objects that aren't fully immutable. These are the high-overhead > objects, and require locking during reference count updates, or > atomic reference count operations if supported by the hardware. > The general idea is to minimize the number of objects in this > zone. > > The zone of an object is determined when the object is created, > and never changes. This is relatively simple to implement. > Tuples (and frozensets, frozendicts, etc.) are normally zone 2 > objects. Only "freeze" creates collections in zones 1 and 3. > Synchronized objects are always created in zone 3. > There are no difficult handoffs, where an object that was previously > thread-local now has to be shared and has to acquire locks during > the transition. > > Existing interlinked data structures, like parse trees and GUIs, > are by default zone 2 objects, with the same semantics as at > present. They can be placed inside a SynchronizedObject if > desired, which makes them usable from multiple threads. > That's optional; they're thread-local otherwise. > > The rationale behind "freezing" some of the language semantics > when the program goes multi-thread comes from two sources - > Adam Olsen's Safethread work, and the acceptance of the > multiprocessing module. Olsen tried to retain all the dynamism of > the language in a multithreaded environment, but locking all the > underlying dictionaries was a boat-anchor on the whole system, > and slowed things down so much that he abandoned the project. > The Unladen Swallow documentation indicates that early thinking > on the project was that Olsen's approach would allow getting > rid of the GIL, but later notes indicate that no path to a > GIL-free JIT system is currently in development. > > The multiprocessing module provides semantics similar to > threading with "freezing". Data passed between processes is "frozen" > by pickling. Processes can't modify each other's code. Restrictive > though the multiprocessing module is, it appears to be useful. > It is sometimes recommended as the Pythonic approach to multi-core CPUs. > This is an indication that "freezing" is not unacceptable to the > user community. > > Most of the real-world use cases for extreme dynamism > involve events that happen during startup. Configuration files are > read, modules are selectively included, functions are overridden, tables > of references to functions are set up, regular expressions are compiled, > and the code is brought into the appropriately configured state. Then > the worker threads are started and the real work starts. The > "newthreading" approach allows all that. > > After two decades of failed attempts remove the Global > Interpreter Lock without making performance worse, it is perhaps > time to take a harder look at scaleable threading semantics. > > John Nagle > Animats -- --Guido van Rossum (python.org/~guido) [View Less]

5 5

Re: [Python-ideas] [Python-Dev] [ANN]: "newthreading" - an approach to simplified thread usage, and a path to getting rid of the GIL
by Adam Olsen June 29, 2010

June 29, 2010

On Sat, Jun 26, 2010 at 10:39, John Nagle <nagle(a)animats.com> wrote: > The rationale behind "freezing" some of the language semantics > when the program goes multi-thread comes from two sources - > Adam Olsen's Safethread work, and the acceptance of the > multiprocessing module. Olsen tried to retain all the dynamism of > the language in a multithreaded environment, but locking all the > underlying dictionaries was a boat-anchor on the whole system, … [View More]

1 0

Rename time module to "posixtime"
by Alexander Belopolsky June 17, 2010

June 17, 2010

One of the common complains about working with time values in Python, is that it some functionality is available in time module, some in datetime module and some in both. I propose a series of steps towards improving this situation. 1. Create posixtime.py initially containing just "from time import *" 2. Add python implementation of time.* functions to posixtime.py. 3. Rename time module to _posixtime and add time.py with a deprecation warning and "from _posixtime import *". Note that #2 … [View More]

10 19

Issue9004 Was:Rename time module to "posixtime"
by Alexander Belopolsky June 17, 2010

June 17, 2010

On Thu, Jun 17, 2010 at 1:01 AM, Bruce Leban <bruce(a)leapyear.org> wrote: .. > When you say "And where in the docs would you explain the following: :-)" > that sounds like you're saying "this is too confusing we shouldn't document > it." To which I can only say :-( I presented what I consider to be a bug. I opened an issue 9004, [1] "datetime.utctimetuple() should not set tm_isdst flag to 0" for that. There is no point in documenting the following as expected behavior: >&… [View More]

1 0

Re: [Python-ideas] Rename time module to "posixtime"
by Bruce Leban June 17, 2010

June 17, 2010

-1 to moving anything The situation is confusing and moving things will add to that confusion for a significant length of time. What I would instead suggest is improving the docs. If I could look in one place to find any time function it would mitigate the fact that they're implemented in multiple places. --- Bruce (via android) On Jun 16, 2010 12:56 AM, "M.-A. Lemburg" <mal(a)egenix.com> wrote: Brett Cannon wrote: > On Tue, Jun 15, 2010 at 16:01, Cameron Simpson <cs(a)zip.com.… [View More]

2 2

Moving calendar.timegm() to time module Was: Rename time module to "posixtime"
by Alexander Belopolsky June 16, 2010

June 16, 2010

On Wed, Jun 16, 2010 at 1:37 PM, Brett Cannon <brett(a)python.org> wrote: .. >> The only improvement I could see, would be to move >> calendar.timegm() to the time module, since that's where >> it belongs (keeping an alias in the calendar module, of >> course). > > That should definitely happen at some point. > This is discussed in Issue 6280 <http://bugs.python.org/issue6280>. There are several issues with this proposal: 1. According to help(time),… [View More]

3 2

Globalize lonely augmented assignment
by Fake Name June 14, 2010

June 14, 2010

Mark Dickinson suggested discussion take place here, rather than at bugs.python http://bugs.python.org/issue8977 To note, his counter example would currently raise an UnboundLocalError _________________________________________________________________ Jeux Messenger : mettez vos amis au défi! http://go.microsoft.com/?linkid=9734397

10 23

reiter: decorator to make generators reiterable
by Mathias Panzenböck June 13, 2010

June 13, 2010

I think this decorator sould be included in itertools: from functools import wraps class ReIter(object): __slots__ = 'f', 'args', 'kwargs' def __init__(self,f,args,kwargs): self.f = f self.args = args self.kwargs = kwargs def __iter__(self): return self.f(*self.args, **self.kwargs) def reiter(f): @wraps(f) def _reiter(*args,**kwargs): return ReIter(f,args,kwargs) return _reiter Using this you can iterate over the return value of a generator decorated with reiter … [View More]

4 3

Re: [Python-ideas] Globalize lonely augmented assignment
by Bruce Leban June 13, 2010

June 13, 2010

Huh? That makes no sense. a[x]=x is not a = a [] x or anything like that. Language decisions shouldn't be made based on wrong understandings of how the language works. As to the idea of turning a guaranteed run time error into a compile time error I'm usually in favor of that. If it doesn't muck up the compiler. --- Bruce (via android) On Jun 13, 2010 5:54 AM, "Demur Rumed" <junkmute(a)hotmail.com> wrote: > > But if the programmer intended a to be global, the *only* > … [View More]

1 0

Moving development out of the standard library
by Ian Bicking June 9, 2010

June 9, 2010

OK... after a bit of off-list discussion I realize what I am really concerned about with respect to the standard library wasn't well expressed. So here's my real assertion: There is no reason any new library or functionality should be tied to a Python release. Outside of a few exceptions (like ast or importlib) functionality in the standard library seldom relies on anything in a particular Python release; e.g., code might use conditional expressions, but it never *has* to use conditional … [View More]

13 56

stdlib upgrades
by Ian Bicking June 8, 2010

June 8, 2010

Threading will probably break here as I wasn't on the list for the first email... My concern with the standard library is that there's a couple things going on: 1. The standard library represents "accepted" functionality, kind of best practice, kind of just conventional. Everyone (roughly) knows what you are talking about when you use things from the standard library. 2. The standard library has some firm backward compatibility guarantees. It also has some firm stability guarantees, … [View More]especially within releases (though in practice, nearly for eternity). 3. The standard library is kind of collectively owned; it's not up to the whims of one person, and can't be abandoned. 4. The standard library is one big chunk of functionality, upgraded all under one version number, and specifically works together (though in practice cross-module refactorings are uncommon). There's positive things about these features, but 4 really drives me nuts, and I think is a strong disincentive to putting stuff into the standard library. For packaging I think 4 actively damages maintainability. Packaging is at the intersection of several systems: * Python versions * Forward and backward compatibility with distributed libraries * System policies (e.g., Debian has changed things around a lot in the last few years) * A whole other ecosystem of libraries outside of Python (e.g., binding to C libraries) * Various developer toolkits, some Python specific (e.g., Cython) some not (gcc) I don't think it's practical to think that we can determine some scope of packaging where it will be stable in the long term, all these things are changing and many are changing without any particular concern for how it affects Python (i.e., packaging must be reactive). And frankly we clearly do not have packaging figured out, we're still circling in on something... and I think the circling will be more like a Strange Attractor than a sink drain. The issues exist for other libraries that aren't packaging-related, of course, it's just worse for packaging. argparse for instance is not "done"... it has bugs that won't be fixed before release, and functionality that it should reasonably include. But there's no path for it to get better. Will it have new and better features in Python 3.3? Who seriously wants to write code that is only compatible with Python 3.3+ just because of some feature in argparse? Instead everyone will work around argparse as it currently exists. In the process they'll probably use undocumented APIs, further calcifying the library and making future improvements disruptive. It's not very specific to argparse, I think ElementTree has similar issues. The json library is fairly unique in that it has a scope that can be "done". I don't know what to say about wsgiref... it's completely irrelevant in Python 3 because it was upgraded along the Python schedule despite being unready to be released (this is relatively harmless as I don't think anyone is using wsgiref in Python 3). So, this is the tension I see. I think aspects of the standard library process and its guarantees are useful, but the current process means releasing code that isn't ready or not releasing code that should be released, and neither is good practice and both compromise those guarantees. Lots of moving versions can indeed be difficult to manage... though it can be made a lot easier with good practices. Though even then distutils2 (and pip) does not even fit into that... they both enter into the workflow before you start working with libraries and versions, making them somewhat unique (though also giving them some more flexibility as they are not so strongly tied to the Python runtime, which is where stability requirements are most needed). -- Ian Bicking | http://blog.ianbicking.org [View Less]

11 14

Callable properties
by George Sakkis June 7, 2010

June 7, 2010

I'm wondering if there is any downside in making properties callable: class callableproperty(property): def __call__(self, obj): return self.fget(obj) class Foo(object): @property def bar(self): return self @callableproperty def baz(self): return self >>> foo = Foo() >>> foo.baz is Foo.baz(foo) True >>> foo.bar is Foo.bar(foo) ... TypeError: 'property' object is not callable As for the motivation, having callable … [View More]

5 7

@setattr(obj, [name])
by George Sakkis June 6, 2010

June 6, 2010

It would be nice if setattr() was extended to allow usage as a decorator: class Foo(object): pass @setattr(Foo) def bar(self): print 'bar' @setattr(Foo, 'baz') def get_baz(self): print 'baz' >>> Foo().bar() bar >>> Foo().baz() baz Here's a pure Python implementation: _setattr = setattr def setattr(obj, *args): if len(args) >= 2: return _setattr(obj, *args) return lambda f: _setattr(obj, args[0] if args else f.__name__, f) or f Thoughts ? George

5 7

Date/time literals
by Marcos Bonci June 4, 2010

June 4, 2010

Hi! I've been working a lot with date/time variables lately, and maybe it's just me, and maybe I just don't get it yet, but it sure doesn't feel like there's one-and-only-one elegant, obvious way to work with them. Feels like working with strings in C. So I was thinking, have date and time literals ever been seriously discussed anywhere in these discussion lists? (If so, I apologize in advance for reviving this subject.) Is there any chance we could ever see date/time literals in Python? … [View More]

14 39

An identity dict
by Benjamin Peterson June 3, 2010

June 3, 2010

In the spirit of collections.OrderedDict and collections.defaultdict, I'd like to propose collections.identitydict. It would function just like a normal dictionary, but ignore hash values and comparison operators and merely lookup keys based on the key's id(). This dict is very useful for keep track of objects that should not be compared by normal comparison methods. For example, I would use an identitydict to hold weak references that would otherwise fallback to their referant object's hash. … [View More]

20 43

stdlib upgrades
by Tarek Ziadé June 3, 2010

June 3, 2010

Hello, That's not a new idea, but I'd like to throw it here again. Some modules/packages in the stdlib are pretty isolated, which means that they could be upgraded with no harm, independently from the rest. For example the unittest package, or the email package. Here's an idea: 1 - add a version number in each package or module of the stdlib that is potentially upgradable 2 - create standalone releases of these modules/packages at PyPI, in a restricted area 'stdlib upgrades' that can … [View More]

6 9

keyfuncdict -- the good, the bad, and the ugly
by Raymond Hettinger June 2, 2010

June 2, 2010

Was thinking of all the things that could be done with Antoine's generalization: The Good ======== d = keyfuncdict(key=str.lower) # case insensitive dict def track(obj): logging.info(obj) return obj d = keyfuncdict(key=track) # monitored dict d = keyfuncdict(key=id) # makes benjamin happy The Bad ======= d = keyfuncdict(key=tuple) # lets you use lists as keys d = keyfuncdict(key=repr) # support … [View More]

2 1

Having unbound methods refer to the classes their defined on
by cool-RR June 2, 2010

June 2, 2010

Hello, I would like to raise an issue here that I've been discussing at python-porting. (And I'd like to preface by saying that I'm not intimately familiar with Python's innards, so if I make any mistakes please correct me.) In Python 2.x there was an "unbound method" type. An unbound method would have an attribute `.im_class` that would refer to the class on which the method was defined. This allowed users to use the `copy_reg` module to pickle unbound methods by name. (In a similar way to … [View More]

3 3