Mailman 3 November 2005 - Python-Dev

DRAFT: python-dev Summary for 2005-10-01 to 2005-10-15
by Tony Meyer 16 Nov '05

16 Nov '05

As you have noticed, there has been a summary delay recently. This is my fault (insert your favourite thesis/work/leisure excuse here). Steve has generously covered my slackness by doing all of the October summaries himself (thanks!). Anyway, if you have some moments to spare, cast your mind back to the start of October, and see if these reflect what happened. Comments/corrections to tony.meyer(a)gmail.com or steven.bethard(a)gmail.com. Thanks! ============= Announcements ============= ---------------------------- QOTF: Quote of the Fortnight ---------------------------- From Phillip J. Eby: So, if threads are "easy" in Python compared to other langauges, it's *because of* the GIL, not in spite of it. Contributing thread: - `Pythonic concurrency <http://mail.python.org/pipermail/python-dev/ 2005-October/057062.html>`__ [SJB] ---------------------------------------- GCC/G++ Issues on Linux: Patch available ---------------------------------------- Christoph Ludwig provided the previously `promised patch`_ to address some of the issues in compiling Python with GCC/G++ on Linux. The patch_ keeps ELF systems like x86 / Linux from having any dependencies on the C++ runtime, and allows systems that require main () to be a C++ function to be configured appropriately. .. _promised patch: http://www.python.org/dev/summary/ 2005-07-01_2005-07-15.html#gcc-g-issues-on-linux .. _patch: http://python.org/sf/1324762 Contributing thread: - `[C++-sig] GCC version compatibility <http://mail.python.org/ pipermail/python-dev/2005-October/057230.html>`__ [SJB] ========= Summaries ========= --------------------- Concurrency in Python --------------------- Michael Sparks spent a bit of time descibing the current state and future goals of the Kamaelia_ project. Mainly, Kamaelia aims to make concurrency as simple and easy to use as possible. A scheduler manages a set of generators that communicate with each other through Queues. The long term goals include being able to farm the various generators off into thread or processes as needed, so that whether your concurrency model is cooperative, threaded or process-based, your code can basically look the same. There was also continued discussion about how "easy" threads are. Shane Hathaway made the point that it's actually locking that's "insanely difficult", and approaches that simplify how much you need to think about locking can keep threading relatively easy -- this was one of the strong points of ZODB. A fairly large camp also got behind the claim that threads are easy if you're limited to only message passing. There were also a few comments about how Python makes threading easier, e.g. through the GIL (see `QOTF: Quote of the Fortnight`_) and through threading.threads's encapsulation of thread- local resources as instance attributes. .. _Kamaelia: http://kamaelia.sourceforge.ne Contributing threads: - `Pythonic concurrency - cooperative MT <http://mail.python.org/ pipermail/python-dev/2005-October/056898.html>`__ - `Pythonic concurrency <http://mail.python.org/pipermail/python-dev/ 2005-October/057023.html>`__ [SJB] ------------------------------------- Organization of modules for threading ------------------------------------- A few people took issue with the current organization of the threading modules into Queue, thread and threading. Guido views Queue as an application of threading, so putting it in the threading module is inappropriate (though with a deeper package structure, it should definitely be a sibling). Nick Coghlan suggested that Queue should be in a threadtools module (in parallel with itertools), while Skip proposed a hierarchy of modules with thread and lock being in the lowest level one, and Thread and Queue being in the highest level. Aahz suggested (and Guido approved) deprecating the thread module and renaming it to _thread at least in Python 3.0. It seems the deprecation may happen sooner though. Contributing threads: - `Making Queue.Queue easier to use <http://mail.python.org/pipermail/ python-dev/2005-October/057184.html>`__ - `Autoloading? (Making Queue.Queue easier to use) <http:// mail.python.org/pipermail/python-dev/2005-October/057216.html>`__ - `threadtools (was Re: Autoloading? (Making Queue.Queue easier to use)) <http://mail.python.org/pipermail/python-dev/2005-October/ 057262.html>`__ - `Threading and synchronization primitives <http://mail.python.org/ pipermail/python-dev/2005-October/057269.html>`__ [SJB] ------------------------- Speed of Unicode decoding ------------------------- Tony Nelson found that decoding with a codec like mac-roman or iso8859-1 can take around ten times as long as decoding with utf-8. Walter Dˆrwald provided a patch_ that implements the mapping using a unicode string of length 256 where undefined characters are mapped to u"\ufffd". This dropped the decode time for mac-roman to nearly the speed of the utf-8 decoding. Hye-Shik Chang showed off a fastmap decoder with comparable performance. In the end, Walter's patch was accepted. .. patch: http://www.python.org/sf/1313939 Contributing thread: - `Unicode charmap decoders slow <http://mail.python.org/pipermail/ python-dev/2005-October/056958.html>`__ [SJB] ------------------ Updates to PEP 343 ------------------ Jason Orendorff proposed replacing the __enter__() and __exit__() methods on context managers with a simple __with__() method instead. While Guido was unconvinced that __enter__() and __exit__() should be dropped, he was convinced that context managers should have a __with__ () method for in parallel with the __iter__() method for iterators. There was some talk of special-casing the @contextmanager decorator on the __with__() method, but no conclusion. Contributing threads: - `Proposed changes to PEP 343 <http://mail.python.org/pipermail/ python-dev/2005-October/057040.html>`__ - `PEP 343 and __with__ <http://mail.python.org/pipermail/python-dev/ 2005-October/056931.html>`__ [SJB] ---------------------- str and unicode issues ---------------------- Martin Blais wanted to completely disable the implicit conversions between unicode and str, so that you would always be forced to call either .encode() or .decode() to convert between one and the other. This is already available through adding ``sys.setdefaultencoding ('undefined')`` to your sitecustomize.py file, but the suggestion started another long discussion over unicode issues. Antoine Pitrou suggested that a good rule of thumb is to convert to unicode everything that is semantically textual, and to only use str for what is to be semantically treated as a string of bytes. Fredrik Lundh argued against this for efficiency reasons -- pure ASCII text would consume more space as a unicode object. There were suggestions that in Python 3.0, opening files in text mode will require an encoding and produce string objects, while opening files in binary mode will produce bytes objects. The bytes() type will be a mutable array of bytes, which can be converted to a string object by specifying an encoding. Contributing threads: - `Divorcing str and unicode (no more implicit conversions). <http:// mail.python.org/pipermail/python-dev/2005-October/056916.html>`__ - `unifying str and unicode <http://mail.python.org/pipermail/python- dev/2005-October/056934.html>`__ - `bytes type <http://mail.python.org/pipermail/python-dev/2005- October/056945.html>`__ [SJB] ---------------------------------------------------------------------- Allowing \*args syntax in tuple unpacking and before keyword arguments ---------------------------------------------------------------------- Gustavo Niemeyer propsed the oft-seen request for allowing the \*args syntax in tuple unpacking, e.g.:: for first, second, *rest in iterator: Guido requested a PEP, saying that he wasn't convinced that there was much of a gain over the already valid:: for item in iterator: (first, second), rest = item[2:], item[:2] Greg Ewing and others didn't like Guido's suggestion as it violates DRY (Don't Repeat Yourself). Others also chimed in with some examples in support of the proposal, but no one has yet put together a PEP. In a related matter, Guido indicated that he wants to be able to write keyword-only arguments after a \*args, so that you could, for example, write:: f(a, b, *args, foo=1, bar=2, **kwds) People seemed almost unanimously in support of this proposal, but, to quote Nick Coghlan, it has still "never bugged anyone enough for them to actaully get around to fixing it". Contributing thread: - `Extending tuple unpacking <http://mail.python.org/pipermail/python- dev/2005-October/057056.html>`__ [SJB] ---------- AST Branch ---------- Guido gave the AST branch a three week ultimatum: either the branch should be merged into MAIN within the next three weeks, or the branch should be abandoned entirely. This jump-started work on the branch, and the team was hoping to merge the changes the weekend of October 15th. Contributing threads: - `Python 2.5a1, ast-branch and PEP 342 and 343 <http:// mail.python.org/pipermail/python-dev/2005-September/056449.html>`__ - `Python 2.5 and ast-branch <http://mail.python.org/pipermail/python- dev/2005-October/056986.html>`__ - `AST branch update <http://mail.python.org/pipermail/python-dev/ 2005-October/057281.html>`__ [SJB] ----------------------------------- Allowing "return obj" in generators ----------------------------------- Piet Delport suggested having ``return obj`` in generators be translated into ``raise StopIteration(obj)``. The return value of a generator function would thus be available as the first arg in the StopIteration exception. Guido asked for some examples to give the idea a better motivation, and felt uncomfortable with the return value being silently ignored in for-loops. The idea was postponed until at least one release after a PEP 342 implementation enters Python, so that people can have some more experience with coroutines. Contributing threads: - `Proposal for 2.5: Returning values from PEP 342 enhanced generators <http://mail.python.org/pipermail/python-dev/2005-October/ 056957.html>`__ - `PEP 342 suggestion: start(), __call__() and unwind_call() methods <http://mail.python.org/pipermail/python-dev/2005-October/ 057042.html>`__ - `New PEP 342 suggestion: result() and allow "return with arguments" in generators (was Re: PEP 342 suggestion: start(), __call__() and unwind_call() methods) <http://mail.python.org/ pipermail/python-dev/2005-October/057116.html>`__ [SJB] ----------------------------- API for the line-number table ----------------------------- Greg Ewing suggested trying to simplify the line-number table (lnotab) by simply matching each byte-code index with a file and line number. Phillip J. Eby pointed out that this would make the stdlib take up an extra megabyte, suggesting two tables instead, one matching bytecodes to line numbers, and one matching the first line- number of a chunk with its file. Michael Hudson suggested that what we really want is an API for accessing the lnotab, so that the implementation that is chosen is less important. The conversation trailed off without a resolution. Contributing thread: - `Simplify lnotab? (AST branch update) <http://mail.python.org/ pipermail/python-dev/2005-October/057285.html>`__ [SJB] ------------------------------ Current directory and sys.path ------------------------------ A question about the status of `the CurrentVersion registry entry`_ led to a discussion about the different behaviors of sys.path across platforms. Apparently, on Windows, sys.path includes the current directory and the directory of the script being executed, while on Linux, it only includes the directory of the script. .. _the CurrentVersion registry entry: http://www.python.org/windows/ python/registry.html Contributing thread: - `PythonCore\CurrentVersion <http://mail.python.org/pipermail/python- dev/2005-October/057095.html>`__ [SJB] ---------------------------------- Changing the __class__ of builtins ---------------------------------- As of Python 2.3, you can no longer change the __class__ of any builtin. Phillip J. Eby suggested that these rules might be overly strict; modules and other mutable objects could probably reasonably have their __class__s changed. No one seemed really opposed to the idea, but no one offered up a patch to make the change either. Contributing thread: - `Assignment to __class__ of module? (Autoloading? (Making Queue.Queue easier to use)) <http://mail.python.org/pipermail/python- dev/2005-October/057253.html>`__ [SJB] ------------------------------------------ exec function specification for Python 3.0 ------------------------------------------ In Python 3.0, exec is slated to become a function (instead of a statement). Currently, the presence of an exec statement in a function can cause some subtle changes since Python has to worry about exec modifying function locals. Guido suggested that the exec () function could require a namespace, basically dumping the exec-in- local-namespace altogether. People seemed generally in favor of the proposal, though no official specification was established. Contributing thread: - `PEP 3000 and exec <http://mail.python.org/pipermail/python-dev/ 2005-October/057135.html>`__ [SJB] ------------------------------------ Adding opcodes to speed up self.attr ------------------------------------ Phillip J. Eby experimented with adding LOAD_SELF and SELF_ATTR opcodes to improve the speed of object-oriented programming. This gained about a 5% improvement in pystone, which isn't organized in a very OO manner. People seemed uncertain as to whether paying the cost of adding two opcodes to gain a 5% speedup was worth it. No decision had been made at the time of this summary. Contributing thread: - `LOAD_SELF and SELF_ATTR opcodes <http://mail.python.org/pipermail/ python-dev/2005-October/057321.html>`__ [SJB] -------------------------------------- Dropping support for --disable-unicode -------------------------------------- Reinhold Birkenfeld tried unsuccessfully to make the test-suite pass with --disable-unicode set. M.-A. Lemburg suggested that the feature should be ripped out entirely, to simplify the code. Martin v. Lˆwis suggested deprecating it to give people a chance to object. The plan is now to add a note to the configure switch that the feature will be removed in Python 2.6. Contributing threads: - `Tests and unicode <http://mail.python.org/pipermail/python-dev/ 2005-October/056897.html>`__ - `--disable-unicode (Tests and unicode) <http://mail.python.org/ pipermail/python-dev/2005-October/056920.html>`__ [SJB] ----------------------------------------- Bug in __getitem__ inheritance at C level ----------------------------------------- Travis Oliphant discovered that the addition of the mp_item and sq_item descriptors and the resolution of any comptetion for __getitem__ calls is done *before* the inheritance of any slots takes place. This means that if you create a type in C that supports the sequence protocol, and tries to inherit the mapping protocol from a parent C type which does not support the sequence protocol, __getitem__ will point to the parent type's __getitem__ instead of the child type's __getitem__. This seemed like more of a bug than a feature, so the behavior may be changed in future Pythons. Contributing thread: - `Why does __getitem__ slot of builtin call sequence methods first? <http://mail.python.org/pipermail/python-dev/2005-October/ 056901.html>`__ [SJB] ================ Deferred Threads ================ - `Early PEP draft (For Python 3000?) <http://mail.python.org/ pipermail/python-dev/2005-October/057251.html>`__ - `Pythonic concurrency - offtopic <http://mail.python.org/pipermail/ python-dev/2005-October/057294.html>`__ =============== Skipped Threads =============== - `PEP 350: Codetags <http://mail.python.org/pipermail/python-dev/ 2005-October/056894.html>`__ - `Active Objects in Python <http://mail.python.org/pipermail/python- dev/2005-October/056896.html>`__ - `IDLE development <http://mail.python.org/pipermail/python-dev/2005- October/056907.html>`__ - `Help needed with MSI permissions <http://mail.python.org/pipermail/ python-dev/2005-October/056908.html>`__ - `C API doc fix <http://mail.python.org/pipermail/python-dev/2005- October/056910.html>`__ - `Static builds on Windows (continued) <http://mail.python.org/ pipermail/python-dev/2005-October/056976.html>`__ - `Removing the block stack (was Re: PEP 343 and __with__) <http:// mail.python.org/pipermail/python-dev/2005-October/057001.html>`__ - `Removing the block stack <http://mail.python.org/pipermail/python- dev/2005-October/057008.html>`__ - `Lexical analysis and NEWLINE tokens <http://mail.python.org/ pipermail/python-dev/2005-October/057014.html>`__ - `PyObject_Init documentation <http://mail.python.org/pipermail/ python-dev/2005-October/057039.html>`__ - `Sourceforge CVS access <http://mail.python.org/pipermail/python- dev/2005-October/057051.html>`__ - `__doc__ behavior in class definitions <http://mail.python.org/ pipermail/python-dev/2005-October/057066.html>`__ - `Sandboxed Threads in Python <http://mail.python.org/pipermail/ python-dev/2005-October/057082.html>`__ - `Weekly Python Patch/Bug Summary <http://mail.python.org/pipermail/ python-dev/2005-October/057092.html>`__ - `test_cmd_line failure on Kubuntu 5.10 with GCC 4.0 <http:// mail.python.org/pipermail/python-dev/2005-October/057094.html>`__ - `defaultproperty (was: Re: RFC: readproperty) <http:// mail.python.org/pipermail/python-dev/2005-October/057120.html>`__ - `async IO and helper threads <http://mail.python.org/pipermail/ python-dev/2005-October/057121.html>`__ - `defaultproperty <http://mail.python.org/pipermail/python-dev/2005- October/057129.html>`__ - `Fwd: defaultproperty <http://mail.python.org/pipermail/python-dev/ 2005-October/057131.html>`__ - `C.E.R. Thoughts <http://mail.python.org/pipermail/python-dev/2005- October/057137.html>`__ - `problem with genexp <http://mail.python.org/pipermail/python-dev/ 2005-October/057175.html>`__ - `Python-Dev Digest, Vol 27, Issue 44 <http://mail.python.org/ pipermail/python-dev/2005-October/057207.html>`__ - `Europeans attention please! <http://mail.python.org/pipermail/ python-dev/2005-October/057233.html>`__

1 0

DRAFT: python-dev Summary for 2005-09-16 to 2005-09-30
by Tony Meyer 16 Nov '05

16 Nov '05

It's been some time (all that concurrency discussion didn't help ;) but here's the second half of September. Many apologies for the delay; hopefully you agree with Guido's 'better late than never', and I promise to try harder in the future. Note that the delay is all my bad, and epithets should be directed at me and not Steve. As usual, please read over if you have a chance, and direct comments/ corrections to tony.meyer(a)gmail.com or steven.bethard(a)gmail.com. (One particular question is whether the concurrency summary is too long). ============= Announcements ============= ----------------------------- QOTF: Quotes of the fortnight ----------------------------- We have two quotes this week, one each from the two biggest threads of this fortnight: concurrency and conditional expressions. The first quote, from Donovan Barda, puts Python's approach to threading into perspective: The reality is threads were invented as a low overhead way of easily implementing concurrent applications... ON A SINGLE PROCESSOR. Taking into account threading's limitations and objectives, Python's GIL is the best way to support threads. When hardware (seriously) moves to multiple processors, other concurrency models will start to shine. Our second QOTF, by yours truly (hey, who could refuse a nomination from Guido?), is a not-so-subtle reminder to leave syntax decisions to Guido: Please no more syntax proposals! ... We need to leave the syntax to Guido. We've already proved that ... we can't as a community agree on a syntax. That's what we have a BDFL for. =) Contributing threads: - `GIL, Python 3, and MP vs. UP <http://mail.python.org/pipermail/ python-dev/2005-September/056609.html>`__ - `Adding a conditional expression in Py3.0 <http://mail.python.org/ pipermail/python-dev/2005-September/056617.html>`__ [SJB] ------------------- Compressed MSI file ------------------- Martin v. Lˆwis discovered that a little more than a `MiB`_ in the Python installer by using LZX:21 instead of the standard MSZIP when compressing the CAB file. After confirmation from several testers that the new format worked, the change (for Python 2.4.2 and beyond) was made. .. _MiB: http://en.wikipedia.org/wiki/Mibibyte Contributing thread: - `Compressing MSI files: 2.4.2 candidate? <http://mail.python.org/ pipermail/python-dev/2005-September/056694.html>`__ [TAM] ========= Summaries ========= ----------------------- Conditional expressions ----------------------- Raymond Hettinger proposed that the ``and`` and ``or`` operators be modified in Python 3.0 to produce only booleans instead of producing objects, motivating this proposal in part by the common (mis-)use of ``<cond> and <true-expr> or <false-expr>`` to emulate a conditional expression. In response, Guido suggested that that the conditional expression discussion of `PEP 308`_ be reopened. This time around, people seemed almost unanimously in support of adding a conditional expression, though as before they disagreed on syntax. Fortunately, this time Guido cut the discussion short and pronounced a new syntax: ``<true-expr> if <cond> else <false-expr>``. Although it has not been implemented yet, the plan is for it to appear in Python 2.5. .. _PEP 308: http://www.python.org/peps/pep-0308.html Contributing threads: - `"and" and "or" operators in Py3.0 <http://mail.python.org/ pipermail/python-dev/2005-September/056510.html>`__ - `Adding a conditional expression in Py3.0 <http://mail.python.org/ pipermail/python-dev/2005-September/056546.html>`__ - `Conditional Expression Resolution <http://mail.python.org/ pipermail/python-dev/2005-September/056846.html>`__ [SJB] --------------------- Concurrency in Python --------------------- Once again, the subject of removing the global interpreter lock (GIL) came up. Sokolov Yura suggested that the GIL be replaced with a system where there are thread-local GILs that cooperate to share writing; Martin v. Lˆwis suggested that he try to implement his ideas, and predicted that he would find that doing so would be a lot of work, would require changes to all extension modules (likely to introduce new bugs, particularly race conditions), and possibly decrease performance. This kicked off several long threads about multi-processor coding. A long time ago (circa Python 1.5), Greg Ward experimented with free threading, which did yield around a 1.6 times speedup on a dual- processor machine. To avoid the overhead of multi-processor locking on a uniprocessor machine, a separate binary could be distributed. Some of the code apparently did make it into Python 1.5, but the issue died off because no-one provided working code, or a strategy for what to do with existing extension modules. Guido pointed out that it is not clear at this time how multiple processors will be used as they become the norm. With the treaded programming model (e.g. in Java) there are problems with concurrent modification errors (without locking) or deadlocks and livelocks (with locking). Guido's hunch (and mine, FWIW) is that instead of writing massively parallel applications, we will continue to write single-threaded applications that are tied together at the process level rather than at the thread level. He also pointed out that it's likely that most problems get little benefit out of multiple processors. Guido threw down the gauntlet: rather than the endless discussion about this topic, someone should come up with a GIL-free Python (not necessarily CPython) and demonstrate its worth. Phillip J. Eby reminded everyone that Jython, IronPython, and PyPy exist, and that someone could, for example, create a multiprocessor-friendly backend for PyPy. Guido also pointed out that fast threading benefits from fast context switches, which benefits from small register sets, and that the current trend in chips is towards larger register sets. In addition, multiple processors with shared memory don't scale all that well (multiple processors with explicit interprocess communication (IPC) channels scale much better). These all favour multi-processing over multi-threading. Donovan Baarda went so far as to say (a QOTF, as above), that Python's GIL is the best way to support threads, which are for single-processor use, and that when multiple-processor platforms have matured more other concurrency models will likewise mature. OTOH, Bob Ippolito pointed out that (in many operating systems) there isn't a lot of difference between threads and processes, and that threads can typically still use IPC. Bob argued that the biggest argument for threading is that lots of existing C/C+ + code uses threads. Simon Percivall argued that the problem is that Python offers ("out of the box") some support for multi-threaded programming, but little for multi-process programming beyond the basics (e.g. data sharing, communication, control over running processes, dealing out tasks to be handled). Simon suggested that the best way to stop people complaining about the GIL is to provide solid, standardized support for multi-process programming. The idea of a "multiprocess" module gained a reasonable amount of support. Phillip J. Eby outlined an idea he is considering PEPifying, in which one could switch all context variables (such as the Decimal context and the sys.* variables) simulaneously and instantaneously when changing execution contexts (like switching between coroutines). He has a prototype implementation of the basic idea, which is less than 200 lines of Python and very fast. However, he pointed out that it's not completely PEP-ready at this point, and he needs to continue considering various parts of the concept. Bruce Eckel joined the thread, and suggested that low-level threads people are only now catching up to objects, but as far as concurrency goes their brains still think in terms of threads, so they naturally apply thread concepts to objects. He believes that pthread-style thinking is two steps backwards: you effectively throw open the innards of the object that you just spent time decoupling from the rest of your system, and the coupling is not unpredictable. Bruce and Guido had discussed offlist "active objects": defining a class as "active" would install a worker thread and concurrent queue in each object of that class, automatically turn method calls into tasks and enqueue them, and prevent any other interaction other than enqueued messages. Guido felt that if multiple active objects could co-exist in the same process, but be prevented (by the language implementation) from sharing data except via channels, and dynamic reallocation of active objects across multiple CPUs were possible, then this might be a solution. He pointed out that an implementation would really be needed to prove this. Phillip and Martin pointed out that preventing any other interacton other than enqueued messages is the difficult part; each active object would, for example, have to have its own sys.modules. Phillip felt that such a solution (which Bruce posed as "a" solution, not "the" solution) wouldn't help with GIL removal, but would help with effective use of multiprocessor machines on platforms where fork() is available, if the API works across processes as well as threads. Bruce then restarted the discussion, putting forth eight criteria that he felt would be necessary for the "pythonic" solution to concurrency. Items on the list were discussed further, with some disagreement about what was possible. The concurrency discussion continues next month... Contributing threads: - `Variant of removing GIL. <http://mail.python.org/pipermail/python- dev/2005-September/056423.html>`__ - `GIL, Python 3, and MP vs. UP (was Re: Variant of removing GIL.) <http://mail.python.org/pipermail/python-dev/2005-September/ 056458.html>`__ - `GIL, Python 3, and MP vs. UP <http://mail.python.org/pipermail/ python-dev/2005-September/056498.html>`__ - `Active Objects in Python <http://mail.python.org/pipermail/python- dev/2005-September/056752.html>`__ - `Pythonic concurrency <http://mail.python.org/pipermail/python-dev/ 2005-September/056801.html>`__ - `Pythonic concurrency - cooperative MT <http://mail.python.org/ pipermail/python-dev/2005-September/056860.html>`__ [TAM] ----------------------------------- Removing nested function parameters ----------------------------------- Brett Cannon proposed removing support for nested function parameters so that instead of being able to write:: def f((x, y)): print x, y you'd have to write something like:: def f(arg): x, y = arg print x, y Brett (with help from Guido) motivated this removal (for Python 3.0) by a few factors: (1) The feature has low visibility: "For every user who is fond of them there are probably ten who have never even heard of it." - Guido (2) The feature can be difficult to read for some people. (3) The feature doesn't add any power to the language; the above functions emit essentially the same byte-code. (4) The feature makes function parameter introspection difficult because tuple unpacking information is not stored in the function object. In general, people were undecided on this proposal. While a number of people said they used the feature and would miss it, many of them also said that their code wouldn't suffer that much if the feature was removed. No decision had been made at the time of the summary. Contributing thread: - `removing nested tuple function parameters <http://mail.python.org/ pipermail/python-dev/2005-September/056459.html>`__ [SJB] ----------------------------------------- Evaluating iterators in a boolean context ----------------------------------------- In Python 2.4 some builtin iterators gained __len__ methods when the number of remaining items could be made available. This broke some of Guido's code that tested iterators for their boolean value (to distinguish them from None). Raymond Hettinger (who supplied the original patch) argued that `testing for None`_ using boolean tests was in general a bad idea, and that knowing the length of an iterator, when possible, had a number of use cases and allowed for some performance gains. However, Guido felt strongly that iterators should not supply __len__ methods, as this would lead to some people writing code expecting this method, which would then break when it received an iterator which could not determine its own length. The feature will be rolled back in Python 2.5, and Raymond will likely move the __len__ methods to private methods in order to maintain the performance gains. .. _testing for None: http://www.python.org/peps/ pep-0290.html#testing-for-none Contributing threads: - `bool(iter([])) changed between 2.3 and 2.4 <http://mail.python.org/ pipermail/python-dev/2005-September/056576.html>`__ - `bool(container) [was bool(iter([])) changed between 2.3 and 2.4] <http://mail.python.org/pipermail/python-dev/2005-September/ 056879.html>`__ [SJB] -------------------------------------------------- Properties that only call the getter function once -------------------------------------------------- Jim Fulton proposed adding a new builtin for a property-like descriptor that would only call the getter method once, so that something like:: class Spam(object): @readproperty def eggs(self): ... expensive computation of eggs self.eggs = result return result would only do the eggs computation once. Currently, you can't do this with a property() because the ``self.eggs = result`` statement tries to call the property's ``fset`` method instead of replacing the property with the result of the eggs() call. A few other people commented that they'd needed similar functionality at times, and Guido seemed moderately interested in the idea, but there was no final resolution. Contributing thread: - `RFC: readproperty <http://mail.python.org/pipermail/python-dev/ 2005-September/056769.html>`__ [SJB] -------- Codetags -------- Micah Elliott submitted his `Codetags PEP 350`_ (after revisions following the comp.lang.python discussion) to python-dev for comment. A common feeling was that this (particularly synonyms) was over-engineering; Guido pointed out that he only uses XXX, and this is certainly the most common (although not only) example in the Python source itself. Some suggestions were made, many of which Micah integrated into the PEP. The suggestion was made that an implementation should precede approval of the PEP. Micah indicated that he would continue development on the tools, and that he encourages anyone interested in using a standard set of codetages to give these a try. .. _Codetags PEP 350: http://python.org/peps/pep-0350.html - `PEP 350: Codetags <http://mail.python.org/pipermail/python-dev/ 2005-September/056744.html>`__ [TAM] ---------------------------- Improving set implementation ---------------------------- Raymond Hettinger suggested a "small, but interesting, C project" to determine whether the setobject.c implementation would be improved by recoding the set_lookkey() function to optimize key insertion order using Brent's variation of Algorithm D (c.f. Knuth vol. III, section 6.4, p525). It has the potential to boost performance for uniquification applications with duplicate keys being identified more quickly, and possibly also more frequent retirement of dummy entires during insertion operations. Andrew Durdin pointed out that Brent's variation depends on the next probe position for a key being derivation from the key and it current position, which is incompatible with the current perturbation system; Raymond replaced perturbation with a secondary hash with linear probing. Antoine Pitrou did some `experimenting with this`_, resulting in a -5% to 2% speedup with various benchmarks. Raymond has also been experimenting with a simpler approach: whenever there are more than three probes, always swap the new key into the first position and then unconditionally re-insert the swapped-out key. He reported that, most of the time, this gives an improvement, and it doesn't require changing the perturbation logic. This simpler approach is cheap to implement, but the benefits are also smaller, with it improving only the worse collisions. .. _experimenting with this: http://pitrou.net/python/sets - `C coding experiment <http://mail.python.org/pipermail/python-dev/ 2005-September/055965.html>`__ [TAM] -------------- Relative paths -------------- Nathan Bullock suggested a ''relpath(path_a, path_b)'' addition to os.path that returns a relative path from path_a to path_b. Trent Mick pointed out that there are a `couple of`_ `recipes for this`_, as well as `Jason Orendorff's Path module`_. Several people supported this idea, and hopefully either Nathan or one of the recipe authors will submit a patch with this functionality. .. _couple of: http://aspn.activestate.com/ASPN/Cookbook/Python/ Recipe/302594 .. _recipes for this: http://aspn.activestate.com/ASPN/Cookbook/ Python/Recipe/208993 .. _Jason Orendorff's Path module: http://www.jorendorff.com/articles/ python/path/ Contributing threads: - `os.path.diff(path1, path2) <http://mail.python.org/pipermail/ python-dev/2005-September/056391.html>`__ - `os.path.diff(path1, path2) (and a first post) <http:// mail.python.org/pipermail/python-dev/2005-September/056703.html>`__ [TAM] ---------------------------------- Adding a vendor-packages directory ---------------------------------- Rich Burridge followed up a `comp.lang.python thread`_ about a "vendor-packages" directory for Python by submitting a `patch`_ and asking for comments about the proposal on python-dev. General consensus was that the proposal needed a better rationale, explaining why this improved on simply adding a .pth file to the site-packages directory. Rich explained that the rationale is that Python files supplied by the vendor (Sun, Apple, RedHat, Microsoft) with their operating system software should go in a separate base directory to differentiate them from Python files installed specifically at the site. However, Bob Ippolito pointed out that, as of OS X 10.4 ("Tiger") Apple already does this via a .pth file ("Extras.pth"), which points to ''/System/Library/Frameworks/Python.framework/ Versions/2.3/Extras/lib/python'' and includes wxPython by default. Bob also pointed out that such a "vendor-packages.pth" should look like ''import site; site.addsitedir('/usr/lib/python2.4/vendor- packages')'' so that packages like Numeric, PIL, and PyObjC, which take advantage of .pth files themselves, work when installed to the vendor-packages location. Phillip J. Eby pointed out that it would be good to have a document for "Python Distributors" that explained these kind of things, and suggested that perhaps a volunteer or two could be found within the distutils-SIG to do this. .. _comp.lang.python thread: http://mail.python.org/pipermail/python- list/2005-September/300029.html .. _patch: http://sourceforge.net/tracker/index.php? func=detail&aid=1298835&group_id=5470&atid=305470 Contributing thread: - `vendor-packages directory <http://mail.python.org/pipermail/python- dev/2005-September/056682.html>`__ [TAM] ======================= Version numbers on OS X ======================= Guido asked if platform.system_alias() could be improved on OS X by mapping uname()'s ''Darwin x.y'' to ''OS X 10.(x-4).y''. Bob Ippolito and others pointed out that this was not a good idea, because uname() only reports on the kernel version number and not the Cocoa API, which is really what OS X 10.x.y refers to. He pointed out that the correct way to do it using a public API is to used gestalt, which is what platform.mac_ver() does. On further inspection, it was discovered that parsing the /System/ Library/CoreServices/SystemVersion.plist property list is also a supported API, and would not rely on access to the Carbon API set. Bob and Wilfredo S·nchez Vega provided sample code that would parse this plist; Marc-Andre Lemburg suggested that a patch be written for system_alias() that would use this method (if possible) for Mac OS. Contributing thread: - `Mapping Darwin 8.2.0 to Mac OS X 10.4.2 in platform.py <http:// mail.python.org/pipermail/python-dev/2005-September/056651.html>`__ [TAM] ================ Deferred Threads ================ - `Python 2.5a1, ast-branch and PEP 342 and 343 <http:// mail.python.org/pipermail/python-dev/2005-September/056449.html>`__ =============== Skipped Threads =============== - `Visibility scope for "for/while/if" statements <http:// mail.python.org/pipermail/python-dev/2005-September/056669.html>`__ - `inplace operators and __setitem__ <http://mail.python.org/ pipermail/python-dev/2005-September/056766.html>`__ - `Repository for python developers <http://mail.python.org/pipermail/ python-dev/2005-September/056717.html>`__ - `For/while/if statements/comprehension/generator expressions unification <http://mail.python.org/pipermail/python-dev/2005- September/056508.html>`__ - `list splicing <http://mail.python.org/pipermail/python-dev/2005- September/056472.html>`__ - `Compatibility between Python 2.3.x and Python 2.4.x <http:// mail.python.org/pipermail/python-dev/2005-September/056437.html>`__ - `python optimization <http://mail.python.org/pipermail/python-dev/ 2005-September/056441.html>`__ - `test__locale on Mac OS X <http://mail.python.org/pipermail/python- dev/2005-September/056463.html>`__ - `possible memory leak on windows (valgrind report) <http:// mail.python.org/pipermail/python-dev/2005-September/056478.html>`__ - `Mixins. <http://mail.python.org/pipermail/python-dev/2005- September/056481.html>`__ - `2.4.2c1 fails test_unicode on HP-UX ia64 <http://mail.python.org/ pipermail/python-dev/2005-September/056551.html>`__ - `2.4.2c1: test_macfs failing on Tiger (Mac OS X 10.4.2) <http:// mail.python.org/pipermail/python-dev/2005-September/056558.html>`__ - `test_ossaudiodev hangs <http://mail.python.org/pipermail/python- dev/2005-September/056559.html>`__ - `unintentional and unsafe use of realpath() <http://mail.python.org/ pipermail/python-dev/2005-September/056616.html>`__ - `Alternative name for str.partition() <http://mail.python.org/ pipermail/python-dev/2005-September/056630.html>`__ - `Weekly Python Patch/Bug Summary <http://mail.python.org/pipermail/ python-dev/2005-September/056713.html>`__ - `Possible bug in urllib.urljoin <http://mail.python.org/pipermail/ python-dev/2005-September/056736.html>`__ - `Trasvesal thought on syntax features <http://mail.python.org/ pipermail/python-dev/2005-September/056741.html>`__ - `Fixing pty.spawn() <http://mail.python.org/pipermail/python-dev/ 2005-September/056750.html>`__ - `64-bit bytecode compatibility (was Re: [PEAK] ez_setup on 64-bit linux problem) <http://mail.python.org/pipermail/python-dev/2005- September/056811.html>`__ - `C API doc fix <http://mail.python.org/pipermail/python-dev/2005- September/056827.html>`__ - `David Mertz on CA state e-voting panel <http://mail.python.org/ pipermail/python-dev/2005-September/056840.html>`__ - `[PATCH][BUG] Segmentation Fault in xml.dom.minidom.parse <http:// mail.python.org/pipermail/python-dev/2005-September/056844.html>`__ - `linecache problem <http://mail.python.org/pipermail/python-dev/ 2005-September/056856.html>`__

1 0

Is some magic required to check out new files from svn?
by skip＠pobox.com 16 Nov '05

16 Nov '05

Is there some magic required to check out new files from the repository? I'm trying to build on the trunk and am getting compilation errors about code.h not being found. If I remember correctly, this is a new file brought over from the ast branch. Using cvs I would have executed something like "cvs up -dPA ." if I found I was missing something (usually a new directory) and wanted to make sure I was in sync with the trunk. I read the developer's FAQ and the output of "svn up --help". Executing "svn up" or "svn info" tells me I'm already at rev 41430, which is the latest rev, right? Creating a fresh build subdirectory followed by configure and make gives me this error: ../Objects/frameobject.c:6:18: code.h: No such file or directory Sure enough, I have no code.h in my Include directory. Before I wipe out Include and svn up again is there any debugging I can do for someone smarter in the ways of Subversion than me? Regarding my checksum problems (which are not appearing at the moment), Martin asked for 1. what specific revision you had checked out (svn info) 2. what the recorded checksum is (see .svn/entries) 3. what the commited-rev is 4. what the actual checksum is on the file on disk (.svn/text-base/filename.base) 5. whether or not the checksums svn reports match the ones you determined yourself. I don't think #2, #4 or #5 apply here. According to .svn/entries I have: <entry committed-rev="41430" name="" committed-date="2005-11-12T15:55:04.419664Z" url="svn+ssh://pythondev@svn.python.org/python/trunk" last-author="fredrik.lundh" kind="dir" uuid="6015fed2-1504-0410-9fe1-9d1591cc4771" prop-time="2005-11-12T18:00:07.000000Z" revision="41430"/> Here's "svn info" output: Path: . URL: svn+ssh://pythondev@svn.python.org/python/trunk Repository UUID: 6015fed2-1504-0410-9fe1-9d1591cc4771 Revision: 41430 Node Kind: directory Schedule: normal Last Changed Author: fredrik.lundh Last Changed Rev: 41430 Last Changed Date: 2005-11-12 09:55:04 -0600 (Sat, 12 Nov 2005) Properties Last Updated: 2005-11-12 12:00:07 -0600 (Sat, 12 Nov 2005) I was running 1.2.0. I just downloaded and built 1.2.3. It made no difference. This is getting kinda frustrating. I haven't got a lot of confidence in Subversion at this point. Skip

6 9

Conclusion: Event loops, PyOS_InputHook, and Tkinter
by Jim Jewett 16 Nov '05

16 Nov '05

Phillip J. Eby: > did you ever try using IPython, and confirm whether it > does or does not address the issue As I understand it, using IPython (or otherwise changing the interactive mode) works fine *if* you just want a point solution -- get something up in some environment chosen by the developer. Michiel is looking to create a component that will work in whatever environment the *user* chooses. Telling users "you must go through this particular interface" is not acceptable. Therefore, IPython is only a workaround, not a solution. On the other hand, IPython is clearly a *good* workaround. The dance described in http://mail.python.org/pipermail/python-dev/2005-November/058057.html is short enough that a real solution might well be built on IPython; it just isn't quite done yet. -jJ

1 0

Behavoir question.
by Scott David Daniels 16 Nov '05

16 Nov '05

Since I am fiddling with int/long conversions to/from string: Is the current behavior intentional (or mandatory?): v = int(' 55555555555555555555555555555555555555555 ') works, but: v = int(' 55555555555555555555555555555555555555555L ') fails. --Scott David Daniels Scott.Daniels(a)Acm.Org

2 1

Event loops, PyOS_InputHook, and Tkinter
by Michiel Jan Laurens de Hoon 15 Nov '05

15 Nov '05

Dear Pythoneers, I use Python heavily for scientific computing, originally for computational physics and nowadays for computational biology. One of the extension modules I support is pygist, a module for scientific visualization. For this (and similar) packages to work, it is important to have an event loop in Python. Currently, event loops are available in Python via PyOS_InputHook, a pointer to a user-defined function that is called when Python is idle (waiting for user input). However, an event loop using PyOS_InputHook has some inherent limitations, so I am thinking about how to improve event loop support in Python. As an example, consider the current implementation of Tkinter. What's nice about it is that events as well as user-typed Python commands are handled without having to call mainloop() explicitly (except on some platforms): "import Tkinter; Tkinter.Tk()" causes a Tk window to pop up that remains responsive throughout. It works as follows (using Tkinter as an example; pygist works essentially the same): 1) Importing Tkinter causes PyOS_InputHook to be set to the EventHook function in _tkinter.c. 2) Before Python calls fgets to read the next Python command typed by the user, it checks PyOS_InputHook and calls it if it is not NULL. 3) The EventHook function in _tkinter runs the following loop: - Check if user input is present; if so, exit the loop - Handle a Tcl/Tk event, if present - Sleep for 20 milliseconds 4) Once the EventHook function returns, Python continues to read the next user command. After executing the command, return to 2). However, this implementation has the following problems: 1) Essentially, the event loop is a busy-wait loop with a 20 ms sleep in between. An event loop using select() (or equivalent on Windows) will give better performance. 2) Since this event loop runs inside Tkinter, there is no way for other extension modules to get their messages handled. Hence, we cannot have more than one extension module that needs an event loop. As an example, it would be nice to have a Tkinter GUI to steer a simulation and a (non-Tk) graphics output window to visualize the simulation. 3) Whereas PyOS_InputHook is called when Python is waiting for user input, it is not called when Python is waiting for anything else, for example one thread waiting for another. For example, IDLE uses two threads, one handling the GUI and one handling the user commands. When the second thread is waiting for the first thread (when waiting for user input to become available), PyOS_InputHook is not being called, and no Tkinter events are being handled. Hence, "import Tkinter; Tkinter.Tk()" does nothing when executed from an IDLE window. Which means that our scientific visualization software can only be run from Python started from the command line, whereas many users (especially on Windows) will want to use IDLE. Now the problem I'm facing is that because of its integration with Tcl, this cannot be fixed easily with Tkinter as the GUI toolkit for Python. If the events to be handled were purely graphical events (move a window, repaint a window, etc.), there would be no harm in handling these events when waiting for e.g. another thread. With Tkinter, however, we cannot enter EventHook while waiting for another thread: a) because EventHook only returns if user input is available (it doesn't wait for threads); b) because EventHook also runs Tcl/Tk commands, and we wouldn't want to run some Tcl commands in some thread while waiting for another thread. Therefore, as far as I can tell, there is no way to set up a true event loop in Python that works nicely with Tkinter, and there is no way to have an event loop function from IDLE. So I'd like to ask the following questions: 1) Did I miss something? Is there some way to get an event loop with Tkinter? 2) Will Tkinter always be the standard toolkit for Python, or are there plans to replace it at some point? I realize that Tkinter has been an important part of Python for some time now, and I don't expect it to be ditched just because of my event loop problems. At the same time, event loop support could use some improvement, so I'd like to call your attention to this issue. Tcl actually has event loops implemented quite nicely, and may serve as an example of how event loops may work in Python. --Michiel. -- Michiel de Hoon Center for Computational Biology and Bioinformatics Columbia University 1150 St Nicholas Avenue New York, NY 10032

16 60

Gothenburg PyPy Sprint II: 7th - 11th December 2005
by Michael Hudson 15 Nov '05

15 Nov '05

Gothenburg PyPy Sprint II: 7th - 11th December 2005 ====================================================== (NOTE: internal EU-only sprint starts on the 5th!) The next PyPy sprint is scheduled to be in December 2005 in Gothenborg, Sweden. Its main focus is heading towards phase 2, which means JIT work, alternate threading models and logic programming (but there are also other possible topics). We'll give newcomer-friendly introductions. To learn more about the new PyPy Python-in-Python implementation look here: http://codespeak.net/pypy Goals and topics of the sprint ------------------------------ We have released pypy-0.8.0_, which is officially a "research base" for future work. The goal of the Gothenburg sprint is to start exploring new directions and continue in the directions started at the Paris sprint. The currently scheduled main topics are: - The L3 interpreter, a small fast interpreter for "assembler-level" flow graphs. This is heading towards JIT work. - Stackless: write an app-level interface, which might be either Tasklets, as in "Stackless CPython", or the more limited Greenlets. - Porting C modules from CPython. (_socket is not finished) - Optimization/debugging work in general. In particular our thread support is far from stable at the moment and unaccountably slow. - Experimentation: logic programming in Python. A first step might be to try to add logic variables to PyPy. .. _`pypy-0.8.0`: http://codespeak.net/pypy/dist/pypy/doc/release-0.8.0.html Location & Accomodation ------------------------ The sprint will be held in the apartment of Laura Creighton and Jacob Halen which is in Gotabergsgatan 22. The location is central in Gothenburg. It is between the tram_ stops of Vasaplatsen and Valand, where many lines call. .. _tram: http://www.vasttrafik.se Probably cheapest and not too far away is to book accomodation at `SGS Veckobostader`_. (You can have a 10% discount there; ask in the pypy-sprint mailing list for details. We also have some possibilites of free accomodation.) .. _`SGS Veckobostader`: http://www.sgsveckobostader.com Exact times ----------- The public PyPy sprint is held Wednesday 7th - Sunday 11th December 2005. There is a sprint for people involved with the EU part of the project on the two days before the "official" sprint. Hours will be from 10:00 until people have had enough. It's a good idea to arrive a day before the sprint starts and leave a day later. In the middle of the sprint there usually is a break day and it's usually ok to take half-days off if you feel like it. Network, Food, currency ------------------------ Sweden is not part of the Euro zone. One SEK (krona in singular, kronor in plural) is roughly 1/10th of a Euro (9.15 SEK to 1 Euro). The venue is central in Gothenburg. There is a large selection of places to get food around, from edible-and-cheap to outstanding. You normally need a wireless network card to access the network, but we can provide a wireless/ethernet bridge. Sweden uses the same kind of plugs as Germany. 230V AC. Registration etc.pp. -------------------- Please subscribe to the `PyPy sprint mailing list`_, introduce yourself and post a note that you want to come. Feel free to ask any questions there! There also is a separate `Gothenburg people`_ page tracking who is already thought to come. If you have commit rights on codespeak then you can modify yourself a checkout of http://codespeak.net/svn/pypy/extradoc/sprintinfo/gothenburg-2005/people.txt .. _`PyPy sprint mailing list`: http://codespeak.net/mailman/listinfo/pypy-sprint .. _`Gothenburg people`: http://codespeak.net/pypy/extradoc/sprintinfo/gothenburg-2005/people.html Cheers, mwh -- Speaking from personal experience, I can attest that the barrel of any firearm (or black powder weapon) pointed at one appears large enough to walk down, hands at full extension above the head, without touching the top. -- Mike Andrews, asr

1 0

Re: [Python-Dev] Inconsistent behaviour in import/zipimport hooks
by Phillip J. Eby 14 Nov '05

14 Nov '05

At 04:33 PM 11/9/2005 -0800, Guido van Rossum wrote: >On 11/9/05, Phillip J. Eby <pje(a)telecommunity.com> wrote: > > By the way, while we're on this subject, can we make the optimization > > options be part of the compile() interface? Right now the distutils has to > > actually exec another Python process whenever you want to compile > > code with > > a different optimization level than what's currently in effect, whereas if > > it could pass the desired level to compile(), this wouldn't be necessary. > >Makes sense to me; we need a patch of course. But before we can do that, it's not clear to me if it should be part of the existing "flags" argument, or whether it should be separate. Similarly, whether it's just going to be a level or an optimization bitmask in its own right might be relevant too. For the current use case, obviously, a level argument suffices, with 'None' meaning "whatever the command-line level was" for backward compatibility. And I guess we could go with that for now easily enough, I'd just like to know whether any of the AST or optimization mavens had anything they were planning in the immediate future that might affect how the API addition should be structured.

5 8

Re: [Python-Dev] Implementation of PEP 341
by Nick Coghlan 14 Nov '05

14 Nov '05

Thomas Lee wrote: > Implemented as you suggested and tested. I'll submit the patch to the > tracker on sourceforge shortly. Are you guys still after contextual > diffs as per the developer pages, or is an svn diff the preferred way to > submit patches now? svn diff should be fine. Although I thought Brett had actually updated those pages after the move to svn. . . > Thanks very much for all your help, Nick. It was extremely informative. I think we can chalk up a respectable win for the AST-based compiler - the trick I suggested wouldn't really have been practical without the AST layer between the parser and the compiler. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org

2 2

Re: [Python-Dev] Event loops, PyOS_InputHook, and Tkinter
by Edward Loper 13 Nov '05

13 Nov '05

As I understand it, you want to improve the performance of interactively run plot commands by queuing up all the plot sub-commands, and then drawing them all at once. Hooking into a python event loop certainly isn't the only way to do this. Perhaps you could consider the following approach: - The plot event loop is in a separate thread, accepting messages from the interactive thread. - These messages can contain plot commands; and they can also contain two new commands: - suspend -- stop plotting, and start saving commands in a queue. - resume -- execute all commands in the queue (with whatever increased efficiency tricks you're using) Then you can either just add functions to generate these messages, and call them at appropriate places; or set PyOS_InputHook to wrap each interactive call with a suspend/resume pair. But note that putting an event loop in a separate thread will be problematic if you want any of the events to generate callbacks into user code -- this could cause all sorts of nasty race-conditions! Using a separate thread for an event loop only seems practical to me if the event loop will never call back into user code (or if you're willing to put the burden on your users of making sure everything is thread safe). -Edward

1 0