Mailman 3 July 2010 - Python-Dev

Proposal: make float.__str__ identical to float__repr__ in Python 3.2
by Mark Dickinson July 30, 2010

July 30, 2010

Now that we've got the short float repr in Python, there's less value in having float.__str__ truncate to 12 significant digits (as it currently does). For Python 3.2, I propose making float.__str__ use the same algorithm as float.__repr__ for its output (and similarly for complex). Apart from simplifying the internals a little bit, one nice feature of this change is that it removes the differences in formatting between printing a float and printing a container of floats: >>> l = [1/3, 1/5, 1/7] >>> print(l) [0.3333333333333333, 0.2, 0.14285714285714285] >>> print(l[0], l[1], l[2]) 0.333333333333 0.2 0.142857142857 Any thoughts or comments on this? There's a working patch at http://bugs.python.org/issue9337 Mark

9 12

Full unicode support for the import machinery
by Victor Stinner July 30, 2010

July 30, 2010

Hi, I'm trying to fix Python to support undecodable bytes in the Python path since some months ago. My first try was really huge and sometimes ugly. When it was possible, I extracted some short and simple patches and applied them to py3k (sometimes with an issue, sometimes directly in the svn). When it was no more possible to split the big patch, I restarted the work from scratch. The main change from my previous try is that I changed import.c to use unicode strings instead of byte strings. With the surrogate hack (PEP 383), unicode is a superset of bytes and so it is "forward compatible". I just created a branch called "import_unicode" (based on py3k) including all my patches. It's still a work in progress. It is possible to start Python installed in an undecodable path (eg. directory with an non-ASCII character with C locale for Linux), which is an huge progress, but some tests are still failing. The last biggest problem is that code object filenames are not reencoded after that the file system encoding is changed (but sys.path and sys.modules filenames are reencoded). I think that I will register all code objects into a list to be able to reencode their filename attribute (and then drop the list). I created an svn branch because I think that it's easier to review short commits than one unique huge patch. The branch also helps me to share the branch between different computers, and allow other people to review the commits (and/or contribute!). Some people will maybe understand better my work with the "whole picture" :-) -- There are at least 4 issues related to this work: #3080: Full unicode import system #4352: imp.find_module() fails with a UnicodeDecodeError when called with non-ASCII search paths #8611: Python3 doesn't support locale different than utf8 and an non-ASCII path (POSIX) #8988: import + coding = failure (3.1.2/win32) -- Some examples of previous issues related to my secret goal (patch import machinery): #8391: os.execvpe() doesn't support surrogates in env #8393: subprocess: support undecodable current working directory on POSIX OS #8412: os.system() doesn't support surrogates nor bytes #8485: Don't accept bytearray as filenames, or simplify the API # 8514: Add fsencode() functions to os module #8610: Python3/POSIX: errors if file system encoding is None (-> create initfsencoding() in pythonrun.c) #8715: Create PyUnicode_EncodeFSDefault() function ... -- Victor Stinner http://www.haypocalc.com/

2 2

[isssue 2001] Pydoc enhancement patch questions
by Ron Adam July 29, 2010

July 29, 2010

This regards the following feature request. http://bugs.python.org/issue2001 Summery: The pydoc "gui" enhancement patch adds a navigation bar on each page with features that correspond to abilities that are currently available to help. ie.. a get field, to get help on an item, a find field, to search for modules by keyword, plus modules, topics, and keyword index links. The file links read the python file in as text and inserts it into a web page instead of relying on the browser to do the right thing. (often enough it doesn't) To achieve this I reworked the existing pydoc server into one that can respond to a navigation bar at the top of the served pages. The new "local_text_server" will exist in the http package where the other server related modules are. The callback function passed to the server does all the pydoc specific work so the server itself is a simple general purpose text server useful for making browser interface front ends. This also removed the need for tkinter in pydoc and allows pydoc -g to work on computers where tk isn't installed. As per discussion on the tracker: > Alexander Belopolsky <belopolsky(a)users.sourceforge.net> added the comment: > > +:program:`pydoc` :option:`-g` will start the server and additionally open a web > +browser to a module index page. Each served page has a navagation bar at the > +top where you can 'get' help on a individual item, 'find' all modules with a > +keyword in thier synopsis line, and goto indexes for 'modules', 'topics' and > +'keywords'. > > I am not sure I like the fact that the browser is started automatically. > Please bring this up on python-dev. This may be an opportunity to > rethink pydoc command line switches. For example, -p and -g are > currently exclusive, but it would make sense for -g to start server on > the port specified by -p. So are any thoughts on starting the web browser automatically, and on how the -g and -p command line switches work? I'm also not sure about the name for the server. I've use local_text_server, but it may also be useful in non-local cases as well. The newest patch is... http://bugs.python.org/file18165/pydoc_server3.diff Any feedback will be welcome. Ron

4 16

Best practice for new namespace (from C/API)
by Campbell Barton July 29, 2010

July 29, 2010

Hi, I'm writing because I'm working on a project where the user can run scripts that don't reference a file but use internal application text data. Otherwise we are not doing anything tricky, only that the scripts should each run independently (no cruft left from the previous scripts namespace, sharing sys.modules etc is fine). Something which is unclear to me even after looking over pythonrun.c is the correct way to setup a namespace. For years we have been doing this and it seemed to work fine... PyObject *d = PyDict_New(); // new namespace PyDict_SetItemString(d, "__builtins__", PyEval_GetBuiltins()); PyDict_SetItemString(d, "__file__", PyString_FromString(filename)); // fake, avoids sys.argv[0] being used for warnings. /* --- snip ---*/ PyEval_EvalCode(compiled_text, d, d); Recently a developer reported a bug where pickle wasn't working, it turns out that in a few places python expects the __main__ modules namespace to match that if the running script: _pickle.c's save_global() in this case eg: >>> spam = 10 >>> print(__import__("__main__").__dict__["spam"]) ... 10 Once I found this was the problem it was simple to use __main__'s namespace however there are still things that are not clear about exactly how this should be done. Simplified code... PyObject *item, *dict= PyModule_GetDict(PyImport_AddModule("__main__")); PyDict_Clear(dict); PyDict_SetItemString(dict, "__builtins__", PyImport_AddModule("builtins")); item = PyUnicode_FromString( "__main__" ); PyDict_SetItemString( dict, "__name__", item ); Py_DECREF(item); PyDict_SetItemString(d, "__file__", PyString_FromString(filename)); // fake, avoids sys.argv[0] being used for warnings. /* --- snip ---*/ PyEval_EvalCode(compiled_text, dict, dict); Still this leaves me with the following questions... - Whats the best way to manage the namespace for running multiple scripts one after another? clear and initialize __main__'s dict each time?, keep a copy and restore it after each run? - should the original __main__ namespace be restored after running a script? - pickle expects: __import__("__main__").__dict__ == ***the current namespace***, is this apart of the python spec or just something specific to pickle? - PyDict_SetItemString(d, "__builtins__", PyEval_GetBuiltins()) acts differently to PyDict_SetItemString(dict, "__builtins__", PyImport_AddModule("builtins")), using the PyEval_GetBuiltins() dict adds every member of __builtins__ when running: print(__import__("__main__").__dict__.keys()), rather then just showing __builtins__. -- - Campbell

3 2

GIL musings (was Re: Thoughts fresh after EuroPython)
by Nick Coghlan July 29, 2010

July 29, 2010

On Tue, Jul 27, 2010 at 12:33 AM, Ronald Oussoren <ronaldoussoren(a)mac.com> wrote: > In my opinion the GIL is a weak point of CPython and it would be nice if it > could be fixed. That is however easier said than done, a number of people > have tried in the past and ran into implementation limitations like our > refcounting garbage collector that make hard to remove the GIL without > either rewriting lots of code, or running into a brick wall > performance-wise. > > The HotPy presentation at EuroPython shows that it is possible to remove the > GIL, although at the cost of replacing the garbage collector and most likely > breaking existing C extensions (although the HotPy author seemed to have a > possible workaround for that). This is the kind of approach that seems to hold the most promise of removing the GIL without incurring the single-threaded performance hit that has been the achilles heel of previous attempts at creating a free-threaded CPython implementation. With first IronClad and now PyPy blazing the trail in interfacing a garbage collected Python implementation with deterministic refcounting for C extension modules, it seems plausible that this kind of approach may eventually prove acceptable. Furthermore, the with statement now provides a superior alternative to application level tricks that previously relied on deterministic refcounting. While multi-threading does break down beyond a certain number of cores, it *is* possible to do safely (particularly using queues to pass references around) and can avoid plenty of serialisation overhead when dealing with sizable data structures. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

6 6

Python Language Summit EuroPython 2010
by Tim Golden July 29, 2010

July 29, 2010

Before the main events of EuroPython 2010 a Python Language Summit took place at the Conference venue in Birmingham. Present were (in the order they sat around the table): * Brett Cannon * Guido van Rossum * Holger Krekel * Amaury Forgeot D'Arc * Georg Brandl * Péter Szabó * Ezio Melotti * Michael Foord * Mark Dickinson * Martin von Loewis * Ronald Oussoren * Tim Golden * Marc Andre Lemburg * Richard Jones Michael initiated a round-up of current and prospective Python versions for various implementations. CPython and IronPython have both just released v2.7 with IronPython offering some Py3 compatibility via a command-line switch. The recent/current migration of Numpy and SciPy to Py3 should give a boost to uptake. Amaury confirmed that PyPy currently supports 2.5.2 but is looking to target 2.7. The PyPy guys also announced a C API bridging layer which should enable a range of Python extension modules to work directly with PyPy. This is only a stepping stone by way of broadening support. Brett suggested that the Unladen Swallow merge to trunk was waiting for some work to complete on the JIT compiler and Georg, as release manager for 3.2, confirmed that Unladen Swallow would not be merged before 3.3. The email module needs some work in Py3. David Murray has been given some money by the PSF but needs more from other sources to complete the work. This is hampered by the legalities around commercial organisations making donations to not-for-profits when those donations are earmarked. Various suggestions were put forward with no-one sure of the legal issues. Guido suggested that we should move forward rather than stall for want of legal advice. A broad discussion arose concerning the issues debated on web-sig concerning the WSGI protocol and the bytes vs string issues. Marc Andre brought up the cgi module which has similar issues under Py3 and other examples were given, including ftplib, urllib and some os functions. Various solutions were put forward including a hybrid bytes-with-encoding object. This proposal was widely unpopular, but two proposals met with broad approval: that certain stdlib functions might be polymorphic, returning the type of their input as output; and that the encoding string should include its error-handling. An example of the first would be that os.getenv ("HOME") would return "/home/tjg" while os.getenv (b"HOME") would return b"/home/tjg". An example of the latter would be "utf8:strict". Something of the sort already works for PYTHONIOENCODING. The issue of a __format__ equivalent for bytes was also raised as was the idea of object methods to render an object as string or bytes, which could be used in the polymorphic functions above. Martin spoke about the state of the stable ABI PEP, indicating that he was targetting 3.2. This work would reduce the need to recompile extension modules separately on Windows for every version of Python -- something especially pertinent when code has been orphaned but is still useful. The versioned .so files PEP being worked out by Barry Warsaw overlaps with this PEP and would only be useful for extensions which don't target the stable ABI. A messy discussion turned on the question of garbage collection of module objects, and the order in which finalisers are called if at all, especially when reference cycles exist. Marc Andre was proposing a __cleanup__ magic function for Python modules, which would enable the implementer to define the order in which resources are released / closed down. This is quite a subtle area and raised the issue of unfinalised objects in a reference cycle whose memory has been freed out from under them but which still exist. Martin described the Java approach where finalisers are called once and then flagged so they are not called again even if their object is resurrected. This sounded like a useful approach for Python but would break code which expected to be able to resurrect an object during its __del__ method... which is not expected to account for much code. Guido pointed out that no-one can be expected to hold enough of the complexities of this area of Python's implementation in their head, and that an implementation of some sort would need to be written so that the corner-cases could emerge. Ronald described the issues around the version and architecture differences on Mac OS X and especially around Tkinter (and therefore IDLE). It was agreed that two installers could be provided: one targetting OS 10.3 on 32-bit Intel/PPC; the other targetting 10.6 on 32 and 64-bit Intel. This latter would then be able to use the system's Tk 8.5. The 10.6 binary would also work for 10.5, which would be indicated in the install docs. The Mercurial migration should move forward once Dirkjan has finished work on his thesis. Martin insisted that a for-real repository would have to be set up so that people can really see how it would work. An outstanding issue in hg-svn prevents the Python history from being imported, but it should be fixable. Martin & Tim brought up the issue of externals which the buildbots use on Windows to bring in and build slightly patched versions of external libraries such as OpenSSL and sqlite3. Brett confirmed that he would like to see the stdlib broken out into its own repository which could then be shared between the different Python implementations. A discussion on the Cheeseshop / Package Index highlighted the fact that the packaging infrastructure has become increasingly important especially since setuptools, buildout and pip all download from it. Richard produced graphs showing the increase in package downloads over time, and attributed the recent slight tail-off to the fact that the toolchains are now becoming more canny with respect to cacheing and mirroring. Martin & Richard confirmed that mirrors are now in place and Marc Andre confirmed that he would be putting together a proposal to have PyPI hosted in the cloud. Guido pointed out that if an AppEngine implementation were desirable, he was sure that AppEngine team would support it with resources as needed. Martin didn't feel that there was a problem with loading on the box in question; it's the uptime that's behind people's concern as it's now so essential to installing and deploying Python applications. Several people outlined the recent heated discussion over the addition of a checkbox to the PyPI user-registration pages. Tarek has already undertaken to patch PyPI to move the checkbox back one step, allowing existing distutils users to register from the command line. At the same time, Brett advised removing that functionality from distutils2 as signing up on a web page is no great hardship.

19 32

Re: [Python-Dev] audience-instructors for Teach Me Python Bugfixing needed
by Catherine Devlin July 29, 2010

July 29, 2010

The PyOhio contribu-palooza starts this Saturday! http://www.pyohio.org/Contribute With two talks and a two-day-four-night sprint, I'm very hopeful that it will recruit and train some new core workers. I'm preparing my portion, the teach-the-newbie (me) -to-fix-a-core-bug session, and I want to make sure that I'm prepared in two ways: 1. Any bulky download/compilation steps are complete I pulled the Py3 trunk with svn co http://svn.python.org/projects/python/branches/py3k python, did the compilation steps, and verified that I can fire up the latest build. I also note that http://www.python.org/dev/ doesn't say anything about hg yet. Is there someplace else I should look for hg-centered docs? Should we just teach it using svn if that's better documented? Then again, if hg is the way of the future... I also built the docs (``cd Doc; make html``) Are there other things that I need to do to configure my machine beforehand? Things that are too long/boring for the audience to sit through while I do it live? 2. Have a good set of questions to ask. Here's what I'm planning so far: - While running ``make test`` on the Python trunk, I got an error on "test_os". Is that a problem with my machine's configuration, or with the build? Do I need to report it? Can I ignore it? Now we'll find a bug. - Do I need an account on bugs.python.org? What do I need to do to get one? - How do I find a bug suitable for me to work on? - entry-level - in Python not C - corresponding to my strong points / interests Now we'll "find" a fake bug that David has planted for us. (David, have you planted it yet?) - Can/should I make my edits directly in the trunk that I just pulled down? - Now we'll make the fix... maybe this should involve using a debugging IDE or pdb? - How do I verify that my fix worked? That it didn't break anything else? That it's written with proper style? That it doesn't generally suck? - How do I send my fix back up to the trunk? - How do I record my work in the bug tracker? (If time permits) now let's try writing a test for a gap in test coverage (not necessarily on the code we just worked on - this doesn't have to be fake) DON'T ANSWER THESE! I need to carefully guard my sincere ignorance through Saturday! (Actually, I already have a pretty good idea about some of them, but I don't want my ignorance to become any less sincere than it already is.) But, if you're David or Dan or anybody else who's going to be there, you may want to ponder how you'll guide me through it. But what I want to know from all of you is: what other questions should be on my list? I was going to address this only to David, my primary audience/instructor volunteer, but I thought it wouldn't hurt to get input from the rest of you. Thank you all! -- - Catherine http://catherinedevlin.blogspot.com/ *** PyOhio 2010 * July 31 - Aug 1 * Columbus, OH * pyohio.org ***

1 0

mkdir -p in python
by Peng Yu July 28, 2010

July 28, 2010

I don't see that there is a function in the library that mimic the behavior of 'mkdir -p'. If 'makedirs' is used, it will generate an error if the file already exists. There are some functions available on the website to close the gap. But I'd prefer this is in the library. Is there any plan to add a function that mimic the behavior of 'mkdir -p'? http://code.activestate.com/recipes/82465-a-friendly-mkdir/ -- Regards, Peng

21 35

New regex module for 3.2?
by MRAB July 28, 2010

July 28, 2010

Hi all, I re-implemented the re module, adding new features and speed improvements. It's available at: http://pypi.python.org/pypi/regex under the name "regex" so that it can be tried alongside "re". I'd be interested in any comments or feedback. How does it compare with "re" in terms of speed on real-world data? The benchmarks suggest it should be faster, or at worst comparable. How much interest would there be in putting it in Python 3.2?

22 50

Re: [Python-Dev] Fast Implementation for ZIP decryption
by CJ Kucera July 28, 2010

July 28, 2010

Hello list, resurrecting a rather old thread from here: http://mail.python.org/pipermail/python-dev/2009-August/091450.html I've actually come across a use case where faster zipfile decryption would be very helpful to me (though one could certainly argue that it's a frivolous case). I'm writing a map editor for a game whose graphics datafiles are stored inside an encrypted zipfile. I've received the game developer's blessing for my editor to be able to read inside the zipfile, so legality isn't an issue here, but the pure-Python decryption currently available in the 'zipfile' module really is dog-slow. When I have my app load directly from already-uncompressed files on the HD, it's rare for a map to take more than 1 second to load; when the app has to use the pure-Python decryption routines to load all the graphics images, it can take over a minute (and this is on a 3GHz Core 2 Duo). Bundling the necessary graphics with my app isn't an option, nor is asking users to crack the zipfile encryption themselves to provide an uncompressed directory to read from. So anyway, I'd at least love to have a peek at the C code mentioned in the thread; if nothing more, perhaps I could bundle it into something that could be installed optionally by someone who wants it. Does anyone still have this code available somewhere? Thanks a bunch, CJ -- WOW: Flemmy | "Happiness isn't good enough for me! I pez(a)apocalyptech.com | demand euphoria!" 24.24.2.3171 | - Calvin

1 1