Mailman 3 May 2008 - Python-Dev

Distutils & configparser rename
by A.M. Kuchling May 16, 2008

May 16, 2008

Python 2.6 renames the ConfigParser module to be configparser. Distutils imports ConfigParser in various places. I just made a commit updating the import in one places, and then noticed that part of commit r63248, which made the same change, was reverted in order to preserve backward-compatibility. Instead, the default path will include lib-old again to keep the old module name available. I suggest dropping that goal, though. We've preserved compatibility but I'm not aware that anyone uses the Python 2.x Distutils with earlier versions of Python. In particular: * There's no standalone distutils package on PyPI, nor can I find such a package with a general web search. Am I missing it? * I do not see users advising other users to use a later version of Distutils to fix their problems. Is anyone actually benefiting from the effort of maintaining backward compatibility? --amk

7 7

SSH connection sharing
by Oleg Broytmann May 16, 2008

May 16, 2008

Hello! I would like to share a useful tip on how to speedup many short ssh (scp, rsync, svn over ssh) sessions (on Unix). Sorry if this information is widely known; I learned the trick a week ago and I am pretty happy with it. OpenSSH4 on Unix allows connection sharing - utilizing one authenticated TCP connection for a number of SSH sessions. The advantage is that subsequent connections don't need to open a new TCP connections and don't need to pass the authentication phase, thus they initialize a session much faster. This is very useful for running a lot of short ssh sessions (scp, sftp, rsync, svn over ssh). Detailed docs are in man ssh_config; see options ControlMaster and ControlPath. There is a lot of information_ on the web. .. _information: http://www.google.com/search?hl=en&ie=utf-8&oe=utf-8&q=ssh%20ControlPath Add the following to your ~/.ssh/config:: ControlMaster auto ControlPath ~/.ssh/.%r@%h:%p *ControlMaster auto* means the first client becomes the master, it opens TCP connection, authenticates and opens a unix socket named in ControlPath. Subsequent clients use the unix socket to connect to the server via the already established connection. In case the user tries to close the first (master) client it will wait until all sessions are finished; in case the user kills the master, it kills all its shared sessions. I use all 3 recommended macros, because I use ssh/scp/sftp/rsync/svn to connect to a number of servers. Drawbacks: the subsequent connections inherit some attributes of the master; agent forwarding, e.g. If ssh (or the OS, or even the entire computer) crashes one needs to remove unix sockets manually; to remedy this I changed ControlPath to ~/.ssh/controls/%r\@%h:%p and added to my crontab:: @reboot rm -f .ssh/controls/* Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd(a)phd.pp.ru Programmers don't die, they just GOSUB without RETURN.

1 0

Python parallel benchmark
by Tom Pinckney May 16, 2008

May 16, 2008

All the discussion recently about pyprocessing got me interested in actually benchmarking Python's multiprocessing performance to see if reality matched my expectations around what would scale up and what would not. I knew Python threads wouldn't be good for compute bound problems, but I was curious to see how well they worked for i/o bound problems. The short answer is that for i/o bound problems, python threads worked just as well as using multiple operating system processes. I wrote two simple benchmarks, one compute bound and the other i/o bound. The compute bound one did a parallel matrix multiply and the i/ o bound one read random records from a remote MySQL database. I ran each benchmark via python's thread module and via MPI (using mpi4py and openmpi and Send()/Recv() for communication). Each test was run multiple times and the numbers were consistent between test runs. I ran the tests on a dual-core Macbook Pro running OS X 10.5 and the included python 2.5.1. 1) Python threads a) compute bound: 1 thread - 16 seconds 2 threads - 21 seconds b) i/o bound: 1 thread -- 13 seconds 4 threads -- 10 seconds 8 threads -- 5 seconds 12 threads - 4 seconds 2) MPI a) compute bound: 1 thread - 17 seconds 2 threads -- 11 seconds b) i/o bound 1 thread -- 13 seconds 4 threads -- 10 seconds 8 threads -- 6 seconds 12 threads -- 4 seconds

7 11

rename of ConfigParser module?
by Mark Hammond May 16, 2008

May 16, 2008

Trying to build pywin32 from the trunk, I see: from distutils.config import PyPIRCCommand File "C:\src\python-svn\lib\distutils\config.py", line 8, in <module> from ConfigParser import ConfigParser ImportError: No module named ConfigParser Digging a little deeper, I see: |r63242 | alexandre.vassalotti | 2008-05-15 08:07:07 +1000 (Thu, 15 May 2008) | 2 lines | |Renamed the ConfigParser module to 'configparser'. Is there something I've missed? Cheers, Mark

7 9

Re: [Python-Dev] [Python-3000] PEP 3138- String representation in Python 3000
by Paul Moore May 16, 2008

May 16, 2008

2008/5/15 Atsuo Ishimoto <ishimoto(a)gembook.org>: > With my proposal, print("Hello\u03C8") prints "Hello\u03C8" instead of > raising an exception. And print(repr("Hello\u03C8")) prints > "'Hello\u03C8'", so no garbage are printed. > > Now, let's say you are Greek and working on Greek version of XP. > print("Hello\u03C8") prints "Hello"+collect Greek character(GREEK > SMALL LETTER PSI). And print(repr("Hello\u03C8")) prints > "'Hello"+collect Greek character+"'". If you have Greek font, you can > try this if you swich your command prompt by "chcp 1253" (change > codepage to 1253) on your command prompt. > [...] > Python detects user's capabilities, since Python 2.x(or 1.6? I forgot.) > On Windows, Python detects user's encoding from codepage. On Unix, > locale is used to detect encoding. Ah, thanks. I hadn't realised this - I've had trouble printing Unicode in the past, and assumed it was a result of Windows' strange console handling (OEM code pages vs Windows code pages confuse me). I use Unicode so rarely that it wasn't worth worrying about it, though. I guess the problem was my understanding, rather than code page detection not working. Sorry for the confusion. >> Like it or not, a large proportion of Python's users still work in >> environments where much of the Unicode character space is not >> displayed readably. >> > > I agree. So rejecting my proposal as "Not common use-case" might be > reasonable. But I should argue to get sympathy, anyway:). As Oleg pointed out, my comment "a large proportion" was a guess, and an unfounded one at that. And regardless, you definitely have my sympathy, this is an issue that needs solving :-) (Heck, just the fact that you have to write your emails to this group in a foreign language is enough to get you my sympathy!!!) > I can understand your aware. Perhaps you don't want see your terminal > flash by escape sequence, beep, endless graphic characters, etc. For > legacy byte-string applications(whether written in C or Python), > printing arbitrary string can cause such mess. But this is unlikely to > happen by printing the Unicode string, since the characters your > terminal cannot understand will be escaped or be converted to > character such as '?'. Ah, that's what the switching of the error mode is for. I understand more clearly now. > Hope this helps. It does - thanks for being patient with me. Paul.

1 0

Help with finishing PEP 3108
by Brett Cannon May 16, 2008

May 16, 2008

I need help to finish implementing PEP 3108. While over 80 modules are now deprecated in Python 2.6 (of which I did over 50 of), there are still over 20 tasks left to do in relation to the PEP. My free time is being sucked away since I have a conference paper deadline of June 1. And I am moving May 31. And I have to fly down to California to help my mother move on June 4. And other personal stuff (see a certain trend in my life at the moment?). So if you have time to help, please see issue 2775 (http://bugs.python.org/issue2775) and the dependencies list. The issues range from doing patch reviews of work people have already done, renaming a module, creating a new package, removing some use from the stdlib, or even backporting some changes made to 3.0 that were never merged into 2.6. In other words a wide variety of things. =) The PEP outlines the steps necessary to deprecate a module for deletion or for a rename in a step-by-step manner so you don't need to worry about forgetting a step. If you can't choose what to do, the issues that will lead to a module be deleted are the highest priority as renames can be handled by 2to3 in a later version while module deletions are harder to get pushed through and accepted. The modules left to still remove are still there because they are still used somehow in the stdlib. The module renames are mostly done at this point, but the new packages have not been handled yet. Obviously I don't want the beta to be held up by this, nor do I want to see any of the work left out because I couldn't get to it all. So any and all help is appreciated. -Brett

1 0

Importing bsddb 4.6.21; with or without AES encryption?
by Trent Nelson May 16, 2008

May 16, 2008

I was about to commit an initial import of the bsddb 4.6.21 source to the 'external' area in the repo, which I obtained from the following URL: http://www.oracle.com/technology/software/products/berkeley-db/index.html I downloaded the source that includes AES encryption, for no reason other than it was first on the list. I'm now wondering if we should only be importing the 'NC' source that doesn't contain any encryption? Jesus, does pybsddb use any of the Berkeley DB encryption facilities? Would anything break if we built the bsddb module without encryption? (Note that all of this only applies to Windows.) Trent.

1 0

PEP 370 extras
by Christian Heimes May 16, 2008

May 16, 2008

I'm working on some extra cool and useful additions to PEP 370. It's going to make compilation and usage of custom libraries very easy. Sit back and watch the slide show. :) The site module has two new options. The --user-base option prints the path to the user base directory to stdout. $ ~/dev/python/trunk/python -m site --user-base /home/heimes/.local I install libxslt-1.1.19 into my ~/.local directory. libxslt-1.1.19$ ./configure --prefix `~/dev/python/trunk/python -m site --user-base` libxslt-1.1.19$ make libxslt-1.1.19$ make install ... test -z "/home/heimes/.local/lib" || mkdir -p -- "/home/heimes/.local/lib" /bin/bash ../libtool --mode=install /usr/bin/install -c 'libxslt.la' '/home/heimes/.local/lib/libxslt.la' /usr/bin/install -c .libs/libxslt.so.1.1.19 /home/heimes/.local/lib/libxslt.so.1.1.19 Now I compile lxml against the libraries and include directories in ~/.local. But first I have to add ~/.local/bin to PATH so lxml picks up the right xslt-config binary. $ export PATH=~/.local/bin:$PATH lxml-1.3.6$ ~/dev/python/trunk/python setup.py build_ext --user Building lxml version 1.3.6-8122 setupinfo.py:137: DeprecationWarning: os.popen3 is deprecated. Use the subprocess module. wf, rf, ef = os.popen3(cmd) /home/heimes/dev/python/trunk/Lib/distutils/dist.py:263: UserWarning: Unknown distribution option: 'zip_safe' warnings.warn(msg) running build_ext building 'lxml.etree' extension creating build creating build/temp.linux-i686-2.6-pydebug creating build/temp.linux-i686-2.6-pydebug/src creating build/temp.linux-i686-2.6-pydebug/src/lxml gcc -pthread -fno-strict-aliasing -g -Wall -Wstrict-prototypes -fPIC -I/home/heimes/.local/include -I/usr/include/libxml2 -I/home/heimes/dev/python/trunk/Include -I/home/heimes/dev/python/trunk -I/home/heimes/.local/include -c src/lxml/etree.c -o build/temp.linux-i686-2.6-pydebug/src/lxml/etree.o -w creating build/lib.linux-i686-2.6-pydebug creating build/lib.linux-i686-2.6-pydebug/lxml gcc -pthread -shared build/temp.linux-i686-2.6-pydebug/src/lxml/etree.o -L/home/heimes/.local/lib -L/home/heimes/.local/lib -Wl,-R/home/heimes/.local/lib -lxslt -lexslt -lxml2 -lz -lm -o build/lib.linux-i686-2.6-pydebug/lxml/etree.so building 'lxml.objectify' extension gcc -pthread -fno-strict-aliasing -g -Wall -Wstrict-prototypes -fPIC -I/home/heimes/.local/include -I/usr/include/libxml2 -I/home/heimes/dev/python/trunk/Include -I/home/heimes/dev/python/trunk -I/home/heimes/.local/include -c src/lxml/objectify.c -o build/temp.linux-i686-2.6-pydebug/src/lxml/objectify.o -w gcc -pthread -shared build/temp.linux-i686-2.6-pydebug/src/lxml/objectify.o -L/home/heimes/.local/lib -L/home/heimes/.local/lib -Wl,-R/home/heimes/.local/lib -lxslt -lexslt -lxml2 -lz -lm -o build/lib.linux-i686-2.6-pydebug/lxml/objectify.so building 'lxml.pyclasslookup' extension gcc -pthread -fno-strict-aliasing -g -Wall -Wstrict-prototypes -fPIC -I/home/heimes/.local/include -I/usr/include/libxml2 -I/home/heimes/dev/python/trunk/Include -I/home/heimes/dev/python/trunk -I/home/heimes/.local/include -c src/lxml/pyclasslookup.c -o build/temp.linux-i686-2.6-pydebug/src/lxml/pyclasslookup.o -w gcc -pthread -shared build/temp.linux-i686-2.6-pydebug/src/lxml/pyclasslookup.o -L/home/heimes/.local/lib -L/home/heimes/.local/lib -Wl,-R/home/heimes/.local/lib -lxslt -lexslt -lxml2 -lz -lm -o build/lib.linux-i686-2.6-pydebug/lxml/pyclasslookup.so lxml-1.3.6$ ~/dev/python/trunk/python setup.py install --user ... As you can see etree.so is going to load my local libxslt library instead of the system one. $ ldd /home/heimes/.local/lib/python2.6/site-packages/lxml/etree.so linux-gate.so.1 => (0xb7fc5000) libxslt.so.1 => /home/heimes/.local/lib/libxslt.so.1 (0xb7e89000) libexslt.so.0 => /home/heimes/.local/lib/libexslt.so.0 (0xb7e78000) libxml2.so.2 => /usr/lib/libxml2.so.2 (0xb7d3d000) libz.so.1 => /usr/lib/libz.so.1 (0xb7d28000) libm.so.6 => /lib/tls/i686/cmov/libm.so.6 (0xb7d03000) libpthread.so.0 => /lib/tls/i686/cmov/libpthread.so.0 (0xb7ceb000) libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7b9c000) libgcrypt.so.11 => /lib/libgcrypt.so.11 (0xb7b4e000) libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0xb7b4a000) /lib/ld-linux.so.2 (0xb7fc6000) libgpg-error.so.0 => /lib/libgpg-error.so.0 (0xb7b46000) Comments? :) Christian

5 6

Buildbots for trunk are all red
by Facundo Batista May 16, 2008

May 16, 2008

All fail in test_compiler.py. -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/

21 51

availability of httplib.HTTPResponse.close
by Thomas Lee May 14, 2008

May 14, 2008

I was debating whether this was truly a question for python-dev or if I should take it to one of the user lists. Ultimately it feels like a question about the implementation of a core module, so hopefully nobody minds me posting it here. :) Although not listed as a public API method in the documentation, it would seem the httplib.HTTPResponse.close method might be useful in the event we don't want to actually read any of the data being sent back from the server. If I'm to assume that an undocumented method is considered "private", currently the only way to force the underlying file-like socket wrapper to close is by calling read() until no more data remains (as per the documentation). What's the reasoning behind requiring callers to read() all the pending data vs. just closing the socket? Is the close method really off limits to code using HTTPResponse? Cheers, Tom

2 1