Mailman 3 March 2010 - Python-Dev

test_bigmem and test_bigaddrspace broken
by Antoine Pitrou 17 Mar '10

17 Mar '10

Hello, The title says it all. test_bigmem and test_bigaddrspace are partly broken under py3k, and no buildbot ever tests that they run successfully. As an example, here is a test_bigmem run under a 16GB machine. It gets "killed" at some point, probably by the Linux kernel's OOM killer, although I instruct the test suite to only use a fraction of the available RAM. There are also a couple of failures: $ ./python -m test.regrtest -v -M 8Gb test_bigmem == CPython 3.2a0 (py3k, Mar 17 2010, 18:38:10) [GCC 4.3.2] == Linux-a.b.c.d-grsec-xxxx-grs-ipv4-64-x86_64-with-debian-lenny-sid == /home/antoine/py3k/py3k/build/test_python_10071 test_bigmem test_capitalize (test.test_bigmem.StrTest) ... Skipping test_capitalize because of memory constraint ok test_center (test.test_bigmem.StrTest) ... ok test_compare (test.test_bigmem.StrTest) ... Skipping test_compare because of memory constraint ok test_concat (test.test_bigmem.StrTest) ... ok test_contains (test.test_bigmem.StrTest) ... Skipping test_contains because of memory constraint ok test_count (test.test_bigmem.StrTest) ... Skipping test_count because of memory constraint ok test_encode (test.test_bigmem.StrTest) ... ERROR test_encode_ascii (test.test_bigmem.StrTest) ... ok test_encode_raw_unicode_escape (test.test_bigmem.StrTest) ... ok test_encode_utf32 (test.test_bigmem.StrTest) ... ok test_encode_utf7 (test.test_bigmem.StrTest) ... ok test_endswith (test.test_bigmem.StrTest) ... Skipping test_endswith because of memory constraint ok test_expandtabs (test.test_bigmem.StrTest) ... Skipping test_expandtabs because of memory constraint ok test_find (test.test_bigmem.StrTest) ... Skipping test_find because of memory constraint ok test_format (test.test_bigmem.StrTest) ... Skipping test_format because of memory constraint ok test_hash (test.test_bigmem.StrTest) ... ok test_index (test.test_bigmem.StrTest) ... Skipping test_index because of memory constraint ok test_isalnum (test.test_bigmem.StrTest) ... Skipping test_isalnum because of memory constraint ok test_isalpha (test.test_bigmem.StrTest) ... Skipping test_isalpha because of memory constraint ok test_isdigit (test.test_bigmem.StrTest) ... Skipping test_isdigit because of memory constraint ok test_islower (test.test_bigmem.StrTest) ... Skipping test_islower because of memory constraint ok test_isspace (test.test_bigmem.StrTest) ... Skipping test_isspace because of memory constraint ok test_istitle (test.test_bigmem.StrTest) ... Skipping test_istitle because of memory constraint ok test_isupper (test.test_bigmem.StrTest) ... Skipping test_isupper because of memory constraint ok test_join (test.test_bigmem.StrTest) ... Skipping test_join because of memory constraint ok test_ljust (test.test_bigmem.StrTest) ... ok test_lower (test.test_bigmem.StrTest) ... Skipping test_lower because of memory constraint ok test_lstrip (test.test_bigmem.StrTest) ... ok test_repeat (test.test_bigmem.StrTest) ... ok test_replace (test.test_bigmem.StrTest) ... Skipping test_replace because of memory constraint ok test_repr_large (test.test_bigmem.StrTest) ... Skipping test_repr_large because of memory constraint ok test_repr_small (test.test_bigmem.StrTest) ... Skipping test_repr_small because of memory constraint ok test_rfind (test.test_bigmem.StrTest) ... Skipping test_rfind because of memory constraint ok test_rindex (test.test_bigmem.StrTest) ... Skipping test_rindex because of memory constraint ok test_rjust (test.test_bigmem.StrTest) ... ok test_rstrip (test.test_bigmem.StrTest) ... ok test_slice_and_getitem (test.test_bigmem.StrTest) ... Skipping test_slice_and_getitem because of memory constraint ok test_split_large (test.test_bigmem.StrTest) ... Skipping test_split_large because of memory constraint ok test_split_small (test.test_bigmem.StrTest) ... Skipping test_split_small because of memory constraint ok test_splitlines (test.test_bigmem.StrTest) ... Skipping test_splitlines because of memory constraint ok test_startswith (test.test_bigmem.StrTest) ... Skipping test_startswith because of memory constraint ok test_strip (test.test_bigmem.StrTest) ... ok test_swapcase (test.test_bigmem.StrTest) ... Skipping test_swapcase because of memory constraint ok test_title (test.test_bigmem.StrTest) ... Skipping test_title because of memory constraint ok test_translate (test.test_bigmem.StrTest) ... Skipping test_translate because of memory constraint ok test_unicode_repr (test.test_bigmem.StrTest) ... Skipping test_unicode_repr because of memory constraint ok test_unicode_repr_overflow (test.test_bigmem.StrTest) ... Skipping test_unicode_repr_overflow because of memory constraint ok test_unicode_repr_wide (test.test_bigmem.StrTest) ... Skipping test_unicode_repr_wide because of memory constraint ok test_upper (test.test_bigmem.StrTest) ... Skipping test_upper because of memory constraint ok test_zfill (test.test_bigmem.StrTest) ... ok test_capitalize (test.test_bigmem.BytesTest) ... ok test_center (test.test_bigmem.BytesTest) ... ok test_compare (test.test_bigmem.BytesTest) ... ok test_concat (test.test_bigmem.BytesTest) ... ok test_contains (test.test_bigmem.BytesTest) ... FAIL test_count (test.test_bigmem.BytesTest) ... ok test_decode (test.test_bigmem.BytesTest) ... ok test_endswith (test.test_bigmem.BytesTest) ... ok test_expandtabs (test.test_bigmem.BytesTest) ... ok test_find (test.test_bigmem.BytesTest) ... ok test_hash (test.test_bigmem.BytesTest) ... ok test_index (test.test_bigmem.BytesTest) ... ok test_isalnum (test.test_bigmem.BytesTest) ... ok test_isalpha (test.test_bigmem.BytesTest) ... ok test_isdigit (test.test_bigmem.BytesTest) ... ok test_islower (test.test_bigmem.BytesTest) ... ok test_isspace (test.test_bigmem.BytesTest) ... ok test_istitle (test.test_bigmem.BytesTest) ... ok test_isupper (test.test_bigmem.BytesTest) ... ok test_join (test.test_bigmem.BytesTest) ... ok test_ljust (test.test_bigmem.BytesTest) ... ok test_lower (test.test_bigmem.BytesTest) ... ok test_lstrip (test.test_bigmem.BytesTest) ... ok test_repeat (test.test_bigmem.BytesTest) ... ok test_replace (test.test_bigmem.BytesTest) ... ok test_rfind (test.test_bigmem.BytesTest) ... ok test_rindex (test.test_bigmem.BytesTest) ... ok test_rjust (test.test_bigmem.BytesTest) ... ok test_rstrip (test.test_bigmem.BytesTest) ... ok test_slice_and_getitem (test.test_bigmem.BytesTest) ... ok test_split_large (test.test_bigmem.BytesTest) ... Skipping test_split_large because of memory constraint ok test_split_small (test.test_bigmem.BytesTest) ... ok test_splitlines (test.test_bigmem.BytesTest) ... ok test_startswith (test.test_bigmem.BytesTest) ... ok test_strip (test.test_bigmem.BytesTest) ... ok test_swapcase (test.test_bigmem.BytesTest) ... ok test_title (test.test_bigmem.BytesTest) ... ok test_translate (test.test_bigmem.BytesTest) ... ok test_upper (test.test_bigmem.BytesTest) ... ok test_zfill (test.test_bigmem.BytesTest) ... ok test_capitalize (test.test_bigmem.BytearrayTest) ... ok test_center (test.test_bigmem.BytearrayTest) ... ok test_compare (test.test_bigmem.BytearrayTest) ... ok test_concat (test.test_bigmem.BytearrayTest) ... ok test_contains (test.test_bigmem.BytearrayTest) ... FAIL test_count (test.test_bigmem.BytearrayTest) ... ok test_decode (test.test_bigmem.BytearrayTest) ... ok test_endswith (test.test_bigmem.BytearrayTest) ... ok test_expandtabs (test.test_bigmem.BytearrayTest) ... ok test_find (test.test_bigmem.BytearrayTest) ... ok test_index (test.test_bigmem.BytearrayTest) ... ok test_isalnum (test.test_bigmem.BytearrayTest) ... ok test_isalpha (test.test_bigmem.BytearrayTest) ... ok test_isdigit (test.test_bigmem.BytearrayTest) ... ok test_islower (test.test_bigmem.BytearrayTest) ... ok test_isspace (test.test_bigmem.BytearrayTest) ... ok test_istitle (test.test_bigmem.BytearrayTest) ... ok test_isupper (test.test_bigmem.BytearrayTest) ... ok test_join (test.test_bigmem.BytearrayTest) ... ok test_ljust (test.test_bigmem.BytearrayTest) ... ok test_lower (test.test_bigmem.BytearrayTest) ... ok test_lstrip (test.test_bigmem.BytearrayTest) ... ok test_repeat (test.test_bigmem.BytearrayTest) ... ok test_replace (test.test_bigmem.BytearrayTest) ... ok test_rfind (test.test_bigmem.BytearrayTest) ... ok test_rindex (test.test_bigmem.BytearrayTest) ... ok test_rjust (test.test_bigmem.BytearrayTest) ... ok test_rstrip (test.test_bigmem.BytearrayTest) ... ok test_slice_and_getitem (test.test_bigmem.BytearrayTest) ... ok test_split_small (test.test_bigmem.BytearrayTest) ... ok test_splitlines (test.test_bigmem.BytearrayTest) ... ok test_startswith (test.test_bigmem.BytearrayTest) ... ok test_strip (test.test_bigmem.BytearrayTest) ... ok test_swapcase (test.test_bigmem.BytearrayTest) ... ok test_title (test.test_bigmem.BytearrayTest) ... ok test_translate (test.test_bigmem.BytearrayTest) ... ok test_upper (test.test_bigmem.BytearrayTest) ... ok test_zfill (test.test_bigmem.BytearrayTest) ... ok test_compare (test.test_bigmem.TupleTest) ... Skipping test_compare because of memory constraint ok test_concat_large (test.test_bigmem.TupleTest) ... Skipping test_concat_large because of memory constraint ok test_concat_small (test.test_bigmem.TupleTest) ... Skipping test_concat_small because of memory constraint ok test_contains (test.test_bigmem.TupleTest) ... Skipping test_contains because of memory constraint ok test_from_2G_generator (test.test_bigmem.TupleTest) ... Skipping test_from_2G_generator because of memory constraint ok test_from_almost_2G_generator (test.test_bigmem.TupleTest) ... Skipping test_from_almost_2G_generator because of memory constraint ok test_hash (test.test_bigmem.TupleTest) ... Skipping test_hash because of memory constraint ok test_index_and_slice (test.test_bigmem.TupleTest) ... Skipping test_index_and_slice because of memory constraint ok test_repeat_large (test.test_bigmem.TupleTest) ... Skipping test_repeat_large because of memory constraint ok test_repeat_large_2 (test.test_bigmem.TupleTest) ... Skipping test_repeat_large_2 because of memory constraint ok test_repeat_small (test.test_bigmem.TupleTest) ... Skipping test_repeat_small because of memory constraint ok test_repr_large (test.test_bigmem.TupleTest) ... Skipping test_repr_large because of memory constraint ok test_repr_small (test.test_bigmem.TupleTest) ... ERROR test_append (test.test_bigmem.ListTest) ... Skipping test_append because of memory constraint ok test_compare (test.test_bigmem.ListTest) ... Skipping test_compare because of memory constraint ok test_concat_large (test.test_bigmem.ListTest) ... Skipping test_concat_large because of memory constraint ok test_concat_small (test.test_bigmem.ListTest) ... Skipping test_concat_small because of memory constraint ok test_contains (test.test_bigmem.ListTest) ... Skipping test_contains because of memory constraint ok test_count (test.test_bigmem.ListTest) ... Skipping test_count because of memory constraint ok test_extend_large (test.test_bigmem.ListTest) ... Skipping test_extend_large because of memory constraint ok test_extend_small (test.test_bigmem.ListTest) ... Skipping test_extend_small because of memory constraint ok test_hash (test.test_bigmem.ListTest) ... Skipping test_hash because of memory constraint ok test_index (test.test_bigmem.ListTest) ... Skipping test_index because of memory constraint ok test_index_and_slice (test.test_bigmem.ListTest) ... Skipping test_index_and_slice because of memory constraint ok test_inplace_concat_large (test.test_bigmem.ListTest) ... Skipping test_inplace_concat_large because of memory constraint ok test_inplace_concat_small (test.test_bigmem.ListTest) ... Skipping test_inplace_concat_small because of memory constraint ok test_inplace_repeat_large (test.test_bigmem.ListTest) ... Skipping test_inplace_repeat_large because of memory constraint ok test_inplace_repeat_small (test.test_bigmem.ListTest) ... Skipping test_inplace_repeat_small because of memory constraint ok test_insert (test.test_bigmem.ListTest) ... Skipping test_insert because of memory constraint ok test_pop (test.test_bigmem.ListTest) ... Skipping test_pop because of memory constraint ok test_remove (test.test_bigmem.ListTest) ... Skipping test_remove because of memory constraint ok test_repeat_large (test.test_bigmem.ListTest) ... Skipping test_repeat_large because of memory constraint ok test_repeat_small (test.test_bigmem.ListTest) ... Skipping test_repeat_small because of memory constraint ok test_repr_large (test.test_bigmem.ListTest) ... Skipping test_repr_large because of memory constraint ok test_repr_small (test.test_bigmem.ListTest) ... Killed

1 0

Re: [Python-Dev] "Fixing" the new GIL
by David Beazley 17 Mar '10

17 Mar '10

I have been following this discussion about fixing the GIL and just wanted to make a few comments about it. To the doubters who don't think this is a real problem worth fixing, I must respectfully disagree. Multicore systems and parallel programming issues are not going away any time in the foreseeable future. Moreover, there are plenty of programmers who are using Python to build applications involving concurrency, distributed computing, and compute intensive processing. While everyone agrees that it would be great if the GIL simply disappeared, the next best option is for the GIL to have well-defined and reasonably predictable behavior (i.e., behavior that can be easily explained and justified). Specifically, if multiple threads happen to be performing CPU intensive work at the same time, it would be nice if they didn't thrash on multiple cores (the problem with the old GIL) and if I/O is mixed with CPU-bound processing it would be nice if I/O requests didn't have their responsiveness and priority penalized (the problem with the new GIL). So, just to be clear about the my bug report, it is directly related to the problem of overlapping I/O requests with CPU-bound processing. This kind of scenario comes up in the context of many applications--especially those based on cooperating processes, multiprocessing, and message passing. Ironically, these are exactly the kinds of parallel programming techniques that people are going to use because of the GIL in the first place. So, why would you want the new GIL to behave poorly here? In any case, if the GIL can be improved in a way that is simple and which either improves or doesn't negatively impact the performance of existing applications, why wouldn't you want to do it? Seems like a no-brainer. Cheers, Dave

8 18

tagging 3.1.2
by Benjamin Peterson 16 Mar '10

16 Mar '10

My plan is to tag 3.1.2 sometime on Thursday, so binaries can be built for the final release on Saturday. Agreeable? -- Regards, Benjamin

3 2

Python 2.6.5 rc 2
by Barry Warsaw 16 Mar '10

16 Mar '10

Hi Python hackateers! It looks like we finally have no more release blockers for 2.6.5rc2. I would like to tag the tree tonight for rc2 so that Martin can build the Windows installer for a release tomorrow. I am also moving the final release back to Friday March 19. -Barry

2 1

Re: [Python-Dev] (no subject)
by David Beazley 15 Mar '10

15 Mar '10

On Mon 15/03/10 4:34 AM , "Martin v. Löwis" martin(a)v.loewis.de sent: > > So, just to be clear about the my bug report, it > is directly related> to the problem of overlapping I/O requests with > CPU-bound processing.> This kind of scenario comes up in the context of > many> applications--especially those based on > cooperating processes,> multiprocessing, and message passing. > > How so? if you have cooperating processes, multiprocessing, and message > passing, you *don't* have CPU bound threads along with IO > "bound"threads in the same process - you may not have threads at all!!! > You're right in that end user will probably not be using threads in this situation. However, threads are still often used inside the associated programming libraries and frameworks that provide communication. For example, threads might be used to implement queuing, background monitoring, event notification, routing, and other similar low-level features. Just as an example, the multiprocessing module currently uses background threads as part of its implementation of queues. In a similar vein, threads are sometimes used in asynchronous I/O frameworks (e.g., Twisted) to handle certain kinds of deferred operations. Bottom line: just because a user isn't directly programming with threads doesn't mean that threads aren't there. > > In any case, if the GIL can be improved in a way > that is simple and> which either improves or doesn't negatively > impact the performance of> existing applications, why wouldn't you want to > do it? Seems like a> no-brainer. > > Unfortunately, a simple improvement doesn't really seem to exist. > Well, I think this problem can be fixed in a manner that is pretty well isolated to just one specific part of Python (the GIL)--especially since several prototypes, including my own, have already demonstrated that it's possible. In any case, as it stands now, the current "new GIL" ignores about 40 years of work concerning operating system thread/process scheduling and the efficient handling of I/O requests. I suspect that I'm not the only one who be disappointed if the Python community simply blew it off and said it's not an issue. Trying to fix it is a worthwhile goal. Cheers, Dave

1 0

interesting article on regex performance
by Neal Becker 14 Mar '10

14 Mar '10

http://swtch.com/~rsc/regexp/regexp1.html

16 22

Patch to telnetlib.py
by gregory dudek 13 Mar '10

13 Mar '10

The Telnet module telnetlib.py can be very slow -- unusably slow -- for large automated data transfers. There are typically done in raw mode. The attached patch greatly increased the speed of telnet interactions in raw mode. I submitted this a couple of year ago, but it was for an older branch of python. There are 2 key things being done: 1) concatenations string with string.join instead of '+' (which is probably a minor issue) 2) wholesale appending the raw and processed buffers when the IAC character is not found. The should be examined carefully since I am not an expert in the Telnet protocol, but it seems to work very well giving me a 5x speedup. --- installed/telnetlib.py 2010-02-02 22:57:58.000000000 -0500 +++ telnetlib.py 2010-03-13 12:17:02.000000000 -0500 @@ -30,6 +30,7 @@ - timeout should be intrinsic to the connection object instead of an option on one of the read calls only +Modified by G. Dudek for greater efficiency. """ @@ -420,6 +421,14 @@ """ buf = ['', ''] try: + if self.rawq: + if not IAC in self.rawq: + # speed hack, no IAC just grab whole queue. --Dudek + buf[self.sb] = "".join((buf[self.sb] , self.rawq)) + self.cookedq = "".join((self.cookedq , buf[0] )) + self.sbdataq = "".join((self.sbdataq , buf[1] )) + self.rawq_flush() + return while self.rawq: c = self.rawq_getchar() if not self.iacseq: @@ -428,7 +437,7 @@ if c == "\021": continue if c != IAC: - buf[self.sb] = buf[self.sb] + c + buf[self.sb] = "".join((buf[self.sb] , c)) continue else: self.iacseq += c @@ -480,8 +489,14 @@ self.iacseq = '' # Reset on EOF self.sb = 0 pass - self.cookedq = self.cookedq + buf[0] - self.sbdataq = self.sbdataq + buf[1] + self.cookedq = "".join((self.cookedq , buf[0] )) + self.sbdataq = "".join((self.sbdataq , buf[1] )) + + + def rawq_flush(self): + """ Set the queue to empty status """ + self.rawq = '' + self.irawq = 0 def rawq_getchar(self): """Get next char from raw queue. @@ -516,7 +531,7 @@ buf = self.sock.recv(50) self.msg("recv %r", buf) self.eof = (not buf) - self.rawq = self.rawq + buf + self.rawq = "".join((self.rawq,buf)) def sock_avail(self): """Test whether data is available on the socket."""

5 4

Desired changes to Hg emails to python-checkins
by Brett Cannon 13 Mar '10

13 Mar '10

1) I miss not having the affected files listed in the subject line. 2) The To field is set to hg(a)python.org which gets rejected as an invalid email address if you reply. Would be better to set it to python-checkins so that the habitual reply to a checkin won't get rejected. At least I would like to see these changes happen. -Brett

9 13

PyPy 1.2, JIT included
by Armin Rigo 12 Mar '10

12 Mar '10

================================== PyPy 1.2: Just-in-Time Compilation ================================== PyPy 1.2 has been released. The highlight of this release is to be the first that ships with a Just-in-Time compiler that is known to be faster than CPython (and unladen swallow) on some real-world applications (or the best benchmarks we could get for them). The main theme for the 1.2 release is speed. Main site: http://pypy.org/ The JIT is stable and we don't observe crashes. Nevertheless we would recommend you to treat it as beta software and as a way to try out the JIT to see how it works for you. Highlights of This Release ========================== * The JIT compiler. * Various interpreter optimizations that improve performance as well as help save memory. * Introducing a new PyPy website at http://pypy.org/ , made by tav and improved by the PyPy team. * Introducing http://speed.pypy.org/ , a new service that monitors our performance nightly, made by Miquel Torres. * There will be ubuntu packages on "PyPy's PPA" made by Bartosz Skowron; however various troubles prevented us from having them as of now. Known JIT problems (or why you should consider this beta software): * The only supported platform is 32bit x86 for now, we're looking for help with other platforms. * It is still memory-hungry. There is no limit on the amount of RAM that the assembler can consume; it is thus possible (although unlikely) that the assembler ends up using unreasonable amounts of memory. If you want to try PyPy, go to the "download page" on our excellent new site at http://pypy.org/download.html and find the binary for your platform. If the binary does not work (e.g. on Linux, because of different versions of external .so dependencies), or if your platform is not supported, you can try building from the source. What is PyPy? ============= Technically, PyPy is both a Python interpreter implementation and an advanced compiler, or more precisely a framework for implementing dynamic languages and generating virtual machines for them. The focus of this release is the introduction of a new transformation, the JIT Compiler Generator, and its application to the Python interpreter. Socially, PyPy is a collaborative effort of many individuals working together in a distributed and sprint-driven way since 2003. PyPy would not have gotten as far as it has without the coding, feedback and general support from numerous people. The PyPy release team, Armin Rigo, Maciej Fijalkowski and Amaury Forgeot d'Arc Together with Antonio Cuni, Carl Friedrich Bolz, Holger Krekel and Samuele Pedroni and many others: http://codespeak.net/pypy/dist/pypy/doc/contributor.html

1 0

Catch SIGINT at Python startup
by Victor Stinner 11 Mar '10

11 Mar '10

Hi, I wrote patches for the issue #3137 and I would like your opinion about my solution (not directly about the patches): I propose to consider a SIGINT during startup as a fatal error. If you press CTRL+c while Python is starting, Python displays an error and exit with the code 1. Currently, if you press CTRL+c while Python is starting: the SIGINT signal will be *ignored* somewhere between Python initialization and the first Python instruction. First problem is that if site import is interrupted, Python is not completly initialized (if a SIGINT is sent just after the call to initsigs(), initsite() is completly skipped). Example: ------------------ $ python ^C Traceback (most recent call last): File "/home/SHARE/SVN/python-trunk/Lib/site.py", line 62, in <module> import os File "/home/SHARE/SVN/python-trunk/Lib/os.py", line 44, in <module> from posix import * KeyboardInterrupt $ echo $? 1 ------------------ It's not easy to reproduce this issue because Python initialization is really fast (yeah really! or at least to fast for me ;-)). I'm using "valgrind --log- file=/dev/null ./python -v" to simulate a very slow computer :-) -- Victor Stinner http://www.haypocalc.com/

7 16