Mailman 3 November 2009 - Python-Dev

buildtime vs runtime in Distutils
by Tarek Ziadé 19 Nov '09

19 Nov '09

Hello, http://bugs.python.org/issue4359 reminds me that Distutils reads build files like Makefile or pyconfig.h to get some environment variables through the sysconfig module at *runtime*. This cannot work on all platforms, when our Makefile is not shipped with python but python-devel. (like Fedora) Since I am already working on the refactoring of this distutils/sysconfig module, so it lands in a stdlib module together with some elements from site.py (see a previous mail on the topic, and my tarek_sysconfig branch - work in progress), I was wondering if we couldn't make this module a template, and inject at its beginning new values extracted from Makefile and pyconfig.h. This could happen when "configure" is called, Regards, Tarek -- Tarek Ziadé | http://ziade.org | オープンソースはすごい! | 开源传万世，因有你参与

16 27

Add an optional timeout to lock operations
by Antoine Pitrou 18 Nov '09

18 Nov '09

Hello, I've submitted a patch (*) to add an optional timeout to locking operations (Lock.acquire() etc.). Since it's a pretty basic functionality, I would like to know if there was any good reason for not doing it. (*) http://bugs.python.org/issue7316 Thank you Antoine.

5 8

Static analysis of CPython using coccinelle/spatch
by David Malcolm 18 Nov '09

18 Nov '09

Has anyone else looked at using Coccinelle/spatch[1] on CPython source code? It's a GPL-licensed tool for matching semantic patterns in C source code. It's been used on the Linux kernel for detecting and fixing problems, and for autogenerating patches when refactoring (http://coccinelle.lip6.fr/impact_linux.php). Although it's implemented in OCaml, it is scriptable using Python. I've been experimenting with using it on CPython code, both on the core implementation, and on C extension modules. As a test, I've written a validator for the mini-language used by PyArg_ParseTuple and its variants. My code examines the types of the variables passed as varargs, and attempts to check that they are correct, according to the rules here http://docs.python.org/c-api/arg.html (and in Python/getargs.c) It can detect this old error (fixed in svn r34931): buggy.c:12:socket_htons:Mismatching type of argument 1 in ""i:htons"": expected "int *" but got "unsigned long *" Similarly, it finds the deliberate error in xxmodule.c: xxmodule.c:207:xx_roj:unknown format char in "O#:roj": '#' (Unfortunately, when run on the full source tree, I see numerous messages, and as far as I can tell, the others are false positives) You can see the code here: http://fedorapeople.org/gitweb?p=dmalcolm/public_git/check-cpython.git;a=tr… and download using anonymous git in this manner: git clone git://fedorapeople.org/home/fedora/dmalcolm/public_git/check-cpython.git The .cocci file detects invocations of PyArg_ParseTuple and determines the types of the arguments. At each matching call site it invokes python code, passing the type information to validate.py's validate_types. (I suspect it's possible to use spatch to detect reference counting antipatterns; I've also attempted 2to3 refactoring of c code using semantic patches, but so far macros tend to get in the way). Alternatively, are there any other non-proprietary static analysis tools for CPython? Thoughts? Dave [1] http://coccinelle.lip6.fr/

4 5

2.7 and 3.2 release schedules
by Benjamin Peterson 17 Nov '09

17 Nov '09

After more thought, I think that separating the 2.7 and 3.2 releases is not as big of an issue as I once thought. Therefore, I'd like to adopt the schedule I posted a few weeks back for 2.7 only. This only means some other lucky victi... I mean volunteer can do 3.2. :) -- Regards, Benjamin

4 3

new unbounded memory leak in exception handling?
by Greg Hewgill 17 Nov '09

17 Nov '09

I've constructed an example program that does not leak memory in Python 2.x, but causes unbounded memory allocation in Python 3.1. Here is the code: import gc import sys class E(Exception): def __init__(self, fn): self.fn = fn def call(self): self.fn() def f(): raise E(f) a = E(f) while True: print(len(gc.get_objects())) try: a.call() except E: # get exception value in a python2/3 portable way a = sys.exc_info()[1] Every time through the loop, the length of gc.get_objects() increases by 7 under Python 3.1. I believe this is a regression error, since Python 2.x does not exhibit the same behaviour. Can anybody confirm this? Greg Hewgill http://hewgill.com

2 2

Status of the Buildbot fleet and related bugs
by R. David Murray 15 Nov '09

15 Nov '09

The buildbot waterfall is much greener now. Thanks to all who have contributed to making it so (and it hasn't just been Mark and Antoine and I, though we've been the most directly active (and yes, Mark, you did contribute several fixes!)). The 'stable builders' fleet is green now except for: (1) issue 7269: occasional 2.6/trunk bsddb3 failures on windows (2) issue 6748: random 3.1/3.x failures on most buidbots. (3) the klose-debian-ppc builder being offline Of these, (2) is by _far_ the biggest issue, and the one that causes the most flap (success-failure-success). And flap is the thing that most harms the buildbot usefulness. Anyone who wants to debug this on a platform where it is consistently reproducible please email me your public key and I'll give you a shell account on my buildbot (Antoine already has one). In the 'unstable' builder fleet, Antoine's new builder seems to be stable across the board, while mine fails consistently on 3.1 and 3.x because of the test_telnetlib bug. Thomas Heller's OS X buildbot is consistently having lots of test failures (the same ones each time I've checked). The master claims the klose-debian-alpha buildbot doesn't know about branches, which is odd since it was working not too long ago. The remaining buildslaves appear to have been offline for some time. Open issues here are: (1) issue 3864: FreeBSD testing hangs consistently. According to the ticket this is a FreeBSD bug fixed in 6.4, so an OS upgrade on the buildslave would probably solve it. (2) issue 4970: consistent signal 32 error on the norwitz-x86 Gentoo buildslave in 3.1 and 3.x. This may be due to the box running an old threading library, but it does make one wonder what changed in 3.x that exposed it. Another issue that I've seen on the buildbots but that doesn't seem to be showing up right now (is it fixed?) is issue 7251, which Mark is working on. So, overall I think the buildbot fleet is in good shape, and if we can nail issue 6748 I think it will be back to being an important resource for sanity checking our checkins. By the way, Georg set up the IRC interface on the #python-dev channel, so you can hang out there if you want to get realtime reports of which buildbots have going from success to failure and vice versa. --David

12 22

global statements outside functions/methods should raise SyntaxError
by Ezio Melotti 15 Nov '09

15 Nov '09

Python currently accepts global statements at the top level: >>> global foo >>> Beside being a meaningless operation, this might lead unexperienced user to make mistakes like: >>> foo = 5 >>> global foo # make foo global >>> def func(): ... print foo # access the global foo ... >>> func() 5 >>> # it works! "global foo" should raise a SyntaxError, similarly to what already happens with "return": >>> return foo File "<stdin>", line 1 SyntaxError: 'return' outside function I opened an issue on the tracker (http://bugs.python.org/issue7329) and Benjamin suggested to discuss this here. The test he mentioned is in test_global.py: def test4(self): prog_text_4 = """\ global x x = 2 """ # this should work compile(prog_text_4, "<test string>", "exec") It just says that "it should work" but it doesn't say /why/. Any thoughts?

7 6

Using itertools in modules that are part of the build chain (Re: [Python-checkins] r76264 - python/branches/py3k/Lib/tokenize.py)
by Nick Coghlan 15 Nov '09

15 Nov '09

benjamin.peterson wrote: > Modified: python/branches/py3k/Lib/tokenize.py > ============================================================================== > --- python/branches/py3k/Lib/tokenize.py (original) > +++ python/branches/py3k/Lib/tokenize.py Sat Nov 14 17:27:26 2009 > @@ -377,17 +377,12 @@ > The first token sequence will always be an ENCODING token > which tells you which encoding was used to decode the bytes stream. > """ > + # This import is here to avoid problems when the itertools module is not > + # built yet and tokenize is imported. > + from itertools import chain This is probably a bad idea - calling tokenize.tokenize() from a thread started as a side effect of importing a module will now deadlock on the import lock if the module import waits for that thread to finish. We tell people not to do that (starting and then waiting on threads as part of module import) for exactly this reason, but it is also the reason we avoid embedding import statements inside functions in the standard library (not to mention encouraging third party developers to also avoid embedding import statements inside functions). This does constrain where we can use itertools - if we want carte blanche to use it anywhere in the standard library, even those parts that are imported as part of the build chain, we'll need to bite the bullet and make it a builtin module rather than a separately built extension module. Cheers, Nick. P.S. The problem is easy to demonstrate on the current Py3k branch: 1. Put this in a module file in your py3k directory (e.g. "deadlock.py"): ----------- import threading import tokenize f = open(__file__, 'rU') def _deadlocks(): tokenize.tokenize(f.readline) t = threading.Thread(target=_deadlocks) t.start() t.join() ----------- 2. Then run: ./python -c "import deadlock" It will, as advertised, deadlock and you'll need to use Ctrl-Brk or kill -9 to get rid of it. (Note that preventing this kind of thing is one of the major reasons why direct execution and even the -m switch *don't* hang onto the import lock while running the __main__ module) -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia ---------------------------------------------------------------

3 3

Modules that run other scripts
by Nick Coghlan 14 Nov '09

14 Nov '09

I'm in the process of adding a "run_path" function to the runpy module that allows Python code to execute scripts using the same rules as the CPython command line (i.e. accepting both source and compiled Python files in addition to zipfiles, directories and other valid sys.path entries containing a __main__.py file). After that I was considering looking at the standard library to see which modules can be used to execute other scripts (e.g. pdb, profile) and determine what would be involved in updating them to support the same flexibility as the main command line (possibly including adding their own "-m" option to run modules by name rather than file path location). For that second part: 1. Is it even worth doing at this stage? I'm not sure to what degree the new command line flexibility has even been adopted by third party application packagers, so I have no idea how large the pool of potential users might be. Should I instead wait until we start seeing complaints that these tools can't be used with script references that the main command line will handle quite happily? 2. Aside from runpy itself, pdb and profile are the only examples that spring to mind when I try to think of standard library modules that execute other Python scripts. Are there any others that I'm missing? (Even if we decide not to do anything at this stage, collating such a list may still be handy for future reference) Regards, Nick. P.S. pdb in particular may be messy to update, since the interaction between the way it provides exception post-mortem debugging support and the way the runpy module tries to avoid mutating the application's own __main__ module will likely require careful handling. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia ---------------------------------------------------------------

2 2

Re: [Python-Dev] PyPI front page
by P.J. Eby 14 Nov '09

14 Nov '09

At 08:33 PM 11/12/2009 -0500, Ian Bicking wrote: >On Thu, Nov 12, 2009 at 7:52 PM, Antoine Pitrou ><<mailto:solipsis@pitrou.net>solipsis(a)pitrou.net> wrote: >Ben Finney <ben+python <at> <http://benfinney.id.au>benfinney.id.au> writes: > > > > There's a problem with the poll's placement: on the front page of the > > PyPI website. > >Speaking of which, why is it that ><http://pypi.python.org/pypi>http://pypi.python.org/pypi and ><http://pypi.python.org/pypi/>http://pypi.python.org/pypi/ (note the >ending slash) return different contents >(the latter being very voluminous)? I always mistake one for the other when >entering the URL directly. > > >easy_install replied on the behavior of /pypi/ (it uses the long >list to do case-insensitive searches).Â Someone changed it, >easy_install broke, and a compromise was to keep /pypi/ the way it >was (but not /pypi). > >Probably this could be removed, as the /simple/ index is already >case-insensitive, so easy_install shouldn't have to hit /pypi/ at all. This was changed over a year ago; easy_install *does* use /simple by default. I would guess enough people are upgraded by now that PyPI need no longer continue to support it.

1 0