I've submitted a patch (*) to add an optional timeout to locking operations
(Lock.acquire() etc.). Since it's a pretty basic functionality, I would like to
know if there was any good reason for not doing it.
Has anyone else looked at using Coccinelle/spatch on CPython source
It's a GPL-licensed tool for matching semantic patterns in C source
code. It's been used on the Linux kernel for detecting and fixing
problems, and for autogenerating patches when refactoring
(http://coccinelle.lip6.fr/impact_linux.php). Although it's implemented
in OCaml, it is scriptable using Python.
I've been experimenting with using it on CPython code, both on the core
implementation, and on C extension modules.
As a test, I've written a validator for the mini-language used by
PyArg_ParseTuple and its variants. My code examines the types of the
variables passed as varargs, and attempts to check that they are
correct, according to the rules here
http://docs.python.org/c-api/arg.html (and in Python/getargs.c)
It can detect this old error (fixed in svn r34931):
buggy.c:12:socket_htons:Mismatching type of argument 1 in ""i:htons"":
expected "int *" but got "unsigned long *"
Similarly, it finds the deliberate error in xxmodule.c:
xxmodule.c:207:xx_roj:unknown format char in "O#:roj": '#'
(Unfortunately, when run on the full source tree, I see numerous
messages, and as far as I can tell, the others are false positives)
You can see the code here:
and download using anonymous git in this manner:
git clone git://fedorapeople.org/home/fedora/dmalcolm/public_git/check-cpython.git
The .cocci file detects invocations of PyArg_ParseTuple and determines
the types of the arguments. At each matching call site it invokes
python code, passing the type information to validate.py's
(I suspect it's possible to use spatch to detect reference counting
antipatterns; I've also attempted 2to3 refactoring of c code using
semantic patches, but so far macros tend to get in the way).
Alternatively, are there any other non-proprietary static analysis tools
After more thought, I think that separating the 2.7 and 3.2 releases
is not as big of an issue as I once thought. Therefore, I'd like to
adopt the schedule I posted a few weeks back for 2.7 only.
This only means some other lucky victi... I mean volunteer can do 3.2. :)
I've constructed an example program that does not leak memory in Python
2.x, but causes unbounded memory allocation in Python 3.1. Here is the
def __init__(self, fn):
self.fn = fn
a = E(f)
# get exception value in a python2/3 portable way
a = sys.exc_info()
Every time through the loop, the length of gc.get_objects() increases
by 7 under Python 3.1. I believe this is a regression error, since
Python 2.x does not exhibit the same behaviour.
Can anybody confirm this?
The buildbot waterfall is much greener now. Thanks to all who have
contributed to making it so (and it hasn't just been Mark and Antoine
and I, though we've been the most directly active (and yes, Mark, you
did contribute several fixes!)).
The 'stable builders' fleet is green now except for:
(1) issue 7269: occasional 2.6/trunk bsddb3 failures on windows
(2) issue 6748: random 3.1/3.x failures on most buidbots.
(3) the klose-debian-ppc builder being offline
Of these, (2) is by _far_ the biggest issue, and the one that causes the
most flap (success-failure-success). And flap is the thing that most
harms the buildbot usefulness.
Anyone who wants to debug this on a platform where it is consistently
reproducible please email me your public key and I'll give you a shell
account on my buildbot (Antoine already has one).
In the 'unstable' builder fleet, Antoine's new builder seems to be
stable across the board, while mine fails consistently on 3.1 and 3.x
because of the test_telnetlib bug. Thomas Heller's OS X buildbot is
consistently having lots of test failures (the same ones each time
I've checked). The master claims the klose-debian-alpha buildbot
doesn't know about branches, which is odd since it was working not
too long ago. The remaining buildslaves appear to have been offline
for some time.
Open issues here are:
(1) issue 3864: FreeBSD testing hangs consistently. According to the
ticket this is a FreeBSD bug fixed in 6.4, so an OS upgrade
on the buildslave would probably solve it.
(2) issue 4970: consistent signal 32 error on the norwitz-x86 Gentoo
buildslave in 3.1 and 3.x. This may be due to the box
running an old threading library, but it does make one wonder what
changed in 3.x that exposed it.
Another issue that I've seen on the buildbots but that doesn't
seem to be showing up right now (is it fixed?) is issue 7251, which
Mark is working on.
So, overall I think the buildbot fleet is in good shape, and if
we can nail issue 6748 I think it will be back to being an
important resource for sanity checking our checkins.
By the way, Georg set up the IRC interface on the #python-dev channel,
so you can hang out there if you want to get realtime reports of which
buildbots have going from success to failure and vice versa.
Python currently accepts global statements at the top level:
>>> global foo
Beside being a meaningless operation, this might lead unexperienced
user to make mistakes like:
>>> foo = 5
>>> global foo # make foo global
>>> def func():
... print foo # access the global foo
>>> # it works!
"global foo" should raise a SyntaxError, similarly to what already
happens with "return":
>>> return foo
File "<stdin>", line 1
SyntaxError: 'return' outside function
I opened an issue on the tracker (http://bugs.python.org/issue7329)
and Benjamin suggested to discuss this here.
The test he mentioned is in test_global.py:
prog_text_4 = """\
x = 2
# this should work
compile(prog_text_4, "<test string>", "exec")
It just says that "it should work" but it doesn't say /why/.
> Modified: python/branches/py3k/Lib/tokenize.py
> --- python/branches/py3k/Lib/tokenize.py (original)
> +++ python/branches/py3k/Lib/tokenize.py Sat Nov 14 17:27:26 2009
> @@ -377,17 +377,12 @@
> The first token sequence will always be an ENCODING token
> which tells you which encoding was used to decode the bytes stream.
> + # This import is here to avoid problems when the itertools module is not
> + # built yet and tokenize is imported.
> + from itertools import chain
This is probably a bad idea - calling tokenize.tokenize() from a thread
started as a side effect of importing a module will now deadlock on the
import lock if the module import waits for that thread to finish.
We tell people not to do that (starting and then waiting on threads as
part of module import) for exactly this reason, but it is also the
reason we avoid embedding import statements inside functions in the
standard library (not to mention encouraging third party developers to
also avoid embedding import statements inside functions).
This does constrain where we can use itertools - if we want carte
blanche to use it anywhere in the standard library, even those parts
that are imported as part of the build chain, we'll need to bite the
bullet and make it a builtin module rather than a separately built
P.S. The problem is easy to demonstrate on the current Py3k branch:
1. Put this in a module file in your py3k directory (e.g. "deadlock.py"):
f = open(__file__, 'rU')
t = threading.Thread(target=_deadlocks)
2. Then run: ./python -c "import deadlock"
It will, as advertised, deadlock and you'll need to use Ctrl-Brk or kill
-9 to get rid of it. (Note that preventing this kind of thing is one of
the major reasons why direct execution and even the -m switch *don't*
hang onto the import lock while running the __main__ module)
Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia
I'm in the process of adding a "run_path" function to the runpy module
that allows Python code to execute scripts using the same rules as the
CPython command line (i.e. accepting both source and compiled Python
files in addition to zipfiles, directories and other valid sys.path
entries containing a __main__.py file).
After that I was considering looking at the standard library to see
which modules can be used to execute other scripts (e.g. pdb, profile)
and determine what would be involved in updating them to support the
same flexibility as the main command line (possibly including adding
their own "-m" option to run modules by name rather than file path
For that second part:
1. Is it even worth doing at this stage? I'm not sure to what degree the
new command line flexibility has even been adopted by third party
application packagers, so I have no idea how large the pool of potential
users might be. Should I instead wait until we start seeing complaints
that these tools can't be used with script references that the main
command line will handle quite happily?
2. Aside from runpy itself, pdb and profile are the only examples that
spring to mind when I try to think of standard library modules that
execute other Python scripts. Are there any others that I'm missing?
(Even if we decide not to do anything at this stage, collating such a
list may still be handy for future reference)
P.S. pdb in particular may be messy to update, since the interaction
between the way it provides exception post-mortem debugging support and
the way the runpy module tries to avoid mutating the application's own
__main__ module will likely require careful handling.
Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia
At 08:33 PM 11/12/2009 -0500, Ian Bicking wrote:
>On Thu, Nov 12, 2009 at 7:52 PM, Antoine Pitrou
>Ben Finney <ben+python <at> <http://benfinney.id.au>benfinney.id.au> writes:
> > There's a problem with the poll's placement: on the front page of the
> > PyPI website.
>Speaking of which, why is it that
><http://pypi.python.org/pypi/>http://pypi.python.org/pypi/ (note the
>ending slash) return different contents
>(the latter being very voluminous)? I always mistake one for the other when
>entering the URL directly.
>easy_install replied on the behavior of /pypi/ (it uses the long
>list to do case-insensitive searches).Â Someone changed it,
>easy_install broke, and a compromise was to keep /pypi/ the way it
>was (but not /pypi).
>Probably this could be removed, as the /simple/ index is already
>case-insensitive, so easy_install shouldn't have to hit /pypi/ at all.
This was changed over a year ago; easy_install *does* use /simple by
default. I would guess enough people are upgraded by now that PyPI
need no longer continue to support it.
On Fri, 13 Nov 2009 04:27:48 am Ludvig Ericson wrote:
> On 12 nov 2009, at 14:38, Steven D'Aprano wrote:
> > On Thu, 12 Nov 2009 08:44:32 pm Ludvig Ericson wrote:
> >> Why are there comments on PyPI? Moreso, why are there comments
> >> which I cannot control as a package author on my very own
> >> packages? That's just absurd.
> > No, what's absurd is thinking that the act of publishing software
> > somehow gives you the right to demand control over what others say
> > about your software.
> ... on my own package's page.
It's your package. It's the community's page about your package.
I think of PyPI as a community-owned noticeboard. Its primary purpose is
to allow the community to find good packages -- the benefit to the
community is why it exists, not the benefit to the author of the
In my opinion, the community is best served by a good comment/review
system, one which avoids the worst trolling, and allows authors the
right of reply, but does not allow authors to censor inconvenient but