Issue http://bugs.python.org/issue1663329 details an annoyance in the
subprocess module that has affected several users, including me.
Essentially, closing hundreds of thousands of file descriptors by
round-tripping through the python exception machinery is very slow,
taking hundreds of milliseconds and at times many seconds. The
proposed fix is to write this loop in c. The c function is but a
handful of lines long. I purposefully kept the implementation
trivial so that it will work on all unix variants (there is another
issue that contains a super-duper optimization for AIX, and other
possibilities exist for Solaris, but the simple fix yields a ten-fold
speedup everywhere but windows, so I didn't think that it was worth
Though technically relating only to performance, I consider this a
bug-fix candidate as mysterious multi-second delays when launching a
subprocess end up making the functionality of close_fds unusable on
some platform configurations (namely, those with high MAX_FD set).
It would be great to see this is 2.5.2. Understanding that issue
evaluation takes significant effort, I've done some evaluation/triage
on other open tickets:
See issues for detailed comments.
http://bugs.python.org/issue1516330: No clear problem, invalid
patch. Recommend rejection.
http://bugs.python.org/issue1516327: No clear problem, no patch.
http://bugs.python.org/issue1705170: reproduced. Conjecture as to
why it is occurring, but I don't know the guts well enough to propose
a decent fix.
http://bugs.python.org/issue1773632: tested patch. Recommend
accepting unless there are things I don't know about this mysterious
_xmlrpclib extension (which there doubtlessly are)
http://bugs.python.org/issue738948: Rather old PEP that has gathered
no comments. Calling it a "PEP" is generous--it is really just a
link to an academic paper with a note about how this might be
integrated into Stackless.
I'm following the issue 1311: http://bugs.python.org/issue1311
There (and always talking in windows), the OP says the in Py2.4
os.path.exists("nul") returned True and now in 2.5 returns False. Note
that "nul" is an special file, something like /dev/null.
We made some tests, and we have inconsisten behaviour in previous
Python versions. For example, in Py2.3.5 in my machine I get a False,
as in Py2.5. But other person in the bug, gets True in 2.3.3 and
False in 2.5.
Even the OP has differents result for the same Python 2.4 in different machines.
Right now (but don't know exactly since when), Python relies in
kernel32.dll functions to make the stat on the file (if stat raises an
error, os.path.exists says that the file does not exist). Of course,
if I call to this function separately, I have the same behaviour.
So, the question is what we should do?:
1. Rely on the kernel32 function and behaves like it says?
2. Return a fixed response for this special file "nul"?
Personally, I prefer the first one, but it changed the semantic of
os.path.exists("nul") (but this semantic is not clear, as we get
different behaviour in different Python versions and windows
Thank you very much!
I'm always daunted by the prospect of trying to implement file locking.
This just came up again in SpamBayes where we have never protected our
pickle files from corruption when multiple processes access them
simultaneously. The presence of networked file systems and
platform-independent locks make it a nasty little problem. Maybe I'm just
showing my age. Does fcntl.flock work over NFS and SMB and on Windows? If
this is still as much of a mess as I remember, should Python provide a
simple file locking module in the standard distribution?
Side note: While reading the fcntl man page on my Mac I came across this
text in the description of F_GETLK/F_SETLK/F_SETLKW.
This interface follows the completely stupid semantics of System V and
IEEE Std 1003.1-1988 (``POSIX.1'') that require that all locks
associated with a file for a given process are removed when any file
descriptor for that file is closed by that process.... Flock(2) is
recommended for applications that want to ensure the integrity of their
locks when using library routines or wish to pass locks to their
I guess the BSD folks were a bit upset when they wrote that.
At 11:09 AM 10/31/2007 -0700, Guido van Rossum wrote:
>Yes, though I think that if prop.fdel is None, we could use func in
Eh? Isn't prop.fdel supposed to be a single-argument function? ISTM
that C is the only place where fset and fdel are the same thing.
> I'd like to make this [propset] a standard built-in,
+1 -- I find this to be an attractive syntax
> I'd also like to change property so that the doc
> string defaults to the doc string of the getter.
+1 -- This should also be done for classmethod,
staticmethod, and anything else that wraps functions.
Alternatively, the doc string could be made a little
__doc__ = 'property of function %s: %s' % (f.__name__, f.__doc__)
> I'm not sure about the name "propset" ...
>Maybe something like "setproperty" would be better.
I think not. Saying "setproperty" has too many ambiguous mental parsings. When does "set" take place -- assigning a value to a property is different defining the property itself. Is "set" a verb so that we're talking about a property of sets/frozensets. Is "set" a completed action so that the property has been set.
Let's stick with "propset" which has precedent as an svn action and serves as a short, simple mnemonic to the functionality.
Also, I find that these unique words are easier to search for. I once co-owned a magazine called Know Your Boston and it was very difficult for clients to find using Google.
The ctypes sources contain the source code for libffi, in Modules/_ctypes/libffi.
These sources were pulled from GCC cvs some time ago, and a new configure system
was written by Perky iirc.
Now, it seems that these sources are showing their age and a newer libffi version
should be used instead.
There are issues on certain platforms like mips64, arm, armeabi; also on other systems
to correctly use the c_longdouble data type that I added recently.
See also http://bugs.python.org/issue1292.
Pythons configure command has a --with-system-ffi option that, if specified, uses
a system installed libffi package (if present). By default, the bundled sources are
The cleanest way to fix this problem probably would be to remove the outdated libffi
sources from Python svn (in trunk and py3k), and require a system installed libffi
for building ctypes. However, I have no idea if there are libffi packages available
for systems like solaris, BSD, or OS X (to name a few that come to mind).
Another way would be to leave the sources in place, and only use them when no
system libffi installation is present - in other words to change the default
of the --with-system-ffi option.
Matthias Klose has already committed a change to configure.in that sets
--with-system-ffi="yes" for Linux/arm* systems. This would be a third way:
do this for all systems that require newer libffi.
PS: As a test, I changed setup.py (in branches/ctypes-branch) so that always
a system installed libffi is used, and manually triggered the build on the
bots that failed or even crashed before: ppc Debian unstable, and S-390 Debian.
PPS: The above does not apply for Windows; the windows libffi suorces work fine
and are in different subdirectories. Also, configure is not used on Windows.
> and have a matching propdel decorator?
-1. That would be a complete waste of builtin space.
Put stuff in when it is really needed. Ideas are
not required to automatically propagate from the
commonly used cases to the rarely used cases.
At 10:08 AM 10/31/2007 -0700, Guido van Rossum wrote:
>I've come up with a relatively unobtrusive pattern for defining
>setters. Given the following definition:
> assert isinstance(prop, property)
> def helper(func):
> return property(prop.__get__, func, func, prop.__doc__)
> return helper
Shouldn't that be property(prop.fget, func, prop.fdel, prop.__doc__),
and have a matching propdel decorator?
Apart from those extremely small nits, a big +1, and I expect to
swipe this idea more or less immediately. :)
>I'd also like to change property so that the doc string defaults to
>the doc string of the getter.
+1 also; I thought it did this already, but am now disappointed to
find it doesn't. :)