I'd like to propose a new function "open_noinherit"
or maybe even a new mode flag "n" for the builtin "open"
(see footnote for the names).
The new function should work exactly like the builtin "open", with one
The open file is not inherited to any child processes
(whereas files opened with "open" will be inherited).
The new function can be implemented (basically) using
os.O_NOINHERIT on MS Windows
resp. fcntl / FD_CLOEXEC on Posix.
I will post a working Python implementation next week.
There are five reasons for the proposal:
1) The builtin "open" causes unexpected problems in conjunction with
in particular in multi-threaded programs.
It can cause file permission errors in the subprocess or in the current
On Microsoft Windows, some of the possible file permission errors are
documented by Microsoft (thus very few programs written for Windows
2) Inheriting open file handles to subprocesses is a security risk.
3) For the developer, finding "cause and effect" is *very* hard, in
multi-threaded programs, when the errors occur only in race-conditions.
4) The problems arise in some of the standard library modules as well,
5) Very few developers are aware of the possible problems.
As a work-around, one can replace open with
os.fdopen (os.open (..., + os.O_NOINHERIT), ... )
on Windows, but that's really ugly, hard to read,
may raise a different exception than open (IOError instead of OSError),
and needs careful work to take platform-specific code into account
Here is a single-threaded example to demonstrate the effect:
outf = open ("blah.tmp", "wt")
subprocess.Popen("notepad.exe") # or whatever program you like, but
# It must be a program that does not exit immediately!
# Now the subprocess has inherited the open file handle
# We can still write:
outf.write ("Hello world!\n")
# But we can not rename the file (at least on Windows)
os.rename ("blah.tmp", "blah.txt")
# this fails with OSError: [Errno 13] Permission denied
# Similar problems with other file operations on non-Windows platforms.
Ok, in this little program one can see what is going wrong easily.
But what if the subprocess exits very quickly?
Then perhaps you see the OSError, perhaps not - depending on the process
of your operation system.
In a commercial multi-theaded daemon application, the error only occured
under heavy load and was hard to reproduce - and it was even harder to find
That's because cause and effect were in two different threads in two
parts of the program:
- Thread A opens a file and starts to write data
- Thread B starts a subprocess (which inherits the file handle from thread
- Thread A continues writing to the file and closes it.
- And now it's a race condition:
- a) Thread A wants to rename the file - b) the subprocess exits.
If a) is first: Error, if b) is first: no error.
To make things more complicated, even two subprocesses can disturb each
The new function should be implemented in C ideally, because the GIL could
prevent a thread-switch between os.open and the fcntl.F_SETFD call.
Note that the problem described here arises not only for files, but for
See bug 1222790: SimpleXMLRPCServer does not set FD_CLOEXEC
Once there is an easy-to-use, platform-independent, documented builtin
"open_noinherit" (or a new mode flag for "open"), the standard library
be considered. For each occurence of "open" or "file", it should be
if it necessary to inherit the file to subprocesses. If not, it should be
One example is shutil.filecopy, where open_noiherit should be used instead
The socket module is another candidate, I think - but I'm not sure about
A nice effect of using "open_noinherit" is that - in many cases - one no
needs to speficy close_fds = True when calling subprocess.Popen.
[Note that close_fds is *terribly* slow if MAX_OPEN_FILES is "big", e.g.
see bug 1663329]
While writing this mail, at least 3 times I typed "nonherit" instead of
So maybe someone can propose a better name?
Or a new mode flag character could be "p" (like "private" or "protected").
PEP 3101 says Py3K should allow item and attribute access syntax
within string templating expressions but "to limit potential security
issues", access to underscore prefixed names within attribute/item
access expressions will be disallowed.
I am a person who has lived with the aftermath of a framework
designed to prevent data access by restricting access to underscore-
prefixed names (Zope 2, ahem), and I've found it's very hard to
explain and justify. As a result, I feel that this is a poor default
policy choice for a framework.
In some cases, underscore names must become part of an object's
external interface. Consider a URL with one or more underscore-
prefixed path segment elements (because prefixing a filename with an
underscore is a perfectly reasonable thing to do on a filesystem, and
path elements are often named after file names) fed to a traversal
algorithm that attempts to resolve each path element into an object
by calling __getitem__ against the parent found by the last path
element's traversal result. Perhaps this is poor design and
__getitem__ should not be consulted here, but I doubt that highly
because there's nothing particularly special about calling a method
named __getitem__ as opposed to some method named "traverse".
The only precedent within Python 2 for this sort of behavior is
limiting access to variables that begin with __ and which do not end
with __ to the scope defined by a class and its instances. I
personally don't believe this is a very useful feature, but it's
still only an advisory policy and you can worm around it with enough
Given that security is a concern at all, the only truly reasonable
way to "limit security issues" is to disallow item and attribute
access completely within the string templating expression syntax. It
seems gratuituous to me to encourage string templating expressions
with item/attribute access, given that you could do it within the
format arguments just as easily in the 99% case, and we've (well...
I've) happily been living with that restriction for years now.
But if this syntax is preserved, there really should be no *default*
restrictions on the traversable names within an expression because
this will almost certainly become a hard-to-explain, hard-to-justify
bug magnet as it has become in Zope.
I haven't responded to this thread because I was hoping some of the
original proponents of the feature would come out to defend it.
(Remember, 3101 is a synthesis of a lot of people's ideas gleaned from
many forum postings - In some cases I am willing to defend particular
aspects of the PEP, and in others I just write down what I think the
general consensus is.)
That being said - from what I've read so far, the evidence on both sides
of the argument seems anecdotal to me. I'd rather wait and see what more
people have to say on the topic.
Aurélien Campéas wrote:
> On Tue, Jun 19, 2007 at 08:20:25AM -0700, Guido van Rossum wrote:
>> Those are valid concerns. I'm cross-posting this to the python-3000
>> list in the hope that the PEP's author and defendents can respond. I'm
>> sure we can work something out.
> Thanks to raise this. It is horrible enough that I feel obliged to
> -10 on this part of PEP3101.
>> Please keep further discussion on the python-3000(a)python.org list.
>> On 6/19/07, Chris McDonough <chrism(a)plope.com> wrote:
>>> Wrt http://www.python.org/dev/peps/pep-3101/
>>> PEP 3101 says Py3K should allow item and attribute access syntax
>>> within string templating expressions but "to limit potential security
>>> issues", access to underscore prefixed names within attribute/item
>>> access expressions will be disallowed.
> People talking about potential security issues should have an
> obligation to show how their proposals *really* improve security (in
> general); this is of course, a hard thing to do; mere hand-waving is
> not sufficient.
>>> I am a person who has lived with the aftermath of a framework
>>> designed to prevent data access by restricting access to underscore-
>>> prefixed names (Zope 2, ahem), and I've found it's very hard to
>>> explain and justify. As a result, I feel that this is a poor default
>>> policy choice for a framework.
> And it's even poorer in the context of a language (for it's probably
> harder to escape language-level restrictions than framework
> obscurities ...).
>>> In some cases, underscore names must become part of an object's
>>> external interface. Consider a URL with one or more underscore-
>>> prefixed path segment elements (because prefixing a filename with an
>>> underscore is a perfectly reasonable thing to do on a filesystem, and
>>> path elements are often named after file names) fed to a traversal
>>> algorithm that attempts to resolve each path element into an object
>>> by calling __getitem__ against the parent found by the last path
>>> element's traversal result. Perhaps this is poor design and
>>> __getitem__ should not be consulted here, but I doubt that highly
>>> because there's nothing particularly special about calling a method
>>> named __getitem__ as opposed to some method named "traverse".
> This is trying to make a technical argument, but the 'consenting
> adults' policy might be enough. In my experience, zope forbiding
> access to _ prefixed attributes just led to work around the
> limitation, thus adding more useless indirection to an already crufty
> code base. The result is more obfuscation and probably even less
> security (as in auditability of the code).
>>> The only precedent within Python 2 for this sort of behavior is
>>> limiting access to variables that begin with __ and which do not end
>>> with __ to the scope defined by a class and its instances. I
>>> personally don't believe this is a very useful feature, but it's
>>> still only an advisory policy and you can worm around it with enough
> FWIW I've come to never use __attrs. The obfuscation feature seems to
> bring nothing but pain (the few times I've fell into that trap as a
> beginner python programmer).
>>> Given that security is a concern at all, the only truly reasonable
>>> way to "limit security issues" is to disallow item and attribute
>>> access completely within the string templating expression syntax. It
>>> seems gratuituous to me to encourage string templating expressions
>>> with item/attribute access, given that you could do it within the
>>> format arguments just as easily in the 99% case, and we've (well...
>>> I've) happily been living with that restriction for years now.
>>> But if this syntax is preserved, there really should be no *default*
>>> restrictions on the traversable names within an expression because
>>> this will almost certainly become a hard-to-explain, hard-to-justify
>>> bug magnet as it has become in Zope.
> I'd add that Zope in general looks to me like a giant collection of
> python anti-patterns and as such can be used as a clue source about
> what not to do, especially what not to include in Py3k.
> I don't want to offense people, well no more than necessary (imho zope
> *is* an offense to common sense in many ways), but that's the opinion
> from someone who earns its living mostly from zope/plone products
> dev. and maintenance (these days, anyway).
> Python-3000 mailing list
> Unsubscribe: http://mail.python.org/mailman/options/python-3000/talin%40acm.org
ACTIVITY SUMMARY (06/17/07 - 06/24/07)
Tracker at http://bugs.python.org/
To view or respond to any of the issues listed below, click on the issue
number. Do NOT respond to this message.
1645 open ( +0) / 8584 closed ( +0) / 10229 total ( +0)
Average duration of open issues: 836 days.
Median duration of open issues: 784 days.
Open Issues Breakdown
open 1645 ( +0)
pending 0 ( +0)
If I do "bzr status" in dinsdale:/etc/apache2, I get
bzr: ERROR: bzrlib.errors.BzrCheckError: Internal check failed: file
u'/etc/init.d/stop-bootlogd' entered as kind 'symlink' id
'stopbootlogd-20070303140018-fe340b888f6e9c69', now of kind 'file'
Traceback (most recent call last):
File "/usr/lib/python2.4/site-packages/bzrlib/commands.py", line 611,
BzrCheckError: Internal check failed: file u'/etc/init.d/stop-bootlogd'
entered as kind 'symlink' id
'stopbootlogd-20070303140018-fe340b888f6e9c69', now of kind 'file'
bzr 0.11.0 on python 2.4.4.final.0 (linux2)
arguments: ['/usr/bin/bzr', 'status']
** please send this report to bazaar-ng(a)lists.ubuntu.com
Can somebody experienced with bzr please help?
Vilnius/Post EuroPython PyPy Sprint 12-14th of July
The PyPy team is sprinting at EuroPython again and we invite
you to participate in our 3 day long sprint at the conference hotel
- Reval Hotel Lietuva.
If you plan to attend the sprint we recommend you to listen to the PyPy
technical talks (`EuroPython schedule`_) during the
conference since it will give you a good overview of the status of development.
On the morning of the first sprint day (12th) we will also have a
tutorial session for those new to PyPy development. As 3 days is relatively
short for a PyPy sprint we suggest to travel back home on the 15th if
possible (but it is ok to attend less than 3 days too).
Goals and topics of the sprint
There are many possible and interesting sprint topics to work on - here
we list some possible task areas:
* completing the missing python 2.5 features and support
* write or port more extension modules (e.g. zlib is missing)
* identify slow areas of PyPy through benchmarking and work on improvements,
possibly moving app-level parts of the Python interpreter to interp-level
* there are some parts of PyPy in need of refactoring, we may spend some time
on those, for example:
- rctypes and the extension compiler need some rethinking
- support for LLVM 2.0 for the llvm backend
* some JIT improvement work
* port the stackless transform to ootypesystem
* other interesting stuff that you would like to work on ...;-)
If you'd like to come, please subscribe to the `pypy-sprint mailing list`_
and drop a note about your interests and post any questions. More
organisational information will be sent to that list.
Please register by adding yourself on the following list (via svn):
or on the pypy-sprint mailing list if you do not yet have check-in rights:
Preparation (if you feel it is needed):
* read the `getting-started`_ pages on http://codespeak.net/pypy
* for inspiration, overview and technical status you are welcome to
read `the technical reports available and other relevant documentation`_
* please direct any technical and/or development oriented questions to
pypy-dev at codespeak.net and any sprint organizing/logistical
questions to pypy-sprint at codespeak.net
* if you need information about the conference, potential hotels,
directions etc we recommend to look at http://www.europython.org.
We are looking forward to meet you at the Vilnius Post EuroPython
The PyPy team
.. See also ..
.. _getting-started: http://codespeak.net/pypy/dist/pypy/doc/getting-started.html
.. _`pypy-sprint mailing list`: http://codespeak.net/mailman/listinfo/pypy-sprint
.. _`the technical reports available and other relevant documentation`: http://codespeak.net/pypy/dist/pypy/doc/index.html
.. _`EuroPython schedule`: http://indico.cern.ch/conferenceTimeTable.py?confId=13919&showDate=all&show…
My first post on this list so I hope this is the right place to post and
Im rewriting parts of the Blender3D python API that has got a bit old
and needs an update.
Im making a PyList subtype with the C/Python API, this involves
intercepting calls to standard list methods to make sure Blenders array
data is in Sync with the list's data.
Iv got it working for tp_as_sequence, tp_as_mapping, iter and dealloc
etc but methods are a problem.
I want to add my own call's before and after PyLists standard functions
but have a proplem with functons that use keywords and have no API
For example, I cant use the API's PyList_Sort because that dosnt support
ls.sort(key=lambda a: a.foo))
And the Problem with PyObject_CallMethod is that it dosnt accept keywords.
PyObject_CallMethod((PyObject *)mylist, "sort", "O", args);
Looking at abstract.c, PyObject_CallMethod uses call_function_tail,
which calls "PyObject_Call(callable, args, NULL);" - so Its not
currently possible with PyObject_CallMethod.
But I cant find any way to do this in a few lines.
I could use PyEval_CallObjectWithKeywords but that would mean Id need to
get the method from the list manually which Ill look into, but unless Im
missing something here, it seems PyObject_CallMethodWithKeywords would
be a nice addition to the Python API that cant be done in a straight
forward way at the moment.
I'm writing an article about the mailbox module for an online
publication, and would like to get comments on the current draft from
people familiar with the module.
If you'd like to take a look, please e-mail me and I'll tell you the