Mailman 3 June 2007 - Python-Dev

handling the granularity of possible Py3K warnings in 2.6
by Brett Cannon June 27, 2007

June 27, 2007

My rewrite of import is written for 2.6. But I am going to try to bootstrap it into 3.0. But I want to bootstrap into 2.6 if it works out well in 3.0. That means coding in 2.6 but constantly forward-porting to 3.0. In other words I am going to be a guinea pig of our transition plan. And being the guinea pig means I want to know where we want to take the Py3K warnings in 2.6. As of right now the -3 option causes some DeprecationWarnings to be raised (mostly about callable and dict.has_key).… [View More]

1 0

Proposal for a new function "open_noinherit" to avoid problems with subprocesses and security risks
by Henning von Bargen June 25, 2007

June 25, 2007

I'd like to propose a new function "open_noinherit" or maybe even a new mode flag "n" for the builtin "open" (see footnote for the names). The new function should work exactly like the builtin "open", with one difference: The open file is not inherited to any child processes (whereas files opened with "open" will be inherited). The new function can be implemented (basically) using os.O_NOINHERIT on MS Windows resp. fcntl / FD_CLOEXEC on Posix. I will post a working Python implementation … [View More]next week. There are five reasons for the proposal: 1) The builtin "open" causes unexpected problems in conjunction with subprocesses, in particular in multi-threaded programs. It can cause file permission errors in the subprocess or in the current process. On Microsoft Windows, some of the possible file permission errors are not documented by Microsoft (thus very few programs written for Windows will react properly). 2) Inheriting open file handles to subprocesses is a security risk. 3) For the developer, finding "cause and effect" is *very* hard, in particular in multi-threaded programs, when the errors occur only in race-conditions. 4) The problems arise in some of the standard library modules as well, i.e. shutil.filecopy. 5) Very few developers are aware of the possible problems. As a work-around, one can replace open with os.fdopen (os.open (..., + os.O_NOINHERIT), ... ) on Windows, but that's really ugly, hard to read, may raise a different exception than open (IOError instead of OSError), and needs careful work to take platform-specific code into account Here is a single-threaded example to demonstrate the effect: import os import subprocess outf = open ("blah.tmp", "wt") subprocess.Popen("notepad.exe") # or whatever program you like, but # It must be a program that does not exit immediately! # Now the subprocess has inherited the open file handle # We can still write: outf.write ("Hello world!\n") outf.close() # But we can not rename the file (at least on Windows) os.rename ("blah.tmp", "blah.txt") # this fails with OSError: [Errno 13] Permission denied # Similar problems with other file operations on non-Windows platforms. Ok, in this little program one can see what is going wrong easily. But what if the subprocess exits very quickly? Then perhaps you see the OSError, perhaps not - depending on the process scheduler of your operation system. In a commercial multi-theaded daemon application, the error only occured under heavy load and was hard to reproduce - and it was even harder to find the cause. That's because cause and effect were in two different threads in two completely different parts of the program: - Thread A opens a file and starts to write data - Thread B starts a subprocess (which inherits the file handle from thread A!) - Thread A continues writing to the file and closes it. - And now it's a race condition: - a) Thread A wants to rename the file - b) the subprocess exits. If a) is first: Error, if b) is first: no error. To make things more complicated, even two subprocesses can disturb each other. The new function should be implemented in C ideally, because the GIL could prevent a thread-switch between os.open and the fcntl.F_SETFD call. Note that the problem described here arises not only for files, but for sockets as well. See bug 1222790: SimpleXMLRPCServer does not set FD_CLOEXEC Once there is an easy-to-use, platform-independent, documented builtin "open_noinherit" (or a new mode flag for "open"), the standard library should be considered. For each occurence of "open" or "file", it should be considered if it necessary to inherit the file to subprocesses. If not, it should be replaced with open_noinherit. One example is shutil.filecopy, where open_noiherit should be used instead of open. The socket module is another candidate, I think - but I'm not sure about that. A nice effect of using "open_noinherit" is that - in many cases - one no longer needs to speficy close_fds = True when calling subprocess.Popen. [Note that close_fds is *terribly* slow if MAX_OPEN_FILES is "big", e.g. 800, see bug 1663329] Footnote: While writing this mail, at least 3 times I typed "nonherit" instead of "noinherit". So maybe someone can propose a better name? Or a new mode flag character could be "p" (like "private" or "protected"). Henning [View Less]

7 16

Issues with PEP 3101 (string formatting)
by Chris McDonough June 24, 2007

June 24, 2007

Wrt http://www.python.org/dev/peps/pep-3101/ PEP 3101 says Py3K should allow item and attribute access syntax within string templating expressions but "to limit potential security issues", access to underscore prefixed names within attribute/item access expressions will be disallowed. I am a person who has lived with the aftermath of a framework designed to prevent data access by restricting access to underscore- prefixed names (Zope 2, ahem), and I've found it's very hard to … [View More]

3 2

Re: [Python-Dev] [Python-3000] Issues with PEP 3101 (string formatting)
by Talin June 24, 2007

June 24, 2007

I haven't responded to this thread because I was hoping some of the original proponents of the feature would come out to defend it. (Remember, 3101 is a synthesis of a lot of people's ideas gleaned from many forum postings - In some cases I am willing to defend particular aspects of the PEP, and in others I just write down what I think the general consensus is.) That being said - from what I've read so far, the evidence on both sides of the argument seems anecdotal to me. I'd rather wait … [View More]and see what more people have to say on the topic. -- Talin Aurélien Campéas wrote: > On Tue, Jun 19, 2007 at 08:20:25AM -0700, Guido van Rossum wrote: >> Those are valid concerns. I'm cross-posting this to the python-3000 >> list in the hope that the PEP's author and defendents can respond. I'm >> sure we can work something out. > > Thanks to raise this. It is horrible enough that I feel obliged to > de-lurk. > > -10 on this part of PEP3101. > > >> Please keep further discussion on the python-3000(a)python.org list. >> >> --Guido >> >> On 6/19/07, Chris McDonough <chrism(a)plope.com> wrote: >>> Wrt http://www.python.org/dev/peps/pep-3101/ >>> >>> PEP 3101 says Py3K should allow item and attribute access syntax >>> within string templating expressions but "to limit potential security >>> issues", access to underscore prefixed names within attribute/item >>> access expressions will be disallowed. > > People talking about potential security issues should have an > obligation to show how their proposals *really* improve security (in > general); this is of course, a hard thing to do; mere hand-waving is > not sufficient. > >>> I am a person who has lived with the aftermath of a framework >>> designed to prevent data access by restricting access to underscore- >>> prefixed names (Zope 2, ahem), and I've found it's very hard to >>> explain and justify. As a result, I feel that this is a poor default >>> policy choice for a framework. > > And it's even poorer in the context of a language (for it's probably > harder to escape language-level restrictions than framework > obscurities ...). > >>> In some cases, underscore names must become part of an object's >>> external interface. Consider a URL with one or more underscore- >>> prefixed path segment elements (because prefixing a filename with an >>> underscore is a perfectly reasonable thing to do on a filesystem, and >>> path elements are often named after file names) fed to a traversal >>> algorithm that attempts to resolve each path element into an object >>> by calling __getitem__ against the parent found by the last path >>> element's traversal result. Perhaps this is poor design and >>> __getitem__ should not be consulted here, but I doubt that highly >>> because there's nothing particularly special about calling a method >>> named __getitem__ as opposed to some method named "traverse". > > This is trying to make a technical argument, but the 'consenting > adults' policy might be enough. In my experience, zope forbiding > access to _ prefixed attributes just led to work around the > limitation, thus adding more useless indirection to an already crufty > code base. The result is more obfuscation and probably even less > security (as in auditability of the code). > >>> The only precedent within Python 2 for this sort of behavior is >>> limiting access to variables that begin with __ and which do not end >>> with __ to the scope defined by a class and its instances. I >>> personally don't believe this is a very useful feature, but it's >>> still only an advisory policy and you can worm around it with enough >>> gyrations. > > FWIW I've come to never use __attrs. The obfuscation feature seems to > bring nothing but pain (the few times I've fell into that trap as a > beginner python programmer). > >>> Given that security is a concern at all, the only truly reasonable >>> way to "limit security issues" is to disallow item and attribute >>> access completely within the string templating expression syntax. It >>> seems gratuituous to me to encourage string templating expressions >>> with item/attribute access, given that you could do it within the >>> format arguments just as easily in the 99% case, and we've (well... >>> I've) happily been living with that restriction for years now. >>> >>> But if this syntax is preserved, there really should be no *default* >>> restrictions on the traversable names within an expression because >>> this will almost certainly become a hard-to-explain, hard-to-justify >>> bug magnet as it has become in Zope. > > I'd add that Zope in general looks to me like a giant collection of > python anti-patterns and as such can be used as a clue source about > what not to do, especially what not to include in Py3k. > > I don't want to offense people, well no more than necessary (imho zope > *is* an offense to common sense in many ways), but that's the opinion > from someone who earns its living mostly from zope/plone products > dev. and maintenance (these days, anyway). > > Regards, > Aurélien. > _______________________________________________ > Python-3000 mailing list > Python-3000(a)python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/talin%40acm.org > [View Less]

1 0

Summary of Tracker Issues
by Tracker June 24, 2007

June 24, 2007

ACTIVITY SUMMARY (06/17/07 - 06/24/07) Tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 1645 open ( +0) / 8584 closed ( +0) / 10229 total ( +0) Average duration of open issues: 836 days. Median duration of open issues: 784 days. Open Issues Breakdown open 1645 ( +0) pending 0 ( +0)

1 0

bzr on dinsdale
by "Martin v. Löwis" June 23, 2007

June 23, 2007

If I do "bzr status" in dinsdale:/etc/apache2, I get bzr: ERROR: bzrlib.errors.BzrCheckError: Internal check failed: file u'/etc/init.d/stop-bootlogd' entered as kind 'symlink' id 'stopbootlogd-20070303140018-fe340b888f6e9c69', now of kind 'file' Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/bzrlib/commands.py", line 611, in run_bzr_catch_errors return run_bzr(argv) ...... BzrCheckError: Internal check failed: file u'/etc/init.d/stop-bootlogd' entered as kind … [View More]

2 2

Weekly Python Patch/Bug Summary
by Kurt B. Kaiser June 23, 2007

June 23, 2007

Patch / Bug Summary ___________________ Patches : 385 open (+21) / 3790 closed (+21) / 4175 total (+42) Bugs : 1029 open (+43) / 6744 closed (+43) / 7773 total (+86) RFE : 262 open ( +4) / 291 closed ( +4) / 553 total ( +8) New / Reopened Patches ______________________ syslog syscall support for SysLogLogger (2007-05-02) http://python.org/sf/1711603 reopened by luke-jr syslog syscall support for SysLogLogger (2007-05-02) http://python.org/sf/1711603 … [View More]

1 0

Vilnius/Post EuroPython PyPy Sprint 12-14th of July
by Armin Rigo June 22, 2007

June 22, 2007

======================================================== Vilnius/Post EuroPython PyPy Sprint 12-14th of July ======================================================== The PyPy team is sprinting at EuroPython again and we invite you to participate in our 3 day long sprint at the conference hotel - Reval Hotel Lietuva. If you plan to attend the sprint we recommend you to listen to the PyPy technical talks (`EuroPython schedule`_) during the conference since it will give you a good overview of … [View More]the status of development. On the morning of the first sprint day (12th) we will also have a tutorial session for those new to PyPy development. As 3 days is relatively short for a PyPy sprint we suggest to travel back home on the 15th if possible (but it is ok to attend less than 3 days too). ------------------------------ Goals and topics of the sprint ------------------------------ There are many possible and interesting sprint topics to work on - here we list some possible task areas: * completing the missing python 2.5 features and support * write or port more extension modules (e.g. zlib is missing) * identify slow areas of PyPy through benchmarking and work on improvements, possibly moving app-level parts of the Python interpreter to interp-level if useful. * there are some parts of PyPy in need of refactoring, we may spend some time on those, for example: - rctypes and the extension compiler need some rethinking - support for LLVM 2.0 for the llvm backend - ... * some JIT improvement work * port the stackless transform to ootypesystem * other interesting stuff that you would like to work on ...;-) ------------ Registration ------------ If you'd like to come, please subscribe to the `pypy-sprint mailing list`_ and drop a note about your interests and post any questions. More organisational information will be sent to that list. Please register by adding yourself on the following list (via svn): http://codespeak.net/svn/pypy/extradoc/sprintinfo/post-ep2007/people.txt or on the pypy-sprint mailing list if you do not yet have check-in rights: http://codespeak.net/mailman/listinfo/pypy-sprint --------------------------------------- Preparation (if you feel it is needed): --------------------------------------- * read the `getting-started`_ pages on http://codespeak.net/pypy * for inspiration, overview and technical status you are welcome to read `the technical reports available and other relevant documentation`_ * please direct any technical and/or development oriented questions to pypy-dev at codespeak.net and any sprint organizing/logistical questions to pypy-sprint at codespeak.net * if you need information about the conference, potential hotels, directions etc we recommend to look at http://www.europython.org. We are looking forward to meet you at the Vilnius Post EuroPython PyPy sprint! The PyPy team .. See also .. .. _getting-started: http://codespeak.net/pypy/dist/pypy/doc/getting-started.html .. _`pypy-sprint mailing list`: http://codespeak.net/mailman/listinfo/pypy-sprint .. _`the technical reports available and other relevant documentation`: http://codespeak.net/pypy/dist/pypy/doc/index.html .. _`EuroPython schedule`: http://indico.cern.ch/conferenceTimeTable.py?confId=13919&showDate=all&show… [View Less]

1 0

Calling Methods from Pythons C API with Keywords
by Campbell Barton June 22, 2007

June 22, 2007

Hey Guys, My first post on this list so I hope this is the right place to post and relevant. Im rewriting parts of the Blender3D python API that has got a bit old and needs an update. Im making a PyList subtype with the C/Python API, this involves intercepting calls to standard list methods to make sure Blenders array data is in Sync with the list's data. Iv got it working for tp_as_sequence, tp_as_mapping, iter and dealloc etc but methods are a problem. I want to add my own call's before … [View More]

3 9

Wanted: readers for a mailbox.py article
by A.M. Kuchling June 21, 2007

June 21, 2007

I'm writing an article about the mailbox module for an online publication, and would like to get comments on the current draft from people familiar with the module. If you'd like to take a look, please e-mail me and I'll tell you the draft's URL. --amk

1 0