Python-Dev January 2011

python-dev@python.org

100 participants
109 discussions

by Guido van Rossum

I've received some enthusiastic emails from someone who wants to revive restricted mode. He started out with a bunch of patches to the CPython runtime using ctypes, which he attached to an App Engine bug: http://code.google.com/p/googleappengine/issues/detail?id=671 Based on his code (the file secure.py is all you need, included in secure.tar.gz) it seems he believes the only security leaks are __subclasses__, gi_frame and gi_code. (I have since convinced him that if we add "restricted" guards to these attributes, he doesn't need the functions added to sys.) I don't recall the exploits that Samuele once posted that caused the death of rexec.py -- does anyone recall, or have a pointer to the threads? -- --Guido van Rossum (home page: http://www.python.org/~guido/)

5 years, 7 months

PEP 3145 (With Contents)

by Eric Pruitt

Alright, I will re-submit with the contents pasted. I never use double backquotes as I think them rather ugly; that is the work of an editor or some automated program in the chain. Plus, it also messed up my line formatting and now I have lines with one word on them... Anyway, the contents of PEP 3145: PEP: 3145 Title: Asynchronous I/O For subprocess.Popen Author: (James) Eric Pruitt, Charles R. McCreary, Josiah Carlson Type: Standards Track Content-Type: text/plain Created: 04-Aug-2009 Python-Version: 3.2 Abstract: In its present form, the subprocess.Popen implementation is prone to dead-locking and blocking of the parent Python script while waiting on data from the child process. Motivation: A search for "python asynchronous subprocess" will turn up numerous accounts of people wanting to execute a child process and communicate with it from time to time reading only the data that is available instead of blocking to wait for the program to produce data [1] [2] [3]. The current behavior of the subprocess module is that when a user sends or receives data via the stdin, stderr and stdout file objects, dead locks are common and documented [4] [5]. While communicate can be used to alleviate some of the buffering issues, it will still cause the parent process to block while attempting to read data when none is available to be read from the child process. Rationale: There is a documented need for asynchronous, non-blocking functionality in subprocess.Popen [6] [7] [2] [3]. Inclusion of the code would improve the utility of the Python standard library that can be used on Unix based and Windows builds of Python. Practically every I/O object in Python has a file-like wrapper of some sort. Sockets already act as such and for strings there is StringIO. Popen can be made to act like a file by simply using the methods attached the the subprocess.Popen.stderr, stdout and stdin file-like objects. But when using the read and write methods of those options, you do not have the benefit of asynchronous I/O. In the proposed solution the wrapper wraps the asynchronous methods to mimic a file object. Reference Implementation: I have been maintaining a Google Code repository that contains all of my changes including tests and documentation [9] as well as blog detailing the problems I have come across in the development process [10]. I have been working on implementing non-blocking asynchronous I/O in the subprocess.Popen module as well as a wrapper class for subprocess.Popen that makes it so that an executed process can take the place of a file by duplicating all of the methods and attributes that file objects have. There are two base functions that have been added to the subprocess.Popen class: Popen.send and Popen._recv, each with two separate implementations, one for Windows and one for Unix based systems. The Windows implementation uses ctypes to access the functions needed to control pipes in the kernel 32 DLL in an asynchronous manner. On Unix based systems, the Python interface for file control serves the same purpose. The different implementations of Popen.send and Popen._recv have identical arguments to make code that uses these functions work across multiple platforms. When calling the Popen._recv function, it requires the pipe name be passed as an argument so there exists the Popen.recv function that passes selects stdout as the pipe for Popen._recv by default. Popen.recv_err selects stderr as the pipe by default. "Popen.recv" and "Popen.recv_err" are much easier to read and understand than "Popen._recv('stdout' ..." and "Popen._recv('stderr' ..." respectively. Since the Popen._recv function does not wait on data to be produced before returning a value, it may return empty bytes. Popen.asyncread handles this issue by returning all data read over a given time interval. The ProcessIOWrapper class uses the asyncread and asyncwrite functions to allow a process to act like a file so that there are no blocking issues that can arise from using the stdout and stdin file objects produced from a subprocess.Popen call. References: [1] [ python-Feature Requests-1191964 ] asynchronous Subprocess http://mail.python.org/pipermail/python-bugs-list/2006-December/ 036524.html [2] Daily Life in an Ivory Basement : /feb-07/problems-with-subprocess http://ivory.idyll.org/blog/feb-07/problems-with-subprocess [3] How can I run an external command asynchronously from Python? - Stack Overflow http://stackoverflow.com/questions/636561/how-can-i-run-an-external- command-asynchronously-from-python [4] 18.1. subprocess - Subprocess management - Python v2.6.2 documentation http://docs.python.org/library/subprocess.html#subprocess.Popen.wait [5] 18.1. subprocess - Subprocess management - Python v2.6.2 documentation http://docs.python.org/library/subprocess.html#subprocess.Popen.kill [6] Issue 1191964: asynchronous Subprocess - Python tracker http://bugs.python.org/issue1191964 [7] Module to allow Asynchronous subprocess use on Windows and Posix platforms - ActiveState Code http://code.activestate.com/recipes/440554/ [8] subprocess.rst - subprocdev - Project Hosting on Google Code http://code.google.com/p/subprocdev/source/browse/doc/subprocess.rst?spec=s… [9] subprocdev - Project Hosting on Google Code http://code.google.com/p/subprocdev [10] Python Subprocess Dev http://subdev.blogspot.com/ Copyright: This P.E.P. is licensed under the Open Publication License; http://www.opencontent.org/openpub/. On Tue, Sep 8, 2009 at 22:56, Benjamin Peterson <benjamin(a)python.org> wrote: > 2009/9/7 Eric Pruitt <eric.pruitt(a)gmail.com>: >> Hello all, >> >> I have been working on adding asynchronous I/O to the Python >> subprocess module as part of my Google Summer of Code project. Now >> that I have finished documenting and pruning the code, I present PEP >> 3145 for its inclusion into the Python core code. Any and all feedback >> on the PEP (http://www.python.org/dev/peps/pep-3145/) is appreciated. > > Hi Eric, > One of the reasons you're not getting many response is that you've not > pasted the contents of the PEP in this message. That makes it really > easy for people to comment on various sections. > > BTW, it seems like you were trying to use reST formatting with the > text PEP layout. Double backquotes only mean something in reST. > > > -- > Regards, > Benjamin >

7 years, 3 months

Solaris family and 64 bits compiling

by Jesus Cea

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 A Solaris installation contains ALWAYS 32 and 64 bits libraries. So in any Solaris you can run 32/64 bits programs, and compile in 32 and 64 bits. For this, libraries are stores in "/usr/lib", for instance, for 32 bits, while the same 64 bits libraries are stored in "/usr/lib/64". Currently, python do not considerate this. We have Solaris 10 buildslaves, but they compile in 32 bits, aparently. For instance <http://www.python.org/dev/buildbot/all/builders/sparc%20solaris10%20gcc%203…>. We now have 32 and 64 bits OpenIndiana buildslaves, so we can actually check this. They were deployed yesterday. Apparently the changes would be pretty simple, adding ".../64" to library paths, to try to find the extra libraries. What do you think?. - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea(a)jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBTOq3yZlgi5GaxT1NAQLQhAP9G2liX+YveYmfYDOuVjWWS8PE7r2wM/XA 5rik9mJM4Z7/wDnY4wrWjG5l3B9sSyrhhNI1YmIcXm4klfYxV9xTkG9dMNL+2bVc +s98rlTdjNlMVTf8Xc7U3tMpdkG/JK0+XWmRfWsf52ATdtxPHazI9L6KvqdYjNuZ 2w3dXNXErZE= =oYXo -----END PGP SIGNATURE-----

9 years

Question about GDB bindings and 32/64 bits

by Jesus Cea

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I have installed GDB 7.2 32 bits and 32 bits buildslaves are green. Nevertheless 64 bits buildslaves are failing test_gdb. Is there any expectation that a 32 bits GDB be able to debug a 64 bits python?. If not, gdb test should compare "platform.architecture()" (for python and gdb in the system) and run only when they are the same. If this should work, I would open a bug and maybe spend some time with it. But before thinking about investing time, I would like to know if this mix is actually expected or not to work. If not, I would consider to install a 64 bits GDB too and do some tricks (like using an "/usr/local/bin/gdb" script wrapper to choose 32/64 "real" gdb version) to actually execute "test_gdb" in both buildslaves (they are running in the same physical machine). Any advice? PS: I am talking about AMD64 OpenIndiana buildbots. Haven't check others. - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea(a)jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBTO8zjJlgi5GaxT1NAQLusgP9GVuhvQJWhPqjzdkZnrMObQg0AD6ggbIR 2B4IstFpD1bKvIcGPJv0Irk3+heaQuFbTzYVLC132d89Ektfib9ZbJ/hzJz2wqd2 lnkfNUCV0tKal3P7kbGYUk828glIrlufSuF1HYIknd2BAzHFl5Zf6q5/AXzYr90D v4Y82b7Wg0k= =NHcR -----END PGP SIGNATURE-----

9 years

Finally fix installer to add Python to %PATH% on Windows

by anatoly techtonik

Hi, I'd like to You probably know that after installation on Windows system it is possible to call Python from Explorer's Run dialog (Win-R). It is because Python path is added to App Paths registry key and Windows Explorer shell checks this key when looking for executable. But Python doesn't work from cmd session and, more importantly, *Python doesn't work from .bat files*. It is because cmd shell doesn't know about App Paths and relies on system PATH to find executable. As far as I remember, there is no function in Python stdlib either, that executes processes and does lookups in App Paths. I never paid much attention to this fact, because I put several .bat files for every 25, 26, 27, 31 and 32 version of Python into PATH manually. But when bootstrap script for build environment of Native Client (NaCl) said that I have no Python available and started to install its own, I've asked myself - "How come? There are 5! possible versions of Python on my system." It appeared that the following .bat file doesn't work: ---cut mypy.bat-- python.exe ---cut mypy.bat-- C:\>mypy.bat C:\>python.exe 'python.exe' is not recognized as an internal or external command, operable program or batch file. I've seen about 7 requests to add Python into %PATH% in installer. All closed with no result, but with some fear and uncertainty. Martin feared that MSI installers are not able to remove entry from PATH and even if they can, they may kill the whole PATH instead of removing just one entry. To prove or dispel these fears, I've just installed/uninstalled Mercurial from mercurial-1.7.3-1-x86.msi and App Engine from GoogleAppEngine-1.4.1.msi several times. Both add entries to PATH and both remove them without any further problems. Should we finally add this to 3.2 installer for Python? -- anatoly t.

9 years, 1 month

http://www.pythonlabs.com/logos.html is gone

by Georg Brandl

Which I noticed since it's cited in the BeOpen license we still refer to in LICENSE. Since pythonlabs.com itself is still up, it probably isn't much work to make the logos.html URI work again, but I don't know who maintains that page. cheer, Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out.

9 years, 1 month

Support for async read/write

by Jesus Cea

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Current Python lacks support for "aio_*" syscalls to do async IO. I think this could be a nice addition for python 3.3. If you agree, I will create an issue in the tracker. If you think the idea is of no value, please say so for me to move on. Maybe an 3th party module, but I think this functionality sould be available in core python. Thanks!. PS: The function calls are: aio_cancel, aio_error, aio_fsync, aio_read, aio_return, aio_write. - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea(a)jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBTL3MUplgi5GaxT1NAQKv5QQAnDan88kXJ67fucz2rT/mZze+065lm9E4 +XJ2JfqGMVE1/qMsXwg81l19RHSYReBgBjd7zyXWE9Fk/1Rfq4gjOZEtoG0IpGZG E3CH+Ab5/o/PjJZNKQaPpe0cwGSXFPKkPpgepKS1d8ZXyf6VXMc8UTSWjMI5jIVe 4M+yvw5m4I0= =nsdO -----END PGP SIGNATURE-----

9 years, 2 months

Issue #11051: system calls per import

by Victor Stinner

Hi, Antoine Pitrou noticed that Python 3.2 tries a lot of filenames to load a module: http://bugs.python.org/issue11051 Python 3.1 does already test many filenames, but with Python 3.2, it is even worse. For each directory in sys.path, it tries 9 suffixes: '', '.cpython-32m.so', 'module.cpython-32m.so', '.abi3.so', 'module.abi3.so', '.so', 'module.so', '.py', '.pyc'. I don't understand why it tests so much .so suffixes. And why it does test with and without "module". Victor

9 years, 2 months

Moving stuff out of Misc and over to the devguide

by Brett Cannon

There is a bunch of stuff in Misc that probably belongs in the devguide (under Resources) instead of in svn. Here are the files I think can be moved (in order of how strongly I think they should be moved): PURIFY.README README.coverty README.klocwork README.valgrind Porting developers.txt maintainers.rst SpecialBuilds.txt Now before anyone yells "that is inconvenient", don't forget that all core developers can check out and edit the devguide, and that almost all of the files listed (SpecialBuilds.txt is the exception) are typically edited and viewed on their own. Anyway, if there is a file listed here you don't think should move out of py3k and into the devguide, speak up.

9 years, 2 months

Re: [Python-Dev] MSI: Remove dependency from win32com.client module (issue4080047)

by techtonik＠gmail.com

There is no b.p.o issue as it's not a bug, but a tiny copy/paste patch to clean up the code a bit while I am trying to understand how to add Python to the PATH. I see no reason for b.p.o bureaucracy. Mercurial-style workflow [1] is more beneficial to development as it doesn't require switching from console to browser for submitting changes. This way tiny changes can be integrated/updated more rapidly. 1. http://mercurial.selenic.com/wiki/ContributingChanges#The_basics:_patches_b… http://codereview.appspot.com/4080047/

9 years, 2 months

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

Python-Dev January 2011