Mailman 3 July 2013 - Python-Dev

Inherance of file descriptor and handles on Windows (PEP 446)
by Victor Stinner 05 Aug '13

05 Aug '13

Hi, Guido van Rossum and others asked me details on how file descriptors and handles are inherited on Windows, for the PEP 446. http://www.python.org/dev/peps/pep-0446/ I hacked Python 3.4 to add a os.get_cloexec() function (extracted from my implementation of the PEP 433), here are some results. Python functions open(), os.open() and os.dup() create file descriptors with the HANDLE_FLAG_INHERIT flag set (cloexec=False), whereas os.pipe() creates 2 file descriptors with the HANDLE_FLAG_INHERIT flag unset (cloexec=False, see also issue #4708). Even if the HANDLE_FLAG_INHERIT flag is set, all handles are closed if subprocess is used with close_fds=True (which is the default value of the parameter), and all file descriptors are closed except 0, 1 and 2. If close_fds=False, handles with the HANDLE_FLAG_INHERIT flag set are inherited, but all file descriptors are still closed except 0, 1 and 2. (I didn't check if file descriptors 0, 1 and 2 are inherited, duplicated or new file descriptors.) The PEP 466 allows you to control which handles are inherited to child process when you use subprocess with close_fds=False. (The subprocess parameter should be called "close_handles" on Windows to avoid confusion.) Said differently: the HANDLE_FLAG_INHERIT flag only has an effect on *handles*, as indicated in its name. On Windows, file *descriptors* are never inherited (are always closed) in child processes. I don't think that it is possible to inherit file descriptors on Windows. By the way, using pass_fds on Windows raises an assertion error ("pass_fds not supported on Windows"). Another example in Python: --- import subprocess, sys code = """ import os, sys fd = int(sys.argv[1]) f = os.fdopen(fd, "rb") print(f.read()) """ f = open(__file__, "rb") fd = f.fileno() subprocess.call([sys.executable, "-c", code, str(fd)], close_fds=False) --- On Unix, the child process will write the script into stdout. On Windows, you just get an OSError(9, "Bad file descriptor") exception. To fix this example on Windows, you have to: * Retrieve the handle of the file using msvcrt.get_osfhandle() ; * Pass the handle, instead of the file descriptor, to the child ; * Create a file descriptor from the handle using msvcrt.open_osfhandle() in the child. The fix would be simpler if Python would provide the handle of a file object (ex: in a method) and if open() supported opening a handle as it does with file descriptors on UNIX. Victor

11 40

PEP 446: Open issues/questions
by Victor Stinner 02 Aug '13

02 Aug '13

Hi, I have a few more questions on the PEP 446: (A) How should we support support where os.set_inheritable() is not supported? Can we announce that os.set_inheritable() is always available or not? Does such platform exist? (B) Should subprocess make the file descriptors of pass_fds inheritable? If yes, should it be done before or after the fork? If it is done after the fork and before exec, it only affects the child process, at least on Linux (the file descriptor is still non-inheritable in the parent process). (C) Should we handle standard streams (0: stdin, 1: stdout, 2: stderr) differently? For example, os.dup2(fd, 0) should make the file descriptor 0 (stdin) inheritable or non-inheritable? On Windows, os.set_inheritable(fd, False) fails (error 87, invalid argument) on standard streams (0, 1, 2) and copies of the standard streams (created by os.dup()). Victor

4 28

PEP 442 aftermath: module globals at shutdown
by Antoine Pitrou 01 Aug '13

01 Aug '13

Hello, PEP 442 has now been committed in time for testing in Alpha 1. This paves the way for the removal of another well-known annoyance: the behaviour of module globals at shutdown. Now that reference cycles aren't a barrier to object finalization anymore, we shouldn't need to set module globals to None before trying to reclaim modules. (and then, we don't need to cache global functions for use in finalizers, either) I have a patch to suppress the hack in http://bugs.python.org/issue18214 Once I get to add some tests, I would like to commit it soon too! Regards Antoine.

4 6

C code coverage report with lcov
by Christian Heimes 31 Jul '13

31 Jul '13

Hi, I have done some experiments with GCC's gcov and lcov to get the C code coverage of our unit test suite. You may find today's report at http://tiran.bitbucket.org/python-lcov/ I'm working on a patch for our Makefile to include all steps in one simple make tag. http://bugs.python.org/issue18481 Christian

4 9

to rename or not...
by Ronald Oussoren 31 Jul '13

31 Jul '13

Hi, The relevant issue for this question is <http://bugs.python.org/issue14455> Plistlib contains object serialization logic that is in spirit simular to json and pickle, but has a different output format and different limitations. The functions in the module don't reflect that though, they are "readPlist" and "writePlist" instead of "load" and "dump". While working on the issue I noticed something uglier than that: plistlib in py3k inherits a design decision that was necessary in py2 and feels decidedly odd in py3k. It represents binary data objects of type plistlib.Data instead of bytes. Those objects have an attribute with the actual data. The distinction was necessary in Python 2 to make it possible to keep binary data and strings apart, but is no longer necessary in Python 3. Because of this I'd like to introduce a new API in plistlib that fixes both problems. In particular: * Add 'load', 'loads', 'dump' and 'dumps', those use "bytes" for binary data by default * Keep and deprecate "readPlist", "writePlist" and the their string equivalents, those still use Data objects (and call the new API to do the actual work). I'd like some feedback on this change. On the one hand the new APIs make it possible to clean up the API of plistlib, on the other hand this is a big API change. Ronald P.S. The issue itself is about adding support for binary plist files, I got a bit carried away while testing and refactoring the original patch :-(

4 4

Re: [Python-Dev] [Python-checkins] cpython (2.7): Issue #18441: Make test.support.requires('gui') skip when it should.
by R. David Murray 31 Jul '13

31 Jul '13

On Sun, 28 Jul 2013 01:09:43 +0200, terry.reedy <python-checkins(a)python.org> wrote: > http://hg.python.org/cpython/rev/dd9941f5fcda > changeset: 84870:dd9941f5fcda > branch: 2.7 > parent: 84865:c0788ee86a65 > user: Terry Jan Reedy <tjreedy(a)udel.edu> > date: Sun Jul 21 20:13:24 2013 -0400 > summary: > Issue #18441: Make test.support.requires('gui') skip when it should. > (Consolidating this check and various checks in tkinter files and moving them > to test.support and test.regrtest will be another issue.) > > files: > Lib/idlelib/idle_test/test_text.py | 5 +--- > Lib/test/test_idle.py | 20 ++++++++++++++--- > 2 files changed, 17 insertions(+), 8 deletions(-) > > > diff --git a/Lib/idlelib/idle_test/test_text.py b/Lib/idlelib/idle_test/test_text.py > --- a/Lib/idlelib/idle_test/test_text.py > +++ b/Lib/idlelib/idle_test/test_text.py > @@ -216,10 +216,7 @@ > requires('gui') > from Tkinter import Tk, Text > cls.Text = Text > - try: > - cls.root = Tk() > - except TclError as msg: > - raise unittest.SkipTest('TclError: %s' % msg) > + cls.root = Tk() > > @classmethod > def tearDownClass(cls): > diff --git a/Lib/test/test_idle.py b/Lib/test/test_idle.py > --- a/Lib/test/test_idle.py > +++ b/Lib/test/test_idle.py > @@ -1,9 +1,21 @@ > -# Skip test if _tkinter or _thread wasn't built or idlelib was deleted. > -from test.test_support import import_module > -import_module('Tkinter') > -import_module('threading') # imported by PyShell, imports _thread > +# Skip test if _thread or _tkinter wasn't built or idlelib was deleted. > +from test.test_support import import_module, use_resources > +import_module('threading') # imported by idlelib.PyShell, imports _thread > +tk = import_module('Tkinter') > idletest = import_module('idlelib.idle_test') > > +# If buildbot improperly sets gui resource (#18365, #18441), remove it > +# so requires('gui') tests are skipped while non-gui tests still run. > +if use_resources and 'gui' in use_resources: > + try: > + root = tk.Tk() > + root.destroy() > + except TclError: > + while True: > + use_resources.delete('gui') > + if 'gui' not in use_resources: > + break I believe that this logic is incorrect. If regrtest is run with "-u gui", given this code the skip message will be "requires gui resource"...but the caller specified the gui resource, which will make the skip message completely confusing. Instead, if it is true that 'gui' always means 'tk must work', then the _is_gui_available function should do the Tk() check. Currently as far as I can see it is indeed the case that requires('gui') always means tk must work. So, I believe that the correct fix is to move check_tk_availability to test.support, and call it from _is_gui_available. If we ever add another gui toolkit, we can deal with moving the tk check out into a separate 'tk' resource at that time. > + > # Without test_main present, regrtest.runtest_inner (line1219) calls > # unittest.TestLoader().loadTestsFromModule(this_module) which calls > # load_tests() if it finds it. (Unittest.main does the same.) > > -- > Repository URL: http://hg.python.org/cpython > _______________________________________________ > Python-checkins mailing list > Python-checkins(a)python.org > http://mail.python.org/mailman/listinfo/python-checkins

2 2

PyPy3 2.1 beta 1 released
by Philip Jenvey 30 Jul '13

30 Jul '13

================ PyPy3 2.1 beta 1 ================ We're pleased to announce the first beta of the upcoming 2.1 release of PyPy3. This is the first release of PyPy which targets Python 3 (3.2.3) compatibility. We would like to thank all of the people who donated_ to the `py3k proposal`_ for supporting the work that went into this and future releases. You can download the PyPy3 2.1 beta 1 release here: http://pypy.org/download.html#pypy3-2-1-beta-1 Highlights ========== * The first release of PyPy3: support for Python 3, targetting CPython 3.2.3! - There are some `known issues`_ including performance regressions (issues `#1540`_ & `#1541`_) slated to be resolved before the final release. What is PyPy? ============== PyPy is a very compliant Python interpreter, almost a drop-in replacement for CPython 2.7.3 or 3.2.3. It's fast due to its integrated tracing JIT compiler. This release supports x86 machines running Linux 32/64, Mac OS X 64 or Windows 32. Also this release supports ARM machines running Linux 32bit - anything with ``ARMv6`` (like the Raspberry Pi) or ``ARMv7`` (like Beagleboard, Chromebook, Cubieboard, etc.) that supports ``VFPv3`` should work. Cheers, the PyPy team

2 1

Adding Python scripts to PATHEXT on Windows
by Paul Moore 30 Jul '13

30 Jul '13

With the addition of the Python launcher (PEP 397), Python scripts on Windows are executable in much the same way as on Unix. However, "out of the box" it is still not quite possible to use Python scripts. This is because the .py extension is not registered with Windows in the PATHEXT environment variable (which holds all file extensions which should be looked for on PATH at the command line). There are two main consequences to this: 1. Under powershell, ".py" files on PATH are not found at all (this is not the case with cmd.exe, and may be technically a bug in powershell - but it's unlikely to change nevertheless). 2. Under both powershell and cmd, users have to explicitly include the extension when invoking the command. I would like to propose that the .py extension be added to PATHEXT as part of the "Register Python Extensions" option in the Python MSI installer. This would mean that Python users had a clean way of writing cross-platform "wrapper scripts" and command-line applications out of the box, just by writing a normal Python script with a #! line at the top. Does anyone have any objections to this? I could try to write a patch, but I know next to nothing about building MSIs, so if any of the installer experts could help that would be fantastic. Paul PS There would still be one difference, in that scripts intended to be used as commands would be named with '.py' on Windows and with no extension on Unix, but that's a pretty trivial difference, and reflects fundamental platform differences rather than configuration. So I don't propose worrying about this.

5 16

MyOpenID.com no longer supported
by Ethan Furman 30 Jul '13

30 Jul '13

Excerpt from http://meta.stackoverflow.com/q/190442/176681: Janrain no longer actively supports MyOpenID, and announced on Twitter that their users should proceed with caution. This decision was made by Janrain, [snip] I know the Python bug tracker allows MyOpenID logins; if that is your only login method you should add another that doesn't depend on MyOpenID. -- ~Ethan~

4 3

importing modules
by syed khalid 29 Jul '13

29 Jul '13

I am attempting to import modules from Shogun to python from a non-standard python directory ie from my /home/xxx directory. is there a way on ubuntu to selectively some modules, scripts, data from one directory and others modules, scripts from another directory. In other words, is there a file(s) that provide pointers to where these modules are located. regards Sy --

2 1