Mailman 3 August 2013 - Python-Dev

Our failure at handling GSoC students
by Antoine Pitrou 07 Aug '13

07 Aug '13

Hello, I would like to point out that we currently fail at handling GSoC projects and bringing them to completion. One cruel example is the set of PEP 3121 / PEP 384 refactorings done by Robin Schreiber: http://bugs.python.org/issue?%40columns=id%2Cactivity%2Ctitle%2Ccreator%2Ca… Robin has produced many patches that seem to reach the stated goal (refactor C extension modules to take advantage of the latest PEPs about module initialization and extension types definition). Unfortunately, tackling both goals at the same time produces big patches with a lot of churn; and it is also not obvious the PEP 384 refactoring is useful for the stdlib (while the PEP 3121 refactoring definitely is). What didn't produce an alarm during Robin's work is that GSoC work is done in private. Therefore, other core developers than the mentor don't get to give an advice early, as would happen with any normal proposal done publicly (on the mailing-list or on the bug tracker). It is also likely that the mentor gets overworked after the GSoC period is over, is unable to finalize the patch and push it, and other core devs have a hard time catching up on the work and don't know what the shortcomings are. Regards Antoine.

12 17

unittest.TestSuite holding references to unittest.TestCase instances too long
by Matt McClure 06 Aug '13

06 Aug '13

It seems unittest.TestSuite holds references to unittest.TestCase instances after the test runs, until the test suite finishes. In a large suite, where the TestCase instances consume memory during execution, that can lead to exhausting all available memory and the OS killing the test process. What do you think of a change like this? $ hg diff diff -r 3bd55ec317a7 Lib/unittest/suite.py --- a/Lib/unittest/suite.py Thu Aug 01 23:57:21 2013 +0200 +++ b/Lib/unittest/suite.py Fri Aug 02 07:42:22 2013 -0400 @@ -90,7 +90,12 @@ if getattr(result, '_testRunEntered', False) is False: result._testRunEntered = topLevel = True - for test in self: + while True: + try: + test = self._tests.pop(0) + except IndexError: + break + if result.shouldStop: break See also the conversation on django-developers[1] that led me here. [1]: https://groups.google.com/forum/#!topic/django-developers/XUMetDSGVT0 -- Matt McClure http://matthewlmcclure.com http://www.mapmyfitness.com/profile/matthewlmcclure

4 12

Make extension module initialisation more like Python module initialisation
by Stefan Behnel 06 Aug '13

06 Aug '13

Hi, I suspect that this will be put into a proper PEP at some point, but I'd like to bring this up for discussion first. This came out of issues 13429 and 16392. http://bugs.python.org/issue13429 http://bugs.python.org/issue16392 Stefan The problem =========== Python modules and extension modules are not being set up in the same way. For Python modules, the module is created and set up first, then the module code is being executed. For extensions, i.e. shared libraries, the module init function is executed straight away and does both the creation and initialisation. This means that it knows neither the __file__ it is being loaded from nor its package (i.e. its FQMN). This hinders relative imports and resource loading. In Py3, it's also not being added to sys.modules, which means that a (potentially transitive) re-import of the module will really try to reimport it and thus run into an infinite loop when it executes the module init function again. And without the FQMN, it's not trivial to correctly add the module to sys.modules either. We specifically run into this for Cython generated modules, for which it's not uncommon that the module init code has the same level of complexity as that of any 'regular' Python module. Also, the lack of a FQMN and correct file path hinders the compilation of __init__.py modules, i.e. packages, especially when relative imports are being used at module init time. The proposal ============ I propose to split the extension module initialisation into two steps in Python 3.4, in a backwards compatible way. Step 1: The current module init function can be reduced to just creating the module instance and returning it (and potentially doing some simple C level setup). Optionally, after creating the module (and this is the new part), the module init code can register a C callback function that will be called after setting up the module. Step 2: The shared library importer receives the module instance from the module init function, adds __file__, __path__, __package__ and friends to the module dict, and then checks for the callback. If non-NULL, it calls it to continue the module initialisation by user code. The callback ============ The callback is defined as follows:: int (*PyModule_init_callback)(PyObject* the_module, PyModuleInitContext* context) "PyModuleInitContext" is a struct that is meant mostly for making the callback more future proof by allowing additional parameters to be passed in. For now, I can see a use case for the following fields:: struct PyModuleInitContext { char* module_name; char* qualified_module_name; } Both names are encoded in UTF-8. As for the file path, I consider it best to retrieve it from the module's __file__ attribute as a Python string object to reduce filename encoding problems. Note that this struct argument is not strictly required, but given that this proposal would have been much simpler if the module init function had accepted such an argument in the first place, I consider it a good idea not to let this chance pass by again. The registration of the callback uses a new C-API function: int PyModule_SetInitFunction(PyObject* module, PyModule_init_callback callback) The function name uses "Set" instead of "Register" to make it clear that there is only one such function per module. An alternative would be a new module creation function "PyModule_Create3()" that takes the callback as third argument, in addition to what "PyModule_Create2()" accepts. This would require users to explicitly pass in the (second) version argument, which might be considered only a minor issue. Implementation ============== The implementation requires local changes to the extension module importer and a new C-API function. In order to store the callback, it should use a new field in the module object struct. Open questions ============== It is not clear how extensions should be handled that register more than one module in their module init function, e.g. compiled packages. One possibility would be to leave the setup to the user, who would have to know all FQMNs anyway in this case, although not the import file path. Alternatively, the import machinery could use a stack to remember for which modules a callback was registered during the last init function call, set up all of them and then call their callbacks. It's not clear if this meets the intention of the user. Alternatives ============ 1) It would be possible to make extension modules optionally export another symbol, e.g. "PyInit2_modulename", that the shared library loader would call in addition to the required function "PyInit_modulename". This would remove the need for a new API that registers the above callback. The drawback is that it also makes it easier to write broken code because a Python version or implementation that does not support this second symbol would simply not call it, without error. The new C-API function would let the build fail instead if it is not supported. 2) The callback could be made available as a Python function in the module dict, thus also removing the need for an explicit registration API. However, this approach would add overhead to both sides, the importer code and the user provided module init code, as it would require additional dictionary handling and the implementation of a one-time Python function in user code. It would also suffer from the problem that missing support in the runtime would pass silently. 3) The callback could be registered statically in the PyModuleDef struct by adding a new field. This is not trivial to do in a backwards compatible way because the struct would grow longer without explicit initialisation by existing user code. Extending PyModuleDef_HEAD_INIT might be possible but would still break at least binary compatibility. 4) Pass a new context argument into the module init function that contains all information necessary to properly and completely set up the module at creation time. This would provide a much simpler and cleaner solution than the proposed solution. However, it will not be possible before Python 4 as it breaks backwards compatibility with all existing extension modules at both the source and binary level.

5 15

Inherance of file descriptor and handles on Windows (PEP 446)
by Victor Stinner 05 Aug '13

05 Aug '13

Hi, Guido van Rossum and others asked me details on how file descriptors and handles are inherited on Windows, for the PEP 446. http://www.python.org/dev/peps/pep-0446/ I hacked Python 3.4 to add a os.get_cloexec() function (extracted from my implementation of the PEP 433), here are some results. Python functions open(), os.open() and os.dup() create file descriptors with the HANDLE_FLAG_INHERIT flag set (cloexec=False), whereas os.pipe() creates 2 file descriptors with the HANDLE_FLAG_INHERIT flag unset (cloexec=False, see also issue #4708). Even if the HANDLE_FLAG_INHERIT flag is set, all handles are closed if subprocess is used with close_fds=True (which is the default value of the parameter), and all file descriptors are closed except 0, 1 and 2. If close_fds=False, handles with the HANDLE_FLAG_INHERIT flag set are inherited, but all file descriptors are still closed except 0, 1 and 2. (I didn't check if file descriptors 0, 1 and 2 are inherited, duplicated or new file descriptors.) The PEP 466 allows you to control which handles are inherited to child process when you use subprocess with close_fds=False. (The subprocess parameter should be called "close_handles" on Windows to avoid confusion.) Said differently: the HANDLE_FLAG_INHERIT flag only has an effect on *handles*, as indicated in its name. On Windows, file *descriptors* are never inherited (are always closed) in child processes. I don't think that it is possible to inherit file descriptors on Windows. By the way, using pass_fds on Windows raises an assertion error ("pass_fds not supported on Windows"). Another example in Python: --- import subprocess, sys code = """ import os, sys fd = int(sys.argv[1]) f = os.fdopen(fd, "rb") print(f.read()) """ f = open(__file__, "rb") fd = f.fileno() subprocess.call([sys.executable, "-c", code, str(fd)], close_fds=False) --- On Unix, the child process will write the script into stdout. On Windows, you just get an OSError(9, "Bad file descriptor") exception. To fix this example on Windows, you have to: * Retrieve the handle of the file using msvcrt.get_osfhandle() ; * Pass the handle, instead of the file descriptor, to the child ; * Create a file descriptor from the handle using msvcrt.open_osfhandle() in the child. The fix would be simpler if Python would provide the handle of a file object (ex: in a method) and if open() supported opening a handle as it does with file descriptors on UNIX. Victor

11 40

Allowing to run certain regression tests in subprocesses
by Eli Bendersky 05 Aug '13

05 Aug '13

Hi All, Today the issue of cross-test global env dependencies showed its ugly head again for me. I recall a previous discussion (http://mail.python.org/pipermail/python-dev/2013-January/123409.html) but there were many more over the years. The core problem is that some tests modify the global env (particularly importing modules) and this sometimes has adverse effects on other tests, because test.regrtest runs all tests in a single process. In the discussion linked above, the particular culprit test__all__ was judged as a candidate to be moved to a subprocess. I want to propose adding a capability to our test harness to run specific tests in subprocesses. Each test will have some simple way of asking to be run in a subprocess, and regrtest will concur (even when running -j1). test__all__ can go there, and it can help solve other problems. My particular case is trying to write a test for http://bugs.python.org/issue14988 - wherein I have to simulate a situation of non-existent pyexpat. It's not hard to write a test for it, but when run in tandem with other tests (where C extensions loaded pyexpat) it becomes seemingly impossible to set up. This should not be the case - there's nothing wrong with wanting to simulate this case, and there's nothing wrong in Python and the stdlib - it's purely an artifact of the way our regression suite works. Thoughts? Eli

3 8

[RELEASED] Python 3.4.0a1
by Larry Hastings 04 Aug '13

04 Aug '13

On behalf of the Python development team, I'm pleased to announce the first alpha release of Python 3.4. This is a preview release, and its use is not recommended for production settings. Python 3.4 includes a range of improvements of the 3.x series, including hundreds of small improvements and bug fixes. Major new features and changes in the 3.4 release series so far include: * PEP 435, a standardized "enum" module * PEP 442, improved semantics for object finalization * PEP 443, adding single-dispatch generic functions to the standard library * PEP 445, a new C API for implementing custom memory allocators To download Python 3.4.0a1 visit: http://www.python.org/download/releases/3.4.0/ Please consider trying Python 3.4.0a1 with your code and reporting any issues you notice to: http://bugs.python.org/ Enjoy! -- Larry Hastings, Release Manager larry at hastings.org (on behalf of the entire python-dev team and 3.4's contributors)

2 3

Re: [Python-Dev] Allowing to run certain regression tests in subprocesses
by R. David Murray 04 Aug '13

04 Aug '13

On Sat, 03 Aug 2013 19:04:21 -0700, Eli Bendersky <eliben(a)gmail.com> wrote: > On Sat, Aug 3, 2013 at 6:57 PM, R. David Murray <rdmurray(a)bitdance.com> wrote: > > On Sat, 03 Aug 2013 16:47:37 -0700, Eli Bendersky <eliben(a)gmail.com> wrote: > >> On Sat, Aug 3, 2013 at 4:36 PM, Eli Bendersky <eliben(a)gmail.com> wrote: > >> > Hi All, > >> > > >> > Today the issue of cross-test global env dependencies showed its ugly > >> > head again for me. I recall a previous discussion > >> > (http://mail.python.org/pipermail/python-dev/2013-January/123409.html) > >> > but there were many more over the years. > >> > > >> > The core problem is that some tests modify the global env > >> > (particularly importing modules) and this sometimes has adverse > >> > effects on other tests, because test.regrtest runs all tests in a > >> > single process. In the discussion linked above, the particular culprit > >> > test__all__ was judged as a candidate to be moved to a subprocess. > >> > > >> > I want to propose adding a capability to our test harness to run > >> > specific tests in subprocesses. Each test will have some simple way of > >> > asking to be run in a subprocess, and regrtest will concur (even when > >> > running -j1). test__all__ can go there, and it can help solve other > >> > problems. > >> > > >> > My particular case is trying to write a test for > >> > http://bugs.python.org/issue14988 - wherein I have to simulate a > >> > situation of non-existent pyexpat. It's not hard to write a test for > >> > it, but when run in tandem with other tests (where C extensions loaded > >> > pyexpat) it becomes seemingly impossible to set up. This should not be > >> > the case - there's nothing wrong with wanting to simulate this case, > >> > and there's nothing wrong in Python and the stdlib - it's purely an > >> > artifact of the way our regression suite works. > >> > > >> > Thoughts? > >> > > >> > Eli > >> > >> FWIW the problem is also discussed here: > >> http://bugs.python.org/issue1674555, w.r.t. test_site > > > > Can't you just launch a subprocess from the test itself using script_helpers? > > > > I can, but such launching will be necessarily duplicated across all > tests that need this functionality (test_site, test___all__, etc). > Since regrtest already has functionality for launching whole > test-suites in subprocesses, it makes sense to reuse it, no? In the case of test_site and test___all___ we are talking about running the entire test file in a subprocess. It sounds like you are only talking about running one individual test function in a subprocess, for which using script_helpers seems the more natural solution. --David

2 1

PEP 8 modernisation
by Nick Coghlan 04 Aug '13

04 Aug '13

With feedback from Guido, Barry, Raymond and others, I have updated PEP 8 to better describe our current development practices. It started as an update to describe the different between public and internal interfaces and to advise against using wildcard imports, but became substantially more :) For those that want full details, the relevant commit and tracker issue are here: http://hg.python.org/peps/rev/fb24c80e9afb http://bugs.python.org/issue18472 If you're responsible for a coding standard that includes PEP 8 by reference, you probably want to take a look at these :) For everyone else, here are the highlights: 1. Made it clear this is a living document (using the approach of creating a tracker issue for major updates and adding a new footnote referencing that issue) 2. Added more specific points to the "foolish consistency" section to help out those folks resisting pointless PEP 8 compliance for code that predates the existence of the PEP's recommendations. 3. Stopped being wishy-washy about tabs vs spaces. Use spaces :) 4. Lines up to 99 characters are now permitted (but 79 is still the preferred limit) 5. The encodings section is now emphatically in favour of UTF-8 (latin-1 is no longer even mentioned) 6. While absolute imports are still favoured, explicit relative imports are deemed acceptable 7. Wildcard imports are strongly discouraged for most cases (with dynamic republishing the only acceptable use case, since PEP 8 doesn't apply at all for the interactive prompt) 8. New section explaining the distinction between public and internal interfaces (and how to tell which is which) 9. Explicit guideline not to assign lambdas to names (use def, that's what it's for) 10. Various tweaks to the exception raising and handling guidelines 11. Explicit recommendation to use a decorator in conjunction with annotations in third party experiments Cheers, Nick. P.S. It's possible this should also be published through python-announce and other channels... -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

18 42

Fwd: Allowing to run certain regression tests in subprocesses
by Eli Bendersky 04 Aug '13

04 Aug '13

On Sat, Aug 3, 2013 at 6:57 PM, R. David Murray <rdmurray(a)bitdance.com> wrote: > On Sat, 03 Aug 2013 16:47:37 -0700, Eli Bendersky <eliben(a)gmail.com> wrote: >> On Sat, Aug 3, 2013 at 4:36 PM, Eli Bendersky <eliben(a)gmail.com> wrote: >> > Hi All, >> > >> > Today the issue of cross-test global env dependencies showed its ugly >> > head again for me. I recall a previous discussion >> > (http://mail.python.org/pipermail/python-dev/2013-January/123409.html) >> > but there were many more over the years. >> > >> > The core problem is that some tests modify the global env >> > (particularly importing modules) and this sometimes has adverse >> > effects on other tests, because test.regrtest runs all tests in a >> > single process. In the discussion linked above, the particular culprit >> > test__all__ was judged as a candidate to be moved to a subprocess. >> > >> > I want to propose adding a capability to our test harness to run >> > specific tests in subprocesses. Each test will have some simple way of >> > asking to be run in a subprocess, and regrtest will concur (even when >> > running -j1). test__all__ can go there, and it can help solve other >> > problems. >> > >> > My particular case is trying to write a test for >> > http://bugs.python.org/issue14988 - wherein I have to simulate a >> > situation of non-existent pyexpat. It's not hard to write a test for >> > it, but when run in tandem with other tests (where C extensions loaded >> > pyexpat) it becomes seemingly impossible to set up. This should not be >> > the case - there's nothing wrong with wanting to simulate this case, >> > and there's nothing wrong in Python and the stdlib - it's purely an >> > artifact of the way our regression suite works. >> > >> > Thoughts? >> > >> > Eli >> >> FWIW the problem is also discussed here: >> http://bugs.python.org/issue1674555, w.r.t. test_site > > Can't you just launch a subprocess from the test itself using script_helpers? > [sorry, sent privately by mistake; forwarding to pydev] I can, but such launching will be necessarily duplicated across all tests that need this functionality (test_site, test___all__, etc). Since regrtest already has functionality for launching whole test-suites in subprocesses, it makes sense to reuse it, no? Eli

1 0

objections to renaming enumobject.h/c in 3.4?
by Eli Bendersky 03 Aug '13

03 Aug '13

Hi all, I was looking around the Objects directory and noticed that we have enumobject.h/c with the enumobject structure for "enumerate" and "reversed". This is somewhat confusing now with Lib/enum.py and will be doubly confusing if we ever decide to have a C implementation of enums. Any objections to renaming the files and the internal structure & static functions with s/enum/enumerate/ ? This would more accurately reflect the use of the code, and avoid confusion with enums. These structures/types are not part of the stable ABI defined by PEP 384. Eli

4 6

Python-Dev August 2013