Mailman 3 August 2013 - Python-Dev

Our failure at handling GSoC students
by Antoine Pitrou Aug. 7, 2013

Aug. 7, 2013

Hello, I would like to point out that we currently fail at handling GSoC projects and bringing them to completion. One cruel example is the set of PEP 3121 / PEP 384 refactorings done by Robin Schreiber: http://bugs.python.org/issue?%40columns=id%2Cactivity%2Ctitle%2Ccreator%2Ca… Robin has produced many patches that seem to reach the stated goal (refactor C extension modules to take advantage of the latest PEPs about module initialization and extension types definition). Unfortunately, … [View More]

12 17

unittest.TestSuite holding references to unittest.TestCase instances too long
by Matt McClure Aug. 6, 2013

Aug. 6, 2013

It seems unittest.TestSuite holds references to unittest.TestCase instances after the test runs, until the test suite finishes. In a large suite, where the TestCase instances consume memory during execution, that can lead to exhausting all available memory and the OS killing the test process. What do you think of a change like this? $ hg diff diff -r 3bd55ec317a7 Lib/unittest/suite.py --- a/Lib/unittest/suite.py Thu Aug 01 23:57:21 2013 +0200 +++ b/Lib/unittest/suite.py Fri Aug 02 07:42:22 … [View More]

4 12

Make extension module initialisation more like Python module initialisation
by Stefan Behnel Aug. 6, 2013

Aug. 6, 2013

Hi, I suspect that this will be put into a proper PEP at some point, but I'd like to bring this up for discussion first. This came out of issues 13429 and 16392. http://bugs.python.org/issue13429 http://bugs.python.org/issue16392 Stefan The problem =========== Python modules and extension modules are not being set up in the same way. For Python modules, the module is created and set up first, then the module code is being executed. For extensions, i.e. shared libraries, the module init … [View More]function is executed straight away and does both the creation and initialisation. This means that it knows neither the __file__ it is being loaded from nor its package (i.e. its FQMN). This hinders relative imports and resource loading. In Py3, it's also not being added to sys.modules, which means that a (potentially transitive) re-import of the module will really try to reimport it and thus run into an infinite loop when it executes the module init function again. And without the FQMN, it's not trivial to correctly add the module to sys.modules either. We specifically run into this for Cython generated modules, for which it's not uncommon that the module init code has the same level of complexity as that of any 'regular' Python module. Also, the lack of a FQMN and correct file path hinders the compilation of __init__.py modules, i.e. packages, especially when relative imports are being used at module init time. The proposal ============ I propose to split the extension module initialisation into two steps in Python 3.4, in a backwards compatible way. Step 1: The current module init function can be reduced to just creating the module instance and returning it (and potentially doing some simple C level setup). Optionally, after creating the module (and this is the new part), the module init code can register a C callback function that will be called after setting up the module. Step 2: The shared library importer receives the module instance from the module init function, adds __file__, __path__, __package__ and friends to the module dict, and then checks for the callback. If non-NULL, it calls it to continue the module initialisation by user code. The callback ============ The callback is defined as follows:: int (*PyModule_init_callback)(PyObject* the_module, PyModuleInitContext* context) "PyModuleInitContext" is a struct that is meant mostly for making the callback more future proof by allowing additional parameters to be passed in. For now, I can see a use case for the following fields:: struct PyModuleInitContext { char* module_name; char* qualified_module_name; } Both names are encoded in UTF-8. As for the file path, I consider it best to retrieve it from the module's __file__ attribute as a Python string object to reduce filename encoding problems. Note that this struct argument is not strictly required, but given that this proposal would have been much simpler if the module init function had accepted such an argument in the first place, I consider it a good idea not to let this chance pass by again. The registration of the callback uses a new C-API function: int PyModule_SetInitFunction(PyObject* module, PyModule_init_callback callback) The function name uses "Set" instead of "Register" to make it clear that there is only one such function per module. An alternative would be a new module creation function "PyModule_Create3()" that takes the callback as third argument, in addition to what "PyModule_Create2()" accepts. This would require users to explicitly pass in the (second) version argument, which might be considered only a minor issue. Implementation ============== The implementation requires local changes to the extension module importer and a new C-API function. In order to store the callback, it should use a new field in the module object struct. Open questions ============== It is not clear how extensions should be handled that register more than one module in their module init function, e.g. compiled packages. One possibility would be to leave the setup to the user, who would have to know all FQMNs anyway in this case, although not the import file path. Alternatively, the import machinery could use a stack to remember for which modules a callback was registered during the last init function call, set up all of them and then call their callbacks. It's not clear if this meets the intention of the user. Alternatives ============ 1) It would be possible to make extension modules optionally export another symbol, e.g. "PyInit2_modulename", that the shared library loader would call in addition to the required function "PyInit_modulename". This would remove the need for a new API that registers the above callback. The drawback is that it also makes it easier to write broken code because a Python version or implementation that does not support this second symbol would simply not call it, without error. The new C-API function would let the build fail instead if it is not supported. 2) The callback could be made available as a Python function in the module dict, thus also removing the need for an explicit registration API. However, this approach would add overhead to both sides, the importer code and the user provided module init code, as it would require additional dictionary handling and the implementation of a one-time Python function in user code. It would also suffer from the problem that missing support in the runtime would pass silently. 3) The callback could be registered statically in the PyModuleDef struct by adding a new field. This is not trivial to do in a backwards compatible way because the struct would grow longer without explicit initialisation by existing user code. Extending PyModuleDef_HEAD_INIT might be possible but would still break at least binary compatibility. 4) Pass a new context argument into the module init function that contains all information necessary to properly and completely set up the module at creation time. This would provide a much simpler and cleaner solution than the proposed solution. However, it will not be possible before Python 4 as it breaks backwards compatibility with all existing extension modules at both the source and binary level. [View Less]

5 15

Inherance of file descriptor and handles on Windows (PEP 446)
by Victor Stinner Aug. 5, 2013

Aug. 5, 2013

Hi, Guido van Rossum and others asked me details on how file descriptors and handles are inherited on Windows, for the PEP 446. http://www.python.org/dev/peps/pep-0446/ I hacked Python 3.4 to add a os.get_cloexec() function (extracted from my implementation of the PEP 433), here are some results. Python functions open(), os.open() and os.dup() create file descriptors with the HANDLE_FLAG_INHERIT flag set (cloexec=False), whereas os.pipe() creates 2 file descriptors with the … [View More]

11 40

Allowing to run certain regression tests in subprocesses
by Eli Bendersky Aug. 4, 2013

Aug. 4, 2013

Hi All, Today the issue of cross-test global env dependencies showed its ugly head again for me. I recall a previous discussion (http://mail.python.org/pipermail/python-dev/2013-January/123409.html) but there were many more over the years. The core problem is that some tests modify the global env (particularly importing modules) and this sometimes has adverse effects on other tests, because test.regrtest runs all tests in a single process. In the discussion linked above, the particular … [View More]

3 8

[RELEASED] Python 3.4.0a1
by Larry Hastings Aug. 4, 2013

Aug. 4, 2013

On behalf of the Python development team, I'm pleased to announce the first alpha release of Python 3.4. This is a preview release, and its use is not recommended for production settings. Python 3.4 includes a range of improvements of the 3.x series, including hundreds of small improvements and bug fixes. Major new features and changes in the 3.4 release series so far include: * PEP 435, a standardized "enum" module * PEP 442, improved semantics for object finalization * PEP 443, adding … [View More]

2 3

Re: [Python-Dev] Allowing to run certain regression tests in subprocesses
by R. David Murray Aug. 4, 2013

Aug. 4, 2013

On Sat, 03 Aug 2013 19:04:21 -0700, Eli Bendersky <eliben(a)gmail.com> wrote: > On Sat, Aug 3, 2013 at 6:57 PM, R. David Murray <rdmurray(a)bitdance.com> wrote: > > On Sat, 03 Aug 2013 16:47:37 -0700, Eli Bendersky <eliben(a)gmail.com> wrote: > >> On Sat, Aug 3, 2013 at 4:36 PM, Eli Bendersky <eliben(a)gmail.com> wrote: > >> > Hi All, > >> > > >> > Today the issue of cross-test global env dependencies showed its ugly … [View More]> >> > head again for me. I recall a previous discussion > >> > (http://mail.python.org/pipermail/python-dev/2013-January/123409.html) > >> > but there were many more over the years. > >> > > >> > The core problem is that some tests modify the global env > >> > (particularly importing modules) and this sometimes has adverse > >> > effects on other tests, because test.regrtest runs all tests in a > >> > single process. In the discussion linked above, the particular culprit > >> > test__all__ was judged as a candidate to be moved to a subprocess. > >> > > >> > I want to propose adding a capability to our test harness to run > >> > specific tests in subprocesses. Each test will have some simple way of > >> > asking to be run in a subprocess, and regrtest will concur (even when > >> > running -j1). test__all__ can go there, and it can help solve other > >> > problems. > >> > > >> > My particular case is trying to write a test for > >> > http://bugs.python.org/issue14988 - wherein I have to simulate a > >> > situation of non-existent pyexpat. It's not hard to write a test for > >> > it, but when run in tandem with other tests (where C extensions loaded > >> > pyexpat) it becomes seemingly impossible to set up. This should not be > >> > the case - there's nothing wrong with wanting to simulate this case, > >> > and there's nothing wrong in Python and the stdlib - it's purely an > >> > artifact of the way our regression suite works. > >> > > >> > Thoughts? > >> > > >> > Eli > >> > >> FWIW the problem is also discussed here: > >> http://bugs.python.org/issue1674555, w.r.t. test_site > > > > Can't you just launch a subprocess from the test itself using script_helpers? > > > > I can, but such launching will be necessarily duplicated across all > tests that need this functionality (test_site, test___all__, etc). > Since regrtest already has functionality for launching whole > test-suites in subprocesses, it makes sense to reuse it, no? In the case of test_site and test___all___ we are talking about running the entire test file in a subprocess. It sounds like you are only talking about running one individual test function in a subprocess, for which using script_helpers seems the more natural solution. --David [View Less]

2 1

PEP 8 modernisation
by Nick Coghlan Aug. 4, 2013

Aug. 4, 2013

With feedback from Guido, Barry, Raymond and others, I have updated PEP 8 to better describe our current development practices. It started as an update to describe the different between public and internal interfaces and to advise against using wildcard imports, but became substantially more :) For those that want full details, the relevant commit and tracker issue are here: http://hg.python.org/peps/rev/fb24c80e9afb http://bugs.python.org/issue18472 If you're responsible for a … [View More]

18 42

Fwd: Allowing to run certain regression tests in subprocesses
by Eli Bendersky Aug. 4, 2013

Aug. 4, 2013

On Sat, Aug 3, 2013 at 6:57 PM, R. David Murray <rdmurray(a)bitdance.com> wrote: > On Sat, 03 Aug 2013 16:47:37 -0700, Eli Bendersky <eliben(a)gmail.com> wrote: >> On Sat, Aug 3, 2013 at 4:36 PM, Eli Bendersky <eliben(a)gmail.com> wrote: >> > Hi All, >> > >> > Today the issue of cross-test global env dependencies showed its ugly >> > head again for me. I recall a previous discussion >> > (http://mail.python.org/pipermail/… [View More]

1 0

objections to renaming enumobject.h/c in 3.4?
by Eli Bendersky Aug. 3, 2013

Aug. 3, 2013

Hi all, I was looking around the Objects directory and noticed that we have enumobject.h/c with the enumobject structure for "enumerate" and "reversed". This is somewhat confusing now with Lib/enum.py and will be doubly confusing if we ever decide to have a C implementation of enums. Any objections to renaming the files and the internal structure & static functions with s/enum/enumerate/ ? This would more accurately reflect the use of the code, and avoid confusion with enums. These … [View More]

4 6