Mailman 3 August 2013 - Python-Dev

Pre-PEP: Redesigning extension modules
by Stefan Behnel 08 Sep '13

08 Sep '13

Hi, this has been subject to a couple of threads on python-dev already, for example: http://thread.gmane.org/gmane.comp.python.devel/135764/focus=140986 http://thread.gmane.org/gmane.comp.python.devel/141037/focus=141046 It originally came out of issues 13429 and 16392. http://bugs.python.org/issue13429 http://bugs.python.org/issue16392 Here's an initial attempt at a PEP for it. It is based on the (unfinished) ModuleSpec PEP, which is being discussed on the import-sig mailing list. http://mail.python.org/pipermail/import-sig/2013-August/000688.html Stefan PEP: 4XX Title: Redesigning extension modules Version: $Revision$ Last-Modified: $Date$ Author: Stefan Behnel <stefan_ml at behnel.de> BDFL-Delegate: ??? Discussions-To: ??? Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 11-Aug-2013 Python-Version: 3.4 Post-History: 23-Aug-2013 Resolution: Abstract ======== This PEP proposes a redesign of the way in which extension modules interact with the interpreter runtime. This was last revised for Python 3.0 in PEP 3121, but did not solve all problems at the time. The goal is to solve them by bringing extension modules closer to the way Python modules behave. An implication of this PEP is that extension modules can use arbitrary types for their module implementation and are no longer restricted to types.ModuleType. This makes it easy to support properties at the module level and to safely store arbitrary global state in the module that is covered by normal garbage collection and supports reloading and sub-interpreters. Motivation ========== Python modules and extension modules are not being set up in the same way. For Python modules, the module is created and set up first, then the module code is being executed. For extensions, i.e. shared libraries, the module init function is executed straight away and does both the creation and initialisation. This means that it knows neither the __file__ it is being loaded from nor its package (i.e. its fully qualified module name, FQMN). This hinders relative imports and resource loading. In Py3, it's also not being added to sys.modules, which means that a (potentially transitive) re-import of the module will really try to reimport it and thus run into an infinite loop when it executes the module init function again. And without the FQMN, it is not trivial to correctly add the module to sys.modules either. This is specifically a problem for Cython generated modules, for which it's not uncommon that the module init code has the same level of complexity as that of any 'regular' Python module. Also, the lack of a FQMN and correct file path hinders the compilation of __init__.py modules, i.e. packages, especially when relative imports are being used at module init time. Furthermore, the majority of currently existing extension modules has problems with sub-interpreter support and/or reloading and it is neither easy nor efficient with the current infrastructure to support these features. This PEP also addresses these issues. The current process =================== Currently, extension modules export an initialisation function named "PyInit_modulename", named after the file name of the shared library. This function is executed by the import machinery and must return either NULL in the case of an exception, or a fully initialised module object. The function receives no arguments, so it has no way of knowing about its import context. During its execution, the module init function creates a module object based on a PyModuleDef struct. It then continues to initialise it by adding attributes to the module dict, creating types, etc. In the back, the shared library loader keeps a note of the fully qualified module name of the last module that it loaded, and when a module gets created that has a matching name, this global variable is used to determine the FQMN of the module object. This is not entirely safe as it relies on the module init function creating its own module object first, but this assumption usually holds in practice. The main problem in this process is the missing support for passing state into the module init function, and for safely passing state through to the module creation code. The proposal ============ The current extension module initialisation will be deprecated in favour of a new initialisation scheme. Since the current scheme will continue to be available, existing code will continue to work unchanged, including binary compatibility. Extension modules that support the new initialisation scheme must export a new public symbol "PyModuleCreate_modulename", where "modulename" is the name of the shared library. This mimics the previous naming convention for the "PyInit_modulename" function. This symbol must resolve to a C function with the following signature:: PyObject* (*PyModuleTypeCreateFunction)(PyObject* module_spec) The "module_spec" argument receives a "ModuleSpec" instance, as defined in PEP 4XX (FIXME). (All names are obviously up for debate and bike-shedding at this point.) When called, this function must create and return a type object, either a Python class or an extension type that is allocated on the heap. This type will be instantiated as module instance by the importer. There is no requirement for this type to be exactly or a subtype of types.ModuleType. Any type can be returned. This follows the current support for allowing arbitrary objects in sys.modules and makes it easier for extension modules to define a type that exactly matches their needs for holding module state. The constructor of this type must have the following signature:: def __init__(self, module_spec): The "module_spec" argument receives the same object as the one passed into the module type creation function. Implementation ============== XXX - not started Reloading and Sub-Interpreters ============================== To "reload" an extension module, the module create function is executed again and returns a new module type. This type is then instantiated as by the original module loader and replaces the previous entry in sys.modules. Once the last references to the previous module and its type are gone, both will be subject to normal garbage collection. Sub-interpreter support is an inherent property of the design. During import in the sub-interpreter, the module create function is executed and returns a new module type that is local to the sub-interpreter. Both the type and its module instance are subject to garbage collection in the sub-interpreter. Open questions ============== It is not immediately obvious how extensions should be handled that want to register more than one module in their module init function, e.g. compiled packages. One possibility would be to leave the setup to the user, who would have to know all FQMNs anyway in this case (or could construct them from the module spec of the current module), although not the import file path. A C-API could be provided to register new module types in the current interpreter, given a user provided ModuleSpec. There is no inherent requirement for the module creation function to actually return a type. It could return a arbitrary callable that creates a 'modulish' object when called. Should there be a type check in place that makes sure that what it returns is a type? I don't currently see a need for this. Copyright ========= This document has been placed in the public domain.

7 46

Add a new tracemalloc module to trace memory allocations
by Victor Stinner 06 Sep '13

06 Sep '13

Hi, Thanks to the PEP 445, it becomes possible to trace easily memory allocations. I propose to add a new tracemalloc module computing the memory usage per file and per line number. It has also a private method to retrieve the location (filename and line number) of a memory allocation of an object. tracemalloc is different than Heapy or PySizer because it is focused on the location of a memory allocation rather that the object type or object content. I have an implementation of the module for Python 2.5-3.4, but it requires to patch and recompile Python: https://pypi.python.org/pypi/pytracemalloc My proposed implementation for Python 3.4 is different: * reuse the PEP 445 to hook memory allocators * use a simple C implementation of an hash table called "cfuhash" (coming from the libcfu project, BSD license) instead of depending on the glib library. I simplified and adapted cfuhash for my usage * no enable() / disable() function: tracemalloc can only be enabled before startup by setting PYTHONTRACEMALLOC=1 environment variable * traces (size of the memory block, Python filename, Python line number) are stored directly in the memory block, not in a separated hash table I chose PYTHONTRACEMALLOC env var instead of enable()/disable() functions to be able to really trace *all* memory allocated by Python, especially memory allocated at startup, during Python initialization. The (high-level) API should be reviewed and discussed. The most interesting part is to take "snapshots" and compare snapshots. The module can load snapshots from disk and compare them later for deeper analysis (ex: ignore some files). For the documentation, see the following page: https://pypi.python.org/pypi/pytracemalloc I created the following issue to track the implementation: http://bugs.python.org/issue18874 The implementation: http://hg.python.org/features/tracemalloc * * * I also created a "pyfailmalloc" project based on the PEP 445 to inject MemoryError exceptions. I used this module to check if Python handles correctly memory allocation failures. The answer is no, I fixed many bugs (see issue #18408). Project homepage: https://bitbucket.org/haypo/pyfailmalloc Charles-François Natali and Serhiy Storchaka asked me to add this module somewhere in Python 3.4: "how about adding pyfailmalloc to the main repo (maybe under Tools), with a script making it easy to run the tests suite with it enabled?" What is the best place for such module? Add it to Modules/ directory but as a private module: "_failmalloc"? * * * Example of tracemalloc output (it is more readable with colors, try in a terminal). The first top is sorted by total size, whereas the second top is sorted (automatically) with the size difference. You can see for example that the linecache module likes caching data (1.5 MB after 10 seconds of tests). $ PYTHONTRACEMALLOC=1 ./python -m test ... == CPython 3.4.0a1+ (default:2ce9e5f6b47c+, Aug 29 2013, 02:03:02) [GCC 4.7.2 20121109 (Red Hat 4.7.2-8)] == Linux-3.9.4-200.fc18.x86_64-x86_64-with-fedora-18-Spherical_Cow little-endian == /home/haypo/prog/python/tracemalloc/build/test_python_11087 ... [ 1/380] test_grammar [ 2/380] test_opcodes [ 3/380] test_dict [ 4/380] test_builtin [ 5/380] test_exceptions [ 6/380] test_types [ 7/380] test_unittest 2013-08-29 02:06:22: Top 25 allocations per file and line #1: <frozen importlib._bootstrap>:704: size=5 MiB, count=56227, average=105 B #2: .../tracemalloc/Lib/linecache.py:127: size=1004 KiB, count=8706, average=118 B #3: .../Lib/unittest/mock.py:1764: size=895 KiB, count=7841, average=116 B #4: .../Lib/unittest/mock.py:1805: size=817 KiB, count=15101, average=55 B #5: .../Lib/test/test_dict.py:35: size=768 KiB, count=8, average=96 KiB #6: <frozen importlib._bootstrap>:274: size=703 KiB, count=4604, average=156 B #7: ???:?: size=511 KiB, count=4445, average=117 B #8: .../Lib/unittest/mock.py:350: size=370 KiB, count=1227, average=308 B #9: .../Lib/unittest/case.py:306: size=343 KiB, count=1390, average=253 B #10: .../Lib/unittest/case.py:496: size=330 KiB, count=650, average=521 B #11: .../Lib/unittest/case.py:327: size=291 KiB, count=717, average=416 B #12: .../Lib/collections/__init__.py:368: size=239 KiB, count=2170, average=113 B #13: .../Lib/test/test_grammar.py:132: size=195 KiB, count=1250, average=159 B #14: .../Lib/unittest/mock.py:379: size=118 KiB, count=152, average=800 B #15: .../tracemalloc/Lib/contextlib.py:37: size=102 KiB, count=672, average=156 B #16: <frozen importlib._bootstrap>:1430: size=91 KiB, count=1193, average=78 B #17: .../tracemalloc/Lib/inspect.py:1399: size=79 KiB, count=104, average=784 B #18: .../tracemalloc/Lib/abc.py:133: size=77 KiB, count=275, average=289 B #19: .../Lib/unittest/case.py:43: size=73 KiB, count=593, average=127 B #20: .../Lib/unittest/mock.py:491: size=67 KiB, count=153, average=450 B #21: <frozen importlib._bootstrap>:1438: size=64 KiB, count=20, average=3321 B #22: .../Lib/unittest/case.py:535: size=56 KiB, count=76, average=766 B #23: .../tracemalloc/Lib/sre_compile.py:508: size=54 KiB, count=115, average=485 B #24: .../Lib/unittest/case.py:300: size=48 KiB, count=616, average=80 B #25: .../Lib/test/regrtest.py:1207: size=48 KiB, count=2, average=24 KiB 7333 more: size=4991 KiB, count=28051, average=182 B Total Python memory: size=17 MiB, count=136358, average=136 B Total process memory: size=42 MiB (ignore tracemalloc: 22 KiB) [ 8/380] test_doctest [ 9/380] test_doctest2 [ 10/380] test_support [ 11/380] test___all__ 2013-08-29 02:08:44: Top 25 allocations per file and line (compared to 2013-08-29 02:08:39) #1: <frozen importlib._bootstrap>:704: size=8 MiB (+2853 KiB), count=80879 (+24652), average=109 B #2: .../tracemalloc/Lib/linecache.py:127: size=1562 KiB (+557 KiB), count=13669 (+4964), average=117 B #3: <frozen importlib._bootstrap>:274: size=955 KiB (+252 KiB), count=6415 (+1811), average=152 B #4: .../Lib/collections/__init__.py:368: size=333 KiB (+93 KiB), count=3136 (+966), average=108 B #5: .../tracemalloc/Lib/abc.py:133: size=148 KiB (+71 KiB), count=483 (+211), average=314 B #6: .../Lib/urllib/parse.py:476: size=71 KiB (+71 KiB), count=969 (+969), average=75 B #7: .../tracemalloc/Lib/base64.py:143: size=59 KiB (+59 KiB), count=1025 (+1025), average=59 B #8: .../tracemalloc/Lib/doctest.py:1283: size=56 KiB (+56 KiB), count=507 (+507), average=113 B #9: .../tracemalloc/Lib/sre_compile.py:508: size=89 KiB (+36 KiB), count=199 (+86), average=460 B #10: <frozen importlib._bootstrap>:53: size=67 KiB (+32 KiB), count=505 (+242), average=136 B #11: <frozen importlib._bootstrap>:1048: size=61 KiB (+27 KiB), count=332 (+138), average=188 B #12: .../Lib/unittest/case.py:496: size=351 KiB (+25 KiB), count=688 (+48), average=522 B #13: .../tracemalloc/Lib/_weakrefset.py:38: size=55 KiB (+24 KiB), count=521 (+236), average=109 B #14: .../Lib/email/quoprimime.py:56: size=24 KiB (+24 KiB), count=190 (+190), average=132 B #15: .../Lib/email/quoprimime.py:57: size=24 KiB (+24 KiB), count=1 (+1) #16: .../test/support/__init__.py:1055: size=24 KiB (+24 KiB), count=1 (+1) #17: .../Lib/test/test___all__.py:48: size=23 KiB (+23 KiB), count=283 (+283), average=84 B #18: .../tracemalloc/Lib/_weakrefset.py:37: size=60 KiB (+22 KiB), count=438 (+174), average=140 B #19: .../tracemalloc/Lib/sre_parse.py:73: size=48 KiB (+20 KiB), count=263 (+114), average=189 B #20: <string>:5: size=61 KiB (+18 KiB), count=173 (+57), average=364 B #21: .../Lib/test/test___all__.py:34: size=16 KiB (+16 KiB), count=164 (+164), average=104 B #22: .../Lib/unittest/mock.py:491: size=79 KiB (+16 KiB), count=145 (+3), average=560 B #23: .../Lib/collections/__init__.py:362: size=50 KiB (+16 KiB), count=23 (+7), average=2255 B #24: .../Lib/test/test___all__.py:59: size=13 KiB (+13 KiB), count=165 (+165), average=84 B #25: .../tracemalloc/Lib/doctest.py:1291: size=13 KiB (+13 KiB), count=170 (+170), average=81 B 10788 more: size=7 MiB (-830 KiB), count=36291 (-16379), average=220 B Total Python memory: size=20 MiB (+3567 KiB), count=147635 (+20805), average=143 B Total process memory: size=49 MiB (+7 MiB) (ignore tracemalloc: 1669 KiB) Victor

6 10

Completing the email6 API changes.
by R. David Murray 04 Sep '13

04 Sep '13

If you've read my blog (eg: on planet python), you will be aware that I dedicated August to full time email package development. At the beginning of the month I worked out a design proposal for the remaining API additions to the email package, dealing with handling message bodies in a more natural way. I posted this to the email-sig, and got...well, no objections. Barry Warsaw did review it, and told me he had no issues with the overall design, but also had no time for a detailed review. Since one way to see if a design holds together is to document and code it, I decided to go ahead and do so. This resulted in a number of small tweaks, but no major changes. I have at this point completed the coding. You can view the whole patch at: http://bugs.python.org/issue18891 which also links to three layered patches that I posted as I went along, if you prefer somewhat smaller patches. I think it would be great if I could check this in for alpha2. Since it is going in as an addition to the existing provisional code, the level of review required is not as high as for non-provisional code, I think. But I would certainly appreciate review from anyone so moved, since I haven't gotten any yet. Of course, if there is serious bikeshedding about the API, I won't make alpha2, but that's fine. The longer term goal, by the way, is to move all of this out of provisional status for 3.5. This code finishes the planned API additions for the email package to bring it fully into the world of Python3 and unicode. It does not "fix" the deep internals, which could be a future development direction (but probably only after the "old" API has been retired, which will take a while). But it does make it so that you can use the email package without having to be a MIME expert. (You can't get away with *no* MIME knowledge, but you no longer have to fuss with the details of the syntax.) To give you the flavor of how the entire new provisional API plays together, here's how you can build a complete message in your application: from email.message import MIMEMessage from email.headerregistry import Address fullmsg = MIMEMessage() fullmsg['To'] = Address('Foö Bar', 'fbar(a)example.com') fullmsg['From'] = "mè <me(a)example.com>" fullmsg['Subject'] = "j'ai un problème de python." fullmsg.set_content("et la il est monté sur moi et il commence" " a m'étouffer.") htmlmsg = MIMEMessage() htmlmsg.set_content("et la il est monté sur moi et il commence" " a m'étouffer.<img src='image1' />", subtype='html') with open('python.jpg', 'rb') as python: htmlmsg.add_related(python.read(), 'image', 'jpg', cid='image1' disposition='inline') fullmsg.make_alternative() fullmsg.attach(htmlmsg) with open('police-report.txt') as report: fullmsg.add_attachment(report.read(), filename='pölice-report.txt', params=dict(wrap='flow'), headers=( 'X-Secret-Level: top', 'X-Authorization: Monty')) Which results in: >>> for line in bytes(fullmsg).splitlines(): >>> print(line) b'To: =?utf-8?q?Fo=C3=B6?= Bar <fbar(a)example.com>' b'From: =?utf-8?q?m=C3=A8?= <me(a)example.com>' b"Subject: j'ai un =?utf-8?q?probl=C3=A8me?= de python." b'MIME-Version: 1.0' b'Content-Type: multipart/mixed; boundary="===============1710006838=="' b'' b'--===============1710006838==' b'Content-Type: multipart/alternative; boundary="===============1811969196=="' b'' b'--===============1811969196==' b'Content-Type: text/plain; charset="utf-8"' b'Content-Transfer-Encoding: 8bit' b'' b"et la il est mont\xc3\xa9 sur moi et il commence a m'\xc3\xa9touffer." b'' b'--===============1811969196==' b'MIME-Version: 1.0' b'Content-Type: multipart/related; boundary="===============1469657937=="' b'' b'--===============1469657937==' b'Content-Type: text/html; charset="utf-8"' b'Content-Transfer-Encoding: quoted-printable' b'' b"et la il est mont=C3=A9 sur moi et il commence a m'=C3=A9touffer.<img =" b"src=3D'image1' />" b'' b'--===============1469657937==' b'MIME-Version: 1.0' b'Content-Type: image/jpg' b'Content-Transfer-Encoding: base64' b'Content-Disposition: inline' b'Content-ID: image1' b'' b'ZmFrZSBpbWFnZSBkYXRhCg==' b'' b'--===============1469657937==--' b'--===============1811969196==--' b'--===============1710006838==' b'MIME-Version: 1.0' b'X-Secret-Level: top' b'X-Authorization: Monty' b'Content-Transfer-Encoding: 7bit' b'Content-Disposition: attachment; filename*=utf-8''p%C3%B6lice-report.txt" b'Content-Type: text/plain; charset="utf-8"; wrap="flow"' b'' b'il est sorti de son vivarium.' b'' b'--===============1710006838==--' If you've used the email package enough to be annoyed by it, you may notice that there are some nice things going on there, such as using CTE 8bit for the text part by default, and quoted-printable instead of base64 for utf8 when the lines are long enough to need wrapping. (Hmm. Looking at that I see I didn't fully fix a bug I had meant to fix: some of the parts have a MIME-Version header that don't need it.) All input strings are unicode, and the library takes care of doing whatever encoding is required. When you pull data out of a parsed message, you get unicode, without having to worry about how to decode it yourself. On the parsing side, after the above message has been parsed into a message object, we can do: >>> print(fullmsg['to'], fullmsg['from']) Foö Bar <"fbar(a)example.com"> mè <me(a)example.com> >>> print(fullmsg['subject']) j'ai un problème de python. >>> print(fullmsg['to'].addresses[0].display_name) Foö Bar >>> print(fullmsg.get_body(('plain',)).get_content()) et la il est monté sur moi et il commence a m'étouffer. >>> for part in fullmsg.get_body().iter_parts(): ... print(part.get_content()) et la il est monté sur moi et il commence a m'étouffer.<img src='image1' /> b'fake image data\n' >>> for attachment in fullmsg.iter_attachments(): ... print(attachment.get_content()) ... print(attachment['Content-Type'].params()) il est sorti de son vivarium. {'charset': 'utf-8', 'wrap': 'flow'} Of course, in a real program you'd actually be checking the mime types via get_content_type() and friends before getting the content and doing anything with it. Please read the new contentmanager module docs in the patch for full details of the content management part of the above API (and the headerregistry docs if you want to review the (new in 3.3) header parsing part of the above API). Feedback welcome, here or on the issue. --David PS: python jokes courtesy of someone doing a drive-by on #python-dev the other day.

4 19

error in test suite
by Ethan Furman 01 Sep '13

01 Sep '13

Am I the only one experiencing this? 262 tests OK. 93 tests failed: test___all__ test_abc test_array test_ast test_asynchat test_asyncore test_bisect test_buffer test_bufio test_bytes test_codeccallbacks test_codecs test_colorsys test_compileall test_configparser test_contextlib test_crypt test_ctypes test_dbm test_dbm_dumb test_dbm_ndbm test_dictcomps test_enum test_exceptions test_faulthandler test_file test_fileinput test_frozen test_future test_future3 test_future5 test_genericpath test_getargs2 test_getpass test_hash test_hashlib test_heapq test_idle test_imaplib test_imp test_import test_index test_io test_ioctl test_ipaddress test_iterlen test_json test_keyword test_largefile test_locale test_macpath test_multiprocessing_fork test_multiprocessing_forkserver test_multiprocessing_spawn test_namespace_pkgs test_ntpath test_operator test_osx_env test_pdb test_pep352 test_posixpath test_print test_py_compile test_random test_regrtest test_robotparser test_runpy test_sched test_set test_shutil test_site test_smtpd test_sndhdr test_source_encoding test_sqlite test_stat test_strftime test_sundry test_tarfile test_textwrap test_threading test_time test_unicode test_univnewlines test_urllib test_urllib2net test_userstring test_uuid test_warnings test_wave test_webbrowser test_xml_dom_minicompat test_zipfile 24 tests skipped: test_codecmaps_cn test_codecmaps_hk test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_curses test_dbm_gnu test_devpoll test_gdb test_kqueue test_lzma test_msilib test_ossaudiodev test_smtpnet test_socketserver test_startfile test_timeout test_tk test_ttk_guionly test_urllibnet test_winreg test_winsound test_xmlrpc_net test_zipfile64 and the failure appears to always be: test [...] crashed -- Traceback (most recent call last): File "/home/ethan/source/python/issue18780/Lib/test/regrtest.py", line 1265, in runtest_inner huntrleaks) File "/home/ethan/source/python/issue18780/Lib/test/regrtest.py", line 1381, in dash_R indirect_test() File "/home/ethan/source/python/issue18780/Lib/test/regrtest.py", line 1261, in <lambda> test_runner = lambda: support.run_unittest(tests) File "/home/ethan/source/python/issue18780/Lib/test/support/__init__.py", line 1683, in run_unittest _run_suite(suite) File "/home/ethan/source/python/issue18780/Lib/test/support/__init__.py", line 1649, in _run_suite result = runner.run(suite) File "/home/ethan/source/python/issue18780/Lib/test/support/__init__.py", line 1548, in run test(result) File "/home/ethan/source/python/issue18780/Lib/unittest/suite.py", line 76, in __call__ return self.run(*args, **kwds) File "/home/ethan/source/python/issue18780/Lib/unittest/suite.py", line 114, in run test(result) File "/home/ethan/source/python/issue18780/Lib/unittest/suite.py", line 76, in __call__ return self.run(*args, **kwds) File "/home/ethan/source/python/issue18780/Lib/unittest/suite.py", line 114, in run test(result) TypeError: 'NoneType' object is not callable

3 2

EINTR handling...
by Charles-François Natali 01 Sep '13

01 Sep '13

Hello, This has been bothering me for years: why don't we properly handle EINTR, by running registered signal handlers and restarting the interrupted syscall (or eventually returning early e.g. for sleep)? EINTR is really a nuisance, and exposing it to Python code is just pointless. Now some people might argue that some code relies on EINTR to interrupt a syscall on purpose, but I don't really buy it: it's highly non-portable (depends on the syscall, SA_RESTART flag...) and subject to race conditions (it it comes before the syscall or if you get a partial read/write you'll deadlock). Furthermore, the stdlib code base is not consistent: some code paths handle EINTR, e.g. subprocess, multiprocessing, sock_sendall() does but not sock_send()... Just grep for EINTR and InterruptedError and you'll be amazed. GHC, the JVM and probably other platforms handle EINTR, maybe it's time for us too? Just for reference, here are some issues due to EINTR popping up: http://bugs.python.org/issue17097 http://bugs.python.org/issue12268 http://bugs.python.org/issue9867 http://bugs.python.org/issue7978 http://bugs.python.org/issue12493 http://bugs.python.org/issue3771 cf

6 6

Re: [Python-Dev] cpython: Issue #17741: Rename IncrementalParser and its methods.
by Antoine Pitrou 31 Aug '13

31 Aug '13

Hello, On Fri, 30 Aug 2013 14:51:42 +0200 (CEST) eli.bendersky <python-checkins(a)python.org> wrote: > diff --git a/Doc/library/xml.etree.elementtree.rst b/Doc/library/xml.etree.elementtree.rst > --- a/Doc/library/xml.etree.elementtree.rst > +++ b/Doc/library/xml.etree.elementtree.rst > @@ -105,37 +105,42 @@ > >>> root[0][1].text > '2008' > > -Incremental parsing > -^^^^^^^^^^^^^^^^^^^ > +Pull API for asynchronous parsing > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ I think the documentation should use another term than "asynchronous": "non-blocking" or "event-driven" would be fine. "Asynchronous" is loaded with various meanings and connotations which IMO make it the wrong term here. For example, in many contexts "asynchronous" means "does the job behind your back, e.g. in a worker thread". POSIX defines some asynchronous I/O APIs which are ostensibly not the same as non-blocking I/O: http://pubs.opengroup.org/onlinepubs/9699919799/functions/aio_read.html Regards Antoine.

5 4

Coverity Scan Spotlight Python
by Christian Heimes 31 Aug '13

31 Aug '13

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Hello, Coverity has published its "Coverity Scan Spotlight Python" a couple of hours ago. It features a summary of Python's ecosystem, an interview with me about Python core development and a defect report. The report is awesome. We have reached a defect density of .005 defects per 1,000 lines of code. In 2012 the average defect density of Open Source Software was 0.69. http://www.coverity.com/company/press-releases/read/coverity-finds-python-s… http://wpcme.coverity.com/wp-content/uploads/2013-Coverity-Scan-Spotlight-P… The internet likes it, too. http://www.prnewswire.com/news-releases/coverity-finds-python-sets-new-leve… http://www.securityweek.com/python-gets-high-marks-open-source-software-sec… Thank you very much to Kristin Brennan and Dakshesh Vyas from Coverity as well as everybody who has helped to fix the remaining issues! Christian -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iQIcBAEBCgAGBQJSH8bEAAoJEMeIxMHUVQ1FFQcQAL1/Tb5PFMdLXwWsMt9D06aP A2qQPunEnfDBMdQz4GTEeDmHPdjs/EgAtUz4sLI48HlAmpdWEtoVPCdg1GvKSvMi IRVHR5LAtxe5p8M42+8DnSFyIOtEsbtv06W5cHvRxr6RuIkY3bTy0SVhtP9JW+N7 wQKsp2cOIOz/FHDWWQWjxwlZmUWEGkvSSggzbYxcdsaJeGHoJgkuzoChQ3mCtUCo w231OTKBZhGQp/VpMK+Q7OXWm78BZdB6d4GcSR3meCU9GpRMfPBxPF7v4IWvDPv9 4l/y922hmLLoOchJG+PDqcDhX1dnFm1t3Q199iqS5c0c+ttgaMRdSJEXZpZrubxe k+frJiOivG4G7BuzgQ39yF01rRHpjs57FW9FBbt4pp2c+4iOEkgARH+L/e2ZwOnk puXE45AfKwJwHLc4RDOhxdaPy/ovOh53HY68UxXoKjeZKWK5ShRopk0muvYG0y5O +8PbAKOYgJbe//NC3ac89V/1eu4rrFhN7xsK2Wc8i+kcbTB2XIVFElLHuV5wjmLd MMXFlm9LDJFOw12E4sF3MPaHyXQYpNJHvbnuxCkcHRQoLKzrcRJ2Y0Jj4HPSUCsj JhfmHX7Zu+/akmT4haqXUdtRrn4wji0OYqGydEqi4aLy7ELrC1EVNZY4OkbUhJO8 gGbpseJXtVThXQ7fymMS =++g9 -----END PGP SIGNATURE-----

6 8

Summary of Python tracker Issues
by Python tracker 31 Aug '13

31 Aug '13

ACTIVITY SUMMARY (2013-08-23 - 2013-08-30) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4182 (+14) closed 26476 (+50) total 30658 (+64) Open issues with patches: 1910 Issues opened (48) ================== #17400: ipaddress should make it easy to identify rfc6598 addresses http://bugs.python.org/issue17400 reopened by ncoghlan #18822: poor proxyval() coverage in test_gdb http://bugs.python.org/issue18822 opened by pitrou #18823: Idle: use pipes instead of sockets to talk with user subproces http://bugs.python.org/issue18823 opened by terry.reedy #18824: Adding LogRecord attribute "traceback" http://bugs.python.org/issue18824 opened by Sworddragon #18825: Making msg optional on logging.exception() and similar variant http://bugs.python.org/issue18825 opened by Sworddragon #18826: reversed() requires a sequence - Could work on any iterator? http://bugs.python.org/issue18826 opened by dstufft #18828: urljoin behaves differently with custom and standard schemas http://bugs.python.org/issue18828 opened by mher.movsisyan #18829: csv produces confusing error message when passed a non-string http://bugs.python.org/issue18829 opened by Thibault.Kruse #18830: Remove duplicates from a result of getclasstree() http://bugs.python.org/issue18830 opened by serhiy.storchaka #18831: importlib.import_module() bypasses builtins.__import__ http://bugs.python.org/issue18831 opened by brett.cannon #18834: Add Clang to distutils to build C/C++ extensions http://bugs.python.org/issue18834 opened by Ryan.Gonzalez #18835: Add aligned memroy variants to the suite of PyMem functions/ma http://bugs.python.org/issue18835 opened by rhettinger #18837: multiprocessing.reduction is undocumented http://bugs.python.org/issue18837 opened by tpievila #18838: The order of interactive prompt and traceback on Windows http://bugs.python.org/issue18838 opened by Drekin #18840: Tutorial recommends pickle module without any warning of insec http://bugs.python.org/issue18840 opened by dstufft #18841: math.isfinite fails with Decimal sNAN http://bugs.python.org/issue18841 opened by stevenjd #18842: Add float.is_finite is_nan is_infinite to match Decimal method http://bugs.python.org/issue18842 opened by stevenjd #18843: Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at P http://bugs.python.org/issue18843 opened by mmokrejs #18844: allow weights in random.choice http://bugs.python.org/issue18844 opened by aisaac #18845: 2.7.5-r2: Fatal Python error: Segmentation fault http://bugs.python.org/issue18845 opened by mmokrejs #18848: In unittest.TestResult .startTestRun() and .stopTestRun() meth http://bugs.python.org/issue18848 opened by py.user #18849: Failure to try another name for tempfile when directory with c http://bugs.python.org/issue18849 opened by vlad #18850: xml.etree.ElementTree accepts control chars. http://bugs.python.org/issue18850 opened by maker #18851: subprocess's Popen closes stdout/stderr filedescriptors used i http://bugs.python.org/issue18851 opened by janwijbrand #18852: site.py does not handle readline.__doc__ being None http://bugs.python.org/issue18852 opened by theller #18853: Got ResourceWarning unclosed file when running Lib/shlex.py de http://bugs.python.org/issue18853 opened by vajrasky #18854: is_multipart and walk should document their treatment of 'mess http://bugs.python.org/issue18854 opened by r.david.murray #18855: Inconsistent README filenames http://bugs.python.org/issue18855 opened by madison.may #18856: Added test coverage for calendar print functions http://bugs.python.org/issue18856 opened by madison.may #18857: urlencode of a None value uses the string 'None' http://bugs.python.org/issue18857 opened by Joshua.Johnston #18858: dummy_threading lacks threading.get_ident() equivalent http://bugs.python.org/issue18858 opened by zuo #18859: README.valgrind should mention --with-valgrind http://bugs.python.org/issue18859 opened by tim.peters #18860: Add content manager API to email package http://bugs.python.org/issue18860 opened by r.david.murray #18861: Problems with recursive automatic exception chaining http://bugs.python.org/issue18861 opened by Nikratio #18862: Implement __subclasshook__() for Finders and Loaders in import http://bugs.python.org/issue18862 opened by eric.snow #18864: Implementation for PEP 451 (importlib.machinery.ModuleSpec) http://bugs.python.org/issue18864 opened by eric.snow #18870: eval() uses latin-1 to decode str http://bugs.python.org/issue18870 opened by valhallasw #18872: platform.linux_distribution() doesn't recognize Amazon Linux http://bugs.python.org/issue18872 opened by Lorin.Hochstein #18873: "Encoding" detected in non-comment lines http://bugs.python.org/issue18873 opened by Paul.Bonser #18874: Add a new tracemalloc module to trace memory allocations http://bugs.python.org/issue18874 opened by haypo #18875: Automatic insertion of the closing parentheses, brackets, and http://bugs.python.org/issue18875 opened by irdb #18876: Problems with files opened in append mode with io module http://bugs.python.org/issue18876 opened by erik.bray #18877: tkinter askopenfilenames does not work in Windows library fold http://bugs.python.org/issue18877 opened by tegavu #18878: Add support of the 'with' statement to sunau.open. http://bugs.python.org/issue18878 opened by serhiy.storchaka #18879: tempfile.NamedTemporaryFile can close the file too early, if n http://bugs.python.org/issue18879 opened by jort.bloem #18880: ssl.SSLSocket shutdown doesn't behave like socket.shutdown http://bugs.python.org/issue18880 opened by zielmicha #18882: Add threading.main_thread() function http://bugs.python.org/issue18882 opened by asvetlov #18885: handle EINTR in the stdlib http://bugs.python.org/issue18885 opened by neologix Most recent 15 issues with no replies (15) ========================================== #18885: handle EINTR in the stdlib http://bugs.python.org/issue18885 #18880: ssl.SSLSocket shutdown doesn't behave like socket.shutdown http://bugs.python.org/issue18880 #18878: Add support of the 'with' statement to sunau.open. http://bugs.python.org/issue18878 #18877: tkinter askopenfilenames does not work in Windows library fold http://bugs.python.org/issue18877 #18875: Automatic insertion of the closing parentheses, brackets, and http://bugs.python.org/issue18875 #18873: "Encoding" detected in non-comment lines http://bugs.python.org/issue18873 #18864: Implementation for PEP 451 (importlib.machinery.ModuleSpec) http://bugs.python.org/issue18864 #18862: Implement __subclasshook__() for Finders and Loaders in import http://bugs.python.org/issue18862 #18860: Add content manager API to email package http://bugs.python.org/issue18860 #18854: is_multipart and walk should document their treatment of 'mess http://bugs.python.org/issue18854 #18853: Got ResourceWarning unclosed file when running Lib/shlex.py de http://bugs.python.org/issue18853 #18849: Failure to try another name for tempfile when directory with c http://bugs.python.org/issue18849 #18842: Add float.is_finite is_nan is_infinite to match Decimal method http://bugs.python.org/issue18842 #18841: math.isfinite fails with Decimal sNAN http://bugs.python.org/issue18841 #18837: multiprocessing.reduction is undocumented http://bugs.python.org/issue18837 Most recent 15 issues waiting for review (15) ============================================= #18882: Add threading.main_thread() function http://bugs.python.org/issue18882 #18880: ssl.SSLSocket shutdown doesn't behave like socket.shutdown http://bugs.python.org/issue18880 #18878: Add support of the 'with' statement to sunau.open. http://bugs.python.org/issue18878 #18876: Problems with files opened in append mode with io module http://bugs.python.org/issue18876 #18874: Add a new tracemalloc module to trace memory allocations http://bugs.python.org/issue18874 #18872: platform.linux_distribution() doesn't recognize Amazon Linux http://bugs.python.org/issue18872 #18870: eval() uses latin-1 to decode str http://bugs.python.org/issue18870 #18860: Add content manager API to email package http://bugs.python.org/issue18860 #18856: Added test coverage for calendar print functions http://bugs.python.org/issue18856 #18853: Got ResourceWarning unclosed file when running Lib/shlex.py de http://bugs.python.org/issue18853 #18851: subprocess's Popen closes stdout/stderr filedescriptors used i http://bugs.python.org/issue18851 #18849: Failure to try another name for tempfile when directory with c http://bugs.python.org/issue18849 #18844: allow weights in random.choice http://bugs.python.org/issue18844 #18834: Add Clang to distutils to build C/C++ extensions http://bugs.python.org/issue18834 #18830: Remove duplicates from a result of getclasstree() http://bugs.python.org/issue18830 Top 10 most discussed issues (10) ================================= #17741: event-driven XML parser http://bugs.python.org/issue17741 59 msgs #18843: Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at P http://bugs.python.org/issue18843 33 msgs #18850: xml.etree.ElementTree accepts control chars. http://bugs.python.org/issue18850 26 msgs #18643: implement socketpair() on Windows http://bugs.python.org/issue18643 14 msgs #18851: subprocess's Popen closes stdout/stderr filedescriptors used i http://bugs.python.org/issue18851 11 msgs #16853: add a Selector to the select module http://bugs.python.org/issue16853 9 msgs #18870: eval() uses latin-1 to decode str http://bugs.python.org/issue18870 8 msgs #5720: ctime: I don't think that word means what you think it means. http://bugs.python.org/issue5720 7 msgs #18756: os.urandom() fails under high load http://bugs.python.org/issue18756 7 msgs #18808: Thread.join returns before PyThreadState is destroyed http://bugs.python.org/issue18808 7 msgs Issues closed (48) ================== #5876: __repr__ returning unicode doesn't work when called implicitly http://bugs.python.org/issue5876 closed by haypo #8713: multiprocessing needs option to eschew fork() under Linux http://bugs.python.org/issue8713 closed by sbt #10115: Support accept4() for atomic setting of flags at socket creati http://bugs.python.org/issue10115 closed by haypo #11798: Test cases not garbage collected after run http://bugs.python.org/issue11798 closed by asvetlov #12107: TCP listening sockets created without FD_CLOEXEC flag http://bugs.python.org/issue12107 closed by haypo #14914: pysetup installed distribute despite dry run option being spec http://bugs.python.org/issue14914 closed by eric.araujo #14974: rename packaging.pypi to packaging.index http://bugs.python.org/issue14974 closed by eric.araujo #15507: test_subprocess assumes SIGINT is not being ignored. http://bugs.python.org/issue15507 closed by gregory.p.smith #16611: Cookie.py does not parse httponly or secure cookie flags http://bugs.python.org/issue16611 closed by r.david.murray #16799: start using argparse.Namespace in regrtest http://bugs.python.org/issue16799 closed by serhiy.storchaka #16946: subprocess: _close_open_fd_range_safe() does not set close-on- http://bugs.python.org/issue16946 closed by haypo #17036: Implementation of the PEP 433: Easier suppression of file desc http://bugs.python.org/issue17036 closed by haypo #17070: PEP 433: Use the new cloexec to improve security and avoid bug http://bugs.python.org/issue17070 closed by haypo #17588: runpy cannot run Unicode path on Windows http://bugs.python.org/issue17588 closed by haypo #17974: Migrate unittest to argparse http://bugs.python.org/issue17974 closed by serhiy.storchaka #18394: cgi.FieldStorage triggers ResourceWarning sometimes http://bugs.python.org/issue18394 closed by brett.cannon #18538: `python -m dis ` should use argparse http://bugs.python.org/issue18538 closed by python-dev #18571: Implementation of the PEP 446: non-inheritable file descriptor http://bugs.python.org/issue18571 closed by haypo #18586: Allow running benchmarks for Python 3 from same directory http://bugs.python.org/issue18586 closed by brett.cannon #18647: re.error: nothing to repeat http://bugs.python.org/issue18647 closed by serhiy.storchaka #18743: References to non-existant "StringIO" module http://bugs.python.org/issue18743 closed by serhiy.storchaka #18757: Fix internal references for concurrent modules http://bugs.python.org/issue18757 closed by serhiy.storchaka #18760: Fix internal doc references for the xml package http://bugs.python.org/issue18760 closed by serhiy.storchaka #18763: subprocess: file descriptors should be closed after preexec_fn http://bugs.python.org/issue18763 closed by neologix #18783: No more refer to Python "long" http://bugs.python.org/issue18783 closed by serhiy.storchaka #18796: Wrong documentation of show_code function from dis module http://bugs.python.org/issue18796 closed by ezio.melotti #18798: Typo and unused variables in test fcntl http://bugs.python.org/issue18798 closed by ezio.melotti #18803: Fix more typos in .py files http://bugs.python.org/issue18803 closed by ezio.melotti #18806: socketmodule: fix/improve setipaddr() numeric addresses handli http://bugs.python.org/issue18806 closed by neologix #18807: Allow venv to create copies, even when symlinks are supported http://bugs.python.org/issue18807 closed by python-dev #18817: Got resource warning when running Lib/aifc.py http://bugs.python.org/issue18817 closed by serhiy.storchaka #18827: mistake in the timedelta.total_seconds docstring http://bugs.python.org/issue18827 closed by allo #18832: New regex module degrades re performance http://bugs.python.org/issue18832 closed by ned.deily #18833: Increased test coverage for telnetlib http://bugs.python.org/issue18833 closed by ezio.melotti #18836: Potential race condition in exceptions http://bugs.python.org/issue18836 closed by r.david.murray #18839: Wrong sentence in sys.exit.__doc__ http://bugs.python.org/issue18839 closed by ezio.melotti #18846: python.exe stdout stderr issues again http://bugs.python.org/issue18846 closed by benjamin.peterson #18847: spam http://bugs.python.org/issue18847 closed by ezio.melotti #18863: Encoding a unicode with unicode() and ignoring errors http://bugs.python.org/issue18863 closed by ned.deily #18865: multiprocessing: remove util.pipe()? http://bugs.python.org/issue18865 closed by sbt #18866: spam http://bugs.python.org/issue18866 closed by ezio.melotti #18867: spam http://bugs.python.org/issue18867 closed by ezio.melotti #18868: Python3 unbuffered stdin http://bugs.python.org/issue18868 closed by pitrou #18869: test suite: faulthandler signal handler is lost http://bugs.python.org/issue18869 closed by neologix #18871: Be more stringent about the test suite http://bugs.python.org/issue18871 closed by pitrou #18881: Can someone try to duplicate corruption on Gentoo? http://bugs.python.org/issue18881 closed by tim.peters #18883: python-3.3.2-r2: Modules/xxlimited.c:17:error: #error Py_LIMIT http://bugs.python.org/issue18883 closed by skrah #18884: python-2.7.5-r3: 40 bytes in 1 blocks are definitely lost http://bugs.python.org/issue18884 closed by haypo

1 0

Add function to signal module for getting main thread id
by Andrew Svetlov 30 Aug '13

30 Aug '13

Main thread is slightly different from others. Signals can be subscribed from main thread only. Tulip has special logic for main thread. In application code we can explicitly know which thread is executed, main or not. But from library it's not easy. Tulip uses check like threading.current_thread().name == 'MainThread' This approach has a problem: thread name is writable attribute and can be changed by user code. My proposition is to add function like get_mainthread_id() -> int which return ident for main thread (I know function name is not perfect, please guess better one). Signal module already has required data as internal variable static long main_thread; I just guess to expose this value to python. Thoughts? -- Thanks, Andrew Svetlov

6 13

Can someone try to duplicate corruption on Gentoo?
by Tim Peters 30 Aug '13

30 Aug '13

In http://bugs.python.org/issue18843 a user reported a debug PyMalloc "bad leading pad byte" memory corruption death while running their code. After some thrashing, they decided to rebuild Python, and got the same kind of error while rebuilding Python. See http://bugs.python.org/msg196481 in that bug report: """ # emerge dev-lang/python:2.7 * IMPORTANT: 11 news items need reading for repository 'gentoo'. * Use eselect news to read news items. Calculating dependencies... done! Debug memory block at address p=0xa7f5900: API 'o' 80 bytes originally requested The 7 pad bytes at p-7 are not all FORBIDDENBYTE (0xfb): at p-7: 0xfb at p-6: 0xfb at p-5: 0xfa *** OUCH at p-4: 0xfb at p-3: 0xfb at p-2: 0xfb at p-1: 0xfb Because memory is corrupted at the start, the count of bytes requested may be bogus, and checking the trailing pad bytes may segfault. The 8 pad bytes at tail=0xa7f5950 are FORBIDDENBYTE, as expected. The block was made by call #21242094 to debug malloc/realloc. Data at p: 73 00 00 00 79 00 00 00 ... 67 00 00 00 00 00 00 00 Fatal Python error: bad leading pad byte Aborted (core dumped) # """ I don't have access to Gentoo, and don't know squat about its `emerge`, but if someone else can do this it might help ;-) The Python used to run `emerge` here was a --with-pydebug Python the bug reporter built earlier.

2 2