Python-Dev July 2012

python-dev@python.org

82 participants
78 discussions

by anatoly techtonik

http://docs.python.org/py3k/howto/pyporting.html#during-installation What's the point in making implicit Python 3 check here: try: # Python 3 from distutils.command.build_py import build_py_2to3 as build_py except ImportError: # Python 2 from distutils.command.build_py import build_py instead of explicit check like: import sys if sys.version_info[0] >= 3: from distutils.command.build_py import build_py_2to3 as build_py ? -- anatoly t.

8 years, 8 months

test_hashlib

by Kristján Valur Jónsson

I was hit by this today. in test_hashlib.py there is this: def test_unknown_hash(self): self.assertRaises(ValueError, hashlib.new, 'spam spam spam spam spam') self.assertRaises(TypeError, hashlib.new, 1) but in hashlib.py, there is this code: except ImportError: pass # no extension module, this hash is unsupported. raise ValueError('unsupported hash type %s' % name) The code will raise ValueError when int(1) is passed in, but the unittests expect a TypeError. So, which is correct? K

8 years, 8 months

four easy documentation patches

by Chris Jerdonek

I was wondering if someone could review the following minor documentation patches: Document that filecmp.dircmp comparisons are "shallow" (7/5/2012): http://bugs.python.org/issue15250 Document dircmp.left and dircmp.right (7/6/2012): http://bugs.python.org/issue15269 Update "PyPI package display" documentation re: rst2html.py (7/5/2012): http://bugs.python.org/issue15231 [Dev Guide] Update documentation around Dev Guide process (7/10/2012): http://bugs.python.org/issue15311 Thanks, --Chris

8 years, 8 months

Re: [Python-Dev] cpython (3.2): ISsue #14988: restore Python 2's behavior of raising ImportError when unable to

by Georg Brandl

This looks like it will give an "error return without exception set" if the "if (expat_capi)" directly before the change is true, and expat_capi is then set to NULL in the branch. Georg On 07/17/2012 01:25 PM, eli.bendersky wrote: > http://hg.python.org/cpython/rev/d896fd0a8ba7 > changeset: 78152:d896fd0a8ba7 > branch: 3.2 > parent: 78142:71537aba3a0a > user: Eli Bendersky <eliben(a)gmail.com> > date: Tue Jul 17 14:20:38 2012 +0300 > summary: > ISsue #14988: restore Python 2's behavior of raising ImportError when unable to load pyexpat, instead of a SystemError/RuntimeError > > files: > Modules/_elementtree.c | 2 ++ > 1 files changed, 2 insertions(+), 0 deletions(-) > > > diff --git a/Modules/_elementtree.c b/Modules/_elementtree.c > --- a/Modules/_elementtree.c > +++ b/Modules/_elementtree.c > @@ -3117,6 +3117,8 @@ > expat_capi->MICRO_VERSION != XML_MICRO_VERSION) > expat_capi = NULL; > } > + if (!expat_capi) > + return NULL; > #endif > > elementtree_parseerror_obj = PyErr_NewException( > > > > _______________________________________________ > Python-checkins mailing list > Python-checkins(a)python.org > http://mail.python.org/mailman/listinfo/python-checkins >

8 years, 8 months

Re: [Python-Dev] cpython: Optimize tostringlist by taking the stream class outside the function. It's now

by Georg Brandl

Uh, optimizations are not exactly what I want to see during feature freeze. Georg On 07/17/2012 02:10 PM, eli.bendersky wrote: > http://hg.python.org/cpython/rev/51978f89e5ed > changeset: 78156:51978f89e5ed > user: Eli Bendersky <eliben(a)gmail.com> > date: Tue Jul 17 15:09:12 2012 +0300 > summary: > Optimize tostringlist by taking the stream class outside the function. It's now 2x faster on short calls. Related to #1767933 > > files: > Lib/xml/etree/ElementTree.py | 38 +++++++++++++---------- > 1 files changed, 22 insertions(+), 16 deletions(-) > > > diff --git a/Lib/xml/etree/ElementTree.py b/Lib/xml/etree/ElementTree.py > --- a/Lib/xml/etree/ElementTree.py > +++ b/Lib/xml/etree/ElementTree.py > @@ -1184,23 +1184,29 @@ > # @defreturn sequence > # @since 1.3 > > +class _ListDataStream(io.BufferedIOBase): > + """ An auxiliary stream accumulating into a list reference > + """ > + def __init__(self, lst): > + self.lst = lst > + > + def writable(self): > + return True > + > + def seekable(self): > + return True > + > + def write(self, b): > + self.lst.append(b) > + > + def tell(self): > + return len(self.lst) > + > def tostringlist(element, encoding=None, method=None): > - data = [] > - class DataStream(io.BufferedIOBase): > - def writable(self): > - return True > - > - def seekable(self): > - return True > - > - def write(self, b): > - data.append(b) > - > - def tell(self): > - return len(data) > - > - ElementTree(element).write(DataStream(), encoding, method=method) > - return data > + lst = [] > + stream = _ListDataStream(lst) > + ElementTree(element).write(stream, encoding, method=method) > + return lst > > ## > # Writes an element tree or element structure to sys.stdout. This > > > > _______________________________________________ > Python-checkins mailing list > Python-checkins(a)python.org > http://mail.python.org/mailman/listinfo/python-checkins >

8 years, 8 months

Re: [Python-Dev] cpython: Issue #15168: Move importlb.test to test.test_importlib.

by Antoine Pitrou

On Fri, 20 Jul 2012 20:49:03 +0200 (CEST) brett.cannon <python-checkins(a)python.org> wrote: > diff --git a/Lib/importlib/test/__init__.py b/Lib/test/test_importlib/__init__.py > rename from Lib/importlib/test/__init__.py > rename to Lib/test/test_importlib/__init__.py > --- a/Lib/importlib/test/__init__.py > +++ b/Lib/test/test_importlib/__init__.py > @@ -1,5 +1,6 @@ > import os > import sys > +from .. import support I don't know if I'm only speaking for myself, but I really have trouble parsing non-trivial relative imports, and I personally prefer when people use absolute imports (e.g. "from test import support"). cheers Antoine. -- Software development and contracting: http://pro.pitrou.net

8 years, 8 months

A new JIT compiler for a faster CPython?

by Victor Stinner

Hi, I would like to write yet another JIT compiler for CPython. Before writing anything, I would like your opinion because I don't know well other Python compilers. I also want to prepare a possible integration into CPython since the beginning of the project, or at least stay very close to the CPython project (and CPython developers!). I did not understand exactly why Unladen Swallow and psyco projects failed, so please tell me if you think that my project is going to fail too! == Why? == CPython is still the reference implementation, new features are first added to this implementation (ex: PyPy is not supporting Python 3 yet, but there is a project to support Python 3). Some projects still rely on low level properties of CPython, especially its C API (ex: numpy; PyPy has a cpyext module to emulate the CPython C API). A JIT is the most promising solution to speed up the main evaluation loop: using a JIT, it is possible to compile a function for a specific type on the fly and so enable deeper optimizations. psyco is no more maintained. It had its own JIT which is complex to maintain. For example, it is hard to port it to a new hardware. LLVM is fast and the next version will be faster. LLVM has a community, a documentation, a lot of tools and is active. There are many Python compilers which are very fast, but most of them only support a subset of Python or require to modify the code (ex: specify the type of all parameters and variables). For example, you cannot run Django with Shredskin. IMO PyPy is complex and hard to maintain. PyPy has a design completly different than CPython and is much faster and has a better memory footprint. I don't expect to be as fast as PyPy, just faster than CPython. == General idea == I don't want to replace CPython. This is an important point. All others Python compilers try to write something completly new, which is an huge task and is a problem to stay compatible with CPython. I would like to reuse as much as possible code of CPython and don't try to fight against the GIL or reference counting, but try to cooperate instead. I would like to use a JIT to generate specialized functions for a combinaison of arguments types. Specialization enables more optimizations. I would like to use LLVM because LLVM is an active project, has many developers and users, is fast and the next version will be faster! LLVM already supports common optimizations like inlining. My idea is to emit the same code than ceval.c from the bytecode to be fully compatible with CPython, and then write a JIT to optimize functions for a specific type. == Roadmap == -- Milestone 1: Proof of concept -- * Use the bytecode produced by CPython parser and compiler * Only compile a single function * Emit the same code than ceval.c using LLVM, but without tracing, exceptions nor signal handling (they will be added later) * Support compiling and calling the following functions: "def func(a, b): return a+b" The pymothoa project can be used as a base to implement quickly such proof of concept. -- Milestone 2: Specialized function for the int type -- * Use type annotation to generate specialized functions for the int type * Use C int with a guard detecting integer overflow to fallback on Python int -- Milestone 3: JIT -- * Depending on the type seen at runtime, recompile the function to generate specialized functions * Use guard to fallback to a generic implementation if the type is not the expected type * Drop maybe the code using function annotations At this step, we can start to benchmark to check if the (JIT) compiler is faster than CPython. -- Later (unsorted ideas) -- * Support exceptions * Full support of Python - classes - list comprehension - etc. * Optimizations: - avoid reference counting when possible - avoid temporary objects when possible - release the GIL when possible - inlining: should be very interesting with list comprehension - unroll loops? - lazy creation of the frame? * Use registers instead of a stack in the "evaluation loop"? * Add code to allow tracing and profiling * Add code to handle signals (pending calls) * Write a compiler using the AST, with a fallback to the bytecode? (would it be faster? easier or more complex to maintain?) * Test LLVM optimizers * Compile a whole module or even a whole program * Reduce memory footprint * Type annotation to help the optimizer? (with guards?) * "const" annotation to help the optimizer? (with guards?) * Support any build option of Python: - support Python 2 (2.5, 2.6, 2.7) and 3 (3.1, 3.2, 3.3, 3.4) - support narrow and wide mode: flag at runtime? - support debug and release mode: flag at runtime? - support 32 and 64 bits mode on Windows? == Other Python VM and compilers == -- Fully Python compliant -- * `PyPy <http://pypy.org/>`_ * `Jython <http://www.jython.org/>`_ based on the JVM * `IronPython <http://ironpython.net/>`_ based on the .NET VM * `Unladen Swallow <http://code.google.com/p/unladen-swallow/>`_ fork of CPython 2.6 using LLVM - `Unladen Swallow Retrospective <http://qinsb.blogspot.com.au/2011/03/unladen-swallow-retrospective.html>`_ - `PEP 3146 <http://python.org/dev/peps/pep-3146/>`_ * `psyco <http://psyco.sourceforge.net/>`_ (fully Python compliant?), no more maintained -- Subset of Python to C++ -- * `Nuitka <http://www.nuitka.net/pages/overview.html>`_ * `Python2C <http://strout.net/info/coding/python/ai/python2c.py>`_ * `Shedskin <http://code.google.com/p/shedskin/>`_ * `pythran <https://github.com/serge-sans-paille/pythran>`_ (no class, set, dict, exception, file handling, ...) -- Subset of Python -- * `pymothoa <http://code.google.com/p/pymothoa/>`_: use LLVM; don't support classes nor exceptions. * `unpython <http://code.google.com/p/unpython/>`_: Python to C * `Perthon <http://perthon.sourceforge.net/>`_: Python to Perl * `Copperhead <http://copperhead.github.com/>`_: Python to GPU (Nvidia) -- Language very close to Python -- * `Cython <http://www.cython.org/>`_: "Cython is a programming language based on Python, with extra syntax allowing for optional static type declarations." Based on `Pyrex <http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/>`_ == See also == * `Volunteer developed free-threaded cross platform virtual machines? <http://www.boredomandlaziness.org/2012/07/volunteer-supported-free-threaded…>`_ Victor Stinner

8 years, 8 months

Summary of Python tracker Issues

by Python tracker

ACTIVITY SUMMARY (2012-07-13 - 2012-07-20) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 3528 ( +8) closed 23658 (+55) total 27186 (+63) Open issues with patches: 1501 Issues opened (39) ================== #9374: urlparse should parse query and fragment for arbitrary schemes http://bugs.python.org/issue9374 reopened by eric.araujo #9803: IDLE closes with save while breakpoint open http://bugs.python.org/issue9803 reopened by serwy #15346: Tkinter dnd has no documentation http://bugs.python.org/issue15346 opened by weirdink13 #15347: IDLE - does not close if the debugger was active http://bugs.python.org/issue15347 opened by serwy #15348: IDLE - shell becomes unresponsive if debugger windows is close http://bugs.python.org/issue15348 opened by serwy #15350: {urllib,urllib.parse}.urlencode.__doc__ is unclear http://bugs.python.org/issue15350 opened by samwyse #15351: Add to unittest.TestCase support for using context managers http://bugs.python.org/issue15351 opened by cjerdonek #15352: importlib.h should be regenerated when the marshaling code cha http://bugs.python.org/issue15352 opened by meador.inge #15353: ld: library not found for -lpython3.3m because of wrong LDFLAG http://bugs.python.org/issue15353 opened by Jerry.Jacobs #15355: generator.__next__() docs should mention exception if already http://bugs.python.org/issue15355 opened by cjerdonek #15358: Test pkgutil.walk_packages in test_pkgutil instead of test_run http://bugs.python.org/issue15358 opened by ncoghlan #15359: Sockets support for CAN_BCM http://bugs.python.org/issue15359 opened by Thorney #15360: Behavior of assigning to __dict__ is not documented http://bugs.python.org/issue15360 opened by davide.rizzo #15362: pyport.h includes antiquated UTF handling for FreeBSD http://bugs.python.org/issue15362 opened by JohnSchneider #15363: Idle/tkinter ~x.py 'save as' fails. closes idle http://bugs.python.org/issue15363 opened by terry.reedy #15364: sysconfig confused by relative paths http://bugs.python.org/issue15364 opened by sbt #15365: Traceback reporting can fail if IO cannot be imported http://bugs.python.org/issue15365 opened by kristjan.jonsson #15369: pybench and test.pystone poorly documented http://bugs.python.org/issue15369 opened by flox #15370: test_runpy should include namespace package tests http://bugs.python.org/issue15370 opened by ncoghlan #15371: test_cmd_line_script should include namespace package tests http://bugs.python.org/issue15371 opened by ncoghlan #15373: copy.copy() does not properly copy os.environment http://bugs.python.org/issue15373 opened by htgoebel #15376: Refactor the test_runpy walk_package support code into a commo http://bugs.python.org/issue15376 opened by cjerdonek #15378: Underported Tools/unicode/comparecodecs.py http://bugs.python.org/issue15378 opened by storchaka #15379: Charmap decoding of no-BMP characters http://bugs.python.org/issue15379 opened by storchaka #15381: Optimize BytesIO to so less reallocations when written, simila http://bugs.python.org/issue15381 opened by eli.bendersky #15382: os.utime() mishandles some arguments http://bugs.python.org/issue15382 opened by Arfrever #15388: SAX parse (ExpatParser) leaks file handle when given filename http://bugs.python.org/issue15388 opened by mth #15389: PEP 3121, 384 refactoring applied to curses module http://bugs.python.org/issue15389 opened by Robin.Schreiber #15390: PEP 3121, 384 refactoring applied to datetime module http://bugs.python.org/issue15390 opened by Robin.Schreiber #15392: Create a unittest framework for IDLE http://bugs.python.org/issue15392 opened by terry.reedy #15393: JSONDecoder.raw_decode breaks on leading whitespace http://bugs.python.org/issue15393 opened by aalien #15397: Unbinding of methods http://bugs.python.org/issue15397 opened by mstefanro #15398: intermittence on UnicodeFileTests.test_rename at test_pep277 o http://bugs.python.org/issue15398 opened by flavio.ribeiro #15402: Correct __sizeof__ support for struct http://bugs.python.org/issue15402 opened by storchaka #15403: Refactor package creation support code into a common location http://bugs.python.org/issue15403 opened by cjerdonek #15406: Deprecation Warnings fixes on test suite http://bugs.python.org/issue15406 opened by flavio.ribeiro #15407: CSV parser fails to iterate properly on 2.6.6 http://bugs.python.org/issue15407 opened by sleepycal #15408: os.fork/os.popen behaviour change between 2.7 and 3.2 http://bugs.python.org/issue15408 opened by doerwalter #1492704: distinct error type if shutil.copyfile() fails because of src http://bugs.python.org/issue1492704 reopened by hynek Most recent 15 issues with no replies (15) ========================================== #15408: os.fork/os.popen behaviour change between 2.7 and 3.2 http://bugs.python.org/issue15408 #15390: PEP 3121, 384 refactoring applied to datetime module http://bugs.python.org/issue15390 #15388: SAX parse (ExpatParser) leaks file handle when given filename http://bugs.python.org/issue15388 #15371: test_cmd_line_script should include namespace package tests http://bugs.python.org/issue15371 #15370: test_runpy should include namespace package tests http://bugs.python.org/issue15370 #15362: pyport.h includes antiquated UTF handling for FreeBSD http://bugs.python.org/issue15362 #15355: generator.__next__() docs should mention exception if already http://bugs.python.org/issue15355 #15353: ld: library not found for -lpython3.3m because of wrong LDFLAG http://bugs.python.org/issue15353 #15350: {urllib,urllib.parse}.urlencode.__doc__ is unclear http://bugs.python.org/issue15350 #15348: IDLE - shell becomes unresponsive if debugger windows is close http://bugs.python.org/issue15348 #15347: IDLE - does not close if the debugger was active http://bugs.python.org/issue15347 #15346: Tkinter dnd has no documentation http://bugs.python.org/issue15346 #15340: OSError with "import random" when /dev/urandom doesn't exist ( http://bugs.python.org/issue15340 #15336: Argparse required arguments incorrectly displayed as optional http://bugs.python.org/issue15336 #15326: --random does not work for regrtest http://bugs.python.org/issue15326 Most recent 15 issues waiting for review (15) ============================================= #15406: Deprecation Warnings fixes on test suite http://bugs.python.org/issue15406 #15403: Refactor package creation support code into a common location http://bugs.python.org/issue15403 #15402: Correct __sizeof__ support for struct http://bugs.python.org/issue15402 #15397: Unbinding of methods http://bugs.python.org/issue15397 #15393: JSONDecoder.raw_decode breaks on leading whitespace http://bugs.python.org/issue15393 #15390: PEP 3121, 384 refactoring applied to datetime module http://bugs.python.org/issue15390 #15389: PEP 3121, 384 refactoring applied to curses module http://bugs.python.org/issue15389 #15382: os.utime() mishandles some arguments http://bugs.python.org/issue15382 #15381: Optimize BytesIO to so less reallocations when written, simila http://bugs.python.org/issue15381 #15379: Charmap decoding of no-BMP characters http://bugs.python.org/issue15379 #15378: Underported Tools/unicode/comparecodecs.py http://bugs.python.org/issue15378 #15373: copy.copy() does not properly copy os.environment http://bugs.python.org/issue15373 #15365: Traceback reporting can fail if IO cannot be imported http://bugs.python.org/issue15365 #15364: sysconfig confused by relative paths http://bugs.python.org/issue15364 #15363: Idle/tkinter ~x.py 'save as' fails. closes idle http://bugs.python.org/issue15363 Top 10 most discussed issues (10) ================================= #1492704: distinct error type if shutil.copyfile() fails because of src http://bugs.python.org/issue1492704 21 msgs #15364: sysconfig confused by relative paths http://bugs.python.org/issue15364 17 msgs #15398: intermittence on UnicodeFileTests.test_rename at test_pep277 o http://bugs.python.org/issue15398 14 msgs #15381: Optimize BytesIO to so less reallocations when written, simila http://bugs.python.org/issue15381 11 msgs #15352: importlib.h should be regenerated when the marshaling code cha http://bugs.python.org/issue15352 10 msgs #15301: os.chown: OverflowError: Python int too large to convert to C http://bugs.python.org/issue15301 9 msgs #14988: _elementtree: Raise ImportError when importing of pyexpat fail http://bugs.python.org/issue14988 8 msgs #15379: Charmap decoding of no-BMP characters http://bugs.python.org/issue15379 7 msgs #15393: JSONDecoder.raw_decode breaks on leading whitespace http://bugs.python.org/issue15393 7 msgs #4130: Intel icc 9.1 does not support __int128_t used by ctypes http://bugs.python.org/issue4130 6 msgs Issues closed (53) ================== #6068: ctypes is not correctly handling bitfields backed by 64 bit in http://bugs.python.org/issue6068 closed by meador.inge #6257: Idle terminates on source save while debugging http://bugs.python.org/issue6257 closed by serwy #6493: Can not set value for structure members larger than 32 bits http://bugs.python.org/issue6493 closed by meador.inge #9254: __import__ docstring should recommend importlib.import_module( http://bugs.python.org/issue9254 closed by python-dev #9458: xml.etree.ElementTree.ElementTree.write(): encoding handling p http://bugs.python.org/issue9458 closed by eli.bendersky #9591: kqueue not reporting EOF under certain circumstances http://bugs.python.org/issue9591 closed by Volodymyr.Kostyrko #13823: xml.etree.ElementTree.ElementTree.write - argument checking http://bugs.python.org/issue13823 closed by eli.bendersky #14600: Change ImportError reference handling, naming http://bugs.python.org/issue14600 closed by brian.curtin #14635: telnetlib uses select instead of poll - limited to FD_SETSIZE http://bugs.python.org/issue14635 closed by gregory.p.smith #14790: use packaging in setup.py http://bugs.python.org/issue14790 closed by loewis #15169: Clear C code under PyImport_ExecCodeModuleObject() http://bugs.python.org/issue15169 closed by brett.cannon #15170: Fix 64-bit building for buildbot scripts (2.7) http://bugs.python.org/issue15170 closed by skrah #15171: Fix 64-bit building for buildbot scripts (3.2) http://bugs.python.org/issue15171 closed by skrah #15180: Cryptic traceback from os.path.join when mixing str & bytes http://bugs.python.org/issue15180 closed by hynek #15230: runpy.run_path doesn't set __package__ correctly http://bugs.python.org/issue15230 closed by python-dev #15238: shutil.copystat should copy Linux extended attributes http://bugs.python.org/issue15238 closed by larry #15297: pkgutil.iter_importers() includes an ImpImporter http://bugs.python.org/issue15297 closed by cjerdonek #15299: pkgutil.ImpImporter(None).iter_modules() does not search sys.p http://bugs.python.org/issue15299 closed by ncoghlan #15307: Patch for --symlink support in pyvenv with framework python http://bugs.python.org/issue15307 closed by python-dev #15314: Use importlib instead of pkgutil in runpy http://bugs.python.org/issue15314 closed by ncoghlan #15334: access denied for HKEY_PERFORMANCE_DATA http://bugs.python.org/issue15334 closed by pitrou #15337: The cmd module incorrectly lists "help" as an undocumented com http://bugs.python.org/issue15337 closed by rhettinger #15338: test_UNC_path failure in test_import http://bugs.python.org/issue15338 closed by pitrou #15343: "pydoc -w <package>" writes out page with empty "Package Conte http://bugs.python.org/issue15343 closed by ncoghlan #15345: HOWTOs Argparse tutorial - code example raises SyntaxError http://bugs.python.org/issue15345 closed by ezio.melotti #15349: SyntaxError b0rked grammar http://bugs.python.org/issue15349 closed by serwy #15354: _PyObject_LengthHint only accepts longs http://bugs.python.org/issue15354 closed by python-dev #15356: '\xa0' isspace returns true while compiling python on solaris http://bugs.python.org/issue15356 closed by r.david.murray #15357: Deprecate redundant pieces of pkgutil http://bugs.python.org/issue15357 closed by ncoghlan #15361: venv's Activate.ps1 causes broken prompt with powershell http://bugs.python.org/issue15361 closed by python-dev #15366: venv assumes header files in sys._home + '/Include' http://bugs.python.org/issue15366 closed by python-dev #15367: build_ext in a venv on Windows assumes pyconfig.h in sys.exec_ http://bugs.python.org/issue15367 closed by python-dev #15368: bytecode generation is not deterministic http://bugs.python.org/issue15368 closed by meador.inge #15372: Python is missing alternative for common quoting character http://bugs.python.org/issue15372 closed by r.david.murray #15374: venv environment variable should follow the conventions http://bugs.python.org/issue15374 closed by pitrou #15375: Trivial for fix in the subprocess documentation http://bugs.python.org/issue15375 closed by r.david.murray #15377: os.path.join() error misleading with path1=None http://bugs.python.org/issue15377 closed by hynek #15380: bytes/str mismatch in distribute http://bugs.python.org/issue15380 closed by ned.deily #15383: Autocompletion crashes Python if the __builtins__ module canno http://bugs.python.org/issue15383 closed by r.david.murray #15384: pkgutil importlib migration test was failing on Windows buildb http://bugs.python.org/issue15384 closed by ncoghlan #15385: Behaviour change in runpy for __file__ attributes between 3.2 http://bugs.python.org/issue15385 closed by ncoghlan #15386: Still getting two copies of importlib._bootstrap http://bugs.python.org/issue15386 closed by python-dev #15387: pkgutil.walk_packages is using a deprecated API http://bugs.python.org/issue15387 closed by python-dev #15391: Add bitlength function to the math module http://bugs.python.org/issue15391 closed by jcea #15394: memory leak in PyModule_Create2 http://bugs.python.org/issue15394 closed by meador.inge #15395: memory leaks in selectmodule.c http://bugs.python.org/issue15395 closed by python-dev #15396: memory leak in tkinter http://bugs.python.org/issue15396 closed by python-dev #15399: processName key is un-/mis-documented in 2.6 and up http://bugs.python.org/issue15399 closed by python-dev #15400: int('12345L', 10) raises ValueError http://bugs.python.org/issue15400 closed by jcea #15401: Typo in inspect.getclosurevars docstring http://bugs.python.org/issue15401 closed by meador.inge #15404: Refleak in PyMethodObject repr http://bugs.python.org/issue15404 closed by asvetlov #15405: Invitation to connect on LinkedIn http://bugs.python.org/issue15405 closed by nadeem.vawda #1767933: Badly formed XML using etree and utf-16 http://bugs.python.org/issue1767933 closed by python-dev

8 years, 9 months

Runtime forks through monkeypatching (was Re: Use function names instead of functions for os.supports_dir_fd?)

by Nick Coghlan

On Fri, Jul 20, 2012 at 4:48 AM, Antoine Pitrou <solipsis(a)pitrou.net> wrote: > On Wed, 18 Jul 2012 02:26:14 +0200 > Victor Stinner <victor.stinner(a)gmail.com> wrote: > >> >> Monkey patching is a common practice in Python. test_os.py replaces >> >> os.exec*() functions temporary for example. >> > >> > Perhaps for testing, but I don't think monkey-patching is common in >> > production code. Perhaps you are thinking of Ruby :) >> >> The gevent library does monkey-patch os.fork (and time.sleep and many >> other functions), but gevent is maybe not ready for production? :-) > > Extensive monkey-patching of core OS functions would certainly make me > weary of using such a third-party library, even if it claims to be > "serious". gevent is one of the well-behaved players in that game - it doesn't monkeypatch anything unless you explicitly tell it to [1]. You do need to have some idea what you're doing if using the monkey patching aspect (as not all third party modules will behave well in that case - just as not all third party modules behave well on other Python implementations), but you can also just use gevent as a normal library without monkey patching anything (however, adhering strictly to that rule leaves you in the same situation as Twisted, where modules that use the standard blocking APIs are completely useless to you. Thus, if your goal is "pure async" code without any need for monkeypatching, Twisted is a much better bet than gevent, just because it's been playing that game for longer and has more fully async aware components available). That's why I call features like the gevent.monkey module "runtime forks" - when they're implemented well, they effectively give you two Python implementations without the complexity of actually having parallel Python installations (compare the complexity of using Stackless, a traditional fork, vs gevent.monkey, a runtime fork - there's a reason the core greenlet component was lifted out of Stackless and made available as a CPython extension module). Applications can decide at startup which Python variant (standard CPython, or CPython+gevent) they want to use without affecting other Python applications installed on the system and without needing to duplicating all installed Python modules into a second interpreter. It's only when libraries monkeypatch standard interfaces *implicitly* as a side effect of import that you start to get seriously broken "action at a distance" behaviour. Or, as happened with distutils, you get a third party project (setuptools) effectively freezing implementation details of the stdlib version - when monkeypatching starts delving too deep into implementation details, it's time to switch to a traditional fork and start figuring out better ways to solve the problem (which process is now grinding away on the distutils2/packaging front). Monkeypatching is a powerful tool - used well, it lets you achieve impressive things that would otherwise be substantially more difficult to either create in the first place or maintain in the long run. As with many powerful tools though, used badly it can lead to a lot of pain. Static languages like Java decide the risk of action at a distance from monkeypatching isn't worth the power it gives you to work around issues in third party libraries without resorting to creating a traditional fork, so they strictly enforce access controls and have completely closed class definitions. Python permits monkeypatching at a language level (because when you need it, you really need it), but discourages it a social level (because there is usually a less magical technique, such as creating and using a wrapper or subclass, that will achieve the desired effect). Cheers, Nick. [1] http://www.gevent.org/intro.html#monkey-patching P.S. for a non-monkeypatching approach to reusing a synchronous library when creating an asynchronous equivalent, take a look at 10gen's attempt to create a *maintainable* Tornado-compatible async wrapper for PyMongo by using greenlets ([2][3]). It's a clever idea, but still a lot more work than just using PyMongo itself with gevent if you're only interested in the "async" part rather than the "Tornado-compatible" part. [2] http://emptysquare.net/blog/motor-internals-how-i-asynchronized-a-synchrono… [3] http://emptysquare.net/blog/motor-four-strategies-for-maintainability/ -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

8 years, 9 months

Unbinding of methods

by M Stefan

Hey, As part of pickle4, I found it interesting to add the possibility of pickling bound functions (instance methods). This is done by pickling f.__self__ and f.__func__ separately, and then adding a BIND opcode to tie them together. While this appears to work fine for python methods (non-builtin), some issues arise with builtins. These are partly caused because not all builtin function types support __func__, partly because not all of them fill __module__ when they should and partly because there are many (7) types a function can actually have: ClassMethodDescriptorType = type(??) BuiltinFunctionType = type(len) FunctionType = type(f) MethodType = type(A().f()) MethodDescriptorType = type(list.append) WrapperDescriptorType = type(list.__add__) MethodWrapperType = type([].__add__) AllFunctionTypes = (ClassMethodDescriptorType, BuiltinFunctionType, FunctionType, MethodType, MethodDescriptorType, WrapperDescriptorType, MethodWrapperType) repr(AllFunctionTypes) = ( <class 'classmethod_descriptor'>, <class 'builtin_function_or_method'>, <class 'function'>, <class 'method'>, <class 'method_descriptor'>, <class 'wrapper_descriptor'>, <class 'method-wrapper'>) I have created a patch at [1], which adds __func__ to some other function types, as well as: 1) adds AllFunctionTypes etc. to Lib/types.py 2) inspect.isanyfunction(), inspect.isanyboundfunction(), inspect.isanyunboundfunction() 3) functools.unbind Note that I am not knowledgeable of cpython internals and therefore the patch needs to be carefully reviewed. Possible issues: Should classmethods be considered bound or unbound? If cm is a classmethod, then should cm.__func__.__self__ = cm.__self__ or cm.__func__.__self__ = None? Currently does the latter: >>> cm.__self__, hasattr(cm,'__self__'), hasattr(cm.__func__, '__self__') (<class '__main__.A'>, True, False) This requires treating classmethods separately when pickling, so I'm not sure if this is ideal. Let me know if I should have opened an issue instead. I look forward to hearing your opinions/suggestions on this matter. Regards, Stefan M [1] https://gist.github.com/3145210

8 years, 9 months

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

Python-Dev July 2012