Mailman 3 November 2012 - Python-Dev

Summary of Python tracker Issues
by Python tracker 23 Nov '12

23 Nov '12

ACTIVITY SUMMARY (2012-11-16 - 2012-11-23) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 3799 ( -2) closed 24518 (+54) total 28317 (+52) Open issues with patches: 1656 Issues opened (37) ================== #8627: Unchecked PyErr_WarnPy3k return value in Objects/typeobject.c http://bugs.python.org/issue8627 reopened by mark.dickinson #16306: Multiple error line for unknown command line parameter http://bugs.python.org/issue16306 reopened by pitrou #16490: "inspect.getargspec()" and "inspect.getcallargs()" don't work http://bugs.python.org/issue16490 opened by jcea #16491: try-except-raise-bug http://bugs.python.org/issue16491 opened by Jocjo #16492: Add a load_parents argument to importlib.find_loader() http://bugs.python.org/issue16492 opened by brett.cannon #16494: Add a method on importlib.SourceLoader for creating bytecode f http://bugs.python.org/issue16494 opened by brett.cannon #16495: bytes_decode() unnecessarily examines encoding http://bugs.python.org/issue16495 opened by chris.jerdonek #16496: Simplify and optimize random_seed() http://bugs.python.org/issue16496 opened by serhiy.storchaka #16499: CLI option for isolated mode http://bugs.python.org/issue16499 opened by christian.heimes #16500: Add an 'afterfork' module http://bugs.python.org/issue16500 opened by christian.heimes #16501: deprecate RISCOS "support" http://bugs.python.org/issue16501 opened by pitrou #16504: IDLE - fatal error when opening a file with certain tokenizing http://bugs.python.org/issue16504 opened by serwy #16507: Patch selectmodule.c to support WSAPoll on Windows http://bugs.python.org/issue16507 opened by trent #16508: include the "object" type in the lists of documented types http://bugs.python.org/issue16508 opened by chris.jerdonek #16509: sqlite3 docs do not explain check_same_thread http://bugs.python.org/issue16509 opened by strcat #16510: Using appropriate checks in tests http://bugs.python.org/issue16510 opened by serhiy.storchaka #16511: IDLE configuration file: blank height and width fields trip up http://bugs.python.org/issue16511 opened by doanviettrung #16512: imghdr doesn't support jpegs with an ICC profile http://bugs.python.org/issue16512 opened by joril #16515: TypeError message incorrect for max() with one keyword arg http://bugs.python.org/issue16515 opened by chris.jerdonek #16516: argparse types (and actions) must be hashable http://bugs.python.org/issue16516 opened by jnothman #16518: add "buffer protocol" to glossary http://bugs.python.org/issue16518 opened by chris.jerdonek #16519: site.venv() should use abspath(executable) http://bugs.python.org/issue16519 opened by christian.heimes #16520: subprocess.Popen() TypeError message incorrect without args ar http://bugs.python.org/issue16520 opened by chris.jerdonek #16521: logging.basicConfig creates empty file when using handlers http://bugs.python.org/issue16521 opened by Jovik #16523: attrgetter and itemgetter signatures in docs need cleanup http://bugs.python.org/issue16523 opened by r.david.murray #16524: File access not always working with Python for Windows 32 bits http://bugs.python.org/issue16524 opened by Antoine.Brodin #16525: wave file module does not support 32bit float format http://bugs.python.org/issue16525 opened by Sebastian.Kraft #16526: Python does not cross compile properly http://bugs.python.org/issue16526 opened by Lothsahn #16529: Compiler error when trying to compile ceval.c on OpenSUSE 11.3 http://bugs.python.org/issue16529 opened by lemburg #16530: documentation of os.wait3 http://bugs.python.org/issue16530 opened by quiver #16531: Allow IPNetwork to take a tuple http://bugs.python.org/issue16531 opened by pitrou #16532: AMD64 Windows 7 build failures http://bugs.python.org/issue16532 opened by pitrou #16533: HPUX: Unable to fork() in thread http://bugs.python.org/issue16533 opened by skrah #16534: test_float failure on IA64 (HPUX) http://bugs.python.org/issue16534 opened by skrah #16535: json encoder unable to handle decimal http://bugs.python.org/issue16535 opened by eric.araujo #16537: setup.py throws a ValueError when self.extensions is empty http://bugs.python.org/issue16537 opened by jhosmer #16540: Make itertools count, cycle, and repeat objects subscriptable http://bugs.python.org/issue16540 opened by neil.g Most recent 15 issues with no replies (15) ========================================== #16533: HPUX: Unable to fork() in thread http://bugs.python.org/issue16533 #16530: documentation of os.wait3 http://bugs.python.org/issue16530 #16523: attrgetter and itemgetter signatures in docs need cleanup http://bugs.python.org/issue16523 #16521: logging.basicConfig creates empty file when using handlers http://bugs.python.org/issue16521 #16519: site.venv() should use abspath(executable) http://bugs.python.org/issue16519 #16518: add "buffer protocol" to glossary http://bugs.python.org/issue16518 #16516: argparse types (and actions) must be hashable http://bugs.python.org/issue16516 #16509: sqlite3 docs do not explain check_same_thread http://bugs.python.org/issue16509 #16508: include the "object" type in the lists of documented types http://bugs.python.org/issue16508 #16494: Add a method on importlib.SourceLoader for creating bytecode f http://bugs.python.org/issue16494 #16492: Add a load_parents argument to importlib.find_loader() http://bugs.python.org/issue16492 #16486: Add context manager support to aifc module http://bugs.python.org/issue16486 #16472: Distutils+mingw links agains msvcr90, while python27.dll is li http://bugs.python.org/issue16472 #16463: test_timeout failure on the RHEL buildbot http://bugs.python.org/issue16463 #16450: test_missing_localfile masks problems in urlopen http://bugs.python.org/issue16450 Most recent 15 issues waiting for review (15) ============================================= #16537: setup.py throws a ValueError when self.extensions is empty http://bugs.python.org/issue16537 #16515: TypeError message incorrect for max() with one keyword arg http://bugs.python.org/issue16515 #16512: imghdr doesn't support jpegs with an ICC profile http://bugs.python.org/issue16512 #16511: IDLE configuration file: blank height and width fields trip up http://bugs.python.org/issue16511 #16510: Using appropriate checks in tests http://bugs.python.org/issue16510 #16507: Patch selectmodule.c to support WSAPoll on Windows http://bugs.python.org/issue16507 #16504: IDLE - fatal error when opening a file with certain tokenizing http://bugs.python.org/issue16504 #16500: Add an 'afterfork' module http://bugs.python.org/issue16500 #16499: CLI option for isolated mode http://bugs.python.org/issue16499 #16496: Simplify and optimize random_seed() http://bugs.python.org/issue16496 #16495: bytes_decode() unnecessarily examines encoding http://bugs.python.org/issue16495 #16488: Add context manager support to epoll object http://bugs.python.org/issue16488 #16487: Allow ssl certificates to be speficfied from memory rather tha http://bugs.python.org/issue16487 #16486: Add context manager support to aifc module http://bugs.python.org/issue16486 #16485: FD leaks in aifc module http://bugs.python.org/issue16485 Top 10 most discussed issues (10) ================================= #16475: Support object instancing and recursion in marshal http://bugs.python.org/issue16475 25 msgs #16496: Simplify and optimize random_seed() http://bugs.python.org/issue16496 18 msgs #15031: Split .pyc parsing from module loading http://bugs.python.org/issue15031 13 msgs #16500: Add an 'afterfork' module http://bugs.python.org/issue16500 13 msgs #16306: Multiple error line for unknown command line parameter http://bugs.python.org/issue16306 10 msgs #16499: CLI option for isolated mode http://bugs.python.org/issue16499 10 msgs #16480: pyvenv 3.3 fails to create symlinks for <virtualenv>/local/{bi http://bugs.python.org/issue16480 9 msgs #16511: IDLE configuration file: blank height and width fields trip up http://bugs.python.org/issue16511 8 msgs #12848: pickle.py treats 32bit lengths as signed, but _pickle.c as uns http://bugs.python.org/issue12848 7 msgs #8627: Unchecked PyErr_WarnPy3k return value in Objects/typeobject.c http://bugs.python.org/issue8627 6 msgs Issues closed (44) ================== #1160: Medium size regexp crashes python http://bugs.python.org/issue1160 closed by pitrou #3580: failures in test_os http://bugs.python.org/issue3580 closed by pitrou #4182: warnings.warn shows the wrong filename and line number for sta http://bugs.python.org/issue4182 closed by brett.cannon #6308: termios fix for QNX breaks HP-UX http://bugs.python.org/issue6308 closed by skrah #6526: importlib.import_module affects permissions of .pyc files subs http://bugs.python.org/issue6526 closed by brett.cannon #7782: new test for test_iter.py http://bugs.python.org/issue7782 closed by ezio.melotti #9761: stale .pyc files aren't cleaned out http://bugs.python.org/issue9761 closed by brett.cannon #10966: eliminate use of ImportError implicitly representing SkipTest http://bugs.python.org/issue10966 closed by brett.cannon #11679: readline interferes with characters beginning with byte \xe9 http://bugs.python.org/issue11679 closed by takluyver #11981: dupe self.fp.tell() in zipfile.ZipFile.writestr http://bugs.python.org/issue11981 closed by ezio.melotti #12005: modulo result of Decimal differs from float/int http://bugs.python.org/issue12005 closed by mark.dickinson #12614: Allow to explicitly set the method of urllib.request.Request http://bugs.python.org/issue12614 closed by orsenthil #13538: Improve doc for str(bytesobject) http://bugs.python.org/issue13538 closed by chris.jerdonek #14313: zipfile should raise an exception for unsupported compression http://bugs.python.org/issue14313 closed by ezio.melotti #14525: ia64-hp-hpux11.31 won't compile 2.7 without -D_TERMIOS_INCLUDE http://bugs.python.org/issue14525 closed by skrah #14631: Instance methods and WeakRefs don't mix. http://bugs.python.org/issue14631 closed by pitrou #15627: Add a method to importlib.abc.SourceLoader for converting sour http://bugs.python.org/issue15627 closed by brett.cannon #16053: "strict" parameter is not documented in csv module http://bugs.python.org/issue16053 closed by ezio.melotti #16157: Irrelevant references to Misc/News http://bugs.python.org/issue16157 closed by ezio.melotti #16213: Expose private functions in marshal used by importlib http://bugs.python.org/issue16213 closed by brett.cannon #16215: Possible double memory free in str.replace http://bugs.python.org/issue16215 closed by pitrou #16240: Document a way to escape metacharacters in glob/fnmatch http://bugs.python.org/issue16240 closed by ezio.melotti #16408: FD leaks in zipfile http://bugs.python.org/issue16408 closed by pitrou #16451: Remove duplication between slice_indices and compute_slice_ind http://bugs.python.org/issue16451 closed by mark.dickinson #16461: wave module: wrong integer format http://bugs.python.org/issue16461 closed by python-dev #16470: Backport set and dictionary comprehensions in tutorial to 2.7 http://bugs.python.org/issue16470 closed by ezio.melotti #16489: importlib find_loader returns a loader for a non existent modu http://bugs.python.org/issue16489 closed by brett.cannon #16493: Document the 'optimize' argument to compile() http://bugs.python.org/issue16493 closed by ezio.melotti #16497: zipimport.zipimporter.find_module() does not work with dotted http://bugs.python.org/issue16497 closed by brett.cannon #16498: Unwanted link between volatile and shelve storage http://bugs.python.org/issue16498 closed by serhiy.storchaka #16502: PEP 305: eaten backslashes http://bugs.python.org/issue16502 closed by ezio.melotti #16503: Unclear documentation regarding apply(), 'extended call syntax http://bugs.python.org/issue16503 closed by ezio.melotti #16505: Drop Py_TPFLAGS_INT_SUBCLASS http://bugs.python.org/issue16505 closed by python-dev #16506: devguide should have table of contents http://bugs.python.org/issue16506 closed by chris.jerdonek #16513: SGMLParser processing <tr> which include two <a> will have pro http://bugs.python.org/issue16513 closed by ezio.melotti #16514: Cryptic traceback when sys.path[0] is None http://bugs.python.org/issue16514 closed by barry #16517: address merge conflicts in devguide null-merge instructions http://bugs.python.org/issue16517 closed by chris.jerdonek #16522: Add 'FAIL_FAST' flag to doctest http://bugs.python.org/issue16522 closed by r.david.murray #16527: (very) long list of elif causes segfault http://bugs.python.org/issue16527 closed by benjamin.peterson #16528: 3.2 docs not updating on docs.python.org http://bugs.python.org/issue16528 closed by r.david.murray #16536: test_cmd_line: failure on Ubuntu Shared http://bugs.python.org/issue16536 closed by ezio.melotti #16538: The docs doesn't describe MAKE_CLOSURE correctly http://bugs.python.org/issue16538 closed by asvetlov #667770: import C API mess http://bugs.python.org/issue667770 closed by brett.cannon #16539: Turn off 'No handlers could be found for logger' message http://bugs.python.org/issue16539 closed by vinay.sajip

1 0

Re: [Python-Dev] [Python-checkins] devguide: Add instructions for handling merge conflicts during null merges.
by Chris Jerdonek 21 Nov '12

21 Nov '12

On Wed, Nov 21, 2012 at 6:07 PM, chris.jerdonek <python-checkins(a)python.org> wrote: > http://hg.python.org/devguide/rev/78a69b929ab7 > changeset: 573:78a69b929ab7 > user: Chris Jerdonek <chris.jerdonek(a)gmail.com> > date: Wed Nov 21 18:04:35 2012 -0800 > summary: > Add instructions for handling merge conflicts during null merges. This was for issue #16517: http://bugs.python.org/issue16517 --Chris > > files: > committing.rst | 11 +++++++++-- > 1 files changed, 9 insertions(+), 2 deletions(-) > > > diff --git a/committing.rst b/committing.rst > --- a/committing.rst > +++ b/committing.rst > @@ -306,17 +306,24 @@ > # Fix any conflicts; compile; run the test suite > hg commit > > +.. index:: null merging > + > .. note:: > - *If the patch shouldn't be ported* from Python 3.3 to Python 3.4, you must > - also make it explicit: merge the changes but revert them before committing:: > + If the patch should *not* be ported from Python 3.3 to Python 3.4, you must > + also make this explicit by doing a *null merge*: merge the changes but > + revert them before committing:: > > hg update default > hg merge 3.3 > hg revert -ar default > + hg resolve -am # needed only if the merge created conflicts > hg commit > > This is necessary so that the merge gets recorded; otherwise, somebody > else will have to make a decision about your patch when they try to merge. > + (Using a three-way merge tool generally makes the ``hg resolve`` step > + in the above unnecessary; also see `this bug report > + <http://bz.selenic.com/show_bug.cgi?id=2706>`__.) > > When you have finished your porting work (you can port several patches one > after another in your local repository), you can push **all** outstanding > > -- > Repository URL: http://hg.python.org/devguide > > _______________________________________________ > Python-checkins mailing list > Python-checkins(a)python.org > http://mail.python.org/mailman/listinfo/python-checkins >

1 0

Re: [Python-Dev] [Python-checkins] cpython (3.3): - Issue #16514: Fix regression causing a traceback when sys.path[0] is None
by Barry Warsaw 21 Nov '12

21 Nov '12

On Nov 20, 2012, at 05:35 PM, Terry Reedy wrote: >On 11/20/2012 3:35 PM, barry.warsaw wrote: > >> for entry in path: >> + if not isinstance(entry, (str, bytes)): >> + continue > >Given that a non-(str,bytes) entry could indicate a programming error, should >a warning be emitted before continuing? That's not what happens in Python 3.2. In fact, this bug report was triggered by someone who was already inserting None to sys.path[0]. It is silently ignored in Python 3.2, but tracebacked in 3.3. Cheers, -Barry

2 1

operator.attrgetter(attr[, args...]) etc.
by Leo 21 Nov '12

21 Nov '12

Sorry the python issue tracker seems broken (I cannot log in). So I am posting it here. In the doc: operator.attrgetter(attr[, args...]) operator.itemgetter(item[, args...]) operator.methodcaller(name[, args...]) The signatures of these functions seem confusing. ARGS is not documented and does not match the python implementation in the doc. Leo

3 2

Re: [Python-Dev] [Python-checkins] cpython (3.3): - Issue #16514: Fix regression causing a traceback when sys.path[0] is None
by Brett Cannon 20 Nov '12

20 Nov '12

Should you also insert None into sys.path_importer_cache to signify there is no finder for the path entry? I guess the real problem with that is there is no guarantee the path entry is hashable, so that probably won't work. So nevermind. =) On Tue, Nov 20, 2012 at 3:35 PM, barry.warsaw <python-checkins(a)python.org>wrote: > http://hg.python.org/cpython/rev/291406748217 > changeset: 80529:291406748217 > branch: 3.3 > parent: 80527:4537dd27b2dc > user: Barry Warsaw <barry(a)python.org> > date: Tue Nov 20 15:22:51 2012 -0500 > summary: > - Issue #16514: Fix regression causing a traceback when sys.path[0] is > None > (actually, any non-string or non-bytes type). > > files: > Doc/library/sys.rst | 4 +- > Doc/reference/import.rst | 24 +- > Lib/importlib/_bootstrap.py | 2 + > Lib/test/test_importlib/import_/test_path.py | 25 +- > Misc/NEWS | 3 + > Python/importlib.h | 130 +++++---- > 6 files changed, 111 insertions(+), 77 deletions(-) > > > diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst > --- a/Doc/library/sys.rst > +++ b/Doc/library/sys.rst > @@ -783,7 +783,9 @@ > current directory first. Notice that the script directory is inserted > *before* > the entries inserted as a result of :envvar:`PYTHONPATH`. > > - A program is free to modify this list for its own purposes. > + A program is free to modify this list for its own purposes. Only > strings > + and bytes should be added to :data:`sys.path`; all other data types are > + ignored during import. > > > .. seealso:: > diff --git a/Doc/reference/import.rst b/Doc/reference/import.rst > --- a/Doc/reference/import.rst > +++ b/Doc/reference/import.rst > @@ -540,7 +540,10 @@ > implementation-specific defaults. Entries in :data:`sys.path` can name > directories on the file system, zip files, and potentially other > "locations" > (see the :mod:`site` module) that should be searched for modules, such as > -URLs, or database queries. > +URLs, or database queries. Only strings and bytes should be present on > +:data:`sys.path`; all other data types are ignored. The encoding of bytes > +entries is determined by the individual :term:`path entry finders <path > entry > +finder>`. > > The :term:`path based finder` is a :term:`meta path finder`, so the import > machinery begins the :term:`import path` search by calling the path > @@ -563,14 +566,17 @@ > the path based finder to perform the path entry search again [#fnpic]_. > > If the path entry is not present in the cache, the path based finder > iterates > -over every callable in :data:`sys.path_hooks`. Each of the > -:term:`path entry hooks <path entry hook>` in this list is called with a > -single argument, the path entry to be searched. This callable may either > -return a :term:`path entry finder` that can handle the path entry, or it > may > -raise :exc:`ImportError`. > -An :exc:`ImportError` is used by the path based finder to signal that the > hook > -cannot find a :term:`path entry finder` for that :term:`path entry`. The > -exception is ignored and :term:`import path` iteration continues. > +over every callable in :data:`sys.path_hooks`. Each of the :term:`path > entry > +hooks <path entry hook>` in this list is called with a single argument, > the > +path entry to be searched. This callable may either return a :term:`path > +entry finder` that can handle the path entry, or it may raise > +:exc:`ImportError`. An :exc:`ImportError` is used by the path based > finder to > +signal that the hook cannot find a :term:`path entry finder` for that > +:term:`path entry`. The exception is ignored and :term:`import path` > +iteration continues. The hook should expect either a string or bytes > object; > +the encoding of bytes objects is up to the hook (e.g. it may be a file > system > +encoding, UTF-8, or something else), and if the hook cannot decode the > +argument, it should raise :exc:`ImportError`. > > If :data:`sys.path_hooks` iteration ends with no :term:`path entry finder` > being returned, then the path based finder's :meth:`find_module()` method > diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py > --- a/Lib/importlib/_bootstrap.py > +++ b/Lib/importlib/_bootstrap.py > @@ -1281,6 +1281,8 @@ > # the list of paths that will become its __path__ > namespace_path = [] > for entry in path: > + if not isinstance(entry, (str, bytes)): > + continue > finder = cls._path_importer_cache(entry) > if finder is not None: > if hasattr(finder, 'find_loader'): > diff --git a/Lib/test/test_importlib/import_/test_path.py > b/Lib/test/test_importlib/import_/test_path.py > --- a/Lib/test/test_importlib/import_/test_path.py > +++ b/Lib/test/test_importlib/import_/test_path.py > @@ -1,15 +1,14 @@ > from importlib import _bootstrap > from importlib import machinery > +from importlib import import_module > from .. import util > from . import util as import_util > -import imp > import os > import sys > -import tempfile > -from test import support > -from types import MethodType > +from types import ModuleType > import unittest > import warnings > +import zipimport > > > class FinderTests(unittest.TestCase): > @@ -89,6 +88,24 @@ > self.assertIs(loader, importer) > self.assertIn(os.curdir, sys.path_importer_cache) > > + def test_None_on_sys_path(self): > + # Putting None in sys.path[0] caused an import regression from > Python > + # 3.2: http://bugs.python.org/issue16514 > + new_path = sys.path[:] > + new_path.insert(0, None) > + new_path_importer_cache = sys.path_importer_cache.copy() > + new_path_importer_cache.pop(None, None) > + new_path_hooks = [zipimport.zipimporter, > + _bootstrap.FileFinder.path_hook( > + *_bootstrap._get_supported_file_loaders())] > + with util.uncache('email'): > + with util.import_state(meta_path=sys.meta_path[:], > + path=new_path, > + > path_importer_cache=new_path_importer_cache, > + path_hooks=new_path_hooks): > + module = import_module('email') > + self.assertIsInstance(module, ModuleType) > + > > def test_main(): > from test.support import run_unittest > diff --git a/Misc/NEWS b/Misc/NEWS > --- a/Misc/NEWS > +++ b/Misc/NEWS > @@ -12,6 +12,9 @@ > Core and Builtins > ----------------- > > +- Issue #16514: Fix regression causing a traceback when sys.path[0] is > None > + (actually, any non-string or non-bytes type). > + > - Issue #16306: Fix multiple error messages when unknown command line > parameters where passed to the interpreter. Patch by Hieu Nguyen. > > diff --git a/Python/importlib.h b/Python/importlib.h > --- a/Python/importlib.h > +++ b/Python/importlib.h > [stripped] > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins(a)python.org > http://mail.python.org/mailman/listinfo/python-checkins > >

2 1

externals?
by Kristján Valur Jónsson 19 Nov '12

19 Nov '12

This may be a silly question, but haven't the python externals been moved to HG yet? I usually work on cpython without bothering with the externals, but I found today that I needed them. On Windows this is a bit of a bother. And I've thrown away all my SVN stuff... K

8 14

need reviewers for #16475 and #16487
by Kristján Valur Jónsson 19 Nov '12

19 Nov '12

Hello there. I'd like to have some pair of eyes on a couple of patches i´ve submitted. http://bugs.python.org/issue16475 This fixes a regression in marshal between 2.x and 3.x, reinstating string reuse and internment support. In addition, it generalizes string reuse to all objects, allowing for data optimizations to be made on code objects before marshaling. This straighforward extension considerably enhances the utility of the marshal module as a low-cost data serialization tool. http://bugs.python.org/issue16487 This allows ssl contexts to be initialized with certificates from memory, rather than having to rely on the openssl performing its own file IO to read them. This allows clients and servers that have their certificates deployed e.g. from a db or verbatim in a module, to use ssl without having to resort to temporary disk files and physical IO. Both of these patches are bourne out of work performed at CCP. The former comes from work on marshal in order to support our own code object optimizer, which helps save memory on the PS3. The second comes from us supporting isolated embedded python servers and clients and not wanting to complicate things with unnecessary temporary files for storing credidentials that are obtained from elsewhere. Both were, of course, 2.7 modifications, that I have now ported to 3.4 for the benefit of the python community. K

2 2

Register-based VM for CPython
by Victor Stinner 18 Nov '12

18 Nov '12

Hi, I'm still looking for the best approach to speedup CPython. I found the following article comparing a stack-based VM to a register-based VM which announces a speed up between 20% and 40% (depending if the instruction dispatch uses a dummy switch or computed goto): http://static.usenix.org/events/vee05/full_papers/p153-yunhe.pdf Yunhe Shi, David Gregg, Andrew Beatty, M. Anton Ertl, 2005 I tried to implement such register-based instructions for CPython in the following fork of Python 3.4: http://hg.python.org/sandbox/registervm/ Contact me if you are interested and/or if you would like to contribute! -- My implementation is not finished and still experimental. It is only enabled explictly by calling registervm.optimize_func(func) where registervm is a new module. This function rewrites the bytecode of the function inplace. I chose this approach because it was simple to implement, it is simple to compare stack-based and register-based instructions, and it is also the approach used in the paper :-) The major drawback of the register approach (at least of my implementation) is that it changes the lifetime of objects. Newly created objects are only "destroyed" at the exit of the function, whereas the stack-based VM destroys "immediatly" objects (thanks to the reference counter). PyPy has similar issues with its different garbage collector. On Linux (with computed goto in ceval.c), there are some interesting speedups in pybench, up to 25% faster. Such speedup can be explained with the bytecode. For example, "c = a + b" is compiled to: LOAD_FAST 'a' LOAD_FAST 'b' BINARY_ADD STORE_FAST 'c' The optimized bytecode becomes: BINARY_ADD_REG 'c', 'a', 'b' So 4 operations are replaced with only 1 operation. So instruction dispatching has a cost, measurable in this microbenchmark. My implementation is incomplete: if you see "PUSH_REG" or "POP_REG" in the assembler, your code will be slower than the original bytecode. There is no register allocator and the optimizer doesn't understand the control flow, so I expect more performance improvment with a more complete implementation. Some optimizations are also disabled by default because the implementation is buggy. The main changes of registervm are done in Python/ceval.c (implement new instructions) and Lib/registervm.py (the new module rewriting the bytecode). Objects/frameobject.c is also modified to allocate registers. See registervm documentation for more information: http://hg.python.org/sandbox/registervm/file/tip/REGISTERVM.txt -- The WPython project is similar to my work (except that it does not use registers). It tries also to reduce the overhead of instruction dispatch by using more complex instructions. http://code.google.com/p/wpython/ Using registers instead of a stack allow to implement more optimizations (than WPython). For example, it's possible to load constants outside loops and merge "duplicate constant loads". I also implemented more aggressive and experimental optimizations (disabled by default) which may break applications: move loads of attributes and globals outside of loops, and replace binary operations with inplace operations. For example, "x=[]; for ...: x.append(...)" is optimized to something like "x=[]; x_append=x.append; for ...: x_append(...)", and "x = x + 1" is replaced with "x += 1". Victor

6 8

Failed issue tracker submission
by Python tracker 17 Nov '12

17 Nov '12

An unexpected error occurred during the processing of your message. The tracker administrator is being notified. Return-Path: <python-dev(a)python.org> X-Original-To: report(a)bugs.python.org Delivered-To: roundup+tracker(a)psf.upfronthosting.co.za Received: from mail.python.org (mail.python.org [IPv6:2001:888:2000:d::a6]) by psf.upfronthosting.co.za (Postfix) with ESMTPS id DFD211C98E for <report(a)bugs.python.org>; Fri, 16 Nov 2012 03:39:49 +0100 (CET) Received: from albatross.python.org (localhost [127.0.0.1]) by mail.python.org (Postfix) with ESMTP id 3Y2kDj4Tk8zRb6 for <report(a)bugs.python.org>; Fri, 16 Nov 2012 03:39:49 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=python.org; s=200901; t=1353033589; bh=H41Haza/UPvj9B16qvXQtqjlb22jRmazur238MqAPOE=; h=MIME-Version:Content-Type:Content-Transfer-Encoding:From:To: Subject:Message-Id:Date; b=yz1YIEtIYuM3H9V1Ok0YJ/SDvU8xtO+cFOApgDH8jPRdxRS6z/CSdZ96bSY1cNFM7 4xxKY3LiUXipe439DPwEjxZr0HcsfX+2JdR4Pzf3OZmdopV6PEl8/twaIFoTHQMy8u dTwpZ7G4OJKCc2IeCq4e5Bl/TEvQvVT9NO8eoa3k= Received: from localhost (HELO mail.python.org) (127.0.0.1) by albatross.python.org with SMTP; 16 Nov 2012 03:39:49 +0100 Received: from virt-7yvsjn.psf.osuosl.org (virt-7yvsjn.psf.osuosl.org [140.211.10.72]) by mail.python.org (Postfix) with ESMTP for <report(a)bugs.python.org>; Fri, 16 Nov 2012 03:39:49 +0100 (CET) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 From: python-dev(a)python.org To: report(a)bugs.python.org Subject: [issue15894] Message-Id: <3Y2kDj4Tk8zRb6(a)mail.python.org> Date: Fri, 16 Nov 2012 03:39:49 +0100 (CET) TmV3IGNoYW5nZXNldCBiZDg1MzMxMWZmZTAgYnkgQnJldHQgQ2Fubm9uIGluIGJyYW5jaCAnZGVm YXVsdCc6Cklzc3VlICMxNTg5NDogRG9jdW1lbnQgd2h5IHdlIGRvbid0IHdvcnJ5IGFib3V0IHJl LWFjcXVpcmluZyB0aGUKaHR0cDovL2hnLnB5dGhvbi5vcmcvY3B5dGhvbi9yZXYvYmQ4NTMzMTFm ZmUwCg==

6 8

Split unicodeobject.c into subfiles?
by Victor Stinner 17 Nov '12

17 Nov '12

Hi, I would like to split the huge unicodeobject.c file into smaller files. It's just the longest C file of CPython: 14,849 lines. I don't know exactly how to split it, but first I would like to know if you would agree with the idea. Example: - Objects/unicode/codecs.c - Objects/unicode/mod_format.c - Objects/unicode/methods.c - Objects/unicode/operators.c - etc. I don't know if it's better to use a subdirectory, or use a prefix for new files: Objects/unicode_methods.c, Objects/unicode_codecs.c, etc. There is already a Python/codecs.c file for example (same filename). I would like to split the unicodeobject.c because it's hard to navigate in this huge file between all functions, variables, types, macros, etc. It's hard to add new code and to fix bugs. For example, the implementation of str%args takes 1000 lines, 2 types and 10 functions (since my refactor yesterday, in Python 3.3 the main function is 500 lines long :-)). I only see one argument against such refactoring: it will be harder to backport/forwardport bugfixes. Victor

10 23