Mailman 3 October 2008 - Python-Dev

Fwd: Removal of GIL through refcounting removal.
by Sigurd Torkel Meldgaard Nov. 5, 2008

Nov. 5, 2008

Hi to all Python developers For a student project in a course on virtual machines, we are evaluating the possibility to experiment with removing the GIL from CPython We have read the arguments against doing this at http://www.python.org/doc/faq/library/#can-t-we-get-rid-of-the-global-inter…. But we think it might be possible to do this with a different approach than what has been tried till now. The main reason for the necessity of the GIL is reference counting. We believe that most of the slowdown in the free threading implementation of Greg Stein was due to the need of atomic refcounting, as this mail seems to confirm: http://mail.python.org/pipermail/python-ideas/2007-April/000414.html So we want to change CPython into having a "real" garbage collector - removing all reference counting, and then the need for locks (or atomic inc/dec ops) should be highly alleviated. Preferably the GC should be a high-performance one for instance a generational one. We believe that it can run quite a lot faster than ref-counting. Shared datastructures would get their lock obviously. Immutable objects (especially shared global objects, like True, False, Null) would not. Most of the interpreter structure would be per-thread, at that point. We do not know how Greg Stein did his locking in the free threads patch, but as a part of the course we learned there exists much faster ways of locking than using OS-locks (faster for the uncontented case) that are used in e.g. the HOT-SPOT java-compiler. This might make "free threading" in python more attractive than some pessimists think. (http://blogs.sun.com/dave/entry/biased_locking_in_hotspot) In particular, we are talking about making the uncontended case go fast, not about the independent part of stack-allocating the mutex structure, which can only be done and is only needed in Java. These ideas are similar to the ones used by Linux fast mutexes (futexes), the implementation of mutexes in NPTL. We have read this mail thread - so it seems that our idea surfaced, but Greg didn't completely love it (he wanted to optimize refcounting instead): http://mail.python.org/pipermail/python-ideas/2007-April/000436.html He was not totally negative however. His main objections are about: - cache locality (He is in our opinion partially right, as seen in some other paper time ago - any GC, copying GC in particular, doubles the amount of used memory, so it's less cache-friendly). But still GCs are overall competitive or faster than explicit management, and surely much faster of refcounting. We know it is the plan for PyPy to work in this way, and also that Jython and Ironpython works like that (using the host vm's GC), so it seems to be somehow agreeable with the python semantics (perhaps not really with __del__ but they are not really nice anyway). Was this ever tried for CPython? Any other comments, encouragements or warnings on the project-idea? Best regards: Paolo, Sigurd

33 66

hg branch gone?
by Brett Cannon Nov. 4, 2008

Nov. 4, 2008

I just tried to update my 3.0 branch in hg from http://code.python.org/hg/branches/py3k/ and hg is telling me it's a 404. Anyone else having trouble? -Brett

8 20

Python2.5 _sre deepcopy regression?
by Andrew McNamara Nov. 4, 2008

Nov. 4, 2008

In version of Python prior to 2.5, it would appear that deepcopying compiled regular expressions worked by accident: 2.4: >>> copy.deepcopy(re.compile('')) <_sre.SRE_Pattern object at 0xb7d53ef0> 2.5: >>> copy.deepcopy(re.compile('')) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python2.5/copy.py", line 173, in deepcopy y = copier(memo) TypeError: cannot deepcopy this pattern object I say "by accident", since the SRE_Pattern object in 2.4 has a __deepcopy__ method which raises the "cannot deepcopy this pattern object" TypeError, however this method isn't found by copy.deepcopy() in 2.4, and copy.deepcopy() falls back to using the pickle logic. The _sre source has #ifdef-out support for __deepcopy__, issue 416670 has the gory details: http://bugs.python.org/issue416670 Changeset 38430 on the release24-maint branch introduced the changes that stopped SRE_Pattern.__deepcopy__ being found. r38430 was a patch forward ported from 2.3, but never ported to the trunk (probably a good thing, too). Thoughts? -- Andrew McNamara, Senior Developer, Object Craft http://www.object-craft.com.au/

1 1

Re: [Python-Dev] [ANN] VPython 0.1
by cadr4u＠gmail.com Nov. 1, 2008

Nov. 1, 2008

Hey, I hope you don't mind my replying in digest form. First off, I guess I should be a little clearer as to what VPthon is and what it does. VPython is essentially a set of patches for CPython (in touches only three files, diff -b is about 800 lines IIRC plus the switch statement in ceval.c's EvalFrameEx()). The main change is moving the VM instruction implementations, in CPython, blocks of code following a case label, into a separate file, adding Vmgen stack comments, and removing the explicit stack manipulation code (plus some minor modification like renaming variables to work with Vmgen's type prefixes and labels to enable the generation of superinstructions). Vmgen parses the stack comments and prints some macros around the provided instruction code (incidentally, this reduced the 1500 line switch body to about 1000 lines). Interested parties should consult ceval.vmg and ceval-vm.i. The nice thing about this is that: a) It's fairly easy to implement different types of dispatch, simply by changing a few macros (and while I haven't done this, it shouldn't be a problem to add some switch dispatch #ifdefs for non-GCC platforms). In particular, direct threaded code leads to less horrible branch prediction than switch dispatch on many machines (exactly how pronounced this effect is depends heavily on the specific architecture). b) Vmgen can generate superinstructions. A quick primer: A sequence of code such as LOAD_CONST LOAD_FAST BINARY_ADD will, in CPython, push some constant onto the stack, push some local onto the stack, then pop both off the stack, add them and push the result back onto the stack. Turning this into a superinstruction means inlining LOAD_CONST and LOAD_FAST, modifying them to store the values they'd otherwise push onto the stack in local variables and adding a version of BINARY_ADD which reads its arguments from those local variables rather than the stack (this reduces dispatch time in addition to pops and pushes). David Gregg (and friends) recently published a paper comparing stack based and register based VMs for Java and found that register based VMs were substantially faster. The main reason for this appears to be the absence of the various LOAD_ instructions in a register VM. They looked at mitigating this using superinstructions but Java would have required (again, IIRC) about a 1000 (which leads to substantial code growth). Since Python doesn't have multiple (typed) versions of every instruction (Java has iadd and so on) much fewer superinstructions are necessary. On my system, superinstructions account for about 10% of the 30% performance gain. As for limitations, as the README points out, currently 2 cases in test_doctest fail, as well as 1 case in test_hotshot, test_inspect, and test_subprocess. And most of the cases in test_trace. The reason for this is, I suspect, that I removed the line tracing code from ceval.c (I didn't want to look at it detail, and it doesn't seem to affect anything else). I expect this would be a bit of work to fix but I don't see it as a huge problem (in fact, if you don't use settrace(?) it shouldn't affect you?). Stack caching: a previous version of VPython supported this, but the performance gain was minimal (maybe 1-2%, though if done really well (e.g. using x as the top of stack cache), who knows, more may be possible). Also, it let to some problems with the garbage collector seeing an out-of-date stack_pointer[-1]. ``Cell'' is, unfortunately, hardcoded into Vmgen. I could either patch that or run ceval-vm.i through sed or something. Finally, to the people who pointed out that VPython (the name) is already taken: Darn! I really should have checked that! Will call it something else in the future. Anyway, HTH, -jakob

11 19

Has python-dev collapsed?
by Guido van Rossum Oct. 31, 2008

Oct. 31, 2008

Or have I been removed from the list? I'm not getting any mail it seems. Or at least very little. Or are we all just tired of trying to fix the release blockers in 3.0? -- --Guido van Rossum (home page: http://www.python.org/~guido/)

7 8

Summary of Python tracker Issues
by Python tracker Oct. 31, 2008

Oct. 31, 2008

ACTIVITY SUMMARY (10/24/08 - 10/31/08) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2153 open (+33) / 13915 closed (+19) / 16068 total (+52) Open issues with patches: 705 Average duration of open issues: 710 days. Median duration of open issues: 1951 days. Open Issues Breakdown open 2137 (+33) pending 16 ( +0) Issues Created Or Reopened (52) _______________________________ Miserable subprocess.Popen performance 10/24/08 http://bugs.python.org/issue4194 created skip.montanaro Regression for executing packages 10/24/08 http://bugs.python.org/issue4195 created vajda patch library documentation errors 10/24/08 CLOSED http://bugs.python.org/issue4196 created LambertDW Doctest module does not work with zipped packages 10/24/08 http://bugs.python.org/issue4197 created belopolsky patch os.path.normcase gets fooled on windows with mapped linux networ 10/24/08 http://bugs.python.org/issue4198 created dawidjoubert add shorthand global and nonlocal statements 10/24/08 http://bugs.python.org/issue4199 created benjamin.peterson patch atexit module not safe in Python 3.0 with multiple interpreters 10/24/08 CLOSED http://bugs.python.org/issue4200 created grahamd patch, needs review Pdb cannot access source code in zipped packages. 10/24/08 http://bugs.python.org/issue4201 created belopolsky patch Multiple interpreters and readline module hook functions. 10/25/08 http://bugs.python.org/issue4202 created grahamd adapt sphinx-quickstart for windows 10/25/08 http://bugs.python.org/issue4203 created timmie Cannot build _multiprocessing, math, mmap and readline of Python 10/25/08 http://bugs.python.org/issue4204 created akitada patch unexpected str.__init__() behaviour on b'' 10/25/08 CLOSED http://bugs.python.org/issue4205 created keldonin multiprocessing docs 10/26/08 http://bugs.python.org/issue4206 created LambertDW Remove backwards compatibility in _heapq for performance 10/26/08 CLOSED http://bugs.python.org/issue4207 created krisvale patch, patch, easy Make multiprocessing compatible with Python 2.4 and 2.5 10/26/08 http://bugs.python.org/issue4208 created christian.heimes patch, patch from __future__ import unicode_literals, print_function doesn't 10/26/08 CLOSED http://bugs.python.org/issue4209 created benjamin.peterson patch os.remove can be used to remove directories on SunOS 10/26/08 CLOSED http://bugs.python.org/issue4210 created giampaolo.rodola frozen packages set an improper __path__ value 10/27/08 http://bugs.python.org/issue4211 created brett.cannon email.LazyImporter does not use absolute imports 10/27/08 http://bugs.python.org/issue4212 created brandonbloom Lower case file system encoding 10/27/08 CLOSED http://bugs.python.org/issue4213 created christian.heimes patch no extension debug info with msvc9compiler.py 10/27/08 http://bugs.python.org/issue4214 created robind Running Python 2.6 GUI on Windows Vista 10/27/08 http://bugs.python.org/issue4215 created Elushin subprocess.Popen hangs at communicate() when child exits 10/27/08 http://bugs.python.org/issue4216 created amy20_z Add file comparisons to the unittest library 10/28/08 http://bugs.python.org/issue4217 created wehart tarfile module fails to correctly parse some .tar.gz archives?! 10/28/08 CLOSED http://bugs.python.org/issue4218 created gerhard_xXx Problem with regular expression 10/28/08 CLOSED http://bugs.python.org/issue4219 created carlosklock Builtins treated as free variables? 10/28/08 http://bugs.python.org/issue4220 created dfugate inconsistent exception from int is confusing 10/28/08 http://bugs.python.org/issue4221 created exarkun dis.findlinestarts is missing from dis.__all__ and from the onli 10/28/08 http://bugs.python.org/issue4222 created exarkun inspect.getsource doesn't work on functions imported from a zipf 10/28/08 http://bugs.python.org/issue4223 created exarkun ** operating incorrectly for negative bases? 10/28/08 CLOSED http://bugs.python.org/issue4224 created martin.speleo unicode_literals doesn't work in exec 10/28/08 CLOSED http://bugs.python.org/issue4225 created benjamin.peterson patch, needs review Python core crashes with associated files. 10/30/08 http://bugs.python.org/issue4226 reopened christian.heimes unicode_literals and print_function do not work together. 10/29/08 CLOSED http://bugs.python.org/issue4227 created wangchun struct.pack('L', -1) 10/29/08 http://bugs.python.org/issue4228 created arigo Can't Start Python 10/29/08 CLOSED http://bugs.python.org/issue4229 created CliffB "__getattr__" can't be a descriptor 10/29/08 http://bugs.python.org/issue4230 created arigo print " should be print(" 10/29/08 CLOSED http://bugs.python.org/issue4231 created LambertDW tempfile.gettempdir() availability not documented 10/29/08 CLOSED http://bugs.python.org/issue4232 created scop open(0, closefd=False) prints 3 warnings 10/30/08 http://bugs.python.org/issue4233 created amaury.forgeotdarc bin missing from documentation 10/30/08 CLOSED http://bugs.python.org/issue4234 created v+python patch Crash when iImporting a builtin module crashes 10/30/08 CLOSED http://bugs.python.org/issue4235 created amaury.forgeotdarc Crash when importing builtin module during interpreter shutdown 10/30/08 http://bugs.python.org/issue4236 created amaury.forgeotdarc patch io.FileIO('foo', 'rt') prints a RuntimeWarning 10/30/08 CLOSED http://bugs.python.org/issue4237 created amaury.forgeotdarc patch, needs review BSD support for multiprocessing.cpu_count 10/30/08 http://bugs.python.org/issue4238 created epg Email example should use SMTP.quit() rather than SMTP.close() 10/30/08 http://bugs.python.org/issue4239 created matt parasite form feed character in email.mime.text 10/30/08 CLOSED http://bugs.python.org/issue4240 created tarek zipfile.py -> is_zipfile leaves file open when error 10/30/08 http://bugs.python.org/issue4241 created eino Classify language vs. impl-detail tests, step 1 10/30/08 http://bugs.python.org/issue4242 created arigo patch, patch has_key doc could be clearer 10/30/08 http://bugs.python.org/issue4243 created tjreedy ihooks incompatible with absolute_import feature 10/31/08 http://bugs.python.org/issue4244 created exarkun threading documentation: reorder sections 10/31/08 http://bugs.python.org/issue4245 created hoffman Issues Now Closed (26) ______________________ ftplib needs a rewrite to use bytes/buffers 388 days http://bugs.python.org/issue1248 haypo backport python 3.0 language functionality to python 2.6 by addi 117 days http://bugs.python.org/issue3238 loewis patch Python turning off assertions (Windows) 77 days http://bugs.python.org/issue3545 kevinwatters python3.0 interpreter on Cygwin ignores all arguments 71 days http://bugs.python.org/issue3626 amaury.forgeotdarc patch Py_NewInterpreter does not work 62 days http://bugs.python.org/issue3723 christian.heimes patch, needs review segfault with defaultdict and pickle 8 days http://bugs.python.org/issue4170 amaury.forgeotdarc patch segfault with pickle if 4th or 5th item of tuple returned by __r 9 days http://bugs.python.org/issue4176 ocean-city patch library documentation errors 0 days http://bugs.python.org/issue4196 benjamin.peterson atexit module not safe in Python 3.0 with multiple interpreters 6 days http://bugs.python.org/issue4200 christian.heimes patch, needs review unexpected str.__init__() behaviour on b'' 0 days http://bugs.python.org/issue4205 keldonin Remove backwards compatibility in _heapq for performance 0 days http://bugs.python.org/issue4207 krisvale patch, patch, easy from __future__ import unicode_literals, print_function doesn't 0 days http://bugs.python.org/issue4209 benjamin.peterson patch os.remove can be used to remove directories on SunOS 0 days http://bugs.python.org/issue4210 giampaolo.rodola Lower case file system encoding 3 days http://bugs.python.org/issue4213 haypo patch tarfile module fails to correctly parse some .tar.gz archives?! 0 days http://bugs.python.org/issue4218 lars.gustaebel Problem with regular expression 0 days http://bugs.python.org/issue4219 georg.brandl ** operating incorrectly for negative bases? 0 days http://bugs.python.org/issue4224 benjamin.peterson unicode_literals doesn't work in exec 2 days http://bugs.python.org/issue4225 benjamin.peterson patch, needs review unicode_literals and print_function do not work together. 0 days http://bugs.python.org/issue4227 christian.heimes Can't Start Python 0 days http://bugs.python.org/issue4229 loewis print " should be print(" 0 days http://bugs.python.org/issue4231 benjamin.peterson tempfile.gettempdir() availability not documented 0 days http://bugs.python.org/issue4232 benjamin.peterson bin missing from documentation 1 days http://bugs.python.org/issue4234 benjamin.peterson patch Crash when iImporting a builtin module crashes 0 days http://bugs.python.org/issue4235 amaury.forgeotdarc io.FileIO('foo', 'rt') prints a RuntimeWarning 1 days http://bugs.python.org/issue4237 christian.heimes patch, needs review parasite form feed character in email.mime.text 0 days http://bugs.python.org/issue4240 skip.montanaro Top Issues Most Discussed (10) ______________________________ 18 Cannot build _multiprocessing, math, mmap and readline of Pytho 6 days open http://bugs.python.org/issue4204 15 Miserable subprocess.Popen performance 7 days open http://bugs.python.org/issue4194 13 atexit module not safe in Python 3.0 with multiple interpreters 6 days closed http://bugs.python.org/issue4200 9 Lower case file system encoding 3 days closed http://bugs.python.org/issue4213 8 io.FileIO('foo', 'rt') prints a RuntimeWarning 1 days closed http://bugs.python.org/issue4237 8 inspect.getsource doesn't work on functions imported from a zip 3 days open http://bugs.python.org/issue4223 6 Classify language vs. impl-detail tests, step 1 1 days open http://bugs.python.org/issue4242 6 Crash when importing builtin module during interpreter shutdown 2 days open http://bugs.python.org/issue4236 6 unicode_literals doesn't work in exec 2 days closed http://bugs.python.org/issue4225 5 has_key doc could be clearer 1 days open http://bugs.python.org/issue4243

1 0

Proper initialization of structs
by Christian Heimes Oct. 31, 2008

Oct. 31, 2008

I like to raise attention for a problem revealed by http://bugs.python.org/issue4237 --- The bug was caused by a design flaw -- which was partly my fault. Some elements of the PyFileIOObject struct were initialized in __new__ while other parts were initialized in __init__. I've moved the initialization to __new__. We should add a rule that all struct members must be properly initialized in __new__. In the past Victor's fuzzying tool has revealed several crashers related to similar design flaws. I'm raising the severity of the bug to release blocker because I can't predict if the problem can be abused to crash the interpreter. We should also review all __new__ and __init__ methods of objects and extension modules for similar issues. --- The same design flaw was responsible for bugs like the pickle crasher http://bugs.python.org/issue3664. I like to establish a rule that *all* struct members must be initialized properly in the type's tp_new function. Comments? Christian

5 8

buildbots
by Thomas Heller Oct. 30, 2008

Oct. 30, 2008

AFAICS there are no buildbots for the release26-maint branch. Thomas

3 3

Classifying language vs. impl-detail tests
by Armin Rigo Oct. 30, 2008

Oct. 30, 2008

Hi all, Here is a first step towards classifying the Python test suite into "real language tests" and "implementation details": http://bugs.python.org/issue4242 If the general approach seems acceptable to people, I would be willing to port more of PyPy's test suite patches. The net result would be to move the test suite towards being more directly useful for alternate Python implementations. (Right now, all of them have some custom tests plus their own similarly patched copy of the stdlib test suite.) Note that the patch above is against 2.7 trunk; the 2.x line is what all non-CPython implementations currently target. The general idea (and to a large extend the patches) can easily be forward-ported to 3.x when it makes sense. Also, the actual decision of what is a "real" or a "detail" test is of course up to discussion. The patch above does the classification for test_descr. It was obtained by running it with PyPy, and for each failure either fixing PyPy, or argumenting (by adding a comment) the reason for which it is a detail -- usually referring to "well-known agreement" expressed on python-dev about the matter. (Of course, "detail" tests still run normally on top of CPython.) A bientot, Armin.

2 1

svnmerge init in the release26-maint branch?
by Thomas Heller Oct. 30, 2008

Oct. 30, 2008

I have a fix for a modulefinder crash that I'm going to commit into the trunk. Since the bug is also in release26-maint I will also backport it. While preparing this I noticed that in the release26-maint branch 'svnmerge init' has not yet been done. I assume that we will use svnmerge to merge from trunk to release26-maint? So, is it ok to do 'svnmerge init' in it? -- Thanks, Thomas

2 2