Mailman 3 December 2009 - Python-Dev

Issues with multiprocessing
by Joshi, Yateen Dec. 9, 2009

Dec. 9, 2009

Hi, I have an application that uses multiprocessing pools (multiprocessing.Pool(processes=.....)). There are multiple such pools and each pool has configurable number of processes. Once the process is spawned, it keeps on executing and does the needed processing. If there is nothing to process (like ftp'ing files from some source, if the files are not there, the process would sleep for some time, and then again check for files, that way, it is a infinite loop with some sleep), the process 'sleeps' for some time and continues. I am using a T5220, Solaris box with Solaris 10. Problem -there are multiple pools and multiple processes, i am seeing that not all the processes get spawned. They get spawned when the sleep time is increased (say from 0.1 sec to 1 sec). If I further increase the number of processes, again some process do not get spawned. For that, I further need to increase the sleep time (say to 2 sec), then the processes get spawned. Typically, in a multiprocessing I would expect that if a process sleeps for even a small time, other processes should get their chance to execute, but this does not seem to be happening here. Can you please throw some light on it? Thanks and Regards, Yateen V. Joshi This email and any attachments may contain legally privileged and/or confidential information of Starent Networks, Corp. and is intended only for the individual or entity named in the message. The information transmitted may not be used to create or change any contractual obligations of Starent Networks, Corp. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this e-mail and its attachments by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify the sender immediately -- by replying to this message or by sending an email to postmaster(a)starentnetworks.com -- and destroy all copies of this message and any attachments without reading or disclosing their contents. Thank you.

3 2

Sort out formatting differences in decimal and float
by Stefan Krah Dec. 6, 2009

Dec. 6, 2009

Hi, I'm in the process of implementing formatting in my C-decimal module. Since testing is quite time consuming, I would appreciate a binding decision on how the following should be formatted: >>> from decimal import * >>> format(float(1234), '020,g') '0,000,000,000,001,234' >>> format(float(1234), '0=20,g') '0,000,000,000,001,234' >>> >>> format(Decimal(1234), '020,g') '0,000,000,000,001,234' >>> format(Decimal(1234), '0=20,g') '0000000000000001,234' >>> format(Decimal('nan'), '020,g') ' NaN' >>> format(Decimal('nan'), '0=20,g') '00000000000000000NaN' You can see that float literally follows PEP-3101: "If the width field is preceded by a zero ('0') character, this enables zero-padding. This is equivalent to an alignment type of '=' and a fill character of '0'." The advantage of decimal is that the user has the option to suppress commas. The behaviour of float is slightly easier to implement in C. So the options are: a) 020 is always equivalent to 0=20 b) 020 is not always equivalent to 0=20 Stefan Krah

7 9

[RELEASED] Python 2.7 alpha 1
by Benjamin Peterson Dec. 5, 2009

Dec. 5, 2009

On behalf of the Python development team, I'm pleased to announce the first alpha release of Python 2.7. Python 2.7 is scheduled to be the last major version in the 2.x series. It includes many features that were first released in Python 3.1. The faster io module, the new nested with statement syntax, improved float repr, and the memoryview object have been backported from 3.1. Other features include an ordered dictionary implementation, unittests improvements, and support for ttk Tile in Tkinter. For a more extensive list of changes in 2.7, see http://doc.python.org/dev/doc/whatsnew/2.7.html or Misc/NEWS in the Python distribution. Please note that this is a development release, intended as a preview of new features for the community, and is thus not suitable for production use. To download Python 2.7 visit: http://www.python.org/download/releases/2.7/ The 2.7 documentation can be found at: http://docs.python.org/2.7 Please consider trying Python 2.7 with your code and reporting any bugs you may notice to: http://bugs.python.org Have fun! -- Benjamin Peterson Release Manager benjamin at python.org (on behalf of the entire python-dev team and 2.7's contributors)

3 3

Rules for the Tools directory
by Antoine Pitrou Dec. 5, 2009

Dec. 5, 2009

Hello, I was going to suggest adding ccbench and iobench to the Tools directory, so I wonder whether there are any rules for putting stuff there? Thank you Antoine.

3 2

who wants to be sprint coach @ PyCon 2010?
by Brett Cannon Dec. 5, 2009

Dec. 5, 2009

I can't afford to attend PyCon beyond the language summit and the conference itself this year (down to about my last year of my PhD and need every last dime I have left in saving to avoid school debt), so for the first time since PyCon started back in 2003, I have to miss out on attending the sprints. That means my traditional role as sprint coach for the core is up for grabs for someone else to take on. Since early-bird registration is open for about a month (ends Jan 6) I wanted to get this email out now instead of wait in case it influences someone's plan to attend the sprints. -Brett

1 0

Windows make_buildinfo changes in 2.6 and cross-compilation
by Tim Lesher Dec. 4, 2009

Dec. 4, 2009

I ran across this while trying to upgrade our company's internal Windows CE build from 2.5.2 to either 2.6 or 3.1. In the 2.5 era, the Windows version of make_buildinfo used to fall back to just copying the getbuildinfo.c file from \Modules if subwcrev failed; since 2.6, make_buildinfo fails the build if the compilation of getbuildinfo.c fails when make_buildinfo is run, without falling back to copying. Any reason not to keep the fallback strategy in the process? The new process fails in cross-compilation scenarios (like building for Windows CE on a Windows host) because, while the VC++ solution can be told to build native (win32) binaries for make_buildinfo and make_versioninfo, when make_buildinfo tries to compile via its hardcoded call to cl.exe, it tries to use the cross-compiler without the required flags from the cross-compilation configs in the vcproj files. Thanks! -- Tim Lesher <tlesher(a)gmail.com>

1 0

Summary of Python tracker Issues
by Python tracker Dec. 4, 2009

Dec. 4, 2009

ACTIVITY SUMMARY (11/27/09 - 12/04/09) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2501 open (+17) / 16755 closed (+14) / 19256 total (+31) Open issues with patches: 998 Average duration of open issues: 695 days. Median duration of open issues: 451 days. Open Issues Breakdown open 2466 (+17) pending 34 ( +0) Issues Created Or Reopened (31) _______________________________ optparse Documentation References Example Files that do not Exis 11/28/09 http://bugs.python.org/issue7404 created MarkRoddy patch compiling python 3.1.1 using cygwin 1.7.0 and gcc 4.3.4 11/28/09 http://bugs.python.org/issue7405 created lopgok int arithmetic relies on C signed overflow behaviour 11/29/09 http://bugs.python.org/issue7406 created mark.dickinson Minor Queue doc improvement 11/29/09 http://bugs.python.org/issue7407 created flub patch test_distutils fails on Mac OS X 10.5 11/29/09 CLOSED http://bugs.python.org/issue7408 created titus cleanup now deepcopy copies instance methods 11/29/09 CLOSED http://bugs.python.org/issue7409 created rbcollins patch deepcopy of itertools.count resets the count 11/30/09 CLOSED http://bugs.python.org/issue7410 created rbcollins allow import from file having name containing hyphen or blank 11/30/09 CLOSED http://bugs.python.org/issue7411 created pfeldman(a)verizon.net distutils install race condition 11/30/09 http://bugs.python.org/issue7412 created illume datetime.datetime.isoformat truncation problem 12/01/09 http://bugs.python.org/issue7413 created exarkun patch, needs review Format code "C" is missing from skipitem() in getargs.c 12/01/09 CLOSED http://bugs.python.org/issue7414 created casevh patch PyUnicode_FromEncodedObject() uses PyObject_AsCharBuffer() 12/01/09 http://bugs.python.org/issue7415 created scoder select module compile errors breaks OS X multi-architecture buil 12/01/09 CLOSED http://bugs.python.org/issue7416 created ned.deily open builtin has no signature in docstring 12/01/09 http://bugs.python.org/issue7417 created englabenny patch hashlib : the names of the different hash algorithms 12/01/09 http://bugs.python.org/issue7418 created chaica_ patch Crash in _locale.setlocale on windows 12/01/09 CLOSED http://bugs.python.org/issue7419 created amaury.forgeotdarc turtle - turtle.update() doesn't override turtle.tracer() 12/01/09 CLOSED http://bugs.python.org/issue7420 created dragon Given 12/01/09 CLOSED http://bugs.python.org/issue7421 created Fallen Document inspect.get(full)argspec limitation to Python function 12/01/09 http://bugs.python.org/issue7422 created tjreedy nested generator expression produces strange results 12/02/09 CLOSED http://bugs.python.org/issue7423 created bogklug segmentation fault in listextend during install 12/02/09 http://bugs.python.org/issue7424 created jon.buller [PATCH] Improve the robustness of "pydoc -k" in the face of brok 12/02/09 http://bugs.python.org/issue7425 created dmalcolm patch StringIO and with statement 12/03/09 http://bugs.python.org/issue7426 created jmfauth BadStatusLine is hell to debug 12/03/09 http://bugs.python.org/issue7427 created djc Possible memory issue with optparse 12/03/09 CLOSED http://bugs.python.org/issue7428 created wfspotz PrettyPrinter cannot print dicts with unsortable keys 12/03/09 CLOSED http://bugs.python.org/issue7429 created maranos "cmp" still sends messages 12/03/09 CLOSED http://bugs.python.org/issue7430 created LambertDW UnboundLocalError during test.test_linecache.LineCacheTests 12/03/09 CLOSED http://bugs.python.org/issue7431 created ivank patch Py3k doc: "from __future__ import division" not necessary 12/03/09 http://bugs.python.org/issue7432 created jaredgrubb MemoryView memory_getbuf causes segfaults, double call to tp_rel 12/03/09 http://bugs.python.org/issue7433 created pv pprint doesn't know how to print a namedtuple 12/04/09 http://bugs.python.org/issue7434 created afoglia Issues Now Closed (40) ______________________ deepcopy doesn't copy instance methods 731 days http://bugs.python.org/issue1515 pitrou patch Automatically disable pymalloc when running under valgrind 624 days http://bugs.python.org/issue2422 benjamin.peterson patch Mac build_install.py script fetches unavailable SQLite version 623 days http://bugs.python.org/issue2441 ghaering patch Make '%F' and float.__format__('F') convert results to upper cas 500 days http://bugs.python.org/issue3382 eric.smith patch, easy PyObject_GetAttr() to get special methods 486 days http://bugs.python.org/issue3471 benjamin.peterson faster long multiplication 431 days http://bugs.python.org/issue3944 mark.dickinson patch Popen(..., cwd=...) does not set PWD environment variable 421 days http://bugs.python.org/issue4057 r.david.murray Do not embed manifest files in *.pyd when compiling with MSVC 415 days http://bugs.python.org/issue4120 loewis patch 10e667.__format__('+') should return 'inf' 366 days http://bugs.python.org/issue4482 eric.smith patch Exception traceback is incorrect for strange exception handling 361 days http://bugs.python.org/issue4486 pitrou patch, needs review Objects/bytesobject.c should include stringdefs.h, instead of de 230 days http://bugs.python.org/issue5748 eric.smith easy Unicode issue with tempfile on Windows 193 days http://bugs.python.org/issue6077 amaury.forgeotdarc patch, needs review test_posix getcwd test leaves tmp dir 70 days http://bugs.python.org/issue6974 r.david.murray range() fails with long integers 70 days http://bugs.python.org/issue6985 mark.dickinson decimal.py: use DivisionImpossible and DivisionUndefined 58 days http://bugs.python.org/issue7046 rhettinger decimal.py: logb: round the result if it is greater than prec 62 days http://bugs.python.org/issue7048 mark.dickinson patch urllib2.urlopen() timeout argument ignored after redirect 48 days http://bugs.python.org/issue7118 orsenthil Unclear warning for subprocess.call 43 days http://bugs.python.org/issue7177 pjenvey select module - kevent ident field 64 bit issue 36 days http://bugs.python.org/issue7211 ned.deily patch test_multiprocessing fails consistently with 'signal 12' on Free 22 days http://bugs.python.org/issue7272 db3l patch, buildbot format: minimum width: UTF-8 separators and decimal points 19 days http://bugs.python.org/issue7327 eric.smith Add initgroups to the posix/os modules 16 days http://bugs.python.org/issue7333 pitrou patch os.write and os.close on pipe from separate threads hangs on Mac 5 days http://bugs.python.org/issue7401 ronaldoussoren Race condition in logging._acquireLock()? 0 days http://bugs.python.org/issue7403 vinay.sajip test_distutils fails on Mac OS X 10.5 0 days http://bugs.python.org/issue7408 titus cleanup now deepcopy copies instance methods 0 days http://bugs.python.org/issue7409 benjamin.peterson patch deepcopy of itertools.count resets the count 1 days http://bugs.python.org/issue7410 rhettinger allow import from file having name containing hyphen or blank 0 days http://bugs.python.org/issue7411 pitrou Format code "C" is missing from skipitem() in getargs.c 2 days http://bugs.python.org/issue7414 mark.dickinson patch select module compile errors breaks OS X multi-architecture buil 0 days http://bugs.python.org/issue7416 ronaldoussoren Crash in _locale.setlocale on windows 0 days http://bugs.python.org/issue7419 amaury.forgeotdarc turtle - turtle.update() doesn't override turtle.tracer() 0 days http://bugs.python.org/issue7420 r.david.murray Given 0 days http://bugs.python.org/issue7421 eric.smith nested generator expression produces strange results 0 days http://bugs.python.org/issue7423 r.david.murray Possible memory issue with optparse 0 days http://bugs.python.org/issue7428 loewis PrettyPrinter cannot print dicts with unsortable keys 0 days http://bugs.python.org/issue7429 r.david.murray "cmp" still sends messages 1 days http://bugs.python.org/issue7430 mark.dickinson UnboundLocalError during test.test_linecache.LineCacheTests 0 days http://bugs.python.org/issue7431 r.david.murray patch 0.0 and -0.0 identified, with surprising results 35 days http://bugs.python.org/issue1678380 mark.dickinson patch chown broken on 64bit 882 days http://bugs.python.org/issue1747858 pitrou 64bit Top Issues Most Discussed (10) ______________________________ 14 test_distutils fails on Mac OS X 10.5 0 days closed http://bugs.python.org/issue7408 14 format: minimum width: UTF-8 separators and decimal points 19 days closed http://bugs.python.org/issue7327 10 open builtin has no signature in docstring 3 days open http://bugs.python.org/issue7417 7 os.write and os.close on pipe from separate threads hangs on Ma 5 days closed http://bugs.python.org/issue7401 6 deepcopy doesn't copy instance methods 731 days closed http://bugs.python.org/issue1515 5 MemoryView memory_getbuf causes segfaults, double call to tp_re 1 days open http://bugs.python.org/issue7433 5 10e667.__format__('+') should return 'inf' 366 days closed http://bugs.python.org/issue4482 5 Popen(..., cwd=...) does not set PWD environment variable 421 days closed http://bugs.python.org/issue4057 5 Make '%F' and float.__format__('F') convert results to upper ca 500 days closed http://bugs.python.org/issue3382 4 datetime.datetime.isoformat truncation problem 4 days open http://bugs.python.org/issue7413

1 0

Troubled by changes to PyPI usage agreement
by Ben Finney Dec. 4, 2009

Dec. 4, 2009

Howdy all, In the Subversion repository for PyPI, this revision appeared: ===== $ bzr info . Repository checkout (format: 2a) Location: repository checkout root: . checkout of branch: https://svn.python.org/packages/trunk/pypi shared repository: /home/bignose/Projects/python/pypi $ bzr log --revision 655 ------------------------------------------------------------ revno: 655 svn revno: 690 (on /trunk/pypi) committer: martin.von.loewis timestamp: Sun 2009-11-29 17:15:12 +0000 message: Update usage agreement according to Van Lindberg's instructions. ===== The revision modifies various files for the web interface, changing the wording of an agreement and requiring an “I agree” checkbox to be checked. ===== $ bzr diff --change 655 | diffstat templates/openid_return.pt | 28 +++++++++++++++++++--------- templates/register.pt | 32 +++++++++++++++++++++++--------- webui.py | 6 ++++++ 3 files changed, 48 insertions(+), 18 deletions(-) ===== The new wording is one that I can't agree to: ===== […] + <li>Content is restricted to Python packages and related information only.</li> + <li>Any content uploaded to PyPI is provided on a non-confidential basis.</li> + <li>The PSF is free to use or disseminate any content that I upload on an + unrestricted basis for any purpose. In particular, the PSF and all other + users of the web site are granted an irrevocable, worldwide, royalty-free, + nonexclusive license to reproduce, distribute, transmit, display, perform, + and publish the content, including in digital form.</li> + <li>I represent and warrant that I have complied with all government + regulations […] ===== The content that I submit to PyPI is licensed under specific license terms. That certainly does *not* allow the PSF to “use or disseminate any content that I upload on an unrestricted basis for any purpose”, etc.; it allows only those acts permitted by the license terms granted in the work. I have already registered an account at PyPI, and never agreed to this wording. (The previous wording was much less broad and unobjectionable.) I would not have noticed it changing if I had not been investigating the PyPI website source code. Will the PSF claim I am bound by it anyway? What about future changes? Why was this wording chosen? How does the PSF propose to reconcile this with copyright holders's chosen license terms for their work? -- \ “Timid men prefer the calm of despotism to the boisterous sea | `\ of liberty.” —Thomas Jefferson | _o__) | Ben Finney

3 4

wpython is back
by Cesare Di Mauro Dec. 3, 2009

Dec. 3, 2009

Hi Mart I'm back with some news about wpython. I completed all the work that I was committed to do till the end of the year. I made a lot of changes to the code, that I'll report here. First, I added several conditional compilation sections that enable or disable almost every optimization I introduced into the project. Everything is controlled by a new include file, wpython.h, which holds a lot of #DEFINEs for each one of them. Every #DEFINE has a brief explanation, and some report an example with Python code disassembled, showing what happens. It can be useful both to document the code (also to access to the interested parts), and to let people test the effect of all optimizations. There are also a couple of #DEFINEs which are useful to enable or disable all superinstructions, or to make wpython work like CPython (with all new optimizations and superinstructions disabled). Full tracing support required a big effort, due to the missing SETUP_LOOP/POP_BLOCK instructions used in FOR_ITER blocks. It was a pain in the neck to let them work, but I think I have found a good solution fot it. If I remember correctly, Collin asked in the past about performance with testing enabled. I believe that speed is comparable to CPython, since I can trace FOR_ITER blocks enter/exit with very little time spent intercepting them; stack unrolling (for forward jumps cases) is fast too. Restoring Python object model required much of the work. I reverted all the changes that I made to many PyObjects, and just added some accessory code only to a few of them. There are no more hacks, and code is quite polite; only CodeObject required one line of code change in the hash function, to let it calculate hash correctly for the constants tuple (because it can hold lists and dictionaries now, which usally aren't hashable). Every file in Include/ and Objects/ that I modified has only 1 diff (except frameobject.c, for tracing code), so it's easy so see what is changed and the extra helper functions that I added to introduce lists and dictionaries in the consts tuple. In the meanwhile I've added a little optimization for lists and dictionaries used in for loops. Writing this: def f(): for x in ['a', 'b', 'c']: print x generates the following (word)code with the previous wpython: LOAD_CONST (['a', 'b', 'c']) DEEP_LIST_COPY GET_ITER FOR_ITER because ['a', 'b', 'c'] is a mutable object, and a copy must be made before using it. Now it'll be: LOAD_CONST (['a', 'b', 'c']) GET_ITER FOR_ITER So code is reduced and memory consumption too, because there's no need clone the list. The trick works only for lists and dictionaries that holds non-mutable objects, but I found it's a common pattern in Python code. I've also updated the source to the latest Python 2.x version, 2.6.4. All tests pass, both with Debug and Release code, on Visual Studio Express with 32 bit code (I can't compile 64 bits versions with it). There are only a few open issues. test_syntax.py required some changes in the doctest (adding full filesystem path) to let them pass correctly. It's really strange, but... works now! test_compile.py has 2 tests disabled in test_compile_ast: #['<forblock>', """for n in [1, 2, 3]:\n print n\n"""], #[fname, fcontents], that's because there's no support for constants (except Num_kind and Str_kind) in the current ASTs code. However code compiles well, except that it cannot make use of the new constant folding code. I haven't updated Doc/library/dis.rst, which is exactly the same of CPython. I'll do it when I stop introducing or changing opcodes. Right now wpython requires manual patching of Include/Python-ast.h, with the following lines: enum _expr_kind {BoolOp_kind=1, BinOp_kind=2, UnaryOp_kind=3, Lambda_kind=4, [...] List_kind=18, Tuple_kind=19, Const_kind=20}; enum _expr_const {no_const=0, mutable_const=1, content_const=3, pure_const=7}; struct _expr { enum _expr_kind kind; union { [...] struct { object c; enum _expr_const constant; } Const; } v; int lineno; int col_offset; }; They are need to let ast.c handle constants for the new constant folding code. I greatly appreciate any help to let it be generated automatically with ASDL grammar. That's all about the new code. Now the weird and stupid part. A few days I got a new gmail account, but accidentally I removed the google account that I've used to create the wpython at Google Code. I definitely lost project ownership, so I can't tag the old code and put the new one in trunk. I'll thank very much if someone that works or has contacts with Google can ask for moving ownership from my old account (cesare at pronto do it) to my new (the one which I've using now to write this mail), so I'll commit ASAP. Alternatively, I need to create a new project at Google Code. I hope that the community will appreciate the work (when I'll upload it :-). I know that it's a young project, but I think it's mature enough to take a look at it. Last but not least, think about it like a starting point. I have many ideas on how to optimize several other parts of Python, and the wordcode structure gives me rooms to do it in an elegant and efficient way thanks to the superinstructions (when needed). For the next release I plan to cleanup opcode.h and ceval.c, grouping some instructions into single superinstructions (CALL_FUNCTIONs and IMPORT_NAME), adding a few opcodes, and tweaking a bit the VM main loop (primarily targeted to reduce the jump-table that compilers produce for the big switch statement). Then I'll consider porting it to python 2.7 and/or python 3.1/3.2 if there'll be interest and feedbacks about it. I'm also at your disposal to discuss any detail about wpython source code, since I know that it isn't a simple patch to apply to some files. Cheers, Cesare 2009/11/4 Mart Sõmermaa <mrts.pydev(a)gmail.com> > Thanks for the recap and for the good work on wpython! > > Best, eagerly waiting for the results of your work to land in mainline > python, > MS >

4 6

Unicode locale values in 2.7
by Eric Smith Dec. 3, 2009

Dec. 3, 2009

While researching http://bugs.python.org/issue7327, I've come to the conclusion that trunk handles locales incorrectly in regards to Unicode. Fixing this would be the first step toward resolving this issue with float and Decimal locale-aware formatting. The issue concerns the locale "cs_CZ.UTF-8", and the "thousands_sep" value (among others). The C struct lconv (in Linux) contains '\xc2\xa0' for thousands_sep. In py3k this is handled by calling mbstowcs (which is locale-aware) and then PyUnicode_FromWideChar, so the value is converted to u"\xa0" (non-breaking space). But in trunk, the value is just used as-is. So when formating a decimal, for example, '\xc2\xa0' is just inserted into the result, such as: >>> format(Decimal('1000'), 'n') '1\xc2\xa0000' This doesn't make much sense, and causes an error when converting it to unicode: >>> format(Decimal('1000'), u'n') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/root/python/trunk/Lib/decimal.py", line 3609, in __format__ return _format_number(self._sign, intpart, fracpart, exp, spec) File "/root/python/trunk/Lib/decimal.py", line 5704, in _format_number return _format_align(sign, intpart+fracpart, spec) File "/root/python/trunk/Lib/decimal.py", line 5595, in _format_align result = unicode(result) UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 1: ordinal not in range(128) I believe that the correct solution is to do what py3k does in locale, which is to convert the struct lconv values to unicode. But since this would be a disruptive change if universally applied, I'd like to propose that we only convert to unicode if the values won't fit into a str. So the algorithm would be something like: 1. call mbstowcs 2. if every value in the result is in the range [32, 126], return a str 3. otherwise, return a unicode This would mean that for most locales, the current behavior in trunk wouldn't change: the locale.localeconv() values would continue to be str. Only for those locales where the values wouldn't fit into a str would unicode be returned. Does this seem like an acceptable change? Eric. PS: Thanks to Mark Dickinson and others on irc and on the issue for helping in formulating this.

4 4