Mailman 3 June 2009 - Python-Dev

Binary Operator for New-Style String Formatting
by Jerry Chen June 22, 2009

June 22, 2009

Hello all, For better or for worse, I have created a patch against the py3k trunk which introduces a binary operator '@' as an alternative syntax for the new string formatting system introduced by PEP 3101 ("Advanced String Formatting"). [1] For common cases, this syntax should be as simple and as elegant as its deprecated [2] predecessor ('%'), while also ensuring that more complex use cases do not suffer needlessly. I would just like to know whether this idea will float before submitting the patch on Roundup and going through the formal PEP process. This is my first foray into the internals of the Python core, and with any luck, I did not overlook any BDFL proclamations banning all new binary operators for string formatting. :-) QUICK EXAMPLES >>> "{} {} {}" @ (1, 2, 3) '1 2 3' >>> "foo {qux} baz" @ {"qux": "bar"} 'foo bar baz' One of the main complaints of a binary operator in PEP 3101 was the inability to mix named and unnamed arguments: The current practice is to use either a dictionary or a tuple as the second argument, but as many people have commented ... this lacks flexibility. To address this, a convention of having the last element of a tuple as the named arguments dictionary is introduced. >>> "{} {qux} {}" @ (1, 3, {"qux": "bar"}) '1 bar 3' Lastly, to print the repr() of a dictionary as an unnamed argument, one would have to append an additional dictionary so there is no ambiguity: >>> "{}" @ {"foo": "bar"} Traceback (most recent call last): File "<stdin>", line 1, in <module> IndexError: tuple index out of range >>> "{}" @ ({"foo": "bar"}, {}) "{'foo': 'bar'}" Admittedly, these workarounds are less than clean, but the understanding is the '@' syntax would indeed be an alternative, so one could easily fall back to the str.format() method or the format() function. IMPLEMENTATION Code-wise, the grammar was edited per PEP 306 [3], and a function was introduced in unicodeobject.c as PyUnicode_FormatPrime (in the mathematical sense of A and A' -- I didn't fully understand or want to intrude upon the *_FormatAdvanced namespace). The PyUnicode_FormatPrime function transforms the incoming arguments, i.e. the operands of the binary '@', and makes the appropriate do_string_format() call. Thus, I have reused as much code as possible. I have done my development with git by using two branches: 'master' and 'subversion', the latter of which can be used to run 'svn update' and merge back into master. This way my code changes and the official ones going into the Subversion repository can stay separate, meanwhile allowing 'svn diff' to produce an accurate patch at any given time. The code is available at: http://github.com/jcsalterego/py3k-atsign/ The SVN patch [4] or related commit [5] are good starting points. References: [1] http://www.python.org/dev/peps/pep-3101 [2] http://docs.python.org/3.0/whatsnew/3.0.html [3] http://www.python.org/dev/peps/pep-0306/ [4] http://github.com/jcsalterego/py3k-atsign/blob/master/py3k-atsign.diff [5] http://github.com/jcsalterego/py3k-atsign/commit/5c8bdf72d9252cea78af2b7809… Thanks, -- Jerry Chen

8 8

run time error anlysis of python source code
by omega_force2003＠yahoo.com June 22, 2009

June 22, 2009

To All, It appears that one possibility of investigation into the development of a safety-critical variant of the python language would be to conduct run time error analysis of the source code that is responsible for producing the python language. Therefore, I will now conduct these run time error analysis of the python source code as if the python environment itself was to be utilized as a FADEC controller within an aircraft engine. I have already conducted some analysis already and it appears to be some concern with memory management within Python. I will redouble my efforts and determine if I am correct and as a promise. I will give my findings to everyone to enjoy if they so want it. I will also give the correct source if anyone would want it for their own purposes. The source code that I will be evaluating is the one responsible for the newer variants of Python 3.0. I believe that I will name this new variant of the python language as Apocalypse Python !! I will also develop a web-page for the development and evolution of Apocalypse Python. If anyone has any questions, please let me know !!! Thank You, David Blubaugh

6 5

xmlrpc improvements
by Kristján Valur Jónsson June 21, 2009

June 21, 2009

Hello there. I've been trying to, in the last weeks, to pass on to the trunk the improvements I've made to XMLRPC. I've created several issues in order to do these changes incrementally but have got no comments. Perhaps it is best to show the whole thing in context, then. I´ve gathered all the changes in a single patch, here: http://codereview.appspot.com/73041/show I've also created a corresponding issue in the issue tracker: http://bugs.python.org/issue6267 I'd really like to get this stuff in. The performance gains allowing http1.1 and gzip for xmlrpc are quite significant. Also, there are bugfixes in there. Cheers, Kristján

4 6

io.BufferedReader.peek() Behaviour in python3.1
by Frederick Reeve June 21, 2009

June 21, 2009

Greetings, As I'm sure you all know there are currently two implementations of the io module one in python and one much faster implementation in C. As I recall the python version was used in python3 and the C version is now used by default in python3.1x. The behavior of the two is different in some ways especially regarding io.BufferedReader.peek(). I wrote an email to the authors of the new C code last Friday. I also sent a copy of it to the python list for comments. I was directed by Antoine Pitrou that I should possibly bring up what I had asked there here or as a bug report. I elected to write here because I am not sure it constitutes a bug. In my former email I stated I was willing to submit patches if the old behavior was desired back and the code author was fine with the changes but didn't want to implement them. Antoine said this, "If people need more sophisticated semantics, I'm open to changing peek() to accommodate it." Antoine: If I do wrong quoting you are free to chastise me. So my basic question is: The behavior of io.BufferedReader.peek() has changed; is that change something that should: stay as is, revert or be different entirely? Here are the two behaviors: The python version of io.BufferedReader.peek() behaves as: If the buffer holds less than requested (upto buffersize) read from the raw stream the difference or up to EOF into the buffer. Return requested number of bytes from the start of the buffer. This may advance the raw stream but not the local stream. This version can guarantee a peek of one chunk (4096 bytes here). The C version behaves as: If the buffer holds 0 bytes fill from the raw stream or up to EOF. Return what is in the buffer. This may advance the raw stream but not the local stream. This version cannot guarantee a peek of over 1 byte if random length reads are being used at all and not tracked. Neither case limits what is possible, though, in my opinion, one makes it easier to accomplish certain things and is more efficient in those cases. Take the following two parser examples: s = io.BufferedReader wrapped stream with no negative seek in most cases. f = output file handler or such. python version work flow: are = re.compile(b'(\r\n|\r|\n)') while True: d = s.peek(4096) # chunk size or so. found = are.search(d) if found: w = d[:found.start()] s.seek(f.write(w)) p = s.peek(74) if p.startswith(multipart_boundary): s.seek(len(multipart_boundary)) # other code containing more possible splits # across boundaries continue w = d[found.start():found.end()] s.seek(f.write(w)) continue f.write(d) #more code continue C version work flow: old = b'' are = re.compile(b'(\r\n|\r|\n)') while True: d = old if old != b'' else s.read1(4096) found = are.search(d) if found: w = d[:found.start()] f.write(w) w = d[found.start():] p = w if len(w) >= 74 else w + s.read(73) if p.startswith(multipart_boundary): # Other code containing more possible splits # across boundaries and joins to p. old = ??? continue f.write(d[found.start():found.end()]) old = dd[found.end():] + p continue old = b'' f.write(d) #more code continue These two examples are not real code but get the point across and are based off code I put into a multipart parser. The former written for python3. I later tried running that parser on 3.1 after the new io layer and found it broken. Then rewrote it to the new interface. That rewrite is represented in the latter some what. This is only one example. Others may vary, of course. Peek seems to me to have little use outside of parsers. Thus I used parsers as an example. My opinion is that it would be better to have a peek function similar to the the python implementation in C like as follows: peek(n): If n is less than 0, None, or not set; return buffer contents with out advancing stream position. If the buffer is empty read a full chunk and return the buffer. Otherwise return exactly n bytes up to _chunk size_(not contents) with out advancing the stream position. If the buffer contents is less than n, buffer an additional chunk from the "raw" stream before hand. If EOF is encountered during any raw read then return as much as we can up to n. (maybe I should write that in code form??) This allows us to obtain the behavior of the current C io implementation easily and would give us the old python implementation's behavior when n is given. The basis for this is: 1. Code reduction and Simplicity Looking at the examples, the code reduction should be obvious. The logic needed to maintain a bytestring of the variously required lengths, so that it may be checked, would not be necessary. The need to hold a bytestring to the next iteration would be done away with as well. Other pieces of data handling would also be simpler. 2. Speed It would require less handling in the "slower" interpreter if we would use the buffer in the buffered reader. Also, all that logic mentioned in 1 is moved to the faster C code or done away with. There is very little necessity for peek outside of parsers, so speed in read-through and random reads would not have to be affected. I have other reasons and arguments, but I want to know what every one else thinks. This will most likely show me what I have missed or am not seeing, if anything. Please I have babbled enough. Thanks so much for the consideration. Frederick Reeve

10 33

Summary of Python tracker Issues
by Python tracker June 19, 2009

June 19, 2009

ACTIVITY SUMMARY (06/12/09 - 06/19/09) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2239 open (+26) / 15895 closed (+14) / 18134 total (+40) Open issues with patches: 886 Average duration of open issues: 653 days. Median duration of open issues: 404 days. Open Issues Breakdown open 2210 (+25) pending 29 ( +1) Issues Created Or Reopened (40) _______________________________ Add client side certificate support to httplib 06/12/09 CLOSED http://bugs.python.org/issue6273 created jnoller patch subprocess.Popen() may leak file descriptors 06/12/09 http://bugs.python.org/issue6274 created facundobatista patch let unittest.assertRaises() return the exception object caught 06/14/09 http://bugs.python.org/issue6275 reopened krisvale patch, patch, easy, needs review Fix contextlib.nested DeprecationWarning for test_signal. 06/13/09 CLOSED http://bugs.python.org/issue6276 created vshenoy patch Add description of new syntax of with to 2.7 whatsnew document. 06/13/09 CLOSED http://bugs.python.org/issue6277 created vshenoy patch http.server, BaseHTTPRequestHandler write string error 06/13/09 CLOSED http://bugs.python.org/issue6278 created System32 datamodel documentation confuses staticmethod with classmethod 06/13/09 CLOSED http://bugs.python.org/issue6279 created segfaulthunter patch calendar.timegm() belongs in time module, next to time.gmtime() 06/13/09 http://bugs.python.org/issue6280 created zooko Bug in hashlib 06/13/09 http://bugs.python.org/issue6281 created Eloff In tarfile, compression level cannot be specified 06/14/09 CLOSED http://bugs.python.org/issue6282 created Tzigi Cannot build extension in amd64 using msvc9compiler 06/14/09 http://bugs.python.org/issue6283 created alexsh C/API PyErr_AsUnicode() 06/14/09 http://bugs.python.org/issue6284 created ideasman42 patch Silent abort on XP help document display 06/14/09 http://bugs.python.org/issue6285 created scott_daniels distutils upload command doesn't work with http proxy 06/15/09 http://bugs.python.org/issue6286 created tarek distutils.core.setup does not document the 'license' meta-data 06/15/09 CLOSED http://bugs.python.org/issue6287 created mattsmart Update contextlib.nested docs in light of deprecation 06/17/09 http://bugs.python.org/issue6288 reopened ncoghlan compile() raises SyntaxError with confusing message for some dec 06/15/09 CLOSED http://bugs.python.org/issue6289 created exarkun cPickle can misread data type 06/15/09 http://bugs.python.org/issue6290 created ac.james TypeError: b2a_base64() argument 1 must be bytes or buffer, not 06/16/09 CLOSED http://bugs.python.org/issue6291 created milivojm Fix tests to work with -OO 06/16/09 http://bugs.python.org/issue6292 created collinwinter patch, easy Have regrtest.py echo back sys.flags 06/16/09 http://bugs.python.org/issue6293 created collinwinter patch, easy Improve shutdown exception ignored message 06/17/09 http://bugs.python.org/issue6294 created tjreedy easy curses doc : getch is blocking by default 06/17/09 CLOSED http://bugs.python.org/issue6295 created nojhan Native (and default) tarfile support for setup.py sdist in distu 06/17/09 http://bugs.python.org/issue6296 created michael.foord Added Misc/python.pc to 'distclean' Rule 06/17/09 http://bugs.python.org/issue6297 created jcsalterego patch http://python.org/download says Python 2.4.5, but I think it mea 06/17/09 CLOSED http://bugs.python.org/issue6298 created zooko pyexpat build failure on Solaris 10 for 2.6.1/2.6.2 06/17/09 http://bugs.python.org/issue6299 created enchanter encode and decode should accept 'errors' as a keyword argument 06/18/09 http://bugs.python.org/issue6300 created r.david.murray easy Error in tutorial section 7.2 06/18/09 CLOSED http://bugs.python.org/issue6301 created melbourne email.header.decode_header data types are inconsistent and incor 06/18/09 http://bugs.python.org/issue6302 created r.david.murray print does not appear in the 3.x documentation index 06/18/09 CLOSED http://bugs.python.org/issue6303 created r.david.murray Confusing error message when passing bytes to print with file po 06/18/09 http://bugs.python.org/issue6304 created r.david.murray islice doesn't accept large stop values 06/18/09 http://bugs.python.org/issue6305 created thomasguest filecmp.cmp can not compare two files from different OS with the 06/18/09 http://bugs.python.org/issue6306 created higer AttributeError in traceback.print_last() 06/18/09 CLOSED http://bugs.python.org/issue6307 created complex termios fix for QNX breaks HP-UX 06/18/09 http://bugs.python.org/issue6308 created haubi Tix needs TCL package require statement 06/19/09 http://bugs.python.org/issue6309 created eantelman Windows "App Paths" key is not checked when installed for curren 06/19/09 http://bugs.python.org/issue6310 created techtonik virtual memory error with archivemail 06/19/09 http://bugs.python.org/issue6311 created helgekraak httplib fails with HEAD requests to pages with "transfer-encodin 06/19/09 http://bugs.python.org/issue6312 created ezio.melotti Issues Now Closed (29) ______________________ linecache .updatecache fails on utf8 encoded files 544 days http://bugs.python.org/issue1685 amaury.forgeotdarc patch, easy test_urllib2_localnet fails on MacOS X 10.4.11 (Intel) 332 days http://bugs.python.org/issue3407 orsenthil Ill-formed surrogates not treated as errors during encoding/deco 295 days http://bugs.python.org/issue3672 hippietrail patch Local variables not freed when Exception raises in function call 76 days http://bugs.python.org/issue5641 georg.brandl Unexpected universal newline behavior (newline duplication) in W 16 days http://bugs.python.org/issue6127 alexandre.vassalotti patch subprocess module not compatible with python 2.5 15 days http://bugs.python.org/issue6189 amaury.forgeotdarc patch sdist doesn't include data_files 10 days http://bugs.python.org/issue6205 purpleidea doctest_aliases doesn't test duplicate removal 8 days http://bugs.python.org/issue6227 amaury.forgeotdarc patch Python compiles dead code 6 days http://bugs.python.org/issue6250 loewis patch PyInt_FromSize_t is undocumented. 7 days http://bugs.python.org/issue6255 georg.brandl distributions built with bdist_msi on 64-bit Windows fail to ins 3 days http://bugs.python.org/issue6258 loewis mmap: don't close file description if fd=-1 2 days http://bugs.python.org/issue6271 ocean-city patch Add client side certificate support to httplib 0 days http://bugs.python.org/issue6273 jnoller patch Fix contextlib.nested DeprecationWarning for test_signal. 6 days http://bugs.python.org/issue6276 georg.brandl patch Add description of new syntax of with to 2.7 whatsnew document. 3 days http://bugs.python.org/issue6277 rhettinger patch http.server, BaseHTTPRequestHandler write string error 2 days http://bugs.python.org/issue6278 amaury.forgeotdarc datamodel documentation confuses staticmethod with classmethod 1 days http://bugs.python.org/issue6279 segfaulthunter patch In tarfile, compression level cannot be specified 1 days http://bugs.python.org/issue6282 Tzigi distutils.core.setup does not document the 'license' meta-data 0 days http://bugs.python.org/issue6287 tarek compile() raises SyntaxError with confusing message for some dec 0 days http://bugs.python.org/issue6289 benjamin.peterson TypeError: b2a_base64() argument 1 must be bytes or buffer, not 0 days http://bugs.python.org/issue6291 r.david.murray curses doc : getch is blocking by default 0 days http://bugs.python.org/issue6295 georg.brandl http://python.org/download says Python 2.4.5, but I think it mea 0 days http://bugs.python.org/issue6298 georg.brandl Error in tutorial section 7.2 1 days http://bugs.python.org/issue6301 pjenvey print does not appear in the 3.x documentation index 0 days http://bugs.python.org/issue6303 r.david.murray AttributeError in traceback.print_last() 0 days http://bugs.python.org/issue6307 r.david.murray current directory in sys.path handles symlinks badly 1665 days http://bugs.python.org/issue1074015 marko enhance unittest to define tests that *should* fail 1253 days http://bugs.python.org/issue1399935 benjamin.peterson patch keyword and topic help broken in Pythonwin IDE 1126 days http://bugs.python.org/issue1489051 georg.brandl Top Issues Most Discussed (10) ______________________________ 18 io.BufferedReader.peek(): Documentation differs from Implementa 58 days open http://bugs.python.org/issue5811 13 Backport the IO lib to trunk 14 days open http://bugs.python.org/issue6215 9 Add "daemon" argument to threading.Thread constructor 31 days open http://bugs.python.org/issue6064 8 Error in tutorial section 7.2 1 days closed http://bugs.python.org/issue6301 8 Update contextlib.nested docs in light of deprecation 2 days open http://bugs.python.org/issue6288 8 Python compiles dead code 6 days closed http://bugs.python.org/issue6250 7 Native (and default) tarfile support for setup.py sdist in dist 2 days open http://bugs.python.org/issue6296 6 Add description of new syntax of with to 2.7 whatsnew document. 3 days closed http://bugs.python.org/issue6277 6 let unittest.assertRaises() return the exception object caught 5 days open http://bugs.python.org/issue6275 6 path separator output ignores shell's path separator: / instead 14 days open http://bugs.python.org/issue6208

1 0

Popen asynchronous input for Windows
by Eric Pruitt June 18, 2009

June 18, 2009

Hello, I am looking for alternatives to Josiah Carlson's asynchronous I/O patch for subprocess.Popen. While his patch seems to work well, it relies on pywin32 which is not part of the standard Python library. If I cannot find an alternative, I will be using cTypes with the parts of Mark Hammond's code that I need, license permitting. Any suggestions are greatly appreciated. Eric

2 2

Avoiding file descriptors leakage in subprocess.Popen()
by Facundo Batista June 18, 2009

June 18, 2009

In a long lived process at work, we started leaking file descriptors. The problem were that subprocess.Popen._execute_child() method creates two files descriptors through a call to os.pipe(), and after some work it closes them. But an os.read() on the descriptor was interrupted (EINTR), so an exception was raised, and the descriptors were not closed... leakage! This problem is easy to fix (I have a patch that fixes it, all tests pass ok, see http://bugs.python.org/issue6274). So, why this mail? I just think that the fix is ugly... it's not elegant. It has the following structure: errpipe_read, errpipe_write = os.pipe() try: try: ..... ..... ..... ..... ..... ..... finally: os.close(errpipe_write) ..... ..... ..... finally: os.close(errpipe_read) I just don't like a huge try/finally... but as FDs are just ints, do you think is there a better way to handle it? Thank you!! Regards, -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/

7 10

ElementTree issues in python 3.1 rc2
by Neil Muller June 18, 2009

June 18, 2009

The following issues in ElementTree still exist in the latest release candidate & all have small patches to fix the issue. Any chance of getting these in before the final release? http://bugs.python.org/issue6231 http://bugs.python.org/issue6233 http://bugs.python.org/issue2746 -- Neil Muller drnlmuller(a)gmail.com

2 1

SSL Certificate Validation
by Devin Cook June 17, 2009

June 17, 2009

Hi all, I have a few questions about validating SSL certificates. From what I gather, this validation occurs in the OpenSSL code called from _ssl.c. Is this correct? Also, I have looked through the docs and code, but haven't been able to figure out exactly what is included in certificate "validation". Is it just validating the chain? Does it check the NotBefore and NotAfter dates? Does it check that the host the socket is connected to is the same as what's given in the CN field in the certificate? Where I'm going with this is I think all this checking needs to be part of certificate validation in the ssl module. If it isn't yet, I'd be happy to work on a patch for it. Please let me know what you think. Thanks! -Devin Cook

4 11

Python script language or not
by abhishek goswami June 17, 2009

June 17, 2009

Hi, I have very basic question about Python that do we consider pyhton as script language. I searched in google but it becomes more confusion for me. After some analysis I came to know that Python support oops . Can anyone clarify me. Please let me know also it is right forum or not. I was looking for Python forum but i did not find any forum. Do we have forum like linux forum so it will be very easy for me to ask this question in forum and i will not ask repeated question becuase i am sure this question may be raised before Abhishek Goswami Chennai Phone No -0996227099 Love Cricket? Check out live scores, photos, video highlights and more. Click here http://cricket.yahoo.com

3 2