Mailman 3 September 2007 - Python-Dev

OpenSSL httplib bug
by Richie Ward 01 Oct '07

01 Oct '07

I was using httplib to power my xml rpc script. I had problems when I wanted to use SSL and I got this error: File "/usr/lib/python2.5/httplib.py", line 1109, in recv return self._ssl.read(len) socket.sslerror: (8, 'EOF occurred in violation of protocol') I figured out this was because of poor error handling in python. May I suggest this as a fix to this bug: $ diff /usr/lib/python2.5/httplib.py /usr/lib/python2.5/httplib.py~ 1109,1112c1109 < try: < return self._ssl.read(len) < except socket.sslerror: < return --- > return self._ssl.read(len) Just a note. I am by no means a python expert, just good enough to get my work done. I use Ubuntu gutsy. -- Thanks, Richie Ward

3 2

building with -Wwrite-strings
by Lisandro Dalcin 01 Oct '07

01 Oct '07

I'm trying to build Python (2.6) with GCC the option -Wwrite-strings. 1 - Is there any interest on this? 2 - What should I do for the very common (taken from int_new): static char *kwlist[] = {"x", "base", 0}; I was able to remove all the warning in Objects/*, except those related to (2). -- Lisandro Dalcín --------------- Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC) Instituto de Desarrollo Tecnológico para la Industria Química (INTEC) Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET) PTLC - Güemes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594

3 3

New lines, carriage returns, and Windows
by Dino Viehland 01 Oct '07

01 Oct '07

We ran into an interesting user-reported issue w/ IronPython and the way Python writes to files and I thought I'd get python-dev's opinion. When writing a string in text mode that contains \r\n we both write \r\r\n because the default write mode is to replace \n with \r\n. This works great as long as you stay within an entirely Python world. Because Python uses \n for everything internally you'll never end up writing out a \r\n that gets transformed into a \r\r\n. But when interoperating with other native code (or .NET code in our case) it's fairly easy to be exposed to a string which contains \r\n. Ultimately we see odd behavior when round tripping the contents of a multi-line text box through a file. So today users have to be aware of the fact that Python internally always uses \n. They also need to be aware of any APIs that they call that might return a string with an embedded \r\n inside of them and transform the string back into the Python version. It could be argued that there's little value in doing the simple transformation from \r\n -> \r\r\n. Ultimately that creates a file that has line endings which aren't good on any platform. On the other hand it could also be argued that Python defines new-lines as \n and there should be no deviation from that. And doing so could be considered a slippery slope, first file deals with it, and next the standard libraries, etc... Finally this might break some apps and if we changed IronPython to behave differently we could introduce incompatibilities which we don't want. So I'm curious: Is there a reason this behavior is useful that I'm missing? Would there be a possibility (or objections to) making \r\n be untransformed in the Py3k timeframe? Or should we just tell our users to open files in binary mode? :)

7 16

Re: [Python-Dev] New lines, carriage returns, and Windows
by Nick Maclaren 01 Oct '07

01 Oct '07

Greg Ewing <greg.ewing(a)canterbury.ac.nz> wrote: > > I don't see how this is different from Unix/C "\n" being > an atomic newline character. Have you used systems with the I/O models I referred to (or ones with newlines being out-of-bound data)? > If you're saying that BCPL is better because it defines > standard semantics for more control characters than just > "\n", that may be true, but C is doing about the best it > can with "\n" as far as I can see, given all the crazy > things that different OSes want to do with line endings. I am afraid that you are wrong - see my other posting for how to do it better. Look, I have implemented both of those two models on systems that are FAR more different than most people can imagine. Both work, and neither causes confusion. The C/Unix/Python one does. > In any case, the problem which started all this isn't > really an I/O problem at all, it's a mismatch between > the world of Python strings which use "\n" and .NET > library code expecting strings which use "\r\n". That's an I/O problem :-) > The correct thing to do with that is to translate whenever > a string crosses a boundary between Python code and > .NET code. This is something that ought to be done > automatically by the Python/.NET interfacing machinery, > maybe by having a different type for .NET strings. Agreed. But the REASON it causes trouble is the inconsistency in the basic C/Unix/Python text I/O model. Let's consider just \f, \r and \n, and a few questions: Exactly what does a free-standing \f mean? Does \n\f\n mean starting at the top of a page or one line down? How do \r and \f interact with line-buffering? Think about MacOS here. I could go on, but those are enough to indicate that the problem is insoluble. The answer "Undefined but not even explicitly discouraged" is a recipe for confusion. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1(a)cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679

2 1

Re: [Python-Dev] [python] Re: New lines, carriage returns, and Windows
by Nick Maclaren 01 Oct '07

01 Oct '07

Greg Ewing <greg.ewing(a)canterbury.ac.nz> wrote: > > > Grrk. That's the problem. You don't get back what you have written > > You do as long as you *don't* use universal newlines mode > for reading. This is the best that can be done, because > universal newlines are inherently ambiguous. I don't know PRECISELY what you mean by "universal newlines mode", and this issue is all about the details, so any response would merely enhance the confusion. > If you want universal newlines, you just have to accept > that you can't also have \r characters meaning something > other than newlines in your files. This is true regardless > of what programming language or I/O model is being used. No, that is not true, and I have used more than one model where it wasn't. Let's stick to models where newlines are special characters - I prefer the ones where they are not, but that is by the way. Model 1: certain characters can be used only in combination. E.g. \f must occur immediately before (or after) a \n, which it modifies. r is either a newline-with-overprint or must be associated with a \n. In both cases, only ONE of the alternatives is permitted in the chosen model - the other use then becomes an error (and raises an exception). Model 2: (BCPL) there are a variety of newline characters, \n for plain newline, \f for newline-with-form-feed and \r for newline- with-overprint. ALL cause a newline, with the associated property. Note that the above is what the program sees - what is written to the outside world and how input is read is another matter. But I can assure you, from my own and many other people's experience, that neither of the above models cause the confusion being shown by the postings in this thread. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1(a)cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679

2 1

Re: [Python-Dev] [python] Re: New lines, carriage returns, and Windows
by Nick Maclaren 30 Sep '07

30 Sep '07

Michael Foord <fuzzyman(a)voidspace.org.uk> wrote: > skip(a)pobox.com wrote: > > > Michael> Actually, I usually get these strings from Windows UI > > Michael> components. A file containing '\r\n' is read in with '\r\n' > > Michael> being translated to '\n'. New user input is added containing > > Michael> '\r\n' line endings. The file is written out and now contains a > > Michael> mix of '\r\n' and '\r\r\n'. > > > > So you need a translation layer between the UI component and your code. > > Treat the component as a text file and perform the desired mapping. Yes? > > Actually the problem was reported by one of the IronPython developers on > behalf of another user. We stick to using the .NET file I/O and so don't > have a problem. The only time it is an issue for us is our tests, where > we have string literals in our test code (where new lines are obviously > '\n') and we do a manual 'replace'. Not very difficult. > > It is just slightly ironic that the time Python 'gets it wrong' (for > some value of wrong) is when you are using text mode for I/O :-) Plus ca change, .... That has been the problem for as long as I have been using the byte stream model (nearly 40 years now). Provided that you can get control, OR there are well-defined semantics, you can sort things out. The semantics "we define only the trivial case, and the programmer must do something arcane, undefined and system-dependent for the rest" means that it is impossible for an interface to do the 'right' translation unless it knows what each side of it is assuming. As I say, there are solutions. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1(a)cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679

1 0

Re: [Python-Dev] New lines, carriage returns, and Windows
by Nick Maclaren 30 Sep '07

30 Sep '07

skip(a)pobox.com wrote: > > I've been thinking about this some more (in lieu of actually writing up any > sort of proposal ;-) and I'm not so sure it would be all that useful. If > you've opened a file in text mode you should only be writing newlines as > '\n' anyway. If you want to translate a text file imported from another > system to use the current system's line ending just open both the input and > output files in text mode. I.e. at least \r, \f and \v are discouraged - i.e. system-dependent, at best. That works. > With universal newlines mode for output, should writing '\r\n' result in one > or two newlines (or one-and-a-half)? Depending on the platform you can > argue that it should write out '\r\r', '\r\n\r\n' or '\n\n' or if on Windows > that it should be left alone as '\r\n'. There is, of course, the current > '\r\r\n' behavior as well. I don't think there's obviously one best answer. Quite. And it has nothing to do with the format the outside system uses - your first question is purely a matter of what the semantics of the Python program are. The question applies as much to zOS as to any of the systems Python supports. > If you want to do something esoteric, open the file in binary mode and do > whatever you like. Er, no. That's the Unix mistake. It works, provided two things are true: 1) You don't need to write portable formatting. 2) The 'outside system' uses the control characters of a byte stream for formatting. Let's skip (1) - but (2) is universally true, nowadays, isn't it? Er, no. Consider reading and writing to an X window (NOT an xterm). Such formatting is out-of-band (sorry, I used out-of-bound in a previous posting). Ouch. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1(a)cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679

1 0

Re: [Python-Dev] New lines, carriage returns, and Windows
by Nick Maclaren 30 Sep '07

30 Sep '07

"Guido van Rossum" <guido(a)python.org> wrote: > > Have you looked at Py3k at all, especially PEP 3116 (new I/O)? No. > Python *does* have its own I/O model. There are binary files and text > files. For binary files, you write bytes and the semantic model is > that of an array of bytes; byte indices are seek positions. That is the same model as C and Unix. It is text files that we are discussing. > For text files, the contents is considered to be Unicode, encoded as > bytes in a binary file. So text file always has an underlying binary > file. Two translations take place, both of which have defaults varying > by platform. One translation is encoding Unicode text into bytes upon > output, and decoding bytes to Unicode text upon input. This can use > any encoding supported by the encodings package. The character code isn't the issue here, and is almost completely irrelevant. > The other translation deals with line endings. Upon input, any of > \r\n, \r, or \n is translated to a single \n by default (this is nhe > "universal newlines" algorithm from Python 2.x). This can be tweaked > or disabled. Upon output, \n is translated into a platform specific > string chosen from \r\n, \r, or \n. This can also be disabled or > overridden. Note that \r, when written, is never treated specially; if > you want special processing for \r on output, you can write your own > translation layer. Grrk. That's the problem. You don't get back what you have written, for a start, which isn't nice. There are other issues, too. > That's all. There is nothing unimplementable or confusing in these > specifications. Nothing unimplementable, I agree. Nothing confusing? Not in the experience of the users I have dealt with. > Python doesn't care about record I/O on legacy OSes; it does care > about variability found in practice between popular OSes. As a short-term solution, that is fine. But I have seen the wheel turn a couple of times in 40 years, and expect it to continue after I am safely 6' under .... > Note that \r, \n and friends in Python 3000 are either ASCII (in bytes > literals) or Unicode (in text literals). Again, no support for legacy > systems that don't use ASCII or a superset. That's not a problem. I don't see that changing in the forseeable future. > Legacy OSes are called that for a reason. Well, I remember when the text I/O model that C, Unix and Python use WAS a feature of legacy OSs :-) Seriously. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1(a)cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679

2 1

Decimal news
by Facundo Batista 29 Sep '07

29 Sep '07

Hi people! After some months, Decimal is now in the trunk again. It's fully updated to the latest Cowlishaw specification, and complying with the latest test cases (from a few days ago, which even take in consideration some feedback from ours). I want to thank so much to Mark Dickinson, who made *a lot* of this work, not only the math part (he's a mathematician himself), but also a lot of cleaning and speeding up. Now we will put our hands in the documentation, for it to be 100% OK way before 2.6 arrives. Py3 will come after that. Regards, -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/

5 9

urllib exception compatibility
by Jim Jewett 29 Sep '07

29 Sep '07

urllib goes to goes to some trouble to ensure that it raises IOError, even when the underlying exception comes from another module.[*] I'm wondering if it would make sense to just have those modules' exceptions inherit from IOError. In particular, should socket.error, ftp.Error and httplib.HTTPException (used in Py3K) inherit from IOError? I'm also wondering whether it would be acceptable to change the details of the exceptions. For example, could raise IOError, ('ftp error', msg), sys.exc_info()[2] be reworded, or is there there too much risk that someone is checking for an "errno" of 'ftp error'? [*] This isn't a heavily tested path; some actually fail with a TypeError since 2.5, because IOError no longer accepts argument tuples longer than 3. http://bugs.python.org/issue1209 Fortunately, this makes me less worried about changing the contents of the specific attributes to something more useful... -jJ

9 17