Mailman 3 April 2008 - Python-Dev

python hangs when parsing a bad-formed email
by Alberto Casado Martín 22 Apr '08

22 Apr '08

Hi all, First of all, sorry if this isn't the list where I have to post this. And sorry for my english. As the subject says, I'm having problems with the attached email, when I try to get a email object reading the attached file, the python process gets hang and gets all cpu. I have debuged my code to find where it happens, and I found that is _parsegen method of the FeedParser class. I know that the email format is wrong but I don't know why python hangs. following paste the code showing where hangs. def _parsegen(self): # Create a new message and start by parsing headers. self._new_message() headers = [] # Collect the headers, searching for a line that doesn't match the RFC # 2822 header or continuation pattern (including an empty line). for line in self._input: if line is NeedMoreData: yield NeedMoreData continue if not headerRE.match(line): # If we saw the RFC defined header/body separator # (i.e. newline), just throw it away. Otherwise the line is # part of the body so push it back. if not NLCRE.match(line): self._input.unreadline(line) break headers.append(line) # Done with the headers, so parse them and figure out what we're # supposed to see in the body of the message. self._parse_headers(headers) # Headers-only parsing is a backwards compatibility hack, which was # necessary in the older parser, which could throw errors. All # remaining lines in the input are thrown into the message body. if self._headersonly: lines = [] while True: line = self._input.readline() if line is NeedMoreData: yield NeedMoreData continue if line == '': break lines.append(line) self._cur.set_payload(EMPTYSTRING.join(lines)) return if self._cur.get_content_type() == 'message/delivery-status': !!!!!! AT THIS POINT HANGS, AND STRAT TO GET ALL CPU FOR THE PROCESS # message/delivery-status contains blocks of headers separated by # a blank line. We'll represent each header block as a separate # nested message object, but the processing is a bit different # than standard message/* types because there is no body for the # nested messages. A blank line separates the subparts. ... ... ... I have workaround the problem adding this line in _parse_headers method def _parse_headers(self, lines): # Passed a list of lines that make up the headers for the current msg lastheader = '' lastvalue = [] for lineno, line in enumerate(lines): # Check for continuation if line[0] in ' \t': if not lastheader: # The first line of the headers was a continuation. This # is illegal, so let's note the defect, store the illegal # line, and ignore it for purposes of headers. defect = errors.FirstHeaderLineIsContinuationDefect(line) self._cur.defects.append(defect) continue if line.strip()!='': !!!!!!! IF THE CONTINUATION LINE IS NOT EMPTY ADD THE LINE TO THE HEADER. lastvalue.append(line) continue if lastheader: ... ... ... I don't know why it hangs and I'm not sure why with this line works...... I have tried to parse this email in python 2.3.3 SunOs, python 2.3.3 gcc python 2.5.1 SunOs,gcc, Windows Xp, and linux SUSE 10. And I have alway the same result. bash-3.00$ python Python 2.5.1 (r251:54863, Feb 28 2008, 07:48:25) [GCC 3.4.6] on sunos5 Type "help", "copyright", "credits" or "license" for more information. >>> import email >>> fp = open('raro.txt') >>> mail = email.message_from_file(fp) never return............ I don't know if someone can tell me what is happening.... Best Regards. Alberto Casado.

3 2

Known doctest bug with unicode?
by Jeroen Ruigrok van der Werven 22 Apr '08

22 Apr '08

Is it a known doctest bug that when you have a dict with Unicode key values that doctest dies with a KeyError? When I excute my code from the regular python interpreter it works as expected. -- Jeroen Ruigrok van der Werven <asmodai(-at-)in-nomine.org> / asmodai イェルーンラウフロックヴァンデルウェルヴェン http://www.in-nomine.org/ | http://www.rangaku.org/ Whenever you meet difficult situations dash forward bravely and joyfully...

4 5

Re: [Python-Dev] Encoding detection in the standard library?
by M.-A. Lemburg 22 Apr '08

22 Apr '08

[CCing python-dev again] On 2008-04-22 12:38, Greg Wilson wrote: >>> I don't think that should be part of the standard library. People >>> will mistake what it tells them for certain. >>> [etc] > > These are all good arguments, but the fact remains that we can't control > our inputs (e.g., we're archiving mail messages sent to lists managed by > DrProject), and some of those inputs *don't* tell us how they're encoded. > Under those circumstances, what would you recommend? I haven't done much research into this, but in general, I think it's better to: * first try to look at other characteristics of a text message, e.g. language, origin, topic, etc., * then narrow down the number of encodings which could apply, * rank them to try to avoid ambiguities and * then try to see what percentage of the text you can decode using each of the encodings in reverse ranking order (ie. more specialized encodings should be tested first, latin-1 last). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 22 2008) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ :::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611

1 0

3k checkin mails to python-checkins
by Georg Brandl 22 Apr '08

22 Apr '08

Since a few days, checkin notifications for the 3k branch seem to be sent to both the python-checkins and the python-3000-checkins lists. Was that a deliberate decision or has some bug crept into the SVN hook? Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out.

2 2

configure error: "rm: conftest.dSYM: is a directory"
by skip＠pobox.com 22 Apr '08

22 Apr '08

I just noticed this error message during configure: checking whether gcc accepts -Olimit 1500... no checking whether gcc supports ParseTuple __format__... no checking whether pthreads are available without options... yes checking whether g++ also accepts flags for thread support... no checking for ANSI C header files... rm: conftest.dSYM: is a directory rm: conftest.dSYM: is a directory yes checking for sys/types.h... yes checking for sys/stat.h... yes checking for stdlib.h... yes checking for string.h... yes Note the "rm: conftest.dSYM: is a directory". This occurred a few times during the configure process. Didn't cause it to conk out, but is annoying. Skip

4 4

Re: [Python-Dev] r62342 - python/branches/py3k/Objects/bytesobject.c
by Christian Heimes 22 Apr '08

22 Apr '08

alexandre.vassalotti schrieb: > Author: alexandre.vassalotti > Date: Mon Apr 14 22:51:05 2008 > New Revision: 62342 > > Log: > Improved bytes_extend() to avoid making a full copy of the temporary > buffer. This also makes the code slightly cleaner. Changes to Objects/bytesobject.c should be applied to the trunk and merged into the py3k branch via svnmerge.py. We need to agree on a policy how we are going to sync the trunk and py3k for new code like bytesobject.c and io.py. The former is easy because the file is almost identical. The later is going to be hard because 2.6 doesn't have annotations. Collin: How hard is it to write a fixer that removes all annotations from functions? A set of small 3to2 fixers for annotations and metaclasses would make the syncing job much easier. Christian

5 8

very bad network performance
by Ralf Schmitt 21 Apr '08

21 Apr '08

Hi all, I'm using mercurial with the release25-maint branch. I noticed that checking out a local repository now takes more than 5 minutes (it should be around 30s). I've tracked it down to this change: http://hgpy.de/py/release25-maint/rev/e9446c6ab3cd this is svn revision 61009. Here is the diff inline: --- a/Lib/socket.py Fri Mar 23 14:27:29 2007 +0100 +++ b/Lib/socket.py Sat Feb 23 20:30:59 2008 +0100 @@ -305,7 +305,7 @@ self._rbuf = "" while True: left = size - buf_len - recv_size = max(self._rbufsize, left) + recv_size = min(self._rbufsize, left) data = self._sock.recv(recv_size) if not data: break self._rbufsize if 1, and so the code reads one byte at a time. this is clearly wrong, I'm posting it to the mailing list, as I don't want this issue to get lost in the bugtracker. - Ralf

7 17

Encoding detection in the standard library?
by Jim Jewett 21 Apr '08

21 Apr '08

David Wolever wrote: > IMO, encoding estimation is something that > many web programs will have to deal with, > so it might as well be built in; I would prefer > the option to run `text=input.encode('guess')` > (or something similar) than relying on an external > dependency or worse yet using a hand-rolled > algorithm The (still draft) html5 spec is trying to get error-correction standardized, so it includes all sort of "if this fails, do X". Encoding detection will be standardized, so there will be an external standard that we can reference. http://dev.w3.org/html5/spec/Overview.html#determining Note that this portion of the spec is probably not stable yet, as there was some new analysis on which "wrong" answers provided better results on real world web pages. e.g., http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2008-March/014127.html http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2008-March/014190.html There was also a recent analysis of how many characters it takes to sniff successfully X% of the time on today's web, though I'm not finding it at the moment. -jJ

1 0

thoughts on having EOFError inherit from EnvironmentError?
by Gregory P. Smith 21 Apr '08

21 Apr '08

http://bugs.python.org/issue1481036 Basically as things are now EOFError is on its own but often wants to be handled the same as other I/O errors that EnvironmentError currently covers. Many uses of EOFError in our code base do not provide it any arguments so it doesn't really fit the (errno, message [, filename]) tuple style that EnvironmentError promises. But we could fudge that with a reasonable default (whats reasonable?) if we rerooted this under EnvironmentError. Alternatively the bug suggests a new parent exception for EnvironmentError and EOFError both to inherit from. Last time changing the heirarchy around this came up there was pushback against adding yet another exception type so I'm thinking the simple re-rooting may be the best answer if anything is done at all. any thoughts? -gps

5 14

unscriptable?
by Jim Jewett 21 Apr '08

21 Apr '08

> I dispute this. Indices aren't necessarily numeric > (think of an A-Z file), Python has recently added an __index__ slot which means "as an integer, and I really am an integer, I'm not just rounding like int(3.4) would do" So in the context of python, an index is numeric, whereas "subscript" has already been used for hashtables with arbitrary keys. -jJ

3 2