Mailman 3 June 2010 - Python-Dev

Re: [Python-Dev] Python Library Support in 3.x (Was: email package status in 3.X)
by Simon de Vlieger June 20, 2010

June 20, 2010

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Dear all, Sorry for the maybe somewhat late response but I am not a subscriber on the python-dev mailinglists. Someone else pointed me towards this thread and I want to shortly clarify a few things regarding the following two statements: > It is not "critical self-evaluation" to repeat "Python 3 is not > ready" as > litany in #Python and your supporting website. I use the word > "litany" here > because #Python refers users … [View More]to what appears to be a religious website > http://python-commandments.org/python3.html > python-commandments.org is owned and hosted by the same person > (Allen Short > aka dash aka washort) as pound-python.org which is the "official" > website > for #Python and which links to it. Both python-commandments.org and pound-python.org are my websites. I own both the domains and I do all administrative tasks regarding these domains. pound-python.org is the official #python website and as such is maintained on Launchpad by a team of volunteers, see: https://launchpad.net/ ~pound-python which is indeed owned by Allen Short. However, Allen Short has nothing to do with the Python Commandments page. That is an endeavor for which I am the sole responsible person. I have asked some people to contribute texts but that doesn't change that I should be spoken to regarding the content on that website. If there are any issues with the content on either website please do not hesitate to contact me at this email address or on IRC where I go by the nickname of ikanobori. As for the potentially harmful text on Python 3 which is included on the python-commandments website I do get the hint that it might not be clear enough that the text does not apply to people who are porting libraries. This is a complaint I have heard before and to which I will take affirmative action by explicitly adding text to clarify that. Hope all is well, Regards, Simon de Vlieger -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) iQIcBAEBAgAGBQJMHSC2AAoJEBBSHP7i+JXf5pMQANPBCUzDwx2xjTP8shA1E4mx 7/OQk27nxt+wOZNT0Ybe/iNXLetF6qa8At7kTau/yU3l/xJWVODjfJUICkDv/0ad ebMKiFeKO8jqdvEe+RL3ck7jTXEM73C2PLNtge9FLTY6HhYrXnOJakNbpWPJR/PG TQQ+mY/8ZvSP+n98RrY9kcVaVJMSmXUJWHvWVh+LkcIDwF/h30EH/e5PUGzylINI NiV5955pNRXTnwdgjsouljUI/rrod3zphnUEyL22QvSUx0b7YXMfC24eRGTpwrLg 9cyQAMjjbuVqkhSJhYFnm+DKwsZEAHxxOvu50Xwuy3i1C7c8L6/QDT1txoSTVuaP 4xw8GSFEblbHviz7hY7KCe5nMpBNHNfcGFHFSWd+WYogRXjpDitlMDNW8HT56pRW lwzs1WENnoOSCAn4Xds+xPJj9JyAGnS8rWz70RVMyrkHDFaJhDlIDNpEFdlAlywT R0uCQrlxs/uWzAXK2IA0wXPtm/m8fYLR3q8mD4++QotZKQcT4ciN7Xv913/ZT2b2 NtR1WEoTZAV+gWrFyFsgmMFAmZhvUdI8Ludxs3l2smHHaCFUkj2Ur9BrkMiEv5Z8 wLN+/LRaHgGnmVT2SF0LOCeOLz97dP728OKBO0DwxqT89Cla8445z7ktdHnJ3amA gjbsfG7W+yx9L2v0IDFC =YDiR -----END PGP SIGNATURE----- [View Less]

3 2

Year 0 and year 10,000 in timetuple
by Alexander Belopolsky June 20, 2010

June 20, 2010

While datetime range is limited to years from 1 through 9999, it is possible to produce time tuple with year 0 or year 10,000: >>> t1 = datetime.min.replace(tzinfo=timezone.max) >>> t2 = datetime.max.replace(tzinfo=timezone.min) >>> t1.utctimetuple().tm_year 0 >>> t2.utctimetuple().tm_year 10000 Most if not all functions consuming timetuples are not designed to handle years beyond 9999 and such timetuples cannot be converted back to datetime. I would like … [View More]

2 3

Re: [Python-Dev] email package status in 3.X
by lutz＠rmi.net June 18, 2010

June 18, 2010

Replying en masse to save bandwidth here... Barry Warsaw <barry(a)python.org> writes: > We know it, we have extensively discussed how to fix it, we have IMO a good > design, and we even have someone willing and able to tackle the problem. We > need to find a sufficient source of funding to enable him to do the work it > will take, and so far that's been the biggest stumbling block. It will take a > focused and determined effort to see this through, and it's obvious that … [View More]> volunteers cannot make it happen. I include myself in the latter category, as > I've tried and failed at least twice to do it in my spare time. All understood, and again, not to disparage anyone here. My comments are directed to the development community at large to underscore the grave p/r problems 3.X faces. I realize email parsing is a known issue; I also realize that most people evaluating 3.X today won't care that it is. Most will care only that the new version of a language reportedly used by Google and YouTube still doesn't support CGI uploads a year and a half after its release. As an author, that's a downright horrible story to have to tell the world. "Stephen J. Turnbull" <stephen(a)xemacs.org> writes: > Email, of course, is a big wart. But guess what? Python 2's email > module doesn't actually work! Yes it does (see next point). > If you live in Kansas, sure, you can concentrate on dodging tornados > and completely forget about Unicode and MIME and text/bogus content. > For the rest of the world, though, the problem is not Python 3 Yes it is, and Kansas is a lot bigger than you seem to think. I want to reiterate that I was able to build a feature rich email client with the email package as it exists in 3.1. This includes support on both the receiving and sending sides for HTML, arbitrary attachments, and decoding and encoding of both text payloads and headers according to email, MIME, and Unicode/I18N standards. It's an amazingly useful package, and does work as is in 3.X. The two main issues I found have been recently fixed. It's unfortunate that this package is also the culprit behind CGI breakage, but it's not clear why it became a critical path for so much utility in the first place. The package might not be aesthetically ideal, but to me it seems that an utterly incompatible overhaul of this in the name of supporting potentially very different data streams is a huge functional overload. And to those people in Kansas who live outside the pydev clique, replacing it with something different at this point will look as if an incompatible Python is already incompatible with releases in its own line. Why in the world would anyone base a new project on that sort of thrashing? For my part, I've had to add far too many notes to the upcoming edition of Programming Python about major pieces of functionality that worked in 2.X but no longer do in 3.X. That's disappointing to me personally, but it will probably seem a lot worse to the book's tens of thousands of readers. Yet this is the reality that 3.X has created for itself. > Should Python 3 have been held back until email was fixed? Dunno, but > I personally am very glad it was not; where I have a choice, I always > use Python 3 now, and have yet to run into a problem. I guess we'll just have to disagree on that. IMHO, Python 3 shot itself in the foot by releasing in half-baked form. And the 3.0 I/O speed issue (remember that?) came very close to blowing its leg clean off. The reality out there in Kansas today is that 3.X is perceived as so bad that it could very well go the way of POP4 if its story does not improve. I don't know what sort of Python world will be left behind in the wake, but I do know it will probably be much smaller. Steve Holden <steve(a)holdenweb.com> writes: > Lest the readership think that the PSF is unaware of this issue, allow > me to point out that we have already partially funded this effort, and > are still offering R. David Murray some further matching funds if he can > raise sponsorship to complete the effort (on which he has made a very > promising start). > > We are also attempting to enable tax-deductible fund raising to increase > the likelihood of David's finding support. Perhaps we need to think > about a broader campaign to increase the quality of the python 3 > libraries. I find it very annoying that the #python IRC group still has > "Don't use Python 3" in it's topic. They adamantly refuse to remove it > until there is better library support, and they are the guys who see the > issues day in day out so it is hard to argue with them (and I don't > think an autocratic decision-making process would be appropriate). I'm all for people getting paid for work they do, but with all due respect, I think this underscores part of the problem in the Python world today. If funding had been as stringent a prerequisite in the 90s, I doubt there would be a Python today. It was about the fun and the code, not the bucks and the bureaucracy. As far as I can recall, there was no notion of creating a task force to get things done. Of course, this may just be the natural evolutionary pattern of human enterprises. As it is today, though, the Python community has a formal diversity statement, but it still does not have a fully functional 3.X almost two years after the fact. I doubt that I'm the only one who sees the irony in that. Again, I mean no disrespect to people contributing to Python today on so many fronts, and I have no answers to offer here. For better or worse, though, this is a personal issue to me too. After spending much of the last 2 years updating the best selling Python books for all the changes this group has seen fit to make, I believe I can say with some authority that 3.X still faces a very uncertain future. --Mark Lutz (http://learning-python.com, http://rmi.net/~lutz) [View Less]

3 2

Re: [Python-Dev] email package status in 3.X
by lutz＠rmi.net June 18, 2010

June 18, 2010

[copied to pydev from email-sig because of the broader scope] Well, it looks like I've stumbled onto the "other shoe" on this issue--that the email package's problems are also apparently behind the fact that CGI binary file uploads don't work in 3.1 (http://bugs.python.org/issue4953). Yikes. I trust that people realize this is a show-stopper for broader Python 3.X adoption. Why 3.0 was rolled out anyhow is beyond me; it seems that it would have been better if Python developers had gotten … [View More]their own code to work with 3.X, before expecting the world at large to do so. FWIW, after rewriting Programming Python for 3.1, 3.x still feels a lot like a beta to me, almost 2 years after its release. How did this happen? Maybe nobody is using 3.X enough to care, but I have a feeling that issues like this are part of the reason why. No offense to people who obviously put in an incredible amount of work on 3.X. As someone who remembers 0.X, though, it's hard not to find the current situation a bit disappointing. --Mark Lutz (http://learning-python.com, http://rmi.net/~lutz) > -----Original Message----- > From: lutz(a)rmi.net > To: "R. David Murray" <rdmurray(a)bitdance.com> > Subject: Re: email package status in 3.X > Date: Sun, 13 Jun 2010 15:30:06 -0000 > > Come to think of it, here was another oddness I just recalled: this > may have been reported already, but header decoding returns mixed types > depending upon the structure of the header. Converting to a str for > display isn't too difficult to handle, but this seems a bit inconsistent > and contrary to Python's type neutrality: > > >>> from email.header import decode_header > >>> S1 = 'Man where did you get that assistant?' > >>> S2 = '=?utf-8?q?Man_where_did_you_get_that_assistant=3F?=' > >>> S3 = 'Man where did you get that =?UTF-8?Q?assistant=3F?=' > > # str: don't decode() > >>> decode_header(S1) > [('Man where did you get that assistant?', None)] > > # bytes: do decode() > >>> decode_header(S2) > [(b'Man where did you get that assistant?', 'utf-8')] > > # bytes: do decode(), using raw-unicode-escape applied in package > >>> decode_header(S3) > [(b'Man where did you get that', None), (b'assistant?', 'utf-8')] > > I can work around this with the following code, but it > feels a bit too tightly coupled to the package's internal details > (further evidence that email.* can be made to work as is today, > even if it may be seen as less than ideal aesthetically): > > parts = email.header.decode_header(rawheader) > decoded = [] > for (part, enc) in parts: # for all substrings > if enc == None: # part unencoded? > if not isinstance(part, bytes): # str: full hdr unencoded > decoded += [part] # else do unicode decode > else: > decoded += [part.decode('raw-unicode-escape')] > else: > decoded += [part.decode(enc)] > return ' '.join(decoded) > > Thanks, > --Mark Lutz (http://learning-python.com, http://rmi.net/~lutz) > > > > -----Original Message----- > > From: lutz(a)rmi.net > > To: "R. David Murray" <rdmurray(a)bitdance.com> > > Subject: Re: email package status in 3.X > > Date: Sat, 12 Jun 2010 16:52:32 -0000 > > > > Hi David, > > > > All sounds good, and thanks again for all your work on this. > > > > I appreciate the difficulties of moving this package to 3.X > > in a backward-compatible way. My suggestions stem from the fact > > that it does work as is today, albeit in a less than ideal way. > > > > That, and I'm seeing that Python 3.X in general is still having > > a great deal of trouble gaining traction in the "real world" > > almost 2 years after its release, and I'd hate to see further > > disincentives for people to migrate. This is a bigger issue > > than both the email package and this thread, of course. > > > > > > 3) Type-dependent text part encoding > > > > > > > ... > > > So, in the next releases of Python all MIMEText input should be string, > > > and it will fail if you pass bytes. I consider this as email previously > > > not living up to its published API, but do you think I should hack > > > in a way for it to accept bytes too, for backward compatibility in the > > > 3 line? > > > > Decoding can probably be safely delegated to package clients. > > Typical email clients will probably have str for display of the > > main text. They may wish to read attachments in binary mode, but > > can always read in text mode instead or decode manualy, because > > they need a known encoding to send the part correctly (my client > > has to ask or use configurations in some cases). > > > > B/W compatibility probably isn't a concern; I suspect that my > > temporary workaround will still work with your patch anyhow, > > and this code didn't work at all for some encodings before. > > > > > > There are some additional cases that now require decoding per mail > > > > headers today due to the str/bytes split, but these are just a > > > > normal artifact of supporting Unicode character sets in general, > > > > ans seem like issues for package client to resolve (e.g., the bytes > > > > returned for decoded payloads in 3.X didn't play well with existing > > > > str-based text processing code written for 2.X). > > > > > > I'm not following you here. Can you give me some more specific > > > examples? Even if these "normal artifacts" must remain with > > > the current API, I'd like to make things as easy as practical when > > > using the new API. > > > > This was just a general statement about things in my own code that > > didn't jive with the 3.X string model. For instance, line wrapping > > logic assumed str; tkinter text widgets do much better rendering str > > than the bytes fetched for decoded payloads; and my Pyedit text editor > > component had to be overhauled to handle display/edit/save of payloads > > of arbitrary encodings. If I remember any more specific issues with > > the email package itself, I'll forward your way. > > > > I'll watch for an opportunity to get the book's new PyMailGUI > > client code to you as a candidate test case, but please ping > > me about it later if I haven't acted on this. It works well, > > but largely because of all the work that went into the email > > package underlying it. > > > > Thanks, > > --Mark Lutz (http://learning-python.com, http://rmi.net/~lutz) > > > > > > > -----Original Message----- > > > From: "R. David Murray" <rdmurray(a)bitdance.com> > > > To: lutz(a)rmi.net > > > Subject: Re: email package status in 3.X > > > Date: Thu, 10 Jun 2010 10:18:48 -0400 > > > > > > On Thu, 10 Jun 2010 09:21:52 -0400, lutz(a)rmi.net wrote: > > > > In other words, some of my concern may have been a bit premature. > > > > I hope that in the future we'll either strive for compatibility > > > > or keep the current version around; it's a lot of very useful code. > > > > > > The plan is to have a compatibility layer that will accept calls based > > > on the old API and forward appropriately to the new API. So far I'm > > > thinking I can succeed in doing this in a fairly straightforward manner, > > > but I won't know for sure until I get some more pieces in place. > > > > > > > In fact, I recommend that any new email package be named distinctly, > > > > > > I'm going to avoid that if I can (though the PyPI package will be > > > named email6 when we publish it for public testing). If, however, > > > it turns out that I can't correctly support both the old and the > > > new API, then I'll have to do that. > > > > > > > and that the current package be retained for a number of releases to > > > > come. After all the breakages that 3.X introduced in general, doing > > > > the same to any email-based code seems a bit too much, especially > > > > given that the current package is largely functional as is. To me, > > > > after having just used it extensively, fixing its few issues seems > > > > a better approach than starting from scratch. > > > > > > Well, the thing is, as you found, existing 2.x code needs to be fixed to > > > correctly handle the distinction between strings and bytes no matter what. > > > The goal is to make it easier to write correct programs, while providing > > > the compatibility layer to make porting smoother. But I doubt that any > > > non-trivial 2.x email program will port without significant changes, > > > even if the compatibility layer is close to 100% compatible with the > > > current Python3 email package, simply because the previous conflation > > > of text and bytes must be untangled in order to work correctly in > > > Python3, and email involves lots of transitions between text and bytes. > > > > > > As for "starting from scratch", it is true that the current plan involves > > > considerable changes in the recommended API (in the direction of greater > > > flexibility and power), but I'm hoping that significant portions of the > > > code will carry forward with minor changes, and that this will make it > > > easier to support the old API. > > > > > > > As far as other issues, the things I found are described below my > > > > signature. I don't know what the utf-8 issue is that you refer > > > > too; I'm able to parse and send with this encoding as is without > > > > problems (both payloads and headers), but I'm probably not using the > > > > interfaces you fixed, and this may be the same as one of item listed. > > > > > > It is, see below. > > > > > > > Another thought: it might be useful to use the book's email client > > > > as a sort of test case for the package; it's much more rigorous in > > > > the new edition because it now has to be given 3.X'Unicode model > > > > (it's abut 4,900 lines of code, though not all is email-related). > > > > I'd be happy to donate the code as soon as I find out what the > > > > copyright will be this time around; it will be at O'Reilly's site > > > > this Fall in any event. > > > > > > That would be great. I am planning to write my own sample ap to > > > demonstrate the new API, but if I can use yours to test the compatibility > > > layer that will help a lot, since I otherwise have no Python3 email > > > application to test against unless I port something from Python2. > > > > > > > Major issues I found... > > > > ------------------------------------------------------------------ > > > > 1) Str required for parsing, but bytes returned from poplib > > > > > > > > The initial decode from bytes to str of full mail text; in > > > > retrospect, probably not a major issue, since original email > > > > standards called for ASCII. A 8-bit encoding like Latin-1 is > > > > probably sufficient for most conforming mails. For the book, > > > > I try a set of different encodings, beginning with an optional > > > > configuration module setting, then ascii, latin-1, and utf-8; > > > > this is probably overkill, but a GUI has to be defensive. > > > > > > This works (mostly) for conforming email, but some important Python email > > > applications need to deal with non-conforming email. That's where the > > > inability to parse bytes directly really causes problems. > > > > > > > 2) Binary attachments encoding > > > > > > > > The binary attachments byte-to-str issue that you've just > > > > fixed. As I mentioned, I worked around this by passing in a > > > > custom encoder that calls the original and runs an extra decode > > > > step. Here's what my fix looked like in the book; your patch > > > > may do better, and I will minimally add a note about the 3.1.3 > > > > and 3.2 fix for this: > > > > > > Yeah, our patch was a lot simpler since we could fix the encoding inside > > > the loop producing the encoded lines :) > > > > > > > 3) Type-dependent text part encoding > > > > > > > > There's a str/bytes confusion issue related to Unicode encodings > > > > in text payload generation: some encodings require the payload to > > > > be str, but others expect bytes. Unfortunately, this means that > > > > clients need to know how the package will react to the encoding > > > > that is used, and special-case based upon that. > > > > > > This was the UTF-8 bug I fixed. I shouldn't have called it "the UTF-8 > > > bug", because it applies equally to the other charsets that use base64, > > > as you note. I called it that because UTF-8 was where the problem was > > > noticed and is mentioned in the title of the bug report. > > > > > > I had a suspicion that the quoted-printable encoding wasn't being done > > > correctly either, so to hear that it is working for you is good news. > > > There may still be bugs to find there, though. > > > > > > So, in the next releases of Python all MIMEText input should be string, > > > and it will fail if you pass bytes. I consider this as email previously > > > not living up to its published API, but do you think I should hack > > > in a way for it to accept bytes too, for backward compatibility in the > > > 3 line? > > > > > > > There are some additional cases that now require decoding per mail > > > > headers today due to the str/bytes split, but these are just a > > > > normal artifact of supporting Unicode character sets in general, > > > > ans seem like issues for package client to resolve (e.g., the bytes > > > > returned for decoded payloads in 3.X didn't play well with existing > > > > str-based text processing code written for 2.X). > > > > > > I'm not following you here. Can you give me some more specific > > > examples? Even if these "normal artifacts" must remain with > > > the current API, I'd like to make things as easy as practical when > > > using the new API. > > > > > > Thanks for all your feedback! > > > > > > --David > > > > > > > > > > > > [View Less]

9 11

Summary of Python tracker Issues
by Python tracker June 18, 2010

June 18, 2010

ACTIVITY SUMMARY (2010-06-11 - 2010-06-18) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2777 open (+43) / 18070 closed (+12) / 20847 total (+55) Open issues with patches: 1122 Average duration of open issues: 713 days. Median duration of open issues: 503 days. Open Issues Breakdown open 2747 (+43) languishing 13 ( +0) pending 16 ( +0) Issues Created Or … [View More]

1 0

cmdline arguments in test_support.run_unittest
by anatoly techtonik June 18, 2010

June 18, 2010

I thought that some arguments to test_support.run_unittest would be useful. Would like to hear your feedback before making anything. http://bugs.python.org/issue9028 -- anatoly t.

1 0

debug and release python
by Kristján Valur Jónsson June 17, 2010

June 17, 2010

Hello there. I'm sure this has come up before, but here it is again: Python exports a different api in debug mode, depending on whether PYMALLOC_DEBUG and WITH_PYMALLOC are exported. This means that _d.pyd files that are used must have been compiled with a version of python using the same settings for these macros. It is unfortunate that the _PyObject_DebugMalloc() api is exposed to external applications using macros in objimpl.h I would suggest two things: 1) provide dummy or … [View More]

3 7

Sharing functions between C extension modules in stdlib
by Alexander Belopolsky June 16, 2010

June 16, 2010

I have learned a long time ago that it is not enough to simply declare a function in some header file if you want to define it in one module and use in another. You have to use what now is known as PyCapsule - an array of pointers to C functions wrapped in a Python object. However, while navigating through the time/datetime maze recently I have come across timefuncs.h which seems to share _PyTime_DoubleToTimet between time and datetime modules. I did not expect this to work, but apparently … [View More]

2 3

Release manager/developer (10 years + experience) would like to help and volunteer time if needed
by Mart June 16, 2010

June 16, 2010

Hi, I have worked 10 years at Adobe Systems as a Release Developer for the LiveCycle ES team and am now employed as a Release Manager (for a team of one, me ;) ) at Nuance Communications since last March. I have put lots of effort to keep Python alive and well at Adobe by providing complete build/release solutions & processes, automation and tooling in my favourite language, Python. I have been promoting, planning and implementing a completely new build/release infrastructure at Nuance, … [View More]

2 2

Re: [Python-Dev] Release manager/developer (10 years + experience) would like to help and volunteer time if needed
by Senthil Kumaran June 16, 2010

June 16, 2010

Welcome! You might just want to hook on to the process mentioned at http://www.python.org/dev That's it. -- Senthil On 16 Jun 2010 16:44, "Mart" <msenecal.sc(a)gmail.com> wrote: Hi, I have worked 10 years at Adobe Systems as a Release Developer for the LiveCycle ES team and am now employed as a Release Manager (for a team of one, me ;) ) at Nuance Communications since last March. I have put lots of effort to keep Python alive and well at Adobe by providing complete build/release … [View More]

1 0