Mailman 3 March 2008 - Python-Dev

3rd party developers: don't change your APIs when porting to Py3k!
by Guido van Rossum March 18, 2008

March 18, 2008

For those who don't read blogs, I just blogged the slides for my keynote, and added an important admonishment to 3rd party developers. Here's the full text of the blog: The slides of my `keynote`_ are now up on python.org. There's both a `PowerPoint`_ and a `PDF`_ file. .. _keynote: http://www.python.org/doc/essays/ppt/ .. _PowerPoint: http://www.python.org/doc/essays/ppt/pycon2008/Py3kAndYou.ppt .. _PDF: http://www.python.org/doc/essays/ppt/pycon2008/Py3kAndYou.pdf I'd like to take this opportunity to remind you of a really important issue that I neglected to mention in the talk: **Don't change your APIs incompatibly when porting to Py3k.** Yes, you heard that right: even though Python 3.0 is changing incompatibly, I implore you (especially if you're maintaining a library that's used by others) *not* to make incompatible changes to your API. If you *have* make API changes, do them *before* you port to 3.0 -- release a version with the new API for Python 2.5, or 2.6 if you must. (Or do it later, *after* you've released a port to 3.0 without adding new features.) Why? Think of your users. Suppose Ima Lumberjack has implemented a web 2.0 app for managing his sawmill. Ima is a happy user of your most excellent web 2.0 framework. Now Ima wants to upgrade his app to Py3k. He waits until you have ported your framework to Py3k. He does everything by the books, runs his source code through the 2to3 tool, and starts testing. Imagine his despair when the tests fail: how is he going to tell whether the breakage is due to your API changes or due to his own code not being Py3k-ready? On the other hand, if port your web 2.0 framework to Py3k *without* making API changes, Ima's task is much more focused: the bugs he is left with after running 2to3 are definitely in his own code, which (presumably :-) he knows how to debug and fix. The same recommendation applies even more strongly if your library is a dependency for other libraries -- due to the fan-out the pain caused to others multiplies. If one of those packages gives up (even temporarily) its Py3k porting effort, this may prevent many other libraries and apps from being ported at all! So, once more for emphasis: **Don't change your APIs at the same time as porting to Py3k!** *PS.* The 3.0final release is now scheduled for September 3, 2008. See `PEP 361`_. .. _PEP 361: http://python.org/dev/peps/pep-0361/ -- --Guido van Rossum (home page: http://www.python.org/~guido/)

3 2

xturtle and 3.0
by Guido van Rossum March 18, 2008

March 18, 2008

I'm changing the subject to keep this separate from the project management tools discussion. On Sun, Mar 16, 2008 at 10:13 AM, Gregor Lingl <gregor.lingl(a)aon.at> wrote: > > Hi everyone, > > with this posting I refer to a paragraph in PEP 361, which says: > > """Each non-trivial feature listed here that is not a PEP must be discussed on python-dev. Other enhancements include: > > - ... > - turtle.py replacement or enhancements > """ > > Some time ago I had offered my xturtle.py as a replacement of or > supplement to turtle.py. The discussion that followed is here: > > http://www.python.org/dev/summary/2006-06-16_2006-06-30/ > > At Europython 2006 I gave a talk on xturtle and there and since then I > had quite positive feedback including encouragement to offer it again to > include it in the python standard distribution by quite a few people > including Guido van Rossum. > > During the last few weeks I did some enhancements to xturtle and put the > current version on the xturtle website for download in order to get same > feedback about the API as well as possible bug reports. This version > still needs some code polishing. > > http://www.rg16.at/~python/xturtle/download.html > > So I'm interested to know if this is still an issue for you. If so there > should be initiated some procedure to decide that. I think that for 3.0, replacing turtle with xturtle is great, *IF* xturtle doesn't add any additional dependencies *AND* it works well on Mac OSX, Linux and Windows. > If this decision were negative, things were done (- and I'd continue to > develop xturtle elsewhere.) > > If the decision were positive, I'd be able to prepare two equivalent > versions for Python2.6 and Python3000 within two or three weeks. (The > port to Python3000 is nearly ready.) These could include say 85% of the > documentation, albeit still not in the correct format. That sounds cool. In 2.6 I'm reluctant to replace the existing turtle module; xturtle can be added as xturtle. > I think these had to be examined my some reviewer(s) and also a > discussion about features to include or not include would be useful. I'd > like to intensivly take part in this discussion and development. > > After a possible decision to include xturtle into Python, which > certainly should take place before the first beta release, there would > be enough time to polish the documentation and to fix bugs. For their > discovery it would certainly be an advantage to put it in some > prerelease as early as possible. > > Of course I know that xturtle is only a side issue in the current > development efforts. Unfortunately I'm not familiar with the procedures > needed to get a new module into Python, so I kindly ask you for your > advice how to proceed, at the same time offering my cooperation. I think that for a library module like this, an email like you've sent is just fine. Maybe Brett has a suggestion on whether it would remain a toplevel module or could be placed in some umbrella package (is Tkinter being moved around?). -- --Guido van Rossum (home page: http://www.python.org/~guido/)

4 5

New/Old class exception pitfall
by Alexander Belopolsky March 17, 2008

March 17, 2008

While discussing issue2291, I presented the following argument: """ Consider the following code: class x: pass class y(x): pass try: raise y except y: print "a" except: print "b" It prints 'b'. Now, suppose in preparation for 3.0 transition someone adds "__metaclass__ = type" to the module with that code. The result: it prints 'a'. Since the difference in behavior is in error handling code, which in my experience is often not thoroughly tested, the bug introduced by a seemingly innocuous move from old to new style classes is likely to trigger in the worst possible moment. (For example a wrong roll-back logic is applied after a failed database commit.) """ http://bugs.python.org/msg63584 This issue is only partially alleviated by the -3 warning because the warning is not issued unless the error condition raising a new style class not deriving from BaseException is actually tested for. It is my understanding that subclass check is skipped for new style classes not derived from BaseException in order to enable the identity check when a string exception is caught. With the deprecation of string exceptions, this logic is hard to justify. In any case, I believe this issue is either code or documentation bug: """ Exceptions are identified by class instances. The except clause is selected depending on the class of the instance: it must reference the class of the instance or a base class thereof. """ http://docs.python.org/dev/reference/executionmodel.html#id2 I don't see anything in the documentation that would suggest that old and new class instances should behave differently.

3 5

Currently adding known 2.6 showstoppers to the tracker
by Brett Cannon March 17, 2008

March 17, 2008

Right now at the sprint I am going through a list of issues Neal and I compiled of what needs to happen to get 2.6/3.0 out the door (although the list is pretty much 2.6-specific). They are all being flagged as "immediate" priority. Hopefully it won't take too long to add all of them (although the list ain't short =). If you want to see the list, Martin added a pre-defined Showstoppers search to make it easy to list everything that is set to "immediate". -Brett

4 4

Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
by Phillip J. Eby March 17, 2008

March 17, 2008

At 01:59 PM 3/17/2008 -0500, Guido van Rossum wrote: >I have certainly personally encountered plenty of situations where I >wasn't able to complete an egg-based install because some dependency >was broken (e.g. not available for the Python version I was using). That's odd -- setuptools-based installs should be able to find and install packages from source. I have noticed a recent phenomenon where new developers upload *only* an egg to PyPI, without the source, but that's usually short-lived until someone points it out to them. Do you happen to know what packages you had this problem with? >I'm okay if setuptools, once it's been installed, runs some setup code >that creates the .egg-info directory and whatever else. This means I'm >also okay with the bootstrap module finding and invoking that setup >code. But I'm *not* okay with building any kind of egg management into >the bootstrap module. The bootstrap module must be be neutral w.r.t. >the package management style. Ok, well then we'll have to invent a new kind of binary package, whose name isn't 'egg'. Supporting distutils source packages is almost certainly a non-starter, if you want to avoid bringing the rest of setuptools into play. The only way to correctly determine what a source package contains is to run its setup script... and running unboxed setup scripts isn't safe because there are people who hardcode paths (or more precisely, use bad ways of computing them) in their setup scripts. I'm not saying the tool needs to guard against *malicious* scripts, just badly-written ones. (Setuptools does this with its "sandboxing" module, when running source packages' setup scripts.) So, if source is out, then some binary format is needed, which means defining the conventions for said format... i.e. "eggs lite" or "egg substitutes". :) > > So, it might be simpler all around to just clear up the > > "controversy". To the best of my recollection, only MAL and MvL have > > ever objected on Python-Dev to the idea of supporting eggs. > >You can add my name to the list. I've heard plenty of people speak >highly of eggs, but I've *also* heard from plenty of people (besides >MAL and MvL) who have serious difficulties with the concept of eggs. I did say "on Python-Dev", and you implied that it was not controversial with you, except for the maintenance-related concerns. I'm not fighting about this, but I would rather you were straight-up with your objections rather than deferring it to a controversy that "might go away in a few years". That way, I could at least attempt to do something about the concerns. OTOH, if your objections were non-specific and likely to stay that way, then I could have at least not wasted your time with any of this.

1 0

Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
by Phillip J. Eby March 17, 2008

March 17, 2008

At 12:17 PM 3/17/2008 -0500, Guido van Rossum wrote: >There will be no egg support in the standard library. Are there any qualifications on that statement, or is this in the same category as "from __future__ import braces"?

2 1

Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
by Phillip J. Eby March 17, 2008

March 17, 2008

At 08:48 AM 3/17/2008 -0500, Guido van Rossum wrote: >On Sun, Mar 16, 2008 at 7:06 PM, Phillip J. Eby <pje(a)telecommunity.com> wrote: > > So, if the consensus is that it would be better to have a module that > > only does bootstrap installs of pure-Python eggs from PyPI, I'm > > totally fine with that. > >Let's just do this; it will avoid a protracted discussion of the >merits of eggs, pkg_resources, and setuptools. Well, it might be replaced by a protracted discussion of how the module should work and what its API should be, but perhaps that would be a better one to have. :) So, the original proposal (from the previous thread about this) was that the module be named easy_install, and that it simply downloads setuptools and delegates to the "real" easy_install. That way, people can simply use "python -m easy_install ...", without worrying about whether setuptools has been installed yet. IIRC, other package management tools such as zc.buildout and workingenv can then be installed using easy_install. Any objections? Should I revise the PEP?

3 4

Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
by Phillip J. Eby March 17, 2008

March 17, 2008

Quick summary of the below: I'm definitely fine with doing a simpler, pure-bootstrap module, if there's some consensus on what should go in it. I just wish we could've had this discussion last year, when OSAF was still able to fund the work... ;-) At 06:13 PM 3/16/2008 -0500, Guido van Rossum wrote: >Phillip asked me to give an opinion on his pkg_resources PEP. While >the PEP is short and sweet, the pkg_resources module itself is huge >(1800 non-blank lines; 16 classes plus 5 exceptions; it exports 67 >names in total according to __all__). And pkg_resources.txt is another >1700 lines of documentation. I find that hard to swallow. Is there >anyone besides Phillip who can claim he understands this module? Bob Ippolito actually wrote the very first version of pkg_resources. Others, such as Philip Jenvey of the Jython project, have provided patches. From previous discussions on the distutils-sig, I know that Jim Fulton has in-depth knowledge of both pkg_resources and easy_install. Of course, that's not the same as any of these guys volunteering to be maintainers. :) >If its inclusion is really meant just as a bootstrap to simplify >installing other package management solutions, as the PEP claims, I >would prefer to see something with a much smaller footprint. Actually, the PEP says: "pkg_resources is a module used to find and manage Python package/version dependencies and access bundled files and resources, including those inside of zipped .egg files. Currently, pkg_resources is only available through installing the entire setuptools distribution, but it does not depend on any other part of setuptools; in effect, it comprises the entire runtime support library for Python Eggs, and is independently useful." This kind of glosses over the part where this is also for runtime support of projects that use eggs. Which, these days, is, well, almost any large Python project, from Chandler to Enthought to Zope. > Surely >there is no need for example to have support for C extensions inside >zip files *as part of the bootstrap module*? It's a runtime; the PEP actually merely proposes that a further addition to be made to support bootstrapping, *also*. Otherwise, the PEP would be even shorter. :) The reason I proposed it this way was for simplicity -- and politics. Currently, people using setuptools in their setup.py have to include a similar bootstrap module to download setuptools if it's not available, and pkg_resources already has version checking logic and everything needed to find dependencies and download them. (Plus, I figured it'd be easier to just use what was already there and stable, rather than creating something different.) That was the simplicity part. The politics part was that: 1. I thought it would be less controversial to include the "runtime for eggs" than to include something that's just a bootstrapper for setuptools. However, MvL surprised me by actually being in *favor* of including a setuptools bootstrapper. 2. I thought that it would have broader acceptance if it was oriented towards bootstrapping *any* package, not just setuptools. So, if the consensus is that it would be better to have a module that only does bootstrap installs of pure-Python eggs from PyPI, I'm totally fine with that. >Unless I find someone besides Phillip who is interested in having this >included and is willing to help maintain it, I don't really think it >would be wise to accept this into the standard library. > >Phillip, in the PEP you mention that there are several other package >management tools that also like to use pkg_resources. Maybe you can >get some folks from those tools to speak up and explain what >pkg_resources means to them, and maybe even volunteer to co-own it >once it's in the standard library? The distutils-sig is the de facto place for discussions regarding those tools, so I've cc'd this there. Hopefully, one or more volunteers will step up if they want this.

3 2

PEP 365 (Adding the pkg_resources module)
by Guido van Rossum March 16, 2008

March 16, 2008

Phillip asked me to give an opinion on his pkg_resources PEP. While the PEP is short and sweet, the pkg_resources module itself is huge (1800 non-blank lines; 16 classes plus 5 exceptions; it exports 67 names in total according to __all__). And pkg_resources.txt is another 1700 lines of documentation. I find that hard to swallow. Is there anyone besides Phillip who can claim he understands this module? If its inclusion is really meant just as a bootstrap to simplify installing other package management solutions, as the PEP claims, I would prefer to see something with a much smaller footprint. Surely there is no need for example to have support for C extensions inside zip files *as part of the bootstrap module*? Unless I find someone besides Phillip who is interested in having this included and is willing to help maintain it, I don't really think it would be wise to accept this into the standard library. Phillip, in the PEP you mention that there are several other package management tools that also like to use pkg_resources. Maybe you can get some folks from those tools to speak up and explain what pkg_resources means to them, and maybe even volunteer to co-own it once it's in the standard library? -- --Guido van Rossum (home page: http://www.python.org/~guido/)

1 0

Fwd: 2.6 and 3.0 project management
by Guido van Rossum March 16, 2008

March 16, 2008

Sorry, forgot to CC this to the list. On Sun, Mar 16, 2008 at 9:31 AM, <skip(a)pobox.com> wrote: > > Guido> It has a much more detailed set of categories, organized as a > Guido> tree. Our project alone probably has 20-30 different bug > Guido> categories. New bugs in those categories are automatically CC'ed > Guido> to our group's mailing list (which isn't the same as > Guido> auto-assignment). > > Adding categories should be easy. Organized in trees? Not so sure. The tree is really useful because it means that end users can assign bugs to the top-level node for a project and the project members can move it to the correct subnode. This can even happen in two triage stages for large projects. > Guido> There are also more "bug states" you can use to track progress of > Guido> a bug through the system: unassigned, assigned, accepted (meaning > Guido> the assignee is actually working on it). (There are also a whole > Guido> bunch that I don't find so useful, and severam that roundup > Guido> already supports.) > > Again, I think this should be easy. It's also the least important one on my list. > Guido> But perhaps the best feature is "hot lists" -- arbitrary, > Guido> ordered, groupings of selected bugs. Each bug can be assigned to > Guido> as many hot lists as you want. Seeing the list of all bugs in a > Guido> particular hot list is one click away. We use this for overlaying > Guido> project management categories and priorities, such as "code", > Guido> "documentation", "configuration" as well as "next internal > Guido> release", "must have", "post launch" etc. > > A hot list sounds like a saved search, which Roundup already supports. It > also supports making these saved searches public. I suspect you could > define one or more saved public searches which correspond to desired hot > lists. Not quite. Items don't automatically end up on a hot list; they must explicitly be put on one. I'm not sure how you'd simulate this via saved searches. Maybe a combination of a custom keyword *and* a saved search would help. However this doesn't scale so well, because keywords show up in everybody's UI. Hot lists are only visible to users who care to subscribe to them. [Georg, in a later post] > Doesn't this match Roundup's keywords? See above answer. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

2 1