Mailman 3 August 2010 - Python-Dev

Is it intentional that "sys.__debug__ = 1" is illegal in Python 2.7?
by Barry Warsaw Aug. 3, 2010

Aug. 3, 2010

In working on making Python 2.7 available in Debian and Ubuntu, we ran into two packages that fail to byte compile against Python 2.7, where they are fine in Python 2.6. The Debian bug tracker issues are: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=590821 http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=590822 Specifically, what fails now is assignment to the __debug__ attribute of a module, e.g. sys -- *not* builtin __debug__. % python2.6 Python 2.6.5 (r265:79063, Apr 16 2010, 13:57:41) [GCC 4.4.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> __debug__ = 1 File "<stdin>", line 1 SyntaxError: can not assign to __debug__ >>> import sys >>> sys.__debug__ = 1 >>> % python2.7 Python 2.7.0+ (release27-maint:83294, Jul 30 2010, 15:49:51) [GCC 4.4.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> __debug__ = 1 File "<stdin>", line 1 SyntaxError: cannot assign to __debug__ >>> import sys >>> sys.__debug__ = 1 File "<stdin>", line 1 SyntaxError: cannot assign to __debug__ >>> I think it's actually somewhat questionable that this is now illegal. It certainly breaks at least two packages. More disturbing though is that I cannot find mention of this change in Misc/NEWS or in the tracker, so I don't actually know if this was intentional or not. It looks like Benjamin's change in r67171 was the relevant diff. Thoughts? Either way I will file a bug. IOW, if this change should *not* have been made, I think it should be reverted for Python 2.7.1 and I can patch our version of Python in the meantime. If this change is intentional, I'll file a bug against (or just fix) Misc/NEWS and the What's New. Cheers, -Barry

9 14

Tracker down
by Terry Reedy Aug. 3, 2010

Aug. 3, 2010

6:22 EDT, tracker down and has been for a couple of minutes. python.org and docs.python.org are fine. Does the daemon program that now checks on Pypi also check the tracker? Is there a particular place to report tracker down? Or should I just assume someone else will notice and do something? -- Terry Jan Reedy

2 1

Re: [Python-Dev] PEP 376 proposed changes for basic plugins support
by P.J. Eby Aug. 2, 2010

Aug. 2, 2010

At 05:08 PM 8/2/2010 +0200, Ãric Araujo wrote: >I wonder if functions in pkgutil or importlib could allow one to >iterate over the plugins (i.e. submodules and subpackages of the >namespace package) without actually loading then. See pkgutil.walk_packages(), available since 2.5. It has to load __init__.py files, especially because of namespace packages, but it doesn't load any non-package modules. That being said, using namespace packages for plugins kind of defeats the true purpose of namespace packages, which is to give developers private package namespaces they can use across multiple projects, like zope.*, peak.*, twisted.*, etc., thereby avoiding naming conflicts in the root package namespace. Granted, you can always re-nest namespaces and do something like someproject.plugins.mynamehere.myplugin, but with entry points you can just register something in mynamehere.mysomeprojectplugin, and flat is better than nested. ;-) (Plus, you can include information about the individual plugins/features residing in that module in the metadata, and avoid importing until/unless you need that feature.)

1 0

Re: [Python-Dev] PEP 376 proposed changes for basic plugins support
by P.J. Eby Aug. 2, 2010

Aug. 2, 2010

At 01:10 PM 8/2/2010 +0200, Tarek Ziadé wrote: >I don't have a specific example in mind, and I must admit that if an >application does the right thing >(provide the right configuration file), this activate feature is not >useful at all. So it seems to be a bad idea. Well, it's not a *bad* idea as such; actually, having conventions for such configuration, and libraries that help to implement the convention are a *good* idea, and I support it. I just don't think it makes much sense to *impose* the convention on the app developers; there are, after all, use cases that don't need the extra configuration. Setuptools was mainly designed to support the "application plugin directory" model for invasive sorts of plugins, and the "global plugin availability" model for the kind of plugins that a user has to explicitly select (e.g. file type converters, special distutils commands, etc.). However, there are definitely use cases for "user-configured plugins", and the apps that do it generally use some sort of configuration file to identify which entry points they'll actually use. >IOW, have entry points like setuptools provides, but in a metadata >field instead of a entry_points.txt file. May I suggest, then, that we keep entry_points.txt, but simply provide a summary in PKG-INFO? (i.e., list the groups and names provided) This would still make it easy for human browsing/discovery of entry points on PyPI, but it would allow easy forward/backward compatibility between setuptools and distutils2, while also providing faster lookup of entry points (because you can skip distributions that don't have an entry points file, vs. having to parse *every* PKG-INFO file). Or to put it another way, when I implement PEP 376 support in setuptools 0.7, I'll only have to change the name of the .egg-info directory and copy the entry point summary into PKG-INFO. And, even more to the point, people who define entry points with distutils2 will then be able to have them work with setuptools-based projects, and vice versa, helping to smooth the transition.

1 0

Re: [Python-Dev] PEP 376 proposed changes for basic plugins support
by P.J. Eby Aug. 2, 2010

Aug. 2, 2010

At 01:53 PM 8/2/2010 +0000, exarkun(a)twistedmatrix.com wrote: >On 01:27 pm, mal(a)egenix.com wrote: >>exarkun(a)twistedmatrix.com wrote: >>>This is also roughly how Twisted's plugin system works. One drawback, >>>though, is that it means potentially executing a large amount of Python >>>in order to load plugins. This can build up to a significant >>>performance issue as more and more plugins are installed. >> >>I'd say that it's up to the application to deal with this problem. >> >>An application which requires lots and lots of plugins could >>define a registration protocol that does not require loading >>all plugins at scanning time. Just for the record, solving this problem is precisely what entry points are for: they provide a discovery mechanism that doesn't require importing anything until you actually need it. >It's not fixable at the application level, at least in Twisted's >plugin system. It sounds like Zope's system has the same problem, >but all I know of that system is what you wrote above. I don't know about Zope in general, but there are certainly Zope corp. projects that use entry points instead of namespaces (buildout, for one), and I believe that there's been a long time push to move third-party code out of the common namespace package. i.e., AFAIK, Zope 3 doesn't use package namespaces as a primary method of extension. > The cost increases with the number of plugins installed on the > system, not the number of plugins the application wants to load. Pretty much any plugin discovery system is going to scale that way, but entry points only require file reads rather than imports, and have a shared cache for all code in use by the application. So if, say, Twisted uses entry points and an application running on Twisted also uses entry points, the loading cost is only paid once for both sets of entry points inspected.

1 0

python-checkins Reply-To now set for python-dev
by Barry Warsaw Aug. 2, 2010

Aug. 2, 2010

Over in #python-dev, Georg reminded us of a change to the python-checkins mailing list that was discussed a few weeks ago: http://mail.python.org/pipermail/python-dev/2010-July/101853.html Despite the mild preference of redirecting python-checkins to python-committers, I noticed that the list was already set up to redirect to python-dev (but the Reply-To munging was disabled). I've now re-enabled redirects of python-checkins to python-dev. I think it's better to default to more openness and it's not really that much traffic anyway. If it gets obnoxious we can narrow things, but let's see how it goes. -Barry

1 0

Python 2.6.6 rc 1 planned for today
by Barry Warsaw Aug. 2, 2010

Aug. 2, 2010

Hi folks, Don't forget that I am planning to cut Python 2.6.6 rc 1 later today (probably starting at around 2200 UTC). We have a number of release blockers currently reported: http://bugs.python.org/issue?@columns=title,id,activity,versions,assignee&@… Feel free to ping me on irc (freenode @ #python-dev) or follow up here if you have any input on these. I'll send the usually notes out to the committers list when I'm ready to freeze the tree. -Barry

1 0

pdb mini-sprint report and questions
by Georg Brandl Aug. 2, 2010

Aug. 2, 2010

Hi, to warm up for tomorrow's 3.2alpha1 release, I did a mini-sprint on pdb issues today. I'm pleased to report that 14 issues could be closed, and pdb got a range of small new features, such as commands on the command line, "until <lineno>" or "longlist" showing all the code for the current function (the latter courtesy of Antonio Cuni's pdb++). One issue that's not yet closed is #7245, which adds a (very nice IMO) feature: when you press Ctrl-C while the program being debugged runs, you will not get a traceback but execution is suspended, and you can debug from the current point of execution -- just like in gdb. However, there were apparently issues with some of the buildbots when the patch was applied for a short time. I also don't know how and if it works on Windows, so I'd need some helpful people testing it. Another question is about a feature of pdb++ that I personally would like, but imagine would make others unhappy: one-letter abbreviations of commands such as c(ontinue) or l(ist) are also often-used variable names, so they are frequently typed without the required "!" or "print" that would distinguish them from the command, and the command is actually executed. The feature in question would default to printing the variable in cases where one exists -- handy enough or too inconsistent? Also, are there any other features you would like to see? One feature of pdb++ that is general enough and has no dependencies would be watch expressions... Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out.

5 6

Re: [Python-Dev] Yield-From Implementation Updated for Python 3
by P.J. Eby Aug. 2, 2010

Aug. 2, 2010

At 09:24 PM 8/1/2010 -0700, Guido van Rossum wrote: >I don't understand all the details and corner >cases (e.g. the concatenation of stacks It's just to ensure that you never have From's iterating over other From's, vs. just iterating whatever's at the top of the stack. >which seems to have to do with the special-casing of From objects in __new__) It isn't connected, actually except that it's another place where I'm keeping From's flat, instead of nested. (I hear that flat is better. ;-) ) >I am curious whether, if you need a trampoline for async I/O anyway, >there would be a swaying argument for integrating this functionality >into the general trampoline (as in the PEP 342 example), Originally, that was why I wasn't very enthusiastic about PEP 380; it didn't seem to me to be adding any new value over what you could do with existing, widely-used libraries. (Twisted's had its own *and* multiple third-party From-ish libraries supporting it for many years now.) After I wrote From(), however (which was originally intended to show why I thought 380 was unnecessary), I realized that having One Obvious Way to implement generator-based pseudothreads independent of an event loop, is actually useful precisely *because* it separates the pseudothreadedness from what you're using the pseudothreadedness for. Essentially, the PEP 380-ish bit is the hardest part of writing an actual pseudothread implementation; connecting that implementation to an I/O framework is actually the relatively simple part. You just write code that steps into the generator, and uses the yielded object to initiate an I/O operation and register a callback. (If you're using Twisted or something else that has "promise"-like deferred results, it's *really* easy, because you only have a couple of types of yielded objects to deal with, and a uniform callback signature.) Indeed, if you're using an existing async I/O framework, you don't even really *have* a "trampoline" as such -- you just have a bit of code that registers callbacks to itself, and the app's main event loop just calls back to that wrapper when the I/O is done. In effect, an I/O framework integration would just give you a single API like, "run(From(geniter))", which then performs one iteration, and then registers whatever callback it's told to by the yield, and the callback it registers would actually be a reinvocation of run() on the same From instance when the I/O is ready, but with a value to pass back into the send(), or an error to throw(). So, the I/O framework's event loop is half of the "trampoline", and the wrapper that sends or throws, then registers an I/O callback, is the other half. Something like: def run(coroutine, value=None, exc_info=()): if exc_info: action = coroutine.throw(*exc_info) else: action = coroutine.send(value) action.registerCallback(partial(run, coroutine)) Where 'action' is some I/O command object, and registerCallback() will call its argument back with a value or exc_info, after the I/O is done. Of course, a real framework integration might actually dispatch on type here rather than using special command objects like this, and there might be more glue code to deal with exceptions, but really, the heart of the thing is just going to look like that. (I just wrote it that way to show the basic structure.) Really, it's just a few functions, maybe a utility routine or two, and maybe a big if-then or dictionary dispatch on types if you just want to be able to 'yield' existing I/O objects provided by the frameworks. IOW, it's a *lot* simpler than actually rolling your own I/O or GUI framework like Twisted or Eventlet or wxPython or tk or some other such thing. >But it seems a bit of a waste to have two different trampolines, >especially since the trampoline itself is so hard to understand >(speaking for myself here :-). ISTM that the single combined >trampoline is easier to understand than the From class. Well, the PEP 342 example was made to look simple, because it doesn't have to actually DO anything (like I/O!) To work for real, it'd need some pluggability, and some things to help it interoperate with different GUI and I/O frameworks and event loops. (Using your own event loop "for real" isn't very useful in a lot of non-trivial applications.) Heck, after writing From(), it gave me an idea that I could just write a trampoline that *could* integrate with other event loops, with an idea to have it be a general-purpose companion to From. But, after several wasted hours, I realized that yes, it *could* be written (I still have the draft), but it was mostly just something that would save a little boilerplate in bolting From()'s onto an existing async I/O framework, and not really anything to write home about. So, I guess what I'm saying is, the benefit of separating the trampoline from control flow, is that people can then use them with their favorite event loop or framework, instead of the stdlib trying to compete with the experts on a slower release cycle. There's additional benefit here in that 1) getting pseudothread implementations correct can be difficult, but once done, they're extremely stable, and 2) having a blessed syntax for identifying pseudothreaded calls and returns is a boon to competition in I/O frameworks. So, instead of everybody having their own versions of From (and I've written more than a couple in my time), there's One Obvious Way To Do It. All that will differ between I/O libraries are the actual API calls for I/O and scheduling and starting up pseudothreads, so there won't be as big of a switching barrier between frameworks.

1 0

Re: [Python-Dev] Yield-From Implementation Updated for Python 3
by P.J. Eby Aug. 2, 2010

Aug. 2, 2010

At 08:49 AM 8/1/2010 -0400, Kevin Jacobs <jacobs(a)bioinformed.com> wrote: >On Sun, Aug 1, 2010 at 3:54 AM, Greg Ewing ><<mailto:greg.ewing@canterbury.ac.nz>greg.ewing(a)canterbury.ac.nz> wrote: >I have updated my prototype yield-from implementation >to work with Python 3.1.2. > > >My work is primarily on the management and analysis of huge genomics >datasets. I use Python generators extensively and intensively to >perform efficient computations and transformations on these datasets >that avoid the need to materialize them in main memory to the extent >possible. I've spent a great deal of effort working around the >lack of an efficient "yield from" construct and would be very >excited to see this feature added. Just so you know, you don't need to wait for this to be added to Python in order to have such a construct; it just won't have the extra syntax sugar. See the sample code I posted here using a "@From.container" decorator, and a "yield From()" call: http://mail.python.org/pipermail/python-dev/2010-July/102320.html This code effectively reduces your generator nesting depth to a constant, no matter how deeply you nest sub-generator invocations. It's not as efficient as the equivalent C implementation, but if you're actually being affected by nesting overhead now, it will nonetheless provide you with some immediate relief, if you backport it to 2.x code. (It's not very 3.x-ish as it sits, really.)

2 1