6:22 EDT, tracker down and has been for a couple of minutes.
python.org and docs.python.org are fine.
Does the daemon program that now checks on Pypi also check the tracker?
Is there a particular place to report tracker down?
Or should I just assume someone else will notice and do something?
Terry Jan Reedy
At 05:08 PM 8/2/2010 +0200, Ãric Araujo wrote:
>I wonder if functions in pkgutil or importlib could allow one to
>iterate over the plugins (i.e. submodules and subpackages of the
>namespace package) without actually loading then.
See pkgutil.walk_packages(), available since 2.5.
It has to load __init__.py files, especially because of namespace
packages, but it doesn't load any non-package modules.
That being said, using namespace packages for plugins kind of defeats
the true purpose of namespace packages, which is to give developers
private package namespaces they can use across multiple projects,
like zope.*, peak.*, twisted.*, etc., thereby avoiding naming
conflicts in the root package namespace.
Granted, you can always re-nest namespaces and do something like
someproject.plugins.mynamehere.myplugin, but with entry points you
can just register something in mynamehere.mysomeprojectplugin, and
flat is better than nested. ;-) (Plus, you can include information
about the individual plugins/features residing in that module in the
metadata, and avoid importing until/unless you need that feature.)
At 01:10 PM 8/2/2010 +0200, Tarek Ziadé wrote:
>I don't have a specific example in mind, and I must admit that if an
>application does the right thing
>(provide the right configuration file), this activate feature is not
>useful at all. So it seems to be a bad idea.
Well, it's not a *bad* idea as such; actually, having conventions for
such configuration, and libraries that help to implement the
convention are a *good* idea, and I support it. I just don't think
it makes much sense to *impose* the convention on the app developers;
there are, after all, use cases that don't need the extra configuration.
Setuptools was mainly designed to support the "application plugin
directory" model for invasive sorts of plugins, and the "global
plugin availability" model for the kind of plugins that a user has to
explicitly select (e.g. file type converters, special distutils
commands, etc.). However, there are definitely use cases for
"user-configured plugins", and the apps that do it generally use some
sort of configuration file to identify which entry points they'll actually use.
>IOW, have entry points like setuptools provides, but in a metadata
>field instead of a entry_points.txt file.
May I suggest, then, that we keep entry_points.txt, but simply
provide a summary in PKG-INFO? (i.e., list the groups and names provided)
This would still make it easy for human browsing/discovery of entry
points on PyPI, but it would allow easy forward/backward
compatibility between setuptools and distutils2, while also providing
faster lookup of entry points (because you can skip distributions
that don't have an entry points file, vs. having to parse *every*
Or to put it another way, when I implement PEP 376 support in
setuptools 0.7, I'll only have to change the name of the .egg-info
directory and copy the entry point summary into PKG-INFO. And, even
more to the point, people who define entry points with distutils2
will then be able to have them work with setuptools-based projects,
and vice versa, helping to smooth the transition.
At 01:53 PM 8/2/2010 +0000, exarkun(a)twistedmatrix.com wrote:
>On 01:27 pm, mal(a)egenix.com wrote:
>>>This is also roughly how Twisted's plugin system works. One drawback,
>>>though, is that it means potentially executing a large amount of Python
>>>in order to load plugins. This can build up to a significant
>>>performance issue as more and more plugins are installed.
>>I'd say that it's up to the application to deal with this problem.
>>An application which requires lots and lots of plugins could
>>define a registration protocol that does not require loading
>>all plugins at scanning time.
Just for the record, solving this problem is precisely what entry
points are for: they provide a discovery mechanism that doesn't
require importing anything until you actually need it.
>It's not fixable at the application level, at least in Twisted's
>plugin system. It sounds like Zope's system has the same problem,
>but all I know of that system is what you wrote above.
I don't know about Zope in general, but there are certainly Zope
corp. projects that use entry points instead of namespaces (buildout,
for one), and I believe that there's been a long time push to move
third-party code out of the common namespace package. i.e., AFAIK,
Zope 3 doesn't use package namespaces as a primary method of extension.
> The cost increases with the number of plugins installed on the
> system, not the number of plugins the application wants to load.
Pretty much any plugin discovery system is going to scale that way,
but entry points only require file reads rather than imports, and
have a shared cache for all code in use by the application. So if,
say, Twisted uses entry points and an application running on Twisted
also uses entry points, the loading cost is only paid once for both
sets of entry points inspected.
Over in #python-dev, Georg reminded us of a change to the python-checkins
mailing list that was discussed a few weeks ago:
Despite the mild preference of redirecting python-checkins to
python-committers, I noticed that the list was already set up to redirect to
python-dev (but the Reply-To munging was disabled). I've now re-enabled
redirects of python-checkins to python-dev. I think it's better to default to
more openness and it's not really that much traffic anyway.
If it gets obnoxious we can narrow things, but let's see how it goes.
Don't forget that I am planning to cut Python 2.6.6 rc 1 later today (probably
starting at around 2200 UTC). We have a number of release blockers currently
Feel free to ping me on irc (freenode @ #python-dev) or follow up here if you
have any input on these.
I'll send the usually notes out to the committers list when I'm ready to
freeze the tree.
to warm up for tomorrow's 3.2alpha1 release, I did a mini-sprint on
pdb issues today. I'm pleased to report that 14 issues could be closed,
and pdb got a range of small new features, such as commands on the
command line, "until <lineno>" or "longlist" showing all the code for the
current function (the latter courtesy of Antonio Cuni's pdb++).
One issue that's not yet closed is #7245, which adds a (very nice IMO)
feature: when you press Ctrl-C while the program being debugged runs,
you will not get a traceback but execution is suspended, and you can
debug from the current point of execution -- just like in gdb.
However, there were apparently issues with some of the buildbots when
the patch was applied for a short time. I also don't know how and if
it works on Windows, so I'd need some helpful people testing it.
Another question is about a feature of pdb++ that I personally would
like, but imagine would make others unhappy: one-letter abbreviations
of commands such as c(ontinue) or l(ist) are also often-used variable
names, so they are frequently typed without the required "!" or "print"
that would distinguish them from the command, and the command is
actually executed. The feature in question would default to printing
the variable in cases where one exists -- handy enough or too
Also, are there any other features you would like to see? One feature
of pdb++ that is general enough and has no dependencies would be watch
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.
At 09:24 PM 8/1/2010 -0700, Guido van Rossum wrote:
>I don't understand all the details and corner
>cases (e.g. the concatenation of stacks
It's just to ensure that you never have From's iterating over other
From's, vs. just iterating whatever's at the top of the stack.
>which seems to have to do with the special-casing of From objects in __new__)
It isn't connected, actually except that it's another place where I'm
keeping From's flat, instead of nested. (I hear that flat is better. ;-) )
>I am curious whether, if you need a trampoline for async I/O anyway,
>there would be a swaying argument for integrating this functionality
>into the general trampoline (as in the PEP 342 example),
Originally, that was why I wasn't very enthusiastic about PEP 380; it
didn't seem to me to be adding any new value over what you could do
with existing, widely-used libraries. (Twisted's had its own *and*
multiple third-party From-ish libraries supporting it for many years now.)
After I wrote From(), however (which was originally intended to show
why I thought 380 was unnecessary), I realized that having One
Obvious Way to implement generator-based pseudothreads independent of
an event loop, is actually useful precisely *because* it separates
the pseudothreadedness from what you're using the pseudothreadedness for.
Essentially, the PEP 380-ish bit is the hardest part of writing an
actual pseudothread implementation; connecting that implementation to
an I/O framework is actually the relatively simple part. You just
write code that steps into the generator, and uses the yielded object
to initiate an I/O operation and register a callback. (If you're
using Twisted or something else that has "promise"-like deferred
results, it's *really* easy, because you only have a couple of types
of yielded objects to deal with, and a uniform callback signature.)
Indeed, if you're using an existing async I/O framework, you don't
even really *have* a "trampoline" as such -- you just have a bit of
code that registers callbacks to itself, and the app's main event
loop just calls back to that wrapper when the I/O is done.
In effect, an I/O framework integration would just give you a single
API like, "run(From(geniter))", which then performs one iteration,
and then registers whatever callback it's told to by the yield, and
the callback it registers would actually be a reinvocation of run()
on the same From instance when the I/O is ready, but with a value to
pass back into the send(), or an error to throw(). So, the I/O
framework's event loop is half of the "trampoline", and the wrapper
that sends or throws, then registers an I/O callback, is the other half.
def run(coroutine, value=None, exc_info=()):
action = coroutine.throw(*exc_info)
action = coroutine.send(value)
Where 'action' is some I/O command object, and registerCallback()
will call its argument back with a value or exc_info, after the I/O is done.
Of course, a real framework integration might actually dispatch on
type here rather than using special command objects like this, and
there might be more glue code to deal with exceptions, but really,
the heart of the thing is just going to look like that. (I just
wrote it that way to show the basic structure.)
Really, it's just a few functions, maybe a utility routine or two,
and maybe a big if-then or dictionary dispatch on types if you just
want to be able to 'yield' existing I/O objects provided by the frameworks.
IOW, it's a *lot* simpler than actually rolling your own I/O or GUI
framework like Twisted or Eventlet or wxPython or tk or some other such thing.
>But it seems a bit of a waste to have two different trampolines,
>especially since the trampoline itself is so hard to understand
>(speaking for myself here :-). ISTM that the single combined
>trampoline is easier to understand than the From class.
Well, the PEP 342 example was made to look simple, because it doesn't
have to actually DO anything (like I/O!) To work for real, it'd need
some pluggability, and some things to help it interoperate with
different GUI and I/O frameworks and event loops. (Using your own
event loop "for real" isn't very useful in a lot of non-trivial applications.)
Heck, after writing From(), it gave me an idea that I could just
write a trampoline that *could* integrate with other event loops,
with an idea to have it be a general-purpose companion to From.
But, after several wasted hours, I realized that yes, it *could* be
written (I still have the draft), but it was mostly just something
that would save a little boilerplate in bolting From()'s onto an
existing async I/O framework, and not really anything to write home about.
So, I guess what I'm saying is, the benefit of separating the
trampoline from control flow, is that people can then use them with
their favorite event loop or framework, instead of the stdlib trying
to compete with the experts on a slower release cycle.
There's additional benefit here in that 1) getting pseudothread
implementations correct can be difficult, but once done, they're
extremely stable, and 2) having a blessed syntax for identifying
pseudothreaded calls and returns is a boon to competition in I/O frameworks.
So, instead of everybody having their own versions of From (and I've
written more than a couple in my time), there's One Obvious Way To Do
It. All that will differ between I/O libraries are the actual API
calls for I/O and scheduling and starting up pseudothreads, so there
won't be as big of a switching barrier between frameworks.
At 08:49 AM 8/1/2010 -0400, Kevin Jacobs <jacobs(a)bioinformed.com> wrote:
>On Sun, Aug 1, 2010 at 3:54 AM, Greg Ewing
>I have updated my prototype yield-from implementation
>to work with Python 3.1.2.
>My work is primarily on the management and analysis of huge genomics
>datasets. I use Python generators extensively and intensively to
>perform efficient computations and transformations on these datasets
>that avoid the need to materialize them in main memory to the extent
>possible. I've spent a great deal of effort working around the
>lack of an efficient "yield from" construct and would be very
>excited to see this feature added.
Just so you know, you don't need to wait for this to be added to
Python in order to have such a construct; it just won't have the
extra syntax sugar. See the sample code I posted here using a
"@From.container" decorator, and a "yield From()" call:
This code effectively reduces your generator nesting depth to a
constant, no matter how deeply you nest sub-generator
invocations. It's not as efficient as the equivalent C
implementation, but if you're actually being affected by nesting
overhead now, it will nonetheless provide you with some immediate
relief, if you backport it to 2.x code. (It's not very 3.x-ish as it
I have updated my prototype yield-from implementation
to work with Python 3.1.2.
I've also fixed a small bug that was affecting one of
the corner cases concerning exceptions thrown into
Interested parties can obtain it here: