On December 28th, an unknown attacker used a previously unknown remote
code exploit on http://wiki.python.org/. The attacker was able to get
shell access as the "moin" user, but no other services were affected.
Some time later, the attacker deleted all files owned by the "moin"
user, including all instance data for both the Python and Jython
wikis. The attack also had full access to all MoinMoin user data on
all wikis. In light of this, the Python Software Foundation encourages
all wiki users to change their password on other sites if the same one
is in use elsewhere. We apologize for the inconvenience and will post
further news as we bring the new and improved wiki.python.org online.
If you have any questions about this incident please contact
jnoller(a)python.org. Thank you for your patience.
There's an interesting python "variant" (more of an overlay actually)
that is rather intriguing on github -- Vigil: a truly safe progamming
>From the readme:
"Infinitely more important than mere syntax and semantics are its
addition of supreme moral vigilance. This is similar to contracts, but
less legal and more medieval."
On Fri, Jan 4, 2013 at 2:38 PM, Dustin Mitchell <djmitche(a)gmail.com> wrote:
> As the maintainer of a pretty large, complex app written in Twisted, I think
> this is great. I look forward to a future of being able to select from a
> broad library of async tools, and being able to write tools that can be used
> outside of Twisted.
Thanks. Me too. :-)
> Buildbot began, lo these many years ago, doing a lot of things in memory on
> on local disk, neither of which require asynchronous IO. So a lot of API
> methods did not originally return Deferreds. Those methods are then used by
> other methods, many of which also do not return Deferreds. Now, we want to
> use a database backend, and parallelize some of the operations, meaning that
> the methods need to return a Deferred. Unfortunately, that requires a
> complete tree traversal of all of the methods and methods that call them,
> rewriting them to take and return Deferreds. There's no "halfway" solution.
> This is a little easier with generators (@inlineCallbacks), since the syntax
> doesn't change much, but it's a significant change to the API (in fact, this
> is a large part of the reason for the big rewrite for Buildbot-0.9.x).
> I bring all this up to say, this PEP will introduce a new "kind" of method
> signature into standard Python, one which the caller must know, and the use
> of which changes the signature of the caller. That can cause sweeping
> changes, and debugging those changes can be tricky.
Yes, and this is the biggest unproven point of the PEP. (The rest is
all backed by a decade or more of experience.)
> Two things can help:
> First, `yield from somemeth()` should work fine even if `somemeth` is not a
> coroutine function, and authors of async tools should be encouraged to use
> this form to assist future-compatibility. Second, `somemeth()` without a
> yield should fail loudly if `somemeth` is a coroutine function. Otherwise,
> the effects can be pretty confusing.
That would be nice. But the way yield from and generators work, that's
hard to accomplish without further changes to the language -- and I
don't want to have to change the language again (at least not
immediately -- maybe in a few releases, after we've learned what the
real issues are). The best I can do for the first requirement is to
define @coroutine in a way that if the decorated function isn't a
generator, it is wrapped in one. For the second requirement, if you
call somemeth() and ignore the result, nothing happens at all -- this
is indeed infuriating but I see no way to change this.(*) If you use
the result, well, Futures have different attributes than most other
objects so hopefully you'll get a loud AttributeError or TypeError
soon, but of course if you pass it into something else which uses it,
it may still be difficult to track. Hopefully these error messages
provide a hint:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'Future' object has no attribute 'foo'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'Future' object is not callable
(*) There's a heavy gun we might use, but I would make this optional,
as a heavy duty debugging mode only. @coroutine could wrap generators
in a lightweight object with a __del__ method and an __iter__ method.
If __del__ is called before __iter__ is ever called, it could raise an
exception or log a warning. But this probably adds too much overhead
to have it always enabled.
> In http://code.google.com/p/uthreads, I accomplished the latter by taking
> advantage of garbage collection: if the generator is garbage collected
> before it's begun, then it's probably not been yielded. This is a bit
> gross, but good enough as a debugging technique.
Eh, yeah, what I said. :-)
> On the topic of debugging, I also took pains to make sure that tracebacks
> looked reasonable, filtering out scheduler code. I haven't looked
> closely at Tulip to see if that's a problem. Most of the "noise" in the
> tracebacks came from the lack of 'yield from', so it may not be an issue at
One of the great advantages of using yield from is that the tracebacks
automatically look nice.
--Guido van Rossum (python.org/~guido)
As the maintainer of a pretty large, complex app written in Twisted, I
think this is great. I look forward to a future of being able to
select from a broad library of async tools, and being able to write
tools that can be used outside of Twisted.
Buildbot began, lo these many years ago, doing a lot of things in
memory on on local disk, neither of which require asynchronous IO. So
a lot of API methods did not originally return Deferreds. Those
methods are then used by other methods, many of which also do not
return Deferreds. Now, we want to use a database backend, and
parallelize some of the operations, meaning that the methods need to
return a Deferred. Unfortunately, that requires a complete tree
traversal of all of the methods and methods that call them, rewriting
them to take and return Deferreds. There's no "halfway" solution.
This is a little easier with generators (@inlineCallbacks), since the
syntax doesn't change much, but it's a significant change to the API
(in fact, this is a large part of the reason for the big rewrite for
I bring all this up to say, this PEP will introduce a new "kind" of
method signature into standard Python, one which the caller must know,
and the use of which changes the signature of the caller. That can
cause sweeping changes, and debugging those changes can be tricky.
Two things can help:
First, `yield from somemeth()` should work fine even if `somemeth` is
not a coroutine function, and authors of async tools should be
encouraged to use this form to assist future-compatibility. Second,
`somemeth()` without a yield should fail loudly if `somemeth` is a
coroutine function. Otherwise, the effects can be pretty confusing.
In http://code.google.com/p/uthreads, I accomplished the latter by
taking advantage of garbage collection: if the generator is garbage
collected before it's begun, then it's probably not been yielded.
This is a bit gross, but good enough as a debugging technique.
On the topic of debugging, I also took pains to make sure that
tracebacks looked reasonable, filtering out scheduler code. I
haven't looked closely at Tulip to see if that's a problem. Most of
the "noise" in the tracebacks came from the lack of 'yield from', so
it may not be an issue at all.
P.S. Apologies for the bad threading - I wasn't on the list when this
was last posted.
[Markus sent this to me off-list, but agreed to me responding on-list,
quoting his entire message.]
On Wed, Dec 26, 2012 at 2:38 PM, Markus <nepenthesdev(a)gmail.com> wrote:
I don't believe we've met before, have we? It would probably help if
you introduced yourself and your experience, since our past
experiences color our judgment.
> as I've been waiting for this to happen, I decided to speak up.
> While I really look forward to this, I disagree with the PEP.
Heh, we can't all agree on everything. :-)
> First shoot should be getting a well established event loop into python.
Perhaps. What is your definition of an event loop?
> libev is great, it takes care of operating system specialities, and
> only does a single job, providing an event loop.
It is also written for C, and I presume much of its API design was
influenced by the conventions and affordabilities of that language.
> This event loop can take care of timers, sockets and signals,
But sockets are not native on Windows, and I am making some effort
with PEP 3156 to efficiently support higher-level abstractions without
tying them to sockets. (The plan is to support IOCP on Windows. The
previous version of Tulip already had a branch that did support that,
as a demonstration of the power of this abstraction.)
> pyev, a
> great python wrapper for libev already provides this simple eventing
> facility in python.
But, being a libev wrapper, it is likely also strongly influenced by C.
> In case you embed python in a c program, the libev default loop of the
> python code and c code can even be shared, providing a great amount of
Only if the C code also uses libev, of course. But C programs may use
other event mechanisms -- e.g. AFAIK there are alternatives to libev
(during the early stages of Tulip development I chatted a bit with one
of the original authors of libevent, Niels Provos, and I believe
there's also something called libuv), and GUI frameworks (e.g. X, Qt,
Gtk, Wx) tend to have their own event loop.
PEP 3156 is designed to let alternative *implementations* of the same
*interface* be selected at run time. Hopefully it is possible to
provide a conforming implementation using libev -- then your goal
(smooth interoperability with C code using libev) is obtained.
It's possible that in order to do that the PEP 3156 interface may have
to be refactored into separate pieces. The Tulip implementation
already has separate "pollster" implementations (which concern
themselves *only* with polling for I/O using select, poll, or other
alternatives). It probably makes sense to factor the part that
implements transports out as well. However, the whole point of
including transports and protocols (and futures) in the PEP is that
some platforms may want to implement the same high-level API (e.g.
create a transport that connects to a certain host/port) using a
different approach altogether, e.g. on Windows the transport might not
even use sockets. OTOH on UNIX it may be possible to add file
descriptors representing pipes and pseudo-ttys.
> libev is great as it is small - it provides exactly what's required,
> and nothing beyond.
Depending on your requirements. :-)
> getaddrinfo/getnameinfo/create_transport are out of scope from a event
> loop point of view.
> This functionality already exists in python, it just does not use a
> event loop and is blocking, as every other io related api.
It wasn't random to add these. The "event loop" in PEP 3156 provides
abstractions that leave the platform free to implement connections
using the appropriate native constructs without letting those
constructs "leak" into the application -- after all, whether you're on
UNIX or on Windows, a TCP connection represents the same abstraction,
but the network stack may have a very different interface.
> I'd propose not to replicate the functionality in the event loop
> namespace, but to extend the existing implementations - by allowing to
> provide an event loop/callback/ctx as optional args which get used.
That's an interface choice that I would regret (I really don't like
writing code using callbacks).
(It would also be harder to implement initially as a 3rd party
framework. At the lowest level, no changes to Python itself are needed
-- it already supports non-blocking sockets, for example. But adding
optional callbacks to existing low-level APIs would require changes
throughout the stdlib.)
> If you specify something like pyev as PEP, you can still come up with
> another PEP which defines the semantics for upper layer protocols like
> udp/tcp on IPv4/6, which can be used to take care of dns and
I could split up the PEP, but that wouldn't really change anything,
since to me it is still a package deal. I am willing to put an effort
into specifying a low-level event loop because I know that I can still
write high-level code which is (mostly) free of callbacks, using
futures, tasks and the yield-from construct. And in order to do that I
need a minimum set of high-level abstractions such as getaddrinfo()
and transport creation (the exact names of the transport creation
methods are still under debate, as are the details of their
signatures, but the need for them is established without a doubt in my
I note that the stdlib socket module has roughly the same set of
abstractions bundled together:
- socket objects
- getaddrinfo(), getnameinfo()
- the makefile() methods on socket objects, which create buffered streams
PEP 3156 offers alternatives for all of these, using higher-level
abstractions that have been developed and proven in practice by
Twisted, *and* offers a path to interop to frameworks that previously
couldn't very well interoperate -- Twisted, Tornado, and others have
traditionally been pretty segregated, but with PEP 3156 they can
interoperate both through the event loop and through Futures (which
are friendly both to a callback style and to yield-from).
> Anyway, I really hope you'll have a look on libev and pyev, both is
> great and well tested software and may give you an idea what people
> who dedicate themselves to event loops came up with already in terms
> of names, subclassing, requirements, guarantees and workarounds for
> platform specific failures (kqueue, epoll ...).
I will certainly have a look! I am not so concerned about naming (it
seems inevitable that everyone uses somewhat different terminology
anyway, and it is probably better not to reuse terms when the meaning
is different), but I do like to look at guarantees (or the absence
thereof!) and best practices for dealing with the differences between
> All together, I'd limit the scope of the PEP to the API of the event
> loop, just focussing on io/timers/signals and propose to extend
> existing API to be usable with an event loop, instead of replicating
You haven't convinced me about this. However, you can help me by
comparing the event loop part of PEP 3156 (ignoring anything that
returns or takes a Future) to libev and pointing out things (either
specific APIs or certain guarantees or requirements) that would be
hard to implement using libev, as well as useful features in libev
that you think every event loop should have.
> For naming I'd prefer 'watcher' over 'Handler'.
Hm, 'watcher' to me sounds more active than the behavior I have in
mind for this class. It is just a reification of a specific function
and some arguments to pass to it, with the ability to cancel the call
Thanks for writing!
--Guido van Rossum (python.org/~guido)
I propose to add new standard collection types: IdentityDict and
IdentitySet. They are almost same as ordinal dict and set, but uses
identity check instead of equality check (and id() or hash(id()) as a
hash). They will be useful for pickling, for implementing __sizeof__()
for compound types, and for other graph algorithms.
Of course, they can be implemented using ordinal dicts:
IdentityDict: key -> value as a dict: id(key) -> (key, value)
IdentitySet as a dict: id(value) -> value
However implementing them directly in the core has advantages, it
consumes less memory and time, and more comfortable for use from C.
IdentityDict and IdentitySet implementations will share almost all code
with implementations of ordinal dict and set, only lookup function and
metainformation will be different. However dict and set already use a
lookup function overloading.
середа 02 січень 2013 21:43:47 Eli Bendersky ви написали:
> I agree that the data structures may be useful, but is there no way to some
> allow the customization of existing data structures instead, without losing
> performance? It's a shame to have another kind of dict just for this
What interface for the customization is possible? Obviously, a dict
constructor can't have a special keyword argument.
Sometimes, I have the flexibility to reduce the memory used by my
program (e.g., by destroying large cached objects, etc.). It would be
great if I could ask Python interpreter to notify me when memory is
running out, so I can take such actions.
Of course, it's nearly impossible for Python to know in advance if the
OS would run out of memory with the next malloc call. Furthermore,
Python shouldn't guess which memory (physical, virtual, etc.) is
relevant in the particular situation (for instance, in my case, I only
care about physical memory, since swapping to disk makes my
application as good as frozen). So the problem as stated above is
But let's say I am willing to do some work to estimate the maximum
amount of memory my application can be allowed to use. If I provide
that number to Python interpreter, it may be possible for it to notify
me when the next memory allocation would exceed this limit by calling
a function I provide it (hopefully passing as arguments the amount of
memory being requested, as well as the amount currently in use). My
callback function could then destroy some objects, and return True to
indicate that some objects were destroyed. At that point, the
intepreter could run its standard garbage collection routines to
release the memory that corresponded to those objects - before
proceeding with whatever it was trying to do originally. (If I
returned False, or if I didn't provide a callback function at all, the
interpreter would simply behave as it does today.) Any memory
allocations that happen while the callback function itself is
executing, would not trigger further calls to it. The whole mechanism
would be disabled for the rest of the session if the memory freed by
the callback function was insufficient to prevent going over the
Would this be worth considering for a future language extension? How
hard would it be to implement?
I have seen many people new to Python stumbling while using the Python
docs due to the order of the search results.
For example, if somebody new to python searches for `tuple`, the
actual section about `tuple` comes in place 39. What is more confusing
for people starting with the language is that all the C functions come
first. I have seen people clicking in PyTupleObject just to be totally
Maybe `tuple` is a silly example. But if somebody wants to know how
does `open` behaves and which arguments it takes, the result comes in
position 16. `property` does not appear in the list at all (but
built-in appears in position 31). This is true for most builtins.
Experienced people will have no trouble navigating through these
results, but new users do. It is not terrible and at the end they get
it, but I think it would be nice to change it to more (new) user
So my suggestion is to put the builtins first, the rest of the
standard lib later including HowTos, FAQ, etc and finally the
c-modules. Additionally, a section with a title matching exactly the
search query should come first. (I am not sure if the last suggestion
belongs in python-ideas or in
the sphinx mailing list, please advice)