[Python-Dev] Fwd: [python-tulip] Need help to finish asyncio documentation

Guido van Rossum guido at python.org
Mon Feb 10 22:21:44 CET 2014


On Mon, Feb 10, 2014 at 12:47 PM, Russell E. Owen <rowen at uw.edu> wrote:

> In article
> <CAP7+vJKmBPYU_E+4TyC3X6oFC_ydmd9K9PXk8atoLL6oJ_8FeA at mail.gmail.com>,
>  Guido van Rossum <guido at python.org> wrote:
>
> > We could really use more help reviewing and finishing asyncio's docs!
> ...
> > http://docs.python.org/dev/library/asyncio.html
>
> I think the documentation desperately needs an overview. I find it very
> confusing without one. I guess the links to PEPs are intended to be
> that, but it would be much more helpful to include the overview in the
> documentation.
>

Thanks very much -- this is the kind of thing that's hard to notice as the
author (of either the code or the docs :-).


> I think a bit of re-ordering of topics could also help:
> - One of the first things presented is a discussion of event loop
> policy, even though most users will never need to change the policy.
> Couldn't this go at the end?
>

Indeed.


> - It tells how to create and set event loops, with no hint as to why one
> would want to do that. Is it common to have multiple event loops?


Shouldn't be that common (but more common than creating or even using the
policy).


> A related oddity: what is the point of
> BaseEventLoop.run_until_complete(future)? It sounds interesting but
> obscure (unless it applies to one of many event loops).
>

That's actually pretty common in a main() program. See several examples in
the Tulip repo, e.g.
http://code.google.com/p/tulip/source/browse/examples/crawl.py#851


> In other words: if it was possible to show basic usage of event loops
> early, then discuss how to do the fancy stuff (new event loops, new even
> loop policies later) that might help.
>
> BaseEventLoop.stop() says:
> "Every callback scheduled before stop() is called will run."
> but it does not say WHEN it will run -- immediately due to the stop, or
> normally as if stop was never called?
>

Yeah, this is confusing -- it only applies to callbacks scheduled with
call_soon(), not for those scheduled with call_later() or call_at() (and
for those scheduled with call_soon_threadsafe() all bets are off, of
course).

it may not even be such a good idea to make this promise (though it's also
in PEP 3156) -- the promise constrains the behavior of stop(). The
implementation is kind of cute though: call_soon() adds things to a FIFO
and the event loop picks items to run off this FIFO -- and stop() just adds
a sentinel item to the FIFO. The FIFO behavior implements the promise. Also
notice that if a callback schedules a further callback using call_soon(),
the latter will end up in the FIFO *after* the sentinel item, so it won't
run.

It's also important to understand that a stopped loop can always be resumed
with no ill effects -- the sentinel having been popped off the FIFO, the
loop will continue to run until it is stopped again. (But there's also a
bit if an issue here if you stop it twice in a row -- the second stop()
call will remain in the FIFO.)

Finally, scheduling something with call_later(0, <cb>) is *not* the same as
call_soon(<cb>). call_later() *always* creates a scheduler item which will
require a separate "tick" of the loop before it gets moved to the
above-mentioned FIFO (a.k.a. the ready queue).

Perhaps the following is helpful (though it gives more implementation
detail than we should probably put in the docs). The event loop repeats the
following steps forever:

1. Wait for I/O ready with a deadline equal to the earliest scheduled item
(zero if the ready queue is non-empty, infinity if nothing is scheduled)
2. Add I/O handlers that should be called to the ready queue
3. Move scheduled items that are now dueue to the ready queue
4. Run everything that is *currently* in the ready queue

When the sentinel from stop() is found in the ready queue, the loop stops,
leaving any remaining items in the ready queue. When the loop is restarted,
we start from the top, with a deadline equal to zero.

The other useful thing to understand is the data structures used by the
event loop:

- a ready queue (the aforementioned FIFO)
- a heapq of scheduled items, sorted by the time they are due
- a data structure to keep track of I/O callbacks (typically a selector,
except when using IOCP on Windows)

Much hand-wringing went into deciding what to do with callbacks put into
the ready queue by call_soon() while step 4 above is running. The decision
was *not* to run these immediately but to loop back to the top, using a
deadline of zero for the I/O wait. This favors responding to I/O over
running callbacks, but still runs the first "wave" of callbacks immediately
without checking back for more I/O.


> That said, I am very excited to have this functionality. I have been
> sticking with Python 2 for now, but this feature may entice me to make
> the switch.
>

That was part of the point. :-)


>  -- Russell
>
>
> >
> > ---------- Forwarded message ----------
> > From: Victor Stinner <victor.stinner at gmail.com>
> > Date: Sat, Feb 8, 2014 at 2:38 PM
> > Subject: [python-tulip] Need help to finish asyncio documentation
> > To: python-tulip <python-tulip at googlegroups.com>
> >
> >
> > Hi,
> >
> > I wrote most parts of the documentation of the asyncio module, but I'm
> > not sure that anyone already read it yet. Can you please at least take
> > at look?
> > http://docs.python.org/dev/library/asyncio.html
> >
> > Tell me if the documentation needs more examples. I don't want to add
> > whole applications, only very short examples to explain one feature or
> > concept, or show how to use one specific API.
> >
> > I just realized that add/remove_reader/writer() methods of the event
> > loop were not documented, sock_recv/sendall/accept/connect() methods
> > neither. I documented them.
> >
> > There are still many functions which only have "XXX" for documentation.
> >
> > If you would like to contribute, send patches on .rst files. The
> > source of the documentation is in the Doc/library/ directory of
> > CPython repository:
> > http://hg.python.org/cpython/
> >
> > Files asyncio-*.rst:
> > http://hg.python.org/cpython/file/default/Doc/library
> >
> > Victor
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140210/547b9632/attachment.html>


More information about the Python-Dev mailing list