asyncore and asynchat are in a difficult position right now since a lot of patches for both modules are pending and no decisions are taken. In detail I'm talking about patches 1519, 1541, 2073 and 1736190 which is the most important one since it includes a lot of fixes for other reported issues (e.g. 1740572, 1736101, 909005). As I said in 1563, IMHO your patch could be reviewed and eventually committed only after 1736190 and others are committed and 2.x related problems are solved. Converting the code to 3.0 should be the last step also because my perception is that your patch changes too much of the existing code: a new iterator_producer, a different collect_incoming_data() behavior, name mangling of internal variables and others. ...A lot of stuff which could be included in a second time. I'm going to inform Josiah Carlson about this topic since he should be the most experienced one about asyncore and asynchat. On 14 Feb, 03:52, Bill Janssen wrote:

It's a big patch, but I'll try applying it to the current py3k branch -- does it apply? -- and try a few things with it.  I'm concerned about how well it behaves with things like Medusa (which probably needs its own py3k update).

Bill _______________________________________________ Python-Dev mailing list Python-...@python.orghttp://mail.python.org/mailman/listinfo/python-dev Unsubscribe:http://mail.python.org/mailman/options/python-dev/python-dev2-garchiv...

First of all, you're conflating my two possible patches. One patch just addresses the problem of strings and bytes, as GvR asked me to do, and adds an 8-line class called iterator_producer that adapts iterators into producers. The patch doesn't change how the module works at all, and iterator_producer is not invoked anywhere within the code; it's purely for user convenience. I consider this the primary patch and would like to focus attention there if possible. The other patch combines the string and bytes fix with a porting of 1736190 and the other things you complain about, most of which scratch personal itches. If the patches you mention are actually going to be applied, then this patch isn't the way to go, and I'll maybe submit parts of it as separate patches. If they're just going to waste away in the bug tracker, though, this patch should be seriously considered. I'm quite willing to re-construct my string and bytes patch against a version of py3k in which the pre-existing patches are already applied. There needs to be forward progress, though: If nothing at all gets done, asyncore is going to be removed from the standard lib. I don't want to see that happen. On Thu, Feb 14, 2008 at 5:55 AM, Giampaolo Rodola' wrote:

(wrong quoting: obvioulsly I was talking to Daniel)

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/djarb%40highenergymagic.or...

Ok, I'll try to take a look at all asyncore/chat reports and try to summarize them by splitting patches which solve bugs and patches which add enhancements or functionalities. On 14 Feb, 16:12, "Facundo Batista" wrote:

2008/2/14, Giampaolo Rodola' :

...

asyncore and asynchat are in a difficult position right now since a  lot of patches for both modules are pending and no decisions are  taken.  In detail I'm talking about patches 1519, 1541, 2073 and 1736190 which  is the most important one since it includes a lot of fixes for other  reported issues (e.g. 1740572, 1736101, 909005).

I took a look to some of these in other opportunity, and IIRC, the main issue here is exactly what you say: lots of issues with problem *and* fixes, but some decisions needs to be taken.

No decision taken, the pile of issues (and real problems behind them) accumulate through time. I think it's a good idea to bring this discussion here, and with luck a result will come up regarding them.

So, it would be great if you (please) summarize here the problems behind those issues and the decisions that must be taken... Thanks!

Regards,

-- .    Facundo

Blog:http://www.taniquetil.com.ar/plog/ PyAr:http://www.python.org/ar/ _______________________________________________ Python-Dev mailing list Python-...@python.orghttp://mail.python.org/mailman/listinfo/python-dev Unsubscribe:http://mail.python.org/mailman/options/python-dev/python-dev2-garchiv...

Hey everyone, Sorry I haven't been available for this recently, I've been working far too much (10-14 hours/day, 6 days/week, since November) to really do any "fun" programming. Also, sorry for top-posting. As I stated 2+ and 6+ months ago, the patches I submitted 9+ months ago work (I've been using them in my own projects without issue). The only issue with the bugfix/rearrangement that I last heard about with regards to the 2.x stuff was that I removed a class that no one was using in the wild, which I believe Giampaolo added in a subsequent patch in the last couple months. My suggestion: 1. Apply the most recent fixes to 2.6 asyncore/asynchat that Giampaolo has updated. 1.a. Figure out what the hell is up with OOB data, how to handle it, and stop making it use handle_expt(). 2. Use the 2.x -> 3.x tool to convert the fixes over. 3. Apply any 3.x specific fixes (for string/bytes, etc.) to the 3.x branch as necessary (make them global constants in both 2.x and 3.x so that they are easy to track). 4. Consider enhancements to 2.6 if they aren't to big, consider slightly larger enhancements to 3.x. * - Josiah * Scheduled tasks are not a valid enhancement for either; anyone who wants them can trivially add them with their own asyncore.loop() variant and call asyncore.poll() as necessary (I did it in about 15 lines in the summer of 2002, I hope others can do better now). If you want my opinion on other async-related features, feel free to email me directly (use the gmail address you see here, then it ends up in my inbox, not the overflowing python folder). On Thu, Feb 14, 2008 at 9:09 AM, Giampaolo Rodola' wrote:

On 14 Feb, 16:36, "Giampaolo Rodola'" wrote:

...

Ok, I'll try to take a look at all asyncore/chat reports and try to summarize them by splitting patches which solve bugs and patches which add enhancements or functionalities.

On 14 Feb, 16:12, "Facundo Batista" wrote:

...

2008/2/14, Giampaolo Rodola' :

...

...

asyncore and asynchat are in a difficult position right now since a lot of patches for both modules are pending and no decisions are taken. In detail I'm talking about patches 1519, 1541, 2073 and 1736190 which is the most important one since it includes a lot of fixes for other reported issues (e.g. 1740572, 1736101, 909005).

...

I took a look to some of these in other opportunity, and IIRC, the main issue here is exactly what you say: lots of issues with problem *and* fixes, but some decisions needs to be taken.

...

No decision taken, the pile of issues (and real problems behind them) accumulate through time. I think it's a good idea to bring this discussion here, and with luck a result will come up regarding them.

...

So, it would be great if you (please) summarize here the problems behind those issues and the decisions that must be taken... Thanks!

...

Regards,

...

-- . Facundo

...

Blog:http://www.taniquetil.com.ar/plog/ PyAr:http://www.python.org/ar/

=== Patches for existing issues ===

- 1736190 which includes fixes for the following issues among other improvements: - 1063924 (asyncore should handle ECONNRESET in send()) - 1736101 (asyncore should handle ECONNABORTED in recv()) - 760475 (handle_error() should call handle_close() instead of close()) - 1740572 (refill_buffer() should call handle_close() rather than close()) - 777588 (wrong "connection refused" behavior on Windows) - 889153 (wrong connect() behavior) - 953599 (asyncore misses socket closes when poll is used) - 1025525 (asyncore.file_dispatcher should not take fd as argument)

- 1519 (async_chat.__init__() and asyncore.dispatcher.__init__ parameters inconsistency) - 1541 (Bad OOB data management when using asyncore with select.poll()) - 2073 (asynchat push always sends 512 bytes (ignoring ac_out_buffer_size))

=== Open issues with no patches (need review) ===

- 658749 (asyncore connect() and winsock errors) - 1161031 (neverending warnings from asyncore)

=== Enhancements & new features ===

- 1641 (add delayed calls feature) - 1563 (conversion to py3k and some other changes)

IMHO the first thing to do should be modifying 1736190 patch to fix the minor issues came out in comments 52767 (re-add simple_producer and fifo classes) and 57581 (look at what is specified in ac_out_buffer_size) and then commit it. I've used that same modified asynchat version in my pyftpdlib project and I can tell that it works pretty good. I guess that Josiah Carlson could do that pretty quickly if he has time to do so.

Independently from all a nice thing to do would be adding tests for many asyncore/chat behaviors which currently aren't covered by the test suite. It could be very useful to know if behaviors between different commits remain the same, hence it would be better working on that (the test suite) before committing 1736190.

-- Giampaolo

On Fri, Feb 15, 2008 at 09:24:14AM -0500, A.M. Kuchling wrote:

On Thu, Feb 14, 2008 at 06:24:04PM -0800, Josiah Carlson wrote:

...

1.a. Figure out what the hell is up with OOB data, how to handle it, and stop making it use handle_expt().

Does OOB data actually need to be supported? For a long time TCP implementations usually had buggy OOB implementations because it was so rarely used; for all I know, that's still the case. Maybe the feature should just be dropped.

Only if you're happy to make it impossible to implement some protocols using asyncore.

Twisted core has been proposed, but I believe the consensus was that it wasn't desirable, generally. I'm also pretty sure that people learn twisted because everyone learns twisted. It's one of those buzz-words ;). As for task scheduling, I believe it was something like... import asyncore import time import heapq tasks = [] def schedule(delta, callme): heapq.heappush(tasks, (time.time()+delta, callme)) def loop_with_schedule(timeout=30): while 1: now = time.time() while tasks and tasks[0][0] < now: callme = heapq.heappop(tasks)[1] try: callme() except (KeyboardInterrupt, SystemExit): raise except: #do something useful pass thistimeout = timeout if tasks: thistimeout = max(min(thistimeout, tasks[0][0]-now), 0) asyncore.poll(timeout=thistimeout) - Josiah On Fri, Feb 15, 2008 at 11:45 AM, Giampaolo Rodola' wrote:

On 15 Feb, 03:24, "Josiah Carlson" wrote:

...

As I stated 2+ and 6+ months ago, the patches I submitted 9+ months ago work (I've been using them in my own projects without issue). The only issue with the bugfix/rearrangement that I last heard about with regards to the 2.x stuff was that I removed a class that no one was using in the wild, which I believe Giampaolo added in a subsequent patch in the last couple months.

I provided the patch for the other issue (look at what is specified in ac_out_buffer_size) but I didn't re-add fifo and simple_producer classes. What should be done here? Re-add or discard them? However, I will send to you by e-mail the modified asynchat version. It is based on your patch therefore a first commit could be finally done.

...

My suggestion: 1. Apply the most recent fixes to 2.6 asyncore/asynchat that Giampaolo has updated.

...

1.a. Figure out what the hell is up with OOB data, how to handle it, and stop making it use handle_expt().

If we want to leave OOB untouched shouldn't handle_expt be the right place where manage such kind of events? Alternatively we could also remove the OOB management at all (e.g. Twisted does that by ignoring such kind of data). OOB could be actually useful to implement some protocols such as FTP (in details ABOR and STAT commands which require OOB data) but it would be probably better not having it than having it but not implemented properly. I am saying this also because later or soonish we might need to care of epoll and kqueue (http://bugs.python.org/issue1657).

...

* Scheduled tasks are not a valid enhancement for either; anyone who wants them can trivially add them with their own asyncore.loop() variant and call asyncore.poll() as necessary (I did it in about 15 lines in the summer of 2002, I hope others can do better now). If you want my opinion on other async-related features, feel free to email me directly (use the gmail address you see here, then it ends up in my inbox, not the overflowing python folder).

How's your solution? Could you post it here or send it to me by mail? IMO scheduled tasks is a very important feature for implementing a lot of interesting stuff like connection timeouts and bandwidth throttling. A lot of people have to learn/use Twisted just because of that. Moreover I think that Bill Janssen could need that feature to make the ssl module work with asyncore.

PS - I have been reading this mailing list for a short time therefore I have no clue whether or not someone has ever thought about including the Twisted core - only itself - in Python stdlib.

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/josiah.carlson%40gmail.com

On 15 Feb, 21:36, Bill Janssen wrote:

I think we should just replace the current "loop" with this (and add the "schedule" function).  Then other folks won't have to figure out how the module works and write their own loop.  I'll be happy to update the documentation to document it.

Bill

I'm -1. This does not permit to reset, cancel or "re-schedule" the scheduled tasks. Something like a connection timeout after a certain time of client's inactivity would not be possible to implement.

I've discussed a lot with Josiah via e-mail and I provided an updated version of the patch proposed in bug #1736190 including a fix for the two issues raised by me in the bug report. The update has been needed also because the original patch has been out-dated by some commits after r53734 involving the test suite and the documentation, which I both took off this patch. I also re-added simple_producer and fifo classes in asynchat.py since it seems they are needed for passing tests. The test suite has passed on Windows XP using Python 2.6 alpha 1. I have also successfully run the test suite of a project of mine based on asynchat which includes over 40 tests.

I'm going to refresh this discussion since it seems no decisions are still taken. Any chance to see a commit finally done? --- Giampaolo http://code.google.com/p/pyftpdlib

On Mon, Mar 24, 2008 at 3:04 PM, Thomas Wouters wrote:

On Fri, Feb 15, 2008 at 9:11 PM, Josiah Carlson wrote:

...

Twisted core has been proposed, but I believe the consensus was that it wasn't desirable, generally.

I remember only a couple of dissenting voices, and only a small number of participants. Of the dissenting voices, I do not recall any actual arguments about undesireability, just misunderstandings of how Twisted actually works. Getting Twisted core (meaning Deferreds, a simple reactor and the Protocol class) into the core is still on my TODO list.

...

I'm also pretty sure that people learn twisted because everyone learns twisted. It's one of those buzz-words ;).

I think that's quite an unfair assessment, even in jest :) Twisted is well worth learning to actually use it, as it's a very versatile event loop and does it best to integrate nicely with other event systems. And including it in the standard library improves integration with other event loops by creating a single interface. It's not a matter of dropping it in, though; it requires some careful structuring to avoid embarrassing situations like we have with the xml package, but still people to provide their own reactor.

In case you're wondering how the twisted reactor in the stdlib is useful to people not using Twisted, take a look at what you currently need to do to combine stdlib modules like urllib and ftplib with event systems like Tkinter and PyGTK. Not to mention that the Twisted implementations of various protocols are really quite, quite good -- in many cases quite a lot better than the stdlib ones. But including those takes yet more time.

In that sense it'd be competing with safethread for inclusion in Python. Whereas safethread requires little if any API changes, twisted requires an entirely new API that can be event-driven. Worse, we'd likely to be stuck maintaining both APIs for a long time, if not forever. Twisted may be one of the best (if not *the* best) ways of writing concurrent programs today, but it doesn't need to be in the stdlib for that. If safethread is going to solve many of the same problems, with less changes required by the users of the language, then this is the wrong time to add twisted. -- Adam Olsen, aka Rhamphoryncus

On Mon, Mar 24, 2008 at 3:37 PM, Thomas Wouters wrote:

On Mon, Mar 24, 2008 at 10:21 PM, Adam Olsen wrote:

...

Twisted may be one of the best (if not *the* best) ways of writing concurrent programs today, but it doesn't need to be in the stdlib for that. If safethread is going to solve many of the same problems, with less changes required by the users of the language, then this is the wrong time to add twisted.

You must have missed the part where we already have a large set of event loops, and not having a single interface to them is in fact hurting people. Twisted goes out of its way to interact nicely with event loops, but it can only do that with ones it knows about (and are versatile enough to hook into.) Having a single event system in the standard library is definitely advantageous, even if safethreads were available everywhere and the performance in the common case was satisfactory. It used to be the case that people thought asyncore was this standard event system, but it's long since ceased to be.

I'm not opposed to standardizing on twisted as the canonical way to do events in python, or to having an event system in python. My concern is that may be used as an excuse to slowly rewrite the entire language into an event-driven model. However, that was based on the assumption that modules like urllib2 weren't already event-driven. Looking now, it seems it already is, and wraps it in a blocking API for simple use cases. So long as we're only talking about replacing existing event-driven stuff, and so long as we retain the existing blocking API (including calling from threads!), I don't think I have any valid opposition. -- Adam Olsen, aka Rhamphoryncus

Let us not get side-tracked in this discussion. Whether or not to include any portion of Twisted into Python 2.6 is well past being a reasonable question; 2.6 alpha 1 has been released. It's a question as to whether someone with commit access can or will commit the patch as posted, run the tests to verify that they aren't broken, and perhaps actually look at the code to verify that we (Giampaolo and I) aren't insane. Mind you, I've been using very similar variants of this patch for months; it fixes outstanding bugs, improves performance, makes the handle* interfaces more consistent, and even offers a 'sample' implementation of a basic protocol in the source (for those who are willing to look). Do we want to fix asyncore/asynchat with work that has already been done and tested? If you want a reason as to why twisted shouldn't *replace* asyncore/asynchat, I'll give you a few: As stated previously by Guido and others (please see previous threads about this over the course of the last 4 years), asyncore/asynchat provide a more or less minimal interface for asynchronous sockets in Python. Any module/package that seeks to implement asynchronous sockets will need to, at least, implement that much. Asyncore/asynchat at present will play nicely with any event loop available, given certain caveats*. Further, if someone spends a half hour reading the source of a reasonably written asyncore server/client, they should understand the basics and be able to begin using it directly (see any one of the simple echo variants). As to whether twisted should be added to the standard library at some point in the future as a "this is better than asyncore in every way, use this instead"; that is a different discussion, not related to 2.6 (perhaps not even related to the 2.x series at all, depending on the future of 2.x). - Josiah * If your application strictly responds to socket IO, then implement your application as part of handle_* methods. If your application does more, then call asyncore.poll() often enough for data to be handled sufficiently fast. If neither offer sufficient performance or flexibility (maybe something like bittorrent + wxPython), use one thread for your GUI, one thread for your sockets, and use Queue.Queue() or some other threadsafe message delivery method. On Mon, Mar 24, 2008 at 3:18 PM, A.M. Kuchling wrote:

On Mon, Mar 24, 2008 at 10:04:20PM +0100, Thomas Wouters wrote:

...

I remember only a couple of dissenting voices, and only a small number of participants. Of the dissenting voices, I do not recall any actual arguments

Weren't some of those dissenting voices the Twisted developers, though?

--amk

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/josiah.carlson%40gmail.com

On Thu, Feb 14, 2008 at 10:09 AM, Giampaolo Rodola' wrote:

On 14 Feb, 16:36, "Giampaolo Rodola'" wrote:

...

Ok, I'll try to take a look at all asyncore/chat reports and try to summarize them by splitting patches which solve bugs and patches which add enhancements or functionalities.

=== Patches for existing issues ===

- 1736190 which includes fixes for the following issues among other improvements: - 1063924 (asyncore should handle ECONNRESET in send()) - 1736101 (asyncore should handle ECONNABORTED in recv()) - 760475 (handle_error() should call handle_close() instead of close()) - 1740572 (refill_buffer() should call handle_close() rather than close()) - 777588 (wrong "connection refused" behavior on Windows) - 889153 (wrong connect() behavior) - 953599 (asyncore misses socket closes when poll is used) - 1025525 (asyncore.file_dispatcher should not take fd as argument)

- 1519 (async_chat.__init__() and asyncore.dispatcher.__init__ parameters inconsistency) - 1541 (Bad OOB data management when using asyncore with select.poll()) - 2073 (asynchat push always sends 512 bytes (ignoring ac_out_buffer_size))

=== Open issues with no patches (need review) ===

- 658749 (asyncore connect() and winsock errors) - 1161031 (neverending warnings from asyncore)

=== Enhancements & new features ===

- 1641 (add delayed calls feature) - 1563 (conversion to py3k and some other changes)

That's a lot of patches. My immediate concern is that test_asynchat is very flaky and fails often. Sometimes it causes other tests to fail. Is there a patch that addresses this? If you need examples, just look through the buildbot mails that mention test_asynchat in: http://mail.python.org/pipermail/python-checkins/ Some platforms have more problems than others, but almost all platforms have failed test_asynchat at one point or another. Please break up the patches into 2 sets and prioritize the patches with the set. Set #1: Patches that have a test and doc updates if required Set #2: Patches that don't have a test or doc updates as required For the patches that fall into Set #1, list them in priority order. Top priority would be a problem that fixes failures seen in the buildbots. Next priority would go to the patches that solve more serious problems. Post the results here. I will try to look at them. For every patch you list, make sure that it conforms to the proper style (e.g, PEP 8) and is essentially perfect and ready for inclusion. This means that there is a single file to download that contains all the modifications. The changes are appropriately commented, lines are less than 80 characters, etc. One of the modifications should be an entry in Misc/NEWS. n