On Sat, Oct 27, 2012 at 1:06 PM, Daniel Foerster wrote:

So, are threads still an option? I feel that many of these problems with generators could be solved with threads.

No, because available operating systems can handle a few orders of magnitude more concurrent IO operations per process than they can handle threads per process. The idea of asynchronous programming is to only use additional threads when you really need them (i.e. for blocking synchronous operations with no asynchronous equivalent), thus providing support for a far greater number of concurrent operations per process than if you rely entirely on threads for concurrency. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Yury Selivanov wrote:

It would be great if we can address those problems with the new async API. I.e. we can use threadpools where necessary, but make the public API look fancy and yield-from-able. Same approach that Joyent uses in their libuv. And when OSes gain more advanced and wide non-blocking support we can decrease use of threads.

This certainly seems to be the plan, though I expect the details will be determined as libraries are updated to support the async API. As long as we ensure that the API itself can support event loops and operations using threads and other OS primitives, then we don't need to specify each and every one at this stage. My design (which I'm writing up now) puts most of the responsibility on the active scheduler, which should make it much easier to have different default schedulers for each platform (and maybe specialised ones that are optimised for more limited situations) while the operations themselves can be built out of existing Python functions. I'll post more details soon, but it basically allows schedulers to optionally support some operations (such as select() or Condition.wait()) 'natively', with the operation only having to implement a fallback (presumably on a thread pool). Cheers, Steve

Yes, thread pools are unfortunately necessary evils. Twisted comes with a few tools to handle the use cases we're discussing. The 1:1 equivalent for call_on_thread would be deferToThread/deferToThreadPool (deferToThread == deferToThreadPool except with the default thread pool instead of a specific one). There are a few other tools: - spawnProcess (equiv to subprocess module, except with async communication with the subprocess) - cooperative multitasking, such (twisted.internet.task.) Cooperator and coiterate: basically resumable tasks that are explicit about where they can be paused/resumed - third party tools such as corotwine, giving stackless-style coroutines, or ampoule, giving remote subprocesses The more I learn about other stuff the more I see that everything is the same because everything is different :) On Sat, Oct 27, 2012 at 12:27 PM, Kristján Valur Jónsson < kristjan@ccpgames.com> wrote:

Yes, stacklesslib provides this functionality with the call_on_thread() api, which turns a blocking operation into a non-blocking one. This is also useful for cpu bound operations, btw. For example, in EVE, when we need to do file operations and zipping of local files, we do it using this api. K

-- cheers lvh

On Sat, Oct 27, 2012 at 6:02 PM, Daniel Foerster wrote:

I'm realizing that I perhaps don't grasp the entirety of Asynchronous programming. However, The only results I have found are for .NET and C#. Would you like to recommend some online sources I could read?

A simple question with a multitude of answers! Presumably you are more interested in the gory details of async programming, whereas most of this discussing has been about what the code looks like. The wikipedia articles, while not fantastic, aren't terrible: https://en.wikipedia.org/wiki/Event_loop https://en.wikipedia.org/wiki/Asynchronous_I/O Also, there's a great introduction at http://krondo.com/?page_id=1327 ; which unfortunately comes wrapped as a twisted tutorial ;) (You can stop reading a while in when it becomes twisted-specific). -- cheers lvh