[Python-ideas] Concurrency Modules

[Nick Coghlan]

On 11 July 2015 at 09:42, Steve Dower <Steve.Dower at microsoft.com> wrote:
> Processes and threads only really enter into asyncio as a "thing that can post messages back to my TODO list/event loop", while asyncio provides an efficient mechanism for interleaving (not parallelising) multiple tasks throughout an entire application (or a very significant self-contained piece of it). The parallelism only comes when all the main thread has to do for a particular task is wait, because another thread/process/service/device/etc. is doing the actual work.
>
> Hopefully that helps clear things up for some people. No example is perfect for everyone, ultimately, so the more we put out there the more likely

I really like this example, so I had a go at expressing it in the
foreground/background terms I use in
http://www.curiousefficiency.org/posts/2015/07/asyncio-tcp-echo-server.html

For folks that already know asyncio, the relevant semantic details of
that proposal for this example are:

    run_in_foreground -> convenience wrapper for run_until_complete
    run_in_background(coroutine) -> convenience wrapper for ensure_future
    run_in_background(callable) -> convenience wrapper for run_in_executor

I quite like the end result:

    # We'll need the concept of an oven
    class Oven:

        # There's a shared pool of ovens we can use
        @classmethod
        async def get_oven():
            ...

        # An oven can only have one set of current settings
        def configure(self, recipe):
            ...

        # An oven can only cook one thing at a time
       def bake(self, mixture):
            ...

    # We stay focused on this task
    def gather_ingredients(recipe):
        ...
        return ingredients

    # Helper to indicate readiness to switch tasks
    def survey_the_kitchen():
        return asyncio.sleep(0)

    # This task may be interleaved with other activities
   async def mix_ingredients(recipe, ingredients):
        mixture = CakeMixture(recipe)
        for ingredient in ingredients:
            mixture.add(ingredient)
            await survey_the_kitchen()
        return mixture

    # This task may be interleaved with other activities
    async def make_cake(recipe):
        # First, we gather and start mixing the ingredients
        ingredients = gather_ingredients(recipe)
        mixture = await mix_ingredients(recipe, ingredients)
        # We wait for a free oven, then configure it for our recipe
        oven = await Oven.get_oven()
        oven.configure(recipe)
        # Baking is synchronous for the *oven*, but *we* don't
        # want to sit around waiting for it the entire time
        bake_cake = functools.partial(oven.bake, mixture)
        return await run_in_background(bake_cake)

    # We have three cakes to make
    make_sponge = make_cake("sponge")
    make_madeira = make_cake("madeira")
    make_chocolate = make_cake("chocolate")

    # Which we'll try to do concurrently
    run_in_foreground(asyncio.wait([make_sponge, make_madeira, make_chocolate]))

    sponge_cake = make_sponge.result()
    madeira_cake = make_madeira.result()
    chocalate_cake = make_chocolate.result()

Now, to upgrade this to full event driven programming: imagine you're
modeling a professional bakery, accepting cake orders from customers.
Then you would need to define a server process that turns orders from
customers into cake making requests, and completed cake notifications
into delivery orders, and your main thread becomes devoted to running
that server, rather than specifying a pre-selected set of cakes to
make.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia