Producer-consumer threading problem
george.sakkis at gmail.com
Thu Jun 12 06:51:05 CEST 2008
On Jun 11, 10:13 am, Jean-Paul Calderone <exar... at divmod.com> wrote:
> On Tue, 10 Jun 2008 22:46:37 -0700 (PDT), George Sakkis <george.sak... at gmail.com> wrote:
> >On Jun 10, 11:47 pm, Larry Bates <larry.ba... at websafe.com`> wrote:
> >> I had a little trouble understanding what exact problem it is that you are
> >> trying to solve but I'm pretty sure that you can do it with one of two methods:
> >Ok, let me try again with a different example: I want to do what can
> >be easily done with 2.5 Queues using Queue.task_done()/Queue.join()
> >(see example athttp://docs.python.org/lib/QueueObjects.html), but
> >instead of having to first put all items and then wait until all are
> >done, get each item as soon as it is done.
> >> 1) Write the producer as a generator using yield method that yields a result
> >> every time it is called (something like os.walk does). I guess you could yield
> >> None if there wasn't anything to consume to prevent blocking.
> >Actually the way items are generated is not part of the problem; it
> >can be abstracted away as an arbitrary iterable input. As with all
> >iterables, "there are no more items" is communicated simply by a
> >> 2) Usw somethink like Twisted insted that uses callbacks instead to handle
> >> multiple asynchronous calls to produce. You could have callbacks that don't do
> >> anything if there is nothing to consume (sort of null objects I guess).
> >Twisted is interesting and very powerful but requires a different way
> >of thinking about the problem and designing a solution. More to the
> >point, callbacks often provide a less flexible and simple API than an
> >iterator that yields results (consumed items). For example, say that
> >you want to store the results to a dictionary. Using callbacks, you
> >would have to explicitly synchronize each access to the dictionary
> >since they may fire independently.
> This isn't true. Access is synchronized automatically by virtue of the
> fact that there is no pre-emptive multithreading in operation. This is
> one of the many advantages of avoiding threads. :) There may be valid
> arguments against callbacks, but this isn't one of them.
Thanks for the correction; that's an important advantage for callbacks
in this case.
More information about the Python-list