[stdlib-sig] futures - a new package for asynchronous execution

[Anh Hai Trinh]

On Sun, Jan 17, 2010 at 5:22 AM, Brian Quinlan <brian at sweetapp.com> wrote:

>>> db_future = executor.submit(setup_database, host, port)
>>> data_future = executor.submit(parse_big_xml_file, data)
>>> # Maybe do something else here.
>>> wait(
>>>   [db_future, data_future],
>>>   timeout=10,
>>>   # If either function raises then we can't complete the operation so
>>>   # there is no reason to make the user wait.
>>>   return_when=FIRST_EXCEPTION)
>>>
>>> db = db_future.result(timeout=0)
>>> data = data.result(timeout=0)
>>> save_data_in_db(data, db)
>
> It is definitely true that you can roll your own implementation using
> threads but the purpose of the futures library is to make that unnecessary.
>
> I don't understand your doubts. To me the example that I gave is simple and
> useful.

What I mean is that your example is simple enough to do with threads. Here:

[...]

def setup_db():
  nonlocal db;
  db = setup_database(host, port)

def parse_xml():
  nonlocal data;
  data = parse_big_xml(file)

db_thread = threading.Thread(target=setup_db)
db_thread.start()

parse_thread = threading.Thread(target=parse_xml)
parse_thread.start()

[...] # Do something else here.

db_thread.join()
parse_thread.join()
save_data_in_db(data, db)

I used "nonlocal" here but you usually do this within a method and
refer to self.db, self.data.


> I don't understand your doubts. To me the example that I gave is simple and useful.

My doubt is about the usefulness of futures' constructs for the kind
of code that "Do several different operations on different data". I
think ThreadPool/ProcessPool is really useful when you do

1. Same operation on different data
2. Different operations on same datum

But

3. Different operations on different data

is perhaps misusing it. It is a too general use case because
dependency comes into play. What if the different operations depend on
each other? A useful thread pool construct for this would be at a more
fundamental level, e.g. Grand Central Dispatch.

Perhaps you would give another example?

Cheers,
-- 
// aht
http://blog.onideas.ws