Too many open files

[Gabriel Genellina]

En Mon, 04 Feb 2008 12:50:15 -0200, Christian Heimes <lists at cheimes.de>  
escribi�:

> Jeff wrote:
>> Why don't you start around 50 threads at a time to do the file
>> writes?  Threads are effective for IO.  You open the source file,
>> start a queue, and start sending data sets to be written to the
>> queue.  Your source file processing can go on while the writes are
>> done in other threads.
>
> I'm sorry, but you are totally wrong. Threads are a very bad idea for IO
> bound operation. Asynchronous event IO is the best answer for any IO
> bound problem. That is select, poll, epoll, kqueue or IOCP.

The OP said that he has this problem on Windows. The available methods  
that I am aware of are:
- using synchronous (blocking) I/O with multiple threads
- asynchronous I/O using OVERLAPPED and wait functions
- asynchronous I/O using IO completion ports

Python does not (natively) support any of the latter ones, only the first.  
I don't have any evidence proving that it's a very bad idea as you claim;  
altough I wouldn't use 50 threads as suggested above, but a few more than  
the number of CPU cores.

-- 
Gabriel Genellina