Python Thread Question

[N.K]

So it seems a Optimal design for crawler is non-blocking sockets +
Ability to run more process/threads if required.i.e,

- Store all URLS to be fetched in queue, 
- One or more process/thread reading que and performing IO using non
blocking sockets
- Decide on number of process/threads is depending on situation 

Comments please,

Thanks in advance,
NKC

Jp Calderone <exarkun at intarweb.us> wrote in message news:<mailman.1051039477.30652.python-list at python.org>...
> On Tue, Apr 22, 2003 at 04:51:20AM -0700, N.K wrote:
> > aahz at pythoncraft.com (Aahz) wrote in message news:<b7meoo$mf6$1 at panix2.panix.com>...
> > > 
> > > This isn't quite true on a multi-CPU box.  Because I/O releases the GIL,
> > > the OS can schedule threads to run on all available CPUs.  So it's
> > > certainly possible for a threaded app to outrun non-blocking I/O.
> > 
> > 
> >   Heard that new Intel Processors support Hyper Threading ( HT)
> > Architecture from which multi-threaded apps can benefit from. Is that
> > true?
> 
>   Some (All?) P4s have this technology.  Practically speaking, this means
> applications see the system as having multiple processors.  More than one
> thread can run at once.  So yes, multi-threaded apps benefit from this in
> much the same way they benefit from traditional SMP systems.
> 
>   Jp
> 
> -- 
> Seduced, shaggy Samson snored.
> She scissored short.  Sorely shorn,
> Soon shackled slave, Samson sighed,
> Silently scheming,
> Sightlessly seeking
> Some savage, spectacular suicide.
>                 -- Stanislaw Lem, "Cyberiad"