[Tutor] Python performance resources & resouce usage hints

[Liam Clarke]

Well, thanks very much Kent, Hugo and Danny.

I went with the "never-ending blocking queues" and sentinel data approach.
When running tests with a continual stream of packets being received
3ms apart, CPU usage peaked at 15%, was usually around 7-9%, and when
deployed the packets will separated by seconds  rather than
milliseconds.

Thanks for the assistance, I've now overcome my fear of blocking I/O :).

Regards,

Lia, Clarke

On 4/8/06, Liam Clarke <ml.cyresse at gmail.com> wrote:
> Thanks very much all. :) I'll have a crack this afternoon and let you know.
>
> Kent - the increase in the queue size for the socket server is to
> allow for any delay in processing packets; it has a default queue size
> of 5 and then it starts rejecting packets; more of a safety policy
> when reducing CPU usage than a direct attempt to reduce CPU usage.
>
> Once again, thanks for the input!
>
> On 4/8/06, Danny Yoo <dyoo at hkn.eecs.berkeley.edu> wrote:
> >
> >
> > On Fri, 7 Apr 2006, Kent Johnson wrote:
> >
> > > Hugo Gonz�lez Monteverde wrote:
> > > > You are not using the optional timeout and blocking which 'get' provides (!)
> > > >
> > > > Try setting it and see your CPU usage go down. This will implement
> > > > blocking, and the queue will be used as soon as data is there.      Set block
> > > > to True and a timeout if you need to use it (looks like you only need
> > > > blocking)
> > > >
> > > >          while True:
> > > >               try:
> > > >                  data = self.queue.get(True)
> > > >                  self.DAO.send_data(data)
> > >
> > > I think he will need the timeout too, otherwise the shutdown flag will
> > > only be checked when there is work which is probably not what he wants.
> >
> >
> > This could be fixed: when setting the 'shutdown' flag, also push a
> > sentinel piece of data into the queue.  That'll wake the thread back up,
> > and that'll give it the opportunity to process a shutdown.
> >
> > The thread here:
> >
> >     http://mail.python.org/pipermail/tutor/2006-January/044557.html
> >
> > has some more examples of this.
> >
> >
> > Hope this helps!
> >
> >
> >
> > _______________________________________________
> > Tutor maillist  -  Tutor at python.org
> > http://mail.python.org/mailman/listinfo/tutor
> >
> >
> >
>