[Catalog-sig] PyPI optimization?

Gerhard Haering gh@ghaering.de
Fri, 25 Apr 2003 13:33:38 +0200


Richard Jones wrote:
> On Friday 25 April 2003 08:07 pm, Thomas Wouters wrote:
> 
>>Well, the machine is a lot heavier loaded now than it was before PyPI was
>>really used; the load used to be ~0.05, it's now hovering around 0.8,
>>sometimes as high as 1.5. That isn't terribly high, though, as this is a
>>dual CPU machine, and Apache for one isn't slowed down by it. I believe
>>it's PyPI doing heavy I/O, or perhaps some form of locking, as the python
>>processes aren't using that much CPU either.
> 
> 
> I haven't looked at the hits that PyPI is getting - does creosote have any web 
> stats automatically generated?
> 
> Perhaps it would be a good idea to switch to an RDBMS that is more 
> multi-user-friendly. Gerhard, is your offer to do this still open? 

Yes, I'd be glad to add this capability to PyPI. It'd help me a little 
if you could send me a dump of the current database, but it's not 
strictly necessary as I could create random test data myself.

> The 
> backend of pypi is all in the store module, and it's quite simple. It'll 
> probably work out of the box on postgresql/mysql.

 From a cursory look, it wouldn't be much work.

Once PyPI is more heavily used, we might also want to get away from the 
dependency on CGI, using something like Quixote (which can run under 
Apache using SCGI/mod_python or standalone with a Medusa or Twisted 
backend, and it can run as plain CGI as well). But first things first.

-- Gerhard