[Catalog-sig] local copy of pypi packages list, and package data, how?

Lukasz Szybalski szybalski at gmail.com
Thu Jan 22 05:52:53 CET 2009

I've been looking into xmlrpc interface that you have for pypi. I am
able to browse all packages and get the data about
them...keywords,etc....I want to have a local version of the catalog
data and keep it in sync daily.

What I'm wondering is how can I keep my app in sync? Here is what I'm
doing right now, I was wondering if this is not overloading your
servers, or is there a faster/more efficient way.

1. I get a list of all 5000+ packages.
2. For each package I get a version number.
3. For each (package,version#) I get the package data.
4. Sync daily using updated_releases

I use the package data to look for certain keywords. Process 2 seem to
take around 20+minutes, process 3 takes more, but after first time I
can just get the new updated packages since the last time and run

1. Is there a xmlrpc function that I can use to search for keywords
and just get the packages I need?
2. Is there a better strategy then what I am doing? I would like to sync daily.



How to create python package?
Bazaar and Launchpad

More information about the Catalog-SIG mailing list