[Catalog-sig] local copy of pypi packages list, and package data, how?

[Lukasz Szybalski]

Hello,
I've been looking into xmlrpc interface that you have for pypi. I am
able to browse all packages and get the data about
them...keywords,etc....I want to have a local version of the catalog
data and keep it in sync daily.

What I'm wondering is how can I keep my app in sync? Here is what I'm
doing right now, I was wondering if this is not overloading your
servers, or is there a faster/more efficient way.

1. I get a list of all 5000+ packages.
2. For each package I get a version number.
3. For each (package,version#) I get the package data.
4. Sync daily using updated_releases

I use the package data to look for certain keywords. Process 2 seem to
take around 20+minutes, process 3 takes more, but after first time I
can just get the new updated packages since the last time and run
these.

1. Is there a xmlrpc function that I can use to search for keywords
and just get the packages I need?
2. Is there a better strategy then what I am doing? I would like to sync daily.


http://lucasmanual.com/blog/2009/how-to-get-information-from-pypi-via-xmlrpc/

Thanks,
Lucas



-- 
How to create python package?
http://lucasmanual.com/mywiki/PythonPaste
Bazaar and Launchpad
http://lucasmanual.com/mywiki/Bazaar