[Distutils] dump of all PyPI project metadata available?
wes.turner at gmail.com
Thu Jul 23 06:12:21 CEST 2015
On Jul 22, 2015 5:12 PM, "Brett Cannon" <bcannon at gmail.com> wrote:
> On Wed, Jul 22, 2015 at 2:19 PM Wes Turner <wes.turner at gmail.com> wrote:
> I'm not quite sure what I'm supposed to get from those links, Wes, as
that code still scrapes every project individually and downloads them while
all I'm trying to avoid having to scrape PyPI and instead just download a
single file (plus I don't want the files but just the metadata already
returned by the JSON API).
An online query or an offline dump?
>> - [ ] a flat bigquery w/ pandas.io.gbq ala GitHub Archive would be great
>> - [ ] it's probably worth it to add RDFa to PyPi and warehouse pages (in
addition to the auxiliary executed/extracted JSON) for #search
>> On Jul 22, 2015 4:08 PM, "Brett Cannon" <bcannon at gmail.com> wrote:
>>> When I wrote https://nothingbutsnark.svbtle.com/python-3-support-on-pypi
I wrote a script to download every project's JSON metadata by scraping the
simple index and then making the appropriate GET request for the JSON
metadata. It worked, but somewhat of a hassle.
>>> Is there some dump somewhere that is built daily, weekly, or monthly of
all the metadata on PyPI for offline analysis?
>>> Distutils-SIG maillist - Distutils-SIG at python.org
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Distutils-SIG