[Catalog-sig] PyPI limitations (was: Deprecate External Links)

[M.-A. Lemburg]

On 27.02.2013 19:11, Noah Kantrowitz wrote:
> 
> On Feb 27, 2013, at 9:28 AM, M.-A. Lemburg wrote:
> 
>> On 27.02.2013 18:05, Noah Kantrowitz wrote:
>>>
>>>
>>> "M.-A. Lemburg" <mal at egenix.com> wrote:
>>>>> I propose we deprecate the external links that PyPI has published
>>>>> on the /simple/ indexes which exist because of the history of PyPI.
>>>>> Ideally in some number of months (1? 2?) we would turn off adding
>>>>> these links from new releases, leaving the existing ones intact and
>>>>> then a few months later the existing links be removed completely.
>>>>
>>>> -1.
>>>>
>>>> There are many reasons for not hosting packages and distributions
>>>> on PyPI itself.
>>>>
>>>
>>> [citation needed]
>>
>> We've been through this discussion a couple of times in the past.
>> I'm sure the reasons will get listed again in this discussion :-)
>>
>> Too many distribution files for PyPI to handle,
> 
> Again, please point at a specific package. I wasn't aware that PyPI limited uploads at all, but if it does we can certainly increase the number if there is a good reason.

PyPI limits the size of the distribution files (at 40MB),
but it doesn't limit the number of distribution files.

However, taking our egenix-mx-base package as example, we have
120 distribution files for every single release. Uploading those
to PyPI would not only take long, but also quickly get the
PyPI storage requirements up to a few TB if just a few package
authors start to do the same.

>> no support for
>> UCS2/UCS4 binary distributions, unsupported distribution file
>> formats (e.g. our prebuilt format),
> 
> Not sure why PyPI would even care what charset the package files use, but if true thats certainly a bug and we can get that fixed. What file formats do pip/buildout support that PyPI doesn't support for uploads?

Not the charset of the package files :-) I'm talking about binary
files for Python UCS2 vs. UCS4 builds. You have to ship both
variants for Unix platforms.

Regarding file formats: PyPI applies a number of checks for
the supported file formats which not only check the extension,
but also look inside the files to only accept a certain number
of formats.

See https://bitbucket.org/loewis/pypi/src/9863fa859e4b/verify_filetype.py?at=default
for details.

I was under the impression that this would filter out our
prebuilt format, but I just tried an upload and it does seem
to pass the tests, so I have to correct the above - our
prebuilt format is supported by PyPI (hey, one problem less
to worry about ;-)).

About the prebuilt format:

We created the prebuilt binary package
format a while ago to overcome issues with eggs not being
flexible enough and not carrying enough information to differentiate
between e.g. UCS2/UCS4 build of Python or properly identifying
platforms.

The format works with easy_install and pip, because the interface
is the same as for sdist files: you unzip the archive, run
"python setup.py ...commands..." and you're done.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Feb 26 2013)
>>> Python Projects, Consulting and Support ...   http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/