[Catalog-sig] Re: Meeting followup- CPAN for Python?

[Ian Bicking]

Richard Jones wrote:
> On Sat, 11 Dec 2004 05:51 am, Ian Bicking wrote:
> 
>>* Track package names into PyPI.  For now these will serve as 
>>identifiers for installation and dependencies.  Conflicting package 
>>names already cause a lot of problems anyway, so it would be nice if 
>>they'd be unique.
> 
> This step is done. In PyPI, a name is owned by someone (or some people).

True.  The downside is that it isn't the Python package name, it's just 
a name.  The Python package name seems like a natural identifier; though 
some projects have no packages (e.g., many applications), and some have 
multiple packages.

>>* Add a field for package dependencies, based on those names.  Forget 
>>about version requirements for now.  Though in theory, since distutil 
>>setup.py files are just Python scripts, this should be extensible.
> 
> 
> Also, handling versioning implies that people retain versioned source 
> downloads - something we can't rely on.

Well, we couldn't necessarily resolve problems, but distutils could at 
least complain obnoxiously.

> Possibly what we need is for the package to be able to indicate which versions 
> it's been *tested* with, so it can warn that the download might not work? 

That seems fairly easy.

> IIRC a problem with CPAN was version incompatibilities breaking things (I 
> know I've hosed my Perl install a couple of times)

Doing multiple installs of different versions in Python isn't that hard, 
but it's not that easy either.  That's a fairly expansive problem.

> Now, having dependencies list is all well and good - what do we do with them? 

I don't think we should look that far ahead ;)  If I was going to leave 
something out of this iteration, the first thing would be the 
dependencies list.  You can always put this at the top of your setup.py:

try:
     import some_package
except ImportError:
     print "You have not installed a required package: some_package"
     sys.exit(1)

> If we have automated checking against those dependencies, then that implies 
> that we remember what's installed. AMK has done some work on this:
> 
> Database of Installed Python Packages
>   http://python.org/peps/pep-0262.html

Certainly useful.

> Other relavent PEPs:
> 
> Package Index and Metadata for Distutils
>   http://python.org/peps/pep-0301.html
>   (*still* marked "under consideration"... sigh)
> Metadata for Python Software Packages v1.1
>   http://python.org/peps/pep-0314.html
> 
> Metadata version 1.1 has fields for "download-url", "requires", "provides", 
> "obsoletes" and "conflicts". I guess we need to change "download-url" to 
> "source-url". Mmm. Ambiguity. 

Indeed, it seems like half of the packages don't have source on the 
other side of their download URL.  Lots don't even have a download page, 
or even a package-specific page.  In cases of ambiguity, authors choose 
to be lazy...

It also needs to support multiple kinds of downloads.  Freshmeat has 
this, though the kinds of downloads are fixed, which seems unnecessary 
for us.  System-specific packages are usually prefered if available. 
Well, sometimes -- usually system-specific packages are harder to use 
for multi-version installs.  Anyway, there's a place for several kinds 
of packages.

> It also removes Platform, and specifies the 
> formatting of the long description field. The only open issue in that PEP is 
> the license field. I believe some people would like it to remain, but only be 
> used when their license doesn't appear in the Classifiers list, and they need 
> to include the full text of their license (or some other statement about it). 
> See PyPI bug 693471.
> 
> Also of interest:
> 
> Module Repository Upload Mechanism
>   http://python.org/peps/pep-0243.html
> 
> 
> 
>>* Add a field for source download; maybe make it a dictionary, so you 
>>can give several links for different sources (e.g., source tarball/zip, 
>>rpm, windows installer, debian package, etc).
> 
> 
> I think just sticking with the source installer is fine. This is all invoked 
> by setup.py, isn't it?

Well, by phrasing this in terms of data, not functionality, we make this 
potentially frontend-neutral.  We can already generate rpm's from 
distutils, and I don't know why debs haven't also been added to that. 
In many cases they would be preferred.

>>* Create a script that can query PyPI, get the link(s), then download 
>>it.  PyPI already has an XML-RPC interface, I believe.
> 
> 
> PyPI's web interface is a complete shambles. I really want to reimplement it 
> in something easier to maintain, and more extensible. As an aside: the pypi 
> sqlite database really needs to move to postgresql.

Would that be hard?  Is it mostly a sysadmin issue, or a coding issue?

What do you want to change in PyPI's interface?

>>Because of SF,  
>>the downloader has to be a little smart about the load balancing page in 
>>that case, but that's relatively easy.
> 
> We'll be special-casing for them, but there will probably be other download 
> sites like this :(

Yeah... well, at least for SF it is worth doing the special case, simply 
because of the number of files hosted there.  After that, who knows.

>>Well, these are some of my ideas.  If people are interested in this in
>>general, we could try to organize a little mini-sprint; a number of us
>>would come together and try to bang it out.  We could try to schedule
>>and coordinate this with Richard or other interested people over IRC.
> 
> 
> Looks like I'll be attending PyCon, so a sprint there would be possible.

Well, people here at ChiPy have been showing some interest, and most of 
them probably won't be going to PyCon.  So people might want to try 
tackling some of these things on a different schedule.  And the PyCon 
sprints won't be during the weekend, so that would also make it hard for 
non-PyCon people to participate.

-- 
Ian Bicking  /  ianb at colorstudy.com  /  http://blog.ianbicking.org