From richardjones at optushome.com.au Mon May 2 00:29:46 2005
From: richardjones at optushome.com.au (Richard Jones)
Date: Mon, 2 May 2005 08:29:46 +1000
Subject: [Catalog-sig] New PEP, pre-submission RFC
In-Reply-To: <20050429095933.GA25977@mylene.ghaering.de>
References: <200504291943.32344.richardjones@optushome.com.au>
<20050429095933.GA25977@mylene.ghaering.de>
Message-ID: <200505020829.50038.richardjones@optushome.com.au>
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On Fri, 29 Apr 2005 07:59 pm, Gerhard Haering wrote:
> For those who have names with characters not in ASCII, is that possible?
> do we just define that all text is UTF-8?
At the moment, the web interface / database is all UTF-8, so it therefore
assumes that submissions are UTF-8. I guess we should actually document
that :)
The one weak link of the current UTF-8-ness is the setup.py file, as you've
identified. In most cases that'll be the default encoding for Python code
(whatever that is, I should look it up ;) but in some cases it'll be
different. Regardless, we really need to UTF-8 encode the submission of the
setup.py-derived metadata to the webserver so there's no incorrect decoding
done at its end.
> Not that I really need it, but would this be the right time to think
> about optional description fields for other languages, too, like
>
> Description-fr, Description-de for French and German ones?
Eeek :)
Richard
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
iD8DBQFCdVhdrGisBEHG6TARAvQIAJwNDP6CaPRPggyEsDcJsMspeKg4MACeOKzo
yU2bfpcOJFWE7YIYW3zCzGY=
=47L7
-----END PGP SIGNATURE-----
From richardjones at optushome.com.au Mon May 2 00:42:50 2005
From: richardjones at optushome.com.au (Richard Jones)
Date: Mon, 2 May 2005 08:42:50 +1000
Subject: [Catalog-sig] "implementation" of PEP 262 as an academic project
In-Reply-To: <42731018.5060700@acm.org>
References: <1114735502.510591.6360@f14g2000cwb.googlegroups.com>
<4272CE91.5080908@acm.org> <42731018.5060700@acm.org>
Message-ID: <200505020842.50769.richardjones@optushome.com.au>
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On Sat, 30 Apr 2005 02:56 pm, Maurice LING wrote:
> > I've just read PEP 262 last night and finds that it does more or less
> > describes what I have in mind. However, I am not sure if there is every
> > need for such a descriptive database file or something slimmer, like
> > Fink's .info files will suffice.
You should make yourself familiar with the latest metadata PEP 314 (specifies
metadata version 1.1). I'm also currently working on metadata version 1.2,
which allows specification of python version requirements, and external
software requirements.
> > Implementing the API specified in PEP 262 is desirable.
> >
> > What I am thinking is this,
> >
> > 1. when user specify a package to install, the package's .info file will
> > be looked up in 'pkginfo' directory (in PEP 262, it is the INSTALLDB
> > directory that holds all these .info files).
> >
> > 2. to the system, only 3 things are crucial: where to get the package?
> > what packages the package needs? how to install? These 3 things are the
> > real critical parts of .info file, the rest are information and metadata.
You might need to prompt the user for handling external dependencies here.
> > Please tell me what you think...
Sounds good to me. I'd like to see the PEP (261) updated with the latest state
of distutils (ie. metadata, the package index), your ideas and re-posted.
Richard
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
iD8DBQFCdVtqrGisBEHG6TARAsxFAJ9906N9GwODpcoNqEcNqXi37tcKiwCeMNu3
5dmJVm97e8ZrsjxHwAcbwnM=
=hhwI
-----END PGP SIGNATURE-----
From mauriceling at acm.org Mon May 2 12:15:58 2005
From: mauriceling at acm.org (Maurice Ling)
Date: Mon, 02 May 2005 20:15:58 +1000
Subject: [Catalog-sig] "implementation" of PEP 262 as an academic project
Message-ID: <4275FDDE.8010006@acm.org>
Thank you for your pointers, Richard.
So now there are
PEP 262
PEP 314
to take note of.
Are there any relevant PEPs towards this purpose? Please kindly tell me.
Thanks
Cheers
Maurice
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mauriceling.vcf
Type: text/x-vcard
Size: 334 bytes
Desc: not available
Url : http://mail.python.org/pipermail/catalog-sig/attachments/20050502/478cf37d/mauriceling.vcf
From richardjones at optusnet.com.au Tue May 3 09:02:05 2005
From: richardjones at optusnet.com.au (richardjones@optusnet.com.au)
Date: Tue, 03 May 2005 17:02:05 +1000
Subject: [Catalog-sig] (no subject)
Message-ID: <200505030702.j437258A025261@mail28.syd.optusnet.com.au>
An embedded and charset-unspecified text was scrubbed...
Name: not available
Url: http://mail.python.org/pipermail/catalog-sig/attachments/20050503/c89d7233/attachment.ksh
From ianb at colorstudy.com Tue May 3 09:54:17 2005
From: ianb at colorstudy.com (Ian Bicking)
Date: Tue, 03 May 2005 02:54:17 -0500
Subject: [Catalog-sig] (no subject)
In-Reply-To: <200505030702.j437258A025261@mail28.syd.optusnet.com.au>
References: <200505030702.j437258A025261@mail28.syd.optusnet.com.au>
Message-ID: <42772E29.90204@colorstudy.com>
richardjones at optusnet.com.au wrote:
> Another point that I'd like to make is that I believe we should avoid reimplementing
> RPM, DEB, fink, whatever. Or at least we should not ignore them. We can produce
> meta-data that supports those packaging formats. Thus a user has the opportunity to
> install Python software using the existing package management mechanisms installed
> in their system, rather than something new and independent just for Python. The
> latest PEP I've proposed adds some meta-data that makes the DEB-alike packagers more
> happy, and should also help out RPM packagers.
>
> Distutils already does a fairly good job of handling the actual installation of
> Python software -- it compiles things, can install data files, with a little extra
> effort other stuff can be done too. It generates RPMs, DEBs, Windows Installers.
> There's a desire from the Ubuntu packagers to add a doc_files option to the setup()
> args, but that's for another discussion.
>
> Anyway, I'm rambling. My point is clear though: I believe we should avoid developing
> Yet Another Packaging System just for the sake of it. And certainly we should play
> well with others where possible.
OTOH, there is a place for distutils installs even in the presence of
native packaging. Because Python doesn't have versioned imports, in
many situations (at least the kind of stuff I do) a system-wide
installation isn't appropriate. And most native packagers don't do well
at localized installations.
> I guess we need to ask what it is that our database of installed Python modules will
> achieve that the existing packaging systems don't.
Of course, I don't know what a database would achieve exactly in this
case either, except perhaps in the presence of versioned imports. Which
I guess Eggs provide, in a way. But then Eggs mix it up too, since
wouldn't be "installed" either, they just get put in the right place,
and so there's no chance for a database to recognize them.
--
Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org
From mauriceling at acm.org Tue May 3 11:20:37 2005
From: mauriceling at acm.org (Maurice Ling)
Date: Tue, 03 May 2005 19:20:37 +1000
Subject: [Catalog-sig] (no subject)
In-Reply-To: <42772E29.90204@colorstudy.com>
References: <200505030702.j437258A025261@mail28.syd.optusnet.com.au>
<42772E29.90204@colorstudy.com>
Message-ID: <42774265.6000006@acm.org>
Ian Bicking wrote:
> richardjones at optusnet.com.au wrote:
>
>> Another point that I'd like to make is that I believe we should avoid
>> reimplementing
>> RPM, DEB, fink, whatever. Or at least we should not ignore them. We
>> can produce
>> meta-data that supports those packaging formats. Thus a user has the
>> opportunity to
>> install Python software using the existing package management
>> mechanisms installed
>> in their system, rather than something new and independent just for
>> Python. The
>> latest PEP I've proposed adds some meta-data that makes the DEB-alike
>> packagers more
>> happy, and should also help out RPM packagers.
>>
>> Distutils already does a fairly good job of handling the actual
>> installation of
>> Python software -- it compiles things, can install data files, with a
>> little extra
>> effort other stuff can be done too. It generates RPMs, DEBs, Windows
>> Installers.
>> There's a desire from the Ubuntu packagers to add a doc_files option
>> to the setup()
>> args, but that's for another discussion.
>>
>> Anyway, I'm rambling. My point is clear though: I believe we should
>> avoid developing
>> Yet Another Packaging System just for the sake of it. And certainly
>> we should play
>> well with others where possible.
>
>
> OTOH, there is a place for distutils installs even in the presence of
> native packaging. Because Python doesn't have versioned imports, in
> many situations (at least the kind of stuff I do) a system-wide
> installation isn't appropriate. And most native packagers don't do
> well at localized installations.
>
>> I guess we need to ask what it is that our database of installed
>> Python modules will
>> achieve that the existing packaging systems don't.
>
>
> Of course, I don't know what a database would achieve exactly in this
> case either, except perhaps in the presence of versioned imports.
> Which I guess Eggs provide, in a way. But then Eggs mix it up too,
> since wouldn't be "installed" either, they just get put in the right
> place, and so there's no chance for a database to recognize them.
>
Hi,
Thank for all your pointers.
I see my proposed work to solve one very fundamental issue. (Please take
a look at the thread "bytecode non-backcompatibility" started in April
2005 in python-list)
Given that (1) there can be multiple versions of Python installed in a
system, (2) each version maintains their own site-packages directory and
(3) if C modules are installed, they are not compatible with other
versions of Python... Imagine a system administrator who had installed
50 libraries and their dependencies in site-package of Python 2.3 and
now has to do it for Python 2.4, can we make his life better?
Building on that, if say I distribute a software which requires 5
3rd-party libraries. Currently, I can only tell the user which libraries
are necessary and hopefully they don't get too frustrated (imagine if
each of the 5 libraries depends on 2 libraries each). If my work is
around (PEP 262 and 314), I can then ask the user to just install ONE
system and in my software distribution, calls that one system to install
all the necessary libraries.
I've not done anything substantial yet.... still feeling around my way.
Cheers
Maurice
--
Maurice Han Tong LING, BSc(Hons)(Biochem), AdvDipComp, CPT, SSN, FIFA,
MASBMB, MAMBIS, MACM
Doctor of Philosophy (Science) Candidate, The University of Melbourne
mobile: +61 4 22781753, +65 96669233
mailing address: Department of Zoology, The University of Melbourne
Royal Parade, Parkville, Victoria 3010, Australia
residence: 9/41 Dover Street, Flemington, Victoria 3031, Australia
resume: http://maurice.vodien.com/maurice_resume.pdf
www: http://www.geocities.com/beldin79/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mauriceling.vcf
Type: text/x-vcard
Size: 334 bytes
Desc: not available
Url : http://mail.python.org/pipermail/catalog-sig/attachments/20050503/2a6166cb/mauriceling.vcf
From richardjones at optushome.com.au Tue May 3 11:58:34 2005
From: richardjones at optushome.com.au (Richard Jones)
Date: Tue, 3 May 2005 19:58:34 +1000
Subject: [Catalog-sig] (no subject)
In-Reply-To: <42774265.6000006@acm.org>
References: <200505030702.j437258A025261@mail28.syd.optusnet.com.au>
<42772E29.90204@colorstudy.com> <42774265.6000006@acm.org>
Message-ID: <200505031958.37920.richardjones@optushome.com.au>
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On Tue, 3 May 2005 07:20 pm, Maurice Ling wrote:
> Given that (1) there can be multiple versions of Python installed in a
> system, (2) each version maintains their own site-packages directory and
> (3) if C modules are installed, they are not compatible with other
> versions of Python... Imagine a system administrator who had installed
> 50 libraries and their dependencies in site-package of Python 2.3 and
> now has to do it for Python 2.4, can we make his life better?
If you could solve that *just* for pure-python packages, you'd already be a
hero :)
More response later (probably tomorrow).
Richard
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
iD8DBQFCd0tNrGisBEHG6TARAqFFAJ9lDxeFNjVzGNfcFHnAbCtxbmdKHwCeK2t4
4mL4uGGw6CoWXht+x+NiWXU=
=TPNR
-----END PGP SIGNATURE-----
From martin at v.loewis.de Thu May 5 14:19:43 2005
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 05 May 2005 14:19:43 +0200
Subject: [Catalog-sig] (no subject)
In-Reply-To: <42774265.6000006@acm.org>
References: <200505030702.j437258A025261@mail28.syd.optusnet.com.au> <42772E29.90204@colorstudy.com>
<42774265.6000006@acm.org>
Message-ID: <427A0F5F.4080009@v.loewis.de>
Maurice Ling wrote:
> Given that (1) there can be multiple versions of Python installed in a
> system, (2) each version maintains their own site-packages directory and
> (3) if C modules are installed, they are not compatible with other
> versions of Python... Imagine a system administrator who had installed
> 50 libraries and their dependencies in site-package of Python 2.3 and
> now has to do it for Python 2.4, can we make his life better?
I'd like to point out that this is a task that the "native" package
format can solve. For example, in Debian, when I use Debian packages
to install the 50 libraries, the current Debian Python policy manages
to update all the libraries from Python 2.3 to Python 2.4, when
the "official" Debian Python version becomes 2.4.
This is achieved by providing, for a library "foo", several packages:
python-foo, python2.2-foo, python2.3-foo, python2.4-foo. As a user,
I install python-foo, which currently depends on python2.3-foo. When
/usr/bin/python becomes 2.4, a new version of python-foo will be
released which depends on python2.4-foo. Just by updating all packages
I have installed, it automatically installs python2.4-foo as well.
Of course, if I don't want to wait for Debian to make /usr/bin/python
Python 2.4, I easily upgrade all libraries myself:
dpkg --get-selections|grep python2.3-|sed s/2.3/2.4/|dpkg --set-selections
will arrange to select all 2.4 libraries for which I have 2.3
versions installed.
So on Debian, I don't need any further support from distutils.
Regards,
Martin
From mauriceling at acm.org Thu May 5 14:57:27 2005
From: mauriceling at acm.org (Maurice Ling)
Date: Thu, 05 May 2005 22:57:27 +1000
Subject: [Catalog-sig] implementing PEP 262 as academic work
In-Reply-To: <427A0F5F.4080009@v.loewis.de>
References: <200505030702.j437258A025261@mail28.syd.optusnet.com.au> <42772E29.90204@colorstudy.com>
<42774265.6000006@acm.org> <427A0F5F.4080009@v.loewis.de>
Message-ID: <427A1837.3060109@acm.org>
Thanks for your pointers, Martin.
From your point of view, I do agree that you do not need more
intervention from disutils. In fact, if the package maintainers so
desire, you do not even need disutils at all. So the question here is,
is PEP 262 really needed at all? I can see that Martin or anyone using
Debian may not need it. However, at where I am (MacOSX with Fink), I
need it.
What is the current plans now?
Cheers
Maurice
>Maurice Ling wrote:
>
>
>>Given that (1) there can be multiple versions of Python installed in a
>>system, (2) each version maintains their own site-packages directory and
>>(3) if C modules are installed, they are not compatible with other
>>versions of Python... Imagine a system administrator who had installed
>>50 libraries and their dependencies in site-package of Python 2.3 and
>>now has to do it for Python 2.4, can we make his life better?
>>
>>
>
>I'd like to point out that this is a task that the "native" package
>format can solve. For example, in Debian, when I use Debian packages
>to install the 50 libraries, the current Debian Python policy manages
>to update all the libraries from Python 2.3 to Python 2.4, when
>the "official" Debian Python version becomes 2.4.
>
>This is achieved by providing, for a library "foo", several packages:
>python-foo, python2.2-foo, python2.3-foo, python2.4-foo. As a user,
>I install python-foo, which currently depends on python2.3-foo. When
>/usr/bin/python becomes 2.4, a new version of python-foo will be
>released which depends on python2.4-foo. Just by updating all packages
>I have installed, it automatically installs python2.4-foo as well.
>
>Of course, if I don't want to wait for Debian to make /usr/bin/python
>Python 2.4, I easily upgrade all libraries myself:
>
>dpkg --get-selections|grep python2.3-|sed s/2.3/2.4/|dpkg --set-selections
>
>will arrange to select all 2.4 libraries for which I have 2.3
>versions installed.
>
>So on Debian, I don't need any further support from distutils.
>
>Regards,
>Martin
>
>
>
--
Maurice Han Tong LING, BSc(Hons)(Biochem), AdvDipComp, CPT, SSN, FIFA,
MASBMB, MAMBIS, MACM
Doctor of Philosophy (Science) Candidate, The University of Melbourne
mobile: +61 4 22781753, +65 96669233
mailing address: Department of Zoology, The University of Melbourne
Royal Parade, Parkville, Victoria 3010, Australia
residence: 9/41 Dover Street, Flemington, Victoria 3031, Australia
resume: http://maurice.vodien.com/maurice_resume.pdf
www: http://www.geocities.com/beldin79/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mauriceling.vcf
Type: text/x-vcard
Size: 334 bytes
Desc: not available
Url : http://mail.python.org/pipermail/catalog-sig/attachments/20050505/83cd7984/mauriceling.vcf
From bob at redivi.com Thu May 5 16:04:43 2005
From: bob at redivi.com (Bob Ippolito)
Date: Thu, 5 May 2005 10:04:43 -0400
Subject: [Catalog-sig] implementing PEP 262 as academic work
In-Reply-To: <427A1837.3060109@acm.org>
References: <200505030702.j437258A025261@mail28.syd.optusnet.com.au> <42772E29.90204@colorstudy.com>
<42774265.6000006@acm.org> <427A0F5F.4080009@v.loewis.de>
<427A1837.3060109@acm.org>
Message-ID: <7D0BC443-0438-43D0-835B-EE9B6B5BD155@redivi.com>
On May 5, 2005, at 8:57 AM, Maurice Ling wrote:
>> Maurice Ling wrote:
>>
>>
>>> Given that (1) there can be multiple versions of Python installed
>>> in a
>>> system, (2) each version maintains their own site-packages
>>> directory and
>>> (3) if C modules are installed, they are not compatible with other
>>> versions of Python... Imagine a system administrator who had
>>> installed
>>> 50 libraries and their dependencies in site-package of Python 2.3
>>> and
>>> now has to do it for Python 2.4, can we make his life better?
>>
>> I'd like to point out that this is a task that the "native" package
>> format can solve. For example, in Debian, when I use Debian packages
>> to install the 50 libraries, the current Debian Python policy manages
>> to update all the libraries from Python 2.3 to Python 2.4, when
>> the "official" Debian Python version becomes 2.4.
>>
> From your point of view, I do agree that you do not need more
> intervention from disutils. In fact, if the package maintainers so
> desire, you do not even need disutils at all. So the question here
> is, is PEP 262 really needed at all? I can see that Martin or
> anyone using Debian may not need it. However, at where I am (MacOSX
> with Fink), I need it.
>
Since Fink uses the same software that Debian does, why shouldn't the
same upgrade pathway work? What did they screw up? They've screwed
up other things, so I don't use or trust Fink, but I'm just curious...
When using Mac OS X without the help of a "trusted third party" like
Darwinports or Fink, or when using Windows (where there isn't such an
option), there is a most definitely a use case for PEP 262.
Additionally, the Python world generally moves a hell of a lot faster
than the Debian world, so there's probably some reasons to have PEP
262 even in such an "ideal" world where dependencies (almost) "just
work" without Python's help.
-bob
From ianb at colorstudy.com Thu May 5 19:38:04 2005
From: ianb at colorstudy.com (Ian Bicking)
Date: Thu, 05 May 2005 12:38:04 -0500
Subject: [Catalog-sig] (no subject)
In-Reply-To: <427A0F5F.4080009@v.loewis.de>
References: <200505030702.j437258A025261@mail28.syd.optusnet.com.au> <42772E29.90204@colorstudy.com>
<42774265.6000006@acm.org> <427A0F5F.4080009@v.loewis.de>
Message-ID: <427A59FC.1000302@colorstudy.com>
Martin v. L?wis wrote:
> Maurice Ling wrote:
>
>>Given that (1) there can be multiple versions of Python installed in a
>>system, (2) each version maintains their own site-packages directory and
>>(3) if C modules are installed, they are not compatible with other
>>versions of Python... Imagine a system administrator who had installed
>>50 libraries and their dependencies in site-package of Python 2.3 and
>>now has to do it for Python 2.4, can we make his life better?
>
>
> I'd like to point out that this is a task that the "native" package
> format can solve. For example, in Debian, when I use Debian packages
> to install the 50 libraries, the current Debian Python policy manages
> to update all the libraries from Python 2.3 to Python 2.4, when
> the "official" Debian Python version becomes 2.4.
Just as a reminder, what Maurice talked about actually doesn't relate to
my initial concern. Or, maybe I'll call it my distutils-love, where
distutils does something currently that system packagers don't do. I'm
not so worried about per-python-version installation of packages -- for
the most part this has worked well for some time. It's the support of
multiple *library* versions that keeps me using distutils, and actually
keeps me from using native packages for most things, except those places
where there's already a centralized system (e.g., I don't install more
than one version of Postgres on a server, so I don't need more than one
version of psycopg). Or things that are stable and practically
equivalent to "standard" for me, like mxDateTime.
But for most libraries I'm very reluctant to install them globally; I
don't want to do monolithic upgrades of systems, I'd rather do
incremental upgrades of specific applications, and handle any large
upgrades through centralized tracking of the installations.
In the C world this is more-or-less solved through a system of symlinks,
e.g., libjpeg.so, libjpeg.so.9, libjpeg.so.9.1, etc. But we don't have
that for Python. So if I have one application that needs SQLObject 0.5,
and another that needs SQLObject 0.6, I just can't use any system
package manager.
At first I was thinking, well, that's because of my specific situation
-- multi-client host machines, applications that go quiet for extended
periods of time and shouldn't be disturbed, and my own tendency to use
lots of in-development libraries (my own and other people's). But as I
think about it, it seems like it applies to most situations. Subway,
for instance, is built against a specific version of SQLObject.
UltraGleeper is built against a different version (just to pick two
projects I know of off the top of my head). Well, technically they used
different package names for the different versions, but the general
problem should be fairly obvious. Making the installation robust and
easy for these kinds of things -- both applications and frameworks -- is
what I think we are trying to do. But as long as we have unversioned
package installation and imports, I think it's a bad idea to do
system-wide installation most packages.
--
Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org
From mauriceling at acm.org Fri May 6 01:36:28 2005
From: mauriceling at acm.org (Maurice Ling)
Date: Fri, 06 May 2005 09:36:28 +1000
Subject: [Catalog-sig] (no subject)
In-Reply-To: <DE61F324-23BE-4008-8437-B5ED09E3CE1A@redivi.com>
References: <200505030702.j437258A025261@mail28.syd.optusnet.com.au> <42772E29.90204@colorstudy.com>
<42774265.6000006@acm.org> <427A0F5F.4080009@v.loewis.de>
<427A59FC.1000302@colorstudy.com>
<DE61F324-23BE-4008-8437-B5ED09E3CE1A@redivi.com>
Message-ID: <427AADFC.9030100@acm.org>
Bob Ippolito wrote:
>>
>> Just as a reminder, what Maurice talked about actually doesn't
>> relate to
>> my initial concern. Or, maybe I'll call it my distutils-love, where
>> distutils does something currently that system packagers don't do. I'm
>> not so worried about per-python-version installation of packages -- for
>> the most part this has worked well for some time. It's the support of
>> multiple *library* versions that keeps me using distutils, and actually
>> keeps me from using native packages for most things, except those
>> places
>> where there's already a centralized system (e.g., I don't install more
>> than one version of Postgres on a server, so I don't need more than one
>> version of psycopg). Or things that are stable and practically
>> equivalent to "standard" for me, like mxDateTime.
>>
>> But for most libraries I'm very reluctant to install them globally; I
>> don't want to do monolithic upgrades of systems, I'd rather do
>> incremental upgrades of specific applications, and handle any large
>> upgrades through centralized tracking of the installations.
>>
>> In the C world this is more-or-less solved through a system of
>> symlinks,
>> e.g., libjpeg.so, libjpeg.so.9, libjpeg.so.9.1, etc. But we don't have
>> that for Python. So if I have one application that needs SQLObject
>> 0.5,
>> and another that needs SQLObject 0.6, I just can't use any system
>> package manager.
>>
>> At first I was thinking, well, that's because of my specific situation
>> -- multi-client host machines, applications that go quiet for
>> extended periods of time and shouldn't be disturbed, and my own
>> tendency to use
>> lots of in-development libraries (my own and other people's). But as I
>> think about it, it seems like it applies to most situations. Subway,
>> for instance, is built against a specific version of SQLObject.
>> UltraGleeper is built against a different version (just to pick two
>> projects I know of off the top of my head). Well, technically they
>> used
>> different package names for the different versions, but the general
>> problem should be fairly obvious. Making the installation robust and
>> easy for these kinds of things -- both applications and frameworks
>> -- is
>> what I think we are trying to do. But as long as we have unversioned
>> package installation and imports, I think it's a bad idea to do
>> system-wide installation most packages.
>
>
> In other words, Python has "DLL hell" and there isn't a damned thing
> that a system package manager can do about it other than say "sorry,
> you can't install FooBar and CheeseFactory at the same time, because
> FooBar needs BazWibble 1.0 and CheeseFactory needs BazWibble 2.0".
>
> Therefore, there needs to be some mechanism to install stuff to an
> "app-packages" because it's not always possible or appropriate to put
> everything in "site-packages".
>
> ... however, the referenced PEP does nothing to help this problem
> really.
>
Yes, I agree that PEP 262 doesn't help to solve local packages
(app-packages) as compared to global packages (site-packages). But it
can be extended into that direction. Given that different applications
may require different versions of a package, and disutils installs newer
versions of the package over older versions (I'm not sure on this one,
although the end results seems pretty much this way), there is a reason
why this feature is useful.
Thanks guys.
Cheers
Maurice
--
Maurice Han Tong LING, BSc(Hons)(Biochem), AdvDipComp, CPT, SSN, FIFA,
MASBMB, MAMBIS, MACM
Doctor of Philosophy (Science) Candidate, The University of Melbourne
mobile: +61 4 22781753, +65 96669233
mailing address: Department of Zoology, The University of Melbourne
Royal Parade, Parkville, Victoria 3010, Australia
residence: 9/41 Dover Street, Flemington, Victoria 3031, Australia
resume: http://maurice.vodien.com/maurice_resume.pdf
www: http://www.geocities.com/beldin79/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mauriceling.vcf
Type: text/x-vcard
Size: 334 bytes
Desc: not available
Url : http://mail.python.org/pipermail/catalog-sig/attachments/20050506/4623bf1d/mauriceling.vcf
From bob at redivi.com Thu May 5 20:37:42 2005
From: bob at redivi.com (Bob Ippolito)
Date: Thu, 5 May 2005 14:37:42 -0400
Subject: [Catalog-sig] (no subject)
In-Reply-To: <427A59FC.1000302@colorstudy.com>
References: <200505030702.j437258A025261@mail28.syd.optusnet.com.au> <42772E29.90204@colorstudy.com>
<42774265.6000006@acm.org> <427A0F5F.4080009@v.loewis.de>
<427A59FC.1000302@colorstudy.com>
Message-ID: <DE61F324-23BE-4008-8437-B5ED09E3CE1A@redivi.com>
On May 5, 2005, at 1:38 PM, Ian Bicking wrote:
> Martin v. L?wis wrote:
>
>> Maurice Ling wrote:
>>
>>
>>> Given that (1) there can be multiple versions of Python installed
>>> in a
>>> system, (2) each version maintains their own site-packages
>>> directory and
>>> (3) if C modules are installed, they are not compatible with other
>>> versions of Python... Imagine a system administrator who had
>>> installed
>>> 50 libraries and their dependencies in site-package of Python 2.3
>>> and
>>> now has to do it for Python 2.4, can we make his life better?
>>>
>>
>>
>> I'd like to point out that this is a task that the "native" package
>> format can solve. For example, in Debian, when I use Debian packages
>> to install the 50 libraries, the current Debian Python policy manages
>> to update all the libraries from Python 2.3 to Python 2.4, when
>> the "official" Debian Python version becomes 2.4.
>>
>
> Just as a reminder, what Maurice talked about actually doesn't
> relate to
> my initial concern. Or, maybe I'll call it my distutils-love, where
> distutils does something currently that system packagers don't do.
> I'm
> not so worried about per-python-version installation of packages --
> for
> the most part this has worked well for some time. It's the support of
> multiple *library* versions that keeps me using distutils, and
> actually
> keeps me from using native packages for most things, except those
> places
> where there's already a centralized system (e.g., I don't install more
> than one version of Postgres on a server, so I don't need more than
> one
> version of psycopg). Or things that are stable and practically
> equivalent to "standard" for me, like mxDateTime.
>
> But for most libraries I'm very reluctant to install them globally; I
> don't want to do monolithic upgrades of systems, I'd rather do
> incremental upgrades of specific applications, and handle any large
> upgrades through centralized tracking of the installations.
>
> In the C world this is more-or-less solved through a system of
> symlinks,
> e.g., libjpeg.so, libjpeg.so.9, libjpeg.so.9.1, etc. But we don't
> have
> that for Python. So if I have one application that needs SQLObject
> 0.5,
> and another that needs SQLObject 0.6, I just can't use any system
> package manager.
>
> At first I was thinking, well, that's because of my specific situation
> -- multi-client host machines, applications that go quiet for
> extended
> periods of time and shouldn't be disturbed, and my own tendency to use
> lots of in-development libraries (my own and other people's). But
> as I
> think about it, it seems like it applies to most situations. Subway,
> for instance, is built against a specific version of SQLObject.
> UltraGleeper is built against a different version (just to pick two
> projects I know of off the top of my head). Well, technically they
> used
> different package names for the different versions, but the general
> problem should be fairly obvious. Making the installation robust and
> easy for these kinds of things -- both applications and frameworks
> -- is
> what I think we are trying to do. But as long as we have unversioned
> package installation and imports, I think it's a bad idea to do
> system-wide installation most packages.
In other words, Python has "DLL hell" and there isn't a damned thing
that a system package manager can do about it other than say "sorry,
you can't install FooBar and CheeseFactory at the same time, because
FooBar needs BazWibble 1.0 and CheeseFactory needs BazWibble 2.0".
Therefore, there needs to be some mechanism to install stuff to an
"app-packages" because it's not always possible or appropriate to put
everything in "site-packages".
... however, the referenced PEP does nothing to help this problem
really.
-bob
From janc13 at gmail.com Fri May 6 19:49:30 2005
From: janc13 at gmail.com (JanC)
Date: Fri, 6 May 2005 19:49:30 +0200
Subject: [Catalog-sig] (no subject)
In-Reply-To: <984838bf050506104419b3ce3@mail.gmail.com>
References: <200505030702.j437258A025261@mail28.syd.optusnet.com.au>
<42772E29.90204@colorstudy.com> <42774265.6000006@acm.org>
<427A0F5F.4080009@v.loewis.de> <427A59FC.1000302@colorstudy.com>
<DE61F324-23BE-4008-8437-B5ED09E3CE1A@redivi.com>
<984838bf050506104419b3ce3@mail.gmail.com>
Message-ID: <984838bf0505061049504a790c@mail.gmail.com>
Oops, sorry for the empty mail... :-/.
On 5/6/05, JanC <janc13 at gmail.com> wrote:
> On 5/5/05, Bob Ippolito <bob at redivi.com> wrote:
> >
> > In other words, Python has "DLL hell" and there isn't a damned thing
> > that a system package manager can do about it other than say "sorry,
> > you can't install FooBar and CheeseFactory at the same time, because
> > FooBar needs BazWibble 1.0 and CheeseFactory needs BazWibble 2.0".
> >
> > Therefore, there needs to be some mechanism to install stuff to an
> > "app-packages" because it's not always possible or appropriate to put
> > everything in "site-packages".
wxPython has solved this more or less with a mechanism that people
allows to use the default version or a specific version.if they want,
with something like this:
import wxversion
wxversion.select("2.4")
import wx
--
JanC
From ianb at colorstudy.com Fri May 6 21:02:41 2005
From: ianb at colorstudy.com (Ian Bicking)
Date: Fri, 06 May 2005 14:02:41 -0500
Subject: [Catalog-sig] (no subject)
In-Reply-To: <984838bf0505061049504a790c@mail.gmail.com>
References: <200505030702.j437258A025261@mail28.syd.optusnet.com.au> <42772E29.90204@colorstudy.com>
<42774265.6000006@acm.org> <427A0F5F.4080009@v.loewis.de>
<427A59FC.1000302@colorstudy.com> <DE61F324-23BE-4008-8437-B5ED09E3CE1A@redivi.com> <984838bf050506104419b3ce3@mail.gmail.com>
<984838bf0505061049504a790c@mail.gmail.com>
Message-ID: <427BBF51.90405@colorstudy.com>
JanC wrote:
>>On 5/5/05, Bob Ippolito <bob at redivi.com> wrote:
>>
>>>In other words, Python has "DLL hell" and there isn't a damned thing
>>>that a system package manager can do about it other than say "sorry,
>>>you can't install FooBar and CheeseFactory at the same time, because
>>>FooBar needs BazWibble 1.0 and CheeseFactory needs BazWibble 2.0".
>>>
>>>Therefore, there needs to be some mechanism to install stuff to an
>>>"app-packages" because it's not always possible or appropriate to put
>>>everything in "site-packages".
>
>
> wxPython has solved this more or less with a mechanism that people
> allows to use the default version or a specific version.if they want,
> with something like this:
>
> import wxversion
> wxversion.select("2.4")
> import wx
Python Eggs also address this, like:
require("Twisted-Internet>=2.0")
import twisted.internet
More on Eggs:
http://peak.telecommunity.com/DevCenter/PythonEggs
They also aren't really "installed", except at runtime by adding them to
sys.path.
I've only played around with them a little, but it's an interesting way
to handle things. It feels like it resolves (through avoidance) a lot
of issues of centralized installation. But this moves things into the
scope of distutils-sig: http://www.python.org/sigs/distutils-sig/
--
Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org
From ianb at colorstudy.com Mon May 23 00:34:37 2005
From: ianb at colorstudy.com (Ian Bicking)
Date: Sun, 22 May 2005 17:34:37 -0500
Subject: [Catalog-sig] Back to RPC
Message-ID: <429108FD.7080405@colorstudy.com>
OK, I'm trying to make package installation easier for Paste, which
comes back to PyPI pretty quickly, of course.
I think all the information I really need is how to get an sdist
package, or a checkout -- right now it's important to be to be able to
install out of repositories, not just released versions.
So, I think all the data is there now. I might search package_urls for
the right release, packagetype 'sdist'. Though there isn't any place
for a repository URL, and that gets a little vague -- in theory you
really could have a per-version repository in the form of a branch. But
that's not very useful information. But maybe that can be disambiguated
with a URL "svn-trunk" or "cvs-trunk", which doesn't imply that the
repository is version-specific, even if the database schema requires it
be put there.
From there it's all client-side work to install the package, so it
doesn't involve PyPI. The only issue is giving URLs to packages that
don't have URLs. It would be a lot easier from my perspective if I
didn't have to go through the package author when URLs were missing,
maybe with notification going back to the owner so they can adjust their
setup.py file. But lets ignore that for a little while, though it's
also very important to me, as I'd like to use this stuff sooner rather
than later.
Anyway, summarizing:
* Agree and document release_urls.packagetype values. I'd like:
* sdist
* bdist
* bdist_dumb
* bdist_rpm
* bdist_wininst
* bdist_egg
* svn-tag
* svn-trunk
* cvs-trunk (is there any agreed-upon format for CVS URLs? What about
branches and tags?)
Also, these really need to be documented somewhere on the pypi site,
that's the only way agreeing on it means anything ;)
* Add some functions to rpc.py:
* package_releases(package_name): list of release versions, as strings
* package_stable_version(package_name): packages.stable_version
* package_urls(package_name, version): A list of {'url': url,
'packagetype': packagetype}
* package_data(package_name, version): A dictionary that basically
summarizes the releases table, plus release_classifiers.
* providing_packages(specifier): A list of (name, version) from
release_provides.
* requiring_packages(specifier): A list of (name, version) from
release_requires.
I don't entirely understand "specifier" and "release_obsoletes". Are
specifiers package names, or a slightly more abstract version of package
names? Or something more structured?
While it isn't important to me now, I think it would be good to change
rpc.search to take a single dictionary argument, which it would pass in
as the query spec. Probably the return value should also be cleaned up,
as store.query_packages returns a kind of odd data structure.
The adding URL thing is a bit more of a policy decision. Maybe we can
just add a form to request a role, which will email the owner and give
them a quick form (maybe just one link) to add the user.
If I can get CVS commit access I can start working on some of these; a
recent database dump would also be useful for testing.
Incidentally, I notice the PyPI page is quite slow. Thoughts? I think
I'm noticing that templates are loaded off disk for each request, which
be really slow. Is PyPI running under mod_python now?
--
Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org
From richardjones at optushome.com.au Mon May 23 01:45:51 2005
From: richardjones at optushome.com.au (Richard Jones)
Date: Mon, 23 May 2005 09:45:51 +1000
Subject: [Catalog-sig] Back to RPC
In-Reply-To: <429108FD.7080405@colorstudy.com>
References: <429108FD.7080405@colorstudy.com>
Message-ID: <200505230945.51529.richardjones@optushome.com.au>
[Barry - SVN question for you buried down below]
On Mon, 23 May 2005 08:34 am, Ian Bicking wrote:
> I think all the information I really need is how to get an sdist
> package, or a checkout -- right now it's important to be to be able to
> install out of repositories, not just released versions.
There's something that's not been on *my* radar :)
Do we really want this in PyPI's database? Related: will people really want to
*maintain* this stuff in PyPI's database?
> So, I think all the data is there now. I might search package_urls for
> the right release, packagetype 'sdist'. Though there isn't any place
> for a repository URL, and that gets a little vague -- in theory you
> really could have a per-version repository in the form of a branch. But
> that's not very useful information. But maybe that can be disambiguated
> with a URL "svn-trunk" or "cvs-trunk", which doesn't imply that the
> repository is version-specific, even if the database schema requires it
> be put there.
Well, as soon as you start talking SVN *and* CVS you talking about totally
different meta-data. For SVN, you've got just a URL (in my understanding of
SVN's branching/tagging) and for CVS you've got a ... well, it's not quite a
URL, and a tag name.
And of course, this ignores the plethora of other revision-control
repositories that people use.
So *perhaps* you just want to mandate SVN and just deal with losing all the
potential Sourceforge contributions? I'm not sure what it is you're actually
trying to do, so I couldn't say :)
> From there it's all client-side work to install the package, so it
> doesn't involve PyPI. The only issue is giving URLs to packages that
> don't have URLs. It would be a lot easier from my perspective if I
> didn't have to go through the package author when URLs were missing,
> maybe with notification going back to the owner so they can adjust their
> setup.py file. But lets ignore that for a little while, though it's
> also very important to me, as I'd like to use this stuff sooner rather
> than later.
URLs/uploaded files for all registered packages is something I don't think
we'll ever achieve.
> Anyway, summarizing:
>
> * Agree and document release_urls.packagetype values. I'd like:
> * sdist
> * bdist
> * bdist_dumb
> * bdist_rpm
> * bdist_wininst
> * bdist_egg
> * svn-tag
> * svn-trunk
> * cvs-trunk (is there any agreed-upon format for CVS URLs? What about
> branches and tags?)
>
> Also, these really need to be documented somewhere on the pypi site,
> that's the only way agreeing on it means anything ;)
They should be listed in the PEP (or non-PEP) design document you're writing
(hint hint <wink>).
> * Add some functions to rpc.py:
> * package_releases(package_name): list of release versions, as strings
> * package_stable_version(package_name): packages.stable_version
> * package_urls(package_name, version): A list of {'url': url,
> 'packagetype': packagetype}
> * package_data(package_name, version): A dictionary that basically
> summarizes the releases table, plus release_classifiers.
> * providing_packages(specifier): A list of (name, version) from
> release_provides.
> * requiring_packages(specifier): A list of (name, version) from
> release_requires.
Good list.
> I don't entirely understand "specifier" and "release_obsoletes". Are
> specifiers package names, or a slightly more abstract version of package
> names? Or something more structured?
See the PEP: http://www.python.org/peps/pep-0314.html
> While it isn't important to me now, I think it would be good to change
> rpc.search to take a single dictionary argument, which it would pass in
> as the query spec. Probably the return value should also be cleaned up,
> as store.query_packages returns a kind of odd data structure.
Sounds fine to me. As I understand it, the code there is more of a
proof-of-concept.
> The adding URL thing is a bit more of a policy decision. Maybe we can
> just add a form to request a role, which will email the owner and give
> them a quick form (maybe just one link) to add the user.
Er, so this is someone asking to have permission to add the URL on behalf of
the package owner?
Seems reasonable to me, I suppose.
> If I can get CVS commit access I can start working on some of these; a
> recent database dump would also be useful for testing.
PyPI's just moved to svn.python.org, but I couldn't say more than that 'cos I
don't know how it works. I've cc'ed Barry Warsaw in case he's not on this
list.
> Incidentally, I notice the PyPI page is quite slow. Thoughts? I think
> I'm noticing that templates are loaded off disk for each request, which
> be really slow. Is PyPI running under mod_python now?
It's being moved right now. AMK's in the final stages of apache config.
Something that concerns me about all this is the referencing of older versions
of software. I can't help this niggling feeling I have of people down the
track asking package developers to fix bugs in ancient branches of their
code. And people on both sides of the equation getting snarky about it.
Richard
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://mail.python.org/pipermail/catalog-sig/attachments/20050523/f287c438/attachment.pgp
From ianb at colorstudy.com Mon May 23 02:54:11 2005
From: ianb at colorstudy.com (Ian Bicking)
Date: Sun, 22 May 2005 19:54:11 -0500
Subject: [Catalog-sig] Back to RPC
In-Reply-To: <200505230945.51529.richardjones@optushome.com.au>
References: <429108FD.7080405@colorstudy.com>
<200505230945.51529.richardjones@optushome.com.au>
Message-ID: <429129B3.4040207@colorstudy.com>
Richard Jones wrote:
> [Barry - SVN question for you buried down below]
>
> On Mon, 23 May 2005 08:34 am, Ian Bicking wrote:
>
>>I think all the information I really need is how to get an sdist
>>package, or a checkout -- right now it's important to be to be able to
>>install out of repositories, not just released versions.
>
>
> There's something that's not been on *my* radar :)
>
> Do we really want this in PyPI's database? Related: will people really want to
> *maintain* this stuff in PyPI's database?
Well, it's a pretty stable URL for most projects, more stable than the
download URL. Also, if we have tools that use the data, they *have* to
be maintained, so they will be maintained ;) -- but it's also why I'm
concerned that these things are correctable regardless of the presence
or interest of the package "owner". There will be workarounds -- I'd
rather those happen in PyPI than to build those into external tools and
external annotations of the index.
>>So, I think all the data is there now. I might search package_urls for
>>the right release, packagetype 'sdist'. Though there isn't any place
>>for a repository URL, and that gets a little vague -- in theory you
>>really could have a per-version repository in the form of a branch. But
>>that's not very useful information. But maybe that can be disambiguated
>>with a URL "svn-trunk" or "cvs-trunk", which doesn't imply that the
>>repository is version-specific, even if the database schema requires it
>>be put there.
>
>
> Well, as soon as you start talking SVN *and* CVS you talking about totally
> different meta-data. For SVN, you've got just a URL (in my understanding of
> SVN's branching/tagging) and for CVS you've got a ... well, it's not quite a
> URL, and a tag name.
Yeah, it's too bad there aren't real URLs for CVS. You could do
pserver:hostname/path.../modulename, or something like that, but it's
just making it up.
> And of course, this ignores the plethora of other revision-control
> repositories that people use.
Yes, but it's a process. Maybe darcs-trunk could be added, and so on.
Right now I'm actually only concerned about svn, because all the
projects I care about installing automatically are in svn. I wouldn't
be implementing anything else in my own client libraries.
It's a problem for me, because I'm starting to maintain or be involved
with several very interrelated projects that are most useful when used
in concert, but have valid independent existances. So if I'm working
with any one of them outside of a released version, there's a good
chance all of them need to be installed from checkouts. (Incidentally,
I'm not expecting to install any of these globally; I've pretty much
given up on globally installing any but a handful of non-stdlib packages)
> So *perhaps* you just want to mandate SVN and just deal with losing all the
> potential Sourceforge contributions? I'm not sure what it is you're actually
> trying to do, so I couldn't say :)
Sure. It's a process; if there's demand (and code), then there's
nothing stopping CVS repositories from having their access encoded into
a single URI-like string. There's a lot of projects still in CVS, but
not a lot of the immature projects where installing non-released
versions is particularly useful.
>>From there it's all client-side work to install the package, so it
>>doesn't involve PyPI. The only issue is giving URLs to packages that
>>don't have URLs. It would be a lot easier from my perspective if I
>>didn't have to go through the package author when URLs were missing,
>>maybe with notification going back to the owner so they can adjust their
>>setup.py file. But lets ignore that for a little while, though it's
>>also very important to me, as I'd like to use this stuff sooner rather
>>than later.
>
>
> URLs/uploaded files for all registered packages is something I don't think
> we'll ever achieve.
If we make it matter, that will help a lot.
>>Anyway, summarizing:
>>
>>* Agree and document release_urls.packagetype values. I'd like:
>> * sdist
>> * bdist
>> * bdist_dumb
>> * bdist_rpm
>> * bdist_wininst
>> * bdist_egg
>> * svn-tag
>> * svn-trunk
>> * cvs-trunk (is there any agreed-upon format for CVS URLs? What about
>> branches and tags?)
>>
>>Also, these really need to be documented somewhere on the pypi site,
>>that's the only way agreeing on it means anything ;)
>
>
> They should be listed in the PEP (or non-PEP) design document you're writing
> (hint hint <wink>).
Oh, sure, I guess I can do that. Does the PEP process really add much
here, or would a document be sufficient? The presence of PyPI
documentation in the PEP index is actually a bit unhelpful, compared to
documentation sitting right in PyPI (and maybe besides distutil-related
documentation).
>>* Add some functions to rpc.py:
>> * package_releases(package_name): list of release versions, as strings
>> * package_stable_version(package_name): packages.stable_version
>> * package_urls(package_name, version): A list of {'url': url,
>> 'packagetype': packagetype}
>> * package_data(package_name, version): A dictionary that basically
>> summarizes the releases table, plus release_classifiers.
>> * providing_packages(specifier): A list of (name, version) from
>> release_provides.
>> * requiring_packages(specifier): A list of (name, version) from
>> release_requires.
>
>
> Good list.
>
>
>
>>I don't entirely understand "specifier" and "release_obsoletes". Are
>>specifiers package names, or a slightly more abstract version of package
>>names? Or something more structured?
>
>
> See the PEP: http://www.python.org/peps/pep-0314.html
OK. I won't plan to do much with this information right now, but maybe
if there's easy access to the information someone will take up the cause.
>>While it isn't important to me now, I think it would be good to change
>>rpc.search to take a single dictionary argument, which it would pass in
>>as the query spec. Probably the return value should also be cleaned up,
>>as store.query_packages returns a kind of odd data structure.
>
>
> Sounds fine to me. As I understand it, the code there is more of a
> proof-of-concept.
Certainly, there's not a useful list of functions currently.
>>The adding URL thing is a bit more of a policy decision. Maybe we can
>>just add a form to request a role, which will email the owner and give
>>them a quick form (maybe just one link) to add the user.
>
>
> Er, so this is someone asking to have permission to add the URL on behalf of
> the package owner?
>
> Seems reasonable to me, I suppose.
Yes, something like that. Maybe just building a contact form into the
package display (or linked off that) is sufficient, that would mail all
owners and maintainers of a package (with language in the form to
indicate that only metadata errors should be reported through that form).
Later, maybe something could be added to record these issues, let the
owners mark them "resolved", and put out whiney messages to owners if
they don't do resolve them, and maybe alert "editors" eventually. But
just a contact form is a start.
>>If I can get CVS commit access I can start working on some of these; a
>>recent database dump would also be useful for testing.
>
>
> PyPI's just moved to svn.python.org, but I couldn't say more than that 'cos I
> don't know how it works. I've cc'ed Barry Warsaw in case he's not on this
> list.
OK. Could I get CVS access too? (username: ianbicking)
>>Incidentally, I notice the PyPI page is quite slow. Thoughts? I think
>>I'm noticing that templates are loaded off disk for each request, which
>>be really slow. Is PyPI running under mod_python now?
>
>
> It's being moved right now. AMK's in the final stages of apache config.
OK, then the speed doesn't really mean anything, and if it's CGI there's
no useful caching of templates that can be done. But I can look at that
too.
> Something that concerns me about all this is the referencing of older versions
> of software. I can't help this niggling feeling I have of people down the
> track asking package developers to fix bugs in ancient branches of their
> code. And people on both sides of the equation getting snarky about it.
Well, you can't predict or solve all future problems ;) I assume most
of the client tools will heavily prefer the "stable version" of the package.
--
Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org
From martin at v.loewis.de Mon May 23 07:19:50 2005
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 23 May 2005 07:19:50 +0200
Subject: [Catalog-sig] Back to RPC
In-Reply-To: <429129B3.4040207@colorstudy.com>
References: <429108FD.7080405@colorstudy.com> <200505230945.51529.richardjones@optushome.com.au>
<429129B3.4040207@colorstudy.com>
Message-ID: <429167F6.7020400@v.loewis.de>
Ian Bicking wrote:
> Oh, sure, I guess I can do that. Does the PEP process really add much
> here, or would a document be sufficient?
It might be sufficient, provided it contains the same information that
a PEP contains. I.e. I would like to see a problem statement, and
a discussion of alternatives (based on the feedback that you get for
the initial document).
I find that your initial draft describes a very complex system, and
I'm concerned that it is too complex for most users. It seems to
be similar to the SF file release system (requiring me to describe
all sorts of meta-information with choices from a set of options
that don't help to describe *my* package), so I wonder whether this
change would make PyPI less usable.
Regards,
Martin
From ianb at colorstudy.com Mon May 23 08:29:59 2005
From: ianb at colorstudy.com (Ian Bicking)
Date: Mon, 23 May 2005 01:29:59 -0500
Subject: [Catalog-sig] Back to RPC
In-Reply-To: <429167F6.7020400@v.loewis.de>
References: <429108FD.7080405@colorstudy.com> <200505230945.51529.richardjones@optushome.com.au>
<429129B3.4040207@colorstudy.com> <429167F6.7020400@v.loewis.de>
Message-ID: <42917867.3000008@colorstudy.com>
Martin v. L?wis wrote:
> Ian Bicking wrote:
>
>>Oh, sure, I guess I can do that. Does the PEP process really add much
>>here, or would a document be sufficient?
>
>
> It might be sufficient, provided it contains the same information that
> a PEP contains. I.e. I would like to see a problem statement, and
> a discussion of alternatives (based on the feedback that you get for
> the initial document).
Sure, I'll try to do that. But like I said, it's something I'd like to
use very soon, and I was actually in the middle of implementing it
separately when I realized that it was closer to the scope of PyPI than
I at first thought. I'd at least like to have a read-only devel copy of
this stuff that I could experiment with. If it's in a branch or
whatever, that's fine. If I have to host my own copy of PyPI for a
while as we discuss the additions that'd even be fine. Whatever -- I
just want to move forward on this, and discuss the good and bad of
specific code; if it means I write code that's thrown away, that doesn't
really bother me.
OTOH, there's a lot of purely internal refactorings to PyPI that would
make many of these things easier while also making PyPI easier to
maintain, and forking PyPI to do apply those would be a total waste.
> I find that your initial draft describes a very complex system, and
> I'm concerned that it is too complex for most users. It seems to
> be similar to the SF file release system (requiring me to describe
> all sorts of meta-information with choices from a set of options
> that don't help to describe *my* package), so I wonder whether this
> change would make PyPI less usable.
I don't see many added complexities at all. It comes down to a couple
of things:
* Document what packagetype's are. Pretty much a no-brainer; what can
be wrong about documentation?
* I think those types should include svn-trunk. I can spell out use
cases if people like -- they have a lot to do with interdependent
packages that track each other's changes, and making that more usable
for package users.
* Some public XML-RPC methods. I think the ones I spelled out are
pretty obvious; not necessarily complete, but a proper start.
* Some resolution process for updating and correcting metadata, probably
just starting with a contact form. This is primarily about
PyPI-the-webapp, which doesn't really need to be that formalized
(compared to PyPI-the-package-database).
packagetype already exists. I don't know what the interface is to set
it, but it's there in the database. I haven't proposed any new fields
(though an optional "title" field for package URLs might be nice, kind
of like a title field to <link> tags).
And I don't think SF is a good example of anything. I can't imagine a
worse designed user interface. So it's not fair to use that as a
counter argument to anything ;)
--
Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org
From ianb at colorstudy.com Wed May 25 08:36:20 2005
From: ianb at colorstudy.com (Ian Bicking)
Date: Wed, 25 May 2005 01:36:20 -0500
Subject: [Catalog-sig] PyPI enhancement doc
Message-ID: <42941CE4.7070206@colorstudy.com>
Here's my first go at a document about the enhancements I asked about
earlier...
Package Types
-------------
PyPI includes this table::
CREATE TABLE release_urls (
name TEXT,
version TEXT,
url TEXT,
packagetype TEXT,
FOREIGN KEY (name, version) REFERENCES releases (name, version)
);
The ``packagetype`` column has no particular type or constraint. I
suggest that it contain these types taken from ``distutils``:
sdist:
a source distribution (tarball, zip file, etc.)
bdist:
a built (binary) distribution
bdist_dumb:
a "dumb" built distribution
bdist_rpm:
an RPM distribution
bdist_wininst:
an executable installer for MS Windows
bdist_egg:
a Python Egg
In addition, I would like these possible values:
svn_trunk:
the URL of a subversion repository; a setup.py file should be
found directly under this URL. This URL will not necessarily
change for different revisions of the package.
svn_tag:
the URL of a subversion repository. A branch or tag for this
specific version.
Issues
~~~~~~
The binary distributions are all potentially platform specific, but
(except for Windows) the platform isn't encoded into the packagetype.
Should packagetype be extended to include information like Python
version? Should it have the same fields as the release_files table?
Potentially even more information is required to fully describe the
package type.
Another change would be to refactor the database, so that
release_files didn't have python_version, packagetype, or md5_digest
fields, but rather a third table (referenced by both release_files and
package_urls) reference this package-metadata field.
Potentially the last portion of the URL is itself self-describing.
The files produced by the bdist* methods generally are, and as long as
the URL doesn't obscure that filename it can be automatically
determined by clients.
The sdist type does not indicate what kind of archive is used -- most
notable, both zip and tar are possible. (Does sdist create zip files
on its own?) However, I think the client can determine the proper way
to unpack the file after downloading; this aspect of files is
self-describing.
An svn trunk is not a package file. However, it is largely equivalent
to an sdist file, though it must be downloaded in a different way;
actually performing the download is a client concern, so it doesn't
really effect this.
XML-RPC methods
---------------
This is an initial list of proposed methods for PyPI to support:
package_releases(package_name):
returns list of release versions, as strings, e.g., ['0.1', '0.2b',
'0.2'], in chronological order.
package_stable_version(package_name):
returns packages.stable_version; the current stable version of the
package. E.g., the string '0.3'
package_urls(package_name, version):
A list of {'url': url, 'packagetype': packagetype}, like [{'url':
'http://svn.pythonpaste.org/Paste/trunk', 'packagetype':
'svn_trunk'}, {'url': 'http://pythonpaste.org/Paste-0.1.tar.gz',
'packagetype': 'sdist'}]
package_data(package_name, version):
A dictionary that summarizes the releases table, plus
release_classifiers. E.g.:
{'name': 'OpenRelease',
'version': '0.1.2',
'author': 'Richard Harris',
'author_email': 'goosequill at users.sourceforge.net',
'maintainer': '',
'maintainer_email': '',
'homepage': 'http://open-release.sourceforge.net',
'download_url':
'http://prdownloads.sourceforge.net/projects/open-release/OpenRelease-0.1.2.tar.gz',
'description': """OpenRelease is a Python module which
automates the packaging, release and announcement of Open Source
software. The pack class creates packages, which are defined by packer
classes, manages versioning, and brings up your notes and changelog in
an editor. The release class uploads the package to SourceForge,
releases it through QRS, announces it on freshmeat and (if appropriate)
on pypi.""",
'license': 'GNU General Public License',
'platform': 'any',
'classifiers': [
'Development Status :: 4 - Beta',
'Environment :: Console',
'Intended Audience :: Developers',
'License :: OSI Approved :: GNU General Public License (GPL)',
'Natural Language :: English',
'Operating System :: OS Independent',
'Programming Language :: Python',
'Topic :: Software Development'],
'summary': '',
'description_html': '',
'keywords': '',
}
All keys are required. None/NULL is translated to ''. Open
issues: will emails be obscured? Is keywords turned into a list?
search(field_specifiers, [operator='and']):
field_specifiers is a dictionary of {fieldname: searchvalue}.
Returns a list like [(name, version)] of matching non-hidden
records. The search values are case-insensitive and match any
substring. The second argument indicates if all the field
specifiers are ANDed or ORed together. The value defaults to
'and' and is case-insensitive.
I'm a little soft on these, since I don't know if specifiers and the
necessary metadata is really ready:
providing_packages(specifier):
A list of (name, version) from release_provides. E.g.,
providing_packages('PageTemplate>=1.0') == [('zpt', '1.0')]. This
will only return non-hidden packages.
requiring_packages(specifier):
A list of (name, version) from release_requires.
Web UI Enhancements
-------------------
Search should be available with a single field, that searches all of
package name, summary, description/description_html, keywords. This
should be on the front page.
There should be a contact form attached to each package that will be
emailed to any people on record as owners or maintainers, for
reporting bad or missing links or other incorrect data. Anything
submitted through that form will also be stored in the database, and
the owner should mark it "resolved" when the issue is corrected.
Until the issue is resolved, the content will show up on the package
page (so other people can see the comment). All links in such emails
should have rel="nofollow" (if HTML anchors are created at all). This
should be clearly marked as being for problem reports only; adding
package comments is another issue.
From exarkun at divmod.com Wed May 25 13:43:28 2005
From: exarkun at divmod.com (Jp Calderone)
Date: Wed, 25 May 2005 11:43:28 GMT
Subject: [Catalog-sig] PyPI enhancement doc
In-Reply-To: <42941CE4.7070206@colorstudy.com>
Message-ID: <20050525114328.559.209089629.divmod.quotient.9347@ohm>
On Wed, 25 May 2005 01:36:20 -0500, Ian Bicking <ianb at colorstudy.com> wrote:
>Here's my first go at a document about the enhancements I asked about
>earlier...
>
>
>Package Types
>-------------
>
>PyPI includes this table::
>
> [snip]
>
>In addition, I would like these possible values:
>
>svn_trunk:
> the URL of a subversion repository; a setup.py file should be
> found directly under this URL. This URL will not necessarily
> change for different revisions of the package.
>svn_tag:
> the URL of a subversion repository. A branch or tag for this
> specific version.
>
Why not just svn_url, taking on values like "svn://svn.twistedmatrix.com/svn/Twisted/tags/2.0.1/@13844" or "svn://svn.twistedmatrix.com/svn/Twisted/trunk/@HEAD"?
Jp
From barry at python.org Wed May 25 13:49:18 2005
From: barry at python.org (Barry Warsaw)
Date: Wed, 25 May 2005 07:49:18 -0400
Subject: [Catalog-sig] Back to RPC
In-Reply-To: <200505230945.51529.richardjones@optushome.com.au>
References: <429108FD.7080405@colorstudy.com>
<200505230945.51529.richardjones@optushome.com.au>
Message-ID: <1117021758.16104.675.camel@presto.wooz.org>
On Sun, 2005-05-22 at 19:45, Richard Jones wrote:
> > If I can get CVS commit access I can start working on some of these; a
> > recent database dump would also be useful for testing.
>
> PyPI's just moved to svn.python.org, but I couldn't say more than that 'cos I
> don't know how it works. I've cc'ed Barry Warsaw in case he's not on this
> list.
(I'm not on this list.)
We're working (slowly) on migrating all the python.org services that are
currently under CVS to be controlled by Subversion. We've started with
the various system files for the two new machines donated by XS4ALL, and
Andrew has been using the same setup to configure the planet.python.org
stuff.
We're still working out the details of how access will work for www
content updates, i.e. whether it will be svn+ssh or https, or both, and
how the site changes will get pushed out, etc. Specifically for
cheeseshop, Andrew's been talking lately about how to set that up for
efficient mirroring.
Unfortunately, I've been too busy with other things to spend much time
on this, but Andrew's been doing a lot lately.
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 307 bytes
Desc: This is a digitally signed message part
Url : http://mail.python.org/pipermail/catalog-sig/attachments/20050525/cfa5fe27/attachment.pgp
From ianb at colorstudy.com Wed May 25 17:17:34 2005
From: ianb at colorstudy.com (Ian Bicking)
Date: Wed, 25 May 2005 10:17:34 -0500
Subject: [Catalog-sig] PyPI enhancement doc
In-Reply-To: <20050525114328.559.209089629.divmod.quotient.9347@ohm>
References: <20050525114328.559.209089629.divmod.quotient.9347@ohm>
Message-ID: <4294970E.1020502@colorstudy.com>
Jp Calderone wrote:
> Why not just svn_url, taking on values like
> "svn://svn.twistedmatrix.com/svn/Twisted/tags/2.0.1/@13844" or
> "svn://svn.twistedmatrix.com/svn/Twisted/trunk/@HEAD"?
I hadn't seen the @ syntax before. Does that work with svn natively,
for all the protocols svn supports?
Also, I don't really see a way to distinguish between the two. And
/tags/2.0.1/@HEAD seems like a more appropriate URL for the tag, since
the directory (not the revision number) indicates the release; in theory
(or by convention) the revision and the tag are identical in svn, but a
branch could have updates to HEAD that actually apply to the given revision.
--
Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org
From martin at v.loewis.de Wed May 25 23:02:10 2005
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 25 May 2005 23:02:10 +0200
Subject: [Catalog-sig] PyPI enhancement doc
In-Reply-To: <4294970E.1020502@colorstudy.com>
References: <20050525114328.559.209089629.divmod.quotient.9347@ohm>
<4294970E.1020502@colorstudy.com>
Message-ID: <4294E7D2.3040908@v.loewis.de>
Ian Bicking wrote:
> I hadn't seen the @ syntax before. Does that work with svn natively,
> for all the protocols svn supports?
AFAICT, it does not support it at all. I can't see where Jp got this
syntax from.
Regards,
Martin
From martin at v.loewis.de Wed May 25 23:23:37 2005
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 25 May 2005 23:23:37 +0200
Subject: [Catalog-sig] PyPI enhancement doc
In-Reply-To: <42941CE4.7070206@colorstudy.com>
References: <42941CE4.7070206@colorstudy.com>
Message-ID: <4294ECD9.5010804@v.loewis.de>
Ian Bicking wrote:
> The ``packagetype`` column has no particular type or constraint. I
> suggest that it contain these types taken from ``distutils``:
I see. I hadn't understood this earlier - documenting that seems
fine to me.
> The binary distributions are all potentially platform specific, but
> (except for Windows) the platform isn't encoded into the packagetype.
> Should packagetype be extended to include information like Python
> version?
It was excluded because it can be implied from the package file
(distutils makes sure the filenames don't conflict). upload.py
puts the platform in the comment.
Standardizing on platform names is a tricky issue: Is "SuSE" the
same platform as "Debian"? Is it the same platform as "SuSE Linux"?
> Another change would be to refactor the database, so that
> release_files didn't have python_version, packagetype, or md5_digest
> fields, but rather a third table (referenced by both release_files and
> package_urls) reference this package-metadata field.
What would be the advantage of doing so? If it is normalization:
I don't see any advantage in normalization at all.
> XML-RPC methods
> ---------------
>
> This is an initial list of proposed methods for PyPI to support:
I think this whole thing is highly debatable. Why XML-RPC? Are HTML
forms not good enough for you? Why not CORBA? etc.
It appears to be a design principle of PyPI to not use XML-RPC if
it can be avoided. I don't know where that principle comes from,
but I think it needs to be followed if indeed it is a principle.
If there is no such principle, I think all the current forms should
be duplicated to simplify the code in the distutils commands.
> Search should be available with a single field, that searches all of
> package name, summary, description/description_html, keywords. This
> should be on the front page.
See
http://sourceforge.net/tracker/index.php?func=detail&aid=989926&group_id=66150&atid=513503
In general, details of the UI and the database can be easily
integrated as long as they don't affect current users and
as long as they don't affect distutils or other programmatic
interfaces.
So contributions are welcome, although I understand this is
meant to be solved by Postgres full text search.
> There should be a contact form attached to each package that will be
> emailed to any people on record as owners or maintainers, for
> reporting bad or missing links or other incorrect data.
Again, contributions are welcome. We should be careful not to implement
another issue tracker here, though, so just sending an email might be
enough for the moment (or else a read-only roundup installation,
writable only to PyPI and the maintainer might be the right solution).
As soon as tracking is available, people will start using it to
report bugs on the actual software itself, which might defeat
the bug reporting channels the maintainers try to establish.
So perhaps a "report problems with this record to" field might
be sufficient, with the feedback form only available if the field
is not filled.
In any case, if you have something that works, please submit
a patch to sf.net/projects/pypi.
Regards,
Martin
From ianb at colorstudy.com Wed May 25 23:59:06 2005
From: ianb at colorstudy.com (Ian Bicking)
Date: Wed, 25 May 2005 16:59:06 -0500
Subject: [Catalog-sig] PyPI enhancement doc
In-Reply-To: <4294ECD9.5010804@v.loewis.de>
References: <42941CE4.7070206@colorstudy.com> <4294ECD9.5010804@v.loewis.de>
Message-ID: <4294F52A.9040904@colorstudy.com>
Incidentally, in case I forget, I should remind the group I have this
file from that last sprint that gives PyPI a WSGI frontend:
http://lonelylion.com/chipy/pypi/pypi_wsgi.py
It's pretty straight-forward. It might be useful in porting PyPI to
mod_python, or some other WSGI server. Incidentally with Paste you
could use the configuration:
publish_app = 'pypi.pypi_wsgi:pypi_app("/path/to/base")'
Martin v. L?wis wrote:
>>The binary distributions are all potentially platform specific, but
>>(except for Windows) the platform isn't encoded into the packagetype.
>>Should packagetype be extended to include information like Python
>>version?
>
>
> It was excluded because it can be implied from the package file
> (distutils makes sure the filenames don't conflict). upload.py
> puts the platform in the comment.
>
> Standardizing on platform names is a tricky issue: Is "SuSE" the
> same platform as "Debian"? Is it the same platform as "SuSE Linux"?
Yes, I agree, it's tricky. Lets say that the URL must include as its
basename the filename that distutils produces, so that this can be used
to recreate that filename:
base = url.split('/')[-1]
base = base.split('?')[0]
>>Another change would be to refactor the database, so that
>>release_files didn't have python_version, packagetype, or md5_digest
>>fields, but rather a third table (referenced by both release_files and
>>package_urls) reference this package-metadata field.
>
>
> What would be the advantage of doing so? If it is normalization:
> I don't see any advantage in normalization at all.
Because those fields can apply to other packages, even if they aren't
stored in PyPI's file repository. Just for normalization's sake I
wouldn't really care. And really, I don't actually care if the fields
are copied either.
But anyway, if we are expecting the filename to be self-describing then
this mostly doesn't matter. Except perhaps an optional md5_digest.
>>XML-RPC methods
>>---------------
>>
>>This is an initial list of proposed methods for PyPI to support:
>
>
> I think this whole thing is highly debatable. Why XML-RPC? Are HTML
> forms not good enough for you? Why not CORBA? etc.
XML-RPC is already in PyPI (and live!), in rpc.py. It just doesn't have
any useful functions right now.
> It appears to be a design principle of PyPI to not use XML-RPC if
> it can be avoided. I don't know where that principle comes from,
> but I think it needs to be followed if indeed it is a principle.
>
> If there is no such principle, I think all the current forms should
> be duplicated to simplify the code in the distutils commands.
You mean the (somewhat awkward, if I remember) construction of POST
requests in distutils? Yes, I think it would be nice to move those to
XML-RPC. That so many old distutils exist means the forms would have to
stay around for some time though :( Or does it; can we ask people to
update at least for submission? Eh, probably not for some time.
>>Search should be available with a single field, that searches all of
>>package name, summary, description/description_html, keywords. This
>>should be on the front page.
>
>
> See
>
> http://sourceforge.net/tracker/index.php?func=detail&aid=989926&group_id=66150&atid=513503
>
> In general, details of the UI and the database can be easily
> integrated as long as they don't affect current users and
> as long as they don't affect distutils or other programmatic
> interfaces.
>
> So contributions are welcome, although I understand this is
> meant to be solved by Postgres full text search.
OK. I expect there's some sysadmin issues to enabling full text search
in Postgres at this point as well.
>>There should be a contact form attached to each package that will be
>>emailed to any people on record as owners or maintainers, for
>>reporting bad or missing links or other incorrect data.
>
>
> Again, contributions are welcome. We should be careful not to implement
> another issue tracker here, though, so just sending an email might be
> enough for the moment (or else a read-only roundup installation,
> writable only to PyPI and the maintainer might be the right solution).
> As soon as tracking is available, people will start using it to
> report bugs on the actual software itself, which might defeat
> the bug reporting channels the maintainers try to establish.
>
> So perhaps a "report problems with this record to" field might
> be sufficient, with the feedback form only available if the field
> is not filled.
Is there a missing negation there? You mean, only allow reporting
problems if that field is filled in? I figure it's good enough to send
mail to everyone who has sufficient permission to resolve the problem,
another field shouldn't be necessary.
I'd like to display the message as well, so that readers can see message
until it is corrected; of course, leaving it forever just builds cruft.
Hence the resolved/unresolved thing.
> In any case, if you have something that works, please submit
> a patch to sf.net/projects/pypi.
Well, for now I'll try to do something with rpc.py, since there's not
enough code their to require refactoring (at which point patches start
to become more challenging).
--
Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org
From richardjones at optushome.com.au Thu May 26 02:15:45 2005
From: richardjones at optushome.com.au (Richard Jones)
Date: Thu, 26 May 2005 10:15:45 +1000
Subject: [Catalog-sig] PyPI development no longer on sf.net
Message-ID: <200505261015.45374.richardjones@optushome.com.au>
The PyPI CVS on sf.net is no longer the repository of PyPI code. It has been
moved to svn.python.org. If you need access to that repository, please
contact me and I'll see whether I can figure something out.
Richard
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://mail.python.org/pipermail/catalog-sig/attachments/20050526/e58eab10/attachment.pgp
From exarkun at divmod.com Thu May 26 04:01:47 2005
From: exarkun at divmod.com (Jp Calderone)
Date: Thu, 26 May 2005 02:01:47 GMT
Subject: [Catalog-sig] PyPI enhancement doc
In-Reply-To: <4294970E.1020502@colorstudy.com>
Message-ID: <20050526020147.559.2045491994.divmod.quotient.9868@ohm>
On Wed, 25 May 2005 10:17:34 -0500, Ian Bicking <ianb at colorstudy.com> wrote:
>Jp Calderone wrote:
>> Why not just svn_url, taking on values like
>> "svn://svn.twistedmatrix.com/svn/Twisted/tags/2.0.1/@13844" or
>> "svn://svn.twistedmatrix.com/svn/Twisted/trunk/@HEAD"?
>
>I hadn't seen the @ syntax before. Does that work with svn natively,
>for all the protocols svn supports?
>
>Also, I don't really see a way to distinguish between the two. And
>/tags/2.0.1/@HEAD seems like a more appropriate URL for the tag, since
>the directory (not the revision number) indicates the release; in theory
>(or by convention) the revision and the tag are identical in svn, but a
>branch could have updates to HEAD that actually apply to the given revision.
>
Hmmm, my mistake, I think. I thought it was general svn URL syntax, but it seems to be a special form just for the diff and merge commands.
Still, if you think the logically right thing to do is to always use @HEAD, then the whole @ bit can be omitted, and it could just be svn_url=".../trunk" and svn_url=".../tags/2.0.1/", since HEAD is the implied revision in those cases.
The reason I prefer @REV when looking at tags is that tags can be deleted. In such cases, trying to look at them @HEAD does not work, but trying to look at them at @REV, where REV is a revision at which they existed, does. But that's moot since it seems it won't work :)
Jp
From richardjones at optushome.com.au Thu May 26 04:50:15 2005
From: richardjones at optushome.com.au (Richard Jones)
Date: Thu, 26 May 2005 12:50:15 +1000
Subject: [Catalog-sig] Distutils projects page
Message-ID: <200505261250.16115.richardjones@optushome.com.au>
In an effort to try to understand what projects are out there trying to
improve distutils / packaging / make the world a better place, I've put up a
wiki page:
http://wiki.python.org/moin/DistutilsProjects
It contains all the projects I'm aware of. I'm not on the distutils SIG list,
so I might have missed some. I did scan the archive for this year, and didn't
see anything.
Please visit the page and edit it for correctness - or include your project if
it's not in there!
Richard
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://mail.python.org/pipermail/catalog-sig/attachments/20050526/cad222d5/attachment-0001.pgp
From richardjones at optushome.com.au Thu May 26 08:20:43 2005
From: richardjones at optushome.com.au (Richard Jones)
Date: Thu, 26 May 2005 16:20:43 +1000
Subject: [Catalog-sig] PyPI enhancement doc
In-Reply-To: <4294ECD9.5010804@v.loewis.de>
References: <42941CE4.7070206@colorstudy.com> <4294ECD9.5010804@v.loewis.de>
Message-ID: <200505261620.43695.richardjones@optushome.com.au>
On Thu, 26 May 2005 07:23 am, Martin v. L?wis wrote:
> It appears to be a design principle of PyPI to not use XML-RPC if
> it can be avoided. I don't know where that principle comes from,
> but I think it needs to be followed if indeed it is a principle.
>
> If there is no such principle, I think all the current forms should
> be duplicated to simplify the code in the distutils commands.
There is no such design principle :)
I just found it easier to reuse the existing HTTP/HTML form interface when
writing the "register" command. Now that people are requesting more complex
interfaces, I see no problem with using XML-RPC for them.
> > There should be a contact form attached to each package that will be
> > emailed to any people on record as owners or maintainers, for
> > reporting bad or missing links or other incorrect data.
>
> Again, contributions are welcome. We should be careful not to implement
> another issue tracker here, though, so just sending an email might be
> enough for the moment
I think that the email would be quite sufficient.
Richard
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://mail.python.org/pipermail/catalog-sig/attachments/20050526/dbb0d366/attachment.pgp
From richardjones at optushome.com.au Thu May 26 08:26:39 2005
From: richardjones at optushome.com.au (Richard Jones)
Date: Thu, 26 May 2005 16:26:39 +1000
Subject: [Catalog-sig] PyPI enhancement doc
In-Reply-To: <42941CE4.7070206@colorstudy.com>
References: <42941CE4.7070206@colorstudy.com>
Message-ID: <200505261626.40111.richardjones@optushome.com.au>
On Wed, 25 May 2005 04:36 pm, Ian Bicking wrote:
> The sdist type does not indicate what kind of archive is used -- most
> notable, both zip and tar are possible. (Does sdist create zip files
> on its own?)
As it turns out, this is the case. I didn't realise it until someone got all
hot under the collar about the unix-centric PyPI... no, I'm getting
sidetracked here. Anyway, yes, on Windows sdists are ZIP files. You can force
sdist to produce a particular format with the --format switch.
> However, I think the client can determine the proper way
> to unpack the file after downloading; this aspect of files is
> self-describing.
The correct meta-type should be supplied to the client too (by Apache, the
file server).
> An svn trunk is not a package file. However, it is largely equivalent
> to an sdist file, though it must be downloaded in a different way;
> actually performing the download is a client concern, so it doesn't
> really effect this.
Are you advocating that release_files somehow point to SVN repos? That doesn't
sound right, so I'll presume I've misunderstood you :)
> I'm a little soft on these, since I don't know if specifiers and the
> necessary metadata is really ready:
>
> providing_packages(specifier):
> A list of (name, version) from release_provides. E.g.,
> providing_packages('PageTemplate>=1.0') == [('zpt', '1.0')]. This
> will only return non-hidden packages.
>
> requiring_packages(specifier):
> A list of (name, version) from release_requires.
There's no code to actually perform this function, but the meta-data
capability is there. We probably won't see much in the way of this stuff
until Python2.5 becomes widespread in use (since additions to the meta-data
require releases of Python).
> Search should be available with a single field, that searches all of
> package name, summary, description/description_html, keywords. This
> should be on the front page.
A laudable goal. TSearch2 might be able to do this --- I don't know enough
about it. I've recently added Xapian support to Roundup --- and it's really,
really easy to use.
Richard
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://mail.python.org/pipermail/catalog-sig/attachments/20050526/885e79cc/attachment.pgp
From ianb at colorstudy.com Thu May 26 08:45:11 2005
From: ianb at colorstudy.com (Ian Bicking)
Date: Thu, 26 May 2005 01:45:11 -0500
Subject: [Catalog-sig] PyPI enhancement doc
In-Reply-To: <200505261626.40111.richardjones@optushome.com.au>
References: <42941CE4.7070206@colorstudy.com>
<200505261626.40111.richardjones@optushome.com.au>
Message-ID: <42957077.2020305@colorstudy.com>
Richard Jones wrote:
>>However, I think the client can determine the proper way
>>to unpack the file after downloading; this aspect of files is
>>self-describing.
>
>
> The correct meta-type should be supplied to the client too (by Apache, the
> file server).
I'd trust magic numbers first. But it doesn't matter -- this is all a
client issue, not a PyPI issue. In general I think if the client can do
it, then the client should do it.
>>An svn trunk is not a package file. However, it is largely equivalent
>>to an sdist file, though it must be downloaded in a different way;
>>actually performing the download is a client concern, so it doesn't
>>really effect this.
>
>
> Are you advocating that release_files somehow point to SVN repos? That doesn't
> sound right, so I'll presume I've misunderstood you :)
I'm advocating package_urls point to a SVN repository, so that you can
install from the svn trunk in addition to formal releases.
>>Search should be available with a single field, that searches all of
>>package name, summary, description/description_html, keywords. This
>>should be on the front page.
>
>
> A laudable goal. TSearch2 might be able to do this --- I don't know enough
> about it. I've recently added Xapian support to Roundup --- and it's really,
> really easy to use.
It's also quite possibly fast enough to simply use OR and LIKE; I don't
know if the data is large enough that it's an issue. I mean, we're just
now moving off CGI, right... I think full text indexing is a relatively
fine point of optimization in comparison ;)
--
Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org
From fdrake at gmail.com Thu May 26 19:04:15 2005
From: fdrake at gmail.com (Fred Drake)
Date: Thu, 26 May 2005 13:04:15 -0400
Subject: [Catalog-sig] Distutils projects page
In-Reply-To: <200505261250.16115.richardjones@optushome.com.au>
References: <200505261250.16115.richardjones@optushome.com.au>
Message-ID: <9cee7ab805052610046c0d20e0@mail.gmail.com>
On 5/25/05, Richard Jones <richardjones at optushome.com.au> wrote:
> In an effort to try to understand what projects are out there trying to
> improve distutils / packaging / make the world a better place, I've put up a
> wiki page:
>
> http://wiki.python.org/moin/DistutilsProjects
I couldn't edit this page; it looks like we're still having wiki
problems on python.org.
I was going to add a link to the zpkg pages:
http://www.zope.org/Members/fdrake/zpkgtools/
The pages there need to be updated; I'll see about getting that done
in the next couple of days. (I'm on travel now.)
-Fred
--
Fred L. Drake, Jr. <fdrake at gmail.com>
From pje at telecommunity.com Sun May 29 06:57:14 2005
From: pje at telecommunity.com (Phillip J. Eby)
Date: Sun, 29 May 2005 00:57:14 -0400
Subject: [Catalog-sig] EasyInstall - automated download/install/manage
Python packages
Message-ID: <5.1.1.6.0.20050529005708.02c461f0@mail.telecommunity.com>
Just a heads-up:
http://peak.telecommunity.com/DevCenter/EasyInstall
I would like to encourage package authors to try using EasyInstall to
download and build the packages they distribute, and to let me know if they
encounter any issues. I'd like to make EasyInstall able to build as many
Python packages "out of the box" as possible.
I've also been updating the Python Eggs documentation:
http://peak.telecommunity.com/DevCenter/PythonEggs
But there is still a fair amount to write about the runtime API, and as you
can see from the "Implementation Status" there are plenty of tasks still
"available" with respect to getting the zip-based runtime as robust as it
should be.
From kenneth.m.mcdonald at gmail.com Tue May 31 03:08:51 2005
From: kenneth.m.mcdonald at gmail.com (Kenneth McDonald)
Date: Mon, 30 May 2005 20:08:51 -0500
Subject: [Catalog-sig] Questions about Qt and wxWindows python packages
Message-ID: <eb45506d0505301808640edcf3@mail.gmail.com>
If this is not an appropriate newsgroup for this type of posting,
please let me know and (if possible) suggest an alternative. The
reading I've done of some of the back articles suggests that
python-relevant questions specific to any python package (whether
distrubuted via something like distutils or not) are acceptable here,
sorry if this is incorrect.
I'd like to start using Python for some GUI programming again. I'd
like to use Tk, but it seems to be dying a slow death, so it's
probably time to look at wxPython and PythonQt.
I'd appreciate an comments on the relevant merits of these two
libraries, both with respect to Python and more generally. I know
about the differences in licensing, that's not an issue. My most
specific concerns are ease of use (which includes how much in the way
of documentation or examples for using through Python are available,
and how easy the Python libs are to use in each case); availability of
a generally capable styled text widget, as one of the things I need to
build is an editor, which needs to be able to handle different fonts,
styles, tabs, and so forth; and ease of installation on a Macintosh,
since that is my preferred dev platform, and installing things made
for other UNIX variants can be a little wonky at times.
Many thanks for any feedback you can give.
Cheers,
Ken
From pje at telecommunity.com Tue May 31 03:14:09 2005
From: pje at telecommunity.com (Phillip J. Eby)
Date: Mon, 30 May 2005 21:14:09 -0400
Subject: [Catalog-sig] [Distutils] EasyInstall 0.3a3 released;
what about PyPI? (was Re: Initial auto-installation support)
In-Reply-To: <5.1.1.6.0.20050530173012.021f2340@mail.telecommunity.com>
References: <429B8133.5070007@colorstudy.com>
Message-ID: <5.1.1.6.0.20050530204156.021f27f8@mail.telecommunity.com>
>At 04:10 PM 5/30/2005 -0500, Ian Bicking wrote:
> > But besides
> >that, this should work now for any packages with a distutils install, so
> >long as those packages are reasonably well behaved. Hrm... except
> >setuptools 0.3a2 doesn't have SourceForge download support, but 0.3a3
> >does and I think PJE will release that soon.
0.3a3 is now released, with a new --build-dir option, sandboxing, more
package workarounds, SourceForge mirror selection, and "installation
reports". See:
http://peak.telecommunity.com/DevCenter/EasyInstall#release-notes-change-history
for more details.
I'm thinking that adding automatic package location via PyPI is probably
pretty doable now, by the way. My plan is to create a PackageFinder class
(subclassing AvailableDistributions) whose obtain() method searches for the
desired package on PyPI, keeping a cache of URLs it has already seen. (It
would also accept a callback argument that it would use to create Installer
objects when it needs to install packages.)
The command-line tool (easy_install.main) would create a PackageFinder with
an interactive installation callback, and in the main loop it would pass it
to each new Installer instance. The Installer would then use it whenever
it gets a non-file, non-URL command line option, and use it to resolve()
such requests.
The PackageFinder.obtain() method would go to the PyPI base URL followed by
the desired distribution name, e.g. 'http://www.python.org/pypi/SQLObject',
and then scrape the page to see if it is a multi-version page, or a
single-version page. If it's multi-version, it would scrape the version
links and select the highest-numbered version that meets all of your criteria.
Once it has a single-version page, it would look for a download URL, and
see if its filename is that of an archive (.egg, .tar, .tgz, etc.) or if
the URL is for subversion. If so, we assume it's the right thing and
invoke the callback to do the install.
If not, then we follow the link anyway, and scrape for links to archives,
checking versions when we get there if possible. If there's still nothing
suitable (or there was no download URL), we apply the same procedure to the
homepage URL.
This should suffice to make a significant number of packages available from
PyPI with autodownload, and packages with dependencies would also be
downloaded, built, and installed.
The hardest parts of this aren't in the screen-scraping per se; it's more
in the heuristics for evaluating whether a specific URL is suitable for
download. Many PyPI download URLs are of the form "foopackage-latest.tgz",
so it's not possible to determine a usable version number from this, unless
I special-case "latest" in the version parser -- which I guess I could do.
We also probably need some kind of heuristic to determine which URLs are
"better" to try, as we don't want to just run through the links in order.
Hm. You know, what if as an interim step we had the command-line tool just
launch a webbrowser pointing you to PyPI? Getting to a page for a suitable
version is easy, so we could then let the user find the right download URL
and then go back to paste it on the command line. That could be a nice
interim addition, although it isn't much of a solution for packages with a
lot of un-installed dependencies. You'd keep getting kicked back to the
web browser a lot, and more to the point you'd have to keep restarting the
tool. So, ultimately we really need a way to actually find the URLs.
There are going to have to be new options for the tool, too. Like a way to
set the PyPI URL to use, and a way to specify what sort of package
revisions are acceptable (e.g. no alphas, no betas, no snapshots).
From ianb at colorstudy.com Tue May 31 03:29:11 2005
From: ianb at colorstudy.com (Ian Bicking)
Date: Mon, 30 May 2005 20:29:11 -0500
Subject: [Catalog-sig] Questions about Qt and wxWindows python packages
In-Reply-To: <eb45506d0505301808640edcf3@mail.gmail.com>
References: <eb45506d0505301808640edcf3@mail.gmail.com>
Message-ID: <429BBDE7.4030108@colorstudy.com>
The list is really meant for questions about packaging. You can ask the
general python-list at python.org -- but first you should read the archive
(http://groups-beta.google.com/group/comp.lang.python), because this
exact question gets asked there often.
Kenneth McDonald wrote:
> If this is not an appropriate newsgroup for this type of posting,
> please let me know and (if possible) suggest an alternative. The
> reading I've done of some of the back articles suggests that
> python-relevant questions specific to any python package (whether
> distrubuted via something like distutils or not) are acceptable here,
> sorry if this is incorrect.
>
> I'd like to start using Python for some GUI programming again. I'd
> like to use Tk, but it seems to be dying a slow death, so it's
> probably time to look at wxPython and PythonQt.
>
> I'd appreciate an comments on the relevant merits of these two
> libraries, both with respect to Python and more generally. I know
> about the differences in licensing, that's not an issue. My most
> specific concerns are ease of use (which includes how much in the way
> of documentation or examples for using through Python are available,
> and how easy the Python libs are to use in each case); availability of
> a generally capable styled text widget, as one of the things I need to
> build is an editor, which needs to be able to handle different fonts,
> styles, tabs, and so forth; and ease of installation on a Macintosh,
> since that is my preferred dev platform, and installing things made
> for other UNIX variants can be a little wonky at times.
>
> Many thanks for any feedback you can give.
From ianb at colorstudy.com Tue May 31 03:47:28 2005
From: ianb at colorstudy.com (Ian Bicking)
Date: Mon, 30 May 2005 20:47:28 -0500
Subject: [Catalog-sig] [Distutils] EasyInstall 0.3a3 released;
what about PyPI? (was Re: Initial auto-installation support)
In-Reply-To: <5.1.1.6.0.20050530204156.021f27f8@mail.telecommunity.com>
References: <429B8133.5070007@colorstudy.com>
<5.1.1.6.0.20050530204156.021f27f8@mail.telecommunity.com>
Message-ID: <429BC230.5040709@colorstudy.com>
Phillip J. Eby wrote:
> The PackageFinder.obtain() method would go to the PyPI base URL followed
> by the desired distribution name, e.g.
> 'http://www.python.org/pypi/SQLObject', and then scrape the page to see
> if it is a multi-version page, or a single-version page. If it's
> multi-version, it would scrape the version links and select the
> highest-numbered version that meets all of your criteria.
Really this would be very easy to add to PyPI as a set of xml-rpc calls,
once the server move gets resolved. I could develop it now, except that
the last pypi database dump I have is from before the move to Postgres,
and there've been a number of changes since then, and I'd like some test
data. I also have a checkout from CVS, but the SF project no longer
lists CVS and svn.python.org doesn't have public access yet, so I can't
point you to the code.
But anyway, if someone on the new or old server can just run pg_dump on
that database and email me the results (or a url to those results) that
would be very helpful.
Getting the data without screenscraping won't instantly give us all the
necessary information. But it does contain good information about
available versions, what the active version is, and per-version download
URLs (which, if nothing else, could be compared against each other to
detect non-version-specific URLs).
> Hm. You know, what if as an interim step we had the command-line tool
> just launch a webbrowser pointing you to PyPI? Getting to a page for a
> suitable version is easy, so we could then let the user find the right
> download URL and then go back to paste it on the command line. That
> could be a nice interim addition, although it isn't much of a solution
> for packages with a lot of un-installed dependencies. You'd keep
> getting kicked back to the web browser a lot, and more to the point
> you'd have to keep restarting the tool. So, ultimately we really need a
> way to actually find the URLs.
That's not a very satisfying experience -- the person might as well just
download the file at that point.
Even with accurate data from PyPI, it's still likely there will be
multiple possible URLs. At that point, at least if you are going
through the command line, displaying all the URLs (numbered) and asking
the user would probably give the user enough information to choose.
--
Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org
From pje at telecommunity.com Tue May 31 06:32:05 2005
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue, 31 May 2005 00:32:05 -0400
Subject: [Catalog-sig] [Distutils] EasyInstall 0.3a3 released;
what about PyPI? (was Re: Initial auto-installation support)
In-Reply-To: <429BC230.5040709@colorstudy.com>
References: <5.1.1.6.0.20050530204156.021f27f8@mail.telecommunity.com>
<429B8133.5070007@colorstudy.com>
<5.1.1.6.0.20050530204156.021f27f8@mail.telecommunity.com>
Message-ID: <5.1.1.6.0.20050531001410.021f3aa0@mail.telecommunity.com>
At 08:47 PM 5/30/2005 -0500, Ian Bicking wrote:
>Getting the data without screenscraping won't instantly give us all the
>necessary information. But it does contain good information about
>available versions, what the active version is, and per-version download
>URLs (which, if nothing else, could be compared against each other to
>detect non-version-specific URLs).
Right; but none of that helps with the real problem (from EasyInstall's
perspective), which is that the current incarnation of PyPI doesn't list
multiple download URLs for a single release of a specific package. For
example, when I release PEAK or PyProtocols, I've been releasing sdists (in
two formats) plus a bdist_wininst -- and in the future I'll probably drop
the bdist_wininst in favor of eggs. But I can't put any of that info on
PyPI, so I just link to my downloads directory - as do 25% of the packages
I surveyed in a random sampling last week.
In order to get at packages like those, a flexible screen scraper is a
must. I agree that PyPI should have better handling of download URLs, but
I'm in a lot better position to improve EasyInstall than PyPI.
>>Hm. You know, what if as an interim step we had the command-line tool
>>just launch a webbrowser pointing you to PyPI? Getting to a page for a
>>suitable version is easy, so we could then let the user find the right
>>download URL and then go back to paste it on the command line. That
>>could be a nice interim addition, although it isn't much of a solution
>>for packages with a lot of un-installed dependencies. You'd keep getting
>>kicked back to the web browser a lot, and more to the point you'd have to
>>keep restarting the tool. So, ultimately we really need a way to
>>actually find the URLs.
>
>That's not a very satisfying experience -- the person might as well just
>download the file at that point.
Tastes differ, I suppose. I'd just right-click the link to copy it, and
then alt-tab, ^R, space, ^K, space, shift-insert, ENTER. But then, I've
been downloading a lot of packages this weekend, so that sequence is
already in my muscle memory. :)
Hm. Maybe somebody could create a Firefox extension that runs EasyInstall
on a selected link. :)
>Even with accurate data from PyPI, it's still likely there will be
>multiple possible URLs. At that point, at least if you are going through
>the command line, displaying all the URLs (numbered) and asking the user
>would probably give the user enough information to choose.
In which case you might as well be back in the web browser. I'm fine with
options being available to fine-tune the selection process, but the
criteria can and should be mechanically processed.
After all unusable versions, platforms, and archive types are eliminated,
the prioritization should be in descending version order, with same version
archives sorted by archive type, eggs first, everything else
second. (Since the eggs don't need to be built.)
By the way, in all of this there's been no discussion about MD5 signatures
or code signing. That's probably because I don't know a whole lot about
that subject. :) But I'm certainly interested in hearing from those who do.