From richardjones at optushome.com.au Fri Jan 2 08:13:17 2009
From: richardjones at optushome.com.au (Richard Jones)
Date: Fri, 2 Jan 2009 18:13:17 +1100
Subject: [Catalog-sig] Application downloads hosted by PyPI?
Message-ID: <200901021813.17776.richardjones@optushome.com.au>
When I initially implemented PyPI file hosting it was specifically designed to
only handle files generated by distutils.
I'm now in a position where I'd personally like to upload a application (well,
a zip file of an application) to the index. I've solved the problem of
generating the application distribution files* but to get it to upload I had
to tell PyPI that it was an sdist, and include a PKG-INFO file (to pass the
basic test I put in place to make sure that sdist files being uploaded really
were sdist files).
Clearly this is not optimal as the file is not really an sdist ;)
I couldn't upload it as a "bdist" because PyPI understands that binary
distributions are Python-version-specific. My application distribution is not
version specific.
What do people think about adding a new file type allowed for upload of
"application" or similar?
Richard
* see my blog entry for more information:
http://www.mechanicalcat.net/richard/log/Python/Sane_Python_application_packaging__initial_solution
From jcea at jcea.es Mon Jan 5 18:03:29 2009
From: jcea at jcea.es (Jesus Cea)
Date: Mon, 05 Jan 2009 18:03:29 +0100
Subject: [Catalog-sig] Replication and security
Message-ID: <49623D61.70707@jcea.es>
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Currently setuptools allows to upload a PGP signature along the package,
to be able to check integrity and security. As far as I know, currently
"easy_install" doesn't check it. That is bad, but life sucks.
My problem now is with mirrors: How can anybody to validate files?.
Beside the possible PGP signatures of authors (a check that should be
integrated in "easy_install"), I would like PYPI main server (I guess it
would be the single point where people upload new packages; the mirrors
would be read-only) to digitally sign each uploaded package. This way,
easy_install can check any package downloaded from any mirror, because
PYPI public key would be a well known value.
I have code in python to digitally sign/verify signatures using ElGamal
algorithm. Any interest?
- --
Jesus Cea Avion _/_/ _/_/_/ _/_/_/
jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/
jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/_/_/_/
. _/_/ _/_/ _/_/ _/_/ _/_/
"Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/
"My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iQCVAwUBSWI9YZlgi5GaxT1NAQLDFAQAjKWWmi9h3E4RvEupi03oAy839iCe7AO5
1nAHs+0aeQbQwskcUSD1RVZ4xP/AeJ+Gva1rvJfr7Ho41FD9WEFO/ErnHyGhEnL3
QK30lXbosnIWoqRiwXijrKtYp+9/pyixuDt7bL8hQ6ZBzgsOnknHaLJhDsNK+AMf
KowdHXxsnPo=
=eTrH
-----END PGP SIGNATURE-----
From martin at v.loewis.de Mon Jan 5 18:42:55 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 05 Jan 2009 18:42:55 +0100
Subject: [Catalog-sig] Replication and security
In-Reply-To: <49623D61.70707@jcea.es>
References: <49623D61.70707@jcea.es>
Message-ID: <4962469F.6070505@v.loewis.de>
> I have code in python to digitally sign/verify signatures using ElGamal
> algorithm. Any interest?
I rather prefer standard PGP signatures (with whatever signature
algorithm the server key uses).
Regards,
Martin
From jcea at jcea.es Mon Jan 5 18:56:36 2009
From: jcea at jcea.es (Jesus Cea)
Date: Mon, 05 Jan 2009 18:56:36 +0100
Subject: [Catalog-sig] Replication and security
In-Reply-To: <4962469F.6070505@v.loewis.de>
References: <49623D61.70707@jcea.es> <4962469F.6070505@v.loewis.de>
Message-ID: <496249D4.9060400@jcea.es>
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Martin v. L?wis wrote:
>> I have code in python to digitally sign/verify signatures using ElGamal
>> algorithm. Any interest?
>
> I rather prefer standard PGP signatures (with whatever signature
> algorithm the server key uses).
Me too, but then you requires an OpenPGP implementation in Python or a
pgp/gpg program around, correctly configured, with the PYPI public key
installed, etc.
Instead, ElGamal signatures are verified in 12 lines of 100% python code.
I am talking about checking that a package actually comes from PyPI, not
the PGP author signature. This is important if anybody can deploy a
mirror... At least "easy_install" can automatically verify that the
downloaded package, from a mirror, was originated in the main PYPI
server and it was not modified in any way.
- --
Jesus Cea Avion _/_/ _/_/_/ _/_/_/
jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/
jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/_/_/_/
. _/_/ _/_/ _/_/ _/_/ _/_/
"Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/
"My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iQCVAwUBSWJJ0Jlgi5GaxT1NAQKKMAP/QZGMLzVq1bBv3BU8HLTtMdygfH4CsH29
dYCxEcgbx7FmrfrdyN9cnAg9xnWR4S0u6ObnfhxVrx0+UyXivtdtTqDxh13TNJay
6U93QbILsrtr2Ey+yFDHg9VwmqNb9LMX/UUvBt2uyd1BEHbiKacPrqshTCyvhdHY
aMW8rspseK4=
=6/Hp
-----END PGP SIGNATURE-----
From ziade.tarek at gmail.com Fri Jan 9 17:24:33 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Fri, 9 Jan 2009 17:24:33 +0100
Subject: [Catalog-sig] [Distutils] [distutils] make the storage of the
password optional in .pypirc
In-Reply-To: <20090109154504.GA25799@fridge.pov.lt>
References: <94bdd2610901040404w6675999exfde5e81f49cbaf0d@mail.gmail.com>
<4960BC4C.7060207@palladion.com>
<94bdd2610901042100g50901aabvd04c67afa67e5710@mail.gmail.com>
<94bdd2610901090032o40116765j96b7f2a68df3791d@mail.gmail.com>
<51f97e530901090708w3105ecf3la220a32347ae126c@mail.gmail.com>
<e5fff6640901090717s74c9618ducf50a80177bd80c8@mail.gmail.com>
<20090109154504.GA25799@fridge.pov.lt>
Message-ID: <94bdd2610901090824r5f13e43sc446665eaea146f3@mail.gmail.com>
On Fri, Jan 9, 2009 at 4:45 PM, Marius Gedminas <marius at pov.lt> wrote:
> On Fri, Jan 09, 2009 at 10:17:50AM -0500, Benji York wrote:
>> On Fri, Jan 9, 2009 at 10:08 AM, Stephen Emslie <stephenemslie at gmail.com> wrote:
>> > A bit OT, but from your blog post on the subject:
>> >
>> >>I'd like to go further and to think about a ssh-agent like system, so there's no need
>> >>to enter the pasword everytime you work with PyPI in the same session.
>> >
>> > Have you had any feedback on this yet?
>>
>> Here's some: how about instead of an ssh-like system, use ssh itself. Front
>> PyPI with an ssh server that users connect to. That way it is both secure and
>> the infrastructure (agent, etc.) is already in place.
>
> Yes please. I'd rather have one agent running and reuse my SSH key for
> authentication.
That would be awesome indeed. But that would involve quite some
changes on server side,
I'll forward this mail to catalog-sig for Richard, Martin and others's feedback
Regards
Tarek
--
Tarek Ziad? | Association AfPy | www.afpy.org
Blog FR | http://programmation-python.org
Blog EN | http://tarekziade.wordpress.com/
From martin at v.loewis.de Fri Jan 9 21:18:20 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 09 Jan 2009 21:18:20 +0100
Subject: [Catalog-sig] [Distutils] [distutils] make the storage of the
password optional in .pypirc
In-Reply-To: <94bdd2610901090824r5f13e43sc446665eaea146f3@mail.gmail.com>
References: <94bdd2610901040404w6675999exfde5e81f49cbaf0d@mail.gmail.com> <4960BC4C.7060207@palladion.com> <94bdd2610901042100g50901aabvd04c67afa67e5710@mail.gmail.com> <94bdd2610901090032o40116765j96b7f2a68df3791d@mail.gmail.com> <51f97e530901090708w3105ecf3la220a32347ae126c@mail.gmail.com> <e5fff6640901090717s74c9618ducf50a80177bd80c8@mail.gmail.com> <20090109154504.GA25799@fridge.pov.lt>
<94bdd2610901090824r5f13e43sc446665eaea146f3@mail.gmail.com>
Message-ID: <4967B10C.6030904@v.loewis.de>
>>> Here's some: how about instead of an ssh-like system, use ssh itself. Front
>>> PyPI with an ssh server that users connect to. That way it is both secure and
>>> the infrastructure (agent, etc.) is already in place.
>> Yes please. I'd rather have one agent running and reuse my SSH key for
>> authentication.
>
> That would be awesome indeed. But that would involve quite some
> changes on server side,
> I'll forward this mail to catalog-sig for Richard, Martin and others's feedback
I'm fairly skeptical. First, the infrastructure is *not* yet in place.
Nobody has uploaded SSH keys to PyPI, and in order to allow SSH access,
we probably would need to create a Unix account, which then runs a fixed
(Python) program on ssh login. That is much less secure than the current
setup, in the sense that this program can probably tricked much easier
than Apache can. So it opens a door for people hacking into the system;
all they have to do is to create a fake PyPI account and upload an SSH
key...
To improve password storage, I think it would be better to use the
platform's secure password storage services where available (e.g.
OSX Keychain, KDE KWallet, etc). Of course, such a library should be
developed independently of distutils. For Keychain, there is already
http://muffinresearch.co.uk/archives/2008/02/05/python-keychainpy-access-to-the-mac-osx-keychain/
Regards,
Martin
From jim at zope.com Fri Jan 9 21:57:55 2009
From: jim at zope.com (Jim Fulton)
Date: Fri, 9 Jan 2009 15:57:55 -0500
Subject: [Catalog-sig] [Distutils] [distutils] make the storage of the
password optional in .pypirc
In-Reply-To: <4967B10C.6030904@v.loewis.de>
References: <94bdd2610901040404w6675999exfde5e81f49cbaf0d@mail.gmail.com> <4960BC4C.7060207@palladion.com> <94bdd2610901042100g50901aabvd04c67afa67e5710@mail.gmail.com> <94bdd2610901090032o40116765j96b7f2a68df3791d@mail.gmail.com> <51f97e530901090708w3105ecf3la220a32347ae126c@mail.gmail.com> <e5fff6640901090717s74c9618ducf50a80177bd80c8@mail.gmail.com> <20090109154504.GA25799@fridge.pov.lt>
<94bdd2610901090824r5f13e43sc446665eaea146f3@mail.gmail.com>
<4967B10C.6030904@v.loewis.de>
Message-ID: <9A77A80A-133F-47F7-AD3B-3CBDB206DE7B@zope.com>
On Jan 9, 2009, at 3:18 PM, Martin v. L?wis wrote:
>>>> Here's some: how about instead of an ssh-like system, use ssh
>>>> itself. Front
>>>> PyPI with an ssh server that users connect to. That way it is
>>>> both secure and
>>>> the infrastructure (agent, etc.) is already in place.
>>> Yes please. I'd rather have one agent running and reuse my SSH
>>> key for
>>> authentication.
>>
>> That would be awesome indeed. But that would involve quite some
>> changes on server side,
>> I'll forward this mail to catalog-sig for Richard, Martin and
>> others's feedback
>
> I'm fairly skeptical. First, the infrastructure is *not* yet in place.
> Nobody has uploaded SSH keys to PyPI,
Right. PyPI would have to grow the ability to manage public keys for
users.
> and in order to allow SSH access,
> we probably would need to create a Unix account,
No, you would not.
> which then runs a fixed
> (Python) program on ssh login. That is much less secure than the
> current
> setup, in the sense that this program can probably tricked much easier
> than Apache can. So it opens a door for people hacking into the
> system;
> all they have to do is to create a fake PyPI account and upload an SSH
> key...
No. You'd have a new server process, written in Python using Twisted
or paramiko, that would would provide a small number of specialized
commands and that would read public keys from the pypi database for
authentication and update the database in response to commands,
Jim
--
Jim Fulton
Zope Corporation
From jim at zope.com Fri Jan 9 22:02:53 2009
From: jim at zope.com (Jim Fulton)
Date: Fri, 9 Jan 2009 16:02:53 -0500
Subject: [Catalog-sig] [Distutils] [distutils] make the storage of the
password optional in .pypirc
In-Reply-To: <9A77A80A-133F-47F7-AD3B-3CBDB206DE7B@zope.com>
References: <94bdd2610901040404w6675999exfde5e81f49cbaf0d@mail.gmail.com> <4960BC4C.7060207@palladion.com> <94bdd2610901042100g50901aabvd04c67afa67e5710@mail.gmail.com> <94bdd2610901090032o40116765j96b7f2a68df3791d@mail.gmail.com> <51f97e530901090708w3105ecf3la220a32347ae126c@mail.gmail.com> <e5fff6640901090717s74c9618ducf50a80177bd80c8@mail.gmail.com> <20090109154504.GA25799@fridge.pov.lt>
<94bdd2610901090824r5f13e43sc446665eaea146f3@mail.gmail.com>
<4967B10C.6030904@v.loewis.de>
<9A77A80A-133F-47F7-AD3B-3CBDB206DE7B@zope.com>
Message-ID: <EA391517-5EB4-426A-81C9-2FBF276FC512@zope.com>
On Jan 9, 2009, at 3:57 PM, Jim Fulton wrote:
> No. You'd have a new server process, written in Python using Twisted
> or paramiko, that would would provide a small number of specialized
> commands
Or better yet, supported scp. Then the upload/register process would
be reduced to just scp-ing a distro to pypi. The server could read
meta-data from the distro, register the release, if necessary, and put
the distro in the right place.
Jim
--
Jim Fulton
Zope Corporation
From martin at v.loewis.de Fri Jan 9 22:03:25 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 09 Jan 2009 22:03:25 +0100
Subject: [Catalog-sig] [Distutils] [distutils] make the storage of the
password optional in .pypirc
In-Reply-To: <9A77A80A-133F-47F7-AD3B-3CBDB206DE7B@zope.com>
References: <94bdd2610901040404w6675999exfde5e81f49cbaf0d@mail.gmail.com> <4960BC4C.7060207@palladion.com> <94bdd2610901042100g50901aabvd04c67afa67e5710@mail.gmail.com> <94bdd2610901090032o40116765j96b7f2a68df3791d@mail.gmail.com> <51f97e530901090708w3105ecf3la220a32347ae126c@mail.gmail.com> <e5fff6640901090717s74c9618ducf50a80177bd80c8@mail.gmail.com> <20090109154504.GA25799@fridge.pov.lt>
<94bdd2610901090824r5f13e43sc446665eaea146f3@mail.gmail.com>
<4967B10C.6030904@v.loewis.de>
<9A77A80A-133F-47F7-AD3B-3CBDB206DE7B@zope.com>
Message-ID: <4967BB9D.6070307@v.loewis.de>
> No. You'd have a new server process, written in Python using Twisted or
> paramiko, that would would provide a small number of specialized
> commands and that would read public keys from the pypi database for
> authentication and update the database in response to commands,
Ok. I guess "contributions are welcome".
Regards,
Martin
From martin at v.loewis.de Fri Jan 9 22:07:36 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 09 Jan 2009 22:07:36 +0100
Subject: [Catalog-sig] [Distutils] [distutils] make the storage of the
password optional in .pypirc
In-Reply-To: <EA391517-5EB4-426A-81C9-2FBF276FC512@zope.com>
References: <94bdd2610901040404w6675999exfde5e81f49cbaf0d@mail.gmail.com> <4960BC4C.7060207@palladion.com> <94bdd2610901042100g50901aabvd04c67afa67e5710@mail.gmail.com> <94bdd2610901090032o40116765j96b7f2a68df3791d@mail.gmail.com> <51f97e530901090708w3105ecf3la220a32347ae126c@mail.gmail.com> <e5fff6640901090717s74c9618ducf50a80177bd80c8@mail.gmail.com> <20090109154504.GA25799@fridge.pov.lt>
<94bdd2610901090824r5f13e43sc446665eaea146f3@mail.gmail.com>
<4967B10C.6030904@v.loewis.de>
<9A77A80A-133F-47F7-AD3B-3CBDB206DE7B@zope.com>
<EA391517-5EB4-426A-81C9-2FBF276FC512@zope.com>
Message-ID: <4967BC98.8070508@v.loewis.de>
> Or better yet, supported scp. Then the upload/register process would be
> reduced to just scp-ing a distro to pypi. The server could read
> meta-data from the distro, register the release, if necessary, and put
> the distro in the right place.
That wouldn't fit too well with the existing "register" and "upload"
commands, I think.
Regards,
Martin
From ziade.tarek at gmail.com Sat Jan 10 11:35:48 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sat, 10 Jan 2009 11:35:48 +0100
Subject: [Catalog-sig] [Distutils] [distutils] make the storage of the
password optional in .pypirc
In-Reply-To: <4967BC98.8070508@v.loewis.de>
References: <94bdd2610901040404w6675999exfde5e81f49cbaf0d@mail.gmail.com>
<94bdd2610901090032o40116765j96b7f2a68df3791d@mail.gmail.com>
<51f97e530901090708w3105ecf3la220a32347ae126c@mail.gmail.com>
<e5fff6640901090717s74c9618ducf50a80177bd80c8@mail.gmail.com>
<20090109154504.GA25799@fridge.pov.lt>
<94bdd2610901090824r5f13e43sc446665eaea146f3@mail.gmail.com>
<4967B10C.6030904@v.loewis.de>
<9A77A80A-133F-47F7-AD3B-3CBDB206DE7B@zope.com>
<EA391517-5EB4-426A-81C9-2FBF276FC512@zope.com>
<4967BC98.8070508@v.loewis.de>
Message-ID: <94bdd2610901100235o6c1544b5u4e94a0fe6111304@mail.gmail.com>
On Fri, Jan 9, 2009 at 10:07 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
>> Or better yet, supported scp. Then the upload/register process would be
>> reduced to just scp-ing a distro to pypi. The server could read
>> meta-data from the distro, register the release, if necessary, and put
>> the distro in the right place.
>
> That wouldn't fit too well with the existing "register" and "upload"
> commands, I think.
+1
and in any case those commands should stay compatible with the current
mechanism and
let people store the password in the pypirc file if they want to, and
use https authentication.
Imho a scp/ssh protocol should be implemented in a new set of commands,
Regards
Tarek
>
> Regards,
> Martin
>
--
Tarek Ziad? | Association AfPy | www.afpy.org
Blog FR | http://programmation-python.org
Blog EN | http://tarekziade.wordpress.com/
From tseaver at palladion.com Sun Jan 11 01:40:54 2009
From: tseaver at palladion.com (Tres Seaver)
Date: Sat, 10 Jan 2009 19:40:54 -0500
Subject: [Catalog-sig] [distutils] make the storage of the password
optional in .pypirc
In-Reply-To: <4967B10C.6030904@v.loewis.de>
References: <94bdd2610901040404w6675999exfde5e81f49cbaf0d@mail.gmail.com> <4960BC4C.7060207@palladion.com> <94bdd2610901042100g50901aabvd04c67afa67e5710@mail.gmail.com> <94bdd2610901090032o40116765j96b7f2a68df3791d@mail.gmail.com> <51f97e530901090708w3105ecf3la220a32347ae126c@mail.gmail.com> <e5fff6640901090717s74c9618ducf50a80177bd80c8@mail.gmail.com> <20090109154504.GA25799@fridge.pov.lt> <94bdd2610901090824r5f13e43sc446665eaea146f3@mail.gmail.com>
<4967B10C.6030904@v.loewis.de>
Message-ID: <49694016.8080302@palladion.com>
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Martin v. L?wis wrote:
>>>> Here's some: how about instead of an ssh-like system, use ssh itself. Front
>>>> PyPI with an ssh server that users connect to. That way it is both secure and
>>>> the infrastructure (agent, etc.) is already in place.
>>> Yes please. I'd rather have one agent running and reuse my SSH key for
>>> authentication.
>> That would be awesome indeed. But that would involve quite some
>> changes on server side,
>> I'll forward this mail to catalog-sig for Richard, Martin and others's feedback
>
> I'm fairly skeptical. First, the infrastructure is *not* yet in place.
> Nobody has uploaded SSH keys to PyPI, and in order to allow SSH access,
> we probably would need to create a Unix account, which then runs a fixed
> (Python) program on ssh login.
Right, a single account with multiple keys (each with 'command='do_pypi
- -u <userid>').
> That is much less secure than the current
> setup, in the sense that this program can probably tricked much easier
> than Apache can. So it opens a door for people hacking into the system;
> all they have to do is to create a fake PyPI account and upload an SSH
> key...
Zope has been using the 'command=' bit to do SSH-protected CVS / SVN
access since 2000 with a lot of success; 370+ committers have keys on
the system. The command being executed is actually a small shell
script, which barfs if the program being run is not one of 'svn', 'cvs',
or 'scp' (for uploading tarballs).
> To improve password storage, I think it would be better to use the
> platform's secure password storage services where available (e.g.
> OSX Keychain, KDE KWallet, etc). Of course, such a library should be
> developed independently of distutils. For Keychain, there is already
>
> http://muffinresearch.co.uk/archives/2008/02/05/python-keychainpy-access-to-the-mac-osx-keychain/
Not only are PyPI passwords stored in the clear on user's hard drives,
they are sent in the clear on every authenticated request to the web
interface (basic auth over unencrypted HTTP): it seems to me we ought
to worry about both those issues more.
Tres.
- --
===================================================================
Tres Seaver +1 540-429-0999 tseaver at palladion.com
Palladion Software "Excellence by Design" http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFJaUAW+gerLs4ltQ4RAhFXAJ47WOzMAe12m+YD5BNu22BzTU+QRQCeLTbX
DSaVk1I96K5mzaZro98HUTU=
=8sRs
-----END PGP SIGNATURE-----
From martin at v.loewis.de Sun Jan 11 04:35:36 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 11 Jan 2009 04:35:36 +0100
Subject: [Catalog-sig] [distutils] make the storage of the password
optional in .pypirc
In-Reply-To: <49694016.8080302@palladion.com>
References: <94bdd2610901040404w6675999exfde5e81f49cbaf0d@mail.gmail.com> <4960BC4C.7060207@palladion.com> <94bdd2610901042100g50901aabvd04c67afa67e5710@mail.gmail.com> <94bdd2610901090032o40116765j96b7f2a68df3791d@mail.gmail.com> <51f97e530901090708w3105ecf3la220a32347ae126c@mail.gmail.com> <e5fff6640901090717s74c9618ducf50a80177bd80c8@mail.gmail.com> <20090109154504.GA25799@fridge.pov.lt> <94bdd2610901090824r5f13e43sc446665eaea146f3@mail.gmail.com>
<4967B10C.6030904@v.loewis.de> <49694016.8080302@palladion.com>
Message-ID: <49696908.1070501@v.loewis.de>
>> That is much less secure than the current
>> setup, in the sense that this program can probably tricked much easier
>> than Apache can. So it opens a door for people hacking into the system;
>> all they have to do is to create a fake PyPI account and upload an SSH
>> key...
>
> Zope has been using the 'command=' bit to do SSH-protected CVS / SVN
> access since 2000 with a lot of success; 370+ committers have keys on
> the system. The command being executed is actually a small shell
> script, which barfs if the program being run is not one of 'svn', 'cvs',
> or 'scp' (for uploading tarballs).
Well, then good luck that nobody has tried to hack your script. E.g.
might it work that I somehow manage to upload a svn binary onto your
system (e.g. by first checking it in, and relying on an automated
checkout process that runs somewhere), then invoke this binary through
the shell account?
> Not only are PyPI passwords stored in the clear on user's hard drives,
> they are sent in the clear on every authenticated request to the web
> interface (basic auth over unencrypted HTTP): it seems to me we ought
> to worry about both those issues more.
Perhaps. Contributions are welcome.
Regards,
Martin
From ziade.tarek at gmail.com Sun Jan 11 10:29:21 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sun, 11 Jan 2009 10:29:21 +0100
Subject: [Catalog-sig] [distutils] make the storage of the password
optional in .pypirc
In-Reply-To: <49696908.1070501@v.loewis.de>
References: <94bdd2610901040404w6675999exfde5e81f49cbaf0d@mail.gmail.com>
<94bdd2610901042100g50901aabvd04c67afa67e5710@mail.gmail.com>
<94bdd2610901090032o40116765j96b7f2a68df3791d@mail.gmail.com>
<51f97e530901090708w3105ecf3la220a32347ae126c@mail.gmail.com>
<e5fff6640901090717s74c9618ducf50a80177bd80c8@mail.gmail.com>
<20090109154504.GA25799@fridge.pov.lt>
<94bdd2610901090824r5f13e43sc446665eaea146f3@mail.gmail.com>
<4967B10C.6030904@v.loewis.de> <49694016.8080302@palladion.com>
<49696908.1070501@v.loewis.de>
Message-ID: <94bdd2610901110129q545346c9ka41fa9319523ee89@mail.gmail.com>
On Sun, Jan 11, 2009 at 4:35 AM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
>> Not only are PyPI passwords stored in the clear on user's hard drives,
>> they are sent in the clear on every authenticated request to the web
>> interface (basic auth over unencrypted HTTP): it seems to me we ought
>> to worry about both those issues more.
>
> Perhaps. Contributions are welcome.
Can we finish on the PyPI mirroring contribution before we start this one ?
(since you are our entry point Martin on these topics)
I have finished my tests on my side. And I have a branch ready here
https://svn.python.org/packages/branches/tarek-pypi/pypi/
I would like to make more tests with a realistic flow of data, and
I am waiting for some feedback/help on this work.
here's how we could proceed:
phase 1 : proving non-regression
1 - I need an access to the pypi log files produced by Apache
(a simple browsable view of the log directory should be enough and
not risky)
2 - on my side I can grab those files daily right and put them on my
PyPI server instance, and run the process like if I was on the real
server.
3 - I will make this version reachable on my server, so we can check
that there's no regression = the count of the package that existed
before the dump I had should be equal and grow the same way on both sides.
phase 2 - testing the mirroring
4 - I will maintain a fake "mirror" that will be registered and will
provide realistic stats (a copy of the pypi apache log, where I will
keep just one hit per package file)
5 - we will validate that the global-stats and local-stats files
generated are right, and that the counts are the sum of pypi and the
mirror. (pypi+1)
If we can do that before Pycon maybe Pycon sprints could be the place where
we launch the mirroring, and start the SSH project if Jean-Paul and
others are willing to jump in ?
Regards
Tarek
--
Tarek Ziad? | Association AfPy | www.afpy.org
Blog FR | http://programmation-python.org
Blog EN | http://tarekziade.wordpress.com/
From rhijnauwen at gmail.com Wed Jan 14 16:06:59 2009
From: rhijnauwen at gmail.com (Bart Spaans)
Date: Wed, 14 Jan 2009 16:06:59 +0100
Subject: [Catalog-sig] How do we change project ownership?
Message-ID: <881c57460901140706x2376ef97n582945226128d96b@mail.gmail.com>
Hi,
I would like to take over the maintenance of a piece of software
(pyFluidSynth), but we can't seem to find a 'change owner' or 'add owner'
option. Is it possible to change the owner of a certain package or should
the original owner completely delete the package first? If so, will that
free up the name?
Best regards,
Bart Spaans.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20090114/299c45e2/attachment.htm>
From lists at zopyx.com Wed Jan 14 16:17:59 2009
From: lists at zopyx.com (Andreas Jung)
Date: Wed, 14 Jan 2009 16:17:59 +0100
Subject: [Catalog-sig] How do we change project ownership?
In-Reply-To: <881c57460901140706x2376ef97n582945226128d96b@mail.gmail.com>
References: <881c57460901140706x2376ef97n582945226128d96b@mail.gmail.com>
Message-ID: <496E0227.5050507@zopyx.com>
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 14.01.2009 16:06 Uhr, Bart Spaans wrote:
> Hi,
>
> I would like to take over the maintenance of a piece of software
> (pyFluidSynth), but we can't seem to find a 'change owner' or 'add
> owner' option. Is it possible to change the owner of a certain package
> or should the original owner completely delete the package first? If so,
> will that free up the name?
>
Please look carefully. There is link "Administer the ROle assigned to
users for this package" after having logged and choosing a package
belonging to you.
- -aj
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkluAicACgkQCJIWIbr9KYzXlQCcDeIIWee6IHtkaOfyh/vKSPuy
tQsAnRTJujPkAoBLgo7mR4wUjjbxRozu
=6Rx+
-----END PGP SIGNATURE-----
-------------- next part --------------
A non-text attachment was scrubbed...
Name: lists.vcf
Type: text/x-vcard
Size: 316 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20090114/56cd79c0/attachment.vcf>
From martin at v.loewis.de Sat Jan 17 15:45:07 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 17 Jan 2009 15:45:07 +0100
Subject: [Catalog-sig] New log record: rename
Message-ID: <4971EEF3.1030406@v.loewis.de>
Those of you monitoring the changelog might notice a
new log record, for when a package gets renamed.
There is currently no UI for renaming packages, so this
is really restricted to the PyPI administrator (renaming
requests can be submitted to the PyPI bug tracker).
Regards,
Martin
From martin at v.loewis.de Sat Jan 17 15:47:47 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 17 Jan 2009 15:47:47 +0100
Subject: [Catalog-sig] Automatic redirects for normalized names
Message-ID: <4971EF93.8020704@v.loewis.de>
Now that PyPI is free of name collisions with respect to
the (setuptools) normalized_name, the "simple" API offers
redirects in cases where a name was misspelled.
E.g. accessing
http://pypi.python.org/simple/pyxml
will redirect to
http://pypi.python.org/simple/PyXML
This should remove the need to download the entire simple
index in setuptools, in most cases.
Regards,
Martin
From jcea at jcea.es Wed Jan 21 17:19:32 2009
From: jcea at jcea.es (Jesus Cea)
Date: Wed, 21 Jan 2009 17:19:32 +0100
Subject: [Catalog-sig] Can not submit new packages
Message-ID: <49774B14.90109@jcea.es>
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
I am trying to upload a new package to PYPI, and the sending is failing
with an authentication failure. I already register again with "python
setup.py register" (using my old credentials), and the server gives "200
OK".. But trying "python setup.py sdist upload --sign --show-response"
gives this error:
Submitting dist/bsddb3-4.7.4.tar.gz to http://pypi.python.org/pypi
Upload failed (401): You must be identified to edit package information
-
---------------------------------------------------------------------------
<strong>Login required</strong><br /><br />
You must be identified to edit package information<br /><br />
<p>If you are a new user, <a href="/pypi?:action=register_form">please
register</a>.</p>
<p>If you have forgotten your password, you can have it
<a href="/pypi?:action=forgotten_password_form">reset for you</a>.</p>
-
---------------------------------------------------------------------------
- --
Jesus Cea Avion _/_/ _/_/_/ _/_/_/
jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/
jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/_/_/_/
. _/_/ _/_/ _/_/ _/_/ _/_/
"Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/
"My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iQCVAwUBSXdLD5lgi5GaxT1NAQKFsgP/R3djp/2pwEsGJ2T0KTqv2xDEEaRzZLkb
A4agXQqQ6VXM1Zd5KPR+z+/jjUAEIsb/Glih6QIrMS2dyNbE4C8w9i1ktqOCHpjQ
h78mbzoWwJ9GaEwlb1vZIjazFdewIYyCsAXjZaB3VGiHSOStGUiPXG1X70eAyujo
VWoSqV/GIbs=
=Bi7t
-----END PGP SIGNATURE-----
From martin at v.loewis.de Wed Jan 21 20:53:08 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 21 Jan 2009 20:53:08 +0100
Subject: [Catalog-sig] Can not submit new packages
In-Reply-To: <49774B14.90109@jcea.es>
References: <49774B14.90109@jcea.es>
Message-ID: <49777D24.7080003@v.loewis.de>
> Submitting dist/bsddb3-4.7.4.tar.gz to http://pypi.python.org/pypi
> Upload failed (401): You must be identified to edit package information
Can you debug this further to find out what is really happening?
Regards,
Martin
From jcea at jcea.es Wed Jan 21 21:38:04 2009
From: jcea at jcea.es (Jesus Cea)
Date: Wed, 21 Jan 2009 21:38:04 +0100
Subject: [Catalog-sig] Can not submit new packages
In-Reply-To: <49777D24.7080003@v.loewis.de>
References: <49774B14.90109@jcea.es> <49777D24.7080003@v.loewis.de>
Message-ID: <497787AC.70804@jcea.es>
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Martin v. L?wis wrote:
>> Submitting dist/bsddb3-4.7.4.tar.gz to http://pypi.python.org/pypi
>> Upload failed (401): You must be identified to edit package information
>
> Can you debug this further to find out what is really happening?
Too late. I uploaded the packages manually via the webpage. It worked
fine. Sorry, the upload was urgent.
Hope this issue is still alive and somebody else hits it :).
- --
Jesus Cea Avion _/_/ _/_/_/ _/_/_/
jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/
jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/_/_/_/
. _/_/ _/_/ _/_/ _/_/ _/_/
"Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/
"My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iQCVAwUBSXeHpplgi5GaxT1NAQJpBwP/QoilGvtklvFVfnQCN58CdKgSGjCxdxw+
u0uAD3/19b3ZNUeyC4Zd4KlJsIp6FRI37XBapg/7+oQF65T4QqwIh+iBIrsY86r4
m5dRNsIBa8zvv9ZzHPe33Ekne5bdVFPr4vF9Lx+ktX1FpMvV/KeukPkk5ZcY7DKT
JIv6kj/IWr4=
=wEBA
-----END PGP SIGNATURE-----
From jcea at jcea.es Wed Jan 21 21:40:34 2009
From: jcea at jcea.es (Jesus Cea)
Date: Wed, 21 Jan 2009 21:40:34 +0100
Subject: [Catalog-sig] Can not submit new packages
In-Reply-To: <497787AC.70804@jcea.es>
References: <49774B14.90109@jcea.es> <49777D24.7080003@v.loewis.de>
<497787AC.70804@jcea.es>
Message-ID: <49778842.6050705@jcea.es>
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Jesus Cea wrote:
> Hope this issue is still alive and somebody else hits it :).
The strange thing is that I had enough privileges to create the release
(4.7.4), but the file upload failed.
- --
Jesus Cea Avion _/_/ _/_/_/ _/_/_/
jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/
jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/_/_/_/
. _/_/ _/_/ _/_/ _/_/ _/_/
"Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/
"My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iQCVAwUBSXeIQplgi5GaxT1NAQIVpQP/dBMgZ3HwNNXiGZoKsEK2f0cuyxByHsEl
9d716pS76ARsIJpkovuV4vX9uD54p52Gp74H6BiMrf9dnFP7W8wJmxrrHBqepbF8
JWxHIHma3FerAuSrh0xilmuM2l4McZ3AzlFPp91Ny0wkpNHdErnNKz4jYWbCr8mg
TsdGM0nlA+g=
=GzIm
-----END PGP SIGNATURE-----
From szybalski at gmail.com Thu Jan 22 05:52:53 2009
From: szybalski at gmail.com (Lukasz Szybalski)
Date: Wed, 21 Jan 2009 22:52:53 -0600
Subject: [Catalog-sig] local copy of pypi packages list, and package data,
how?
Message-ID: <804e5c70901212052j2e9eb544o102e2b929e2441db@mail.gmail.com>
Hello,
I've been looking into xmlrpc interface that you have for pypi. I am
able to browse all packages and get the data about
them...keywords,etc....I want to have a local version of the catalog
data and keep it in sync daily.
What I'm wondering is how can I keep my app in sync? Here is what I'm
doing right now, I was wondering if this is not overloading your
servers, or is there a faster/more efficient way.
1. I get a list of all 5000+ packages.
2. For each package I get a version number.
3. For each (package,version#) I get the package data.
4. Sync daily using updated_releases
I use the package data to look for certain keywords. Process 2 seem to
take around 20+minutes, process 3 takes more, but after first time I
can just get the new updated packages since the last time and run
these.
1. Is there a xmlrpc function that I can use to search for keywords
and just get the packages I need?
2. Is there a better strategy then what I am doing? I would like to sync daily.
http://lucasmanual.com/blog/2009/how-to-get-information-from-pypi-via-xmlrpc/
Thanks,
Lucas
--
How to create python package?
http://lucasmanual.com/mywiki/PythonPaste
Bazaar and Launchpad
http://lucasmanual.com/mywiki/Bazaar
From amk at amk.ca Thu Jan 22 13:02:36 2009
From: amk at amk.ca (A.M. Kuchling)
Date: Thu, 22 Jan 2009 07:02:36 -0500
Subject: [Catalog-sig] PyPI load this morning
Message-ID: <20090122120236.GA8545@amk.local>
Some sort of script was hitting the top page on PyPI this morning,
driving the machine's load average to 15. I added:
# 2009-01-22
deny from 213.41.97.133
to the cheeseshop config and reloaded Apache, and the load average has
now dropped to around .3.
--amk
From lists at zopyx.com Fri Jan 23 15:09:31 2009
From: lists at zopyx.com (Andreas Jung)
Date: Fri, 23 Jan 2009 15:09:31 +0100
Subject: [Catalog-sig] [PyPI] Hidden release shown on simple index
Message-ID: <4979CF9B.1080001@zopyx.com>
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi there,
the simplex index also contains stuff belonging to hidden releases.
Is this intentional?
Example (threadframe package):
http://pypi.python.org/pypi/threadframe
The XMLRPC API reveals only one published version: 0.2
The simple index contains also a 1.0 release which is hidden:
http://pypi.python.org/simple/threadframe
I suggest that the simple index should only show stuff belong to
un-hidden releases.
Andreas
- --
ZOPYX Ltd. & Co. KG - Charlottenstr. 37/1 - 72070 T?bingen - Germany
Web: www.zopyx.com - Email: info at zopyx.com - Phone +49 - 7071 - 793376
Registergericht: Amtsgericht Stuttgart, Handelsregister A 381535
Gesch?ftsf?hrer/Gesellschafter: ZOPYX Limited, Birmingham, UK
- ------------------------------------------------------------------------
E-Publishing, Python, Zope & Plone development, Consulting
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkl5z5sACgkQCJIWIbr9KYxbxQCfZsJ7jfBWA4tjBA/uG0/4CbhP
NkcAn0Guz5UvZj6axswJBwlOieJAzOS4
=YoCs
-----END PGP SIGNATURE-----
-------------- next part --------------
A non-text attachment was scrubbed...
Name: lists.vcf
Type: text/x-vcard
Size: 316 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20090123/76678817/attachment.vcf>
From fdrake at gmail.com Fri Jan 23 15:15:47 2009
From: fdrake at gmail.com (Fred Drake)
Date: Fri, 23 Jan 2009 09:15:47 -0500
Subject: [Catalog-sig] [PyPI] Hidden release shown on simple index
In-Reply-To: <4979CF9B.1080001@zopyx.com>
References: <4979CF9B.1080001@zopyx.com>
Message-ID: <9cee7ab80901230615g580611a5p18d4cf2e5fedc226@mail.gmail.com>
On Fri, Jan 23, 2009 at 9:09 AM, Andreas Jung <lists at zopyx.com> wrote:
> the simplex index also contains stuff belonging to hidden releases.
> Is this intentional?
Yes.
Projects that are already using a version still need to be able to
find the releases; this is common for projects that specify exact
versions for deployment purposes.
The simple index is the right place for this; tools like setuptools
and zc.buildout use the simple index for automated operations by
default.
The web UI doesn't show hidden releases, which is fine for interactive users.
-Fred
--
Fred L. Drake, Jr. <fdrake at gmail.com>
"Chaos is the score upon which reality is written." --Henry Miller
From benji at benjiyork.com Fri Jan 23 16:11:53 2009
From: benji at benjiyork.com (Benji York)
Date: Fri, 23 Jan 2009 10:11:53 -0500
Subject: [Catalog-sig] [PyPI] Hidden release shown on simple index
In-Reply-To: <9cee7ab80901230615g580611a5p18d4cf2e5fedc226@mail.gmail.com>
References: <4979CF9B.1080001@zopyx.com>
<9cee7ab80901230615g580611a5p18d4cf2e5fedc226@mail.gmail.com>
Message-ID: <e5fff6640901230711l7e200346qdcc9cc0d4f8a493c@mail.gmail.com>
On Fri, Jan 23, 2009 at 9:15 AM, Fred Drake <fdrake at gmail.com> wrote:
> On Fri, Jan 23, 2009 at 9:09 AM, Andreas Jung <lists at zopyx.com> wrote:
>> the simplex index also contains stuff belonging to hidden releases.
>> Is this intentional?
>
> Yes.
>
> Projects that are already using a version still need to be able to
> find the releases; this is common for projects that specify exact
> versions for deployment purposes.
Exactly.
> The web UI doesn't show hidden releases, which is fine for interactive users.
As an interactive user, I've never understood why the hidden release
functionality exists; especially automatically hiding old releases.
--
Benji York
From fdrake at gmail.com Fri Jan 23 16:14:01 2009
From: fdrake at gmail.com (Fred Drake)
Date: Fri, 23 Jan 2009 10:14:01 -0500
Subject: [Catalog-sig] [PyPI] Hidden release shown on simple index
In-Reply-To: <e5fff6640901230711l7e200346qdcc9cc0d4f8a493c@mail.gmail.com>
References: <4979CF9B.1080001@zopyx.com>
<9cee7ab80901230615g580611a5p18d4cf2e5fedc226@mail.gmail.com>
<e5fff6640901230711l7e200346qdcc9cc0d4f8a493c@mail.gmail.com>
Message-ID: <9cee7ab80901230714r76f9ac43vadfe12cccfd6cdc6@mail.gmail.com>
On Fri, Jan 23, 2009 at 10:11 AM, Benji York <benji at benjiyork.com> wrote:
> As an interactive user, I've never understood why the hidden release
> functionality exists; especially automatically hiding old releases.
I can only guess at the original motivations myself, and that's not
really helpful.
The automatic hiding of older releases is definitely a mistake; I've
seen no good come of it, and it's surprising.
-Fred
--
Fred L. Drake, Jr. <fdrake at gmail.com>
"Chaos is the score upon which reality is written." --Henry Miller
From richardjones at optushome.com.au Sat Jan 24 02:44:30 2009
From: richardjones at optushome.com.au (Richard Jones)
Date: Sat, 24 Jan 2009 12:44:30 +1100
Subject: [Catalog-sig] [PyPI] Hidden release shown on simple index
In-Reply-To: <9cee7ab80901230714r76f9ac43vadfe12cccfd6cdc6@mail.gmail.com>
References: <4979CF9B.1080001@zopyx.com>
<e5fff6640901230711l7e200346qdcc9cc0d4f8a493c@mail.gmail.com>
<9cee7ab80901230714r76f9ac43vadfe12cccfd6cdc6@mail.gmail.com>
Message-ID: <200901241244.30479.richardjones@optushome.com.au>
On Sat, 24 Jan 2009, Fred Drake wrote:
> On Fri, Jan 23, 2009 at 10:11 AM, Benji York <benji at benjiyork.com> wrote:
> > As an interactive user, I've never understood why the hidden release
> > functionality exists; especially automatically hiding old releases.
>
> I can only guess at the original motivations myself, and that's not
> really helpful.
It seemed like a good idea at the time :)
> The automatic hiding of older releases is definitely a mistake; I've
> seen no good come of it, and it's surprising.
Proposals to change it are, as with everything PyPI, welcome :)
Richard
From jim at zope.com Sat Jan 24 16:57:48 2009
From: jim at zope.com (Jim Fulton)
Date: Sat, 24 Jan 2009 10:57:48 -0500
Subject: [Catalog-sig] [PyPI] Hidden release shown on simple index
In-Reply-To: <200901241244.30479.richardjones@optushome.com.au>
References: <4979CF9B.1080001@zopyx.com>
<e5fff6640901230711l7e200346qdcc9cc0d4f8a493c@mail.gmail.com>
<9cee7ab80901230714r76f9ac43vadfe12cccfd6cdc6@mail.gmail.com>
<200901241244.30479.richardjones@optushome.com.au>
Message-ID: <AA53C7FF-0534-40A4-83AC-6B703760CCAA@zope.com>
On Jan 23, 2009, at 8:44 PM, Richard Jones wrote:
...
>> The automatic hiding of older releases is definitely a mistake; I've
>> seen no good come of it, and it's surprising.
>
> Proposals to change it are, as with everything PyPI, welcome :)
What about:
http://mail.python.org/pipermail/catalog-sig/2007-April/001083.html
:)
Jim
--
Jim Fulton
Zope Corporation
From fdrake at gmail.com Sat Jan 24 17:27:41 2009
From: fdrake at gmail.com (Fred Drake)
Date: Sat, 24 Jan 2009 11:27:41 -0500
Subject: [Catalog-sig] [PyPI] Hidden release shown on simple index
In-Reply-To: <AA53C7FF-0534-40A4-83AC-6B703760CCAA@zope.com>
References: <4979CF9B.1080001@zopyx.com>
<e5fff6640901230711l7e200346qdcc9cc0d4f8a493c@mail.gmail.com>
<9cee7ab80901230714r76f9ac43vadfe12cccfd6cdc6@mail.gmail.com>
<200901241244.30479.richardjones@optushome.com.au>
<AA53C7FF-0534-40A4-83AC-6B703760CCAA@zope.com>
Message-ID: <9cee7ab80901240827g39b37bd3s5098702daa0e4f46@mail.gmail.com>
On Sat, Jan 24, 2009 at 10:57 AM, Jim Fulton <jim at zope.com> wrote:
> What about:
>
> http://mail.python.org/pipermail/catalog-sig/2007-April/001083.html
I think Richard and Martin are waiting for the patch I've never had
time to look into. :-(
-Fred
--
Fred L. Drake, Jr. <fdrake at gmail.com>
"Chaos is the score upon which reality is written." --Henry Miller
From martin at v.loewis.de Sat Jan 24 20:34:02 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 24 Jan 2009 20:34:02 +0100
Subject: [Catalog-sig] [PyPI] Hidden release shown on simple index
In-Reply-To: <9cee7ab80901240827g39b37bd3s5098702daa0e4f46@mail.gmail.com>
References: <4979CF9B.1080001@zopyx.com> <e5fff6640901230711l7e200346qdcc9cc0d4f8a493c@mail.gmail.com> <9cee7ab80901230714r76f9ac43vadfe12cccfd6cdc6@mail.gmail.com> <200901241244.30479.richardjones@optushome.com.au> <AA53C7FF-0534-40A4-83AC-6B703760CCAA@zope.com>
<9cee7ab80901240827g39b37bd3s5098702daa0e4f46@mail.gmail.com>
Message-ID: <497B6D2A.9080905@v.loewis.de>
Fred Drake wrote:
> On Sat, Jan 24, 2009 at 10:57 AM, Jim Fulton <jim at zope.com> wrote:
>> What about:
>>
>> http://mail.python.org/pipermail/catalog-sig/2007-April/001083.html
>
> I think Richard and Martin are waiting for the patch I've never had
> time to look into. :-(
It's a bit different: in absence of a patch, I had simply forgotten
about it (IOW, I wasn't waiting).
I have now implemented that feature; there is a checkbox on each
package telling whether old revisions should automatically get hidden.
Regards,
Martin
From martin at v.loewis.de Sat Jan 24 20:36:29 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 24 Jan 2009 20:36:29 +0100
Subject: [Catalog-sig] [PyPI] Hidden release shown on simple index
In-Reply-To: <4979CF9B.1080001@zopyx.com>
References: <4979CF9B.1080001@zopyx.com>
Message-ID: <497B6DBD.2080409@v.loewis.de>
> the simplex index also contains stuff belonging to hidden releases.
> Is this intentional?
As others have already explained: yes. It was made the way it is
specifically on request of setuptools users (as was the entire /simple)
index.
> I suggest that the simple index should only show stuff belong to
> un-hidden releases.
Chances of changing PyPI would be slightly (but not much) higher
if you explained *why* you want this changed.
Regards,
Martin
From martin at v.loewis.de Sat Jan 24 20:48:52 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 24 Jan 2009 20:48:52 +0100
Subject: [Catalog-sig] local copy of pypi packages list, and package data,
how?
In-Reply-To: <804e5c70901212052j2e9eb544o102e2b929e2441db@mail.gmail.com>
References: <804e5c70901212052j2e9eb544o102e2b929e2441db@mail.gmail.com>
Message-ID: <497B70A4.8010009@v.loewis.de>
> What I'm wondering is how can I keep my app in sync?
There are a number of PyPI mirroring solutions out there; I suggest
you use one of them:
http://pypi.python.org/pypi/z3c.pypimirror
https://launchpad.net/~pypi-mirror
> Here is what I'm
> doing right now, I was wondering if this is not overloading your
> servers, or is there a faster/more efficient way.
See AMK's recent message - perhaps it was you who was overloading the
server.
> 1. Is there a xmlrpc function that I can use to search for keywords
> and just get the packages I need?
No; you might use the regular UI search function, of course, but please
do restrict this to a small number of queries per hour.
> 2. Is there a better strategy then what I am doing? I would like to sync daily.
For downloading all files for a package, you might want to use the
simple API (/simple). For keeping in sync, you might want to use
changelog; updated_releases will only tell you whether a new release was
made, not whether a file has been added or replaced.
Regards,
Martin
From fdrake at gmail.com Sat Jan 24 20:49:00 2009
From: fdrake at gmail.com (Fred Drake)
Date: Sat, 24 Jan 2009 14:49:00 -0500
Subject: [Catalog-sig] [PyPI] Hidden release shown on simple index
In-Reply-To: <497B6D2A.9080905@v.loewis.de>
References: <4979CF9B.1080001@zopyx.com>
<e5fff6640901230711l7e200346qdcc9cc0d4f8a493c@mail.gmail.com>
<9cee7ab80901230714r76f9ac43vadfe12cccfd6cdc6@mail.gmail.com>
<200901241244.30479.richardjones@optushome.com.au>
<AA53C7FF-0534-40A4-83AC-6B703760CCAA@zope.com>
<9cee7ab80901240827g39b37bd3s5098702daa0e4f46@mail.gmail.com>
<497B6D2A.9080905@v.loewis.de>
Message-ID: <9cee7ab80901241149w4f655987t15ca4fc0a32fa84c@mail.gmail.com>
On Sat, Jan 24, 2009 at 2:34 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> I have now implemented that feature; there is a checkbox on each
> package telling whether old revisions should automatically get hidden.
Wonderful! This is really nice; I've started switching projects I
manage that should have this toggled.
-Fred
--
Fred L. Drake, Jr. <fdrake at gmail.com>
"Chaos is the score upon which reality is written." --Henry Miller
From benji at benjiyork.com Sat Jan 24 21:24:30 2009
From: benji at benjiyork.com (Benji York)
Date: Sat, 24 Jan 2009 15:24:30 -0500
Subject: [Catalog-sig] [PyPI] Hidden release shown on simple index
In-Reply-To: <497B6D2A.9080905@v.loewis.de>
References: <4979CF9B.1080001@zopyx.com>
<e5fff6640901230711l7e200346qdcc9cc0d4f8a493c@mail.gmail.com>
<9cee7ab80901230714r76f9ac43vadfe12cccfd6cdc6@mail.gmail.com>
<200901241244.30479.richardjones@optushome.com.au>
<AA53C7FF-0534-40A4-83AC-6B703760CCAA@zope.com>
<9cee7ab80901240827g39b37bd3s5098702daa0e4f46@mail.gmail.com>
<497B6D2A.9080905@v.loewis.de>
Message-ID: <e5fff6640901241224ja4351fasf73c62096563ef8c@mail.gmail.com>
On Sat, Jan 24, 2009 at 2:34 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> Fred Drake wrote:
>> On Sat, Jan 24, 2009 at 10:57 AM, Jim Fulton <jim at zope.com> wrote:
>>> What about:
>>>
>>> http://mail.python.org/pipermail/catalog-sig/2007-April/001083.html
>>
>> I think Richard and Martin are waiting for the patch I've never had
>> time to look into. :-(
>
> It's a bit different: in absence of a patch, I had simply forgotten
> about it (IOW, I wasn't waiting).
>
> I have now implemented that feature; there is a checkbox on each
> package telling whether old revisions should automatically get hidden.
Great!
--
Benji York
From lists at zopyx.com Sun Jan 25 12:12:51 2009
From: lists at zopyx.com (Andreas Jung)
Date: Sun, 25 Jan 2009 12:12:51 +0100
Subject: [Catalog-sig] [PyPI] Hidden release shown on simple index
In-Reply-To: <4979CF9B.1080001@zopyx.com>
References: <4979CF9B.1080001@zopyx.com>
Message-ID: <497C4933.1020400@zopyx.com>
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Also related to my initial question:
how to deal with broken package releases if the package maintainer(s)
don't response within a reasonable timeframe? My particular case:
the 1.0 release of 'threadframe' is hidden but visible within the simple
index. This 1.0 release is broken since the download URL actually points
to the 0.1 release of the package. This is highly confusing in the
context of zc.buildout and PyPI mirror because the 1.0 release appears
as being the most current version however no threadframe-1.0* package is
available from the download page.
Andreas
On 23.01.2009 15:09 Uhr, Andreas Jung wrote:
> Hi there,
>
> the simplex index also contains stuff belonging to hidden releases.
> Is this intentional?
>
> Example (threadframe package):
>
> http://pypi.python.org/pypi/threadframe
>
> The XMLRPC API reveals only one published version: 0.2
>
> The simple index contains also a 1.0 release which is hidden:
>
> http://pypi.python.org/simple/threadframe
>
> I suggest that the simple index should only show stuff belong to
> un-hidden releases.
>
> Andreas
>
- ------------------------------------------------------------------------
_______________________________________________
Catalog-SIG mailing list
Catalog-SIG at python.org
http://mail.python.org/mailman/listinfo/catalog-sig
- --
ZOPYX Ltd. & Co. KG - Charlottenstr. 37/1 - 72070 T?bingen - Germany
Web: www.zopyx.com - Email: info at zopyx.com - Phone +49 - 7071 - 793376
Registergericht: Amtsgericht Stuttgart, Handelsregister A 381535
Gesch?ftsf?hrer/Gesellschafter: ZOPYX Limited, Birmingham, UK
- ------------------------------------------------------------------------
E-Publishing, Python, Zope & Plone development, Consulting
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkl8STMACgkQCJIWIbr9KYwa/wCgjhc1TnF1nGDuFnweNjpXbbyU
qJ8An3DuiEyok942mKORKFV5WLEyr0aI
=uUh1
-----END PGP SIGNATURE-----
-------------- next part --------------
A non-text attachment was scrubbed...
Name: lists.vcf
Type: text/x-vcard
Size: 316 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20090125/0a4bb71d/attachment.vcf>
From lists at zopyx.com Sun Jan 25 13:50:44 2009
From: lists at zopyx.com (Andreas Jung)
Date: Sun, 25 Jan 2009 13:50:44 +0100
Subject: [Catalog-sig] [PyPI] Creation date of releases and release files?
Message-ID: <497C6024.5090509@zopyx.com>
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Would it take much effort including the creation date of packages (and
their release files) on PyPI?
Andreas
- --
ZOPYX Ltd. & Co. KG - Charlottenstr. 37/1 - 72070 T?bingen - Germany
Web: www.zopyx.com - Email: info at zopyx.com - Phone +49 - 7071 - 793376
Registergericht: Amtsgericht Stuttgart, Handelsregister A 381535
Gesch?ftsf?hrer/Gesellschafter: ZOPYX Limited, Birmingham, UK
- ------------------------------------------------------------------------
E-Publishing, Python, Zope & Plone development, Consulting
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkl8YCQACgkQCJIWIbr9KYwLGgCdHkNNG0DXnDwv97D8ioc6OvN+
hggAn2k6zNTj4O+bYWCsL3ukLexMR3z/
=ErMO
-----END PGP SIGNATURE-----
-------------- next part --------------
A non-text attachment was scrubbed...
Name: lists.vcf
Type: text/x-vcard
Size: 316 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20090125/aaee7681/attachment.vcf>
From tarek.ziade at ingeniweb.com Sun Jan 25 14:05:28 2009
From: tarek.ziade at ingeniweb.com (Tarek Ziade)
Date: Sun, 25 Jan 2009 14:05:28 +0100
Subject: [Catalog-sig] [PyPI] Creation date of releases and release
files?
In-Reply-To: <497C6024.5090509@zopyx.com>
References: <497C6024.5090509@zopyx.com>
Message-ID: <a26746990901250505m6f2f6dd2x71f3e535f97a1925@mail.gmail.com>
2009/1/25 Andreas Jung <lists at zopyx.com>:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Would it take much effort including the creation date of packages (and
> their release files) on PyPI?
I guess not because the date is already stored in the database in the
"journals" table, so this
would just require to change the UI to display it.
But maybe it would be better to include a date field in the "releases"
table imho to avoid an extra query join when
displaying release infos.
Tarek
--
Tarek Ziad? - Directeur Technique
INGENIWEB (TM) - SAS 50000 Euros - RC B 438 725 632
Bureaux de la Colline - 1 rue Royale - B?timent D - 9?me ?tage
92210 Saint Cloud - France
Phone : 01.78.15.24.00 / Fax : 01 46 02 44 04
http://www.ingeniweb.com - une soci?t? du groupe Alter Way
From lists at zopyx.com Sun Jan 25 14:19:38 2009
From: lists at zopyx.com (Andreas Jung)
Date: Sun, 25 Jan 2009 14:19:38 +0100
Subject: [Catalog-sig] [zc.buildout] Dealing with building (large) libraries
Message-ID: <497C66EA.3070109@zopyx.com>
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi there,
it had become a common pattern with buildout compiling almost all and
everything within one buildout. E.g. the Deliverance integration using
plone.recipe.deliverance downloads and compiles libxml2/libxslt which
takes a lot of time. In addition we have seen unmotivated
uninstall/install orgies of parts (possibly the related recipes are to
blame) causing a lot of turnaround time for developers (and frustration
about using buildout).
Anyone having similar experiences and/or hints how deal with such larger
buildouts? We are having a company internal sprint next week where we
are thinking about a 2-stage buildout for some of our projects where the
fat parts will be moved to a dedicated buildout configuration and
installed/maintained as as global resources. This will at least reduce
the number of pointless uninstall/install cycles.
Andreas
- --
ZOPYX Ltd. & Co. KG - Charlottenstr. 37/1 - 72070 T?bingen - Germany
Web: www.zopyx.com - Email: info at zopyx.com - Phone +49 - 7071 - 793376
Registergericht: Amtsgericht Stuttgart, Handelsregister A 381535
Gesch?ftsf?hrer/Gesellschafter: ZOPYX Limited, Birmingham, UK
- ------------------------------------------------------------------------
E-Publishing, Python, Zope & Plone development, Consulting
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkl8ZuoACgkQCJIWIbr9KYwQ6QCg5IuQ8aw+tIp7rgfBnT45A7VK
b1QAoOoFv9w0w+iYOMWtUR4BzZ2t0Qad
=bxE1
-----END PGP SIGNATURE-----
-------------- next part --------------
A non-text attachment was scrubbed...
Name: lists.vcf
Type: text/x-vcard
Size: 316 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20090125/d046a05b/attachment.vcf>
From jim at zope.com Sun Jan 25 17:42:37 2009
From: jim at zope.com (Jim Fulton)
Date: Sun, 25 Jan 2009 11:42:37 -0500
Subject: [Catalog-sig] [zc.buildout] Dealing with building (large)
libraries
In-Reply-To: <497C66EA.3070109@zopyx.com>
References: <497C66EA.3070109@zopyx.com>
Message-ID: <5EF42A12-D48F-4772-A51A-CFEF93AE1A16@zope.com>
On Jan 25, 2009, at 8:19 AM, Andreas Jung wrote:
> it had become a common pattern with buildout compiling almost all and
> everything within one buildout. E.g. the Deliverance integration using
> plone.recipe.deliverance downloads and compiles libxml2/libxslt which
> takes a lot of time. In addition we have seen unmotivated
> uninstall/install orgies of parts (possibly the related recipes are to
> blame) causing a lot of turnaround time for developers (and
> frustration
> about using buildout).
Buildout takes a conservative approach when deciding whether a part
needs to be reinstalled. In particular, a change to a part's recipe
(like a new recipe egg) or a package the recipe depends on (e.g.
buildout itself) will cause a part to be reinstalled.
> Anyone having similar experiences and/or hints how deal with such
> larger
> buildouts?
I don't think it's really a question of the size of the buildout so
much as the expense of individual parts. Many or most parts aren't
expensive to reinstall. Certain parts, like those that build a big
external library can be especially painful.
> We are having a company internal sprint next week where we
> are thinking about a 2-stage buildout for some of our projects where
> the
> fat parts will be moved to a dedicated buildout configuration and
> installed/maintained as as global resources. This will at least reduce
> the number of pointless uninstall/install cycles.
That's a reasonable approach. Another approach might be to add an
option to make buildout less conservative about certain parts. For
example, there might be an option to, for a given list of parts to
only reinstall a part if
an option changes, ignoring changes to the version of the part recipe
or it's dependencies. Alternatively, we could change buildout to use a
provided value __buildout_signature__, rather than computing one
itself if the option is provided. Then, for expensive parts, like one
building a library, once could simply provide this option, giving a
static value. I think this would be more effective that managing
separate buildouts to compute expensive parts.
Jim
--
Jim Fulton
Zope Corporation
From jim at zope.com Sun Jan 25 17:46:30 2009
From: jim at zope.com (Jim Fulton)
Date: Sun, 25 Jan 2009 11:46:30 -0500
Subject: [Catalog-sig] [PyPI] Hidden release shown on simple index
In-Reply-To: <497C4933.1020400@zopyx.com>
References: <4979CF9B.1080001@zopyx.com> <497C4933.1020400@zopyx.com>
Message-ID: <6CF36ABE-8CCC-4A6F-9F9C-08DF5FAD5263@zope.com>
On Jan 25, 2009, at 6:12 AM, Andreas Jung wrote:
> how to deal with broken package releases if the package maintainer(s)
> don't response within a reasonable timeframe? My particular case:
> the 1.0 release of 'threadframe' is hidden but visible within the
> simple
> index. This 1.0 release is broken since the download URL actually
> points
> to the 0.1 release of the package. This is highly confusing in the
> context of zc.buildout and PyPI mirror because the 1.0 release appears
> as being the most current version however no threadframe-1.0*
> package is
> available from the download page.
To whom is this confusing? I don't see how buildout or setuptools
would be confused by this situation. All they ultimately care about is
the actual distributions they find.
Jim
--
Jim Fulton
Zope Corporation
From martin at v.loewis.de Sun Jan 25 19:20:16 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 25 Jan 2009 19:20:16 +0100
Subject: [Catalog-sig] [PyPI] Creation date of releases and release
files?
In-Reply-To: <497C6024.5090509@zopyx.com>
References: <497C6024.5090509@zopyx.com>
Message-ID: <497CAD60.5060300@v.loewis.de>
Andreas Jung wrote:
> Would it take much effort including the creation date of packages (and
> their release files) on PyPI?
As Tarek says: the creation date is already available from the
changelog. The upload dates are available from the changelog also,
but more easily so directly from the file system.
So I don't feel inclined to do anything about this - if you need it, the
information is there. If you want to have it available more
conveniently, contribute patches.
Regards,
Martin
From lists at zopyx.com Sun Jan 25 19:42:24 2009
From: lists at zopyx.com (Andreas Jung)
Date: Sun, 25 Jan 2009 19:42:24 +0100
Subject: [Catalog-sig] [PyPI] Creation date of releases and release
files?
In-Reply-To: <497CAD60.5060300@v.loewis.de>
References: <497C6024.5090509@zopyx.com> <497CAD60.5060300@v.loewis.de>
Message-ID: <497CB290.7050908@zopyx.com>
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 25.01.2009 19:20 Uhr, Martin v. L?wis wrote:
> Andreas Jung wrote:
>> Would it take much effort including the creation date of packages (and
>> their release files) on PyPI?
>
> As Tarek says: the creation date is already available from the
> changelog. The upload dates are available from the changelog also,
> but more easily so directly from the file system.
>
> So I don't feel inclined to do anything about this - if you need it, the
> information is there. If you want to have it available more
> conveniently, contribute patches.
Huh? The change is likely a one-liner within the related template code
or whatever is behind the PyPI web UI - but anyway...
- -aj
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkl8spAACgkQCJIWIbr9KYx7HwCeM3UD3Dafi2+YhVVdgJyDBU/2
32cAoMr0OGxFQ5jdgso3fErCRAKOSTKo
=YMW+
-----END PGP SIGNATURE-----
-------------- next part --------------
A non-text attachment was scrubbed...
Name: lists.vcf
Type: text/x-vcard
Size: 316 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20090125/6eaaae0b/attachment.vcf>
From martin at v.loewis.de Sun Jan 25 20:24:41 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 25 Jan 2009 20:24:41 +0100
Subject: [Catalog-sig] [PyPI] Creation date of releases and release
files?
In-Reply-To: <497CB290.7050908@zopyx.com>
References: <497C6024.5090509@zopyx.com> <497CAD60.5060300@v.loewis.de>
<497CB290.7050908@zopyx.com>
Message-ID: <497CBC79.3070105@v.loewis.de>
> Huh? The change is likely a one-liner within the related template code
> or whatever is behind the PyPI web UI - but anyway...
The last change took me several our, because I had to fight ZPT;
I try to reduce that to a minimum.
Again, patches welcome.
Regards,
Martin
From adam.boduch at gmail.com Wed Jan 28 15:01:12 2009
From: adam.boduch at gmail.com (Adam Boduch)
Date: Wed, 28 Jan 2009 09:01:12 -0500
Subject: [Catalog-sig] API Documentation
Message-ID: <1233151272.6653.3.camel@adam-laptop>
Hi,
I'm just wondering how to use the hosted API documentation feature on
pypi. I followed the instructions for uploading a zip file containing
the documentation. I now get HTTP forbidden when visiting
http://packages.python.org/boduch/
Am I missing a step?
Thanks in advance,
Adam
From martin at v.loewis.de Wed Jan 28 18:44:10 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 28 Jan 2009 18:44:10 +0100
Subject: [Catalog-sig] API Documentation
In-Reply-To: <1233151272.6653.3.camel@adam-laptop>
References: <1233151272.6653.3.camel@adam-laptop>
Message-ID: <4980996A.7030702@v.loewis.de>
Adam Boduch wrote:
> Hi,
>
> I'm just wondering how to use the hosted API documentation feature on
> pypi. I followed the instructions for uploading a zip file containing
> the documentation. I now get HTTP forbidden when visiting
> http://packages.python.org/boduch/
>
> Am I missing a step?
No - you just have the URL wrong. It's
http://packages.python.org/boduch/boduch/
(apparently, your zipfile contains a single directory "boduch", and no
index.html beside it)
Regards,
Martin
From adam.boduch at gmail.com Wed Jan 28 19:10:49 2009
From: adam.boduch at gmail.com (Adam Boduch)
Date: Wed, 28 Jan 2009 13:10:49 -0500
Subject: [Catalog-sig] API Documentation
In-Reply-To: <4980996A.7030702@v.loewis.de>
References: <1233151272.6653.3.camel@adam-laptop>
<4980996A.7030702@v.loewis.de>
Message-ID: <1233166249.6489.0.camel@adam-laptop>
Excellent. Thanks so much!
On Wed, 2009-01-28 at 18:44 +0100, "Martin v. L?wis" wrote:
> Adam Boduch wrote:
> > Hi,
> >
> > I'm just wondering how to use the hosted API documentation feature on
> > pypi. I followed the instructions for uploading a zip file containing
> > the documentation. I now get HTTP forbidden when visiting
> > http://packages.python.org/boduch/
> >
> > Am I missing a step?
>
> No - you just have the URL wrong. It's
>
> http://packages.python.org/boduch/boduch/
>
> (apparently, your zipfile contains a single directory "boduch", and no
> index.html beside it)
>
> Regards,
> Martin
From martin at v.loewis.de Wed Jan 28 19:15:33 2009
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Wed, 28 Jan 2009 19:15:33 +0100
Subject: [Catalog-sig] API Documentation
In-Reply-To: <1233166249.6489.0.camel@adam-laptop>
References: <1233151272.6653.3.camel@adam-laptop> <4980996A.7030702@v.loewis.de>
<1233166249.6489.0.camel@adam-laptop>
Message-ID: <4980A0C5.6000800@v.loewis.de>
Adam Boduch wrote:
> Excellent. Thanks so much!
You are welcome. I do recommend that you change the zipfile, though, to
remove one URL level.
Regards,
Martin
From szybalski at gmail.com Wed Jan 28 19:55:36 2009
From: szybalski at gmail.com (Lukasz Szybalski)
Date: Wed, 28 Jan 2009 12:55:36 -0600
Subject: [Catalog-sig] API Documentation
In-Reply-To: <4980A0C5.6000800@v.loewis.de>
References: <1233151272.6653.3.camel@adam-laptop>
<4980996A.7030702@v.loewis.de> <1233166249.6489.0.camel@adam-laptop>
<4980A0C5.6000800@v.loewis.de>
Message-ID: <804e5c70901281055w64020000l78fbaa4086dbbb81@mail.gmail.com>
How do you upload the documentation? Is there a link somewhere that
has instructions on getting these docs in. I might use it for my
package.
Thanks,
Lucas
On Wed, Jan 28, 2009 at 12:15 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> Adam Boduch wrote:
>> Excellent. Thanks so much!
>
> You are welcome. I do recommend that you change the zipfile, though, to
> remove one URL level.
>
> Regards,
> Martin
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG at python.org
> http://mail.python.org/mailman/listinfo/catalog-sig
>
--
How to create python package?
http://lucasmanual.com/mywiki/PythonPaste
Bazaar and Launchpad
http://lucasmanual.com/mywiki/Bazaar
From adam.boduch at gmail.com Wed Jan 28 20:12:31 2009
From: adam.boduch at gmail.com (Adam Boduch)
Date: Wed, 28 Jan 2009 14:12:31 -0500
Subject: [Catalog-sig] API Documentation
In-Reply-To: <4980A0C5.6000800@v.loewis.de>
References: <1233151272.6653.3.camel@adam-laptop>
<4980996A.7030702@v.loewis.de> <1233166249.6489.0.camel@adam-laptop>
<4980A0C5.6000800@v.loewis.de>
Message-ID: <1233169951.6489.1.camel@adam-laptop>
Yep. Thats the plan, and, thanks again.
On Wed, 2009-01-28 at 19:15 +0100, "Martin v. L?wis" wrote:
> Adam Boduch wrote:
> > Excellent. Thanks so much!
>
> You are welcome. I do recommend that you change the zipfile, though, to
> remove one URL level.
>
> Regards,
> Martin
From martin at v.loewis.de Wed Jan 28 20:21:47 2009
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Wed, 28 Jan 2009 20:21:47 +0100
Subject: [Catalog-sig] API Documentation
In-Reply-To: <804e5c70901281055w64020000l78fbaa4086dbbb81@mail.gmail.com>
References: <1233151272.6653.3.camel@adam-laptop> <4980996A.7030702@v.loewis.de>
<1233166249.6489.0.camel@adam-laptop> <4980A0C5.6000800@v.loewis.de>
<804e5c70901281055w64020000l78fbaa4086dbbb81@mail.gmail.com>
Message-ID: <4980B04B.6050804@v.loewis.de>
Lukasz Szybalski wrote:
> How do you upload the documentation? Is there a link somewhere that
> has instructions on getting these docs in. I might use it for my
> package.
Just go to your package's page, and it should be all obvious.
Regards,
Martin
From szybalski at gmail.com Fri Jan 30 05:24:09 2009
From: szybalski at gmail.com (Lukasz Szybalski)
Date: Thu, 29 Jan 2009 22:24:09 -0600
Subject: [Catalog-sig] threads and xmlrpc?
In-Reply-To: <804e5c70901282208ud623b83x85bbdc403fb97c07@mail.gmail.com>
References: <804e5c70901282208ud623b83x85bbdc403fb97c07@mail.gmail.com>
Message-ID: <804e5c70901292024u39782f74wbd5701f7b7ff1e81@mail.gmail.com>
Hello,
I'm running a threaded app using some calls via xmlrpc to pypi. What
I'm trying to get is a to get a littler more responses in a shorter
time, as I see that the bandwidth used by xmlrpc calls are minimal
(<kb). The problem I run into is that connection is reset by peer
after about 10min (~500 calls). I use a single connection and a queue
of 8 threads to get the data. Would anybody have an example on how to
run xmlrpc in a thread? Do I set multiple connections, or is there a
setting to keep the connection live or reconnect if disconnected?
I've tried to catch the error but that is not working well. I don't
want to connect connect for each request, It would be best if somehow
I keep the connection live and issue multiple calls via threads.
Also, please advice if you think that somehow I am overloading your
servers. I've tasted some downloads speeds and I am sure you web
browser can accept 100+ requests per second, but what about xmlrpc?
Without threads I get <5 requests per second.
Thanks,
Lucas
Traceback (most recent call last):
File "/usr/lib/python2.5/threading.py", line 486, in __bootstrap_inner
self.run()
File "pypi.py", line 29, in run
version=pypi.package_releases(package)[0]
File "/usr/lib/python2.5/xmlrpclib.py", line 1147, in __call__
return self.__send(self.__name, args)
File "/usr/lib/python2.5/xmlrpclib.py", line 1437, in __request
verbose=self.__verbose
File "/usr/lib/python2.5/xmlrpclib.py", line 1185, in request
errcode, errmsg, headers = h.getreply()
File "/usr/lib/python2.5/httplib.py", line 1199, in getreply
response = self._conn.getresponse()
File "/usr/lib/python2.5/httplib.py", line 928, in getresponse
response.begin()
File "/usr/lib/python2.5/httplib.py", line 385, in begin
version, status, reason = self._read_status()
File "/usr/lib/python2.5/httplib.py", line 343, in _read_status
line = self.fp.readline()
File "/usr/lib/python2.5/socket.py", line 372, in readline
data = recv(1)
error: (104, 'Connection reset by peer')
--
How to create python package?
http://lucasmanual.com/mywiki/PythonPaste
Bazaar and Launchpad
http://lucasmanual.com/mywiki/Bazaar
--
How to create python package?
http://lucasmanual.com/mywiki/PythonPaste
Bazaar and Launchpad
http://lucasmanual.com/mywiki/Bazaar
From martin at v.loewis.de Fri Jan 30 06:44:46 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 30 Jan 2009 06:44:46 +0100
Subject: [Catalog-sig] threads and xmlrpc?
In-Reply-To: <804e5c70901292024u39782f74wbd5701f7b7ff1e81@mail.gmail.com>
References: <804e5c70901282208ud623b83x85bbdc403fb97c07@mail.gmail.com>
<804e5c70901292024u39782f74wbd5701f7b7ff1e81@mail.gmail.com>
Message-ID: <498293CE.4070606@v.loewis.de>
> I'm running a threaded app using some calls via xmlrpc to pypi. What
> I'm trying to get is a to get a littler more responses in a shorter
> time, as I see that the bandwidth used by xmlrpc calls are minimal
> (<kb). The problem I run into is that connection is reset by peer
> after about 10min (~500 calls). I use a single connection and a queue
> of 8 threads to get the data. Would anybody have an example on how to
> run xmlrpc in a thread? Do I set multiple connections, or is there a
> setting to keep the connection live or reconnect if disconnected?
Using threads will not at all make it faster to communicate over a
single connection. For a single connection, all communication must
be serialized; you cannot issue a new request until the previous
request has completed. So you might as well just issue the requests
from a single thread.
> Also, please advice if you think that somehow I am overloading your
> servers. I've tasted some downloads speeds and I am sure you web
> browser can accept 100+ requests per second, but what about xmlrpc?
> Without threads I get <5 requests per second.
I think 5 requests per second is fairly fast.
Regards,
Martin
From szybalski at gmail.com Sat Jan 31 06:46:28 2009
From: szybalski at gmail.com (Lukasz Szybalski)
Date: Fri, 30 Jan 2009 23:46:28 -0600
Subject: [Catalog-sig] threads and xmlrpc?
In-Reply-To: <498293CE.4070606@v.loewis.de>
References: <804e5c70901282208ud623b83x85bbdc403fb97c07@mail.gmail.com>
<804e5c70901292024u39782f74wbd5701f7b7ff1e81@mail.gmail.com>
<498293CE.4070606@v.loewis.de>
Message-ID: <804e5c70901302146v43bb982bx222fd785bdff8af5@mail.gmail.com>
On Thu, Jan 29, 2009 at 11:44 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
>> I'm running a threaded app using some calls via xmlrpc to pypi. What
>> I'm trying to get is a to get a littler more responses in a shorter
>> time, as I see that the bandwidth used by xmlrpc calls are minimal
>> (<kb). The problem I run into is that connection is reset by peer
>> after about 10min (~500 calls). I use a single connection and a queue
>> of 8 threads to get the data. Would anybody have an example on how to
>> run xmlrpc in a thread? Do I set multiple connections, or is there a
>> setting to keep the connection live or reconnect if disconnected?
>
> Using threads will not at all make it faster to communicate over a
> single connection. For a single connection, all communication must
> be serialized; you cannot issue a new request until the previous
> request has completed. So you might as well just issue the requests
> from a single thread.
>
>> Also, please advice if you think that somehow I am overloading your
>> servers. I've tasted some downloads speeds and I am sure you web
>> browser can accept 100+ requests per second, but what about xmlrpc?
>> Without threads I get <5 requests per second.
>
> I think 5 requests per second is fairly fast.
>
Its more like 2 requests per second.
If I set it to 2 threads I can list each package version in about an
hour, but I lost connection when I was at a z packages.
If I used 5-8 I can get half way in about 25min but I lose connection.
("Connection reset by peer")
Would you know how can I issue more requests, and/or increase the
number of connections?
I know "http://www.faqs.org/rfcs/rfc2068.html See section 8.1.4.
The RFC says "should limit 2 connections per server" and a lot of http
client libraries obey this."
Does xmlrpc lib used by pypi does the same?
Does pypi use http://docs.python.org/library/xmlrpclib.html#multicall-objects
This is my last try. I was hoping that I can increase the number of
connections to at least 10/second ~20min but I can't seem to find any
performance increases on xmlrpc.
Is there another way to get:
pypi.list_packages()
pypi.package_releases('xyz')
pypi.release_data(' xyz' ,' 0.7.79dev' )
If not then I guess I will go back to the regular for loop and loop
through all the records in a serialized manner. (Its been 1h 15min and
I am on packages starting with letter R.)
Cpickle file coming soon for the metadata available in release_data
for all packages.
Thanks,
Lucas