From monitor at jacobian.org  Thu Sep  1 09:57:21 2011
From: monitor at jacobian.org (monitor at jacobian.org)
Date: Thu, 01 Sep 2011 02:57:21 -0500
Subject: [Catalog-sig] [monit] c.pypi.python.org - Connection failed
Message-ID: <1314863845.1@jacobian.org>

Connection failed Service c.pypi.python.org 

	Date:        Thu, 01 Sep 2011 02:57:21 -0500
	Action:      alert
	Host:        jacobian.org
	Description: failed, cannot open a connection to INET[c.pypi.python.org:80] via TCP

Your faithful employee,
monit


From monitor at jacobian.org  Thu Sep  1 10:14:42 2011
From: monitor at jacobian.org (monitor at jacobian.org)
Date: Thu, 01 Sep 2011 03:14:42 -0500
Subject: [Catalog-sig] [monit] c.pypi.python.org - Connection succeeded
Message-ID: <1314864884.1@jacobian.org>

Connection succeeded Service c.pypi.python.org 

	Date:        Thu, 01 Sep 2011 03:14:41 -0500
	Action:      alert
	Host:        jacobian.org
	Description: connection succeeded to INET[c.pypi.python.org:80] via TCP

Your faithful employee,
monit


From richard at python.org  Fri Sep  2 04:49:08 2011
From: richard at python.org (Richard Jones)
Date: Fri, 2 Sep 2011 12:49:08 +1000
Subject: [Catalog-sig] Fwd: Cannot receive RSS feeds on latest updates
	to Packages
In-Reply-To: <B58FACD3-0C3D-4C2E-952E-6247055FC341@voidspace.org.uk>
References: <CAL_Nh-zpgNYnUWTZWSpZMxXm-NJ-Wmr1XQujGWB3w+AoYOHVfw@mail.gmail.com>
	<B58FACD3-0C3D-4C2E-952E-6247055FC341@voidspace.org.uk>
Message-ID: <CAHrZfZBWSB+dDeXv9-4Hd9=nWdu2TZFJmGW1Gs6=XoHjs=JYaw@mail.gmail.com>

>From Melton Low <melton.low at gmail.com>:
> I am on a Mac running Safari. ?In the last few days, I get notifications of
> new updates to the Packages on my link to the RSS URL?PyPI recent updates.
> ?When I clink on my bookmarked link, no new items are downloaded.
> The RSS feed on new additions to the Packages works fine.

I can't see a problem. Please confirm the two urls you're referring to are:

not working: http://pypi.python.org/pypi?%3Aaction=rss
working: http://pypi.python.org/pypi?%3Aaction=packages_rss

Looking at the RSS being generated it looks fine - plenty of recent
content there. The last-modified and etag information appears to be
correct as well, so caching is *probably* not an issue...


    Richard

From monitor at jacobian.org  Thu Sep  1 10:34:47 2011
From: monitor at jacobian.org (monitor at jacobian.org)
Date: Thu, 01 Sep 2011 03:34:47 -0500
Subject: [Catalog-sig] [monit] c.pypi.python.org - Connection succeeded
Message-ID: <1314866089.1@jacobian.org>

Connection succeeded Service c.pypi.python.org 

	Date:        Thu, 01 Sep 2011 03:34:47 -0500
	Action:      alert
	Host:        jacobian.org
	Description: connection succeeded to INET[c.pypi.python.org:80] via TCP

Your faithful employee,
monit


From monitor at jacobian.org  Thu Sep  1 10:26:16 2011
From: monitor at jacobian.org (monitor at jacobian.org)
Date: Thu, 01 Sep 2011 03:26:16 -0500
Subject: [Catalog-sig] [monit] c.pypi.python.org - Connection failed
Message-ID: <1314865579.1@jacobian.org>

Connection failed Service c.pypi.python.org 

	Date:        Thu, 01 Sep 2011 03:26:16 -0500
	Action:      alert
	Host:        jacobian.org
	Description: failed, cannot open a connection to INET[c.pypi.python.org:80] via TCP

Your faithful employee,
monit


From monitor at jacobian.org  Fri Sep  9 18:59:16 2011
From: monitor at jacobian.org (monitor at jacobian.org)
Date: Fri, 09 Sep 2011 11:59:16 -0500
Subject: [Catalog-sig] [monit] d.pypi.python.org - Connection succeeded
Message-ID: <1315587557.1@jacobian.org>

Connection succeeded Service d.pypi.python.org 

	Date:        Fri, 09 Sep 2011 11:59:16 -0500
	Action:      alert
	Host:        jacobian.org
	Description: connection succeeded to INET[d.pypi.python.org:80] via TCP

Your faithful employee,
monit


From monitor at jacobian.org  Fri Sep  9 18:45:28 2011
From: monitor at jacobian.org (monitor at jacobian.org)
Date: Fri, 09 Sep 2011 11:45:28 -0500
Subject: [Catalog-sig] [monit] d.pypi.python.org - Connection failed
Message-ID: <1315586731.1@jacobian.org>

Connection failed Service d.pypi.python.org 

	Date:        Fri, 09 Sep 2011 11:45:28 -0500
	Action:      alert
	Host:        jacobian.org
	Description: failed, cannot open a connection to INET[d.pypi.python.org:80] via TCP

Your faithful employee,
monit


From djay at pretaweb.com  Mon Sep 12 10:43:11 2011
From: djay at pretaweb.com (Dylan Jay)
Date: Mon, 12 Sep 2011 18:43:11 +1000
Subject: [Catalog-sig] parsing setuptools style requires.txt
Message-ID: <25305383-BC7D-4626-9399-A9D2F7D8D0EB@pretaweb.com>

Hi,

A recent request[1] (which I didn't submit) was rejected with the  
following comment:

"Assuming you are talking about setuptools-style dependencies: this  
has been
rejected by the community, as hurting PEP 345. PEP-345-style  
dependencies
are already parsed and provided. If you want to discuss further, post to
catalog-sig."

I had a search of the archives and couldn't locate the discussion,  
unless it's this one [2] which seemed to indicate that there was a  
suitable way to publish both pep345 compatible requirements as well as  
PIP and setuptools requirements via PYPI.

It strikes me that
1. if someone is prepared to write a patch to pypi to handle  
setuptools style requirements
2. if there is a lot of packages out there and will continue to be for  
a long time, that have setuptools style
   dependency specification
3. if PEP345 isn't implemented in any tools yet (excuse my ignorance.  
I'm assuming PEP345 tool support is in distutils2 and that isn't  
finished yet)

then why not allow a change to parsing of setuptools style dependencies?
or perhaps point me to the discussion that explains what I'm missing.

Its also been mentioned that in order to make this parsing work you  
need to run setup.py to get the requires.txt for setuptools packages  
and this is a security concern. However many packages already have the  
egg-info commend run before upload so there is no need to run  
setup.py. For those packages where there is a need I think security  
concerns could be overcome with the use of the restrictedpython  
package. Anything trying to import anything but the bare minimum is  
skipped.

My interest in this is the idea that we could get distutils2 and/or  
zc.buildout to be able to download regular updates of metadata  
including dependencies, then perhaps those tools could avoid certain  
kinds of conflict errors which are a pain to debug without that  
information. For instance, the current design of zc.buildout means that:

Download Bob. Bob 1.0 requires Fred >= 2.0.
Download Fred 3.0
Download Marry. Marry 1.0 requires Fred < 2.5
Conflict error. Marry 1.0 requires Fred < 2.5 but we already have Fred  
3.0.

If instead we knew in advance of this conflict we could have chosen to  
download Fred 2.4 or at least warned the user there was a potential  
conflict and they should pick a compatible version. In the case of  
preinstalled packages, it could offer to downgrade Fred from 3.0 to 2.4.


[1] http://sourceforge.net/tracker/?func=detail&aid=3407539&group_id=66150&atid=513503
[2] http://mail.python.org/pipermail/catalog-sig/2011-January/003452.html



---
Dylan Jay
Technical Solutions Manager
PretaWeb: reducing duplication in the government web.
P: +612 80819071 | M: +61421477460 | twitter.com/djay75 | linkedin.com/ 
in/djay75


From alexis at notmyidea.org  Mon Sep 12 14:01:43 2011
From: alexis at notmyidea.org (=?ISO-8859-1?Q?Alexis_M=E9taireau?=)
Date: Mon, 12 Sep 2011 14:01:43 +0200
Subject: [Catalog-sig] parsing setuptools style requires.txt
In-Reply-To: <25305383-BC7D-4626-9399-A9D2F7D8D0EB@pretaweb.com>
References: <25305383-BC7D-4626-9399-A9D2F7D8D0EB@pretaweb.com>
Message-ID: <4E6DF4A7.80900@notmyidea.org>

Hi Dylan,

On 09/12/2011 10:43 AM, Dylan Jay wrote:
> ts also been mentioned that in order to make this parsing work you need
> to run setup.py to get the requires.txt for setuptools packages and this
> is a security concern. However many packages already have the egg-info
> commend run before upload so there is no need to run setup.py. For those
> packages where there is a need I think security concerns could be
> overcome with the use of the restrictedpython package. Anything trying
> to import anything but the bare minimum is skipped.

One problem I can think about is the fact that it is not possible to get 
platform independent information coming from a setup.py, in the sense 
that setup.py is executed and can provide different metadata regarding 
the platform the setup.py is executing onto. PEP 345 and environment 
markers aims to resolve this problem.

Bu, what are you trying to do exacty? Do you want to get the list of 
dependencies coming from setuptools? If yes, then the way we are doing 
it for the setuptools compatibility layer in distutils2/packaging is the 
following:

1. monkey-patch setuptools.setup with your own setup, storing the given 
metadata somewhere
2. run the setup.py which will call this monkey patched setup function

This has been implemented here: 
http://bazaar.launchpad.net/~anybox/+junk/anybox.recipe.openerp/view/head:/anybox/recipe/openerp/__init__.py#L53

I agree this is not as easy as it could be with uploading metadata to 
pypi (as it will be with distutils2 and PEP 345) but having platform 
specific dependent metadata information on PyPI is only half answering 
the problem, so the solution is probably for now to do this on the 
client side.

Hope this helps.
-- 
Alexis

From ziade.tarek at gmail.com  Mon Sep 12 15:01:19 2011
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 12 Sep 2011 15:01:19 +0200
Subject: [Catalog-sig] parsing setuptools style requires.txt
In-Reply-To: <25305383-BC7D-4626-9399-A9D2F7D8D0EB@pretaweb.com>
References: <25305383-BC7D-4626-9399-A9D2F7D8D0EB@pretaweb.com>
Message-ID: <CAGSi+Q50hb=po9zhhxQBtkdbzjpBaR8jpinmReLRA5aEVow7Aw@mail.gmail.com>

On Mon, Sep 12, 2011 at 10:43 AM, Dylan Jay <djay at pretaweb.com> wrote:
...
> then why not allow a change to parsing of setuptools style dependencies?
> or perhaps point me to the discussion that explains what I'm missing.
>
> Its also been mentioned that in order to make this parsing work you need to
> run setup.py to get the requires.txt for setuptools packages and this is a
> security concern. However many packages already have the egg-info commend
> run before upload so there is no need to run setup.py. For those packages
> where there is a need I think security concerns could be overcome with the
> use of the restrictedpython package. Anything trying to import anything but
> the bare minimum is skipped.

the egg-info produced by setuptools is dependent on the platform it's
run on, so running on the target client is what you need to get the
right values for sure.

with PEP 345-style metadata, you have ways to define static
per-platform metadata.

In fact, one of the goal is to have setup.cfg published at pypi


>
> My interest in this is the idea that we could get distutils2 and/or
> zc.buildout to be able to download regular updates of metadata including
> dependencies, then perhaps those tools could avoid certain kinds of conflict
> errors which are a pain to debug without that information. For instance, the
> current design of zc.buildout means that:
>
> Download Bob. Bob 1.0 requires Fred >= 2.0.
> Download Fred 3.0
> Download Marry. Marry 1.0 requires Fred < 2.5
> Conflict error. Marry 1.0 requires Fred < 2.5 but we already have Fred 3.0.
>
> If instead we knew in advance of this conflict we could have chosen to
> download Fred 2.4 or at least warned the user there was a potential conflict
> and they should pick a compatible version. In the case of preinstalled
> packages, it could offer to downgrade Fred from 3.0 to 2.4.

I think Pip solves this by downloading all dependencies beforehand, no
? and it works for all platforms.

The bottom line is that spending efforts on adding on server side
platform-specific metadata that are not guaranteed to be accurate for
your target platform, is not worth it since you can already do
something accurate on client side at this time.

PEP 345 will provide an improvement in the long term since you will be
able to read dependencies, but also the file list, without having to
download the package.

IOW, installers will speed up because they will be able to build
dependency graphs on the fly -- as long as all packages are PEP 345,
which won't happen before a while.

Cheers
Tarek

-- 
Tarek Ziad? | http://ziade.org

From jim at zope.com  Mon Sep 12 15:31:17 2011
From: jim at zope.com (Jim Fulton)
Date: Mon, 12 Sep 2011 09:31:17 -0400
Subject: [Catalog-sig] parsing setuptools style requires.txt
In-Reply-To: <25305383-BC7D-4626-9399-A9D2F7D8D0EB@pretaweb.com>
References: <25305383-BC7D-4626-9399-A9D2F7D8D0EB@pretaweb.com>
Message-ID: <CAPDm-Fi+Sq3VkY+=xT1e6CjSJ-mBnFwiE=sYWYoDWg9yynKFWw@mail.gmail.com>

On Mon, Sep 12, 2011 at 4:43 AM, Dylan Jay <djay at pretaweb.com> wrote:
...
> My interest in this is the idea that we could get distutils2 and/or
> zc.buildout to be able to download regular updates of metadata including
> dependencies, then perhaps those tools could avoid certain kinds of conflict
> errors which are a pain to debug without that information. For instance, the
> current design of zc.buildout means that:
>
> Download Bob. Bob 1.0 requires Fred >= 2.0.
> Download Fred 3.0
> Download Marry. Marry 1.0 requires Fred < 2.5
> Conflict error. Marry 1.0 requires Fred < 2.5 but we already have Fred 3.0.

> If instead we knew in advance of this conflict we could have chosen to
> download Fred 2.4 or at least warned the user there was a potential conflict
> and they should pick a compatible version. In the case of preinstalled
> packages, it could offer to downgrade Fred from 3.0 to 2.4.

The issue here isn't that buildout has to download packages
to get meta data. The problem is that it doesn't backtrack.  This is
compounded by fact that when computing and fetching
requirements, it uses a depth-first strategy and doesn't use
information about requirements of other packages in the working
set until too late.  There's definitely room for improvement in the
strategy it uses, even short of backtracking.

In any case, allowing buildout to get requirements info before
downloading packages would be a useful optimization, but wouldn't
solve the conflict problem.

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton

From james.p.turk at gmail.com  Tue Sep 13 00:37:48 2011
From: james.p.turk at gmail.com (James Turk)
Date: Mon, 12 Sep 2011 18:37:48 -0400
Subject: [Catalog-sig] change package ownership on some packages
Message-ID: <CAOaGJB9BcOc2dXizmVTo5y-0bhsNThsf3QQ0zyAyRh39QKmFwA@mail.gmail.com>

Hi,

I'm hoping that I can get ownership of a few packages that were uploaded to
PyPI by my coworker that has departed and is currently out of contact (due
to personal reasons, I can elaborate but would prefer not to on a public
mailing list for his privacy).

Due to the sudden nature of his departure this was one of the things
overlooked.

The packages in question are:

The packages in question are:
  http://pypi.python.org/pypi/scrapelib
  http://pypi.python.org/pypi/jellyfish
  http://pypi.python.org/pypi/python-openstates
  http://pypi.python.org/pypi/nimsp
  http://pypi.python.org/pypi/name_tools

You'll see that I'm listed (sometimes as jturk at sunlightfoundation.com) as
co-author on most of these, they are also all hosted on the sunlightlabs
github account (http://github.com/sunlightlabs), for which I'm one of the
owners.  I'd be happy to provide any additional verification that would be
helpful to assert both of these facts.

My pypi name is jamesturk, if that account could be added as a maintainer on
these 5 I'd greatly appreciate it.

Thanks,

James
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20110912/e043f940/attachment.html>

From djay at pretaweb.com  Tue Sep 13 03:48:15 2011
From: djay at pretaweb.com (Dylan Jay)
Date: Tue, 13 Sep 2011 11:48:15 +1000
Subject: [Catalog-sig] parsing setuptools style requires.txt
In-Reply-To: <CAPDm-Fi+Sq3VkY+=xT1e6CjSJ-mBnFwiE=sYWYoDWg9yynKFWw@mail.gmail.com>
References: <25305383-BC7D-4626-9399-A9D2F7D8D0EB@pretaweb.com>
	<CAPDm-Fi+Sq3VkY+=xT1e6CjSJ-mBnFwiE=sYWYoDWg9yynKFWw@mail.gmail.com>
Message-ID: <E962F96F-5722-465A-9AFE-01C0864B7E48@pretaweb.com>

On 12/09/2011, at 11:31 PM, Jim Fulton wrote:

> On Mon, Sep 12, 2011 at 4:43 AM, Dylan Jay <djay at pretaweb.com> wrote:
> ...
>> My interest in this is the idea that we could get distutils2 and/or
>> zc.buildout to be able to download regular updates of metadata  
>> including
>> dependencies, then perhaps those tools could avoid certain kinds of  
>> conflict
>> errors which are a pain to debug without that information. For  
>> instance, the
>> current design of zc.buildout means that:
>>
>> Download Bob. Bob 1.0 requires Fred >= 2.0.
>> Download Fred 3.0
>> Download Marry. Marry 1.0 requires Fred < 2.5
>> Conflict error. Marry 1.0 requires Fred < 2.5 but we already have  
>> Fred 3.0.
>
>> If instead we knew in advance of this conflict we could have chosen  
>> to
>> download Fred 2.4 or at least warned the user there was a potential  
>> conflict
>> and they should pick a compatible version. In the case of  
>> preinstalled
>> packages, it could offer to downgrade Fred from 3.0 to 2.4.
>
> The issue here isn't that buildout has to download packages
> to get meta data. The problem is that it doesn't backtrack.  This is
> compounded by fact that when computing and fetching
> requirements, it uses a depth-first strategy and doesn't use
> information about requirements of other packages in the working
> set until too late.  There's definitely room for improvement in the
> strategy it uses, even short of backtracking.

true but to do backtracking properly could result in downloading many  
different available versions of potentially large packages which isn't  
so nice.


>
> In any case, allowing buildout to get requirements info before
> downloading packages would be a useful optimization, but wouldn't
> solve the conflict problem.

yes, backtracking would :) or at least it would solve some of them. Do  
you think it's doable even if there's no pypi api to get requirements?
I think this would be a great feature as conflict errors are very  
frustrating for new users. They also tend come up in buildout more  
than any other tool due to the way to it's normally used and new users  
tend to blame buildout for this.

>
> Jim
>
> -- 
> Jim Fulton
> http://www.linkedin.com/in/jimfulton


From djay at pretaweb.com  Tue Sep 13 03:53:37 2011
From: djay at pretaweb.com (Dylan Jay)
Date: Tue, 13 Sep 2011 11:53:37 +1000
Subject: [Catalog-sig] parsing setuptools style requires.txt
In-Reply-To: <CAGSi+Q50hb=po9zhhxQBtkdbzjpBaR8jpinmReLRA5aEVow7Aw@mail.gmail.com>
References: <25305383-BC7D-4626-9399-A9D2F7D8D0EB@pretaweb.com>
	<CAGSi+Q50hb=po9zhhxQBtkdbzjpBaR8jpinmReLRA5aEVow7Aw@mail.gmail.com>
Message-ID: <F93A509F-F047-4F3E-A173-61E3421E3626@pretaweb.com>

On 12/09/2011, at 11:01 PM, Tarek Ziad? wrote:

> On Mon, Sep 12, 2011 at 10:43 AM, Dylan Jay <djay at pretaweb.com> wrote:
> ...
>> then why not allow a change to parsing of setuptools style  
>> dependencies?
>> or perhaps point me to the discussion that explains what I'm missing.
>>
>> Its also been mentioned that in order to make this parsing work you  
>> need to
>> run setup.py to get the requires.txt for setuptools packages and  
>> this is a
>> security concern. However many packages already have the egg-info  
>> commend
>> run before upload so there is no need to run setup.py. For those  
>> packages
>> where there is a need I think security concerns could be overcome  
>> with the
>> use of the restrictedpython package. Anything trying to import  
>> anything but
>> the bare minimum is skipped.
>
> the egg-info produced by setuptools is dependent on the platform it's
> run on, so running on the target client is what you need to get the
> right values for sure.

true. but I'd guess that 99% of eggs have the same set of requirements  
regardless of platform.
And if we had someway to label these requirements on pypi as "dodgy- 
setuptools" and should be used only as a hint rather the exact set of  
requirements for your particular platform?


>
> with PEP 345-style metadata, you have ways to define static
> per-platform metadata.
>
> In fact, one of the goal is to have setup.cfg published at pypi
>
>
>>
>> My interest in this is the idea that we could get distutils2 and/or
>> zc.buildout to be able to download regular updates of metadata  
>> including
>> dependencies, then perhaps those tools could avoid certain kinds of  
>> conflict
>> errors which are a pain to debug without that information. For  
>> instance, the
>> current design of zc.buildout means that:
>>
>> Download Bob. Bob 1.0 requires Fred >= 2.0.
>> Download Fred 3.0
>> Download Marry. Marry 1.0 requires Fred < 2.5
>> Conflict error. Marry 1.0 requires Fred < 2.5 but we already have  
>> Fred 3.0.
>>
>> If instead we knew in advance of this conflict we could have chosen  
>> to
>> download Fred 2.4 or at least warned the user there was a potential  
>> conflict
>> and they should pick a compatible version. In the case of  
>> preinstalled
>> packages, it could offer to downgrade Fred from 3.0 to 2.4.
>
> I think Pip solves this by downloading all dependencies beforehand, no
> ? and it works for all platforms.

yes it can be solved by downloading all dependencies but you'd have to  
download all versions of all dependencies potentially to avoid a  
conflict in the worst case.

>
> The bottom line is that spending efforts on adding on server side
> platform-specific metadata that are not guaranteed to be accurate for
> your target platform, is not worth it since you can already do
> something accurate on client side at this time.

see above.


>
> PEP 345 will provide an improvement in the long term since you will be
> able to read dependencies, but also the file list, without having to
> download the package.
>
> IOW, installers will speed up because they will be able to build
> dependency graphs on the fly -- as long as all packages are PEP 345,
> which won't happen before a while.

can't wait, but in the meantime I can't see why installers can't use  
these hints to speed things up as long as they use it with the caveat  
that requirements could potentially differ once actually installed.

>
> Cheers
> Tarek
>
> -- 
> Tarek Ziad? | http://ziade.org


From jim at zope.com  Tue Sep 13 15:33:08 2011
From: jim at zope.com (Jim Fulton)
Date: Tue, 13 Sep 2011 09:33:08 -0400
Subject: [Catalog-sig] parsing setuptools style requires.txt
In-Reply-To: <E962F96F-5722-465A-9AFE-01C0864B7E48@pretaweb.com>
References: <25305383-BC7D-4626-9399-A9D2F7D8D0EB@pretaweb.com>
	<CAPDm-Fi+Sq3VkY+=xT1e6CjSJ-mBnFwiE=sYWYoDWg9yynKFWw@mail.gmail.com>
	<E962F96F-5722-465A-9AFE-01C0864B7E48@pretaweb.com>
Message-ID: <CAPDm-FgMiBuWBzb6puXeMUz6bez4t2OW4zGqgfrmtP=X3PkoXA@mail.gmail.com>

On Mon, Sep 12, 2011 at 9:48 PM, Dylan Jay <djay at pretaweb.com> wrote:
> On 12/09/2011, at 11:31 PM, Jim Fulton wrote:
>
>> On Mon, Sep 12, 2011 at 4:43 AM, Dylan Jay <djay at pretaweb.com> wrote:

...

>> In any case, allowing buildout to get requirements info before
>> downloading packages would be a useful optimization, but wouldn't
>> solve the conflict problem.
>
> yes, backtracking would :) or at least it would solve some of them. Do you
> think it's doable even if there's no pypi api to get requirements?

I'm sure backtracking is doable. I suspect it's rather hard. In any case,
it's orthogonal to avoiding downloading to get requirements.

> I think this would be a great feature as conflict errors are very
> frustrating for new users. They also tend come up in buildout more than any
> other tool due to the way to it's normally used and new users tend to blame
> buildout for this.

Maybe. In the interest of being incremental, I think it would be better
start with improving the current algorithm short of backtracking.

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton

From ziade.tarek at gmail.com  Tue Sep 13 17:24:06 2011
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 13 Sep 2011 17:24:06 +0200
Subject: [Catalog-sig] parsing setuptools style requires.txt
In-Reply-To: <F93A509F-F047-4F3E-A173-61E3421E3626@pretaweb.com>
References: <25305383-BC7D-4626-9399-A9D2F7D8D0EB@pretaweb.com>
	<CAGSi+Q50hb=po9zhhxQBtkdbzjpBaR8jpinmReLRA5aEVow7Aw@mail.gmail.com>
	<F93A509F-F047-4F3E-A173-61E3421E3626@pretaweb.com>
Message-ID: <CAGSi+Q4F1E3qOXo9YaMjJtfYc54Mvu5dRN60Tz_WK0b3t+nZsQ@mail.gmail.com>

On Tue, Sep 13, 2011 at 3:53 AM, Dylan Jay <djay at pretaweb.com> wrote:
> On 12/09/2011, at 11:01 PM, Tarek Ziad? wrote:
>
>> On Mon, Sep 12, 2011 at 10:43 AM, Dylan Jay <djay at pretaweb.com> wrote:
>> ...
>>>
>>> then why not allow a change to parsing of setuptools style dependencies?
>>> or perhaps point me to the discussion that explains what I'm missing.
>>>
>>> Its also been mentioned that in order to make this parsing work you need
>>> to
>>> run setup.py to get the requires.txt for setuptools packages and this is
>>> a
>>> security concern. However many packages already have the egg-info commend
>>> run before upload so there is no need to run setup.py. For those packages
>>> where there is a need I think security concerns could be overcome with
>>> the
>>> use of the restrictedpython package. Anything trying to import anything
>>> but
>>> the bare minimum is skipped.
>>
>> the egg-info produced by setuptools is dependent on the platform it's
>> run on, so running on the target client is what you need to get the
>> right values for sure.
>
> true. but I'd guess that 99% of eggs have the same set of requirements
> regardless of platform.

where did you get this percentage from ?

I see this pattern more often than in 1% of the cases:

if sys.platform == 'win32':
    reqs.append('pywin32')

(I have more examples)


> And if we had someway to label these requirements on pypi as
> "dodgy-setuptools" and should be used only as a hint rather the exact set of
> requirements for your particular platform?

how would you detect them ?

...
>> I think Pip solves this by downloading all dependencies beforehand, no
>> ? and it works for all platforms.
>
> yes it can be solved by downloading all dependencies but you'd have to
> download all versions of all dependencies potentially to avoid a conflict in
> the worst case.
>
>>
>> The bottom line is that spending efforts on adding on server side
>> platform-specific metadata that are not guaranteed to be accurate for
>> your target platform, is not worth it since you can already do
>> something accurate on client side at this time.
>
> see above.
>
>
>>
>> PEP 345 will provide an improvement in the long term since you will be
>> able to read dependencies, but also the file list, without having to
>> download the package.
>>
>> IOW, installers will speed up because they will be able to build
>> dependency graphs on the fly -- as long as all packages are PEP 345,
>> which won't happen before a while.
>
> can't wait, but in the meantime I can't see why installers can't use these
> hints to speed things up as long as they use it with the caveat that
> requirements could potentially differ once actually installed.

a speed up with a potential non-working installation, does not sound
like a good idea me.

now, installers could keep a cache of platform-specific metadata on
the client side to build a dependency graph that's looked up and
completed on any new installation. That would save some calls and
downloads.

Cheers
Tarek

-- 
Tarek Ziad? | http://ziade.org

From techtonik at gmail.com  Wed Sep 14 14:12:35 2011
From: techtonik at gmail.com (anatoly techtonik)
Date: Wed, 14 Sep 2011 15:12:35 +0300
Subject: [Catalog-sig] PyPI in Mercurial,
	conversion cleanup and state of Selenium tests
Message-ID: <CAPkN8xJVh2vbDJFRUNW1T0he3ois57L6w+L40uoTOZ=p=9FQ-g@mail.gmail.com>

Hi,

Please, CC.

I've converted PyPI repository to Mercurial to enable public posting
of changes to BitBucket or Google Code. You can find it at
https://bitbucket.org/techtonik/pypi
If people like to use Mercurial for PyPI development, it may worth to
switch completely. For that purpose it is worthy to cleanup the clone.

I've noticed a pypi_selenium/ dir dated 2005, which contains whole
JUnit with Xerces binary.What is the status of this stuff?
http://wiki.python.org/moin/CheeseShopDev is silent about any testing
at all.

It is also interesting to know about status of these branches:
egginfo                     1675:b5ea39a3f498
tarek-pep-345-support       1455:cbf5f43e3917
tarek-pypi                  1114:1db82f5e78b2
name-mangling                767:e6d118808d09
redesign                     694:88eb8de4246c
MichaelTwomey                466:c56de66b3da2
richard                        4:84a72b2a8497
pypi                        1672:fa3073ebc9a9 (inactive)

--
anatoly t.

From drkjam at gmail.com  Wed Sep 14 14:36:57 2011
From: drkjam at gmail.com (drkjam)
Date: Wed, 14 Sep 2011 13:36:57 +0100
Subject: [Catalog-sig] PyPI trove classifiers for alternate language
	implementations
References: <CADiSq7dNhexnM56-u3W1Y_i3u1vAdHrFBtFVLbrri0JXLVf1XQ@mail.gmail.com>
Message-ID: <DF04F544-0CA8-426F-A9F2-DAFA7EBCDC14@gmail.com>

Would it be possible to have trove classifiers added to PyPI specifically for PyPy and possibly also Jython and IronPython?

Regards,

David Moss
> 

From martin at v.loewis.de  Wed Sep 14 15:22:13 2011
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 14 Sep 2011 15:22:13 +0200
Subject: [Catalog-sig] PyPI trove classifiers for alternate
	language	implementations
In-Reply-To: <DF04F544-0CA8-426F-A9F2-DAFA7EBCDC14@gmail.com>
References: <CADiSq7dNhexnM56-u3W1Y_i3u1vAdHrFBtFVLbrri0JXLVf1XQ@mail.gmail.com>
	<DF04F544-0CA8-426F-A9F2-DAFA7EBCDC14@gmail.com>
Message-ID: <4E70AA85.6090700@v.loewis.de>

Am 14.09.11 14:36, schrieb drkjam:
> Would it be possible to have trove classifiers added to PyPI specifically for PyPy and possibly also Jython and IronPython?

Here's the standard set of questions for proposers of new classifiers:

* what specific classifiers are you proposing?
* what specific semantics would you imply for packages tagged with these 
classifiers?
* what specific packages that are already on PyPI would be tagged with
these specific classifiers?

Regards,
Martin

From fuzzyman at gmail.com  Thu Sep 15 13:55:42 2011
From: fuzzyman at gmail.com (Michael Foord)
Date: Thu, 15 Sep 2011 12:55:42 +0100
Subject: [Catalog-sig] PyPI trove classifiers for alternate language
	implementations
In-Reply-To: <4E70AA85.6090700@v.loewis.de>
References: <CADiSq7dNhexnM56-u3W1Y_i3u1vAdHrFBtFVLbrri0JXLVf1XQ@mail.gmail.com>
	<DF04F544-0CA8-426F-A9F2-DAFA7EBCDC14@gmail.com>
	<4E70AA85.6090700@v.loewis.de>
Message-ID: <CAKCKLWx2Tqc_fU+N2F=saAFT0VrSjSaC1PR9WE6jgnjWeC+N5Q@mail.gmail.com>

On 14 September 2011 14:22, "Martin v. L?wis" <martin at v.loewis.de> wrote:

> Am 14.09.11 14:36, schrieb drkjam:
>
>  Would it be possible to have trove classifiers added to PyPI specifically
>> for PyPy and possibly also Jython and IronPython?
>>
>
> Here's the standard set of questions for proposers of new classifiers:
>
> * what specific classifiers are you proposing?
> * what specific semantics would you imply for packages tagged with these
> classifiers?
> * what specific packages that are already on PyPI would be tagged with
> these specific classifiers?
>


Addressing the last 2 of these questions particularly:

It would be useful to be able to specify that a package is tested with
(known to work with) alternative implementations. So, for example, I would
tag "mock" with pypy and Jython as I routinely test on those implementations
and know that it works.

As a user (or potential user) of pypy, it would be very useful to know what
packages are known work with pypy. At the moment pypy has to maintain its
own list or users have to check individual package documentation or proceed
by trial and error.

For Jython and IronPython it is possible to have packages that *only* work
on those implementations because they integrate with or use features of the
underlying platform. For example rst2xaml is a package only useful for those
using IronPython (either on the desktop or with Silverlight).

All the best,

Michael





>
> Regards,
> Martin
>
> ______________________________**_________________
> Catalog-SIG mailing list
> Catalog-SIG at python.org
> http://mail.python.org/**mailman/listinfo/catalog-sig<http://mail.python.org/mailman/listinfo/catalog-sig>
>



-- 

http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20110915/6efe9554/attachment.html>

From fuzzyman at gmail.com  Thu Sep 15 14:09:08 2011
From: fuzzyman at gmail.com (Michael Foord)
Date: Thu, 15 Sep 2011 13:09:08 +0100
Subject: [Catalog-sig] PyPI trove classifiers for alternate language
	implementations
In-Reply-To: <4E70AA85.6090700@v.loewis.de>
References: <CADiSq7dNhexnM56-u3W1Y_i3u1vAdHrFBtFVLbrri0JXLVf1XQ@mail.gmail.com>
	<DF04F544-0CA8-426F-A9F2-DAFA7EBCDC14@gmail.com>
	<4E70AA85.6090700@v.loewis.de>
Message-ID: <CAKCKLWz=13LuP=h0NsYRQ55uy8suM46viKh9+0HKeSdEv7ZvUg@mail.gmail.com>

On 14 September 2011 14:22, "Martin v. L?wis" <martin at v.loewis.de> wrote:

> Am 14.09.11 14:36, schrieb drkjam:
>
>  Would it be possible to have trove classifiers added to PyPI specifically
>> for PyPy and possibly also Jython and IronPython?
>>
>
> Here's the standard set of questions for proposers of new classifiers:
>
> * what specific classifiers are you proposing?
>

Suggestion:

Programming Language - Python - Implementation - CPython
Programming Language - Python - Implementation - pypy
Programming Language - Python - Implementation - jython
Programming Language - Python - Implementation - IronPython

Michael Foord


> * what specific semantics would you imply for packages tagged with these
> classifiers?
> * what specific packages that are already on PyPI would be tagged with
> these specific classifiers?
>
> Regards,
> Martin
>
> ______________________________**_________________
> Catalog-SIG mailing list
> Catalog-SIG at python.org
> http://mail.python.org/**mailman/listinfo/catalog-sig<http://mail.python.org/mailman/listinfo/catalog-sig>
>



-- 

http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20110915/b671079f/attachment.html>

From martin at v.loewis.de  Thu Sep 15 16:12:02 2011
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 15 Sep 2011 16:12:02 +0200
Subject: [Catalog-sig] PyPI trove classifiers for alternate language
	implementations
In-Reply-To: <CAKCKLWz=13LuP=h0NsYRQ55uy8suM46viKh9+0HKeSdEv7ZvUg@mail.gmail.com>
References: <CADiSq7dNhexnM56-u3W1Y_i3u1vAdHrFBtFVLbrri0JXLVf1XQ@mail.gmail.com>	<DF04F544-0CA8-426F-A9F2-DAFA7EBCDC14@gmail.com>	<4E70AA85.6090700@v.loewis.de>
	<CAKCKLWz=13LuP=h0NsYRQ55uy8suM46viKh9+0HKeSdEv7ZvUg@mail.gmail.com>
Message-ID: <4E7207B2.4070404@v.loewis.de>


> Programming Language - Python - Implementation - CPython
> Programming Language - Python - Implementation - pypy
> Programming Language - Python - Implementation - jython
> Programming Language - Python - Implementation - IronPython

Opinions on this proposal? (including the specific spelling,
leaving alone that the separator is ::, not -)

Regards,
Martin


From mal at egenix.com  Thu Sep 15 16:24:05 2011
From: mal at egenix.com (M.-A. Lemburg)
Date: Thu, 15 Sep 2011 16:24:05 +0200
Subject: [Catalog-sig] PyPI trove classifiers for alternate
	language	implementations
In-Reply-To: <4E7207B2.4070404@v.loewis.de>
References: <CADiSq7dNhexnM56-u3W1Y_i3u1vAdHrFBtFVLbrri0JXLVf1XQ@mail.gmail.com>	<DF04F544-0CA8-426F-A9F2-DAFA7EBCDC14@gmail.com>	<4E70AA85.6090700@v.loewis.de>	<CAKCKLWz=13LuP=h0NsYRQ55uy8suM46viKh9+0HKeSdEv7ZvUg@mail.gmail.com>
	<4E7207B2.4070404@v.loewis.de>
Message-ID: <4E720A85.2070501@egenix.com>

"Martin v. L?wis" wrote:
> 
>> Programming Language - Python - Implementation - CPython
>> Programming Language - Python - Implementation - pypy
>> Programming Language - Python - Implementation - jython
>> Programming Language - Python - Implementation - IronPython
> 
> Opinions on this proposal? (including the specific spelling,
> leaving alone that the separator is ::, not -)

Better user CPython, PyPy and Jython for consistency with the
other Trove spellings.

I'm -0 on the "Implementation" part. Do we really need this ?

Also: What about release versions of those implementations ?

Jython and IronPython appear to follow the CPython release
versions, but PyPy uses its own version scheme.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Sep 15 2011)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2011-10-04: PyCon DE 2011, Leipzig, Germany                19 days to go

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From fuzzyman at gmail.com  Thu Sep 15 16:30:46 2011
From: fuzzyman at gmail.com (Michael Foord)
Date: Thu, 15 Sep 2011 15:30:46 +0100
Subject: [Catalog-sig] PyPI trove classifiers for alternate language
	implementations
In-Reply-To: <4E720A85.2070501@egenix.com>
References: <CADiSq7dNhexnM56-u3W1Y_i3u1vAdHrFBtFVLbrri0JXLVf1XQ@mail.gmail.com>
	<DF04F544-0CA8-426F-A9F2-DAFA7EBCDC14@gmail.com>
	<4E70AA85.6090700@v.loewis.de>
	<CAKCKLWz=13LuP=h0NsYRQ55uy8suM46viKh9+0HKeSdEv7ZvUg@mail.gmail.com>
	<4E7207B2.4070404@v.loewis.de> <4E720A85.2070501@egenix.com>
Message-ID: <CAKCKLWxDj3zh5-Crw8v7e_8GJ2HQdvpBC2QRteQdK9umYQrPpA@mail.gmail.com>

On 15 September 2011 15:24, M.-A. Lemburg <mal at egenix.com> wrote:

> "Martin v. L?wis" wrote:
> >
> >> Programming Language - Python - Implementation - CPython
> >> Programming Language - Python - Implementation - pypy
> >> Programming Language - Python - Implementation - jython
> >> Programming Language - Python - Implementation - IronPython
> >
> > Opinions on this proposal? (including the specific spelling,
> > leaving alone that the separator is ::, not -)
>
> Better user CPython, PyPy and Jython for consistency with the
> other Trove spellings.
>
> I'm -0 on the "Implementation" part. Do we really need this ?
>


Jython, PyPy and CPython are not "programming languages" they're
implementations of the Python programming language - so I would prefer to
differentiate like this.


>
> Also: What about release versions of those implementations ?
>
>
We could always add those later if anyone really wanted them (for Python we
have the broad "Python" *plus* version specific categories). In other words,
YAGNI (for now).

Michael


> Jython and IronPython appear to follow the CPython release
> versions, but PyPy uses its own version scheme.
>
> --
> Marc-Andre Lemburg
> eGenix.com
>
> Professional Python Services directly from the Source  (#1, Sep 15 2011)
> >>> Python/Zope Consulting and Support ...        http://www.egenix.com/
> >>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
> >>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
> ________________________________________________________________________
> 2011-10-04: PyCon DE 2011, Leipzig, Germany                19 days to go
>
> ::: Try our new mxODBC.Connect Python Database Interface for free ! ::::
>
>
>   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
>    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
>           Registered at Amtsgericht Duesseldorf: HRB 46611
>               http://www.egenix.com/company/contact/
>



-- 

http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20110915/a2cfe70d/attachment-0001.html>

From lac at openend.se  Thu Sep 15 16:57:54 2011
From: lac at openend.se (Laura Creighton)
Date: Thu, 15 Sep 2011 16:57:54 +0200
Subject: [Catalog-sig] PyPI trove classifiers for alternate language
	implementations
In-Reply-To: Message from =?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=
	<martin@v.loewis.de>
	of "Thu, 15 Sep 2011 16:12:02 +0200." <4E7207B2.4070404@v.loewis.de>
References: <CADiSq7dNhexnM56-u3W1Y_i3u1vAdHrFBtFVLbrri0JXLVf1XQ@mail.gmail.com>
	<DF04F544-0CA8-426F-A9F2-DAFA7EBCDC14@gmail.com>
	<4E70AA85.6090700@v.loewis.de>
	<CAKCKLWz=13LuP=h0NsYRQ55uy8suM46viKh9+0HKeSdEv7ZvUg@mail.gmail.com><4E7207B2.4070404@v.loewis.de>
Message-ID: <201109151457.p8FEvs6M021638@theraft.openend.se>

In a message of Thu, 15 Sep 2011 16:12:02 +0200, "Martin v. L?wis" writes:
>
>> Programming Language - Python - Implementation - CPython
>> Programming Language - Python - Implementation - pypy
>> Programming Language - Python - Implementation - jython
>> Programming Language - Python - Implementation - IronPython
>
>Opinions on this proposal? (including the specific spelling,
>leaving alone that the separator is ::, not -)
>
>Regards,
>Martin

PyPy should be spelled like that.

Laura


From exarkun at twistedmatrix.com  Thu Sep 15 16:36:10 2011
From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com)
Date: Thu, 15 Sep 2011 14:36:10 -0000
Subject: [Catalog-sig] PyPI trove classifiers for alternate
	language	implementations
In-Reply-To: <4E720A85.2070501@egenix.com>
References: <CADiSq7dNhexnM56-u3W1Y_i3u1vAdHrFBtFVLbrri0JXLVf1XQ@mail.gmail.com>
	<DF04F544-0CA8-426F-A9F2-DAFA7EBCDC14@gmail.com>
	<4E70AA85.6090700@v.loewis.de>
	<CAKCKLWz=13LuP=h0NsYRQ55uy8suM46viKh9+0HKeSdEv7ZvUg@mail.gmail.com>
	<4E7207B2.4070404@v.loewis.de> <4E720A85.2070501@egenix.com>
Message-ID: <20110915143610.1808.320688955.divmod.xquotient.870@localhost.localdomain>

On 02:24 pm, mal at egenix.com wrote:
>"Martin v. L?wis" wrote:
>>
>>>Programming Language - Python - Implementation - CPython
>>>Programming Language - Python - Implementation - pypy
>>>Programming Language - Python - Implementation - jython
>>>Programming Language - Python - Implementation - IronPython
>>
>>Opinions on this proposal? (including the specific spelling,
>>leaving alone that the separator is ::, not -)
>
>Better user CPython, PyPy and Jython for consistency with the
>other Trove spellings.
>
>I'm -0 on the "Implementation" part. Do we really need this ?

These seem like a different sort of thing than the existing classifiers 
like

    Programming Language :: Python :: 2.7

It seems that each implementation is trying to provide compatibility 
with a particular CPython release.  You could imagine a project 
specifying

    Programming Language :: Python :: 2.7
    Programming Language :: Python :: Implementation :: Jython

To indicate that a Jython release implementing Python 2.7 is required, 
and only the Jython runtime is supported.
>Also: What about release versions of those implementations ?
>
>Jython and IronPython appear to follow the CPython release
>versions, but PyPy uses its own version scheme.

The current release of PyPy is 1.6, but it is intended to be compatible 
with CPython 2.7.  If the classifiers are interpreted how I've 
interpreted them above, then combining two or more is sufficient, and 
having PyPy :: 1.5, PyPy 1.6, PyPy :: 1.7 doesn't offer much.

However, as usual, interpretation is up in the air, since a classifier 
is only a name, with no associated documented meaning, beyond what 
people manage to glean from mailing lists or by looking at existing 
projects that have decided to use them.

Jean-Paul

From mal at egenix.com  Fri Sep 16 01:18:34 2011
From: mal at egenix.com (M.-A. Lemburg)
Date: Fri, 16 Sep 2011 01:18:34 +0200
Subject: [Catalog-sig] PyPI trove classifiers for alternate
	language	implementations
In-Reply-To: <CAKCKLWxDj3zh5-Crw8v7e_8GJ2HQdvpBC2QRteQdK9umYQrPpA@mail.gmail.com>
References: <CADiSq7dNhexnM56-u3W1Y_i3u1vAdHrFBtFVLbrri0JXLVf1XQ@mail.gmail.com>	<DF04F544-0CA8-426F-A9F2-DAFA7EBCDC14@gmail.com>	<4E70AA85.6090700@v.loewis.de>	<CAKCKLWz=13LuP=h0NsYRQ55uy8suM46viKh9+0HKeSdEv7ZvUg@mail.gmail.com>	<4E7207B2.4070404@v.loewis.de>
	<4E720A85.2070501@egenix.com>
	<CAKCKLWxDj3zh5-Crw8v7e_8GJ2HQdvpBC2QRteQdK9umYQrPpA@mail.gmail.com>
Message-ID: <4E7287CA.6070900@egenix.com>

Michael Foord wrote:
> On 15 September 2011 15:24, M.-A. Lemburg <mal at egenix.com> wrote:
> 
>> "Martin v. L?wis" wrote:
>>>
>>>> Programming Language - Python - Implementation - CPython
>>>> Programming Language - Python - Implementation - pypy
>>>> Programming Language - Python - Implementation - jython
>>>> Programming Language - Python - Implementation - IronPython
>>>
>>> Opinions on this proposal? (including the specific spelling,
>>> leaving alone that the separator is ::, not -)
>>
>> Better user CPython, PyPy and Jython for consistency with the
>> other Trove spellings.
>>
>> I'm -0 on the "Implementation" part. Do we really need this ?
>>
> 
> 
> Jython, PyPy and CPython are not "programming languages" they're
> implementations of the Python programming language - so I would prefer to
> differentiate like this.

Well, yes, but if you look at the OS section of the Trove list
you also find different implementations of BSD under the
BSD category:

Operating System :: POSIX :: BSD
Operating System :: POSIX :: BSD :: BSD/OS
Operating System :: POSIX :: BSD :: FreeBSD
Operating System :: POSIX :: BSD :: NetBSD
Operating System :: POSIX :: BSD :: OpenBSD

(just to take one exmaple)

See http://pypi.python.org/pypi?%3Aaction=list_classifiers for the full
list.

Also note that Cython is listed as:

Programming Language :: Cython

even Zope is listed as programming language:

Programming Language :: Zope

so there isn't all that much consistency in the naming.

BTW: Stackless is missing from your list.

>> Also: What about release versions of those implementations ?
>>
>>
> We could always add those later if anyone really wanted them (for Python we
> have the broad "Python" *plus* version specific categories). In other words,
> YAGNI (for now).
> 
> Michael
> 
> 
>> Jython and IronPython appear to follow the CPython release
>> versions, but PyPy uses its own version scheme.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Sep 16 2011)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2011-10-04: PyCon DE 2011, Leipzig, Germany                18 days to go

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From ziade.tarek at gmail.com  Tue Sep 27 11:40:44 2011
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 27 Sep 2011 11:40:44 +0200
Subject: [Catalog-sig] Fwd: The state of PyPI
In-Reply-To: <CAGSi+Q6ukuHOi4ukO_mPZv3uxi7OQmsmMwNMrfYk-HG+e_sTLA@mail.gmail.com>
References: <CAGSi+Q6ukuHOi4ukO_mPZv3uxi7OQmsmMwNMrfYk-HG+e_sTLA@mail.gmail.com>
Message-ID: <CAGSi+Q7LY_dLxffUW7QY1Xju=8VXH0z2GkRrYSwLki-e5jwDfw@mail.gmail.com>

I have sent that to the PSF list because there's a PSF project about PyPI infra.

But someone complained, saying that I was doing this discussion
"behind closed doors"

SInce this is not my goal, I am now spamming more lists...


---------- Forwarded message ----------
From: Tarek Ziad? <ziade.tarek at gmail.com>
Date: Tue, Sep 27, 2011 at 10:37 AM
Subject: The state of PyPI
To: PSF Members List <psf-members at python.org>
Cc: Richard Jones <r1chardj0n3s at gmail.com>, Steve Holden <steve at holdenweb.com>


Hey

This is just a mail that summarizes the current state of PyPI, the
existing features, and what can be done next to improve stuff.

I am sending this in the PSF members list because we had a project of
an infrastructure going on, and I want to make sure all involved
parties are in the same page.

1/ stability and high availability
2/ private mirrors
3/ private projects
4/ tutorial ?


= stability and high availability =

we went in two directions to improve PyPI :

1/ add the mirroring protocol
2/ make the PyPI server more reliable by pushing its storage in a
redundant cloud.

== mirroring ==

The mirroring protocol (PEP 381) is implemented on server-side, I've
worked with Martin on this, and we have mirrors now:

Look at http://pypi.python.org/mirrors

Also, there's a client that anyone can use to set up a mirror:
http://pypi.python.org/pypi/pep381client

The idea is that anyone in the community willing to maintain a mirror
can do so. We add the mirror in the CND, and make it available for
client tools to use. What's really missing right now is more
integration on client-side.

- Pip supports the mirroring protocol, and can fall back to a mirror,
but I am not 100% sure this is a default behavior. ?(please correct me
if it is now)
- Buildout knows how to use *another server* than the main PyPI, so
can manually switch to a mirror, but I don't think it's transparent.
It should.
- Distribute/Setuptools does not do anything for this, and should.
- everything is already implemented in packaging/distutils2

The effect of the mirrors is that PyPI being down should not impact
the community. This will be true once all tool are transparently using
the mirrors.

== better infra ==

I think the project is staled right now.


= private mirrors =

Having a private mirror makes a lot of sense, when companies need to
make sure their build systems are not relying on external services
like PyPI or a mirror. It's also a good way to dramatically reduce the
load for the community servers.

The idea is that a Jenkins server that builds hundreds of Python apps
every hour should not hammer PyPI.

We have everything needed these days to set up this kind of system,
with zc.buildout or pip good practices.

What we need is a good tutorial or a guide [*]

= private projects =

The part that we do not address in the community is private projects:
since we don't have any permissions/group/roles system in PyPI,
everything is public.

One way to solve this is to have a local repository for private
packages, that is looked by tools like pip or easy_install, with the
--find-links option.


What we need is a good tutorial or a guide [*]

= tutorial =

[*] If this helps, I am willing to work on a tutorial day for Pycon
US, that goes through all of this, to help people set up their dev.
environment the best way possible.

The material could then be published at python.org/pypi to help out.

I know Richard has some material already, so maybe this could be a
joint tutorial ?

HTH

Cheers
Tarek





--
Tarek Ziad? | http://ziade.org



-- 
Tarek Ziad? | http://ziade.org

From mal at egenix.com  Tue Sep 27 12:04:25 2011
From: mal at egenix.com (M.-A. Lemburg)
Date: Tue, 27 Sep 2011 12:04:25 +0200
Subject: [Catalog-sig] Fwd: The state of PyPI
In-Reply-To: <CAGSi+Q7LY_dLxffUW7QY1Xju=8VXH0z2GkRrYSwLki-e5jwDfw@mail.gmail.com>
References: <CAGSi+Q6ukuHOi4ukO_mPZv3uxi7OQmsmMwNMrfYk-HG+e_sTLA@mail.gmail.com>
	<CAGSi+Q7LY_dLxffUW7QY1Xju=8VXH0z2GkRrYSwLki-e5jwDfw@mail.gmail.com>
Message-ID: <4E819FA9.7020300@egenix.com>

Tarek Ziad? wrote:
> = stability and high availability =
> 
> we went in two directions to improve PyPI :
> 
> 1/ add the mirroring protocol
> 2/ make the PyPI server more reliable by pushing its storage in a
> redundant cloud.
> ...
> == better infra ==
> 
> I think the project is staled right now.

True, I don't have the needed cycles to make progress and neither
do the team members. We do have the Amazon infrastructure setup
(EC2, CloudFront, S3), and some progress has been made to sync
the S3 storage with the local copies of the packages.

The next steps would be to implement a trigger based sync scheme
for packages and the simple/ index, i.e. new uploads should trigger
an S3 copy process and update the simple/ index on S3 as well.

For more info, please have a look at the wiki page:

    http://wiki.python.org/moin/CloudPyPI

Here's the original proposal:

    http://wiki.python.org/moin/CloudPyPI/Proposal

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Sep 27 2011)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2011-10-04: PyCon DE 2011, Leipzig, Germany                 7 days to go

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From carl at oddbird.net  Tue Sep 27 16:36:19 2011
From: carl at oddbird.net (Carl Meyer)
Date: Tue, 27 Sep 2011 08:36:19 -0600
Subject: [Catalog-sig] Fwd: The state of PyPI
In-Reply-To: <CAGSi+Q7LY_dLxffUW7QY1Xju=8VXH0z2GkRrYSwLki-e5jwDfw@mail.gmail.com>
References: <CAGSi+Q6ukuHOi4ukO_mPZv3uxi7OQmsmMwNMrfYk-HG+e_sTLA@mail.gmail.com>
	<CAGSi+Q7LY_dLxffUW7QY1Xju=8VXH0z2GkRrYSwLki-e5jwDfw@mail.gmail.com>
Message-ID: <4E81DF63.9080703@oddbird.net>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Tarek,

On 09/27/2011 03:40 AM, Tarek Ziad? wrote:
> = private projects =
> 
> The part that we do not address in the community is private projects:
> since we don't have any permissions/group/roles system in PyPI,
> everything is public.
> 
> One way to solve this is to have a local repository for private
> packages, that is looked by tools like pip or easy_install, with the
> --find-links option.
> 
> 
> What we need is a good tutorial or a guide [*]
> 
> = tutorial =
> 
> [*] If this helps, I am willing to work on a tutorial day for Pycon
> US, that goes through all of this, to help people set up their dev.
> environment the best way possible.
> 
> The material could then be published at python.org/pypi to help out.
> 
> I know Richard has some material already, so maybe this could be a
> joint tutorial ?

The talk I gave at djangocon.us a couple weeks ago [1] covers this
material, though certainly not as in-depth as it could be covered, and
with a focus primarily on pip.

I would be willing/interested in doing (or co-presenting) a PyCon
tutorial on solutions for managing local package repositories.

Carl

 [1] https://carljm.github.com/tamingdeps/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk6B32MACgkQ8W4rlRKtE2dOdACeNnd7egS3z3agCkvJMAXIjNbS
90sAoLrd8X50uA7n1Vz6uBcRtXd5Fpyc
=RRy1
-----END PGP SIGNATURE-----

From ziade.tarek at gmail.com  Tue Sep 27 17:17:25 2011
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 27 Sep 2011 17:17:25 +0200
Subject: [Catalog-sig] Fwd: The state of PyPI
In-Reply-To: <4E81DF63.9080703@oddbird.net>
References: <CAGSi+Q6ukuHOi4ukO_mPZv3uxi7OQmsmMwNMrfYk-HG+e_sTLA@mail.gmail.com>
	<CAGSi+Q7LY_dLxffUW7QY1Xju=8VXH0z2GkRrYSwLki-e5jwDfw@mail.gmail.com>
	<4E81DF63.9080703@oddbird.net>
Message-ID: <CAGSi+Q4L5UhpQ2oaU0YWc-TY7fS9s9HtzsvObmdaXpbNa-hPWg@mail.gmail.com>

On Tue, Sep 27, 2011 at 4:36 PM, Carl Meyer <carl at oddbird.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi Tarek,
>
> On 09/27/2011 03:40 AM, Tarek Ziad? wrote:
>> = private projects =
>>
>> The part that we do not address in the community is private projects:
>> since we don't have any permissions/group/roles system in PyPI,
>> everything is public.
>>
>> One way to solve this is to have a local repository for private
>> packages, that is looked by tools like pip or easy_install, with the
>> --find-links option.
>>
>>
>> What we need is a good tutorial or a guide [*]
>>
>> = tutorial =
>>
>> [*] If this helps, I am willing to work on a tutorial day for Pycon
>> US, that goes through all of this, to help people set up their dev.
>> environment the best way possible.
>>
>> The material could then be published at python.org/pypi to help out.
>>
>> I know Richard has some material already, so maybe this could be a
>> joint tutorial ?
>
> The talk I gave at djangocon.us a couple weeks ago [1] covers this
> material, though certainly not as in-depth as it could be covered, and
> with a focus primarily on pip.
>
> I would be willing/interested in doing (or co-presenting) a PyCon
> tutorial on solutions for managing local package repositories.
>
> Carl
>
> ?[1] https://carljm.github.com/tamingdeps/

Awesome.

Richard and Alexis are also interested, so I propose that we work on a
joint proposal on an etherpad and submit it.

I do believe half a day would be enough




> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAk6B32MACgkQ8W4rlRKtE2dOdACeNnd7egS3z3agCkvJMAXIjNbS
> 90sAoLrd8X50uA7n1Vz6uBcRtXd5Fpyc
> =RRy1
> -----END PGP SIGNATURE-----
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG at python.org
> http://mail.python.org/mailman/listinfo/catalog-sig
>



-- 
Tarek Ziad? | http://ziade.org

From noah at coderanger.net  Tue Sep 27 19:24:02 2011
From: noah at coderanger.net (Noah Kantrowitz)
Date: Tue, 27 Sep 2011 10:24:02 -0700
Subject: [Catalog-sig] Fwd: The state of PyPI
In-Reply-To: <CAGSi+Q7LY_dLxffUW7QY1Xju=8VXH0z2GkRrYSwLki-e5jwDfw@mail.gmail.com>
References: <CAGSi+Q6ukuHOi4ukO_mPZv3uxi7OQmsmMwNMrfYk-HG+e_sTLA@mail.gmail.com>
	<CAGSi+Q7LY_dLxffUW7QY1Xju=8VXH0z2GkRrYSwLki-e5jwDfw@mail.gmail.com>
Message-ID: <DDF6FA6D-2EB1-4957-B871-590DAA750AFA@coderanger.net>

I've copied our current draft plan for the structure of the PyPI infra. I stress this is just a draft as it stands today and is not final.

On Sep 27, 2011, at 2:40 AM, Tarek Ziad? wrote:
> 
> == better infra ==
> 


The current plan is to have two primary load balancer VMs running Nginx acting as both balancer and SSL termination points. These will share a set of floating IPs using Heartbeat. Behind this will be the same Apache configuration currently in use (Apache serving static files and PyPI running as an FCGI script controlled by Apache) running on two VMs, both talking to the same master-slave Postgres 9 replication setup. Package files will initially be handled by a shared DRBD drive, however this may be obsoleted by the project to move file hosting to Cloudfront or another CDN.

A currently open question is how best to provide reliability and security for the SSH-based file upload system currently deployed on ximenez. Most likely we can setup the initial SSH endpoint on the load balancers to run a proxy to one of the main PyPI application servers, however failover would have to be semi-manual (possibly driven by Chef, meaning a chef-client run would have to happen before the tunnel would be updated, or anywhere up to 30 minutes). Given the relatively minimal public knowledge of this service, I think this is acceptable as a first-pass but a future solution involving HAProxy or another TCP load balancer to handle the SSH traffic might be appropriate. Similarly, automated database failover is not planned at this time due to the extra application-side complexity however the process will be well documented and able to be executed by all tier one on-call operations staff if the Postgres master goes offline for some reason.

This will only partially address the current reliability issues as many of the current problems are linked to Apache or mod_fcgid needing to be restarted. In that light I would like to see PyPI and the catalog-sig group investigate moving the codebase to work against mod_wsgi or gunicorn (no real preference between the two) to create a more reliable runtime environment.

--Noah
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 163 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20110927/f03dcb94/attachment.pgp>

From ziade.tarek at gmail.com  Tue Sep 27 19:33:58 2011
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 27 Sep 2011 19:33:58 +0200
Subject: [Catalog-sig] Fwd: The state of PyPI
In-Reply-To: <DDF6FA6D-2EB1-4957-B871-590DAA750AFA@coderanger.net>
References: <CAGSi+Q6ukuHOi4ukO_mPZv3uxi7OQmsmMwNMrfYk-HG+e_sTLA@mail.gmail.com>
	<CAGSi+Q7LY_dLxffUW7QY1Xju=8VXH0z2GkRrYSwLki-e5jwDfw@mail.gmail.com>
	<DDF6FA6D-2EB1-4957-B871-590DAA750AFA@coderanger.net>
Message-ID: <CAGSi+Q7YAfcoYRRsp=FUjPGnb1iazRv41JUO74kEHkm88qaaxw@mail.gmail.com>

On Tue, Sep 27, 2011 at 7:24 PM, Noah Kantrowitz <noah at coderanger.net> wrote:
> I've copied our current draft plan for the structure of the PyPI infra. I stress this is just a draft as it stands today and is not final.

Thanks,

I've started this page with all the content :
http://wiki.python.org/moin/BetterPyPI





-- 
Tarek Ziad? | http://ziade.org

From noah at coderanger.net  Tue Sep 27 19:34:55 2011
From: noah at coderanger.net (Noah Kantrowitz)
Date: Tue, 27 Sep 2011 10:34:55 -0700
Subject: [Catalog-sig] Fwd: The state of PyPI
In-Reply-To: <CAGSi+Q7YAfcoYRRsp=FUjPGnb1iazRv41JUO74kEHkm88qaaxw@mail.gmail.com>
References: <CAGSi+Q6ukuHOi4ukO_mPZv3uxi7OQmsmMwNMrfYk-HG+e_sTLA@mail.gmail.com>
	<CAGSi+Q7LY_dLxffUW7QY1Xju=8VXH0z2GkRrYSwLki-e5jwDfw@mail.gmail.com>
	<DDF6FA6D-2EB1-4957-B871-590DAA750AFA@coderanger.net>
	<CAGSi+Q7YAfcoYRRsp=FUjPGnb1iazRv41JUO74kEHkm88qaaxw@mail.gmail.com>
Message-ID: <5F826E99-A368-47E2-A26D-B691F8A05649@coderanger.net>


On Sep 27, 2011, at 10:33 AM, Tarek Ziad? wrote:

> On Tue, Sep 27, 2011 at 7:24 PM, Noah Kantrowitz <noah at coderanger.net> wrote:
>> I've copied our current draft plan for the structure of the PyPI infra. I stress this is just a draft as it stands today and is not final.
> 
> Thanks,
> 
> I've started this page with all the content :
> http://wiki.python.org/moin/BetterPyPI

If people have feedback please also make sure to let me know so I we can discuss it.

--Noah
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 163 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20110927/18b10466/attachment-0001.pgp>