Mailman 3 March 2011 - Distutils-SIG

Re: [Distutils] zc.buildout: new option for requiring MD5 sums for downloads
by Thomas Lotze March 11, 2011

March 11, 2011

"Filip M. Noetzel" <filip(a)j03.de> schrieb: > (I'm replying out of band, [...] I hope you don't mind if I send a copy of my reply back to the list, though. > I think wrote what you are describing in your post a few months ago: > > http://pypi.python.org/pypi/buildout-md5sums (Source at https://github.com/peritus/buildout-md5sums ) It has a very similar purpose indeed. Nice to see that this is something I'm not the only one to want to have. Thank you for pointing it out! > I'd love feedback on it (I use it on a day-to-day basis for my buildouts, but don't know other users). The problems I see with your approach: - Patching the download API is technically less than optimal. - Anchoring MD5 enforcement that deeply within the mechanics means that client code cannot decide whether its associated configuration needs to honour the allow-picked-downloads flag. I'm not sure whether that's a good thing or bad - that's part of what I'm hoping to discuss. I could imagine that one wants to enforce checksums for, e.g., source packages downloaded by a cmmi recipe while avoiding them for base configuration files downloaded by buildout itself. - As a less technical aspect, you might want to consider a more serious licence for your package if you hope for more wide-spread use. Thomas

1 0

pypi/packages/docs.python.org
by Lennart Regebro March 11, 2011

March 11, 2011

On Sun, Mar 6, 2011 at 20:49, cool-RR <cool-rr(a)cool-rr.com> wrote: > Speaking of names, I would rename PyPI to packages.python.org, maybe move > the existing documentation center to docs.python.org, and then move the docs > of Python itself to a `/python` folder... > But that's just me. No, I do think docs.python.org should be for documentation of Python. I agree having package documentation on packages.python.org may not be very pedagogical, though. :-) //Lennart

10 29

Re: [Distutils] Name the software! Package quality tester.
by P.J. Eby March 10, 2011

March 10, 2011

At 08:08 AM 3/10/2011 -0500, Jim Fulton wrote: >I really should let this rest ....... really .... I notice you seem to have already let rest defending your proposal, as opposed to opposing mine. ;-) That is, I don't see where you've included any counterargument for why convincing people to *stop* using "package" for "python package" isn't going to be a lot harder than simply *adding* the term "project" to existing docs, via simple explanations like this one: "If you would like to distribute your python package or module, you need to create a project. A project consists of a directory containing the package(s) or module(s) you want to distribute, along with a setup.py/.cfg, specifying the project's name, current version number, and other information. If you wish to distribute your package via PyPI, then you must choose a unique project name and register it. If you are only distributing one module or package, you can name the project after it, as long as it doesn't conflict with any other project names already registered on PyPI." Context-specific explanations like this are easy to give, at a place where the recipient of the explanation *wants* to learn something. In contrast, to convince everybody in the world to replace "package" with "module", you must send out a constant broadcast to people who don't really care. (your other points are addressed below) >On Wed, Mar 9, 2011 at 11:43 AM, P.J. Eby <pje(a)telecommunity.com> wrote: > > At 07:06 AM 3/9/2011 -0500, Jim Fulton wrote: > >> > >> They certainly aren't "projects" in any sense that most people would > >> understand. > > > > I don't follow you. Sourceforge hosts projects. Freshmeat indexes > > projects. Mozdev.org hosts projects. The Apache Foundation hosts projects. > > "Project", IOW, is *exactly* the word people use for these things, in the > > larger world of software, and that's precisely why I chose it for my > > original terminology proposal. > >But the things in PyPI are not projects. Projects have bug trackers, >and mailing lists, and source code repositories. And which of those things have to be hosted on the same site as the project's index page? Freshmeat, for example, is not a project *hosting* system, it's a project *index*. PyPI is in that sense, a project index. Some index pages may link to all the stuff associated with that project, some may not. (i.e. the set of all possible project features is not equal to the intersection of all projects' actual features: some project have mailing lists and not bug trackers, and vice versa. This does not make them non-projects.) > Projects are organizations of people -- not software distributions. I'm an organization of people; am I a project? No, the various things I have listed on PyPI are my projects, and I distribute releases of them. >The things in PyPI that you call "projects" are things that get >produced by projects. There isn't a buildout.recipe.egg >"project". buildout.recipe.egg is just a um packaging of some software >components. It's not a "project" in any usual sense of the word. I don't get that. You work on it, and it is distinct from other works. That makes it a project. >No. End users need to know about this. They need to know what these >names mean and that the "package name" (distutils) or "project name" >(setuptools) doesn't imply that the distribution will provide a python >package of the same name. Trust me on this. Users care about this. Actually, I believe distutils calls it a "distribution name", but I could be wrong about that. ;-) > > For one thing, if the distutils documentation simply starts out like: > > > > "If you want to distribute your work to the larger Python community, you > > need to create a project for it. In practical terms, this means having a > > directory with a setup.py and the code, data, or documentation you wish to > > distribute. Your project will also need a unique name, if you want to make > > it accessible to others via the Python Project Index (PyPI)." (replace > > setup.py w/setup.cfg for distutils2, of course) > >It does? Where? Can you give a link? I don't see this text anywhere. Of course not; it was a proposal for a hypothetical introduction. That's why it says "if" up there. ;-) > > This usage of project also maps onto common IDE usage of the term > project -- > > a directory of stuff that you're going to edit and build. > >That use of project is equivalent to "working directory" and not >relevant, IMO. Huh? In an IDE, a project is a thing you build and distribute, usually with source control. It's distinct from other projects, and it has a name. It is usually not tied to a specific code unit such as a module or package (regardless of what the code units of the relevant programming language are called). It has additional metadata, over and above the actual code units, and possibly includes non-code resources such as documentation, data, images, etc. IOW, it's *exactly* the sort of project we're talking about here. So, the term project is consistent with both the IDE usage of the word, *and* the mozdev/freshmeat/sourceforge et al meaning of project. It is thus unlikely to be confusing or surprising in the least. (Side question: were you *really* confused by the term project when you first began working with setuptools? I don't remember you having any issues with it at the time, but perhaps I've forgotten?) The hard part of introducing the term *here*, on this mailing list, is precisely the same part that is hard for your proposal: it is usually happening at a point where people are *not* as open to learning something. When somebody is on the list and we need to explain the project/package distinction, it's noise to them because they're trying to solve some other problem, and the distinction doesn't seem important. At the point, however, where someone is reading the distutils docs for the first time, or reading a "how to distribute your stuff" tutorial, that is precisely the place where that distinction will be perceived as valuable. That is, they are asking, "what do I need to do to get my code out there?" And the answer is, "make a project, register it, and upload some files." This answer is sensible and satisfying, so the term slips right in. Arguably, you could call it a "fooblymuffin" at that same point of introduction, and the "it's an answer to their question" argument mostly still applies. But the pre-existing meaning of "project" from both IDE usage and project-sites usage means that anyone who's encountered the term in either of those uses will likely have an immediate "aha" or "of course" reaction, thereby making the learning and acceptance even easier. Renaming package to module, on the other hand, has no similarly obvious point of motivation to learn, except for people who are *not yet python developers*... which means that it's going to take a long, long time to get the terminology changed.

1 0

Re: [Distutils] Name the software! Package quality tester.
by P.J. Eby March 10, 2011

March 10, 2011

At 07:06 AM 3/9/2011 -0500, Jim Fulton wrote: >They certainly aren't "projects" in any sense that most people would >understand. I don't follow you. Sourceforge hosts projects. Freshmeat indexes projects. Mozdev.org hosts projects. The Apache Foundation hosts projects. "Project", IOW, is *exactly* the word people use for these things, in the larger world of software, and that's precisely why I chose it for my original terminology proposal. Many PyPI listings also only describe a project, in the sense that there is no release information at all, or the code is only available via a revision control system, etc. >The term "project" has has never stuck, despite being discussed >repeatedly. It's only been discussed here, on this list. It hasn't been put in official documentation, or really blessed by anybody. When it has been discussed, it's been considered a reasonable name, and when people have needed to make the distinction, they've tended to use it. The primary reason it hasn't caught on in a larger sense is that people don't know about it, and have no motivation to learn it unless they end up in a situation where the distinction makes a difference... And let's face it, it really only makes a difference if you are creating some kind of packaging or distribution tool, or if you have a corner case of using one. >I think given the historical use of the term "package" it >was reasonable to find another term. OK, we tried. It didn't work. I don't think that's an accurate assessment on two points. First, "we" didn't try -- *I* did. (It's possible that someone else has promoted it, but I don't recall any instances of that.) So, there actually being a "we" behind it -- where "we" includes an approved PEP, documentation, etc., would be helpful. Second, I don't think it's accurate to say "it didn't work". It's not like people who need the distinction have objected to the word or proposed any alternatives. >We can pretent that if we work hard enough, we can make people adopt >our confusing terminology. Good luck with that. No matter which way we go (assuming there aren't any other alternatives), we will be trying to get some group of people to change their terminology. I'm just pointing out that the group that would need to change to use "project" is both smaller *and* inherently more motivated to change their usage, than the group that would need to replace "package" with "module". Thus, if getting people to use "project" is an uphill battle, getting people to stop using "package" is going to be a much higher vertical cliff. ;-) For one thing, if the distutils documentation simply starts out like: "If you want to distribute your work to the larger Python community, you need to create a project for it. In practical terms, this means having a directory with a setup.py and the code, data, or documentation you wish to distribute. Your project will also need a unique name, if you want to make it accessible to others via the Python Project Index (PyPI)." (replace setup.py w/setup.cfg for distutils2, of course) This usage of project also maps onto common IDE usage of the term project -- a directory of stuff that you're going to edit and build. And, it immediately introduces the term to a superset of the audience that will need to know it. Having PyPI describe its contents as "projects" is pretty much the other half of the indoctrination needed. In contrast, to make the package->module change, you'd have to not only change the official language reference and tutorial, but also every third-party book or tutorial about Python. Sure, some of those third party references would also exist for "package" as "project", but that's simply an *imprecise* usage, rather than an *incorrect* one.

5 5

Re: [Distutils] Name the software! Package quality tester.
by P.J. Eby March 9, 2011

March 9, 2011

At 05:18 PM 3/7/2011 -0500, Jim Fulton wrote: >If what we now call "packages" were called "modules", then we could >start using the term "package" the way everyone else does. I think >lots of people would be less confused. It seems to me that in order to make that change, you have to get more people to change their terminology, since the set of people who need to refer to package[module] is larger than the set of people who need to refer to package[project]. (There is also a larger body of documentation associated with package[module].) IOW, I think this proposal is a heavy uphill battle, both in the number of people to be convinced and the amount of documentation. In addition, the people who are calling a project a package can more easily understand the need to call it a project, than the people who are calling a package a package, will understand the need to call it a module. ;-) >Otherwise, I prefer we try hard to use the precise definitions >above. This topic can be confusing enough without making it more so >through sloppy terminology. I think that this approach is more achievable: it requires only an official blessing, a relatively small amount of documentation to be changed, and the renaming of PyPI et al. (i.e. "Python Projects Index", projects.python.org, etc.) ("Projects Index" is a better name anyway, since some things on PyPI are not packages at all but applications, scripts, modules, plugins, etc.)

3 2

Re: [Distutils] Name the software! Package quality tester.
by P.J. Eby March 9, 2011

March 9, 2011

At 11:46 AM 3/7/2011 +0100, Lennart Regebro wrote: >I'll make a note of this in the documentation to clear it up. >Distutils2 is definitely in the minority at the moment when it comes >to calling them "projects". The term has been in use in setuptools since around 2005, but it hasn't caught on much outside of the small group of people who need to be able to speak precisely about the concept. ;-) If you search the sig archives, though, you will find that it gets proposed and mostly-approved every time the topic comes up, of what to call these "things we distribute releases of". The more-or-less consensus terminology (for people who need a precise terminology): package = thing you import in Python that contains modules project = thing you make releases of release = one version of a project distribution = a file that embodies the release of a project (may be source or binary) People who don't care about precision just call everything a package, pretty much. Heck, a lot of times I find myself starting to type "package" when I'm really talking about a project, release, or distribution, despite promoted the more-precise terminology for half a decade or so. ;-)

6 8

zc.buildout: new option for requiring MD5 sums for downloads
by Thomas Lotze March 9, 2011

March 9, 2011

Hi, I wonder whether it makes sense to introduce a new option to the buildout section, named downloads-require-md5sum or similar. Recipes such as zc.recipe.cmmi or gocept.download could then be made to require an MD5 checksum for every downloaded resource if that option is set to true, and treat checksums as optional otherwise. The responsibility for evaluating that option would have to lie with recipes, not buildout's download API for two reasons: the download API should stay as simple as possible, and it must remain possible to use it for downloading resources such as base configurations (by the extends option) whose MD5 checksum isn't known beforehand. Any opinions on whether such functionality would be worth the added weight of such an option, or maybe insights into why requiring checksums depending on a buildout option might not be useful after all? Thank you. -- Thomas

1 0

Re: [Distutils] Name the software! Package quality tester.
by Lennart Regebro March 8, 2011

March 8, 2011

On Mon, Mar 7, 2011 at 21:47, P.J. Eby <pje(a)telecommunity.com> wrote: > The term has been in use in setuptools since around 2005, but it hasn't > caught on much outside of the small group of people who need to be able to > speak precisely about the concept. ;-) Well, I've changed the terminology in pyroma, to hopefully spread it around a bit. //Lennart

1 0

Name the software! Package quality tester.
by Lennart Regebro March 7, 2011

March 7, 2011

I've started working on a little utility to give a quality rating on packages, expressed in 0-10 points, and also in cheese types, according to smellyness. It's going to check for things like that it has all meta data it should have, such as author_email, specifies Python versions via the trove classifiers (currently works) and that it specifies all dependencies (still todo). It will support both checking on a package (works currently) a distribution file and PyPI (still to do). It's not a uniqe idea, it overlaps with Andreas Jungs zopyx.trashfinder in scope, and it will also in the case of checking a package on PyPI check that there are several people that have owner access, and hence include the functionality of mr.parker. (In fact when checking on PyPI it will also check if there are documentation on packages.python.org, that the distribution files are uploaded to PyPI, etc, but this is all still todo). But I didn't find anything else, and I wanted bigger scopes than both these in what to check in and which cases. But, before I move this to a public repository and upload it to PyPI, there is one important thing to be determined: What should it be called? Currently I'm calling it "pypilib.quality". I don't mind this kind of boring names, but there is currently not a pypilib namespace, and I don't want to just create top level namespaces left and right for no reason. So other names are welcome. It doesn't have to have a namespace either. In the long run I would not mind to see this utility integrated into a general pypi/cheeseshop script with other utility commands, which even could include installing and removing, thusly giving Perl people what they think they want a "CPAN" for Python. :-) -- Lennart Regebro: http://regebro.wordpress.com/ The Python 3 Porting book is out: http://python3porting.com/ +33 661 58 14 64

8 15

Proposed change to buildout configuration-file handling
by Jim Fulton March 7, 2011

March 7, 2011

An annoyance in buildout configurations, resulting from it's use of ConfigParser, is that configuration data lines are stripped of leading spaces. This means configurations like this: [paste.ini] recipe = zc.recipe.deployment:configuration text = [app:main] use = egg:bobo bobo_resources = cmsapp filter-with = zodb [filter:zodb] use = egg:zc.zodbwsgi configuration = <zodb main> <zeoclient> server ${main-db-server:address} </zeoclient> </zodb> <zodb index> <zeoclient> server ${index-db-server:address} </zeoclient> </zodb> [server:main] use = egg:Paste#http host = localhost port = 8080 Don't work because leading whitespace is stripped, forcing me to use hacks like this:: [paste.ini] recipe = zc.recipe.deployment:configuration s = text = ${:s}[app:main] ${:s}use = egg:bobo ${:s}bobo_resources = cmsapp ${:s}filter-with = zodb ${:s} ${:s}[filter:zodb] ${:s}use = egg:zc.zodbwsgi ${:s}configuration = ${:s} <zodb main> ${:s} <zeoclient> ${:s} server ${main-db-server:address} ${:s} </zeoclient> ${:s} </zodb> ${:s} <zodb index> ${:s} <zeoclient> ${:s} server ${index-db-server:address} ${:s} </zeoclient> ${:s} </zodb> ${:s} ${:s}[server:main] ${:s}use = egg:Paste#http ${:s}host = localhost ${:s}port = 8080 :( I propose to give buildout it's own parser that begaves like ConfigParser, with the exeption that leading whitespace won't be stripped, but will be dedented with textwrap.dedent. There is some potential for breakage as some option values without leading whitespace before would get it now, but my sense is that this would be unlikely. Thoughts? If I did that (and I really want to do this) I'd release a beta to help people assess the impact. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton

6 9