On Tue, Jan 27, 2009 at 5:29 PM, Akira Kitada <akitada@gmail.com> wrote:

Hello Tarek,

I think "apt, yum, etc" would be also used for packaging/distributing apps.

There is already a command that let you create a rpm package (bdist_rpm) out of a python package, There were also a bdist_deb project but it never made it to distutils, also for Debian there's a policy on how to work with python packages : http://www.debian.org/doc/packaging-manuals/python-policy/ Last, this mailing list had a lot of threads about the fact that there's no standard way in Python to work with resources that could be installed in the system, using a LSB-compliant approach. So I don't have (I think no one does at this point) any clear view of what could be done in this area. Tarek -- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/

On Tue, Jan 27, 2009 at 11:18 PM, zooko <zooko@zooko.com> wrote:

...

So I don't have (I think no one does at this point) any clear view of what could be done in this area.

I don't understand what are the potential problems, but so far I've been happy using stdeb to produce .deb's from my Python sdists.

I guess it is perfectly fine if you are your own debian packager. The problem I see is when people that builds packages and are not their own debian packagers. For example, if my application has a log file, it is better under Debian to have it in /var/log/xxx In the meantime, for the same application, I don't want to bother under win32 about that, the log can leave inside the package for instance. So how can I describe in my package, that I will have a log file, and how can the Debian packager can tell to my package that it has to be in /var/log/... ? In other words, what would be the new metadata we could put in the setup.py in the package to minimize the work to be done on stdeb side ? Regards Tarek -- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/

zooko wrote:

On Jan 27, 2009, at 15:00 PM, Tarek Ziadé wrote:

...

There were also a bdist_deb project but it never made it to distutils, also for Debian there's a policy on how to work with python packages : http://www.debian.org/doc/packaging-manuals/python-policy/

Last, this mailing list had a lot of threads about the fact that there's no standard way in Python to work with resources that could be installed in the system, using a LSB-compliant approach.

So I don't have (I think no one does at this point) any clear view of what could be done in this area.

I don't understand what are the potential problems, but so far I've been happy using stdeb to produce .deb's from my Python sdists.

This is not the right solution for distributions maintainers: it is a good tool for individual (it gives you uninstallation, etec...) but .deb packages produced by stddeb are not debian-compatible, and cannot be included in debian proper. This is not a critic of stddeb, I think it is a very good tool and useful tool. The *only* right solution for packaging python modules on Linux distribution is to make it as "easy" for packagers as it is for autoconf packages. Meaning having clear differences between installation, binary, libraries, etc... (what's called resources by setuptools, IIUC), so that maintainers can set it up how they want. This way, python developers do not have to care about debian, and distributions maintainers do not have to care about python (well, not more than now). It is a solved problem: autoconf does it well, and has all the required features, David

Tarek Ziadé wrote:

On Wed, Jan 28, 2009 at 11:45 AM, David Cournapeau <david@ar.media.kyoto-u.ac.jp> wrote:

...

It is a solved problem: autoconf does it well, and has all the required features,

So does it mean that having Distutils generate some kind of "configure.in" template that could be used by autoconf could be the right approach ?

I meant that instead of installing almost everything indistinctly like we do now with distutils/setuptools, we should have something like: python setup.py install --bindir=foodir --sbindir=bardir --mandir=mandir .... E.g. just copying the categories from autoconf (the ones which make sense for python packages). So making a FHS-compliant or something like currently done is the responsibility of the packagers - assuming the directories are correctly set by the developer in the setup.py. The main problem is how to retrieve the different files when it is needed in the package - I would guess pkg_resources would need to be modified as well for that purpose, cheers, David

Tarek Ziadé wrote:

On Wed, Jan 28, 2009 at 12:05 PM, David Cournapeau <david@ar.media.kyoto-u.ac.jp> wrote:

...

Tarek Ziadé wrote:

...

On Wed, Jan 28, 2009 at 11:45 AM, David Cournapeau <david@ar.media.kyoto-u.ac.jp> wrote:

...

It is a solved problem: autoconf does it well, and has all the required features,

So does it mean that having Distutils generate some kind of "configure.in" template that could be used by autoconf could be the right approach ?

I meant that instead of installing almost everything indistinctly like we do now with distutils/setuptools, we should have something like:

python setup.py install --bindir=foodir --sbindir=bardir --mandir=mandir ....

Sounds like new metadata to me

Yes - sorry if I don't use the right terms. Although I am quite familiar with distutils for the build parts (compilation and co) from my work on numpy/scipy, I am not so familiar with the installation parts and its vocabulary.

but besides the executable scripts location, how would you deal with man file from the Python developer point of view ?

What's the problem with man files for python developers ? I am not saying that any developer is required to add a manpage, only that if she/he does though, that should be marked as man. That's quite simple, really: the python developer should not - does not - have to care about FHS, windows, mac os X, whatever, only about marking correctly the different files: mark the pdf as pdf, html doc as html, etc... so that the *user* (the one executing setup.py) can choose whatever he wants, like for autoconf. You *can* do ./configure --mandir=bla --bindir=foo, but most people are fine with ./configure --prefix. It also means if the developer made a mistake in the setup.py, the package maintainer can fix the problem in the setup.py, and sent a patch upstream (so any platform is fixed). IMHO, that's really a part of distutils where explicitly avoiding any policy (where to install what) makes sense, and should be relatively easy to implement.

Yes that was my example (the log file), and the feature needeed to have it should be thaught and implemented on its own, because setuptools's pkg_resources does myriad of things.

Maybe, I don't claim to be familiar with pkg_resources - I am not a user of setuptools, except when I am forced to.

Would you like to write down a detail description of that feature as well ?

I am not sure I will have time to do it - I may underestimate the difficult of some things because I am not familiar with distutils implementation for install-related commands. Doing a description without any design proposal would certainly be possible, though. cheers, David

Tarek Ziadé <ziade.tarek@gmail.com> writes:

Man files seem rather specific to Linux

Hardly. Man pages have been part of every Unix-like operating system (which is to say, all major operating systems except one) since before Linus even discovered a C compiler.

unless we have some kind of script that would know how to transform a reStructuredText file into a man file ?

There is an experimental ‘manpage’ writer for docutils, that is packaged for (at least) Debian [0]. It has a corresponding program, ‘rst2man’ [1], for exactly the task you describe. [0]: <URL:http://packages.debian.org/sid/docutils-writer-manpage> [1]: <URL:http://packages.debian.org/sid/rst2man> -- \ “When cryptography is outlawed, bayl bhgynjf jvyy unir | `\ cevinpl.” —Anonymous | _o__) | Ben Finney

Yesterday I spoke to a couple of hackers who use my allmydata-tahoe project, which is packaged with setuptools. They complained about setuptools getting in their way, and I asked them each to name their top two complaints. The first hacker, David, said: David 1. He can't easily install eggs into an arbitrary directory. Setuptools yells at him loudly when he tries. .pth files work only in magical directories. easy_install.pth makes it impossible to use GNU stow. site.py should just just look for .eggs in all directories on sys.path. David 2. entry_points doesn't work when the svn checkout is just in PYTHONPATH or ".", and fails if there is a non-existent directory on his PYTHONPATH. The second hacker, Nathan, said: Nathan 1: You cannot easily install in a non-standard location like / usr/local/stow/$pkg_name. Nathan 2: You cannot uninstall. Both David's and Nathan's first desire would be somewhat improved by my patch http://bugs.python.org/setuptools/issue54 (be more like distutils with regard to --prefix=). Both of them would be *more* improved by my proposed extension to the standard Python import mechanism: "how to easily consume just the parts of eggs that are good for you" [1] (which is exactly what David suggested, except he suggested site.py do that instead of python importer doing it). David's second problem I don't understand. Perhaps it is a bug in setuptools? Nathan's second problem -- you can't uninstall -- is widely known. Note that using GNU stow is one excellent solution to this problem, so perhaps if the first problem were solved then we would have another tool against the second. Also, most users of setuptools don't seem to realize that they can uninstall almost everything (everything except scripts) simply by removing the .egg. I see that PJE has replied to http://bugs.python.org/setuptools/ issue54 . I'll follow-up on that ticket. Regards, Zooko [1] http://mail.python.org/pipermail/python-dev/2008-March/078243.html --- Tahoe, the Least-Authority Filesystem -- http://allmydata.org store your data: $10/month -- http://allmydata.com/?tracking=zsig

On Wed, Jan 28, 2009 at 11:03 AM, zooko <zooko@zooko.com> wrote:

David 2. entry_points doesn't work when the svn checkout is just in PYTHONPATH or ".", and fails if there is a non-existent directory on his PYTHONPATH.

Probably this is because the package isn't activated, and if it's not activated you can't see its entry point. When a .pth file is on PYTHONPATH, Python won't load it up (it only loads .pth files in some specific locations). So while easy-install.pth would normally activate a package (by adding it to sys.path), with PYTHONPATH that doesn't work. I think the site.py that Setuptools will sometimes create is intended to address this, but it might not always work. Or there might be some entirely different problem I'm unaware of. -- Ian Bicking | http://blog.ianbicking.org

On Wed, Jan 28, 2009 at 1:38 PM, Ben Finney <ben+python@benfinney.id.au> wrote:

Tarek Ziadé <ziade.tarek@gmail.com> writes:

...

...

...

...

It is a solved problem: autoconf does it well, and has all the required features,

Would you like to write down a detailed description of these elements ?

I second this. David has several times invoked "setuptools should do it like autoconf", but that isn't much use without a detailed specification of *exactly what* setuptools should do, written by people who know what is being asked for (like David) and targetted to people who don't know anything about autoconf.

It's a lot of work to come up with such a specification, and will require input from experts and much discussion, but until it's done such requests for change carry little weight.

Right, maybe the first step would be for David to write down the desired change in the wiki, to be discussed in this list. The threads that we had in the past in these topics did not produce any output unfortunately, because it was hard for everyone to read +100 messages If a wiki page is started with simple requirements, it can be used to start to work things out ++ Tarek -- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/

On Wed, Jan 28, 2009 at 5:05 AM, David Cournapeau < david@ar.media.kyoto-u.ac.jp> wrote:

I meant that instead of installing almost everything indistinctly like we do now with distutils/setuptools, we should have something like:

python setup.py install --bindir=foodir --sbindir=bardir --mandir=mandir ....

E.g. just copying the categories from autoconf (the ones which make sense for python packages). So making a FHS-compliant or something like currently done is the responsibility of the packagers - assuming the directories are correctly set by the developer in the setup.py. The main problem is how to retrieve the different files when it is needed in the package - I would guess pkg_resources would need to be modified as well for that purpose,

In the previous discussion (somewhere, I'm too lazy to look it up) people started getting interested in the idea of an improved sdist format. I think the basic idea was to tag files by type. Then (I guess) you'd just do: python setup.py build to get the platform-specific parts of the library built, and then move the files into place based on the descriptions of the files. The actual tool to do this would be external to setup.py and distutils; e.g., you'd have a python-sdist-to-rpm tool, developed entirely separately from distutils or setuptools. Potentially this would make it easier to provide your own file tags, so you could adapt a library without requiring immediate upstream support or patches. As you mention, there would have to be some extension to pkg_resources (or an equivalent library) to handle finding these files at runtime. Getting a runtime in place is probably the harder thing, as it is more intrusive for the upstream developers. -- Ian Bicking | http://blog.ianbicking.org

On Thu, Jan 29, 2009 at 7:53 AM, Josselin Mouette <joss@debian.org> wrote:

Le mercredi 28 janvier 2009 à 13:16 -0600, Ian Bicking a écrit :

...

As you mention, there would have to be some extension to pkg_resources (or an equivalent library) to handle finding these files at runtime. Getting a runtime in place is probably the harder thing, as it is more intrusive for the upstream developers.

As I have already explained in the previous discussion, this could easily be solved just like autoconf does, with an automatically generated config.py file that would hold all variables set at build time.

"Easily solved" is a misstatement -- you can lead a horse to water, but you can't make him drink. In this case, you can provide infrastructure for library authors, but you can't make them use it. The solution has to be sufficiently simple that people who don't care at all about FHS or Linux packages won't find it onerous to use, because if they do find it onerous then they won't use it. But anyway, where would this config.py go and what would it look like? -- Ian Bicking | http://blog.ianbicking.org

On Thu, Jan 29, 2009 at 10:51 PM, Ben Finney <ben+python@benfinney.id.au> wrote:

Josselin Mouette <joss@debian.org> writes:

...

Le mercredi 28 janvier 2009 à 13:16 -0600, Ian Bicking a écrit :

...

As you mention, there would have to be some extension to pkg_resources (or an equivalent library) to handle finding these files at runtime. Getting a runtime in place is probably the harder thing, as it is more intrusive for the upstream developers.

As I have already explained in the previous discussion, this could easily be solved just like autoconf does, with an automatically generated config.py file that would hold all variables set at build time.

I would recommend choosing (earlier rather than later) a different name for that file. 'config.py' suggests rather the run-time configuration for the program.

Perhaps 'setup_config.py'? Something that makes it clear that the configuration is intended for the setup and installation, *not* the running Python package.

what about "metadata.py" (that can be used by setup.py as well) ++ Tarek

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Ben Finney wrote:

Josselin Mouette <joss@debian.org> writes:

...

Le mercredi 28 janvier 2009 à 13:16 -0600, Ian Bicking a écrit :

...

As you mention, there would have to be some extension to pkg_resources (or an equivalent library) to handle finding these files at runtime. Getting a runtime in place is probably the harder thing, as it is more intrusive for the upstream developers. As I have already explained in the previous discussion, this could easily be solved just like autoconf does, with an automatically generated config.py file that would hold all variables set at build time.

I would recommend choosing (earlier rather than later) a different name for that file. ‘config.py’ suggests rather the run-time configuration for the program.

Perhaps ‘setup_config.py’? Something that makes it clear that the configuration is intended for the setup and installation, *not* the running Python package.

I would argue against making it a .py file at all: make it an INI file, or something. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJgjKg+gerLs4ltQ4RAicLAKC0/IJnCA7dQqXrE58mQpLHNyJE9wCfQ3LO nW98d8KkYqszyQNUAGOnsik= =JHM5 -----END PGP SIGNATURE-----

On Thu, Jan 29, 2009 at 11:54:38PM +0100, Tarek Ziadé wrote:

On Thu, Jan 29, 2009 at 11:50 PM, Tres Seaver <tseaver@palladion.com> wrote:

...

...

Perhaps 'setup_config.py'? Something that makes it clear that the configuration is intended for the setup and installation, *not* the running Python package.

I would argue against making it a .py file at all: make it an INI file, or something.

setup.cfg could be used in that case.

setup.cfg is a configuration file to be edited by developers. Going to modify that at build time is guaranteed to result in unhappy people at some stage I reckon.

But having a .py file would allow having some values calculated dynamically,

But those could be calculated at the time setup.py runs too. If it's not directly a .py file there needs to be an stdlib module to access it easily, manually using ConfigParser is not an option IMHO, I'd rather do `import foo; foo.prefix'. An argument against a generated .py file is that this won't work for single-module python distributions. But having a .cfg file or something next to the module/package might defeat the point of trying to help the FHS in violating it already. OTOH .egg-info does that too and seems to be accepted currently. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org

On Fri, Jan 30, 2009 at 01:21:12AM +0100, Tarek Ziadé wrote:

On Fri, Jan 30, 2009 at 12:34 AM, Floris Bruynooghe <floris.bruynooghe@gmail.com> wrote:

...

An argument against a generated .py file is that this won't work for single-module python distributions. But having a .cfg file or something next to the module/package might defeat the point of trying to help the FHS in violating it already. OTOH .egg-info does that too and seems to be accepted currently.

Well, the egg-info file is the static version of setup.py metadata in some way, I would find a new file redundant.

I am really curious though, to see what the file would contain..

I imagine things like libdir, prefix, datadir, docdir and other things copied from autoconf. Where the defaults would be something like: prefix = sys.prefix libdir = sys.prefix/lib/pythonX.Y/site-packages/pkgname datadir = sys.prefix/share/mypackage docdir = sys.prefix/share/doc/mypackage

from my current understanding, and having in mind to introduce such a file in a way things can gently evolve:

What if an egg-info-like file was present in the package from the very beginning, describing the package metadata, and used by setup.py, *and* at build time by third party tools. (with setup.py knowing that it can be changed at run time)

I don't see why moving all the metadata to egg-info would improve things. You could easily(?) deprecate the `package_data' and `data_files' keywords to setup() and replace them with `doc_files', `data_files', `man_files', `config_files', etc. And depending on which --sysconfdir, --datadir, etc options where used to setup.py install_egg_info would write the correct values for datadir etc. The runtime would then use your pkginfo.get_metadata() to find the files. It might even be possible to do all of this inside distutils without breaking backwards compatibility, just deprecating some keywords once people start liking and using it. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org

On Fri, Jan 30, 2009 at 08:35:07PM +0100, Tarek Ziadé wrote:

David Cournapeau has sent me a first draft of a document, you might want to look at.

http://wiki.python.org/moin/distutils-autoconf

Might be a good basis for a common understanding

Looks useful! Not sure where to comment on this so doing it here... Under Categories I think "configuration files (sysconfdir)" and "statedir" are missing. I also don't know in how far it's useful to provide a separate "logdir", autotools doesn't seem to do this it rather let's you construct one using statedir. As for "html doc" and "pdf doc" I'm not sure why they are separate, surely they can both just be constructed from "docdir"? The trick is to make it fine grained enough so packagers will be able to customise it enough and leave it coarse enough so that developers will want to classify their files properly. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org

At 06:39 PM 1/30/2009 +0000, Floris Bruynooghe wrote:

...

On Fri, Jan 30, 2009 at 12:34 AM, Floris Bruynooghe <floris.bruynooghe@gmail.com> wrote:

...

An argument against a generated .py file is that this won't work for single-module python distributions. But having a .cfg file or something next to the module/package might defeat the point of trying to help the FHS in violating it already. OTOH .egg-info does that too and seems to be accepted currently.

Well, the egg-info file is the static version of setup.py

On Fri, Jan 30, 2009 at 01:21:12AM +0100, Tarek Ziadé wrote: metadata in some way,

...

I would find a new file redundant.

I am really curious though, to see what the file would contain..

I imagine things like libdir, prefix, datadir, docdir and other things copied from autoconf. Where the defaults would be something like:

prefix = sys.prefix libdir = sys.prefix/lib/pythonX.Y/site-packages/pkgname datadir = sys.prefix/share/mypackage docdir = sys.prefix/share/doc/mypackage

I'm confused by the above lines: do you mean the *project* name, or the name of some package within the project? What if the project contains no packages, only modules? What is libdir for?

...

from my current understanding, and having in mind to introduce such a file in a way things can gently evolve:

What if an egg-info-like file was present in the package from the very beginning, describing the package metadata, and used by setup.py, *and* at build time by third party tools. (with setup.py knowing that it can be changed at run time)

I don't see why moving all the metadata to egg-info would improve things.

Moving from code to data means better tool interoperability. setup.py sucks as a format for obtaining data, especially since many distutils newbs hardcode all sorts of rubbish in their setup.py files, like writing to files without paying attention to the command line.

You could easily(?) deprecate the `package_data' and `data_files' keywords to setup() and replace them with `doc_files', `data_files', `man_files', `config_files', etc. And depending on which --sysconfdir, --datadir, etc options where used to setup.py install_egg_info would write the correct values for datadir etc. The runtime would then use your pkginfo.get_metadata() to find the files.

...in which case, why not just put all the files in the .egg-info to begin with? Meanwhile, getting rid of package_data or *requiring* a runtime API to access files is going to be a major barrier to adoption in the short run.

"P.J. Eby" <pje@telecommunity.com> writes:

At 06:39 PM 1/30/2009 +0000, Floris Bruynooghe wrote:

...

I imagine things like libdir, prefix, datadir, docdir and other things copied from autoconf. Where the defaults would be something like:

prefix = sys.prefix libdir = sys.prefix/lib/pythonX.Y/site-packages/pkgname datadir = sys.prefix/share/mypackage docdir = sys.prefix/share/doc/mypackage

I'm confused by the above lines: do you mean the *project* name, or the name of some package within the project? What if the project contains no packages, only modules? What is libdir for?

To clarify: Here again we come up against the unfortunate choice of the term “package” in Python to mean something confisingly different from what it means to most people dealing with software. I imagine Floris meant what most hackers mean by “package”, which is what Python perversely calls a “distribution”. That may or may not be what setuptools calls a “project”, I've never been clear on that :-/ -- \ “Everything is futile.” —Marvin of Borg | `\ | _o__) | Ben Finney

Tarek Ziadé <ziade.tarek@gmail.com> writes:

On Sat, Jan 31, 2009 at 12:09 AM, Ben Finney <ben+python@benfinney.id.au> wrote:

...

I imagine Floris meant what most hackers mean by "package", which is what Python perversely calls a "distribution". That may or may not be what setuptools calls a "project", I've never been clear on that

let's all use this then maybe ?

http://wiki.python.org/moin/PythonPackagingTerminology

Who is “us”? You and I know about those terms and their non-standard meanings in Python, but how is anyone supposed to know who comes to Python expecting things to mean what they mean in the majority of other software projects? Given the inertia on both sides of this terminological schism, and the fact that these concepts are not exactly things that an outsider would even expect to have to learn new terms for, I don't think it matters how prominent such a “Python packaging terminology” web page is. No newcomer is going to be looking for it, and IMO rightly so. The damage is done and, short of educating everyone the first time they learn about software, we in the Python community have the burden of teaching these non-standard meanings indefinitely every time the conflicting meanings raise themselves. -- \ “Pinky, are you pondering what I'm pondering?” “Well, I think | `\ so (hiccup), but Kevin Costner with an English accent?” —_Pinky | _o__) and The Brain_ | Ben Finney

Tarek Ziadé <ziade.tarek@gmail.com> writes:

On Sat, Jan 31, 2009 at 12:32 AM, Ben Finney <ben+python@benfinney.id.au> wrote:

...

Given the inertia on both sides of this terminological schism, and the fact that these concepts are not exactly things that an outsider would even expect to have to learn new terms for, I don't think it matters how prominent such a "Python packaging terminology" web page is. No newcomer is going to be looking for it, and IMO rightly so.

For the newcomers:

Well, between reading some thread where there is +100 mails and reading 4 lines of a definition for each word in a FAQ or Taxonomy...

I'm not sure what your point is with that paragraph; I presume you're saying that a short online definition is quicker to read than a long discussion thread. If so, I've already addressed that: The online definition document won't *be* read by the representative newcomer until it's already caused them confusion, most likely in a long discussion thread. That's because it won't even occur to most of them that Python would be perverse enough to use these terms in a contradictory meaning to the majority of the software world. And, in my opinion, they're right to have that expectation. To think otherwise is to blame the victim. From that, it follows that we can't expect the newcomer to have any awareness that this terminological issue even *exists* until it bites them. -- \ “I went to a fancy French restaurant called ‘Déjà Vu’. The head | `\ waiter said, ‘Don't I know you?’” —Steven Wright | _o__) | Ben Finney

On Fri, Jan 30, 2009 at 03:19:05PM -0600, Ian Bicking wrote:

On Fri, Jan 30, 2009 at 12:39 PM, Floris Bruynooghe < floris.bruynooghe@gmail.com> wrote:

...

I imagine things like libdir, prefix, datadir, docdir and other things copied from autoconf. Where the defaults would be something like:

prefix = sys.prefix libdir = sys.prefix/lib/pythonX.Y/site-packages/pkgname datadir = sys.prefix/share/mypackage docdir = sys.prefix/share/doc/mypackage

Ok, that was supposed to read: prefix = sys.prefix libdir = sys.prefix/lib/pythonX.Y/site-packages/pkgname datadir = sys.prefix/share/mypackage docdir = sys.prefix/share/doc/mypackage

I wouldn't want to use those. What goes in libdir, what goes in datadir? I don't know, and frankly the distinctions start getting really arbitrary.

You won't have to know or care. What you define with `package_data', `py_modules' and `package_dir' would just be translated to it (they all go to libdir), it's a distutils (or other tool that's consuming the metadata) implementation detail. If you'd opt to tag a file as going into "datadir" for example instead of using `data_files' then you'd have to use a pkg_resources-like API to retrieve that (with one more argument to say that you want a "datadir" resource). Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org

Le samedi 31 janvier 2009 à 14:10 +0000, Floris Bruynooghe a écrit :

On Fri, Jan 30, 2009 at 11:02:36PM +0000, Floris Bruynooghe wrote:

...

Ok, that was supposed to read:

prefix = sys.prefix libdir = sys.prefix/lib/pythonX.Y/site-packages/pkgname datadir = sys.prefix/share/mypackage docdir = sys.prefix/share/doc/mypackage

Just how many time will I get this wrong??

libdir = sys.prefix/lib/pythonX.Y/site-packages datadir = sys.prefix/share/<project_name> docdir = sys.prefix/share/doc/<project_name>

Please don’t add more confusion by adding a libdir meaning something different from the existing autoconf macros. prefix = sys.prefix libdir = prefix+"/lib" # not sys.prefix; redefining prefix must lead to redefining the others pkglibdir = libdir+"/"+pkgname datadir = prefix+"/share" pkgdatadir = datadir+"/"+pkgname pythondir = libdir+"pythonX.Y/site-packages" pkgpythondir = pythondir+modulename ... (And you can add the prefix / exec_prefix distinction for systems that require it.) -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `- our own. Resistance is futile.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Ben Finney wrote:

Ian Bicking <ianb@colorstudy.com> writes:

...

On Fri, Jan 30, 2009 at 12:39 PM, Floris Bruynooghe < floris.bruynooghe@gmail.com> wrote:

...

I imagine things like libdir, prefix, datadir, docdir and other things copied from autoconf. Where the defaults would be something like:

prefix = sys.prefix libdir = sys.prefix/lib/pythonX.Y/site-packages/pkgname datadir = sys.prefix/share/mypackage docdir = sys.prefix/share/doc/mypackage I wouldn't want to use those. What goes in libdir, what goes in datadir?

Again, I expect Floris is using these terms in their traditional meanings.

“library” would be a collection of executable code intended for use as a unit; what Pythoncalls a “package” (or the degerate case of a “module”).

“data” would be non-executable files used by the package that don't fit any other (e.g. “documentation”) classification.

...

I don't know, and frankly the distinctions start getting really arbitrary.

Hopefully that clarifies.

...

I would rather see something like pkg_resources existing API, where there is some file that maps out how the local names of files (where they'd be in a checkout) map to their installed location, then the pkg_resources code could finds the real location of the file.

This loses the indirection that is so sorely needed of tagging resources by *type*, so that their install location can be decided on that basis.

Deciding on a file-by-file basis where files get installed is something that distributors and packagers already have to do, and it's a massive pain.

Allowing the developer to tag resources by type is a sensible division of responsibility: the developer knows the *purpose* (and therefore type) of the resource, and the packager knows the appropriate *location* on the filesystem for resources of a particular type.

The need for those distinctions derives from a particular distributor's / packager's "religion": e.g., the insistance that "template" files (software, but not "executable" by some criterion) mut not be kept in the same directory with the other software. That rule is important for some packagers, and completely mystifying / irrelevant to the original develpoers or to packagers with a different model. Right now, making the 'pkg_resources' jump can be motivated by cross-platform concerns, e.g. a desire to allow the "package_data" files to be loaded from a zipfile / egg. Anything more complicated at *runtime* (as opposed to metadata added to the 'sdist') is harder to motivate. The "uptake" problem here is both to motivate the non-believer to help with the distinctions, and to make the burden of doing so as minimal as possible. This discussion has broken down before at exactly this point, IIRC. As a packaging-agnostic developer, I'm +0 on the idea of making it easier to package my software: - - I do appreciate having folks able to install the software easily; - - However, complexifying the software itself to make some packagers happy is a burden, and a potential source of bugs; - - Sometimes, packagesrs *break* my software in ways I can't support. If we can drive the first concern down, I can live better with the second (and perhaps it will become less likely). Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJhBMY+gerLs4ltQ4RAl5CAKCHP1Y3ja65uQZGRowdu8C+R2lcJwCgmNbV JjXgoibfZumIRJjYmlsr9XE= =//L/ -----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 David Cournapeau wrote:

Tres Seaver wrote:

...

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

Ben Finney wrote:

...

Ian Bicking <ianb@colorstudy.com> writes:

...

On Fri, Jan 30, 2009 at 12:39 PM, Floris Bruynooghe < floris.bruynooghe@gmail.com> wrote:

...

I imagine things like libdir, prefix, datadir, docdir and other things copied from autoconf. Where the defaults would be something like:

prefix = sys.prefix libdir = sys.prefix/lib/pythonX.Y/site-packages/pkgname datadir = sys.prefix/share/mypackage docdir = sys.prefix/share/doc/mypackage

I wouldn't want to use those. What goes in libdir, what goes in datadir?

Again, I expect Floris is using these terms in their traditional meanings.

“library” would be a collection of executable code intended for use as a unit; what Pythoncalls a “package” (or the degerate case of a “module”).

“data” would be non-executable files used by the package that don't fit any other (e.g. “documentation”) classification.

...

I don't know, and frankly the distinctions start getting really arbitrary.

Hopefully that clarifies.

...

I would rather see something like pkg_resources existing API, where there is some file that maps out how the local names of files (where they'd be in a checkout) map to their installed location, then the pkg_resources code could finds the real location of the file.

This loses the indirection that is so sorely needed of tagging resources by *type*, so that their install location can be decided on that basis.

Deciding on a file-by-file basis where files get installed is something that distributors and packagers already have to do, and it's a massive pain.

Allowing the developer to tag resources by type is a sensible division of responsibility: the developer knows the *purpose* (and therefore type) of the resource, and the packager knows the appropriate *location* on the filesystem for resources of a particular type.

The need for those distinctions derives from a particular distributor's / packager's "religion": e.g., the insistance that "template" files (software, but not "executable" by some criterion) mut not be kept in the same directory with the other software. That rule is important for some packagers, and completely mystifying / irrelevant to the original develpoers or to packagers with a different model.

That's not a religion: it has rationales. You may not agree with them, both those splits are not arbitrary.

I don't equate "religious" with "irrational": I mean it in the sense that people have different assumptions ("postulate sheaths", if you like a more mathematical approach) which are taken as "given." For isntance, the whole idea of dividing architecure-specific bits from architecture-independent bits is legacy (from my POV, of course) of a day when disk space was expensive, and when sysadmins wanted / needed to share installed software across multiple, heterogenous machines. In today's world, I bet that is a percent-of-a-percent minority of all installed Unix-like boxes. Calling .py files "data" is another choice which seems (to me) arbitrary: in my view, .py files, as well as others, are "software", not data, and should be kept together with the other software (e.g., templates used for rendering HTML) that they are distributed with.

Each platform has its own way of dealing with this, and doing against it is both futile and counter-productive.

I'm arguing that making it easier for platform-focused people to apply their policies is a good thing, but is in tension with keeping the software itself simple and supportable across multiple platforms. For instance, adding standardized metadata to setup.py to help packagers is a reasonable thing to ask of Python developers. It is a harder thing to ask them to use an API to load support files (like the templates I mentioned), but that can be motivated by cross-platform concerns. On the other hand, changing my software so that it uses some Debian- (or BSD-, or Windows-specific) policy at runtime to load those files is Right Out (TM).

Of course some of the distributions aspects are irrelevant to you - but so are some of your interests to packagers. At the risk of sounding obvious: developers and packagers do not have the same goals, and they sometimes conflict. But you cannot expect "not to care" and "my software will be available everywhere" at the same time. That's why those discussion are tiresome and difficult: there has to be some compromise. There is not right solution,

My original post higlighted the tension in those goals, and suggested some of the compromises (repeated above) which packagers might reasonably ask developers to make. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJhLHn+gerLs4ltQ4RAhduAKDQw0iJK40uR/CdoPAweMsX0cXjjQCeMQKb nnINTY3DHvN5EWbzUTqdyPY= =fsQ6 -----END PGP SIGNATURE-----

Ian Bicking wrote:

On Fri, Jan 30, 2009 at 12:39 PM, Floris Bruynooghe <floris.bruynooghe@gmail.com <mailto:floris.bruynooghe@gmail.com>> wrote:

I imagine things like libdir, prefix, datadir, docdir and other things copied from autoconf. Where the defaults would be something like:

prefix = sys.prefix libdir = sys.prefix/lib/pythonX.Y/site-packages/pkgname datadir = sys.prefix/share/mypackage docdir = sys.prefix/share/doc/mypackage

I wouldn't want to use those. What goes in libdir, what goes in datadir? I don't know, and frankly the distinctions start getting really arbitrary.

They are not arbitrary - they come from standard usage and have a rationale, at least on Unix (datadir for arch independent, and libdir for arch dependent, to simplify). But you mostly do not need to care, as a developer: .py files would be considered as data files, extensions as arch-dependent, etc... The main category which needs special care is documentation, and I think I am not the only one thinking that's one thing missing in distutils ATM.

I would rather see something like pkg_resources existing API, where there is some file that maps out how the local names of files (where they'd be in a checkout) map to their installed location, then the pkg_resources code could finds the real location of the file.

I am not sure I understand how this would help OS packagers - this does not sound as the same problem at all. David

On Sat, Jan 31, 2009 at 10:37:56AM -0500, P.J. Eby wrote:

At 05:40 PM 1/31/2009 +0900, David Cournapeau wrote:

...

But you mostly do not need to care, as a developer: .py files would be considered as data files, extensions as arch-dependent, etc...

If this is true, then there's no need to distinguish between .py files and any other data files - they both belong in /share to begin with, not in /lib. Or else they ALL belong in /lib. The entire "FHS demands they be split" concept is wrong from the get-go, under that interpretation.

I don't think trying to split off .py and .so files is a good idea. Python will just break because of namespace package issues etc. I recon all .py/.pyc/.pyo/.so/.pyd files should be treated as arch dependent, and other data files like templates and INI files or anything else you have should be tagged as data files. Then there's of course tagging things as binaries, documentation, configuration and state files etc. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org

On Sun, Feb 01, 2009 at 02:32:06PM +0900, David Cournapeau wrote:

Floris Bruynooghe wrote:

...

On Sat, Jan 31, 2009 at 10:37:56AM -0500, P.J. Eby wrote:

...

At 05:40 PM 1/31/2009 +0900, David Cournapeau wrote:

...

But you mostly do not need to care, as a developer: .py files would be considered as data files, extensions as arch-dependent, etc...

If this is true, then there's no need to distinguish between .py files and any other data files - they both belong in /share to begin with, not in /lib. Or else they ALL belong in /lib. The entire "FHS demands they be split" concept is wrong from the get-go, under that interpretation.

I don't think trying to split off .py and .so files is a good idea.

That's how it is done in debian and ubuntu packages. So if you want to make it easier for those distributions, you don't have a choice.

Hmm fair enough, I must have missed that last time I looked at the implementation of python-support and python-central. But both take the burden of having to create a symlink farm because of this though. And to be honest I think the motivation for this is supporting multiple python versions without having to duplicate all .py files rather then the FHS. But the great thing about the proposal is that it doesn't even matter. Files will be easily tagged/detected as "python modules" and "python extension modules" and the tools consuming this information can decide to do what they want with them. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org

On Sat, 31 Jan 2009 10:37:56 -0500, "P.J. Eby" <pje@telecommunity.com> wrote:

If this is true, then there's no need to distinguish between .py files and any other data files - they both belong in /share to begin with, not in /lib. Or else they ALL belong in /lib. The entire "FHS demands they be split" concept is wrong from the get-go, under that interpretation.

Not taking away from that.... but in windows things have their own directories too and python files should go in the right places also. Best place for packages -> \Program Files\Common Files\Python x.x\Packages Best place for python -> \Program Files\Python x.x Under linux - Best place for packages -> /usr/local/Python x.x/Packages. (a python developer shouldn't go looking in system directories for a source package he may want to look at) Putting files where people expect to find them, doesn't cost more. The only reason things are the way they are is because of history. But at the same time, if we want our software to "live" in an operating system we must follow their rules to be regarded as good citizens. :-) Regards David

David Cournapeau <david@ar.media.kyoto-u.ac.jp> writes:

They are not arbitrary - they come from standard usage and have a rationale, at least on Unix

Agreed with this.

(datadir for arch independent, and libdir for arch dependent, to simplify).

I think you've not only simplified, you've done so in the wrong direction. I'd say instead that “datadir” is for *non-executable* files, and “libdir” for executable. That something is non-executable usually means that it's “architecture-independent”, and traditionally, “executable” has meant “architecture-dependent”, but I don't think that's a necessary distinction here. Viz. the placement of the Python standard library, including mostly architecture-independent executable files, in a “libdir”.

But you mostly do not need to care, as a developer: .py files would be considered as data files, […]

That would lead to a misguided attempt to keep ‘foo.py’ separate from the resulting ‘foo.pyc’, which would be far more pain than it's worth AFAICT. Rather, ‘foo.py’ (and ‘foo.pyc’) files would both be identified as *executable* resources, and thence installed to the same location. -- \ “Too many Indians spoil the golden egg.” —Sir Joh | `\ Bjelke-Petersen | _o__) | Ben Finney

David Cournapeau <david@ar.media.kyoto-u.ac.jp> writes:

Ben Finney wrote:

...

I think you've not only simplified, you've done so in the wrong direction. I'd say instead that “datadir” is for *non-executable* files, and “libdir” for executable.

It depends on what you mean by executable: it is obviously wrong when executable = have the executable bit, so what do you mean by executable ?

We could diverge into a long discussion about what “executable” means; but let's not. I'm willing to allow a common sense meaning to serve for the purpose of this discussion. The issue is less about what Ben Finney or David Cornapeau think it means. The issue is more whether it's a useful distinction.

...

That something is non-executable usually means that it's “architecture-independent”, and traditionally, “executable” has meant “architecture-dependent”,

I don't understand this in the context of python: most python executables are architecture independent (python scripts).

Yes, that's mainly my point. Whether a particular set of bits is “architecture-dependent” or “architecture-independent” is rather orthogonal, in Python, to whether it's an executable program library. -- \ “Jealousy: The theory that some other fellow has just as little | `\ taste.” —Henry L. Mencken | _o__) | Ben Finney

On Sun, Feb 01, 2009 at 02:15:25PM +0900, David Cournapeau wrote:

The arch independent/dependent is much clearer IMHO, and is the vocabulary used by the FHS, AFAIK. It is certainly the one used by the configure script help:

--libdir=DIR object code libraries [EPREFIX/lib] --datarootdir=DIR read-only arch.-independent data root [PREFIX/share] --datadir=DIR read-only architecture-independent data [DATAROOTDIR]

The FHS, just like autoconf, does say that datadir must be arch-independent. But AFAIK they never say that libdir is not allowed to be architecture independent. But I'm suspecting this is very much off-topic by now. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org

Ian Bicking <ianb@colorstudy.com> writes:

I'm just about ready to run screaming from this discussion... so no, I want no part of determining what the "right" place for these files is

As I understand it, you are speaking from the perspective of a “developer”; that is, someone who intends to share the software work you've written, but leave the packaging of that work for particular OSen to other people. In that case, you're *not* being asked to determine what the “right” installation location of files is; that's exactly the shift in control that's being requested here, by divorcing the decision about what the *purpose* of a file is from what its installed *location* will be. Instead, you're being asked only to determine which category a file belongs to; and those categories are (modulo some historical quirks, due to David's obvious strong preference for the C-language-focussed ‘autoconf’) mostly about the *purpose* of a file — precisely to allow that determination to be made without the same person needing to know about install locations.

I imagine some file like package-data.conf, containing:

data mypackage/templates/ docs docs/_build/

At least in this example the first word is some tag, and the second is the directory, or files, or maybe a wildcard or something determining what files that tag applies to. Everything not declared but present in a package (or as a module) would default to "lib", and everything outside of that (like the setup.py file) would be "ignore".

This is (with implementation differences) entirely compatible with what David's proposing, AFAICT. The message you responded to, though, was discussing the complementary part: the install location of (in your example) ‘docs’, ‘data’, ‘lib’, etc. If you want no part of that, you're welcome to avoid it; and that's exactly the kind of partition that is being striven for here.

On installation, you'd write something like mypackage.egg-info/file-locations.txt, that might look like:

mypackage/templates/ -> /usr/share/mypackage/templates/ docs/_build/ -> /usr/share/doc/mypackage

(I'm not sure what the syntax would look like, but whatever.)

Here, though, you lose the very useful categorisation you set up above. AIUI, David is proposing that install location be a mapping not from file path to file path, but from file *purpose* (as categorised above) to install file path. -- \ “For every complex problem, there is a solution that is simple, | `\ neat, and wrong.” —Henry L. Mencken | _o__) | Ben Finney

Ben Finney wrote:

Instead, you're being asked only to determine which category a file belongs to; and those categories are (modulo some historical quirks, due to David's obvious strong preference for the C-language-focussed ‘autoconf’) mostly about the *purpose* of a file — precisely to allow that determination to be made without the same person needing to know about install locations.

I've just found out that cabal ("distutils for haskell") supports most of those options, BTW: http://www.haskell.org/cabal/release/latest/doc/users-guide/builders.html#se... And haskell is not really a C-language-focused language :) I don't know anything about haskell, but it may be interesting to look at how they are doing things, David

Hi David, I am learning a lot from this list.... On Mon, 02 Feb 2009 12:31:18 +0900, David Cournapeau <david@ar.media.kyoto-u.ac.jp> >> > prefix = sys.prefix

...

> libdir = sys.prefix/lib/pythonX.Y/site-packages/pkgname > datadir = sys.prefix/share/mypackage > docdir = sys.prefix/share/doc/mypackage

.. I never said it was self-evident. And about the part of determining what the right place is: the whole point is that you, as a developer, won't have to care :)

As a python developer... I think you do.... :-) First thing you want to go looking for is the documentation. omg - that can be impossible to find for most packages. Lucky I was trained in the school of 'if you don't understand the program - go debug the code'.... Sorry to say this, but I didn't know packages even included documentation. It is d&*med difficult to find in linux.

I think this does not have to be as complicated. We already need to tag some files in the setup.py file, so we could do:

from distutils.core import setup, Extension

setup(name='example', version='1.0', description='Example', author='You', author_email='me@python.net', packages=['example', 'example.foo'], ext_modules=[Extension('example.foo._bar', ['example/foo/bar.c'])], pdfdoc=['doc/doc.pdf'], htmldoc=['doc/html'], pacakge_data=['example/data/data1.dat', 'example/data/data2.dat'] )

Note that except pdfdoc and htmldoc, all this is already there in distutils. Then, you could do

python setup.py install --prefix=tmp

ending up in (on Linux):

tmp/lib/python-$PYVER/site-packages/example/__init__.py ... tmp/lib/python-$PYVER/site-packages/example/foo/bar/_bar.so tmp/lib/python-$PYVER/site-packages/example/data/data1.dat ... tmp/lib/python-$PYVER/site-packages/example/doc/doc.pdf

Very interesting... Regards David

On Sun, Feb 01, 2009 at 03:29:52PM -0600, Ian Bicking wrote:

I'm just about ready to run screaming from this discussion...

Thanks for not doing so

Sorry, I didn't describe what I meant.

I imagine some file like package-data.conf, containing:

data mypackage/templates/ docs docs/_build/

At least in this example the first word is some tag, and the second is the directory, or files, or maybe a wildcard or something determining what files that tag applies to. Everything not declared but present in a package (or as a module) would default to "lib", and everything outside of that (like the setup.py file) would be "ignore".

On installation, you'd write something like mypackage.egg-info/file-locations.txt, that might look like:

mypackage/templates/ -> /usr/share/mypackage/templates/ docs/_build/ -> /usr/share/doc/mypackage

(I'm not sure what the syntax would look like, but whatever.) Then when I did something like pkg_resources.resource_string('mypackage', 'mypackage/templates') it'd look up this file to find the location (in the absence of the file it'd act like it does currently)

I really like this idea. It seems to solve most of the issues. For some reason my brain kept thinking that categorising data by it's purpose is a really natural thing to do, but that seems not so. With this solution both packagers and developers can be happy and the required patch to be added by a packager is really simple (add a file with the tags). It is even entirely possible to provide two APIs with this system, the one you just described and one for people like me who'd rather say get_resource('mypackage', 'datafile', 'templates/foo.xml'). Cheers Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Jan 28, 2009, at 5:45 AM, David Cournapeau wrote:

This is not the right solution for distributions maintainers: it is a good tool for individual (it gives you uninstallation, etec...) but .deb packages produced by stddeb are not debian-compatible, and cannot be included in debian proper. This is not a critic of stddeb, I think it is a very good tool and useful tool.

The *only* right solution for packaging python modules on Linux distribution is to make it as "easy" for packagers as it is for autoconf packages. Meaning having clear differences between installation, binary, libraries, etc... (what's called resources by setuptools, IIUC), so that maintainers can set it up how they want. This way, python developers do not have to care about debian, and distributions maintainers do not have to care about python (well, not more than now).

It is a solved problem: autoconf does it well, and has all the required features,

I'd like to make a radical suggestion: upstream authors should never have to worry about building distribution blobs. In my ideal world, I would make a release by tagging my release in whatever vcs I'm using, then I would tell cheeseshop, "hey I just released version 3.1 and it is here" where "here" means whatever native vcs syntax points to the revision I just released. Then PyPI would do the "coagulation" into distribution formats and distribute it amongst its worldwide mirrors, all automatically. I as the Python developer don't want to know about eggs, tarballs, debs, rpms, whatevers, I just want to write some software. I'm happy to add a bit of metadata to my setup.py to play ball, but otherwise I really just want one command to release my code and then magically have it appear available to everybody. If that's not feasible <wink> then the next best thing would be to just spin the source tarball and upload that. Tarballs I can handle. <wink> On the other end, when I zc.buildout, or paver, or easy_install, or aptitude install, or emerge, or port install, or whatever, it would go out to the Worldwide Python Distribution Network and pull down the proper blobby thing to install based on what I'm trying to do, e.g. an egg if I'm developing, a deb if I'm installing into my system Python, etc. Again, I don't really care and shouldn't have to know. Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) iQCVAwUBSYBYgnEjvBPtnXfVAQKM/wQAnoINOJlvFhFB7yHemIujIveFKsZv4cm7 eekwAwEnaIp8fUP2xAuvMZrnGmz51cjRGh7ih24DcZtqPG3pwjrha8+DR7vY9/08 Oa0LKyTvWXDoqC/eCeyTxSvGHe733eDMSp+G283o/XLbgtNUtENfr17+DruRi+9E pYIJZSYIffc= =DZic -----END PGP SIGNATURE-----

Barry Warsaw wrote:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Jan 28, 2009, at 5:45 AM, David Cournapeau wrote:

...

This is not the right solution for distributions maintainers: it is a good tool for individual (it gives you uninstallation, etec...) but .deb packages produced by stddeb are not debian-compatible, and cannot be included in debian proper. This is not a critic of stddeb, I think it is a very good tool and useful tool.

The *only* right solution for packaging python modules on Linux distribution is to make it as "easy" for packagers as it is for autoconf packages. Meaning having clear differences between installation, binary, libraries, etc... (what's called resources by setuptools, IIUC), so that maintainers can set it up how they want. This way, python developers do not have to care about debian, and distributions maintainers do not have to care about python (well, not more than now).

It is a solved problem: autoconf does it well, and has all the required features,

I'd like to make a radical suggestion: upstream authors should never have to worry about building distribution blobs.

that you think it is radical is quite saying about the state of affairs in python IMHO :) For me, it is obvious that the upstream author should not have to worry about debian if he is not producing .deb. The problem is just that today, we make it much harder for packagers than it needs to be. cheers, David

Le mercredi 28 janvier 2009 à 21:18 -0500, david.lyon@preisshare.net a écrit :

Sometimes it is easier under linux to get an operating system python package (from debian for example) installed than it is to get a pure python package...

That seems very wrong to me.....

So I am agreeing...

What can we do to fix this?

As the whole point of packaging is to ease the installation of software, it is a pretty normal situation to find packages are easier to install. If you know cases where they are not, that would definitely be something to fix. -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `- our own. Resistance is futile.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Jan 29, 2009, at 8:57 AM, Josselin Mouette wrote:

Le mercredi 28 janvier 2009 à 08:07 -0500, Barry Warsaw a écrit :

...

I'd like to make a radical suggestion: upstream authors should never have to worry about building distribution blobs.

This is just silly. You don’t have to worry about the distribution’s internals, and what is specific to each package format, but the very idea of developing without any kind of knowledge about how the software will integrate on a system is a guaranteed recipe for a development disaster.

That's not what I'm saying. What I'm really saying is that I don't want to have to run 5 different setup.py commands every time I do a release in order to upload all the possible distribution formats that my users may want. Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) iQCVAwUBSYG+pnEjvBPtnXfVAQIc2wP/Q3Fks9V6KX7LtUBEw2S4FVcjIOmb2iII H3fnb8D5hft7BiGbRQ+jLakzVSff9cV9F1TIsFLepXI+buBAupJ7XmiKIyVcfDRT Y6KIQQfO150Tlw5K/L9XdnJjPuEmXji+f0LcY8DjI2SqYZkQZBxV7+XX35CdqbYK dQgCgAgtaRU= =zSqr -----END PGP SIGNATURE-----

Lennart Regebro wrote:

On Thu, Jan 29, 2009 at 15:35, Barry Warsaw <barry@python.org> wrote:

...

That's not what I'm saying. What I'm really saying is that I don't want to have to run 5 different setup.py commands every time I do a release in order to upload all the possible distribution formats that my users may want.

Well, you are gonna have to, because you also need to test that they work anyway. We can't distribute autogenerated distribution packages that nobody ever tested that they actually work.

Someone should, true. But it need not be the original author, who might not have the ability to test (or even produce) every desirable distribution format.

On Thu, Jan 29, 2009 at 09:55:29AM -0500, Eric Smith wrote:

Lennart Regebro wrote:

...

On Thu, Jan 29, 2009 at 15:35, Barry Warsaw <barry@python.org> wrote:

...

That's not what I'm saying. What I'm really saying is that I don't want to have to run 5 different setup.py commands every time I do a release in order to upload all the possible distribution formats that my users may want.

Well, you are gonna have to, because you also need to test that they work anyway. We can't distribute autogenerated distribution packages that nobody ever tested that they actually work.

Someone should, true. But it need not be the original author, who might not have the ability to test (or even produce) every desirable distribution format.

In my ideal world python package developers would only ever have to worry about uploading an sdist. It's up to whoever wants to consume these to re-pack them in the appropriate format if that's what they want. If I am correct the only reason people started to go away from this is because Windows users don't tend to have compilers installed. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Jan 29, 2009, at 9:51 AM, Lennart Regebro wrote:

On Thu, Jan 29, 2009 at 15:35, Barry Warsaw <barry@python.org> wrote:

...

That's not what I'm saying. What I'm really saying is that I don't want to have to run 5 different setup.py commands every time I do a release in order to upload all the possible distribution formats that my users may want.

Well, you are gonna have to, because you also need to test that they work anyway. We can't distribute autogenerated distribution packages that nobody ever tested that they actually work.

That's what users, continuous integration, and and bug reports are for! Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) iQCVAwUBSYHP8nEjvBPtnXfVAQJKbwP9G6f2QjKvf7lv0frBuNTJPzryOG6Riu/8 g3Kh/NZ1fkrNerS8Qvzcn7Az9pbwWGkIYjgBTXOjgd0WHlyTsEUG7vMqnTNB3jM4 m3UUH+WbndZKaaQHim9g1xikKKk+2b90atrQ6Ss5mVOYRLHuH6ftILetsfMz1IUL NG5HUGtRz2Y= =DrTe -----END PGP SIGNATURE-----

On Thu, 29 Jan 2009 09:35:18 -0500, Barry Warsaw <barry@python.org> wrote:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Jan 29, 2009, at 8:57 AM, Josselin Mouette wrote:

...

Le mercredi 28 janvier 2009 à 08:07 -0500, Barry Warsaw a écrit :

...

I'd like to make a radical suggestion: upstream authors should never have to worry about building distribution blobs.

This is just silly. You don’t have to worry about the distribution’s internals, and what is specific to each package format, but the very idea of developing without any kind of knowledge about how the software will integrate on a system is a guaranteed recipe for a development disaster.

That's not what I'm saying. What I'm really saying is that I don't want to have to run 5 different setup.py commands every time I do a release in order to upload all the possible distribution formats that my users may want.

Barry, just have your buildbot do it for you - and not just when you do a release, but every time you commit. As a bonus, it can then install the packages and run your test suite and tell you if you've got installation problems (or any other bugs) on any of the platforms. :) Jean-Paul

Barry Warsaw wrote:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Jan 29, 2009, at 10:03 AM, Jean-Paul Calderone wrote:

...

Barry, just have your buildbot do it for you - and not just when you do a release, but every time you commit. As a bonus, it can then install the packages and run your test suite and tell you if you've got installation problems (or any other bugs) on any of the platforms. :)

I feel like I need to be snake bitten. :)

http://www.snakebite.org/ and http://mail.python.org/pipermail/python-committers/2009-January/000331.html for those who didn't get it ;-) Seriously, I think that's the way forward. Raphael

Barry

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSYHKGHEjvBPtnXfVAQJB+gQAllkJ7g+OzVu6PpXLsyuuHNyOkEhSmcb9 7+aHtoDuW1Brj6u+aiUrZOt8K5tRkTWTmkyOqJMrGa1/JnssU25qsGb8SODUbfO3 xZ9jcO9P6VKx/kRDBLUt7kNebR5aYUS7Z5MhjdM/at66CWhcVHaTlMuIkHyvZwuG 8/hP301QAE0= =Dx6o -----END PGP SIGNATURE----- _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Jan 29, 2009, at 12:17 PM, Tres Seaver wrote:

Barry Warsaw wrote:

...

On Jan 29, 2009, at 8:57 AM, Josselin Mouette wrote:

...

Le mercredi 28 janvier 2009 à 08:07 -0500, Barry Warsaw a écrit :

...

I'd like to make a radical suggestion: upstream authors should never have to worry about building distribution blobs. This is just silly. You dont have to worry about the distributions internals, and what is specific to each package format, but the very idea of developing without any kind of knowledge about how the software will integrate on a system is a guaranteed recipe for a development disaster.

That's not what I'm saying. What I'm really saying is that I don't want to have to run 5 different setup.py commands every time I do a release in order to upload all the possible distribution formats that my users may want.

I would argue that the upstream developer should (almost) never be uploading anything but a metadata-rich sdist: except for packages with C extensions, nobody really needs anything else for "library" packages, and it's really only the Windows folks who can't build those binaries for themselves.

People distributing applications might want to provide installers, I guess, for the command-line challenged.

Otherwise, if we provided enough metadata in the sdist, packagers can build the other formats (.deb, .rpm, etc.) for us, assuming that we can solve the "resource file" problem.

+1-enthusiastical-ly y'rs, Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) iQCVAwUBSYHlVnEjvBPtnXfVAQL18QP/TtIRsN2f93yAs1hNBl/Rb31q1Dpe6NnT R6vIZU9yGw0XyFaEEb9a/f8TC/EUKUDoaq2/g+RnCoZCABWBo3tiFrbmUxZs5C9n 1sbP1dFYSVGkOThtPz6gBa5hqYMwiA/l/UuCJAXVJdeRdVmiWMMJDRBftHIbMITB Mp53xK/NEmQ= =zeh5 -----END PGP SIGNATURE-----

Barry Warsaw wrote:

I'd like to make a radical suggestion: upstream authors should never have to worry about building distribution blobs.

In my ideal world, I would make a release by tagging my release in whatever vcs I'm using, then I would tell cheeseshop, "hey I just released version 3.1 and it is here" where "here" means whatever native vcs syntax points to the revision I just released. Then PyPI would do the "coagulation" into distribution formats and distribute it amongst its worldwide mirrors, all automatically.

I as the Python developer don't want to know about eggs, tarballs, debs, rpms, whatevers, I just want to write some software. I'm happy to add a bit of metadata to my setup.py to play ball, but otherwise I really just want one command to release my code and then magically have it appear available to everybody.

If that's not feasible <wink> then the next best thing would be to just spin the source tarball and upload that. Tarballs I can handle. <wink>

On the other end, when I zc.buildout, or paver, or easy_install, or aptitude install, or emerge, or port install, or whatever, it would go out to the Worldwide Python Distribution Network and pull down the proper blobby thing to install based on what I'm trying to do, e.g. an egg if I'm developing, a deb if I'm installing into my system Python, etc. Again, I don't really care and shouldn't have to know.

+sys.maxint to all of the above. ...and I'm damned sure the same is true for anyone using a package. Of course, here is where the distributions (debian, I'm looking at you!) start causing problems by only having ancient versions of packages, arbitarily stripping out dev headers and (in the case of xlwt) even removing actual source code from the package because they so chose to! ...and they then wonder why people who write python apps and libraries tell people not to bother using debian packages and the like? ;-) </rant> Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk

Hello Zooko On Wed, Jan 28, 2009 at 07:44:04AM -0700, zooko wrote:

On Jan 28, 2009, at 3:45 AM, David Cournapeau wrote:

...

...

I don't understand what are the potential problems, but so far I've been happy using stdeb to produce .deb's from my Python sdists.

This is not the right solution for distributions maintainers: it is a good tool for individual (it gives you uninstallation, etec...) but .deb packages produced by stddeb are not debian-compatible, and cannot be included in debian proper. This is not a critic of stddeb, I think it is a very good tool and useful tool.

I've heard things like this from Debian developers before, and I don't understand. Please provide me with more explanation.

The answer (or the part that comes to my mind at this moment at least) is: Policy, Policy and Policy. Debian is so good in no small part because each package has to follow the Policy (and all applicable sub-policies). * I don't trust a random python developer who might never even have heard of Debian to pick the correct name of their python project that will ensure the resulting package name will be correct and not conflict. * I know for a fact that distutils capabilities are not good enough to fully automatically follow the FHS (discussed many a times on this list already). * setup.py allows for arbitrary code to be executed, if you don't know the proper distutils way to do something you can make it do it in many different ways. These will break assumptions automated tools will make. * <more variations on the same themes here>

I don't intend to put words in your mouth, but I will offer a few guesses as to why you say stddeb can't be used for Debian proper:

This is pretty much flamebait but I'll answer anyway, again it boils down to the above points though.

1. You want the production of .deb's from Python packages to be done by a human instead of automatedly, therefore stdeb can't do it.

Yes, currently distutils is just not good enough. Maybe if the tools (including distutils) improve this could be wittled down to just checking setup.py (or equivalent) declares everything properly before letting the automated tools loose.

2. You want the production of .deb's from Python packages to be done by a Debian developer instead of by the upstream developer of the Python package.

Yes, someone who cares about the Debian Policy.

3. It would be okay for this process to be automated (or semi- automated), but there's some flaw in the design of stdeb which means it will never be able to do it right unless stdeb is rewritten with a new design.

I have no idea about stdeb as I've never used it, but distutils just doesn't provide enough information as I mentioned earlier. So yes.

4. It would be okay, and the general design of stdeb is okay, but there are some bugs in stdeb which currently cause it to produce the wrong results.

Ugh, really repeating things by now. But for a change I'll go for "no" as this is simply not applicable. There are probably many more reasons that I'm not thinking off right now. Most of those can probably be found sprayed around in various long threads on this list. IIRC Josselin Mouette did once summarise Debian's issues quite decently on this list. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org

Forgive me if I am new... but I totally agree.... On Thu, 29 Jan 2009 11:00:40 +0900, David Cournapeau <david@ar.media.kyoto-u.ac.jp> wrote:

zooko wrote:

...

On Jan 28, 2009, at 3:45 AM, David Cournapeau wrote:

...

...

I don't understand what are the potential problems, but so far I've been happy using stdeb to produce .deb's from my Python sdists.

This is not the right solution for distributions maintainers: it is a good tool for individual (it gives you uninstallation, etec...) but .deb packages produced by stddeb are not debian-compatible, and cannot be included in debian proper. This is not a critic of stddeb, I think it is a very good tool and useful tool.

I've heard things like this from Debian developers before, and I don't understand. Please provide me with more explanation. I don't intend to put words in your mouth, but I will offer a few guesses as to why you say stddeb can't be used for Debian proper:

1. You want the production of .deb's from Python packages to be done by a human instead of automatedly, therefore stdeb can't do it.

I don't *want* human production (I think some Debian developers want to - but that's not something that we need to care about I think). But for non trivial packages, human intervention will always be needed: packages which use autoconf cannot be automated either, because you may need post/pre install scripts, you need to split doc/non doc parts, devel and non devel parts, debug/release, etc... All of this is seen as a good thing and some even required by Debian policy.

...

2. You want the production of .deb's from Python packages to be done by a Debian developer instead of by the upstream developer of the Python package.

That's mandatory, indeed. An *official* debian package can only be done by a debian developer, almost by definition - only official debian developers can upload .deb to official debian repositories. This has no consequences for the python developer, though.

...

3. It would be okay for this process to be automated (or semi-automated), but there's some flaw in the design of stdeb which means it will never be able to do it right unless stdeb is rewritten with a new design.

There are some fundamental issues in *distutils* which make it impossible to do it correctly at the moment, mainly the lack of metadata about the installed files, but it looks like this point is already understood and agreed on. I will work on the requirements,

cheers,

David _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

Thanks for the information, Josselin: On Jan 29, 2009, at 7:10 AM, Josselin Mouette wrote:

Le mercredi 28 janvier 2009 à 07:44 -0700, zooko a écrit :

...

3. It would be okay for this process to be automated (or semi- automated), but there's some flaw in the design of stdeb which means it will never be able to do it right unless stdeb is rewritten with a new design.

This is the one. BTW, I don’t consider this a flaw in stdeb, it’s just that stdeb was not designed with the goal to produce packages suitable for Debian itself.

What we need is the equivalent of dh_make_perl [0]. That is, a script that will generate the debian/ structure in a semi-automated fashion, leading to a package ready to be installed after minor tweaks and a human’s review. Bonus points would go for providing a script suggesting changes in the description and/or dependencies when updating the package for a new upstream release.

A-ha! I think I understand the disagreement now! It hinges on the subtle distinction between "manual", "automated", and "semi-automated". Unless I'm misunderstanding something (which is quite possible), stdeb already does exactly what you just suggested. It produces a debian/ subdirectory and a .dsc file, which you can them feed into "dpkg-buildpackage" to produce a .deb. You can, of course, inspect and modify those files after stdeb produced then and before dpkg- buildpackage consumes them. http://github.com/astraw/stdeb/tree/master Perhaps there is some confusion on this point because I like to *talk* about stdeb as though it sucks in Python source trees and spits out .deb's. That's how I like to use it -- I try not to look at or change the .dsc or debian/ files. However, there's no fundamental reason that I am aware of that it couldn't be used by a real Debian developer to ease his task of producing completely Policy- Compliant, high-quality Debian packages. Apparently the Perl and Haskell Debian developers have already started using semi-automation this way to maintain large numbers of Policy-Compliant Perl and Haskell .deb's produced from Perl and Haskell source trees. If you, or anyone, tries this, please post to this list and also Cc: me (the list is just too high-volume for me to keep up :-() and I'll try to help. Regards, Zooko --- Tahoe, the Least-Authority Filesystem -- http://allmydata.org store your data: $10/month -- http://allmydata.com/?tracking=zsig

Akira Kitada wrote:

In my opinion, bdist_rpm and the like are "nice hacks" at best and nothing more. Peoplo who love rigorous distribution or control freaks would probably prefer to bother packaging themselves and that will leads them to use apt, yum...

I am both a control freak and love rigorous distribution. I use bdist_rpm all the time. There's no reason it can't be as good as creating a .spec file by hand, as long as setup.py has all of the required metadata. And I think that's the point here, to identify all of the metadata.

On Wed, Jan 28, 2009 at 7:00 AM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:

...

On Tue, Jan 27, 2009 at 5:29 PM, Akira Kitada <akitada@gmail.com> wrote:

...

Hello Tarek,

I think "apt, yum, etc" would be also used for packaging/distributing apps.

There is already a command that let you create a rpm package (bdist_rpm) out of a python package,

There were also a bdist_deb project but it never made it to distutils, also for Debian there's a policy on how to work with python packages : http://www.debian.org/doc/packaging-manuals/python-policy/

Last, this mailing list had a lot of threads about the fact that there's no standard way in Python to work with resources that could be installed in the system, using a LSB-compliant approach.

So I don't have (I think no one does at this point) any clear view of what could be done in this area.

Tarek -- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig