Tarek Ziadé writes:

It doesn't make sense to have the list of the files in .svn or .hg files. for your package.

Again, why not? If I'm using a VCS for my source files, then the VCS is the Single Point of Truth for the inventory of source files. That doesn't mean the VCS metadata goes *into* the source package; it means that {distutils, setuptools} *queries* the VCS to get the inventory when building the source package.

It should be agnostic imho and explicitely defined in the package.

That duplicates state information in multiple locations, which means it's either going to get out of date, or is mechanically duplicated for little benefit. -- \ “Of course, everybody says they're for peace. Hitler was for | `\ peace. Everybody is for peace. The question is: what kind of | _o__) peace?” —Noam Chomsky, 1984-05-14 | Ben Finney

David Cournapeau writes:

Ben Finney wrote:

...

Tarek Ziadé writes:

...

It doesn't make sense to have the list of the files in .svn or .hg files. for your package.

Again, why not? If I'm using a VCS for my source files, then the VCS is the Single Point of Truth for the inventory of source files.

depending on your definition of sources, it is not. The VCS is there to track a project, but sdist is a *distribution* mean. It is a packaged version of your software - simple, but packaged nonetheless. There has to be a (potential) difference between what goes in sdist and what is recorded in the VCS. Python packaging is the only tool I am aware which say both are equal.

Okay, but that's a far cry from saying that it “doesn't make sense” to use the VCS inventory.

The duplication argument could be made the other way: why duplicating with sdist what the VCS offers you ?

Yes, that's exactly my point. The sdist needs to be generated and distributed, since that's the common interface for recipients to get at the software for installation. My point is that that process of generating the sdist should re-use available information, e.g. from the existing VCS information which the developer has access to, it should be used from that source. -- \ “Pinky, are you pondering what I'm pondering?” “Um, I think so, | `\ Brainie, but why would anyone want to Pierce Brosnan?” —_Pinky | _o__) and The Brain_ | Ben Finney

David Cournapeau writes:

Ben Finney wrote:

...

Okay, but that's a far cry from saying that it “doesn't make sense” to use the VCS inventory.

Not that a far cry: python packaging tools are the only systems that I know of which does this. For example, in autotools, the tarball generated by make dist is never generated from the VCS.

I'm not advocating using the VCS to *generate the tarball*. I'm talking about using the VCS to *determine what files to put in the tarball*.

I may have some weird tools in my VCS, or some huge test data, things which do not make sense to distribute.

Okay. I was addressing the claim that it makes no sense (i.e. it's *always* wrong) to use the VCS inventory.

Maybe I was not clear, but my point cannot be further from your suggestion :) sdist should not use the VCS, because a source tarball generated from the VCS and from sdist are not the same thing at all.

Your leap from “use the VCS inventory” to “cause the VCS to generate the sdist tarball”, which is wrong. I'm saying sdist should (in some cases, i.e. it makes sense sometimes, i.e. it's not something that doesn't make sense) use the VCS inventory to know what files form the source distribution.

If you want a source checkout (which is what sdist does if you use the VCS), then use the VCS. Most good systems offer a service to automatically generate the source from the VCS if you need to make it available to people wo the VCS (trac, git and hg web frontends, etc...).

As you're no doubt aware, there's more to making an sdist than merely making a tarball of the files; if nothing else, the metadata needs to be assembled by {distutils, setuptools} and isn't stored directly in the VCS. So I'm really not following your point here. -- \ “I was trying to daydream, but my mind kept wandering.” —Steven | `\ Wright | _o__) | Ben Finney

At 03:02 PM 4/6/2009 +0900, David Cournapeau wrote:

Ben Finney wrote:

...

I'm not advocating using the VCS to *generate the tarball*. I'm talking about using the VCS to *determine what files to put in the tarball*.

Currently, using the vcs plugin does include everything as far as I know. So what do you have in mind ?

You can *exclude* files from the sdist, however, using MANIFEST.in.

Lennart Regebro writes:

On Mon, Apr 6, 2009 at 08:02, Ben Finney wrote:

...

I'm not advocating using the VCS to *generate the tarball*. I'm talking about using the VCS to *determine what files to put in the tarball*.

What's the difference?

The former is having the sdist process tell the VCS “generate a tarball export of the versioned files” which isn't much use since the result still isn't a complete sdist. The latter is having the sdist process query the VCS “tell me the set of version files by name” and then using that information in the rest of the normal sdist creation process. David Cournapeau writes:

Currently, using the vcs plugin does include everything as far as I know. So what do you have in mind ?

That, in the presence of a VCS and in the absence of a ‘MANIFEST.in’ file, the “create an sdist” procedure should query the VCS inventory for the information it would otherwise get from the ‘MANIFEST.in’. Thus making that file unnecessary in many cases that I care about. -- \ “A fine is a tax for doing wrong. A tax is a fine for doing | `\ well.” —anonymous | _o__) | Ben Finney

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Ben Finney wrote:

Lennart Regebro writes:

...

On Mon, Apr 6, 2009 at 08:02, Ben Finney wrote:

...

I'm not advocating using the VCS to *generate the tarball*. I'm talking about using the VCS to *determine what files to put in the tarball*. What's the difference?

The former is having the sdist process tell the VCS “generate a tarball export of the versioned files” which isn't much use since the result still isn't a complete sdist.

The latter is having the sdist process query the VCS “tell me the set of version files by name” and then using that information in the rest of the normal sdist creation process.

David Cournapeau writes:

...

Currently, using the vcs plugin does include everything as far as I know. So what do you have in mind ?

That, in the presence of a VCS and in the absence of a ‘MANIFEST.in’ file, the “create an sdist” procedure should query the VCS inventory for the information it would otherwise get from the ‘MANIFEST.in’. Thus making that file unnecessary in many cases that I care about.

In *any* case I know of: including non-VCS-controleld files in an sdist is an anti-usecase for me. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJ2k51+gerLs4ltQ4RAjdQAJ0cURt1pAcGVrZseKTNZ+2OUcVDWwCgzw8q n7Pxy/CMbRUatp8WQkbpTzk= =HwNM -----END PGP SIGNATURE-----

David Cournapeau writes:

Tres Seaver wrote:

...

In *any* case I know of: including non-VCS-controleld files in an sdist is an anti-usecase for me.

A simple example: including sphinx-generated documentation.

That's a good counter-point to Tres's point, true.

If including non-VCS controled stuff is an anti-case, then why having sdist at all ? Any half decent VCS can generate a tarball directly from the files it controls.

This continues to be irrelevant. I don't know why you keep repeating it. An sdist is *not* just a tarball of the source files. The important difference is that it contains metadata generated and assembled by distutils (or an equivalent) that is specific to Python's distutils system. Hence, the job of creating an sdist is *not* something for the VCS to do. -- \ “Many are stubborn in pursuit of the path they have chosen, few | `\ in pursuit of the goal.” —Friedrich Nietzsche | _o__) | Ben Finney

David Cournapeau writes:

Ben Finney wrote:

...

An sdist is *not* just a tarball of the source files.

It is if you consider any non-tracked file to be an anti-usecase,

No. In *no case* is an sdist equal to a tarball of the source files. Please stop blurring the distinction. -- \ “One of the most important things you learn from the internet | `\ is that there is no ‘them’ out there. It's just an awful lot of | _o__) ‘us’.” —Douglas Adams | Ben Finney

Lennart Regebro wrote:

Except for files generated by the sdist/build-process, yes. Why would you want to include files in a distribution that are neither controlled source files or a result of the sdist/build process?

Depends on what you mean by the build process. I sometimes need to alleviate distutils limitations with some scripts, or because of incompatible monkey patching between numpy.distutils, paver and setuptools.

Now, in which of these cases will the sdist created by me an d by Bobo be identical? Right. In the one where the file list is determined by which files are version controlled. Can we then really say that this case is more dependent on the environment?

It depends on how you set your wildcard. If you use *.$ext, seems to me that it would alleviate those cases, or do you mean something else ?

Yes, for me, the sdist will be different if I have a checkout or not, since I have the project file in the directory. But that's a rather unusual usecase, don't you think?

Well, no, I don't think so. You can't make assumptions about what I need, as I can't make assumption about what you need. But if we can agree on a "low level communication format", then you can use your tools to support your usecases, and I can use mine for my usecases, so that we can end up with something which works everywhere.

Why would I have a non-VCS directory of a mdule with a project file in it, that I distribute? If I modify a project, it should be checked in.

What if the file is generated ? For example, I have some .c files which are generated from a script from some templates. I want to include the generated file in the sdist so people don't need the scripts (and potential dependencies) to build the software. That's something which is well supported on autotools, for example.

Asking the VCS for which files are checked in i sthe most reliable way of figuring out which files are actually source files I can think of.

I am not denying that it works for you. I am just trying to argue that it does not work for me, as I have given several examples already (generated sources, built documentation, sources obtained from a different VCS than the original source). So instead of trying to impose each solution to each other, we should focus on the commonalities, cheers, David

Lennart Regebro wrote:

On Tue, Apr 7, 2009 at 07:20, David Cournapeau wrote:

...

It depends on how you set your wildcard. If you use *.$ext, seems to me that it would alleviate those cases, or do you mean something else ?

Sure, but now we are getting closer and closer to having to specify each file separately. Which is not a practical state of affairs. This is about which process to use to determine which files should be included when we don't specify things.

I think it is obvious that we can't agree on the process :)

I've never claimed it works for you. You are saying that this method should *not* be supported.

I am not saying that it should not be possible - but that it should be built on top of a lower system which we can all agree upon. Then, you can use your tool to depend on the VCS to feed this lower system, and I can reuse this lower system myself without depending on the VCS thing, and everybody is happy ? cheers, David

Lennart Regebro wrote:

"Lower system"? What would that be? This is about if we should support getting a file list from the VCS or not to determine what files should go in a sdist.

We can argue all day long - but since there is a disagreement, there is only two issues: either someone decides for one behavior or the other, or we acknowledge our difference and decide on less ambitious goals to build other tools above.

I have a hard time envisioning anything "lower" than that.

Why so ? Few packaging system use the VCS as the lower system to know what goes in a distribution, most of them (regardless of the platform: windows, unix, mac os x) uses something more low level than this at the implementation level. cheers, David

Lennart Regebro wrote:

Possibly. I can't say we are arguing though. You are just saying that you don't agree, but you still come with no arguments for your standpoint or against mine.

Ok. David

On Tue, Apr 7, 2009 at 3:02 PM, Floris Bruynooghe wrote:

...

Where in this is there a "lower level"?

Number 1 could be the simple lower level, a list of files (I'm not saying this is a good choice, I haven't studied all the issues good enough).  That way the tool you use to consume setup.py (or whatever it becomes) can construct this list from your VCS in an automated fashion if you do like that.

That's an egg-and-chicken issue. This is the MANIFEST final file, built by all the different mentioned techniques.

But it avoids having the VCS dependency inside distutils.

Maybe we could think about a plugin system to build this list. (a bit the way vcs plugins works in setuptools) Distutils would provide a default plugin and people will be able to register their own plugins. I'll write a small proposal for this. (I hope this won't create a new flame here) Regards Tarek

On Tue, Apr 07, 2009 at 02:41:07PM +0900, David Cournapeau wrote:

Lennart Regebro wrote:

...

On Tue, Apr 7, 2009 at 07:20, David Cournapeau wrote:

...

It depends on how you set your wildcard. If you use *.$ext, seems to me that it would alleviate those cases, or do you mean something else ?

Sure, but now we are getting closer and closer to having to specify each file separately. Which is not a practical state of affairs. This is about which process to use to determine which files should be included when we don't specify things.

I think it is obvious that we can't agree on the process :)

...

I've never claimed it works for you. You are saying that this method should *not* be supported.

I am not saying that it should not be possible - but that it should be built on top of a lower system which we can all agree upon. Then, you can use your tool to depend on the VCS to feed this lower system, and I can reuse this lower system myself without depending on the VCS thing, and everybody is happy ?

+1 Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org

On Tue, Apr 7, 2009 at 07:36, Robert Kern wrote:

Unfortunately, the setup.py file is not always capable of performing all of the build processes (at least, not easily). One might use a Makefile or other build tool to build the docs or other assets. So you need a manual way of telling the setup.py file what additional files to include.

OK, this is a good usecase for including non-VCS files. But then again, nobody ever said that this would *not* be possible. In this case you are likely to have several intermediate files that you are not interested in including, and the question is if you want to include the source files used for this external build tool at all. So would automatic globbing be better? Wouldn't you need to specify which files that should be included here? -- Lennart Regebro: Pythonista, Barista, Notsotrista. http://regebro.wordpress.com/ +33 661 58 14 64

On Tue, Apr 07, 2009 at 08:12:39PM +0300, Marius Gedminas wrote:

On Tue, Apr 07, 2009 at 07:20:11AM +0200, Lennart Regebro wrote:

...

OK, here is a usecase. I have a directoy with a module, foo.bar. I have also in that directory saved a project file from my IDE, because naturally I save a project file into the directory of every project I have. But since that's not a part of the module itself I never check it in.

My friend Bobo, however, does not use an IDE, and does not create a project file.

In case number one, I specify the files to be included with a wildcard.

In case number two, I do not specify the files to be included at all, but let sdist determine the files by looking at which files are version controlled.

Now, in which of these cases will the sdist created by me an d by Bobo be identical? Right. In the one where the file list is determined by which files are version controlled. Can we then really say that this case is more dependent on the environment?

Consider a different use case: my friend's friend Alice does not like svn. She uses git-svn to get a local git repository containing a copy of the subversion code. She then uses the usual 'python setup.py sdist' build process. It produces a broken distribution (silently) because setuptools doesn't support git, and Alice didn't know she had to install a plugin.

Users don't expect version control system metadata to have an influence on the build process (other than recording the revision number/ID somewhere in the built package).

I guess, what I'm saying boils down to: the failure should not be silent and unnoticed. If the package's maintainer wants to use $VCS metadata to define the contents of the sdist, that's fine, but (1) it should be specified explicitly, e.g. setup(use_vcs_for_manifest=True) and (2) if setuptools is unable to find VCS metadata for any of the VCSes it supports (including through plugins), it should abort with an error (suggesting the user to look for plugins for the appropriate VCS) The remaining question is: what do you do when the tree is included in multiple VCSes? (I've been known to place bzr trees into svn for backup purposes. It didn't work out too well.) Marius Gedminas -- Only great masters of style can succeed in being obtuse. -- Oscar Wilde Most UNIX programmers are great masters of style. -- The Unnamed Usenetter

On Tue, Apr 07, 2009 at 07:27:29PM +0200, Lennart Regebro wrote:

On Tue, Apr 7, 2009 at 19:12, Marius Gedminas wrote:

...

Consider a different use case: my friend's friend Alice does not like svn.  She uses git-svn to get a local git repository containing a copy of the subversion code.  She then uses the usual 'python setup.py sdist' build process.  It produces a broken distribution (silently) because setuptools doesn't support git, and Alice didn't know she had to install a plugin.

But how would it break? It would in that case possibly include *more* files, as it instead would default to including everything, and not only VCS files, so you might get intermediary build files included as well, in worst case. Or Alice's project file, or similar.

Are we talking about the current design (which doesn't include everything), or about some proposed change? I kind of lost track. Some redundant background (already seen upthread): My personal meeting with this issue was when the application failed to work correctly because it was missing some Javascript files due to the fact that the released version was built from a tarball produced with svn export. And no, that tarball could not have been built with setup.py sdist instead of svn export---it contained more than one package and some of those weren't setuptoolized yet (and some of those weren't really Python packages). I'm repeating this to indicate that in my experience, svn-metadata-less builds with no explicit MANIFEST.in do not in fact include everything by default.

But the distribution would not be exactly the same, this is true. Is that a problem?

Assuming the distribution is complete and has extra files: no, it wouldn't necessarily be a problem. Including extra cruft is inelegant, but not broken (unless that extra cruft breaks builds somewhere else, because the files in question are arch-specific and do not get rebuilt---a hypothetical use case probably not applicable to Python packages). Marius Gedminas -- Favorite MAC error message: "Not enough memory to eject disk!"

On Apr 7, 2009, at 11:12 AM, Marius Gedminas wrote:

Consider a different use case: my friend's friend Alice does not like svn. She uses git-svn to get a local git repository containing a copy of the subversion code. She then uses the usual 'python setup.py sdist' build process. It produces a broken distribution (silently) because setuptools doesn't support git, and Alice didn't know she had to install a plugin.

For the Tahoe project, users can build from sdists (because they come with SOURCES.txt file listing all files) or from the official revision control tool (which is darcs). Getting a copy of the files which didn't come from the official revision control tool and didn't come from an sdist is not supported, and as far as I know no-one has ever done it. Nonetheless I worry about those sorts of rare, quiet failures so with PJE's help I added a clause in setuptools_darcs which will emit a warning message if there is no SOURCES.txt and also it fails to invoke the darcs executable: http://allmydata.org/trac/setuptools_darcs/browser/setuptools_darcs/ setuptools_darcs.py?rev=54#L28 Here is the documentation about this issue in the setuptools_darcs plugin (I copied it from the setuptools-git plugin): http://allmydata.org/trac/setuptools_darcs/browser/README.txt? rev=63#L103 Here is the documentation about this issue in the tahoe setup.py: http://allmydata.org/trac/tahoe/browser/setup.py?rev=3669#L122 I have not heard any complaints about this particular detail. I have heard plenty of complaint about lots of other setuptools issues from lots of people. The Big Two are still: http://bugs.python.org/setuptools/issue53 # respect the PYTHONPATH and http://bugs.python.org/setuptools/issue54 # be more like distutils with regard to --prefix= Regards, Zooko

Tres Seaver wrote:

David Cournapeau wrote:

...

Ben Finney wrote:

...

An sdist is *not* just a tarball of the source files. It is if you consider any non-tracked file to be an anti-usecase, which is what I was answering to :)

There are three classes of files here:

- "source" files, which should be under version control, and included (by default) in the sdist.

- "derived" files, which should *not* be under version control, but which might (in certain circumstances) be included in the sdist. E.g., the C file generated from a Pyrex / Cython source file, to enable builds on systems without Pyrex / Cython installed.

- "stray" files, not under version control. These shuold *never* be in an sdist.

Why those assumptions on what people can do. That's exactly what's wrong with distutils for me, all those untold assumptions, which are 'obviously' correct. As I said previously, including built documentation is something I use MANIFEST.in for, and should be supported, or at least possible. Making things easier for some people is fine - as long as it does not prevent other usage. Anything which does not solve the fact that sdist command gives a different tarball depending on whether it is generated by distutils, setuptools, paver and co is useless to me. Maybe it is not important for distutils to solve this, but for me, that's much more important than support from VCS or any convenience feature which can be solved by a simple script anyway.

...

Moving away from "using VCS is good" vs "using VCS is not good", maybe we should focus on the workflows to support. The most significant to me is reproducibility (python setup.py sdist output should depend as little as possible on its environment - whether the sources are under VCS or not, etc....). At least one person said this is not a requirement. Can we reach a consensus on this point, and then focus on the technical solutions ?

I don't understand the goal of "not depending on the environment" (it will depend on the version of Python / distutils / setuptools / Cython / Pyrex you use, for instance, just as C-based packages depend on the version of autogen and any local autoconf macros).

Environment was a poorly chosen word. Of course, if I use a different cython, etc... version, the files are different. But on a same environment (same tools, compilers, etc...), having a different tarball depending on the sources are under a VCS or another or not is a fundamental misfeature, at least for me. Whether the sources are under cvs or not, make dist works the same for any autotools package. Would people still think it is expected if bdist_msi or bdist_wininst targets where different if built under VCS or not ? cheers, Davi

On Wed, Apr 8, 2009 at 04:58, David Cournapeau wrote:

Why those assumptions on what people can do. That's exactly what's wrong with distutils for me, all those untold assumptions, which are 'obviously' correct. As I said previously, including built documentation is something I use MANIFEST.in for, and should be supported, or at least possible.

That's covered by the "derived" case.

Making things easier for some people is fine - as long as it does not prevent other usage.

Nobody has anywhere in this discussion suggested anything that in any shape, form or way prevents any sort of other usage. You still aren't listening.

Anything which does not solve the fact that sdist command gives a different tarball depending on whether it is generated by distutils, setuptools, paver and co is useless to me.

You agree that distutils have a problem in this area. Setuptools overrides the behaviour so that it is less broken, to fix your usecase and include files you want included, but which distutils would not include. And then you complain that setuptools doesn't create the exact same sdist as distutils. Yeah, that makes sense. Not.

Anything which does not solve the fact

And exactly which anything are you referring to here? Again, you are not being specific, but you come with implicit general accusations against everything and everyone. I am not aware on any suggestion done in this discussion which does *not* solve this problem.

Environment was a poorly chosen word. Of course, if I use a different cython, etc... version, the files are different. But on a same environment (same tools, compilers, etc...), having a different tarball depending on the sources are under a VCS or another or not is a fundamental misfeature, at least for me.

Well, fine. So specify what files you want included then. Problem solved. You still aren't listening, and as a result, you are wasting peoples time by arguing against straw men. -- Lennart Regebro: Pythonista, Barista, Notsotrista. http://regebro.wordpress.com/ +33 661 58 14 64

On Wed, Apr 8, 2009 at 06:04, David Cournapeau wrote:

It does if the underlying implementation is not shared correctly between all the tools.

Since we are discussing what the underlying implementation to be shared by all tools should be, that's a rather pointless comment.

Now, how can I use the distribution files produced by setuptools/paver in another tool. If there is no way to share this data reliably, that's a misfeature. That's why a common, underlying low level thing which can be merged reliably by different tools is needed.

Your are fighting windmills. -- Lennart Regebro: Pythonista, Barista, Notsotrista. http://regebro.wordpress.com/ +33 661 58 14 64

David Cournapeau writes:

Ok. Looks like you feel insulted for some reasons to keep insulting me back. Since I can't make my point clearly, and nobody in this discussion seems to care anyway, I will leave the discussion.

I hope you can reconsider (but certainly a little time away from the discussion would be good to help clear the mood). From what I've seen in this discussion, it's not that nobody else cares, but rather that we all care but aren't communicating too well. -- \ “I went to the hardware store and bought some used paint. It | `\ was in the shape of a house.” —Steven Wright | _o__) | Ben Finney

2009/4/8 Paul Moore :

As a bystander, what *I* care most about is that Python ends up with a single approach to creating and distributing extensions. What concerns me most about this whole discussion, is that it seems like no-one is attempting to compromise, and participants are pulling further apart rather than converging on a solution.

Two solutions is significantly worse than none, in my book.

BTW, I share David's pain here - I have spent some time in the past explaining *my* requirements, and have found that I repeatedly come up against either blank "I don't understand why you want that" responses, or suggestions that I change how I work to match what's proposed. Nobody seems to be attempting to collect requirements here. Paul. PS No, I won't state my requirements again. That'll only restart the flames. Maybe later, if someone starts a genuine, unbiased, attempt to collect requirements...

Meta discussions are generally a complete waste of time. This is the only thing I intend to say on the issue. On Wed, Apr 8, 2009 at 11:44, Paul Moore wrote:

As a bystander, what *I* care most about is that Python ends up with a single approach to creating and distributing extensions. What concerns me most about this whole discussion, is that it seems like no-one is attempting to compromise, and participants are pulling further apart rather than converging on a solution.

It's bad if it seems like that to you, because I would say that this is completely incorrect. There is no pulling apart, because there are no standpoints that can pull apart and there are no diverging proposals. There has been two related proposals afaik. Tareks talk about plugins, and mine where I said I would like the default behavior to be: 1. The file list is gotten from a specification in setup.py. 2. If no such specification, the file list is gotten from the VCS. 3. If no VCS, include all files except well known temporary file extensions. This is completely compatible with Tareks proposal for plugins, as this simply could be the three default plugins, if no other plugins are specified. And there has not come up any real arguments against either mine or Tareks proposals. So how could this be "pulling apart"?

PS No, I won't state my requirements again. That'll only restart the flames. Maybe later, if someone starts a genuine, unbiased, attempt to collect requirements...

Tarek has collected requirements and are still doing so, and everybody is completely open for new requirements, or explanations of why proposed solutions doesn't work. Regards -- Lennart Regebro: Pythonista, Barista, Notsotrista. http://regebro.wordpress.com/ +33 661 58 14 64

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Lennart Regebro wrote:

There has been two related proposals afaik. Tareks talk about plugins, and mine where I said I would like the default behavior to be:

1. The file list is gotten from a specification in setup.py. 2. If no such specification, the file list is gotten from the VCS.

2a. There should be a way to specify deltas from the list generated via the VCS (to include particular derived files, or exclude some sources).

3. If no VCS, include all files except well known temporary file extensions.

I would never want #3, myself, but I have no problem with it being available as an option for the VCS-averse. It should probably use the same "deltas" model as in #2a. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJ3JV5+gerLs4ltQ4RAn42AKC2TbBtwPNeYKKLJO2UZiXk4vUCuACgtHho sMemhM5bb2E8kjT9ELg44rw= =Pq4q -----END PGP SIGNATURE-----

2009/4/6 Lennart Regebro :

On Mon, Apr 6, 2009 at 08:02, Ben Finney wrote:

...

I'm not advocating using the VCS to *generate the tarball*. I'm talking about using the VCS to *determine what files to put in the tarball*.

What's the difference?

Personally, I agree that if there is a VCS, and no other information on what files to include, using the VCS file info makes sense to me.

I don't understand a large part of this discussion, but it seems to me that the expected and reasonable way of determening what files to include would be to:

1. Specify files in setup.py 2. Use the files that is a part of the VCS.

What if you don't want to include some files that are present in the VCS ? This is an implicit behavior. Tarek

Lennart Regebro wrote:

It's a practical default.

It is practical on the short term, but a pain in the long term. The single most problematic aspect of distutils (and the tools above it) is all this magic which works in some simple cases, and breaks in other, without any way to circumvent it. A distribution tool has to be simple and customizable, not magic. For example, all those different, slightly different behaviors of what goes where means that depending on whether you use distutils, setup.py sdist, paver sdist, setuptools setup.py sdist, your tarball is different. That's just insane, and the only way to make it work is to have *one simple way* to tell what goes where. Distutils should be simplified, not made even more complicated. Tarball generation is typically an area where there is too much magic. If there is one simple way to make sdist, then you can build your own tools to integrate with your DVCS if you want. But the data produced and fed by/from tools have to be simple. Put the magic in the tools, not in the data :) cheers, David

Lennart Regebro wrote:

Yes. And can you point out what in my suggestion is not simple, and what is magic, and what is not circumventable?

- Not simple: you need a plugin for each VCS. To be consistent, it needs to be included in distutils, so when the VCS format changes, it breaks - you can't update it easily because it is in the stdlib. And if the sources are not under a VCS, it breaks, again. - Magic: it depends on the VCS. So if the same sources is under a different VCS, it may be different. How do you handle ignored files ? Etc... I have experienced all those problems for almost any package I am distributing, be it big (numpy/scipy) or small (my owns). Now, if on the contrary, you have a simple format, then you can always have your own tool to generate this format from the DVCS. Why should python be different from every other type of software ? cheers, David

Lennart Regebro wrote:

Sure, if the plugin can't use a stable API then it should not be included in the stdlib.

I was not talking about the plugin stability, but about the situation where the VCS detection is broken. As it has happened with setuptools very recently.

How does it break?

If I have say the git plugin, and I run sdist in a git repo, but later someone run sdist from my tarball: how can I be sure it is the same ?

How is this different from any non-VCS situation?

It is totally different: in the simple situation, I say which file to include/not to include. So it is reproducible, whatever the situation is. I won't have some ignored files because my own import does not have the same ignored files as the original package.

Can you explain what is different and when?

No other distribution mechanism that I know of uses magic to detect which files to distribute. Neither debian tools, nor rpm tools, nor autotools. Not being explicit about files for distribution is a terrible idea; a packaging tool should be explicit, to avoid as much as possible to do something unexpected. cheers, David

rpm(build) is a build tool, doens't deal with source distribution (but it generates srpms from the build sources). There's a mild autodetection in rpmbuild when used in a restricted environment: it detects installed but unpackaged files under $RPM_BUILD_ROOT. It fails apart when used to build software under root. I still think that a flowchart wuold be beneficial in identify the usage scenarios better. Regards, Antonio On Mon, Apr 6, 2009 at 11:18 AM, Jeff Rush wrote:

David Cournapeau wrote:

...

No other distribution mechanism that I know of uses magic to detect which files to distribute. Neither debian tools, nor rpm tools, nor autotools. Not being explicit about files for distribution is a terrible idea; a packaging tool should be explicit, to avoid as much as possible to do something unexpected.

While true that the rpm tools do not magically detect which files to distribute, there are many that wish it did and have tried various approaches to adding it.  The following except from the RPM book speaks of the various, failed approaches.  Still it speaks of a essential need on the part of developers to automate this aspect of packaging, even if the perfect solution has not yet been found.

If we're going to be explicit on what to include, I would argue we should NOT allow wildcards, since in subsequent uses of sdist by another person, it could cause files to be included that otherwise would not be there.  The only way to be sure precisely the same set of files is always included is to explicitly list them one by one.  A big pain.

--- cut here --- excerpt from Maximum RPM book

How Do You Create the File List?

Since RPM automates so many aspects of software installation, it's easy to fall into the trap of assuming that RPM does everything for you. Not so! One task that is still a manual process is creating the file list. While it may seem at first glance, that it could be automated somehow, it's actually a more difficult problem than it seems.

Since the majority of an application's files are installed by its makefile, RPM has no control over that part of the build process, and therefore, cannot automatically determine which files should be part of the package. Some people have attempted to use a modified version of install that logs the name of every file it installs. But not every makefile uses install, or if it does, uses it sporadically.

Another approach tried was to obtain a list of every file on the build system, immediately before and after a build, and use the differences as the file list. While this approach will certainly find every file that the application installed, it can also pick up extraneous files, such as system logs, files in /tmp, and the like. The only way to begin to make this approach workable would be to do nothing else on the build system, which is highly inconvenient. This approach also precludes building more than one package on the system at any given time.

At present, the best way to create the file list is to read the makefile to see what files it installs, verify this against the files installed on the build system, and create the list.

_______________________________________________ Distutils-SIG maillist  -  Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

Lennart Regebro wrote:

Well, because the tarball will include the files in the VCS. But sure, if you in the tarball case add files in the directory that are not in the VCS, that would be included too. Maybe that's a problem, I don't know. I can't see the problem myself, but maybe somebody else would have a case for that.

My use case is very simple, and yet very common. If you have your sources in a VCS system, say svn: python setup.py sdist # put everything under svn into the tarball cd dist && uncompress tarball && python setup.py sdist # the tarball is not the same That's a concrete example of what I mean by magic. The distributed files depends on how where you build it. It means that someone who get this tarball, modify it, and regenerate it cannot do it correctly. The whole point of packaging tools is reproducibility. Using the VCS as is done currently breaks this. cheers, David

On Apr 6, 2009, at 8:31 AM, Lennart Regebro wrote:

2009/4/6 David Cournapeau :

...

python setup.py sdist # put everything under svn into the tarball cd dist && uncompress tarball && python setup.py sdist # the tarball is not the same

Why not?

It will be missing the .hg,.svn or whatever it used to generate the file list. I think the non magic way would be to have a tool to extract a file listing from (d)vcs and using that to generate a file, or maybe the api be made so that you have to explicitly put "file_list = distutils.generate_files_hg()" somewhere, then it is not magic anymore, and changing is just creating a list or a function that returns a list. -- Leonardo Santagada santagada at gmail.com

At 11:17 AM 4/6/2009 -0300, Leonardo Santagada wrote:

On Apr 6, 2009, at 8:31 AM, Lennart Regebro wrote:

...

2009/4/6 David Cournapeau :

...

python setup.py sdist # put everything under svn into the tarball cd dist && uncompress tarball && python setup.py sdist # the tarball is not the same

Why not?

It will be missing the .hg,.svn or whatever it used to generate the file list.

But it *will* have a SOURCES.txt, that lists all those files in it.

P.J. Eby wrote:

At 08:08 PM 4/6/2009 +0900, David Cournapeau wrote:

...

My use case is very simple, and yet very common. If you have your sources in a VCS system, say svn:

python setup.py sdist # put everything under svn into the tarball cd dist && uncompress tarball && python setup.py sdist # the tarball is not the same

Does this actually fail under setuptools? It should not, as RPM building from an sdist would then fail, and there was code specifically added to ensure that use case works.

Have you actually tested this, or are you simply theorizing?

I had some recent failure with paver + numpy.distutils + setuptools combination as late as last week. I am not saying it is setuptools's fault (it may well be a problem in numpy), but I can't know for sure because there are many different ways to generate this list of sources. The problem is not doing it with VCS as much as there are many ways in different conditions that only a very few people understand as of today. As a numpy developer who messes quite a bit with distutils and numpy.distutils extensions, I hope we can get rid of all this magic as much as possible. It is ok for *tools* to be magic, as long as they write their *data* to a commonly understood and specified format. But for now, setuptools enforce some behavior that other tools can't mimic, because setuptools' way is implementation defined. And for other parts of packaging, maybe numpy.distutils is implementation defined, without the possibility for other tools to mimic this either. That's why IMHO distutils should be as straightforward as possible, so you can build up things on top of it. Some people like setuptools' behavior w.r.t VCS; but it should be possible for my package and for my potential distutils extensions to stay compatible with distutils, whether I am using setuptools or not. Otherwise, when there are N tools which are all implementation defined for packaging, it becomes intractable very quickly. cheers, David

P.J. Eby wrote:

That means creating an installation manifest format that is 100% explicit, and then having tools (including distutils/setuptools commands, and plugins for other build systems) that can generate that manifest, as well as tools to install files according to different platforms' predilections.

I am 100 % in agreement with the above (I have been thinking a bit about something a bit more powerful that MANIFEST and PKG-INFO for the static-metadata, as a path for a 'next'-gen distutils, but I don't have anything working yet), cheers, David

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 David Cournapeau wrote:

Lennart Regebro wrote:

...

Well, because the tarball will include the files in the VCS. But sure, if you in the tarball case add files in the directory that are not in the VCS, that would be included too. Maybe that's a problem, I don't know. I can't see the problem myself, but maybe somebody else would have a case for that.

My use case is very simple, and yet very common. If you have your sources in a VCS system, say svn:

python setup.py sdist # put everything under svn into the tarball cd dist && uncompress tarball && python setup.py sdist # the tarball is not the same

That is an "iced tea spoon"[1]: there is no guarantee that running sdist will (or even should) work the same way. Note that under setuptools, the actual files included in the original 'sdist' are generated into the 'sources.txt' file in the EGG-INFO directory, It is often true for *lots* of release management strategies that the release managers have to do extra work to create a release tarball (e.g., re-run autogen, etc., or flex / bison, etc.), and that the released tarball does not include enough information to rebuild itself (as opposed to re-tgz'ing it).

That's a concrete example of what I mean by magic. The distributed files depends on how where you build it. It means that someone who get this tarball, modify it, and regenerate it cannot do it correctly.

The whole point of packaging tools is reproducibility. Using the VCS as is done currently breaks this.

Nobody but a package maintainer should be making sdists. If you are making private sdists of a package maintained elsewhere, they you should be prepared to do extra work, like checking the sources into a VCS, or hacking the packaging data yourself. [1] "Doctor! Doctor! whenever I drink iced tea, I get a cold stabbing pain in my eye!" "Take out the spoon!" Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJ2lFk+gerLs4ltQ4RAt3pAJ96zikIssJkAtifbZUNf40ZFVvryQCfVl9d 0+1vwn8q91zzTBRyPeBcViY= =ydao -----END PGP SIGNATURE-----

On Mon, Apr 6, 2009 at 9:30 AM, Lennart Regebro wrote:

On Mon, Apr 6, 2009 at 09:23, Tarek Ziadé wrote:

...

...

I don't understand a large part of this discussion, but it seems to me that the expected and reasonable way of determening what files to include would be to:

1. Specify files in setup.py 2. Use the files that is a part of the VCS.

What if you don't want to include some files that are present in the VCS ?

Well, then you need to specify stuff in setup.py.

So part of your file list is declared implicitely in your (D)VCS and part explicitely in your setup. This is getting complicated.

...

This is an implicit behavior.

It's a practical default.

For sure. But that means there will be still more than one way to do it Unless the resulting MANIFEST file is always stored in the (D)VCS besides setup.py maybe.. Tarek

On Mon, Apr 6, 2009 at 10:05 AM, Lennart Regebro wrote:

On Mon, Apr 6, 2009 at 09:51, Tarek Ziadé wrote:

...

So part of your file list is declared implicitely in your (D)VCS and part explicitely in your setup.

I don't see how that would work. Reasonably, the VCS is a fallback of nothing is declared in setup.py.

Ok so, in version 1.2 of your package, you use the implicit way (VCS based) then you introduce a new file, that forces your to be explicit in 1.3 so you use setup.py then in 1.4 you're back in an implicit list... this is not a long-term solution indeed.

I don't see why you should be forced to explicitly declare every file in the setup.py, a sensible default is not a problem.

With glob-style patterns this is not a burden at all. and it's all in clear text there, not depending on any magic. Cheers Tarej

On Mon, Apr 6, 2009 at 11:03 AM, Lennart Regebro wrote:

On Mon, Apr 6, 2009 at 10:14, Tarek Ziadé wrote:

...

Ok so, in version 1.2 of your package, you use the implicit way (VCS based) then you introduce a new file, that forces your to be explicit in 1.3

Why?

because in 1.2 you don't want to add one file that is present in your VCS.

...

so you use setup.py then in 1.4 you're back in an implicit list...  this is not a long-term solution indeed.

No, that's a very strange solution. I don't believe I have suggested anything like that.

I am just describing a scenario using your logic. I guess I'll write something down, it's simpler Cheers Tarek

I know it might look sooo web 0.1 but has anyone thought to put the whole sdist stuff under a flow chart (as well the distutils stuffs)? Much of this discussion it would be easier in some way. About the vcs magics. Some vcs systems do not have "special" directories under the source structures (it come to my mind perforce): they hold meta data into a remote server. Regards, Antonio Cavallo

Could we in this discussion, instead of making fuzzy general statements explain exactly what the problems are?

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Tarek Ziadé wrote:

On Mon, Apr 6, 2009 at 9:30 AM, Lennart Regebro wrote:

...

On Mon, Apr 6, 2009 at 09:23, Tarek Ziadé wrote:

...

...

I don't understand a large part of this discussion, but it seems to me that the expected and reasonable way of determening what files to include would be to:

1. Specify files in setup.py 2. Use the files that is a part of the VCS. What if you don't want to include some files that are present in the VCS ? Well, then you need to specify stuff in setup.py.

So part of your file list is declared implicitely in your (D)VCS and part explicitely in your setup. This is getting complicated.

Or we add support allow specifying the exclusions directly.

...

...

This is an implicit behavior. It's a practical default.

For sure. But that means there will be still more than one way to do it

Unless the resulting MANIFEST file is always stored in the (D)VCS besides setup.py maybe..

- -1 to any file which looks like the current MANIFEST{,.in} at all, anywhere, in the "one way" model (as opposed to BBB). Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJ2k8T+gerLs4ltQ4RAnJ0AKDCbdc66fA4npvYhs1iQ9+h74tzXwCdFU1Z /GH9J+mXkyFAw9cyZ7CiAv8= =OAxe -----END PGP SIGNATURE-----

One particular annoyance is that the svn integration built-in to setuptools tends to break every time a major release of Subversion comes out (e.g. setuptools broke when 1.5 came out, is probably currently broken with 1.6, etc.). The "stable" version of setuptools was broken wrt. svn 1.5 for so long that someone made a temporary fork of setuptools and provided a svn 1.5 compatible build. It was annoying to say the least, because it prevented us from upgrading to 1.5 for a while.

I'm the maintainer of the setuptools_darcs plugin, and it broke when darcs v2.0 came out, but since it is a little separately-packaged plugin it was easy for me to fix it and distribute a fixed plugin. Regards, Zooko