Hello After some discussions with people at Pycon, we think that the MANIFEST template should be removed. I would like to deprecate MANIFEST.in in Distutils, in favor of a package_data that can match directories with a glob-style pattern see : http://bugs.python.org/issue5302 If no one objects I will: - deprecate it with a warning in Python 2.7 - add the glob-style pattern for package_data Cheers Tarek -- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/
On Sun, Apr 05, 2009 at 06:45:45PM +0200, Tarek Ziadé wrote:
After some discussions with people at Pycon, we think that the MANIFEST template should be removed.
+1 for fixing the mess that setuptools + distutils manage to make out of the manifest (MANIFEST.in + old MANIFEST lying around + .svn metadata => lots of confusion and lack of reproducibility)
I would like to deprecate MANIFEST.in in Distutils, in favor of a package_data that can match directories with a glob-style pattern see : http://bugs.python.org/issue5302
How would the new syntax look like? Would it allow you to separately specify the files that 1) need to be included in the installed Python packages (e.g. templates and static resources for web pages) 2) need to be included only in the sdist (e.g. HACKING.txt, shell scripts for setting up a dev environment, things like buildout.cfg)
If no one objects I will:
- deprecate it with a warning in Python 2.7 - add the glob-style pattern for package_data
Marius Gedminas -- "Linux was made by foreign terrorists to take money from true US companies like Microsoft." -Some AOL'er. "To this end we dedicate ourselves..." -Don (From the sig of "Don" <don@cs.byu.edu>)
At 11:24 PM 4/5/2009 +0300, Marius Gedminas wrote:
On Sun, Apr 05, 2009 at 06:45:45PM +0200, Tarek Ziadé wrote:
After some discussions with people at Pycon, we think that the MANIFEST template should be removed.
+1 for fixing the mess that setuptools + distutils manage to make out of the manifest (MANIFEST.in + old MANIFEST lying around + .svn metadata => lots of confusion and lack of reproducibility)
If you've reported this as a problem or filed a bug for it, I appear to have missed it. Steps to reproduce, and the expected vs. actual behavior would be helpful.
On Sun, Apr 05, 2009 at 07:51:41PM -0400, P.J. Eby wrote:
At 11:24 PM 4/5/2009 +0300, Marius Gedminas wrote:
On Sun, Apr 05, 2009 at 06:45:45PM +0200, Tarek Ziadé wrote:
After some discussions with people at Pycon, we think that the MANIFEST template should be removed.
+1 for fixing the mess that setuptools + distutils manage to make out of the manifest (MANIFEST.in + old MANIFEST lying around + .svn metadata => lots of confusion and lack of reproducibility)
If you've reported this as a problem or filed a bug for it, I appear to have missed it. Steps to reproduce, and the expected vs. actual behavior would be helpful.
Sorry about that, I though it was working as designed (having seen discussions about it here on this list). 1. Build a package from an svn checkout -> it works 2. Build a package from a tree exported with svn export -> it fails, since the MANIFEST.in is nonexistent/incomplete, but you never notice that while developing 3. Change the test fixture to ignore .svn subdirectories when tarring up the working tree and copying it into the virtual machine for building (cannot use svn export here, since the tests need to test your working tree with uncommitted changes). Watch it *not* fail. Spend an hour until you figure out the old *output* file pkg.egg-info/MANIFEST is now suddenly an *input* file for the manifest generation process and therefore masks errors in your MANIFEST.in. Change the tar logic to skip *.egg-info, watch the test fail, fix MANIFEST.in, watch the test pass, be happy. Usage of .svn dirs and including the previous version of MANIFEST when generating a new one are features intended to make the life simpler for the developer, but when they break down, it's not fun figuring out what happened or why. Do you want me to report this as a bug? The proposed fix cannot be "remove code dealing with .svn and reading the old MANIFEST, instead require MANIFEST.in to be always complete"---that would break backwards compatibility and make people unhappy. Tarek's plan of replacing MANIFEST.in with a new mechanism sounds like an opportunity to get rid of the magic, whence my +1 with the ill-phrased rationale. Marius Gedminas -- "Linux: the operating system with a CLUE... Command Line User Environment". (seen in a posting in comp.software.testing)
At 12:25 PM 4/6/2009 +0300, Marius Gedminas wrote:
On Sun, Apr 05, 2009 at 07:51:41PM -0400, P.J. Eby wrote:
At 11:24 PM 4/5/2009 +0300, Marius Gedminas wrote:
On Sun, Apr 05, 2009 at 06:45:45PM +0200, Tarek Ziadé wrote:
After some discussions with people at Pycon, we think that the MANIFEST template should be removed.
+1 for fixing the mess that setuptools + distutils manage to make out of the manifest (MANIFEST.in + old MANIFEST lying around + .svn metadata => lots of confusion and lack of reproducibility)
If you've reported this as a problem or filed a bug for it, I appear to have missed it. Steps to reproduce, and the expected vs. actual behavior would be helpful.
Sorry about that, I though it was working as designed (having seen discussions about it here on this list).
1. Build a package from an svn checkout -> it works 2. Build a package from a tree exported with svn export -> it fails, since the MANIFEST.in is nonexistent/incomplete, but you never notice that while developing 3. Change the test fixture to ignore .svn subdirectories when tarring up the working tree and copying it into the virtual machine for building (cannot use svn export here, since the tests need to test your working tree with uncommitted changes).
Can't you use an sdist here instead of a manually-created tarball? Or would that not fix the problem for some reason?
On Mon, Apr 06, 2009 at 09:55:47AM -0400, P.J. Eby wrote:
At 12:25 PM 4/6/2009 +0300, Marius Gedminas wrote:
On Sun, Apr 05, 2009 at 07:51:41PM -0400, P.J. Eby wrote:
At 11:24 PM 4/5/2009 +0300, Marius Gedminas wrote:
On Sun, Apr 05, 2009 at 06:45:45PM +0200, Tarek Ziadé wrote:
After some discussions with people at Pycon, we think that the MANIFEST template should be removed.
+1 for fixing the mess that setuptools + distutils manage to make out of the manifest (MANIFEST.in + old MANIFEST lying around + .svn metadata => lots of confusion and lack of reproducibility)
If you've reported this as a problem or filed a bug for it, I appear to have missed it. Steps to reproduce, and the expected vs. actual behavior would be helpful.
Sorry about that, I though it was working as designed (having seen discussions about it here on this list).
1. Build a package from an svn checkout -> it works 2. Build a package from a tree exported with svn export -> it fails, since the MANIFEST.in is nonexistent/incomplete, but you never notice that while developing 3. Change the test fixture to ignore .svn subdirectories when tarring up the working tree and copying it into the virtual machine for building (cannot use svn export here, since the tests need to test your working tree with uncommitted changes).
Can't you use an sdist here instead of a manually-created tarball? Or would that not fix the problem for some reason?
Not really; this is a tarball of a larger system which is not eggified (and now contains a couple of setuptoolized packages in subdirectories). The thing that worries me most is that when I listen to arguments on this list, it all makes sense, and the sun is shining, and the feature feels good. But then you get new users unfamiliar with all the intricacies, and they get confused, and sometimes release broken packages into the wild. Marius Gedminas -- If it weren't for the last minute, nothing would ever get done.
On Sunday 05 April 2009 10:45:45 am Tarek Ziadé wrote:
I would like to deprecate MANIFEST.in in Distutils, in favor of a package_data that can match directories with a glob-style pattern see : http://bugs.python.org/issue5302
Not all files that are released in an sdist would be considered for installation (ala package_data). Think of top-level README, COPYRIGHT, LICENSE or files (modules) used only by the setup.py script (ala custom commands). It would become an abuse of a keyword `package_data` to indicate files that are not actually *package* data. Remember not all files are meant to be used by the Python code, some, like C header files, would only be used at build time. This just strikes me as being an over-simplification that really benefits no one. Note, however, `sdist` does need to be updated to include more files automatically, like the aforementioned C header files. With those fixes in place, then, maybe, MANIFEST.in would not be required. -- Jeremy Kloth 4Suite Developer http://4suite.org/
On Sun, Apr 5, 2009 at 10:37 PM, Jeremy Kloth <jeremy.kloth@4suite.org> wrote:
On Sunday 05 April 2009 10:45:45 am Tarek Ziadé wrote:
I would like to deprecate MANIFEST.in in Distutils, in favor of a package_data that can match directories with a glob-style pattern see : http://bugs.python.org/issue5302
Not all files that are released in an sdist would be considered for installation (ala package_data). Think of top-level README, COPYRIGHT, LICENSE or files (modules) used only by the setup.py script (ala custom commands). It would become an abuse of a keyword `package_data` to indicate files that are not actually *package* data.
Remember not all files are meant to be used by the Python code, some, like C header files, would only be used at build time. This just strikes me as being an over-simplification that really benefits no one.
The benefit I am seeing is having one and only one explicit way to define what goes into your package distribution. We have many right now, and the worst is : now people define some packages that depends on the source control system they use (like svn) ! So basically, if you get a source distribution out there and work on it in your own DVCS, you are unable to create a distro.
Note, however, `sdist` does need to be updated to include more files automatically, like the aforementioned C header files. With those fixes in place, then, maybe, MANIFEST.in would not be required.
Answering to Marius here too. Right now, when you create a sdist, without any template file, it will include by default these files if it finds them : - README or README.txt - setup.py - test/test*.py - all pure Python modules mentioned in setup script - all files pointed by package_data (build_py) - all files defined in data_files.(build_py) - all files defined as scripts. - all C sources listed as part of extensions or C libraries in the setup script (doesn't catch C headers!) So basically, build_py does most of the hard work, by computing package_data and data_files to build the file list. What build_py misses is the ability to recursively add elements found in subdirectories, which is the new feature I would like to add. People would be able to provide glob-style patterns, to macth all their files. But maybe we would need to define another name for some files that would not fit under "package_data" and "data_files" Last, I find it a bit odd to have README and test/test*.py added explicitely while others have to be defined.
-- Jeremy Kloth 4Suite Developer http://4suite.org/
-- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/
Tarek Ziadé wrote:
Hello
After some discussions with people at Pycon, we think that the MANIFEST template should be removed.
I am all for removing MANIFEST.in, but we should first make sure all its uses are covered by distutils. Currently, they are far from being fullfilled. The ones which come to mind are: - a reliable way to include package_data - a reliable way to add documentation (and install them somewhere which can be customized at build time and installation time) It is important to be able to notify where to put them in sdist and at installation time, cheers, David
On Mon, Apr 6, 2009 at 5:42 AM, David Cournapeau <david@ar.media.kyoto-u.ac.jp> wrote:
Tarek Ziadé wrote:
Hello
After some discussions with people at Pycon, we think that the MANIFEST template should be removed.
I am all for removing MANIFEST.in, but we should first make sure all its uses are covered by distutils. Currently, they are far from being fullfilled. The ones which come to mind are: - a reliable way to include package_data
what do you mean ? (beside the feature that is decribed in the ticket - which enables adding files recursively)
- a reliable way to add documentation (and install them somewhere which can be customized at build time and installation time)
Yes, beside the "how to include it in sdist" there's the "where to install it" This is somehow related to the work started here http://wiki.python.org/moin/Distutils/StaticMetadata Tres and Matthias were talking about adding an "install_doc" new command. Cheers, Tarek
Tarek Ziadé wrote:
what do you mean ? (beside the feature that is decribed in the ticket - which enables adding files recursively)
I mean something more fine-grained than package_data, and more explicit. So that we can separate tests, documentation, etc... as every linux packaging expects. For people who do not care, we can keep the package_data which put everything together. cheers, David
participants (5)
-
David Cournapeau -
Jeremy Kloth -
Marius Gedminas -
P.J. Eby -
Tarek Ziadé