Hello, I've sent a mail after Pycon to request some comments on the work we have done during Pycon but I had no feedback at all :) http://mail.python.org/pipermail/distutils-sig/2010-March/015684.html I realize that this piece of work is quite dense, and hard to follow. (As a reminder the draft is here: http://hg.python.org/distutils2/file/tip/docs/design/wiki.rst) I propose to make it a PEP on its own, and to reduce the scope of PEP 376 to what is contains already. This new PEP will be an extension to the work done in PEP 376 and will not really interfer : the new piece of work might add a new file in the egg-info directory, but the rest of the work is elsewhere. (in sysconfig) Then I propose to focus on PEP 376, and to go ahead and try to have it accepted before the first beta of Python 2.7. So basically, PEP 376 defines a directory containing the metadata of an installed package, and all information about the files installed. The goal is to provide some APIs in pkgutil, in order to query what's installed. In other words, add in Python's stdlib what setuptools provides with pkg_resources. So what about starting a new round of comments on this PEP, keeping in mind that all the unsolved issues about resource files are being worked out in the other document ? There's one thing I would like to comment : Someone at Pycon asked me why the structure was called "PROJECT_NAME.egg-info". After explaining to him the historical reasons, I realized that there's no benefits at all to keep this name: - we've removed the python version from the name of the directory - we stated that the new APIs will not recognize the old formats (setuptools', distutils) - any existing software that want to use the new format will have to change its code anyways, and a name change don't really add more work. So I am proposing to replace "egg" by "pkg". For example: - docutils-0.5.egg-info/ PKG-INFO RECORD Becomes: - docutils-0.5.pkg-info/ PKG-INFO RECORD I am also proposing to change the name of the file "PKG-INFO" into "METADATA" to stay consistent, because that's what it is.. : - docutils-0.5.pkg-info/ METADATA RECORD With these definitions: - pkg-info directory : information about a Python software package that is installed on the system - METADATA file: PEP 345 file about a Python software package that is installed on the system Any comment on this ? Or on anything else on PEP 376 ? Regards Tarek -- Tarek Ziadé | http://ziade.org
On 3/25/2010 6:20 AM, Tarek Ziadé wrote: <snip>
So I am proposing to replace "egg" by "pkg". For example:
- docutils-0.5.egg-info/ PKG-INFO RECORD
Becomes:
- docutils-0.5.pkg-info/ PKG-INFO RECORD
Given issue #8 (http://bitbucket.org/tarek/hitchhiker-guide-packaging/issue/8/confusing-use-...), might this be a more accurate renaming? docutils-0.5.dist-info/ DIST-INFO RECORD -John
At 11:05 AM 3/25/2010 -0400, John Posner wrote:
Given issue #8 (http://bitbucket.org/tarek/hitchhiker-guide-packaging/issue/8/confusing-use-...), might this be a more accurate renaming?
docutils-0.5.dist-info/ DIST-INFO RECORD
+1 (though DIST-INFO could be METADATA)
2010/3/25 P.J. Eby <pje@telecommunity.com>:
At 11:05 AM 3/25/2010 -0400, John Posner wrote:
Given issue #8 (http://bitbucket.org/tarek/hitchhiker-guide-packaging/issue/8/confusing-use-...), might this be a more accurate renaming?
docutils-0.5.dist-info/ DIST-INFO RECORD
+1 (though DIST-INFO could be METADATA)
Yes so : docutils-0.5.dist-info/ METADATA RECORD Sounds reasonable -- Tarek Ziadé | http://ziade.org
On 2010-03-25, at 10:21 AM, Tarek Ziadé wrote:
2010/3/25 P.J. Eby <pje@telecommunity.com>:
At 11:05 AM 3/25/2010 -0400, John Posner wrote:
Given issue #8 (http://bitbucket.org/tarek/hitchhiker-guide-packaging/issue/8/confusing-use-...), might this be a more accurate renaming?
docutils-0.5.dist-info/ DIST-INFO RECORD
+1 (though DIST-INFO could be METADATA)
Yes so :
docutils-0.5.dist-info/ METADATA RECORD
+1 I presume sdist too, if generated with 'python setup.py sdist', will contain this .dist-info directory? -srid
On Thu, Mar 25, 2010 at 8:56 PM, Sridhar Ratnakumar <sridharr@activestate.com> wrote: [..]
+1 I presume sdist too, if generated with 'python setup.py sdist', will contain this .dist-info directory?
I don't think so, because part of the files are generated at installation time. Do you have something in mind ? -- Tarek Ziadé | http://ziade.org
On 2010-03-29, at 12:32 AM, Tarek Ziadé wrote:
On Thu, Mar 25, 2010 at 8:56 PM, Sridhar Ratnakumar <sridharr@activestate.com> wrote: [..]
+1 I presume sdist too, if generated with 'python setup.py sdist', will contain this .dist-info directory?
I don't think so, because part of the files are generated at installation time.
Do you have something in mind ?
Yes, I read metadata of source distributions by trying to find the PKG-INFO files and extracting at least Name and Version from it (using filenames is unreliable -- eg: "latest.zip"). I thought that due to environment markers and such changes hinting static metadata, the source distribution will at least have METADATA (formerly PKG-INFO) in it. Will that be the case with distutuils2? I know that distutils/setuptools includes PKG-INFO. -srid
On Mon, Mar 29, 2010 at 6:20 PM, Sridhar Ratnakumar <sridharr@activestate.com> wrote:
On 2010-03-29, at 12:32 AM, Tarek Ziadé wrote:
On Thu, Mar 25, 2010 at 8:56 PM, Sridhar Ratnakumar <sridharr@activestate.com> wrote: [..]
+1
I presume sdist too, if generated with 'python setup.py sdist', will contain
this .dist-info directory?
I don't think so, because part of the files are generated at installation time.
Do you have something in mind ?
Yes, I read metadata of source distributions by trying to find the PKG-INFO files and extracting at least Name and Version from it (using filenames is unreliable -- eg: "latest.zip"). I thought that due to environment markers and such changes hinting static metadata, the source distribution will at least have METADATA (formerly PKG-INFO) in it. Will that be the case with distutuils2? I know that distutils/setuptools includes PKG-INFO.
I guess we could generate METADATA through sdist but not the other files. I am a bit worried of having a partial .dist-info directory though, not sure why Tarek -- Tarek Ziadé | http://ziade.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Tarek Ziadé wrote:
On Mon, Mar 29, 2010 at 6:20 PM, Sridhar Ratnakumar <sridharr@activestate.com> wrote:
On 2010-03-29, at 12:32 AM, Tarek Ziadé wrote:
On Thu, Mar 25, 2010 at 8:56 PM, Sridhar Ratnakumar <sridharr@activestate.com> wrote: [..]
+1
I presume sdist too, if generated with 'python setup.py sdist', will contain
this .dist-info directory?
I don't think so, because part of the files are generated at installation time.
Do you have something in mind ?
Yes, I read metadata of source distributions by trying to find the PKG-INFO files and extracting at least Name and Version from it (using filenames is unreliable -- eg: "latest.zip"). I thought that due to environment markers and such changes hinting static metadata, the source distribution will at least have METADATA (formerly PKG-INFO) in it. Will that be the case with distutuils2? I know that distutils/setuptools includes PKG-INFO.
I guess we could generate METADATA through sdist but not the other files.
I am a bit worried of having a partial .dist-info directory though, not sure why
It seems to me that sdists and binary dists need to be introspectable in the same way that and installed and "develop" dists are. Shouldn't the information needed to generate the RECORD file be present in the sdist somehow? E.g., the CSV file could be there, but with placeholder / logical names for the target files? I have real use cases for treating them all as "equal animals": - http://packages.python.org/pkginfo - http://packages.python.org/compoze Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkuxLjkACgkQ+gerLs4ltQ7IQACdEYme7rVRbzSgYF1tQLaHsgFY XCEAn18ur5U5H08hLAdPTaQ1M7UmEhOH =OleV -----END PGP SIGNATURE-----
[..]
I am a bit worried of having a partial .dist-info directory though, not sure why
It seems to me that sdists and binary dists need to be introspectable in the same way that and installed and "develop" dists are. Shouldn't the information needed to generate the RECORD file be present in the sdist somehow? E.g., the CSV file could be there, but with placeholder / logical names for the target files?
For a binary distribution, the file will be there since a bdist does an install call in a local root, For the sdist, I think that's basically heading to the work we have started at Pycon : a list of files with expandable variables. The variables values will be taken from the global sysconfig.cfg file, as described in wiki.rst. We said during Pycon that this could be another file alongside RECORD, that would be created by reading the (future) [resources] and [files] sections in setup.cfg for example.
I have real use cases for treating them all as "equal animals":
Yes. OS packagers have similar use cases -- Tarek Ziadé | http://ziade.org
On Tue, Mar 30, 2010 at 7:48 AM, Tres Seaver <tseaver@palladion.com> wrote:
It seems to me that sdists and binary dists need to be introspectable in the same way that and installed and "develop" dists are. Shouldn't the information needed to generate the RECORD file be present in the sdist somehow? E.g., the CSV file could be there, but with placeholder / logical names for the target files?
I would strongly suggest at looking at existing projects which do the exact same thing before inventing something from scratch. For example, cabal has the notion of Installed Package Info, which is produced at build time and shipped in binaries (and somewhere at install time), for this exact purpose. The syntax does not matter, but the key points are: - any installer can be regenerated from this manifest (+ the built files of course) - it contains everything necessary to index the corresponding package when installed (when installed, the path variables should be resolved to the actual variables) - it should be reasonably fast to parse because most packagers will need to deal with those quite extensively (although one could imagine putting this info in a db to speed things up). FWIW, I used to use a custom format for toydist, but I am now converting to a simple ConfigParser-like format, which will look like: [meta] name = Sphinx # All the PKG-INFO metadata here .... [variables] # All path variables, + a few toydist specific stuff prefix = some_path site_packages = $libdir/$py_version/site-packages ... [file:pythonfiles:library] source_dir = build/lib.linux-2.6/ target_dir = $sitedir files = ..... David
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Dear all, I am not sure if this is the correct mailinglist to address my question to but it was the closest thing I could find (I see a few names from the PyPi-commits mailing list in the archives here). If this is not the correct place please let me know so I can post my question to the correct mailing list. The XMLRPC API documentation for PyPi at http://wiki.python.org/moin/PyPiXmlRpc contains a 'downloads' key in the example. However in the documentation below it is missing and my own tries at the API confirm this. Now I would very much like to have access to the data (especially the download count). Is there any possibility of these values returning? Regards, Simon de Vlieger -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) iQIcBAEBAgAGBQJLsfp/AAoJEBBSHP7i+JXfeaMQAKPRKalf/kiFXzi0yCuyKM9Z VJzHdtB5lE3F6EyvWj9iVML1in7HNnO1yvBRfwVTT9QWLyqrn62MFz/wd05NGP6l 8WVGJXNGvne3evUdDAj4aHSf6ZkAtO14MMjjW49gi52MDK9Gncdj0xxH8xG9VCaP MAE5EJf7EPJZ0BR3j6ta4L40mTJKwJFFCLTq6OrxRdcx67ua0BUGtfkyCI4ucakv yzq9vP4mnx7riV2cLkEY7IR5nkl2GzlC8iPjP9ORgeDRxvy1qYHcOnyn5OsNuv79 zs2Exq0wGXSa3moPMS+q05X+zrmAf5sSR8It8/qeVaSKehdAlP9LQfFpIYF8J7kE w7clasPhWyH1Q4v24lvCBblR8LKkDSBNFEl14rY2VUIFvM9onmq1KBuQJc1Gl5Fl WAnQK19MpaqZgEBdHXNUtIcuygm036hQOChQVhXYqSgzPARp45Gh9AhfU+Wjtt5V v+FFrHSOs9LBWyJWoahIYSj7JlcKNPvsjfB/F4PeQSJaRFhbgD2arYOD11r//p60 SUvszUbTBvcLr0cQ6HqQCymhOOTC/baG9TU4ikbAoYwoD6FEPsVxe6VEGEBwXiUI eY0bEWZ3nwUm6dgYITDvLebI4APIQxMIlzgVkuCZeyoil5wFSFO9e082Qh0hemRP yn/0APlqzM1lxOPQx3Tv =UdMs -----END PGP SIGNATURE-----
Hi Simon, The right list is the Catalog-SIG Regards, Tarek
I am not sure if this is the correct mailinglist to address my question to but it was the closest thing I could find (I see a few names from the PyPi-commits mailing list in the archives here). If this is not the correct place please let me know so I can post my question to the correct mailing list.
catalog-sig would be a better place for discussion.
Now I would very much like to have access to the data (especially the download count). Is there any possibility of these values returning?
Please submit a feature request to sf.net/projects/pypi. Regards, Martin
Hi On Thu, Mar 25, 2010 at 11:20:41AM +0100, Tarek Ziadé wrote:
I've sent a mail after Pycon to request some comments on the work we have done during Pycon but I had no feedback at all :)
http://mail.python.org/pipermail/distutils-sig/2010-March/015684.html
I realize that this piece of work is quite dense, and hard to follow.
(As a reminder the draft is here: http://hg.python.org/distutils2/file/tip/docs/design/wiki.rst)
This work calls for a systemwide sysconfig.cfg file which maps the categories onto system locations. I notice the example one contains sections for "posix" and "posix_home" (among others), presumably the "posix_home" one is for when using "setup.py --home". But what about the user site-packages directory (setup.py --user)? Would it make sense to have a sysconfig.cfg (or equivalent) file in the user site-packages too? On a tangent it seems to like --home should be deprecated in favour of --user. Any toughts on this? As for the 1st point of the open issues - what about pkgutil.open() without having to use a "setup.py develop"; here a half baked idea: If the sys.path entry from which the current module came from does not contain the setup.cfg metadata required can pkgutil.open() not assume that it's a non-installed package/module and simply use the relative path to find the resource? Altough I can still see this getting messy with non-inplace builds of extension modules. Lastly has toydist's "toysetup.info" from David Cournapeau been considered in the discussions on setup.cfg for distutils2? It's a pretty nice format and it actually has an implementation too. I do think there is value in that format to superseed the setup.py/setup.cfg combination (there is the problem of setup.cfg now becoming both a config file for the developer as well as a formal description for the package distribution). Currently it seems a lot more verbose for tagging files under categories compared to the proposed setup.cfg, but that might not be set in stone and I hope David will be happy to incorporate reasonable proposals if this is preceived to be a problem. Regards Floris PS: Thanks for the work of writing up the PyCon packaging sprint! -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org
On a tangent it seems to like --home should be deprecated in favour of --user. Any toughts on this?
I personally like to use ~/.local as prefix, but I’m sure there are a lot of people used to have bin, lib and co. in their home directory. I think it’s clutter but everyone is entitled to their preferences, so I don’t think it’s feasible to deprecate --home. Regards
On 2010-03-29, at 2:00 PM, Éric Araujo wrote:
On a tangent it seems to like --home should be deprecated in favour of --user. Any toughts on this?
I personally like to use ~/.local as prefix, but I’m sure there are a lot of people used to have bin, lib and co. in their home directory. I think it’s clutter but everyone is entitled to their preferences, so I don’t think it’s feasible to deprecate --home.
Well, since PEP 370 is more than --home (adds ~/.local to sys.path automatically), I favour deprecation of --home with a message suggesting the user to set "site.USER_BASE" to his home directory. -srid
On Mon, Mar 29, 2010 at 10:37 PM, Floris Bruynooghe <floris.bruynooghe@gmail.com> wrote:
Hi
On Thu, Mar 25, 2010 at 11:20:41AM +0100, Tarek Ziadé wrote:
I've sent a mail after Pycon to request some comments on the work we have done during Pycon but I had no feedback at all :)
http://mail.python.org/pipermail/distutils-sig/2010-March/015684.html
I realize that this piece of work is quite dense, and hard to follow.
(As a reminder the draft is here: http://hg.python.org/distutils2/file/tip/docs/design/wiki.rst)
This work calls for a systemwide sysconfig.cfg file which maps the categories onto system locations. I notice the example one contains sections for "posix" and "posix_home" (among others), presumably the "posix_home" one is for when using "setup.py --home".
Yes,
But what about the user site-packages directory (setup.py --user)? Would it make sense to have a sysconfig.cfg (or equivalent) file in the user site-packages too?
What about having a user section in sysconfig.cfg ?
On a tangent it seems to like --home should be deprecated in favour of --user. Any toughts on this?
Why deprecating --home ? It's a target for unix-only systems, but not specific to a user
As for the 1st point of the open issues - what about pkgutil.open() without having to use a "setup.py develop"; here a half baked idea: If the sys.path entry from which the current module came from does not contain the setup.cfg metadata required can pkgutil.open() not assume that it's a non-installed package/module and simply use the relative path to find the resource? Altough I can still see this getting messy with non-inplace builds of extension modules.
I think it's a bit what we've try to do: pkgutil.open() works with paths in the development tree, relative to setup.py/cfg. If its' not what you mean, can you provide examples ?
Lastly has toydist's "toysetup.info" from David Cournapeau been considered in the discussions on setup.cfg for distutils2? It's a pretty nice format and it actually has an implementation too.
No. I havn't looked at it yet, I think its pretty new (like, a few months maybe?)
I do think there is value in that format to superseed the setup.py/setup.cfg combination (there is the problem of setup.cfg now becoming both a config file for the developer as well as a formal description for the package distribution). Currently it seems a lot more verbose for tagging files under categories compared to the proposed setup.cfg, but that might not be set in stone and I hope David will be happy to incorporate reasonable proposals if this is preceived to be a problem.
If David could participate to this discussion and present his ideas on this, that would be great. We could compare both syntaxes and try to build the best one.
Regards Floris
PS: Thanks for the work of writing up the PyCon packaging sprint!
For the wiki.rst, that's mainly other people from the sprint group that did this summary ;) So +1 in the thanks -- Tarek Ziadé | http://ziade.org
On Tue, Mar 30, 2010 at 5:37 AM, Floris Bruynooghe <floris.bruynooghe@gmail.com> wrote:
This work calls for a systemwide sysconfig.cfg file which maps the categories onto system locations. I notice the example one contains sections for "posix" and "posix_home" (among others), presumably the "posix_home" one is for when using "setup.py --home". But what about the user site-packages directory (setup.py --user)? Would it make sense to have a sysconfig.cfg (or equivalent) file in the user site-packages too?
I am not sure what you mean but systemwide sysconfig. Do you mean to override defaults ? Concerning toydist, the scheme is as simple as I could make it. First, toydist has an internal dict which copies autoconf: scheme = { 'prefix': '/usr/local', 'eprefix': '$prefix', 'bindir': '$eprefix/bin', 'sbindir': '$eprefix/sbin', .... } Then, the build command of the command line interface to toydist produces a manifest file which produces the list of files to be installed, classified category-wise (python files, extensions, bytecode, etc...). Each category has one installation path attached to it (which are *not* resolved at that point, i.e. the values still contain the $variables form). The $variables are resolved at *install* time: this enables production of both "normal" installation and eggs/wininst/etc... from the same manifest file, as the installation path is resolved by the corresponding install command. With this system, I can also support the FHS-compliant scheme (default on unix), the Mac OS X-way on mac os X, the windows way on windows, and --user/--prefix options without changing any of the logic. This is also quite flexible from a user POV, because toydist has the notion of datafile sections, which can have an arbitrary installation path attached to it (and toydist allows the creation of new $path variable), and can be itself referenced as a $path variable.
Lastly has toydist's "toysetup.info" from David Cournapeau been considered in the discussions on setup.cfg for distutils2? It's a pretty nice format and it actually has an implementation too.
Note that most of the syntax is stolen from cabal, so I don't deserve the credit. The implementation still has issues, though: - there is no support for Unicode (but this should not be too difficult to solve, mostly a matter of tweaking a few regex in the lexer) - the parser uses PLY to define the lexer and grammar - it is relatively slow I am sure people familiar with parsing algos can come up with something much better here - although PLY is quite small as a dependency. There is also experimental support to automatically convert setup.py to the format (it cannot be reliable because of inherent distutils fragility, but it can already convert reasonable packages such as virtualenv, Jinja2 or sphinx).
Currently it seems a lot more verbose for tagging files under categories compared to the proposed setup.cfg
I am not sure to understand what you are referring to ? Do you mean this: DataFiles: mandoc SourceDir: doc/man TargetDir: $manpath Files: man1/foo1, man3/foo3 I would be glad to hear suggestion on a better syntax, but note that both SourceDir and TargetDir are necessary to support all the features I want to (in particular nobase install mode as available in automake).
but that might not be set in stone and I hope David will be happy to incorporate reasonable proposals if this is preceived to be a problem.
Toydist is completely experimental at that point, and anything subject to change - as suggested by its name, it is more a playground to experiment things than a real tool (although I certainly intend to make a real alternative to distutils/setuptools from it in the future). David
After reading through the PEP, one thing that I'd like to see is the use of relative filenames in RECORD. I've used this functionality myself with pip and Silver Lining, where I am transferring code (in its installed form) between systems; so long as you stick to pure-Python code and have relative filenames (as pip uses, and with some fixups of .pth files) this works well. Systems like App Engine also encourage this kind of behavior. Uninstall actually becomes more interesting to me as well in this model, as I have these packages installed for a much longer amount of time (I'm not simply reinstalling everything on new computers). After a bit of IRC discussion, it seems like files that are outside of sys.prefix should not be relative (e.g., a file installed in /etc shouldn't be ../../../../etc/package/some-file), but anything inside sys.prefix can be relative. This would work well with virtualenv, for instance (where most everything is under sys.prefix, and relative files work nicely for all those cases). -- Ian Bicking | http://blog.ianbicking.org | http://twitter.com/ianbicking
On Mon, Mar 29, 2010 at 11:07 PM, Ian Bicking <ianb@colorstudy.com> wrote:
After reading through the PEP, one thing that I'd like to see is the use of relative filenames in RECORD. I've used this functionality myself with pip and Silver Lining, where I am transferring code (in its installed form) between systems; so long as you stick to pure-Python code and have relative filenames (as pip uses, and with some fixups of .pth files) this works well. Systems like App Engine also encourage this kind of behavior. Uninstall actually becomes more interesting to me as well in this model, as I have these packages installed for a much longer amount of time (I'm not simply reinstalling everything on new computers).
After a bit of IRC discussion, it seems like files that are outside of sys.prefix should not be relative (e.g., a file installed in /etc shouldn't be ../../../../etc/package/some-file), but anything inside sys.prefix can be relative. This would work well with virtualenv, for instance (where most everything is under sys.prefix, and relative files work nicely for all those cases).
If no one objects, I'll change it this way. -- Tarek Ziadé | http://ziade.org
On Thu, Mar 25, 2010 at 11:20 AM, Tarek Ziadé <ziade.tarek@gmail.com> wrote: [..]
Then I propose to focus on PEP 376, and to go ahead and try to have it accepted before the first beta of Python 2.7.
I think we are through with this, I'll send a mail to python-dev later today, proposing the PEP. (minor typo/editing tasks are in progress but I think the PEP is now in a good enough shape to be pushed in python-dev) Then I propose to focus on the installation schemes details (pycon sprint + sysconfig work etc..) and build a new PEP out of it :) Regards Tarek -- Tarek Ziadé | http://ziade.org
participants (12)
-
"Martin v. Löwis" -
David Cournapeau -
Floris Bruynooghe -
Fred Drake -
Ian Bicking -
John Posner -
P.J. Eby -
Simon de Vlieger -
Sridhar Ratnakumar -
Tarek Ziadé -
Tres Seaver -
Éric Araujo