Re: [Python-Dev] "setuptools has divided the Python community"
At 03:28 PM 3/26/2009 -0500, Guido van Rossum wrote:
As someone else suggested, moving some of the functionality to PEP 302 interfaces would also help. Most of the code, though, deals with locating/inspecting installed distributions, resolving version requirements, and managing sys.path. And most of the nastiest complexity comes from trying to support true filename access to resources -- if that were dropped from the stdlib, there'd be no need for egg caches and the like, along with all the complexity entailed. Application environments such as Chandler, Trac, Zope, etc. that want their plugins to live in .egg files wouldn't necessarily be able to use such an API, but the independent pkg_resources wouldn't be disappearing. (Of course, they could also implement application-specific file extraction, if the stdlib API included the ability to inspect and open zipped resources.) The other significant source of complexity is dynamic management of namespace packages; specifically, trying to handle the situation where new sys.path entries (e.g. .egg files added as plugins) need to have their contents added to existing sys.modules __path__ entries. This is perhaps another feature that could be dropped from the stdlib version, given a way to interop with pkg_resources or a replacement.
On Thu, Mar 26, 2009 at 4:33 PM, P.J. Eby <pje@telecommunity.com> wrote:
We had a pretty long discussion about this during the packages session at the language workshop today at PyCon. Tarek should probably report more extensively on this (hopefully he won't take it straight to distutils-sig) but in summary, the plan is roughly: - standardize more metadata, especially including dependencies, and APIs for accessing the metadata and dependencies - there should be an API to get metadata for a package without actually executing any of the package's installation script - this will go into the stdlib for Python 2.7 and 3.1 - it will be provided as separately downloadable backports for prior versions (maybe using a bootstrap not unlike ez_install) - keep distutils, but start deprecating certain higher-level functionality in it (e.g. bdist_rpm) - don't try to provide higher-level functionality in the stdlib, but instead let third party tools built on top of these core APIs compete It will be a while, but once this is a reality, it should be easy enough for the existing crop of tools (like setuptools, pip, zc.buildout) to be retargeted for the new API, and the benefit would be that for *consumers* of packages it doesn't matter which tool was used. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
2009/3/27 Guido van Rossum <guido@python.org>:
Please don't move bdist_wininst out of the core, though! I'd argue that Windows is a special case, as many Windows users don't have the ability to build their own extensions, so they rely heavily on binary installers. And there's no "Windows packagers" organisation to produce such things in the way that Linux has people building debs, rpms, etc. Making it as easy as possible for a random developer to build a Windows installer (by including bdist_wininst or equivalent in the core) is therefore significantly more beneficial than doing the same for a Linux/Mac/Solaris/... installer (Mac users or anyone else speak up here if I'm misrepresenting you!) policies. As Windows has no such policies (or any that do exist are routinely ignored :-)) there is no pressure to "fix" bdist_wininst as policies change. Sorry if this was discussed at the session already. Paul. PS I'm using bdist_wininst as an example here. If someone were to come up with an alternative (core) means of building Windows installers for Python packages (maybe based on a common binary format such as eggs) that would be equally good. It's having no way in the core of building Windows installers that I'm arguing against.
Paul Moore <p.f.moore@gmail.com> writes:
This is the case for many free-software projects; yet such projects (depending on popularity) *do* accumulate volunteers capable of building binary packages for their platform, and then contributing them back to the project for use by all. None of this needs explicit special handling in the ‘distutils’ core, IIUC.
I would argue that the Python community has a wealth of people quite capable of taking on this particular task, and if it makes the core architecture and maintenance of ‘distutils’ simpler to remove special cases for binary installers, I think that's a pearl of great price. -- \ “Holy uncanny photographic mental processes, Batman!” —Robin | `\ | _o__) | Ben Finney
On Fri, Mar 27, 2009 at 8:28 PM, Ben Finney <bignose+hates-spam@benfinney.id.au> wrote:
I think there are two points making binary installers pluggable, so that they are independent of a core distutils, and including such plugins in the stdlib. Nobody argues against the first case: that's certainly a common complain that distutils is a big ball of code where everything is intervened. Concerning contribution for windows binaries: as the current numpy developer in charge of windows binaries and windows support for a while, my experience is that the windows situation for contribution is very different from the other platforms. The mentality is just different. At the risk of an overly broad and unfair generalization, my experience is that on windows, people just want things to work, complain when they do not, and almost never contribute back to make it work, or when they do, they are almost never familiar with how things work on other platforms, so they suggest broken fixes. To say it differently: I mostly use Linux, the less time I am on windows, the happier I am, but bdist_wininst is the only distutils bdist_* command I care about. cheers, David
2009/3/27 David Cournapeau <cournape@gmail.com>:
As a Windows user, I'll actually agree with this :-) As a bit of self-justification, I'll point out that it's *hard* to get things (specifically, C extensions, in this context) to work on Windows. The lack of a C compiler available "as standard" can be a nuisance (although nowadays, MSVC and mingw are good free options, so it's *only* a nuisance these days, rather than a showstopper). The lack of any reasonable standard for external libraries (zlib, jpeg, whatever) can be a huge issue [1]. So it's often a case of "give me a working binary, or it's too hard for me to use this module and I'll find another solution". Not contributing back is probably a cultural thing. When the norm is to pay a lot of money for software, you get used to the idea that you expect not to have to fix the bugs in it yourself :-) I try hard not to forget that this doesn't apply when I get something for free, but I don't always remember... Windows users don't understand Unix, sure. But Unix users don't understand Windows either. And one person's "broken fix" is another's "works for me". Would you feel better if I posted a fix and explicitly said "this works for me, but I don't know anything about your system, so it may need tweaking to apply generally"? The effect is identical, it's just how I express it (and FWIW I've seen exactly this type of scenario where it's a Linux-specific fix not working for Windows (or Macs, for that matter)). The alternative is not accepting contributions unless they are perfect, or expecting every contributor to be willing to spend a lot of time learning the intricacies of systems they'll never use, to improve their contribution in ways that make no difference to them, and which they can't even test. Anyway, this is straying well off-topic, into Linux vs Windows territory, so I shall say no more! Paul [1] As a concrete example: My current PC is newly built, with Python 2.6 and MS Visual Studio 2008 Express and mingw installed. That's all the development tools I presently have. How much time would I have to spend to get to a point where I could build numpy for myself, from this point? Based on previous experience, my instinct is that the answer is "too much effort to justify it for the limited interest I have in numpy"...)
On 2009-03-27 04:19, Guido van Rossum wrote:
Should this be read as: - remove bdist_rpm from the stdlib and let it live on PyPI ? Perhaps I just misunderstand the comment. I think that esp. the bdist_* commands help developers a lot by removing the need to know how to build e.g. RPMs or Windows installers and let distutils deal with it. The bdist_* commands don't really provide any higher level functionality. They only provide interfaces to certain packaging formats commonly used on the various platforms. Instead of removing such functionality, I think we should add more support for standard packaging formats to distutils, e.g. bdist_deb, bdist_pkg, etc. And for eggs, there should be a standard bdist_egg, written against the core distutils APIs (*), creating archives which other Python package managers can then use in whatever way they seem fit. Just please don't tie eggs to one specific package manager, e.g. having to install setuptools just to run eggified packages is just plain wrong. The format itself doesn't require this and neither should the software shipped with those eggs. (*) I've had a go at this a few months ago and then found out that the egg format itself is not documented anywhere. As a result you have to dig deep into setuptools to find out which files are needed and where. That's something that needs to change (Tarek is already working on a PEP for this, AFAIK). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 27 2009)
2009-03-19: Released mxODBC.Connect 1.0.1 http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On Fri, Mar 27, 2009 at 9:49 PM, M.-A. Lemburg <mal@egenix.com> wrote:
I think it is a big dangerous to build rpm/deb without knowing how to build them, because contrary to windows .exe, rpm/deb install things system-wide, and you could easily break something. I don't think you can build deb/rpm without knowing quite a lot about them.
It is "documented" here: http://peak.telecommunity.com/DevCenter/EggFormats But as said in the preambule, people are not supposed to rely on this. I for once would be really happy if I could build eggs without setuptools - for example to build eggs from scons, scripts, etc... cheers, David
On 2009-03-27 13:58, David Cournapeau wrote:
Well, at least the bdist_rpm command did a pretty good job: RedHat used the RPM spec generated by it to ship egenix-mx-base in their distribution.
Thanks for the link. I must have missed that in my search for a format spec.
Right, the .egg format should be a formal PEP standard. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 27 2009)
2009-03-19: Released mxODBC.Connect 1.0.1 http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
M.-A. Lemburg wrote:
As one of the people who proposed this, I think it means: move bdist_rpm, bdist_msi, etc. out of distutils, but provide some of them with the standard Python installation. I'm certain that as part of the refactoring and simplification of distutils we'll gradually move the existing bdist_* commands into separate, stand-alone "things" (scripts, callable modules, or something). We'll need to do this if only for testing, so we may as well make them work.
Agreed.
Agreed.
I think that whatever we produce will need to be supported by the standalone version of the installer portion that will be backported and separately available.
On Fri, Mar 27, 2009 at 8:02 AM, Eric Smith <eric@trueblade.com> wrote:
One of the motivations for deprecating this (and for using this specific example) was that Matthias Klose, the Python packager for Debian, said he never uses bdist_rpm. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
On Mar 27, 2009, at 3:56 PM, Guido van Rossum wrote:
Given that Debian doesn't use RPMs, isn't that expected? I'm actually in favor of removing the bdist_* from the standard library, and allowing 3rd-party tools to implement whatever they need for the distros. But I don't think what you're presenting there supports it. -Fred -- Fred Drake <fdrake at acm.org>
On 2009-03-27 20:56, Guido van Rossum wrote:
Why would a Debian developer want to use bdist_rpm ? Perhaps you meant a bdist_deb command, but that's not part of the stdlib distutils. More importantly: Why is the non-use of a command by a single Python developer enough motivation to remove a useful feature of distutils that's been in use by many others for years ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 27 2009)
2009-03-19: Released mxODBC.Connect 1.0.1 http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On Fri, Mar 27, 2009 at 3:48 PM, M.-A. Lemburg <mal@egenix.com> wrote:
From the discussions I had with RPM packagers, bdist_rpm is hard to use to comply with all the different RPM-based systems out there,
Fedora, Red Hat, etc.. I think that each OS community should maintain its own tool, that complies to the OS standard (wich has its own evolution cycle) Of course this will be possible as long as Distutils let the system packager find/change the metadata in an easy way. I think this is the same rationale for debian packages. Right now people tend to use external tools like stdeb and they are OK with it (but still gets problems extracting stuff out of Python packages at this point) Regards Tarek -- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/
In the specific case of RPMs, I still think that a distutils command is the right solution. It may be that bdist_rpm is a bit too general, and that bdist_fedora and bdist_suse might be more useful. It all comes down to whether the .spec file is written manually or not. *If* it is written manually, there is IMO no need to have the packager's metadata readily available. Whoever writes the spec file can also look at setup.py. OTOH, if the spec file is automatically generated, I can't see why a bdist_ command couldn't do that - and a bdist_ command can easily get at all the package (meta) data it needs. So in this case, I think separate access to the meta-data isn't needed/doesn't help. For bdist_deb, things might be different, as the packager will certainly want to maintain parts of the debian/ directory manually; other parts would be generated. However, I still believe that a bdist_ command would be appropriate (e.g. bdist_dpkg). As I understand Matthias Klose, the tricky part here is that the packaging sometimes wants to reorganize the files in a different layout, and for that, it needs to know what files have what function (documentation, regular package, test suite, etc). If that file classification isn't provided by the package author, then there would be still a manually-maintained step to reorganize the files. The same is potentially true for RPM: if the files need to be layout differently than the package author intended, automatic generation of the spec file will also fail.
Explicit is better than implicit: What is your list of problems with stdeb? Regards, Martin
2009/3/29 "Martin v. Löwis" <martin@v.loewis.de>:
Maybe I don't understand what is meant by metadata, but I don't understand why we can't provide the same metadata as autotools, so that distutils could be customized from the command line to put data where they belong (according to each OS convention). So that making a FHS compliant package would be as simple as python setup.py distutils --datadir=bla --htmldir=foo .... I spent some time looking at cabal this afternoon ("haskell distutils"), and from my current understanding, that's exactly what they are doing: http://www.haskell.org/cabal/release/cabal-latest/doc/users-guide/authors.ht... This way, if some metadata are not provided by upstream, the downstream packager can fix it, and send patches upstream so that other packagers benefit from it. (FWIW, from the reading of cabal documentation, it looks like cabal provides everything I would like distutils to provide: static metadata, good documentation, sane handling of options, etc... Maybe that's something worth looking into as inspiration for improving/fixing distutils) cheers, David
Maybe I don't understand what is meant by metadata, but I don't understand why we can't provide the same metadata as autotools
Likewise, *this* I do not understand. In what way does autotools *provide* metadata? I can understand that it *uses* certain metadata, but it doesn't *provide* them...
What's the meaning of the distutils command? Regards, Martin
On Sun, Mar 29, 2009 at 10:42 PM, "Martin v. Löwis" <martin@v.loewis.de> wrote:
It does not provide them to external tools, true. Let me rephrase this: why cannot distutils use and provide metadata corresponding to the different categories as available in autotools ? It provides both customization from the command line and a relatively straightforward way to set which files go where. Last time this point was discussed on distutils-sig, the main worry came from people who do not care about tagging things appropriately. I don't think it is a big problem, because people already do it in setup.py, or distutils can do it semi automatically (it already has different categories for .py, .pyc, data files, libraries). Also, since packagers have to do it anyway, I think they would prefer something which enable them to send those changes upstream instead of every OS packager having to do it.
python setup.py distutils --datadir=bla --htmldir=foo ....
What's the meaning of the distutils command?
Sorry, this should read python setup.py install ... cheers, David
On Fri, Mar 27, 2009 at 7:49 AM, M.-A. Lemburg <mal@egenix.com> wrote:
+1 ... for this ...
If not the eggs we have today ... the eggs we may incubate for tomorrow ... XD
... partly, because of this ... ;) -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Comandos : Pipe Viewer ... ¿Qué está pasando por esta tubería?
On 27 Mar, 2009, at 7:49, M.-A. Lemburg wrote:
IIRC the reason for wanting to deprecate bdist_rpm (and not adding bdist_deb, ...) is that the variour Linux distributions have varying policies for how to package Python code and those policies tend to vary on another schedule than the Python development schedule. The result of this is that the Linux distributors are incapable to use bdist_rpm. It would therefore be better to ensure that Python packages / distutils can provide the metadata that's needed to build packages and move the actual creation of OS installers outside of the core where the tool can be maintained by people that have detailed knowlegde about the needs of the packaging system and system policies. Ronald
On 2009-03-27 15:00, Ronald Oussoren wrote:
Hmm, I have heard different things - see my reply to Olemis :-) Besides, the bdist_rpm command as well as the others are not only for Linux distributors to use, but to help developers ship their packages to their users *without* the help of some distribution, ie. by providing an RPM or DEB file for download.
Fair enough, though, I'm sure we have enough developer knowledge on board to maintain bdist_rpm and add a bdist_deb. Perhaps even an bdist_dmg :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 27 2009)
2009-03-19: Released mxODBC.Connect 1.0.1 http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
At 01:49 PM 3/27/2009 +0100, M.-A. Lemburg wrote:
(*) I've had a go at this a few months ago and then found out that the egg format itself is not documented anywhere.
It's been documented for just under three years now. Here's where you quoted the email where I announced that documentation, provided links to it, and asked you to let me know if there's anything else you'd need in it: http://mail.python.org/pipermail/python-dev/2006-April/064496.html
On 2009-03-27 17:19, P.J. Eby wrote:
Thanks for reminding me. I must have forgotten about that wiki page and instead looked on the setuptools page. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 27 2009)
2009-03-19: Released mxODBC.Connect 1.0.1 http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
At 08:12 PM 3/27/2009 +0100, M.-A. Lemburg wrote:
For future reference, you can also Google "Python egg formats" or "Python egg file format" or other such phrases, and it always comes up as the first or second hit. Ironically, that's how I found it the other day myself, as I didn't remember right off what the wiki page name was. ;-)
At 11:37 PM 3/26/2009 -0500, Eric Smith wrote:
If a plugin includes C code (.so/.dll), or uses a library that operates on filenames rather than bytes in memory (e.g. gettext), then the resources would need to be extracted from the .egg. pkg_resources transparently extracts such resources to a cache directory when you ask for a resource's filename, rather than asking for a stream or string of its contents. This feature represents a significant chunk of the complexity and code size of pkg_resources -- and I was proposing ways to cut down on that complexity and code size, for a (limited) stdlib version of the functionality.
On 2009-03-27 17:07, P.J. Eby wrote:
This functionality is one of the more annoying setuptools "features". It causes each and every user of e.g. Trac to have their own little version of the same piece of software in their home dir cache. The solution to this is simple: don't use ZIP files for installed packages, instead unzip them into normal directories on sys.path. This makes all these problems go away and allows users to access embedded documentation, configuration, etc. Zip files are great for shipping packages to the end-user, but there's no need to keep them zipped once they get there. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 27 2009)
2009-03-19: Released mxODBC.Connect 1.0.1 http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
M.-A. Lemburg wrote:
I also think the feature should go. If you want functionality that's so difficult to provide when you install as a zip file, the answer is not to make things more complex, but to not install as zip files.
On Fri, Mar 27, 2009 at 1:21 PM, Eric Smith <eric@trueblade.com> wrote:
What about environments like Google App Engine ? I mean, AFAIK using ZIP files is the *official* solution to meet some quotas & requirements ... CMIIW anyway ... I mean, for example: what if someone writes an app containing resources like message catalogs for i18n, uploads it to GAE using ZIP files and still wants to use the MO files (i.e. resources) for translation (i.e. for anything else ...) ... Hmmmm ? -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Comandos : Pipe Viewer ... ¿Qué está pasando por esta tubería?
Olemis Lang wrote:
I mentioned yesterday (in the language summit) that we really need to get all the requirements together before we start any work. I fear that there are many hidden requirements (or ones that I'm personally not aware of, at least). I don't know gettext well enough give an answer yet.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Mar 27, 2009, at 2:27 PM, Eric Smith wrote:
The class-based API for gettext takes streams, so resource_stream() would work just fine. I think i18n plugins for Python do not necessarily need to use the classic gettext API. Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) iQCVAwUBSc1E2XEjvBPtnXfVAQKc5QP+NOgx0Q/yZ47d1HKR2M/TP4gwT08hr1K2 rTZ+YsGVQ6B+pO36uBi3dwOWNzVbPd3VMBCPNE/TBiGRHklqNiI4AxjRyougdmcw 6nhZomS0GxlE//r27nKx/Wl+qS/b8eQ6UWB/jpGRGL1+K/TuI4Zij0yUYBL7PUgn 7JZxeLkueOY= =MI9F -----END PGP SIGNATURE-----
On Fri, Mar 27, 2009 at 4:27 PM, Barry Warsaw <barry@python.org> wrote:
I remembered another similar use case (which is mentioned in PEP 302 -motivation AFAICR- ) ... what about importing Py modules/pkgs packaged in .JAR files to be used in Jython ? ... Hmmmmm? I dont think its a good idea so far to remove zip imports | installs and similar ... at least I'll need further arguments and solutions to concrete use cases to understand this decision a little bit better... ;)
In fact ... I use Babel (... especially Translations & Locale classes ;) quite often ... :P Besides web frameworks and apps also need specific artifacts (e.g. Genshi template filters for i18n ...) to fully accomplish these tasks ... ;o) -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Comandos : Pipe Viewer ... ¿Qué está pasando por esta tubería?
mal> Zip files are great for shipping packages to the end-user, but mal> there's no need to keep them zipped once they get there. I thought one of the arguments for zip files was a performance increase (reduced stat(2) calls, etc). I may misremember though. Skip
On Mar 27, 2009, at 3:24 PM, skip@pobox.com wrote:
You're memory is working fine, but I don't think the way eggs are used accomplishes that. The measurements that supported that performance assertion were made by putting the standard library in a single ZIP, and using that. I don't have measurements handy, but I recall being told by someone who did that using a *lot* of separate ZIP files wasn't as effective. -Fred -- Fred Drake <fdrake at acm.org>
On 2009-03-27 20:24, skip@pobox.com wrote:
True and like Fred already mentioned that's the main reason why we have ZIP file package imports. Putting the stdlib into a ZIP file does make our favorite interpreter start up faster. E.g. py2exe makes use of this feature. However, using eggs (which are ZIP files) directly on the sys.path causes these to get scanned on every startup - regardless of whether you use any of their content. Very much unlike standard Python package directories that only get scanned if they are referenced. Due to the nature of eggs (many small packages), you usually end up having a whole Easter basket full of them in your sys.path. Unless you want to maintain a separate Python installation per task, your overall Python startup time will increase noticeably for every single script you run with it. Perhaps someone should start working on a tool called "FryingPan" to create "Omelettes", ie. all eggs squashed into a single ZIP file... ;-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 27 2009)
2009-03-19: Released mxODBC.Connect 1.0.1 http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On Mar 27, 2009, at 5:22 PM, M.-A. Lemburg wrote:
I've certainly suggested such a tool in various conversations, but it usually comes down to not wanting to build a separate ZIP for (potentially) every script. In practice, I suspect it's a lot better than this. -Fred -- Fred Drake <fdrake at acm.org>
At 10:22 PM 3/27/2009 +0100, M.-A. Lemburg wrote:
Perhaps someone should start working on a tool called "FryingPan" to create "Omelettes", ie. all eggs squashed into a single ZIP file... ;-)
They're called baskets actually. ;-) There's no tool to do it, but pkg_resources does support multiple eggs being contained in the same zipfile.
On Mar 27, 2009, at 6:22 PM, P.J. Eby wrote:
And omelettes are when you take a collection of unzipped eggs and symlink into them to create a "flat" view of the files those eggs provide. (or at least that's the term used in Plone land after they switched to eggs and wrote the collective.recipe.omelette recipe for Buildout to make creating these symlinks easy) http://pypi.python.org/pypi/collective.recipe.omelette There is also a recipe for use with Buildout for gathering a collection of eggs and putting them all in a basket called z3c.recipe.eggbasket. Although that's a little different than having a basket as a zip file with multiple eggs in an installed location, since the z3c.recipe.eggbasket is for sending a bunch of eggs over the internet all in one go, with the idea that that file is uncompressed once the eggs are installed ... because you need to take the eggs out of the basket before you bake an omelette :P http://pypi.python.org/pypi/z3c.recipe.eggbasket
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 skip@pobox.com wrote:
Empirical evidence suggests that the win for zipfiles is only for a *very* small set of them (basically, the whole app and all libraries in a single zipfile). The two lines cross pretty fast after N == 1. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJzWy6+gerLs4ltQ4RAjYUAKCc/yiv5noitSwWavsH3voK9veYnwCfe9qn 16tnRWGZjMazB2fP5G3Q2N4= =LUFx -----END PGP SIGNATURE-----
On Thu, Mar 26, 2009 at 4:33 PM, P.J. Eby <pje@telecommunity.com> wrote:
We had a pretty long discussion about this during the packages session at the language workshop today at PyCon. Tarek should probably report more extensively on this (hopefully he won't take it straight to distutils-sig) but in summary, the plan is roughly: - standardize more metadata, especially including dependencies, and APIs for accessing the metadata and dependencies - there should be an API to get metadata for a package without actually executing any of the package's installation script - this will go into the stdlib for Python 2.7 and 3.1 - it will be provided as separately downloadable backports for prior versions (maybe using a bootstrap not unlike ez_install) - keep distutils, but start deprecating certain higher-level functionality in it (e.g. bdist_rpm) - don't try to provide higher-level functionality in the stdlib, but instead let third party tools built on top of these core APIs compete It will be a while, but once this is a reality, it should be easy enough for the existing crop of tools (like setuptools, pip, zc.buildout) to be retargeted for the new API, and the benefit would be that for *consumers* of packages it doesn't matter which tool was used. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
2009/3/27 Guido van Rossum <guido@python.org>:
Please don't move bdist_wininst out of the core, though! I'd argue that Windows is a special case, as many Windows users don't have the ability to build their own extensions, so they rely heavily on binary installers. And there's no "Windows packagers" organisation to produce such things in the way that Linux has people building debs, rpms, etc. Making it as easy as possible for a random developer to build a Windows installer (by including bdist_wininst or equivalent in the core) is therefore significantly more beneficial than doing the same for a Linux/Mac/Solaris/... installer (Mac users or anyone else speak up here if I'm misrepresenting you!) policies. As Windows has no such policies (or any that do exist are routinely ignored :-)) there is no pressure to "fix" bdist_wininst as policies change. Sorry if this was discussed at the session already. Paul. PS I'm using bdist_wininst as an example here. If someone were to come up with an alternative (core) means of building Windows installers for Python packages (maybe based on a common binary format such as eggs) that would be equally good. It's having no way in the core of building Windows installers that I'm arguing against.
Paul Moore <p.f.moore@gmail.com> writes:
This is the case for many free-software projects; yet such projects (depending on popularity) *do* accumulate volunteers capable of building binary packages for their platform, and then contributing them back to the project for use by all. None of this needs explicit special handling in the ‘distutils’ core, IIUC.
I would argue that the Python community has a wealth of people quite capable of taking on this particular task, and if it makes the core architecture and maintenance of ‘distutils’ simpler to remove special cases for binary installers, I think that's a pearl of great price. -- \ “Holy uncanny photographic mental processes, Batman!” —Robin | `\ | _o__) | Ben Finney
On Fri, Mar 27, 2009 at 8:28 PM, Ben Finney <bignose+hates-spam@benfinney.id.au> wrote:
I think there are two points making binary installers pluggable, so that they are independent of a core distutils, and including such plugins in the stdlib. Nobody argues against the first case: that's certainly a common complain that distutils is a big ball of code where everything is intervened. Concerning contribution for windows binaries: as the current numpy developer in charge of windows binaries and windows support for a while, my experience is that the windows situation for contribution is very different from the other platforms. The mentality is just different. At the risk of an overly broad and unfair generalization, my experience is that on windows, people just want things to work, complain when they do not, and almost never contribute back to make it work, or when they do, they are almost never familiar with how things work on other platforms, so they suggest broken fixes. To say it differently: I mostly use Linux, the less time I am on windows, the happier I am, but bdist_wininst is the only distutils bdist_* command I care about. cheers, David
2009/3/27 David Cournapeau <cournape@gmail.com>:
As a Windows user, I'll actually agree with this :-) As a bit of self-justification, I'll point out that it's *hard* to get things (specifically, C extensions, in this context) to work on Windows. The lack of a C compiler available "as standard" can be a nuisance (although nowadays, MSVC and mingw are good free options, so it's *only* a nuisance these days, rather than a showstopper). The lack of any reasonable standard for external libraries (zlib, jpeg, whatever) can be a huge issue [1]. So it's often a case of "give me a working binary, or it's too hard for me to use this module and I'll find another solution". Not contributing back is probably a cultural thing. When the norm is to pay a lot of money for software, you get used to the idea that you expect not to have to fix the bugs in it yourself :-) I try hard not to forget that this doesn't apply when I get something for free, but I don't always remember... Windows users don't understand Unix, sure. But Unix users don't understand Windows either. And one person's "broken fix" is another's "works for me". Would you feel better if I posted a fix and explicitly said "this works for me, but I don't know anything about your system, so it may need tweaking to apply generally"? The effect is identical, it's just how I express it (and FWIW I've seen exactly this type of scenario where it's a Linux-specific fix not working for Windows (or Macs, for that matter)). The alternative is not accepting contributions unless they are perfect, or expecting every contributor to be willing to spend a lot of time learning the intricacies of systems they'll never use, to improve their contribution in ways that make no difference to them, and which they can't even test. Anyway, this is straying well off-topic, into Linux vs Windows territory, so I shall say no more! Paul [1] As a concrete example: My current PC is newly built, with Python 2.6 and MS Visual Studio 2008 Express and mingw installed. That's all the development tools I presently have. How much time would I have to spend to get to a point where I could build numpy for myself, from this point? Based on previous experience, my instinct is that the answer is "too much effort to justify it for the limited interest I have in numpy"...)
On 2009-03-27 04:19, Guido van Rossum wrote:
Should this be read as: - remove bdist_rpm from the stdlib and let it live on PyPI ? Perhaps I just misunderstand the comment. I think that esp. the bdist_* commands help developers a lot by removing the need to know how to build e.g. RPMs or Windows installers and let distutils deal with it. The bdist_* commands don't really provide any higher level functionality. They only provide interfaces to certain packaging formats commonly used on the various platforms. Instead of removing such functionality, I think we should add more support for standard packaging formats to distutils, e.g. bdist_deb, bdist_pkg, etc. And for eggs, there should be a standard bdist_egg, written against the core distutils APIs (*), creating archives which other Python package managers can then use in whatever way they seem fit. Just please don't tie eggs to one specific package manager, e.g. having to install setuptools just to run eggified packages is just plain wrong. The format itself doesn't require this and neither should the software shipped with those eggs. (*) I've had a go at this a few months ago and then found out that the egg format itself is not documented anywhere. As a result you have to dig deep into setuptools to find out which files are needed and where. That's something that needs to change (Tarek is already working on a PEP for this, AFAIK). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 27 2009)
2009-03-19: Released mxODBC.Connect 1.0.1 http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On Fri, Mar 27, 2009 at 9:49 PM, M.-A. Lemburg <mal@egenix.com> wrote:
I think it is a big dangerous to build rpm/deb without knowing how to build them, because contrary to windows .exe, rpm/deb install things system-wide, and you could easily break something. I don't think you can build deb/rpm without knowing quite a lot about them.
It is "documented" here: http://peak.telecommunity.com/DevCenter/EggFormats But as said in the preambule, people are not supposed to rely on this. I for once would be really happy if I could build eggs without setuptools - for example to build eggs from scons, scripts, etc... cheers, David
On 2009-03-27 13:58, David Cournapeau wrote:
Well, at least the bdist_rpm command did a pretty good job: RedHat used the RPM spec generated by it to ship egenix-mx-base in their distribution.
Thanks for the link. I must have missed that in my search for a format spec.
Right, the .egg format should be a formal PEP standard. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 27 2009)
2009-03-19: Released mxODBC.Connect 1.0.1 http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
M.-A. Lemburg wrote:
As one of the people who proposed this, I think it means: move bdist_rpm, bdist_msi, etc. out of distutils, but provide some of them with the standard Python installation. I'm certain that as part of the refactoring and simplification of distutils we'll gradually move the existing bdist_* commands into separate, stand-alone "things" (scripts, callable modules, or something). We'll need to do this if only for testing, so we may as well make them work.
Agreed.
Agreed.
I think that whatever we produce will need to be supported by the standalone version of the installer portion that will be backported and separately available.
On Fri, Mar 27, 2009 at 8:02 AM, Eric Smith <eric@trueblade.com> wrote:
One of the motivations for deprecating this (and for using this specific example) was that Matthias Klose, the Python packager for Debian, said he never uses bdist_rpm. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
On Mar 27, 2009, at 3:56 PM, Guido van Rossum wrote:
Given that Debian doesn't use RPMs, isn't that expected? I'm actually in favor of removing the bdist_* from the standard library, and allowing 3rd-party tools to implement whatever they need for the distros. But I don't think what you're presenting there supports it. -Fred -- Fred Drake <fdrake at acm.org>
On 2009-03-27 20:56, Guido van Rossum wrote:
Why would a Debian developer want to use bdist_rpm ? Perhaps you meant a bdist_deb command, but that's not part of the stdlib distutils. More importantly: Why is the non-use of a command by a single Python developer enough motivation to remove a useful feature of distutils that's been in use by many others for years ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 27 2009)
2009-03-19: Released mxODBC.Connect 1.0.1 http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On Fri, Mar 27, 2009 at 3:48 PM, M.-A. Lemburg <mal@egenix.com> wrote:
From the discussions I had with RPM packagers, bdist_rpm is hard to use to comply with all the different RPM-based systems out there,
Fedora, Red Hat, etc.. I think that each OS community should maintain its own tool, that complies to the OS standard (wich has its own evolution cycle) Of course this will be possible as long as Distutils let the system packager find/change the metadata in an easy way. I think this is the same rationale for debian packages. Right now people tend to use external tools like stdeb and they are OK with it (but still gets problems extracting stuff out of Python packages at this point) Regards Tarek -- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/
In the specific case of RPMs, I still think that a distutils command is the right solution. It may be that bdist_rpm is a bit too general, and that bdist_fedora and bdist_suse might be more useful. It all comes down to whether the .spec file is written manually or not. *If* it is written manually, there is IMO no need to have the packager's metadata readily available. Whoever writes the spec file can also look at setup.py. OTOH, if the spec file is automatically generated, I can't see why a bdist_ command couldn't do that - and a bdist_ command can easily get at all the package (meta) data it needs. So in this case, I think separate access to the meta-data isn't needed/doesn't help. For bdist_deb, things might be different, as the packager will certainly want to maintain parts of the debian/ directory manually; other parts would be generated. However, I still believe that a bdist_ command would be appropriate (e.g. bdist_dpkg). As I understand Matthias Klose, the tricky part here is that the packaging sometimes wants to reorganize the files in a different layout, and for that, it needs to know what files have what function (documentation, regular package, test suite, etc). If that file classification isn't provided by the package author, then there would be still a manually-maintained step to reorganize the files. The same is potentially true for RPM: if the files need to be layout differently than the package author intended, automatic generation of the spec file will also fail.
Explicit is better than implicit: What is your list of problems with stdeb? Regards, Martin
2009/3/29 "Martin v. Löwis" <martin@v.loewis.de>:
Maybe I don't understand what is meant by metadata, but I don't understand why we can't provide the same metadata as autotools, so that distutils could be customized from the command line to put data where they belong (according to each OS convention). So that making a FHS compliant package would be as simple as python setup.py distutils --datadir=bla --htmldir=foo .... I spent some time looking at cabal this afternoon ("haskell distutils"), and from my current understanding, that's exactly what they are doing: http://www.haskell.org/cabal/release/cabal-latest/doc/users-guide/authors.ht... This way, if some metadata are not provided by upstream, the downstream packager can fix it, and send patches upstream so that other packagers benefit from it. (FWIW, from the reading of cabal documentation, it looks like cabal provides everything I would like distutils to provide: static metadata, good documentation, sane handling of options, etc... Maybe that's something worth looking into as inspiration for improving/fixing distutils) cheers, David
Maybe I don't understand what is meant by metadata, but I don't understand why we can't provide the same metadata as autotools
Likewise, *this* I do not understand. In what way does autotools *provide* metadata? I can understand that it *uses* certain metadata, but it doesn't *provide* them...
What's the meaning of the distutils command? Regards, Martin
On Sun, Mar 29, 2009 at 10:42 PM, "Martin v. Löwis" <martin@v.loewis.de> wrote:
It does not provide them to external tools, true. Let me rephrase this: why cannot distutils use and provide metadata corresponding to the different categories as available in autotools ? It provides both customization from the command line and a relatively straightforward way to set which files go where. Last time this point was discussed on distutils-sig, the main worry came from people who do not care about tagging things appropriately. I don't think it is a big problem, because people already do it in setup.py, or distutils can do it semi automatically (it already has different categories for .py, .pyc, data files, libraries). Also, since packagers have to do it anyway, I think they would prefer something which enable them to send those changes upstream instead of every OS packager having to do it.
python setup.py distutils --datadir=bla --htmldir=foo ....
What's the meaning of the distutils command?
Sorry, this should read python setup.py install ... cheers, David
On Fri, Mar 27, 2009 at 7:49 AM, M.-A. Lemburg <mal@egenix.com> wrote:
+1 ... for this ...
If not the eggs we have today ... the eggs we may incubate for tomorrow ... XD
... partly, because of this ... ;) -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Comandos : Pipe Viewer ... ¿Qué está pasando por esta tubería?
On 27 Mar, 2009, at 7:49, M.-A. Lemburg wrote:
IIRC the reason for wanting to deprecate bdist_rpm (and not adding bdist_deb, ...) is that the variour Linux distributions have varying policies for how to package Python code and those policies tend to vary on another schedule than the Python development schedule. The result of this is that the Linux distributors are incapable to use bdist_rpm. It would therefore be better to ensure that Python packages / distutils can provide the metadata that's needed to build packages and move the actual creation of OS installers outside of the core where the tool can be maintained by people that have detailed knowlegde about the needs of the packaging system and system policies. Ronald
On 2009-03-27 15:00, Ronald Oussoren wrote:
Hmm, I have heard different things - see my reply to Olemis :-) Besides, the bdist_rpm command as well as the others are not only for Linux distributors to use, but to help developers ship their packages to their users *without* the help of some distribution, ie. by providing an RPM or DEB file for download.
Fair enough, though, I'm sure we have enough developer knowledge on board to maintain bdist_rpm and add a bdist_deb. Perhaps even an bdist_dmg :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 27 2009)
2009-03-19: Released mxODBC.Connect 1.0.1 http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
At 01:49 PM 3/27/2009 +0100, M.-A. Lemburg wrote:
(*) I've had a go at this a few months ago and then found out that the egg format itself is not documented anywhere.
It's been documented for just under three years now. Here's where you quoted the email where I announced that documentation, provided links to it, and asked you to let me know if there's anything else you'd need in it: http://mail.python.org/pipermail/python-dev/2006-April/064496.html
On 2009-03-27 17:19, P.J. Eby wrote:
Thanks for reminding me. I must have forgotten about that wiki page and instead looked on the setuptools page. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 27 2009)
2009-03-19: Released mxODBC.Connect 1.0.1 http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
At 08:12 PM 3/27/2009 +0100, M.-A. Lemburg wrote:
For future reference, you can also Google "Python egg formats" or "Python egg file format" or other such phrases, and it always comes up as the first or second hit. Ironically, that's how I found it the other day myself, as I didn't remember right off what the wiki page name was. ;-)
At 11:37 PM 3/26/2009 -0500, Eric Smith wrote:
If a plugin includes C code (.so/.dll), or uses a library that operates on filenames rather than bytes in memory (e.g. gettext), then the resources would need to be extracted from the .egg. pkg_resources transparently extracts such resources to a cache directory when you ask for a resource's filename, rather than asking for a stream or string of its contents. This feature represents a significant chunk of the complexity and code size of pkg_resources -- and I was proposing ways to cut down on that complexity and code size, for a (limited) stdlib version of the functionality.
On 2009-03-27 17:07, P.J. Eby wrote:
This functionality is one of the more annoying setuptools "features". It causes each and every user of e.g. Trac to have their own little version of the same piece of software in their home dir cache. The solution to this is simple: don't use ZIP files for installed packages, instead unzip them into normal directories on sys.path. This makes all these problems go away and allows users to access embedded documentation, configuration, etc. Zip files are great for shipping packages to the end-user, but there's no need to keep them zipped once they get there. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 27 2009)
2009-03-19: Released mxODBC.Connect 1.0.1 http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
M.-A. Lemburg wrote:
I also think the feature should go. If you want functionality that's so difficult to provide when you install as a zip file, the answer is not to make things more complex, but to not install as zip files.
On Fri, Mar 27, 2009 at 1:21 PM, Eric Smith <eric@trueblade.com> wrote:
What about environments like Google App Engine ? I mean, AFAIK using ZIP files is the *official* solution to meet some quotas & requirements ... CMIIW anyway ... I mean, for example: what if someone writes an app containing resources like message catalogs for i18n, uploads it to GAE using ZIP files and still wants to use the MO files (i.e. resources) for translation (i.e. for anything else ...) ... Hmmmm ? -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Comandos : Pipe Viewer ... ¿Qué está pasando por esta tubería?
Olemis Lang wrote:
I mentioned yesterday (in the language summit) that we really need to get all the requirements together before we start any work. I fear that there are many hidden requirements (or ones that I'm personally not aware of, at least). I don't know gettext well enough give an answer yet.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Mar 27, 2009, at 2:27 PM, Eric Smith wrote:
The class-based API for gettext takes streams, so resource_stream() would work just fine. I think i18n plugins for Python do not necessarily need to use the classic gettext API. Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) iQCVAwUBSc1E2XEjvBPtnXfVAQKc5QP+NOgx0Q/yZ47d1HKR2M/TP4gwT08hr1K2 rTZ+YsGVQ6B+pO36uBi3dwOWNzVbPd3VMBCPNE/TBiGRHklqNiI4AxjRyougdmcw 6nhZomS0GxlE//r27nKx/Wl+qS/b8eQ6UWB/jpGRGL1+K/TuI4Zij0yUYBL7PUgn 7JZxeLkueOY= =MI9F -----END PGP SIGNATURE-----
On Fri, Mar 27, 2009 at 4:27 PM, Barry Warsaw <barry@python.org> wrote:
I remembered another similar use case (which is mentioned in PEP 302 -motivation AFAICR- ) ... what about importing Py modules/pkgs packaged in .JAR files to be used in Jython ? ... Hmmmmm? I dont think its a good idea so far to remove zip imports | installs and similar ... at least I'll need further arguments and solutions to concrete use cases to understand this decision a little bit better... ;)
In fact ... I use Babel (... especially Translations & Locale classes ;) quite often ... :P Besides web frameworks and apps also need specific artifacts (e.g. Genshi template filters for i18n ...) to fully accomplish these tasks ... ;o) -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Comandos : Pipe Viewer ... ¿Qué está pasando por esta tubería?
mal> Zip files are great for shipping packages to the end-user, but mal> there's no need to keep them zipped once they get there. I thought one of the arguments for zip files was a performance increase (reduced stat(2) calls, etc). I may misremember though. Skip
On Mar 27, 2009, at 3:24 PM, skip@pobox.com wrote:
You're memory is working fine, but I don't think the way eggs are used accomplishes that. The measurements that supported that performance assertion were made by putting the standard library in a single ZIP, and using that. I don't have measurements handy, but I recall being told by someone who did that using a *lot* of separate ZIP files wasn't as effective. -Fred -- Fred Drake <fdrake at acm.org>
On 2009-03-27 20:24, skip@pobox.com wrote:
True and like Fred already mentioned that's the main reason why we have ZIP file package imports. Putting the stdlib into a ZIP file does make our favorite interpreter start up faster. E.g. py2exe makes use of this feature. However, using eggs (which are ZIP files) directly on the sys.path causes these to get scanned on every startup - regardless of whether you use any of their content. Very much unlike standard Python package directories that only get scanned if they are referenced. Due to the nature of eggs (many small packages), you usually end up having a whole Easter basket full of them in your sys.path. Unless you want to maintain a separate Python installation per task, your overall Python startup time will increase noticeably for every single script you run with it. Perhaps someone should start working on a tool called "FryingPan" to create "Omelettes", ie. all eggs squashed into a single ZIP file... ;-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 27 2009)
2009-03-19: Released mxODBC.Connect 1.0.1 http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On Mar 27, 2009, at 5:22 PM, M.-A. Lemburg wrote:
I've certainly suggested such a tool in various conversations, but it usually comes down to not wanting to build a separate ZIP for (potentially) every script. In practice, I suspect it's a lot better than this. -Fred -- Fred Drake <fdrake at acm.org>
At 10:22 PM 3/27/2009 +0100, M.-A. Lemburg wrote:
Perhaps someone should start working on a tool called "FryingPan" to create "Omelettes", ie. all eggs squashed into a single ZIP file... ;-)
They're called baskets actually. ;-) There's no tool to do it, but pkg_resources does support multiple eggs being contained in the same zipfile.
On Mar 27, 2009, at 6:22 PM, P.J. Eby wrote:
And omelettes are when you take a collection of unzipped eggs and symlink into them to create a "flat" view of the files those eggs provide. (or at least that's the term used in Plone land after they switched to eggs and wrote the collective.recipe.omelette recipe for Buildout to make creating these symlinks easy) http://pypi.python.org/pypi/collective.recipe.omelette There is also a recipe for use with Buildout for gathering a collection of eggs and putting them all in a basket called z3c.recipe.eggbasket. Although that's a little different than having a basket as a zip file with multiple eggs in an installed location, since the z3c.recipe.eggbasket is for sending a bunch of eggs over the internet all in one go, with the idea that that file is uncompressed once the eggs are installed ... because you need to take the eggs out of the basket before you bake an omelette :P http://pypi.python.org/pypi/z3c.recipe.eggbasket
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 skip@pobox.com wrote:
Empirical evidence suggests that the win for zipfiles is only for a *very* small set of them (basically, the whole app and all libraries in a single zipfile). The two lines cross pretty fast after N == 1. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJzWy6+gerLs4ltQ4RAjYUAKCc/yiv5noitSwWavsH3voK9veYnwCfe9qn 16tnRWGZjMazB2fP5G3Q2N4= =LUFx -----END PGP SIGNATURE-----
participants (16)
-
"Martin v. Löwis" -
Barry Warsaw -
Ben Finney -
David Cournapeau -
Eric Smith -
Fred Drake -
Guido van Rossum -
Kevin Teague -
M.-A. Lemburg -
Olemis Lang -
P.J. Eby -
Paul Moore -
Ronald Oussoren -
skip＠pobox.com -
Tarek Ziadé -
Tres Seaver