Re: [Distutils] Distutils and Distribute roadmap (and some words on Virtualenv, Pip)
Paul Moore wrote:
2009/10/20 Ian Bicking <ianb@colorstudy.com>:
Well, I wanted to move it off python-dev, and I didn't really feel like moving it onto distutils-sig because I felt the debate was not particularly important to introduce the confusion of a half-started thread.
Sorry, I hadn't realised it was on python-dev rather than distutils-sig. You're right in that case.
My point is the bdist_win installers work fine for both your use case, Paul, and Chris' use case, so there's no real conflict.
OK, and yet people still seem to argue for eggs rather than bdist_wininst, and projects provide eggs rather than bdist_wininst. Something's not working here, even though logic says it should.
Well, speaking only for myself, I never use bdist_wininst packages because they do no dependency management. If I release a package that has no external dependencies, I'll stick a bdist_msi or bdist_wininst if asked, but it's rarely asked for. From that I infer that most people use tools like buildout or pip to manage installation, even on Windows. Again, as a package consumer, I prefer one cross-platform package management method for Python packages, regardless of what operating system I happen to be working on at the time... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
2009/10/27 Chris Withers <chris@simplistix.co.uk>:
My point is the bdist_win installers work fine for both your use case, Paul, and Chris' use case, so there's no real conflict.
OK, and yet people still seem to argue for eggs rather than bdist_wininst, and projects provide eggs rather than bdist_wininst. Something's not working here, even though logic says it should.
Well, speaking only for myself, I never use bdist_wininst packages because they do no dependency management.
Maybe someone has confused me. I understood that easy_install can consume bdist_wininst format .exe files just as easily as .egg format, and easy_install will do your dependency management for you. So, if you provide a bdist_wininst .exe format file: 1. People who want integration with add/remove just run the .exe and are happy. 2. People who want dependency management use easy_install which transparently uses the .exe. If you provide egg format files, only group (2) are satisfied. And if you provide both, you just make extra work for yourself. Obviously, this only applies to Windows. By all means provide eggs, rpms, debs, or whatever suits your users for other platforms. If this scenario is wrong, then I've seriously misunderstood the situation (for which I apologise, but I don't think it's just me). I can't recall where I got this understanding from, but I *think* it was from the easy_install documentation. No time right now to check this, sorry.
If I release a package that has no external dependencies, I'll stick a bdist_msi or bdist_wininst if asked, but it's rarely asked for.
My point is that you could stick up the bdist_wininst at no cost to your existing users, and some benefit to certain other users. I suspect, based on my own behaviour, that there's a proportion of people who won't ask for a bdist_wininst, but will just not bother with your package (depending on lots of other things, not least of which is the level of competition).
From that I infer that most people use tools like buildout or pip to manage installation, even on Windows. Again, as a package consumer, I prefer one cross-platform package management method for Python packages, regardless of what operating system I happen to be working on at the time...
This inference probably depends on your user base and the area you're working in. My impression is that buildout/pip are favoured among web developers, but not so much people writing console scripts or standalone applications. I don't think anyone has a clear picture of the overall situation. My focus is very, very narrow here, and I acknowledge that freely. All I care is that Windows users without the tools, time or expertise to build non-trivial C extensions have access to workable binary builds. Consider the following list of packages: wxPython, PyQT, PyGame, cx_Oracle, pywin32, PIL, lxml, numpy, pyCrypto I can't build these myself, and if the projects only provide eggs, I can't use them (unless I use easy_install or something similar, and accept the implications of that). I had to move one project away from mod_python because there's no binary installer for Python 2.6. That's where my interest lies. For pure-python packages or packages with C code with no external dependencies, I have no problem building my own bdist_wininst so I don't care, any more than someone who exclusively uses python setup.py install would care. It's when I rely on someone else building binaries for me that their choice of binary format impacts me. Paul.
On Tue, Oct 27, 2009 at 7:16 PM, Paul Moore <p.f.moore@gmail.com> wrote:
Maybe someone has confused me. I understood that easy_install can consume bdist_wininst format .exe files just as easily as .egg format, and easy_install will do your dependency management for you.
That's exactly the case, you are not wrong. $ easy_install your_bdist_wininst_dist.exe will install it and process the dependencies from the install_requires option. And pip should be compatible soon too. That makes this format a perfect binary format for win32. Regards Tarek
On Tue, 27 Oct 2009 20:51:10 +0100, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
$ easy_install your_bdist_wininst_dist.exe
will install it and process the dependencies from the install_requires option.
And pip should be compatible soon too. That makes this format a perfect binary format for win32.
I'm not sure about that Tarek.. An .exe installer as a perfect binary format for python packages? Are you serious? That is the biggest security threat I can think of, asking python users to run unverified, unsigned, un-trusted executable files on their systems. No Tarek, don't dictate that untrusted .exe files present no security threat to windows users. In this case, there *couldn't be* a bigger security threat than untrusted, unverified executables. David
On 2009-10-27 18:41 PM, David Lyon wrote:
On Tue, 27 Oct 2009 20:51:10 +0100, Tarek Ziadé<ziade.tarek@gmail.com> wrote:
$ easy_install your_bdist_wininst_dist.exe
will install it and process the dependencies from the install_requires option.
And pip should be compatible soon too. That makes this format a perfect binary format for win32.
I'm not sure about that Tarek..
An .exe installer as a perfect binary format for python packages?
Are you serious?
That is the biggest security threat I can think of, asking python users to run unverified, unsigned, un-trusted executable files on their systems.
easy_install does not execute the executable. bdist_wininst installers are zip files concatenated with an executable header. easy_install just unzips the file as if it were a zipped egg and ignores the executable part. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
On Oct 27, 2009, at 7:41 PM, David Lyon wrote:
I'm not sure about that Tarek..
An .exe installer as a perfect binary format for python packages?
Are you serious?
That is the biggest security threat I can think of, asking python users to run unverified, unsigned, un-trusted executable files on their systems.
easy_install, pip, and indeed all of PyPI is basically a system for executing untrusted code, usually as a system administrator, straight off of what is effectively a wiki. If you're concerned about security and distutils, there is a _lot_ of work to do. There is no particular additional danger in executing a .exe rather than a setup.py.
easy_install does not execute the executable. bdist_wininst installers are zip files concatenated with an executable header. easy_install just unzips
On Tue, 27 Oct 2009 18:51:07 -0500, Robert Kern <robert.kern@gmail.com> wrote: the
file as if it were a zipped egg and ignores the executable part.
Ok - well thanks for the information and I learned something. A typical windows user, if they see an .exe installer, they would just install it by double-clicking on it. I don't know how many winodw users would automatically think to open a command shell and type the command line. In all seriousness, a bdist_wininst installer would be ok for installing a python application, but the support for applications in distutils for windows is incomplete. .egg files are perfect for windows. But the neccessary python support to make them work properly isn't complete either. What I am referring to here, is the shell extensions to the registry to make them load automatically. There isn't any inherent requirement in windows saying that setuptools need to be loaded for eggs to work. It would be so nice if we could double click on an egg file on pypi, download it, and install it automatically. I can't think of anything simpler. That would be proper native windows support. We would only have to add one script to distutils and a few registry settings for this could be accomplished. It could be progressively rolled out with new python versions.
From my perspective, PJE did a heck of a lot of good work. EGG files are a good idea. As long as they get unpacked and installed as any other python package on arrival.
David
On Tue, 27 Oct 2009 20:06:11 -0400, Glyph Lefkowitz
If you're concerned about security and distutils, there is a _lot_ of work to do. There is no particular additional danger in executing a .exe rather than a setup.py.
Come to think of it.. you're totally right.. :-( David
Chris Withers wrote:
From that I infer that most people use tools like buildout or pip to manage installation, even on Windows.
I am skeptical about this claim because up to recently, virtualenv did not even work correctly for windows when you needed to build a C extension (bug which has been fixed since then), and setuptools from 2 weeks ago could not even produce workable entry points for scripts on 64 bits windows (fixed in last setuptools 0.6.9c11). In the corporate world, msi rules (because of group policy and all), and generally, wininst are much more similar to what people expect on windows. I think it is also important to keep in mind that not everybody use the same workflow - I myself build a lot of non trivial python softwares on many platforms, and the whole virtualenv/buildout/pip workflows is very foreign to me, and do not fit at all how I want to build and deploy things. Data-free generalization on how people do things should be avoided for those discussions, and we should focus on the requirements and use-cases. David
Paul Moore wrote:
Maybe someone has confused me. I understood that easy_install can consume bdist_wininst format .exe files just as easily as .egg format, and easy_install will do your dependency management for you.
So, if you provide a bdist_wininst .exe format file:
1. People who want integration with add/remove just run the .exe and are happy.
But will people not be surprised that none of the dependencies are installed if they do this?
If you provide egg format files, only group (2) are satisfied. And if you provide both, you just make extra work for yourself.
It generally equates to typing a whole extra len('bdist_wininst') characters ;-)
suspect, based on my own behaviour, that there's a proportion of people who won't ask for a bdist_wininst, but will just not bother with your package (depending on lots of other things, not least of which is the level of competition).
They're welcome to go elsewhere if they can't be bothered to use what's available or ask for what they think they need.
wxPython, PyQT, PyGame, cx_Oracle, pywin32, PIL, lxml, numpy, pyCrypto
I can't build these myself, and if the projects only provide eggs, I can't use them (unless I use easy_install or something similar, and accept the implications of that).
..or you just place the egg on your python path... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
David Cournapeau wrote:
Chris Withers wrote:
From that I infer that most people use tools like buildout or pip to manage installation, even on Windows.
I am skeptical about this claim because up to recently, virtualenv did not even work correctly for windows when you needed to build a C extension (bug which has been fixed since then),
I don't use virtualenv and I didn't mention it for that reason. Buildout has behaved just fine in all my use of it...
and setuptools from 2 weeks ago could not even produce workable entry points for scripts on 64 bits windows (fixed in last setuptools 0.6.9c11).
"even" seems to imply this is a basic requirement. I wonder how many people are rolling out python apps on Win64? I'm certainly not...
I think it is also important to keep in mind that not everybody use the same workflow - I myself build a lot of non trivial python softwares on many platforms, and the whole virtualenv/buildout/pip workflows is very foreign to me, and do not fit at all how I want to build and deploy things.
How do you build and deploy things? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
2009/10/28 Chris Withers <chris@simplistix.co.uk>:
Paul Moore wrote:
Maybe someone has confused me. I understood that easy_install can consume bdist_wininst format .exe files just as easily as .egg format, and easy_install will do your dependency management for you.
So, if you provide a bdist_wininst .exe format file:
1. People who want integration with add/remove just run the .exe and are happy.
But will people not be surprised that none of the dependencies are installed if they do this?
No. That's the nature of Windows installers. I accept that your view is that automatic installation of dependencies is a key feature - please accept in turn that I prefer to manage dependencies manually.
If you provide egg format files, only group (2) are satisfied. And if you provide both, you just make extra work for yourself.
It generally equates to typing a whole extra len('bdist_wininst') characters ;-)
Hmm, OK, fair point :-) It's a pity more people don't just do it then...
suspect, based on my own behaviour, that there's a proportion of people who won't ask for a bdist_wininst, but will just not bother with your package (depending on lots of other things, not least of which is the level of competition).
They're welcome to go elsewhere if they can't be bothered to use what's available or ask for what they think they need.
True. Certainly in the case of "ask for what they think they need". I'm less convinced that "use what's available" is always appropriate (see below).
wxPython, PyQT, PyGame, cx_Oracle, pywin32, PIL, lxml, numpy, pyCrypto
I can't build these myself, and if the projects only provide eggs, I can't use them (unless I use easy_install or something similar, and accept the implications of that).
..or you just place the egg on your python path...
My personal issue with this is that I lose the one listing of what I have installed from add/remove programs, and I lose the one-click uninstall of add/remove. This adds to my management overhead for my Python installation. Lots of this is personal preference. No argument there. Paul.
Chris Withers wrote:
"even" seems to imply this is a basic requirement. I wonder how many people are rolling out python apps on Win64? I'm certainly not...
Even was maybe a bit strong. Concerning win64, the first release of numpy we did for win64 has been downloaded 3000 times, even though it is experimental and not working. Obviously, this is linked to 'our' community: in scientific software, 64 bits arch has mattered for a long time, and a lot of people use windows, for better or worse. In numpy/scipy, even though the vast majority of dev are linux/mac os x users, windows installers are by far the most downloaded (the numbers are obviously skewed since many people can build from svn on linux/mac os x, whereas very few people will do so on windows).
I think it is also important to keep in mind that not everybody use the same workflow - I myself build a lot of non trivial python softwares on many platforms, and the whole virtualenv/buildout/pip workflows is very foreign to me, and do not fit at all how I want to build and deploy things.
How do you build and deploy things?
To answer your question: I use distutils, scons, and native OS package tools. Sandboxing at build time is done through chrooting (and some hacked wine setup for windows). I think the main issue is that we don't mean the same thing by "deploy things". For web developers, it seems that deploying mean repeat an exact environment with all its dependencies on a similar machine (or set of machines), often by the same person (or at least a technically competent person). In that scenario, if there is a problem (wrong pythonpath, double entry from an egg, etc...), you can easily fix it, and being able to use latest version of some libraries + developer convenience is the key. By deploying, I mean being able to give a single file which people can install by themselves, people who are often ignorant about PYTHONPATH, import mechanisms, what a .pth file is. In that scenario, altering the environment in a non workable state is a total failure. Developer convenience is a problem in many cases (concrete case: using the VCS to get which files to put in a distribution). This article is about ruby, but totally nails it IMHO: http://www.madstop.com/ruby/ruby_has_a_distribution_problem.html Given that web dev are the ones doing the work, it is natural and certainly legitimate that current distutils-related work is focused on this workflow. But this workflow is not always applicable to more 'traditional' deployments scenario, and it often does make life more difficult. It certainly did for me. David
A typical windows user, if they see an .exe installer, they would just install it by double-clicking on it. I don't know how many winodw users would automatically think to open a command shell and type the command line.
None in fact and it is a make or break requirement in so many companies that if anyone is thinking of dropping bdist_wininst should rethink of it. If it goes away I have to discourage use of python because in no case a language specific way to install things will ever replace the platform standardised way. I'll give you an anecdotal example: I worked in a company developing systems for an extremely larger one. Our software needn't any installer: just unzip the thing and start to using it. Simple? You bet it cannot be any easier than that..... One of the contractual requirements was about providing an "installer" for windows: no matter how easy it was to deploy the thing. This makes sense because their way to manage systems has been standardised in that way (the windows way, like it or hate it): no matter how clever our system was, it was extremely expensive to them doing it in "our" way.
In all seriousness, a bdist_wininst installer would be ok for installing a python application, but the support for applications in distutils for windows is incomplete.
It is just enough to allow people to work: dependencies are handled manually and this goes in a project requirement document. An exception on a missing module it is just fine to give a hint about what is missing and trigger a user action.
.egg files are perfect for windows. But the neccessary python support to make them work properly isn't complete either. What I am referring to here, is the shell extensions to the registry to make them load automatically. It would be so nice if we could double click on an egg file on pypi, download it, and install it automatically. I can't think of anything simpler. That would be proper native windows support.
I disagree here and reinventing a package/dependencies manager to compete with cpan and rpm/dpkg/yum/zypper/synaptic/windows installers ... thanks no I don't feel any need for it. Ever heard of SuSE one click installer? It does exactly this for rpms. In windows clicking on a web link with an *.exe trigger the action you're just asking for already: no need to support another language-specific layer.
We would only have to add one script to distutils and a few registry settings for this could be accomplished. It could be progressively rolled out with new python versions.
Sure and imagine a company stretching across three continents with hundred developers and many more users and slightly different settings/python versions: do you think anything like this will ever work in real life? How does anybody going to handle the chain of trust (yes sooner or later somebody will ask THIS question)? Again my solution will be easy: drop python off the list of language to develop with.
From my perspective, PJE did a heck of a lot of good work. EGG files
are a good idea. As long as they get unpacked and installed as any other python package on arrival.
Excellent last point aiming straight to the problem: I cannot agree more. Regards, Antonio
Antonio, I completely get what you're saying. Anyway, we shouldn't be considering dropping bdist_wininst installers for windows unless we actually have something viable to replace them with. As much as I don't think they are the ideal solution, I'd hate to think about having to to package installation on windows without them. So from a pure practical point.. I totally agree with what you're saying.. David On Thu, 29 Oct 2009 07:44:03 +0000, "A. Cavallo" <a.cavallo@cavallinux.eu> wrote:
None in fact and it is a make or break requirement in so many companies that if anyone is thinking of dropping bdist_wininst should rethink of it.
If it goes away I have to discourage use of python because in no case a language specific way to install things will ever replace the platform standardised way.
I'll give you an anecdotal example: I worked in a company developing systems for an extremely larger one. Our software needn't any installer: just unzip
the thing and start to using it. Simple? You bet it cannot be any easier than that..... One of the contractual requirements was about providing an "installer" for windows: no matter how easy it was to deploy the thing.
This makes sense because their way to manage systems has been standardised in that way (the windows way, like it or hate it): no matter how clever our system was, it was extremely expensive to them doing it in "our" way.
In all seriousness, a bdist_wininst installer would be ok for installing a python application, but the support for applications in distutils for windows is incomplete.
It is just enough to allow people to work: dependencies are handled manually and this goes in a project requirement document. An exception on a missing module it is just fine to give a hint about what is missing and trigger a user action.
.egg files are perfect for windows. But the neccessary python support to make them work properly isn't complete either. What I am referring to here, is the shell extensions to the registry to make them load automatically. It would be so nice if we could double click on an egg file on pypi, download it, and install it automatically. I can't think of anything simpler. That would be proper native windows support.
I disagree here and reinventing a package/dependencies manager to compete with cpan and rpm/dpkg/yum/zypper/synaptic/windows installers ... thanks no I don't feel any need for it.
Ever heard of SuSE one click installer? It does exactly this for rpms. In windows clicking on a web link with an *.exe trigger the action you're just asking for already: no need to support another language-specific layer.
We would only have to add one script to distutils and a few registry settings for this could be accomplished. It could be progressively rolled out with new python versions.
Sure and imagine a company stretching across three continents with hundred developers and many more users and slightly different settings/python versions: do you think anything like this will ever work in real life? How does anybody going to handle the chain of trust (yes sooner or later somebody will ask THIS question)? Again my solution will be easy: drop python off the list of language to develop with.
From my perspective, PJE did a heck of a lot of good work. EGG files
are a good idea. As long as they get unpacked and installed as any other python package on arrival.
Excellent last point aiming straight to the problem: I cannot agree more.
Regards, Antonio _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
David Cournapeau wrote:
Chris Withers wrote:
"even" seems to imply this is a basic requirement. I wonder how many people are rolling out python apps on Win64? I'm certainly not...
Even was maybe a bit strong. Concerning win64, the first release of numpy we did for win64 has been downloaded 3000 times,
By what? I'm always sceptical about the download figures on PyPI, and other places. I wonder how many bots are sucking down every .exe or .msi they can find to check for viruses or the like?
I think it is also important to keep in mind that not everybody use the same workflow - I myself build a lot of non trivial python softwares on many platforms, and the whole virtualenv/buildout/pip workflows is very foreign to me, and do not fit at all how I want to build and deploy things. How do you build and deploy things?
To answer your question: I use distutils, scons, and native OS package tools. Sandboxing at build time is done through chrooting (and some hacked wine setup for windows).
You should probably look at buildout, you may find your like getting a lot easier. Yes, the docs suck. I even promised to do something about that back at PyCon, sadly time has not been kind to my promises...
I think the main issue is that we don't mean the same thing by "deploy things". For web developers, it seems that deploying mean repeat an exact environment with all its dependencies on a similar machine (or set of machines), often by the same person (or at least a technically competent person). In that scenario, if there is a problem (wrong pythonpath, double entry from an egg, etc...), you can easily fix it, and being able to use latest version of some libraries + developer convenience is the key.
By deploying, I mean being able to give a single file which people can install by themselves, people who are often ignorant about PYTHONPATH, import mechanisms, what a .pth file is.
Well, buildout.cfg is exacltly that file for me, for both cases ;-) Chris - for the record: .pth files are evil and should be punished... -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
Chris Withers wrote:
David Cournapeau wrote:
Chris Withers wrote:
"even" seems to imply this is a basic requirement. I wonder how many people are rolling out python apps on Win64? I'm certainly not...
Even was maybe a bit strong. Concerning win64, the first release of numpy we did for win64 has been downloaded 3000 times,
By what? I'm always sceptical about the download figures on PyPI, and other places. I wonder how many bots are sucking down every .exe or .msi they can find to check for viruses or the like?
Sourceforge numbers. Besides, I can assure you that there is a lot of interest in windows 64 for scientific computing (in terms of people willing to pay).
You should probably look at buildout, you may find your like getting a lot easier. Yes, the docs suck. I even promised to do something about that back at PyCon, sadly time has not been kind to my promises...
The docs are indeed less than ideal - everytime I tried to read something about buildout, I could not even understand what it was about :) But even though, I doubt buildout is what I am after: chroot-like sandboxing for example, is crucial (and for what it worths, it is a standard practice in the unix deployment world). What I use is kind of besides the point, though. What matters is to have a standard core on top of which people can build their tools: a core in the stdlib, and buildout-based, scons-based, whatever-based tools on top. If you have an API to retrieve where things are put, how they are built and so on, everyone can be happy with the tool he prefers.
Well, buildout.cfg is exacltly that file for me, for both cases ;-)
I may not have been clear: by single file, I mean an installer, which needs to work offline, and is as integrated as possible in the user system. This mean msi, dmg, rpm, etc... A buildout.cfg can't possible be further from what I want in those cases. David
David Cournapeau a écrit :
But even though, I doubt buildout is what I am after: chroot-like sandboxing for example, is crucial (and for what it worths, it is a standard practice in the unix deployment world).
What I use is kind of besides the point, though. What matters is to have a standard core on top of which people can build their tools: a core in the stdlib, and buildout-based, scons-based, whatever-based tools on top. If you have an API to retrieve where things are put, how they are built and so on, everyone can be happy with the tool he prefers.
For your chrooting/API purposes, you can have a look to minitage, it's all his goals. But even buildout can provide you a basic isolation layer.
Well, buildout.cfg is exacltly that file for me, for both cases ;-)
I may not have been clear: by single file, I mean an installer, which needs to work offline, and is as integrated as possible in the user system. This mean msi, dmg, rpm, etc... A buildout.cfg can't possible be further from what I want in those cases.
What about using both ? If it is for releasing final softwares for "non-developers", just archive the resulting buildout infra. from the targeted host and make a package from there. Many people there do in that way (rpm, or so). As a developer, you ll have the flexibility and reliability of buildout, and you ll assure to your users to use their lovely system package manager which install a locally tarball of you whole buildout based packaged software. No need to be online there. On the other hand, buildout can work totally offline if you have all your requirements fulfilled. -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF
participants (9)
-
A. Cavallo -
Chris Withers -
David Cournapeau -
David Lyon -
Glyph Lefkowitz -
kiorky -
Paul Moore -
Robert Kern -
Tarek Ziadé