On Tue, Dec 29, 2009 at 3:49 AM, Dag Sverre Seljebotn wrote:

Do you here mean automatic generation of Ubuntu debs, Debian debs, Windows MSI installer, Windows EXE installer, and so on? (If so then great!)

Yes (although this is not yet implemented). In particular on windows, I want to implement a scheme so that you can convert from eggs to .exe and vice et versa, so people can still install as exe (or msi), even though the method would default to eggs.

If this is the goal, I wonder if one looks outside of Python-land one might find something that already does this -- there's a lot of different package format, "Linux meta-distributions", "install everywhere packages" and so on.

Yes, there are things like 0install or autopackage. I think those are deemed to fail, as long as it is not supported thoroughly by the distribution. Instead, my goal here is much simpler: producing rpm/deb. It does not solve every issue (install by non root, multiple // versions), but one has to be realistic :) I think automatically built rpm/deb, easy integration with native method can solve a lot of issues already.

 - Currently I'm making a Sage SPKG for CHOLMOD. This essentially gets the job done by not bothering about the problem, not even using the OS-installed Python.

Something that would spit out both Sage SPKGs, Ubuntu debs, Windows installers, both with Python code and C/Fortran code or a mix (and put both in the place preferred by the system in question), seems ideal. Of course one would still need to make sure that the code builds properly everywhere, but just solving the distribution part of this would be a huge step ahead.

On windows, this issue may be solved using eggs: enstaller has a feature where dll put in a special location of an egg are installed in python such as they are found by the OS loader. One could have mechanisms based on $ORIGIN + rpath on linux to solve this issue for local installs on Linux, etc... But again, one has to be realistic on the goals. With toydist, I want to remove all the pile of magic, hacks built on top of distutils so that people can again hack their own solutions, as it should have been from the start (that's a big plus of python in general). It won't magically solve every issue out there, but it would hopefully help people to make their own. Bundling solutions like SAGE, EPD, etc... are still the most robust ways to deal with those issues in general, and I do not intended to replace those.

What I'm saying is that this is a software distribution problem in general, and I'm afraid that Python-specific solutions are too narrow.

Distribution is a hard problem. Instead of pushing a very narrow (and mostly ill-funded) view of how people should do things like distutils/setuptools/pip/buildout do, I want people to be able to be able to build their own solutions. No more "use this magic stick v 4.0.3.3.14svn1234, trust me it work you don't have to understand" which is too prevalant with those tools, which has always felt deeply unpythonic to me. David

hello again, On Tue, Dec 29, 2009 at 2:22 PM, David Cournapeau wrote:

On Tue, Dec 29, 2009 at 10:27 PM, René Dudfield wrote:

...

Hi,

In the toydist proposal/release notes, I would address 'what does toydist do better' more explicitly.

**** A big problem for science users is that numpy does not work with pypi + (easy_install, buildout or pip) and python 2.6. ****

Working with the rest of the python community as much as possible is likely a good goal.

Yes, but it is hopeless. Most of what is being discussed on distutils-sig is useless for us, and what matters is ignored at best. I think most people on distutils-sig are misguided, and I don't think the community is representative of people concerned with packaging anyway - most of the participants seem to be around web development, and are mostly dismissive of other's concerns (OS packagers, etc...).

Sitting down with Tarek(who is one of the current distutils maintainers) in Berlin we had a little discussion about packaging over pizza and beer... and he was quite mindful of OS packagers problems and issues. He was also interested to hear about game developers issues with packaging (which are different again to scientific users... but similar in many ways). However these systems were developed by the zope/plone/web crowd, so they are naturally going to be thinking a lot about zope/plone/web issues. Debian, and ubuntu packages for them are mostly useless because of the age. Waiting a couple of years for your package to be released is just not an option (waiting even an hour for bug fixes is sometimes not an option). Also isolation of packages is needed for machines that have 100s of different applications running, written by different people, each with dozens of packages used by each application. Tools like checkinstall and stdeb ( http://pypi.python.org/pypi/stdeb/ ) can help with older style packaging systems like deb/rpm. I think perhaps if toydist included something like stdeb as not an extension to distutils, but a standalone tool (like toydist) there would be less problems with it. One thing the various zope related communities do is make sure all the relevant and needed packages are built/tested by their compile farms. This makes pypi work for them a lot better than a non-coordinated effort does. There are also lots of people trying out new versions all of the time.

I want to note that I am not starting this out of thin air - I know most of distutils code very well, I have been the mostly sole maintainer of numpy.distutils for 2 years now. I have written extensive distutils extensions, in particular numscons which is able to fully build numpy, scipy and matplotlib on every platform that matters.

Simply put, distutils code is horrible (this is an objective fact) and  flawed beyond repair (this is more controversial). IMHO, it has almost no useful feature, except being standard.

yes, I have also battled with distutils over the years. However it is simpler than autotools (for me... maybe distutils has perverted my fragile mind), and works on more platforms for python than any other current system. It is much worse for C/C++ modules though. It needs dependency, and configuration tools for it to work better (like what many C/C++ projects hack into distutils themselves). Monkey patching, and extensions are especially a problem... as is the horrible code quality of distutils by modern standards. However distutils has had more tests and testing systems added, so that refactoring/cleaning up of distutils can happen more so.

If you want a more detailed explanation of why I think distutils and all tools on top are deeply flawed, you can look here:

http://cournape.wordpress.com/2009/04/01/python-packaging-a-few-observations...

I agree with many things in that post. Except your conclusion on multiple versions of packages in isolation. Package isolation is like processes, and package sharing is like threads - and threads are evil! Leave my python site-packages directory alone I say... especially don't let setuptools infect it :) Many people currently find the multi versions of packages in isolation approach works well for them - so for some use cases the tools are working wonderfully.

...

numpy used to work with buildout in python2.5, but not with 2.6. buildout lets other team members get up to speed with a project by running one command.  It installs things in the local directory, not system wide.  So you can have different dependencies per project.

I don't think it is a very useful feature, honestly. It seems to me that they created a huge infrastructure to split packages into tiny pieces, and then try to get them back together, imaganing that multiple installed versions is a replacement for backward compatibility. Anyone with extensive packaging experience knows that's a deeply flawed model in general.

Science is supposed to allow repeatability. Without the same versions of packages, repeating experiments is harder. This is a big problem in science that multiple versions of packages in _isolation_ can help get to a solution to the repeatability problem. Just pick some random paper and try to reproduce their results. It's generally very hard, unless the software is quite well packaged. Especially for graphics related papers, there are often many different types of environments, so setting up the environments to try out their techniques, and verify results quickly is difficult. Multiple versions are not a replacement for backwards compatibility, just a way to avoid the problem in the short term to avoid being blocked. If a new package version breaks your app, then you can either pin it to an old version, fix your app, or fix the package. It is also not a replacement for building on stable high quality components, but helps you work with less stable, and less high quality components - at a much faster rate of change, with a much larger dependency list.

...

Plenty of good work is going on with python packaging.

That's the opposite of my experience. What I care about is:  - tools which are hackable and easily extensible  - robust install/uninstall  - real, DAG-based build system  - explicit and repeatability

None of this is supported by the tools, and the current directions go even further away. When I have to explain at length why the command-based design of distutils is a nightmare to work with, I don't feel very confident that the current maintainers are aware of the issues, for example. It shows that they never had to extend distutils much.

All agreed! I'd add to the list parallel builds/tests (make -j 16), and outputting to native build systems. eg, xcode, msvc projects, and makefiles. It would interesting to know your thoughts on buildout recipes ( see creating recipes http://www.buildout.org/docs/recipe.html ). They seem to work better from my perspective. However, that is probably because of isolation. The recipe are only used by those projects that require them. So the chance of them interacting are lower, as they are not installed in the main python. How will you handle toydist extensions so that multiple extensions do not have problems with each other? I don't think this is possible without isolation, and even then it's still a problem. Note, the section in the distutils docs on creating command extensions is only around three paragraphs. There is also no central place to go looking for extra commands (that I know of). Or a place to document or share each others command extensions. Many of the methods for extending distutils are not very well documented either. For example, 'how do I you change compiler command line arguments for certain source files?' Basic things like that are possible with disutils, but not documented (very well).

...

There are build farms for windows packages and OSX uploaded to pypi. Start uploading pre releases to pypi, and you get these for free (once you make numpy compile out of the box on those compile farms).  There are compile farms for other OSes too... like ubuntu/debian, macports etc.  Some distributions even automatically download, compile and package new releases once they spot a new file on your ftp/web site.

I am familiar with some of those systems (PPA and opensuse build service in particular). One of the goal of my proposal is to make it easier to interoperate with those tools.

yeah, cool.

I think Pypi is mostly useless. The lack of enforced metadata is a big no-no IMHO. The fact that Pypi is miles beyond CRAN for example is quite significant. I want CRAN for scientific python, and I don't see Pypi becoming it in the near future.

The point of having our own Pypi-like server is that we could do the following:  - enforcing metadata  - making it easy to extend the service to support our needs

Yeah, cool. Many other projects have their own servers too. pygame.org, plone, etc etc, which meet their own needs. Patches are accepted for pypi btw. What type of enforcements of meta data, and how would they help? I imagine this could be done in a number of ways to pypi. - a distutils command extension that people could use. - change pypi source code. - check the metadata for certain packages, then email their authors telling them about issues.

...

pypm:  http://pypm.activestate.com/list-n.html#numpy

It is interesting to note that one of the maintainer of pypm has recently quitted the discussion about Pypi, most likely out of frustration from the other participants.

yeah, big mailing list discussions hardly ever help I think :) oops, this is turning into one.

...

Documentation projects are being worked on to document, give tutorials and make python packaging be easier all round.  As witnessed by 20 or so releases on pypi every day(and growing), lots of people are using the python packaging tools successfully.

This does not mean much IMO. Uploading on Pypi is almost required to use virtualenv, buildout, etc.. An interesting metric is not how many packages are uploaded, but how much it is used outside developers.

Yeah, it only means that there are lots of developers able to use the packaging system to put their own packages up there. However there are over 500 science related packages on there now - which is pretty cool. A way to measure packages being used would be by downloads, and by which packages depend on which other packages. I think the science ones would be reused lower than normal, since a much higher percentage are C/C++ based, and are likely to be more fragile packages.

...

I'm not sure making a separate build tool is a good idea.  I think going with the rest of the python community, and improving the tools there is a better idea.

It has been tried, and IMHO has been proved to have failed. You can look at the recent discussion (the one started by Guido in particular).

I don't think 500+ science related packages is a total failure really.

...

pps. some notes on toydist itself. - toydist convert is cool for people converting a setup.py .  This means that most people can try out toydist right away.  but what does it gain these people who convert their setup.py files?

Not much ATM, except that it is easier to write a toysetup.info compared to setup.py IMO, and that it supports a simple way to include data files (something which is currently *impossible* to do without writing your own distutils extensions). It has also the ability to build eggs without using setuptools (I consider not using setuptools a feature, given the too many failure modes of this package).

yeah, I always make setuptools not used in my packages by default. However I use command line arguments to use the features of setuptools required (eggs, bdist_mpkg etc etc). Having a tool to create eggs without setuptools would be great in itself. Definitely list this in the feature list :)

The main goals though are to make it easier to build your own tools on top of if, and to integrate with real build systems.

yeah, cool.

...

- a toydist convert that generates a setup.py file might be cool :)

toydist started like this, actually: you would write a setup.py file which loads the package from toysetup.info, and can be converted to a dict argument to distutils.core.setup. I have not updated it recently, but that's definitely on the TODO list for a first alpha, as it would enable people to benefit from the format, with 100 % backward compatibility with distutils.

yeah, cool. That would let you develop things incrementally too, and still have toydist be useful for the whole development period until it catches up with the features of distutils needed.

...

- arbitrary code execution happens when building or testing with toydist.

You are right for testing, but wrong for building. As long as the build is entirely driven by toysetup.info, you only have to trust toydist (which is not safe ATM, but that's an implementation detail), and your build tools of course.

If you execute build tools on arbitrary code, then arbitrary code execution is easy for someone who wants to do bad things. Trust and secondarily sandboxing are the best ways to solve these problems imho.

Obviously, if you have a package which uses an external build tool on top of toysetup.info (as will be required for numpy itself for example), all bets are off. But I think that's a tiny fraction of the interesting packages for scientific computing.

yeah, currently 1/5th of science packages use C/C++/fortran/cython etc (see http://pypi.python.org/pypi?:action=browse&c=40 110/458 on that page ). There seems to be a lot more using C/C++ compared to other types of pakages on there (eg zope3 packages list 0 out of 900 packages using C/C++). So the hight number of C/C++ science related packages on pypi demonstrate that better C/C++ tools for scientific packages is a big need. Especially getting compile/testing farms for all these packages. Getting compile farms is a big need compared to python packages - since C/C++ is MUCH harder to write/test in a portable way. I would say it is close to impossible to get code to work without quite good knowledge on multiple platforms without errors. There are many times with pygame development that I make changes on an osx, windows or linux box, commit the change, then wait for the compile/tests to run on the build farm ( http://thorbrian.com/pygame/builds.php ). Releasing packages otherwise makes the process *heaps* longer... and many times I still get errors on different platforms, despite many years of multi platform coding.

Sandboxing is particularly an issue on windows - I don't know a good solution for windows sandboxing, outside of full vms, which are heavy-weights.

yeah, VMs are the way to go. If only to make the copies a fresh install each time. However I think automated distributed building, and trust are more useful. ie, only build those packages where you trust the authors, and let anyone download, build and then post their build/test results. MS have given out copies of windows to some people to set up VMs for building to different members of the python community in the past. By automated distributed building, I mean what happens with mailing lists usually. Where people post their test results when they have a problem. Except in a more automated manner. Adding a 'Do you want to upload your build/test results?' at the end of a setup.py for subversion builds would give you dozens or hundreds of test results daily from all sorts of machines. Making it easy for people to set up package builders which also upload their packages somewhere gives you distributed package building, in a fairly safe automated manner. (more details here: http://renesd.blogspot.com/2009/09/python-build-bots-down-maybe-they-need.ht... )

...

- it should be possible to build this toydist functionality as a distutils/distribute/buildout extension.

No, it cannot, at least as far as distutils/distribute are concerned (I know nothing about buildout). Extending distutils is horrible, and fragile in general. Even autotools with its mix of generated sh scripts through m4 and perl is a breeze compared to distutils.

...

- extending toydist?  How are extensions made?  there are 175 buildout packages which extend buildout, and many that extend distutils/setuptools - so extension of build tools in a necessary thing.

See my answer earlier about interoperation with build tools.

I'm still not clear on how toydist will be extended. I am however, a lot clearer about its goals. cheers,

On Wed, Dec 30, 2009 at 9:26 AM, Ravi wrote:

On Wednesday 30 December 2009 06:15:45 René Dudfield wrote:

...

I agree with many things in that post.  Except your conclusion on multiple versions of packages in isolation.  Package isolation is like processes, and package sharing is like threads - and threads are evil!

I don't think this is an appropriate analogy, and hyperbolic statements like "threads are evil!" are unlikely to persuade a scientific audience.

You have stated this several times, but is there any evidence that this is the desire of the majority of users? In the scientific community, interactive experimentation is critical and users are typically not seasoned systems administrators. For such users, almost all packages installed after installing python itself are packages they use. In particular, all I want to do is to use apt/yum to get the packages (or ask my sysadmin, who rightfully has no interest in learning the intricacies of python package installation, to do so) and continue with my work. "Packages-in-isolation" is for people whose job is to run server farms, not interactive experimenters.

I agree.

...

 Leave my python site-packages directory alone I say... especially don't let setuptools infect it :)

There are already mechanisms in place for this. "python setup.py install --user" or "easy_install --prefix=/usr/local" for example. Darren

René Dudfield wrote:

On Wed, Dec 30, 2009 at 2:26 PM, Ravi wrote:

...

On Wednesday 30 December 2009 06:15:45 René Dudfield wrote:

...

I agree with many things in that post. Except your conclusion on multiple versions of packages in isolation. Package isolation is like processes, and package sharing is like threads - and threads are evil! You have stated this several times, but is there any evidence that this is the desire of the majority of users? In the scientific community, interactive experimentation is critical and users are typically not seasoned systems administrators. For such users, almost all packages installed after installing python itself are packages they use. In particular, all I want to do is to use apt/yum to get the packages (or ask my sysadmin, who rightfully has no interest in learning the intricacies of python package installation, to do so) and continue with my work. "Packages-in-isolation" is for people whose job is to run server farms, not interactive experimenters.

500+ packages on pypi. Provide a counter point, otherwise the evidence is against your position - overwhelmingly.

?!? Wouldn't you need to measure the number of downloads (and also figure out something else to measure that relative to)? Uploading something to PyPI is easy enough to do and probably done by default by a lot of package authors -- that doesn't mean that it is the main distribution method. -- Dag Sverre

On Wed, Dec 30, 2009 at 12:47, René Dudfield wrote:

On Wed, Dec 30, 2009 at 2:26 PM, Ravi wrote:

...

On Wednesday 30 December 2009 06:15:45 René Dudfield wrote:

...

I agree with many things in that post.  Except your conclusion on multiple versions of packages in isolation.  Package isolation is like processes, and package sharing is like threads - and threads are evil!

You have stated this several times, but is there any evidence that this is the desire of the majority of users? In the scientific community, interactive experimentation is critical and users are typically not seasoned systems administrators. For such users, almost all packages installed after installing python itself are packages they use. In particular, all I want to do is to use apt/yum to get the packages (or ask my sysadmin, who rightfully has no interest in learning the intricacies of python package installation, to do so) and continue with my work. "Packages-in-isolation" is for people whose job is to run server farms, not interactive experimenters.

500+ packages on pypi.   Provide a counter point, otherwise the evidence is against your position - overwhelmingly.

Linux distributions, which are much, much more popular than any collection of packages on PyPI you might care to name. Isolated environments have their uses, but they are the exception, not the rule. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

On Wed, Dec 30, 2009 at 11:10 AM, René Dudfield wrote:

On Wed, Dec 30, 2009 at 7:08 PM, Robert Kern wrote:

...

On Wed, Dec 30, 2009 at 12:47, René Dudfield wrote:

...

On Wed, Dec 30, 2009 at 2:26 PM, Ravi wrote:

...

On Wednesday 30 December 2009 06:15:45 René Dudfield wrote:

...

I agree with many things in that post.  Except your conclusion on multiple versions of packages in isolation.  Package isolation is like processes, and package sharing is like threads - and threads are evil!

You have stated this several times, but is there any evidence that this is the desire of the majority of users? In the scientific community, interactive experimentation is critical and users are typically not seasoned systems administrators. For such users, almost all packages installed after installing python itself are packages they use. In particular, all I want to do is to use apt/yum to get the packages (or ask my sysadmin, who rightfully has no interest in learning the intricacies of python package installation, to do so) and continue with my work. "Packages-in-isolation" is for people whose job is to run server farms, not interactive experimenters.

500+ packages on pypi.   Provide a counter point, otherwise the evidence is against your position - overwhelmingly.

Linux distributions, which are much, much more popular than any collection of packages on PyPI you might care to name. Isolated environments have their uses, but they are the exception, not the rule.

wrong.  pypi has way more python packages than any linux distribution.  8500+ listed, compared to how many in debian?

Debian has over 30k packages. But I think he was talking about popularity, not the number of packages.

On Wed, Dec 30, 2009 at 13:10, René Dudfield wrote:

On Wed, Dec 30, 2009 at 7:08 PM, Robert Kern wrote:

...

On Wed, Dec 30, 2009 at 12:47, René Dudfield wrote:

...

...

500+ packages on pypi.   Provide a counter point, otherwise the evidence is against your position - overwhelmingly.

Linux distributions, which are much, much more popular than any collection of packages on PyPI you might care to name. Isolated environments have their uses, but they are the exception, not the rule.

wrong.  pypi has way more python packages than any linux distribution.  8500+ listed, compared to how many in debian?

I said "more popular". As in "more users", not "more packages". But if you insist, Debian has ~30000 or so, depending on the architecture and release and how you count. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

David Cournapeau wrote:

You really should not try to make the point that Pypi is working for the scipy community:

I think the evidence supports that pypi is useful, and therefor better than nothing -- which in no way means it couldn't be much better. My personal experience is that I always try: easy_install whatever first, and when it works, I'm happy and a bit surprised. It virtually never works for more complex packages, particularly on OS-X: - PIL - scipy - matplotlib - netcdf4 - gdal I don't now if wxPython is on PyPi, I've never even tried. many of these fail because of other non-python dependencies. So yes -- we really could use something better! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On Wed, Dec 30, 2009 at 8:15 PM, René Dudfield wrote:

Sitting down with Tarek(who is one of the current distutils maintainers) in Berlin we had a little discussion about packaging over pizza and beer... and he was quite mindful of OS packagers problems and issues.

This has been said many times on distutils-sig, but no concrete action has ever been taken in that direction. For example, toydist already supports the FHS better than distutils, and is more flexible. I have tried several times to explain why this matters on distutils-sig, but you then have the peanuts gallery interfering with unrelated nonsense (like it would break windows, as if it could not be implemented independently). Also, retrofitting support for --*dir in distutils would be *very* difficult, unless you are ready to break backward compatibility (there are 6 ways to install data files, and each of them has some corner cases, for example - it is a real pain to support this correctly in the convert command of toydist, and you simply cannot recover missing information to comply with the FHS in every case).

However these systems were developed by the zope/plone/web crowd, so they are naturally going to be thinking a lot about zope/plone/web issues.

Agreed - it is natural that they care about their problems first, that's how it works in open source. What I find difficult is when our concern are constantly dismissed by people who have no clue about our issues - and later claim we are not cooperative.

 Debian, and ubuntu packages for them are mostly useless because of the age.

That's where the build farm enters. This is known issue, that's why the build service or PPA exist in the first place.

 I think perhaps if toydist included something like stdeb as not an extension to distutils, but a standalone tool (like toydist) there would be less problems with it.

That's pretty much how I intend to do things. Currently, in toydist, you can do something like: from toydist.core import PackageDescription pkg = PackageDescription.from_file("toysetup.info") # pkg now gives you access to metadata, as well as extensions, python modules, etc... I think this gives almost everything that is needed to implement a sdist_dsc command. Contrary to the Distribution class in distutils, this class would not need to be subclassed/monkey-patched by extensions, as it only cares about the description, and is 100 % uncoupled from the build part.

yes, I have also battled with distutils over the years.  However it is simpler than autotools (for me... maybe distutils has perverted my fragile mind), and works on more platforms for python than any other current system.

Autotools certainly works on more platforms (windows notwhistanding), if only because python itself is built with autoconf. Distutils simplicity is a trap: it is simpler only if you restrict to what distutils gives you. Don't get me wrong, autotools are horrible, but I have never encountered cases where I had to spend hours to do trivial tasks, as has been the case with distutils. Numpy build system would be much, much easier to implement through autotools, and would be much more reliable.

 However distutils has had more tests and testing systems added, so that refactoring/cleaning up of distutils can happen more so.

You can't refactor distutils without breaking backward compatibility, because distutils has no API. The whole implementation is the API. That's one of the fundamental disagreement I and other scipy dev have with current contributors on distutils-sig: the starting point (distutils) and the goal are so far away from each other that getting there step by step is hopeless.

I agree with many things in that post.  Except your conclusion on multiple versions of packages in isolation.  Package isolation is like processes, and package sharing is like threads - and threads are evil!

I don't find the comparison very helpful (for once, you can share data between processes, whereas virtualenv cannot see each other AFAIK).

Science is supposed to allow repeatability.  Without the same versions of packages, repeating experiments is harder.  This is a big problem in science that multiple versions of packages in _isolation_ can help get to a solution to the repeatability problem.

I don't think that's true - at least it does not reflect my experience at all. But then, I don't pretend to have an extensive experience either. From most of my discussions at scipy conferences, I know most people are dissatisfied with the current python solutions.

...

...

Plenty of good work is going on with python packaging.

That's the opposite of my experience. What I care about is:  - tools which are hackable and easily extensible  - robust install/uninstall  - real, DAG-based build system  - explicit and repeatability

None of this is supported by the tools, and the current directions go even further away. When I have to explain at length why the command-based design of distutils is a nightmare to work with, I don't feel very confident that the current maintainers are aware of the issues, for example. It shows that they never had to extend distutils much.

All agreed!  I'd add to the list parallel builds/tests (make -j 16), and outputting to native build systems.  eg, xcode, msvc projects, and makefiles.

Yep - I got quite far with numscons already. It cannot be used as a general solution, but as a dev tool for my own work on numpy/scipy, it has been a huge time saver, especially given the top notch dependency tracking system. It supports // builds, and I can build full debug builds of scipy < 1 minute on a fast machine. That's a real productivity booster.

How will you handle toydist extensions so that multiple extensions do not have problems with each other?  I don't think this is possible without isolation, and even then it's still a problem.

By doing it mostly the Unix way, through protocols and file format, not through API. Good API is hard, but for build tools, it is much, much harder. When talking about extensions, I mostly think about the following: - adding a new compiler/new platform - adding a new installer format - adding a new kind of source file/target (say ctypes extension, cython compilation, etc...) Instead of using classes for compilers/tools, I am considering using python modules for each tool, and each tool would be registered through a source file extension (associate a function to ".c", for example). Actual compilation steps would be done through strings ("$CC ...."). The system would be kept simple, because for complex projects, one should forward all this to a real build system (like waf or scons). There is also the problem of post/pre hooks, adding new steps in toymaker: I have not thought much about this, but I like waf's way of doing it, and it may be applicable. In waf, the main script (called wscript) defines a function for each build step: def configure(): pass def build(): pass .... And undefined functions are considered unmodified. What I know for sure is that the distutils-way of extending through inheritance does not work at all. As soon as two extensions subclass the same base class, you're done.

Yeah, cool.  Many other projects have their own servers too. pygame.org, plone, etc etc, which meet their own needs.  Patches are accepted for pypi btw.

Yes, but how long before the patch is accepted and deployed ?

What type of enforcements of meta data, and how would they help?  I imagine this could be done in a number of ways to pypi. - a distutils command extension that people could use. - change pypi source code. - check the metadata for certain packages, then email their authors telling them about issues.

First, packages with malformed metadata would be rejected, and it would not be possible to register a package without uploading the sources. I simply do not want to publish a package which does not even have a name or a version, for example. The current way of doing things in pypi in insane if you ask me. For example, if you want to install a package with its dependencies, you need to download the package, which may be in another website, and you need to execute setup.py just to know its dependencies. This has so many failures modes, I don't understand how this can seriously be considered, really. Every other system has an index to do this kind of things (curiously, both EPD and pypm have an index as well AFAIK). Again, a typical example of NIH, with inferior solutions implemented in the case of python.

yeah, cool.  That would let you develop things incrementally too, and still have toydist be useful for the whole development period until it catches up with the features of distutils needed.

Initially, toydist was started to show that writing something compatible with distutils without being tight to distutils was possible.

If you execute build tools on arbitrary code, then arbitrary code execution is easy for someone who wants to do bad things.

Well, you could surely exploit built tools bugs. But at least, I can query metadata and packages features in a safe way - and this is very useful already (cf my points about being able to query packages metadata in one "query").

and many times I still get errors on different platforms, despite many years of multi platform coding.

Yes, that's a difficult process. We cannot fix this - but having automatically built (and hopefully tested) installers on major platforms would be a significant step in the right direction. That's one of the killer feature of CRAN (whenever you submit a package for CRAN, a windows installer is built, and tested). cheers, David

On Tue, Dec 29, 2009 at 11:34:44PM +0900, David Cournapeau wrote:

Buildout, virtualenv all work by sandboxing from the system python: each of them do not see each other, which may be useful for development, but as a deployment solution to the casual user who may not be familiar with python, it is useless. A scientist who installs numpy, scipy, etc... to try things out want to have everything available in one python interpreter, and does not want to jump to different virtualenvs and whatnot to try different packages.

I think that you are pointing out a large source of misunderstanding in packaging discussion. People behind setuptools, pip or buildout care to have a working ensemble of packages that deliver an application (often a web application)[1]. You and I, and many scientific developers see libraries as building blocks that need to be assembled by the user, the scientist using them to do new science. Thus the idea of isolation is not something that we can accept, because it means that we are restricting the user to a set of libraries. Our definition of user is not the same as the user targeted by buildout. Our user does not push buttons, but he writes code. However, unlike the developer targeted by buildout and distutils, our user does not want or need to learn about packaging. Trying to make the debate clearer... Gaël [1] I know your position on why simply focusing on sandboxing working ensemble of libraries is not a replacement for backward compatibility, and will only create impossible problems in the long run. While I agree with you, this is not my point here.

On Tue, Dec 29, 2009 at 2:34 PM, David Cournapeau wrote:

On Tue, Dec 29, 2009 at 10:27 PM, René Dudfield wrote:

...

Buildout is what a lot of the python community are using now.

I would like to note that buildout is a solution to a problem that I don't care to solve. This issue is particularly difficult to explain to people accustomed with buildout in my experience - I have not found a way to explain it very well yet.

Hello, The main problem buildout solves is getting developers up to speed very quickly on a project. They should be able to call one command and get dozens of packages, and everything else needed ready to go, completely isolated from the rest of the system. If a project does not want to upgrade to the latest versions of packages, they do not have to. This reduces the dependency problem a lot. As one package does not have to block on waiting for 20 other packages. It makes iterating packages daily, or even hourly to not be a problem - even with dozens of different packages used. This is not theoretical, many projects iterate this quickly, and do not have problems. Backwards compatibility is of course a great thing to keep up... but harder to do with dozens of packages, some of which are third party ones. For example, some people are running pygame applications written 8 years ago that are still running today on the latest versions of pygame. I don't think people in the python world understand API, and ABI compatibility as much as those in the C world. However buildout is a solution to their problem, and allows them to iterate quickly with many participants, on many different projects. Many of these people work on maybe 20-100 different projects at once, and some machines may be running that many applications at once too. So using the system pythons packages is completely out of the question for them.

A scientist who installs numpy, scipy, etc... to try things out want to have everything available in one python interpreter, and does not want to jump to different virtualenvs and whatnot to try different packages.

It is very easy to include a dozen packages in a buildout, so that you have all the packages required. Anyway... here is a skeleton buildout project that uses numpy if anyone wants to have a play. http://renesd.blogspot.com/2009/12/buildout-project-that-uses-numpy.html cheers,

On Tue, Dec 29, 2009 at 11:20 PM, David Cournapeau wrote:

On Wed, Dec 30, 2009 at 3:36 AM, René Dudfield wrote:

...

On Tue, Dec 29, 2009 at 2:34 PM, David Cournapeau wrote:

...

On Tue, Dec 29, 2009 at 10:27 PM, René Dudfield wrote:

...

Buildout is what a lot of the python community are using now.

I would like to note that buildout is a solution to a problem that I don't care to solve. This issue is particularly difficult to explain to people accustomed with buildout in my experience - I have not found a way to explain it very well yet.

Hello,

The main problem buildout solves is getting developers up to speed very quickly on a project.  They should be able to call one command and get dozens of packages, and everything else needed ready to go, completely isolated from the rest of the system.

If a project does not want to upgrade to the latest versions of packages, they do not have to.  This reduces the dependency problem a lot.  As one package does not have to block on waiting for 20 other packages.  It makes iterating packages daily, or even hourly to not be a problem - even with dozens of different packages used.  This is not theoretical, many projects iterate this quickly, and do not have problems.

Backwards compatibility is of course a great thing to keep up... but harder to do with dozens of packages, some of which are third party ones.  For example, some people are running pygame applications written 8 years ago that are still running today on the latest versions of pygame.  I don't think people in the python world understand API, and ABI compatibility as much as those in the C world.

However buildout is a solution to their problem, and allows them to iterate quickly with many participants, on many different projects. Many of these people work on maybe 20-100 different projects at once, and some machines may be running that many applications at once too. So using the system pythons packages is completely out of the question for them.

This is all great, but I don't care about solving this issue, this is a *developer* issue. I don't mean this is not an important issue, it is just totally out of scope.

The developer issues I care about are much more fine-grained (corrent dependency handling between target, toolchain customization, etc...). Note however that hopefully, by simplifying the packaging tools, the problems you see with numpy on 2.6 would be less common. The whole distutils/setuptools/distribute stack is hopelessly intractable, given how messy the code is.

The numpy issue is because of the change in package handling for 2.6, which numpy 1.3 was not developed for.

...

It is very easy to include a dozen packages in a buildout, so that you have all the packages required.

I think there is a confusion - I mostly care about *end users*. People who may not have compilers, who want to be able to easily upgrade one package, etc...

I was just describing the problems that buildout solves(for others). If I have a project that depends on numpy and 12 other packages, I can send it to other people who can get their project up and running fairly quickly (assuming everything installs ok). btw, numpy 1.4 works with buildout! (at least on my ubuntu box) sweet :) cd /tmp/ bzr branch lp:numpybuildout cd numpybuildout/trunk/ python bootstrap.py -d ./bin/buildout ./bin/py

...

...

import numpy numpy.__file__ '/tmp/numpybuildout/trunk/eggs/numpy-1.4.0-py2.6-linux-i686.egg/numpy/__init__.pyc'

On Tue, Dec 29, 2009 at 6:34 AM, David Cournapeau wrote:

Buildout, virtualenv all work by sandboxing from the system python: each of them do not see each other, which may be useful for development, but as a deployment solution to the casual user who may not be familiar with python, it is useless. A scientist who installs numpy, scipy, etc... to try things out want to have everything available in one python interpreter, and does not want to jump to different virtualenvs and whatnot to try different packages.

What I do -- and documented for people in my lab to do -- is set up one virtualenv in my user account, and use it as my default python. (I 'activate' it from my login scripts.) The advantage of this is that easy_install (or pip) just works, without any hassle about permissions etc. This should be easier, but I think the basic approach is sound. "Integration with the package system" is useless; the advantage of distribution packages is that distributions can provide a single coherent system with consistent version numbers across all packages, etc., and the only way to "integrate" with that is to, well, get the packages into the distribution. On another note, I hope toydist will provide a "source prepare" step, that allows arbitrary code to be run on the source tree. (For, e.g., cython->C conversion, ad-hoc template languages, etc.) IME this is a very common pain point with distutils; there is just no good way to do it, and it has to be supported in the distribution utility in order to get everything right. In particular: -- Generated files should never be written to the source tree itself, but only the build directory -- Building from a source checkout should run the "source prepare" step automatically -- Building a source distribution should also run the "source prepare" step, and stash the results in such a way that when later building the source distribution, this step can be skipped. This is a common requirement for user convenience, and necessary if you want to avoid arbitrary code execution during builds. And if you just set up the distribution util so that the only place you can specify arbitrary code execution is in the "source prepare" step, then even people who know nothing about packaging will automatically get all of the above right. Cheers, -- Nathaniel

On Sun, Jan 3, 2010 at 8:05 PM, Nathaniel Smith wrote:

On Tue, Dec 29, 2009 at 6:34 AM, David Cournapeau wrote:

...

Buildout, virtualenv all work by sandboxing from the system python: each of them do not see each other, which may be useful for development, but as a deployment solution to the casual user who may not be familiar with python, it is useless. A scientist who installs numpy, scipy, etc... to try things out want to have everything available in one python interpreter, and does not want to jump to different virtualenvs and whatnot to try different packages.

What I do -- and documented for people in my lab to do -- is set up one virtualenv in my user account, and use it as my default python. (I 'activate' it from my login scripts.) The advantage of this is that easy_install (or pip) just works, without any hassle about permissions etc.

It just works if you happen to be able to build everything from sources. That alone means you ignore the majority of users I intend to target. No other community (except maybe Ruby) push those isolated install solutions as a general deployment solutions. If it were such a great idea, other people would have picked up those solutions.

This should be easier, but I think the basic approach is sound. "Integration with the package system" is useless; the advantage of distribution packages is that distributions can provide a single coherent system with consistent version numbers across all packages, etc., and the only way to "integrate" with that is to, well, get the packages into the distribution.

Another way is to provide our own repository for a few major distributions, with automatically built packages. This is how most open source providers work. Miguel de Icaza explains this well: http://tirania.org/blog/archive/2007/Jan-26.html I hope we will be able to reuse much of the opensuse build service infrastructure.

On another note, I hope toydist will provide a "source prepare" step, that allows arbitrary code to be run on the source tree. (For, e.g., cython->C conversion, ad-hoc template languages, etc.) IME this is a very common pain point with distutils; there is just no good way to do it, and it has to be supported in the distribution utility in order to get everything right. In particular:  -- Generated files should never be written to the source tree itself, but only the build directory  -- Building from a source checkout should run the "source prepare" step automatically  -- Building a source distribution should also run the "source prepare" step, and stash the results in such a way that when later building the source distribution, this step can be skipped. This is a common requirement for user convenience, and necessary if you want to avoid arbitrary code execution during builds.

Build directories are hard to implement right. I don't think toydist will support this directly. IMO, those advanced builds warrant a real build tool - one main goal of toydist is to make integration with waf or scons much easier. Both waf and scons have the concept of a build directory, which should do everything you described. David

On Sun, Jan 3, 2010 at 17:42, Nathaniel Smith wrote:

On Sun, Jan 3, 2010 at 4:23 AM, David Cournapeau wrote:

...

On Sun, Jan 3, 2010 at 8:05 PM, Nathaniel Smith wrote:

...

What I do -- and documented for people in my lab to do -- is set up one virtualenv in my user account, and use it as my default python. (I 'activate' it from my login scripts.) The advantage of this is that easy_install (or pip) just works, without any hassle about permissions etc.

It just works if you happen to be able to build everything from sources. That alone means you ignore the majority of users I intend to target.

No other community (except maybe Ruby) push those isolated install solutions as a general deployment solutions. If it were such a great idea, other people would have picked up those solutions.

AFAICT, R works more-or-less identically (once I convinced it to use a per-user library directory); install.packages() builds from source, and doesn't automatically pull in and build random C library dependencies.

That's not quite the same. That is the R equivalent of Python's recent per-user site-packages feature (every user get's their own sandbox), not virtualenv (every project gets it's own sandbox). The former feature has a long history in the multiuser UNIX world and is not really controversial. http://www.python.org/dev/peps/pep-0370/ -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

Nathaniel Smith wrote:

On Sun, Jan 3, 2010 at 4:23 AM, David Cournapeau wrote:

...

Another way is to provide our own repository for a few major distributions, with automatically built packages. This is how most open source providers work. Miguel de Icaza explains this well:

http://tirania.org/blog/archive/2007/Jan-26.html

I hope we will be able to reuse much of the opensuse build service infrastructure.

Sure, I'm aware of the opensuse build service, have built third-party packages for my projects, etc. It's a good attempt, but also has a lot of problems, and when talking about scientific software it's totally useless to me :-). First, I don't have root on our compute cluster.

I use Sage for this very reason, and others use EPD or FEMHub or Python(x,y) for the same reasons. Rolling this into the Python package distribution scheme seems backwards though, since a lot of binary packages that have nothing to do with Python are used as well -- the Python stuff is simply thin wrappers around what should ideally be located in /usr/lib or similar (but are nowadays compiled into the Python extension .so because of distribution problems). To solve the exact problem you (and me) have I think the best solution is to integrate the tools mentioned above with what David is planning (SciPI etc.). Or if that isn't good enough, find generic "userland package manager" that has nothing to do with Python (I'm sure a dozen half-finished ones must have been written but didn't look), finish it, and connect it to SciPI. What David does (I think) is seperate the concerns. This makes the task feasible, and also has the advantage of convenience for the people that *do* want to use Ubuntu, Red Hat or whatever to roll out scientific software on hundreds of clients. Dag Sverre

On Wed, Dec 30, 2009 at 11:26 PM, Darren Dale wrote:

Hi David,

On Mon, Dec 28, 2009 at 9:03 AM, David Cournapeau wrote:

...

Executable: grin    module: grin    function: grin_main

Executable: grind    module: grin    function: grind_main

Have you thought at all about operations that are currently performed by post-installation scripts? For example, it might be desirable for the ipython or MayaVi windows installers to create a folder in the Start menu that contains links the the executable and the documentation. This is probably a secondary issue at this point in toydist's development, but I think it is an important feature in the long run.

Also, have you considered support for package extras (package variants in Ports, allowing you to specify features that pull in additional dependencies like traits[qt4])? Enthought makes good use of them in ETS, and I think they would be worth keeping.

Does this example covers what you have in mind ? I am not so familiar with this feature of setuptools: Name: hello Version: 1.0 Library: BuildRequires: paver, sphinx, numpy if os(windows) BuildRequires: pywin32 Packages: hello Extension: hello._bar sources: src/hellomodule.c if os(linux) Extension: hello._linux_backend sources: src/linbackend.c Note that instead of os(os_name), you can use flag(flag_name), where flag are boolean variables which can be user defined: http://github.com/cournape/toydist/blob/master/examples/simples/conditional/... http://github.com/cournape/toydist/blob/master/examples/var_example/toysetup... David

On Thu, Dec 31, 2009 at 6:06 AM, Darren Dale wrote:

I should defer to the description of extras in the setuptools documentation. It is only a few paragraphs long:

http://peak.telecommunity.com/DevCenter/setuptools#declaring-extras-optional...

Ok, so there are two issues related to this feature: - supporting variant at the build stage - supporting different variants of the same package in the dependency graph at install time The first issue is definitely supported - I fixed a bug in toydist to support this correctly, and this will be used when converting setuptools-based setup.py which use the features argument. The second issue is more challenging. It complicates the dependency handling quite a bit, and may cause difficult situations to happen at dependency resolution time. This becomes particularly messy if you mix packages you build yourself with packages grabbed from a repository. I wonder if there is a simpler solution which would give a similar feature set. cheers, David

On Sat, Jan 2, 2010 at 4:58 PM, Gael Varoquaux wrote:

On Sat, Jan 02, 2010 at 11:32:00AM +0900, David Cournapeau wrote:

...

[snip]   - supporting different variants of the same package in the dependency graph at install time

...

[snip]

...

The second issue is more challenging. It complicates the dependency handling quite a bit, and may cause difficult situations to happen at dependency resolution time. This becomes particularly messy if you mix packages you build yourself with packages grabbed from a repository. I wonder if there is a simpler solution which would give a similar feature set.

AFAICT, in Debian, the same feature is given via virtual packages: you would have:

I don't think virtual-packages entirely fix the issue. AFAIK, virtual packages have two uses: - handle dependencies where several packages may resolve one particular dependency in an equivalent way (one good example is LAPACK: both liblapack and libatlas provides the lapack feature) - closer to this discussion, you can build several variants of the same package, and each variant would resolve the dependency on a virtual package handling the commonalities. For example, say we have two numpy packages, one built with lapack (python-numpy-full), the other without (python-numpy-core). What happens when a package foo depends on numpy-full, but numpy-core is installed ? AFAICS, this can only work as long as the set containing every variant can be ordered (in the conventional set ordering sense), and the dependency can be satisfied by the smallest one. cheers, David