Mailman 3 December 2009 - SciPy-Dev

SLSQP Constrained Optimizer Status
by Rob Falck 24 Nov '13

24 Nov '13

I'm currently implementing the Sequential Least Squares Quadratic Programming (SLSQP) optimizer, by Dieter Kraft, for use in Scipy. The Fortran code being wrapped with F2PY is here: http://www.netlib.org/toms/733 (its use within Scipy has been cleared) SLSQP provides for bounds on the independent variables, as well as equality and inequality constraint functions, which is a capability that doesn't exist in scipy.optimize. Currently the code works, although the constraint normals are being generated by approximation. I'm working on a way to pass these in. I think the most elegant way will be a single function that returns the matrix of constraint normals. For a demonstration of what the code can do, here is an optimization of f(x,y) = 2xy + 2x - x**2 - 2y**2 Example 14.2 in Chapra & Canale gives the maximum as x=2.0, y=1.0. The unbounded optimization tests find this solution. As expected, its faster when derivatives are provided rather than approximated. Unbounded optimization. Derivatives approximated. Elapsed time: 1.45792961121 ms Results [[1.9999999515712266, 0.99999996181577444], -1.9999999999999984, 4, 0, 'Optimization terminated successfully.'] Unbounded optimization. Derivatives provided. Elapsed time: 1.03211402893 ms Results [[1.9999999515712266, 0.99999996181577444], -1.9999999999999984, 4, 0, 'Optimization terminated successfully.'] The following example uses an equality constraint to find the optimal when x=y. Bound optimization. Derivatives approximated. Elapsed time: 1.384973526 ms Results [[0.99999996845920858, 0.99999996845920858], -0.99999999999999889, 4, 0, 'Optimization terminated successfully.'] I've tried to conform to the syntax used by the other optimizers in scipy.optimize. The function definition and doc string are below. If anyone is interested in testing it out, let me know. def fmin_slsqp( func, x0 , eqcons=[], f_eqcons=None, ieqcons=[], f_ieqcons=None, bounds = [], fprime = None, fprime_cons=None,args = (), iter = 100, acc = 1.0E-6, iprint = 1, full_output = 0, epsilon = _epsilon ): """ Minimize a function using Sequential Least SQuares Programming Description: Python interface function for the SLSQP Optimization subroutine originally implemented by Dieter Kraft. Inputs: func - Objective function (in the form func(x, *args)) x0 - Initial guess for the independent variable(s). eqcons - A list of functions of length n such that eqcons[j](x0,*args) == 0.0 in a successfully optimized problem f_eqcons - A function of the form f_eqcons(x, *args) that returns an array in which each element must equal 0.0 in a successfully optimized problem. If f_eqcons is specified, eqcons is ignored. ieqcons - A list of functions of length n such that ieqcons[j](x0,*args) >= 0.0 in a successfully optimized problem f_ieqcons - A function of the form f_ieqcons(x0, *args) that returns an array in which each element must be greater or equal to 0.0 in a successfully optimized problem. If f_ieqcons is specified, ieqcons is ignored. bounds - A list of tuples specifying the lower and upper bound for each independent variable [ (xl0, xu0), (xl1, xu1), ...] fprime - A function that evaluates the partial derivatives of func fprime_cons - A function of the form f(x, *args) that returns the m by n array of constraint normals. If not provided, the normals will be approximated. Equality constraint normals precede inequality constraint normals. args - A sequence of additional arguments passed to func and fprime iter - The maximum number of iterations (int) acc - Requested accuracy (float) iprint - The verbosity of fmin_slsqp. iprint <= 0 : Silent operation iprint == 1 : Print summary upon completion (default) iprint >= 2 : Print status of each iterate and summary full_output - If 0, return only the minimizer of func (default). Otherwise, output final objective function and summary information. epsilon - The step size for finite-difference derivative estimates. Outputs: ( x, { fx, gnorm, its, imode, smode }) x - The final minimizer of func. fx - The final value of the objective function. its - The number of iterations. imode - The exit mode from the optimizer, as an integer. smode - A string describing the exit mode from the optimizer. Exit modes are defined as follows: -1 : Gradient evaluation required (g & a) 0 : Optimization terminated successfully. 1 : Function evaluation required (f & c) 2 : Number of equality constraints larger than number of independent variables 3 : More than 3*n iterations in LSQ subproblem 4 : Inequality constraints incompatible 5 : Singular matrix E in LSQ subproblem 6 : Singular matrix C in LSQ subproblem 7 : Rank-deficient equality constraint subproblem HFTI 8 : Positive directional derivative for linesearch 9 : Iteration limit exceeded -- - Rob Falck

4 16

np.savetxt: apply patch in enhancement ticket 1079 to add headers?
by Skipper Seabold 08 Jun '10

08 Jun '10

Hi all, I am assuming that this is ok to request via the list... Could we discuss or could someone apply the patch in enhancement ticket 1079? http://projects.scipy.org/numpy/ticket/1079 I needed this functionality recently, and this is a quick and easy fix that may have been overlooked. There is also another enhancement request about this here: http://projects.scipy.org/numpy/ticket/1236 The only thing that I can think of that might need to be added is a test to see that the header length is the same as the number of columns, but really that might just be up to the user to supply the right headers. It might also be nice to have a header = True, that uses the field names for a structured array, but I can live without that. Cheers, Skipper

3 11

Re: [SciPy-dev] [Numpy-discussion] Announcing toydist, improving distribution and packaging situation
by David Cournapeau 04 Jan '10

04 Jan '10

On Tue, Dec 29, 2009 at 10:27 PM, René Dudfield <renesd(a)gmail.com> wrote: > Buildout is what a lot of the python community are using now. I would like to note that buildout is a solution to a problem that I don't care to solve. This issue is particularly difficult to explain to people accustomed with buildout in my experience - I have not found a way to explain it very well yet. Buildout, virtualenv all work by sandboxing from the system python: each of them do not see each other, which may be useful for development, but as a deployment solution to the casual user who may not be familiar with python, it is useless. A scientist who installs numpy, scipy, etc... to try things out want to have everything available in one python interpreter, and does not want to jump to different virtualenvs and whatnot to try different packages. This has strong consequences on how you look at things from a packaging POV: - uninstall is crucial - a package bringing down python is a big no no (this happens way too often when you install things through setuptools) - if something fails, the recovery should be trivial - the person doing the installation may not know much about python - you cannot use sandboxing as a replacement for backward compatibility (that's why I don't care much about all the discussion about versioning - I don't think it is very useful as long as python itself does not support it natively). In the context of ruby, this article makes a similar point: http://www.madstop.com/ruby/ruby_has_a_distribution_problem.html David

4 8

Re: [SciPy-dev] [Numpy-discussion] Announcing toydist, improving distribution and packaging situation
by David Cournapeau 03 Jan '10

03 Jan '10

On Tue, Dec 29, 2009 at 10:27 PM, René Dudfield <renesd(a)gmail.com> wrote: > Hi, > > In the toydist proposal/release notes, I would address 'what does > toydist do better' more explicitly. > > > > **** A big problem for science users is that numpy does not work with > pypi + (easy_install, buildout or pip) and python 2.6. **** > > > > Working with the rest of the python community as much as possible is > likely a good goal. Yes, but it is hopeless. Most of what is being discussed on distutils-sig is useless for us, and what matters is ignored at best. I think most people on distutils-sig are misguided, and I don't think the community is representative of people concerned with packaging anyway - most of the participants seem to be around web development, and are mostly dismissive of other's concerns (OS packagers, etc...). I want to note that I am not starting this out of thin air - I know most of distutils code very well, I have been the mostly sole maintainer of numpy.distutils for 2 years now. I have written extensive distutils extensions, in particular numscons which is able to fully build numpy, scipy and matplotlib on every platform that matters. Simply put, distutils code is horrible (this is an objective fact) and flawed beyond repair (this is more controversial). IMHO, it has almost no useful feature, except being standard. If you want a more detailed explanation of why I think distutils and all tools on top are deeply flawed, you can look here: http://cournape.wordpress.com/2009/04/01/python-packaging-a-few-observation… > numpy used to work with buildout in python2.5, but not with 2.6. > buildout lets other team members get up to speed with a project by > running one command. It installs things in the local directory, not > system wide. So you can have different dependencies per project. I don't think it is a very useful feature, honestly. It seems to me that they created a huge infrastructure to split packages into tiny pieces, and then try to get them back together, imaganing that multiple installed versions is a replacement for backward compatibility. Anyone with extensive packaging experience knows that's a deeply flawed model in general. > Plenty of good work is going on with python packaging. That's the opposite of my experience. What I care about is: - tools which are hackable and easily extensible - robust install/uninstall - real, DAG-based build system - explicit and repeatability None of this is supported by the tools, and the current directions go even further away. When I have to explain at length why the command-based design of distutils is a nightmare to work with, I don't feel very confident that the current maintainers are aware of the issues, for example. It shows that they never had to extend distutils much. > > There are build farms for windows packages and OSX uploaded to pypi. > Start uploading pre releases to pypi, and you get these for free (once > you make numpy compile out of the box on those compile farms). There > are compile farms for other OSes too... like ubuntu/debian, macports > etc. Some distributions even automatically download, compile and > package new releases once they spot a new file on your ftp/web site. I am familiar with some of those systems (PPA and opensuse build service in particular). One of the goal of my proposal is to make it easier to interoperate with those tools. I think Pypi is mostly useless. The lack of enforced metadata is a big no-no IMHO. The fact that Pypi is miles beyond CRAN for example is quite significant. I want CRAN for scientific python, and I don't see Pypi becoming it in the near future. The point of having our own Pypi-like server is that we could do the following: - enforcing metadata - making it easy to extend the service to support our needs > > pypm: http://pypm.activestate.com/list-n.html#numpy It is interesting to note that one of the maintainer of pypm has recently quitted the discussion about Pypi, most likely out of frustration from the other participants. > Documentation projects are being worked on to document, give tutorials > and make python packaging be easier all round. As witnessed by 20 or > so releases on pypi every day(and growing), lots of people are using > the python packaging tools successfully. This does not mean much IMO. Uploading on Pypi is almost required to use virtualenv, buildout, etc.. An interesting metric is not how many packages are uploaded, but how much it is used outside developers. > > I'm not sure making a separate build tool is a good idea. I think > going with the rest of the python community, and improving the tools > there is a better idea. It has been tried, and IMHO has been proved to have failed. You can look at the recent discussion (the one started by Guido in particular). > pps. some notes on toydist itself. > - toydist convert is cool for people converting a setup.py . This > means that most people can try out toydist right away. but what does > it gain these people who convert their setup.py files? Not much ATM, except that it is easier to write a toysetup.info compared to setup.py IMO, and that it supports a simple way to include data files (something which is currently *impossible* to do without writing your own distutils extensions). It has also the ability to build eggs without using setuptools (I consider not using setuptools a feature, given the too many failure modes of this package). The main goals though are to make it easier to build your own tools on top of if, and to integrate with real build systems. > - a toydist convert that generates a setup.py file might be cool :) toydist started like this, actually: you would write a setup.py file which loads the package from toysetup.info, and can be converted to a dict argument to distutils.core.setup. I have not updated it recently, but that's definitely on the TODO list for a first alpha, as it would enable people to benefit from the format, with 100 % backward compatibility with distutils. > - arbitrary code execution happens when building or testing with > toydist. You are right for testing, but wrong for building. As long as the build is entirely driven by toysetup.info, you only have to trust toydist (which is not safe ATM, but that's an implementation detail), and your build tools of course. Obviously, if you have a package which uses an external build tool on top of toysetup.info (as will be required for numpy itself for example), all bets are off. But I think that's a tiny fraction of the interesting packages for scientific computing. Sandboxing is particularly an issue on windows - I don't know a good solution for windows sandboxing, outside of full vms, which are heavy-weights. > - it should be possible to build this toydist functionality as a > distutils/distribute/buildout extension. No, it cannot, at least as far as distutils/distribute are concerned (I know nothing about buildout). Extending distutils is horrible, and fragile in general. Even autotools with its mix of generated sh scripts through m4 and perl is a breeze compared to distutils. > - extending toydist? How are extensions made? there are 175 buildout > packages which extend buildout, and many that extend > distutils/setuptools - so extension of build tools in a necessary > thing. See my answer earlier about interoperation with build tools. cheers, David

2 1

Announcing toydist, improving distribution and packaging situation
by David Cournapeau 02 Jan '10

02 Jan '10

(warning, long post) Hi there, As some of you already know, the packaging and distributions of scientific python packages have been a constant source of frustration. Open source is about making it easy for anyone to use software how they see fit, and I think python packaging infrastructure has not been very successfull for people not intimately familiar with python. A few weeks ago, after Guido visited Berkeley and was told how those issues were still there for the scientific community, he wrote an email asking whether current efforts on distutils-sig will be enough (see http://aspn.activestate.com/ASPN/Mail/Message/distutils-sig/3775972). Several of us have been participating to this discussion, but I feel like the divide between current efforts on distutils-sig and us (the SciPy community) is not getting smaller. At best, their efforts will be more work for us to track the new distribute fork, and more likely, it will be all for nothing as it won't solve any deep issue. To be honest, most of what is considered on distutils-sig sounds like anti-goals to me. Instead of keeping up with the frustrating process of "improving" distutils, I think we have enough smart people and manpower in the scientific community to go with our own solution. I am convinced it is doable because R or haskell, with a much smaller community than python, managed to pull out something with is miles ahead compared to pypi. The SciPy community is hopefully big enough so that a SciPy-specific solution may reach critical mass. Ideally, I wish we had something with the following capabilities: - easy to understand tools - http-based package repository ala CRAN, which would be easy to mirror and backup (through rsync-like tools) - decoupling the building, packaging and distribution of code and data - reliable install/uninstall/query of what is installed locally - facilities for building windows/max os x binaries - making the life of OS vendors (Linux, *BSD, etc...) easier The packaging part ============== Speaking is easy, so I started coding part of this toolset, called toydist (temporary name), which I presented at Scipy India a few days ago: http://github.com/cournape/toydist/ Toydist is more or less a rip off of cabal (http://www.haskell.org/cabal/), and consist of three parts: - a core which builds a package description from a declarative file similar to cabal files. The file is almost purely declarative, and can be parsed so that no arbitrary code is executed, thus making it easy to sandbox packages builds (e.g. on a build farm). - a set of command line tools to configure, build, install, build installers (egg only for now) etc... from the declarative file - backward compatibility tools: a tool to convert existing setup.py to the new format has been written, and a tool to use distutils through the new format for backward compatibility with complex distutils extensions should be relatively easy. The core idea is to make the format just rich enough to describe most packages out there, but simple enough so interfacing it with external tools is possible and reliable. As a regular contributor to scons, I am all too aware that a build tool is a very complex beast to get right, and repeating their efforts does not make sense. Typically, I envision that complex packages such as numpy, scipy or matplotlib would use make/waf/scons for the build - in a sense, toydist is written so that writing something like numscons would be easier. OTOH, most if not all scikits should be buildable from a purely declarative file. To give you a feel of the format, here is a snippet for the grin package from Robert K. (automatically converted): Name: grin Version: 1.1.1 Summary: A grep program configured the way I like it. Description: ==== grin ==== I wrote grin to help me search directories full of source code. The venerable GNU grep_ and find_ are great tools, but they fall just a little short for my normal use cases. <snip> License: BSD Platforms: UNKNOWN Classifiers: License :: OSI Approved :: BSD License, Development Status :: 5 - Production/Stable, Environment :: Console, Intended Audience :: Developers, Operating System :: OS Independent, Programming Language :: Python, Topic :: Utilities, ExtraSourceFiles: README.txt, setup.cfg, setup.py, Library: InstallDepends: argparse, Modules: grin, Executable: grin module: grin function: grin_main Executable: grind module: grin function: grind_main Although still very much experimental at this point, toydist already makes some things much easier than with distutils/setuptools: - path customization for any target can be done easily: you can easily add an option in the file so that configure --mynewdir=value works and is accessible at every step. - making packages FHS compliant is not a PITA anymore, and the scheme can be adapted to any OS, be it traditional FHS-like unix, mac os x, windows, etc... - All the options are accessible at every step (no more distutils commands nonsense) - data files can finally be handled correctly and consistently, instead of the 5 or 6 magics methods currently available in distutils/setuptools/numpy.distutils - building eggs does not involve setuptools anymore - not much coupling between package description and build infrastructure (building extensions is actually done through distutils ATM). Repository ======== The goal here is to have something like CRAN (http://cran.r-project.org/web/views/), ideally with a build farm so that whenever anyone submits a package to our repository, it would automatically be checked, and built for windows/mac os x and maybe a few major linux distributions. One could investigate the build service from open suse to that end (http://en.opensuse.org/Build_Service), which is based on xen VM to build installers in a reproducible way. Installed package db =============== I believe that the current open source enstaller package from Enthought can be a good starting point. It is based on eggs, but eggs are only used as a distribution format (eggs are never installed as eggs AFAIK). You can easily remove packages, query installed versions, etc... Since toydist produces eggs, interoperation between toydist and enstaller should not be too difficult. What's next ? ========== At this point, I would like to ask for help and comments, in particular: - Does all this make sense, or hopelessly intractable ? - Besides the points I have mentioned, what else do you think is needed ? - There has already been some work for the scikits webportal, but I think we should bypass pypi entirely (the current philosophy of not enforcing consistent metadata does not make much sense to me, and is at the opposite of most other similar system out there). - I think a build farm for at least windows packages would be a killer feature, and enough incentive to push some people to use our new infrastructure. It would be good to have a windows guy familiar with windows sandboxing/virtualization to do something there. The people working on the opensuse build service have started working on windows support - I think being able to automatically convert most of scientific packages is a significant feature, and needs to be more robust - so anyone is welcomed to try converting existing setup.py with toydist (see toydist readme). thanks, David

4 10

cephes docstrings editable?
by josef.pktd＠gmail.com 01 Jan '10

01 Jan '10

is it possible to edit the cephes doc strings in scipy.special? for example: http://docs.scipy.org/scipy/docs/scipy.special._cephes.chdtr/#scipy-special… The doc editor allows editing, but doesn't show the source. Are edits propagated back to wherever the generated (?) doc strings are hiding? Josef

2 1

trunk failures
by josef.pktd＠gmail.com 31 Dec '09

31 Dec '09

just FYI, after upgrading to numpy release Josef >>> numpy.version.version '1.4.0' >>> scipy.version.version '0.8.0.dev6156' ====================================================================== ERROR: test_decomp.test_lapack_misaligned(<function solve at 0x01FA10F0>, (array ([[ 1.734e-255, 8.189e-217, 4.025e-178, 1.903e-139, 9.344e-101, ---------------------------------------------------------------------- Traceback (most recent call last): File "c:\programs\python25\lib\site-packages\nose-0.11.1-py2.5.egg\nose\case.p y", line 183, in runTest self.test(*self.arg) File "c:\josef\_progs\subversion\scipy-trunk_after\trunk\dist\scipy-0.8.0.dev6 156.win32\programs\python25\lib\site-packages\scipy\linalg\tests\test_decomp.py" , line 1106, in check_lapack_misaligned func(*a,**kwargs) File "\Programs\Python25\Lib\site-packages\scipy\linalg\basic.py", line 127, i n solve File "C:\Programs\Python25\Lib\site-packages\numpy\lib\function_base.py", line 586, in asarray_chkfinite raise ValueError, "array must not contain infs or NaNs" ValueError: array must not contain infs or NaNs ====================================================================== ERROR: test_mpmath.test_expi_complex ---------------------------------------------------------------------- Traceback (most recent call last): File "c:\programs\python25\lib\site-packages\nose-0.11.1-py2.5.egg\nose\case.p y", line 183, in runTest self.test(*self.arg) File "C:\Programs\Python25\Lib\site-packages\numpy\testing\decorators.py", lin e 146, in skipper_func return f(*args, **kwargs) File "c:\josef\_progs\subversion\scipy-trunk_after\trunk\dist\scipy-0.8.0.dev6 156.win32\programs\python25\lib\site-packages\scipy\special\tests\test_mpmath.py ", line 46, in test_expi_complex dataset = np.array(dataset, dtype=np.complex_) TypeError: a float is required ====================================================================== FAIL: test_lambertw.test_values ---------------------------------------------------------------------- Traceback (most recent call last): File "c:\programs\python25\lib\site-packages\nose-0.11.1-py2.5.egg\nose\case.p y", line 183, in runTest self.test(*self.arg) File "c:\josef\_progs\subversion\scipy-trunk_after\trunk\dist\scipy-0.8.0.dev6 156.win32\programs\python25\lib\site-packages\scipy\special\tests\test_lambertw. py", line 80, in test_values FuncData(w, data, (0,1), 2, rtol=1e-10, atol=1e-13).check() File "c:\josef\_progs\subversion\scipy-trunk_after\trunk\dist\scipy-0.8.0.dev6 156.win32\programs\python25\lib\site-packages\scipy\special\tests\testutils.py", line 187, in check assert False, "\n".join(msg) AssertionError: Max |adiff|: 2.5797 Max |rdiff|: 3.81511 Bad results for the following points (in output 0): (-0.44800000000000001+0.40000000000000002j) 0j => ( -1.2370928928166736-1.6588828572971361j) != (-0.11855133765652383+0.665705343135 83418j) (rdiff 3.8151122286225245) ---------------------------------------------------------------------- Ran 4259 tests in 91.016s FAILED (KNOWNFAIL=10, SKIP=33, errors=3, failures=1) <nose.result.TextTestResult run=4259 errors=3 failures=1> I'm also getting several special.gammaincinv warnings in stats.distributions, that I think are new, but they don't cause any test failure c:\josef\_progs\subversion\scipy-trunk_after\trunk\dist\scipy-0.8.0.dev6156.win3 2\programs\python25\lib\site-packages\scipy\stats\distributions.py:2050: Special FunctionWarning: gammaincinv: failed to converge at (a, y) = (1.1844749658973861 , 0.083706058837762387): 3 val2 = special.gammaincinv(a,1.0-q)

2 1

Re: [SciPy-dev] [Numpy-discussion] Announcing toydist, improving distribution and packaging situation
by David Cournapeau 30 Dec '09

30 Dec '09

On Wed, Dec 30, 2009 at 8:15 PM, René Dudfield <renesd(a)gmail.com> wrote: > > Sitting down with Tarek(who is one of the current distutils > maintainers) in Berlin we had a little discussion about packaging over > pizza and beer... and he was quite mindful of OS packagers problems > and issues. This has been said many times on distutils-sig, but no concrete action has ever been taken in that direction. For example, toydist already supports the FHS better than distutils, and is more flexible. I have tried several times to explain why this matters on distutils-sig, but you then have the peanuts gallery interfering with unrelated nonsense (like it would break windows, as if it could not be implemented independently). Also, retrofitting support for --*dir in distutils would be *very* difficult, unless you are ready to break backward compatibility (there are 6 ways to install data files, and each of them has some corner cases, for example - it is a real pain to support this correctly in the convert command of toydist, and you simply cannot recover missing information to comply with the FHS in every case). > > However these systems were developed by the zope/plone/web crowd, so > they are naturally going to be thinking a lot about zope/plone/web > issues. Agreed - it is natural that they care about their problems first, that's how it works in open source. What I find difficult is when our concern are constantly dismissed by people who have no clue about our issues - and later claim we are not cooperative. > Debian, and ubuntu packages for them are mostly useless > because of the age. That's where the build farm enters. This is known issue, that's why the build service or PPA exist in the first place. > I think > perhaps if toydist included something like stdeb as not an extension > to distutils, but a standalone tool (like toydist) there would be less > problems with it. That's pretty much how I intend to do things. Currently, in toydist, you can do something like: from toydist.core import PackageDescription pkg = PackageDescription.from_file("toysetup.info") # pkg now gives you access to metadata, as well as extensions, python modules, etc... I think this gives almost everything that is needed to implement a sdist_dsc command. Contrary to the Distribution class in distutils, this class would not need to be subclassed/monkey-patched by extensions, as it only cares about the description, and is 100 % uncoupled from the build part. > yes, I have also battled with distutils over the years. However it is > simpler than autotools (for me... maybe distutils has perverted my > fragile mind), and works on more platforms for python than any other > current system. Autotools certainly works on more platforms (windows notwhistanding), if only because python itself is built with autoconf. Distutils simplicity is a trap: it is simpler only if you restrict to what distutils gives you. Don't get me wrong, autotools are horrible, but I have never encountered cases where I had to spend hours to do trivial tasks, as has been the case with distutils. Numpy build system would be much, much easier to implement through autotools, and would be much more reliable. > However > distutils has had more tests and testing systems added, so that > refactoring/cleaning up of distutils can happen more so. You can't refactor distutils without breaking backward compatibility, because distutils has no API. The whole implementation is the API. That's one of the fundamental disagreement I and other scipy dev have with current contributors on distutils-sig: the starting point (distutils) and the goal are so far away from each other that getting there step by step is hopeless. > I agree with many things in that post. Except your conclusion on > multiple versions of packages in isolation. Package isolation is like > processes, and package sharing is like threads - and threads are evil! I don't find the comparison very helpful (for once, you can share data between processes, whereas virtualenv cannot see each other AFAIK). > Science is supposed to allow repeatability. Without the same versions > of packages, repeating experiments is harder. This is a big problem > in science that multiple versions of packages in _isolation_ can help > get to a solution to the repeatability problem. I don't think that's true - at least it does not reflect my experience at all. But then, I don't pretend to have an extensive experience either. From most of my discussions at scipy conferences, I know most people are dissatisfied with the current python solutions. > >>> Plenty of good work is going on with python packaging. >> >> That's the opposite of my experience. What I care about is: >> - tools which are hackable and easily extensible >> - robust install/uninstall >> - real, DAG-based build system >> - explicit and repeatability >> >> None of this is supported by the tools, and the current directions go >> even further away. When I have to explain at length why the >> command-based design of distutils is a nightmare to work with, I don't >> feel very confident that the current maintainers are aware of the >> issues, for example. It shows that they never had to extend distutils >> much. >> > > All agreed! I'd add to the list parallel builds/tests (make -j 16), > and outputting to native build systems. eg, xcode, msvc projects, and > makefiles. Yep - I got quite far with numscons already. It cannot be used as a general solution, but as a dev tool for my own work on numpy/scipy, it has been a huge time saver, especially given the top notch dependency tracking system. It supports // builds, and I can build full debug builds of scipy < 1 minute on a fast machine. That's a real productivity booster. > > How will you handle toydist extensions so that multiple extensions do > not have problems with each other? I don't think this is possible > without isolation, and even then it's still a problem. By doing it mostly the Unix way, through protocols and file format, not through API. Good API is hard, but for build tools, it is much, much harder. When talking about extensions, I mostly think about the following: - adding a new compiler/new platform - adding a new installer format - adding a new kind of source file/target (say ctypes extension, cython compilation, etc...) Instead of using classes for compilers/tools, I am considering using python modules for each tool, and each tool would be registered through a source file extension (associate a function to ".c", for example). Actual compilation steps would be done through strings ("$CC ...."). The system would be kept simple, because for complex projects, one should forward all this to a real build system (like waf or scons). There is also the problem of post/pre hooks, adding new steps in toymaker: I have not thought much about this, but I like waf's way of doing it, and it may be applicable. In waf, the main script (called wscript) defines a function for each build step: def configure(): pass def build(): pass .... And undefined functions are considered unmodified. What I know for sure is that the distutils-way of extending through inheritance does not work at all. As soon as two extensions subclass the same base class, you're done. > > Yeah, cool. Many other projects have their own servers too. > pygame.org, plone, etc etc, which meet their own needs. Patches are > accepted for pypi btw. Yes, but how long before the patch is accepted and deployed ? > What type of enforcements of meta data, and how would they help? I > imagine this could be done in a number of ways to pypi. > - a distutils command extension that people could use. > - change pypi source code. > - check the metadata for certain packages, then email their authors > telling them about issues. First, packages with malformed metadata would be rejected, and it would not be possible to register a package without uploading the sources. I simply do not want to publish a package which does not even have a name or a version, for example. The current way of doing things in pypi in insane if you ask me. For example, if you want to install a package with its dependencies, you need to download the package, which may be in another website, and you need to execute setup.py just to know its dependencies. This has so many failures modes, I don't understand how this can seriously be considered, really. Every other system has an index to do this kind of things (curiously, both EPD and pypm have an index as well AFAIK). Again, a typical example of NIH, with inferior solutions implemented in the case of python. > > yeah, cool. That would let you develop things incrementally too, and > still have toydist be useful for the whole development period until it > catches up with the features of distutils needed. Initially, toydist was started to show that writing something compatible with distutils without being tight to distutils was possible. > If you execute build tools on arbitrary code, then arbitrary code > execution is easy for someone who wants to do bad things. Well, you could surely exploit built tools bugs. But at least, I can query metadata and packages features in a safe way - and this is very useful already (cf my points about being able to query packages metadata in one "query"). > and many times I still > get errors on different platforms, despite many years of multi > platform coding. Yes, that's a difficult process. We cannot fix this - but having automatically built (and hopefully tested) installers on major platforms would be a significant step in the right direction. That's one of the killer feature of CRAN (whenever you submit a package for CRAN, a windows installer is built, and tested). cheers, David

1 0

Re: [SciPy-dev] [Numpy-discussion] Announcing toydist, improving distribution and packaging situation
by David Cournapeau 29 Dec '09

29 Dec '09

On Wed, Dec 30, 2009 at 3:36 AM, René Dudfield <renesd(a)gmail.com> wrote: > On Tue, Dec 29, 2009 at 2:34 PM, David Cournapeau <cournape(a)gmail.com> wrote: >> On Tue, Dec 29, 2009 at 10:27 PM, René Dudfield <renesd(a)gmail.com> wrote: >> >>> Buildout is what a lot of the python community are using now. >> >> I would like to note that buildout is a solution to a problem that I >> don't care to solve. This issue is particularly difficult to explain >> to people accustomed with buildout in my experience - I have not found >> a way to explain it very well yet. > > Hello, > > The main problem buildout solves is getting developers up to speed > very quickly on a project. They should be able to call one command > and get dozens of packages, and everything else needed ready to go, > completely isolated from the rest of the system. > > If a project does not want to upgrade to the latest versions of > packages, they do not have to. This reduces the dependency problem a > lot. As one package does not have to block on waiting for 20 other > packages. It makes iterating packages daily, or even hourly to not be > a problem - even with dozens of different packages used. This is not > theoretical, many projects iterate this quickly, and do not have > problems. > > Backwards compatibility is of course a great thing to keep up... but > harder to do with dozens of packages, some of which are third party > ones. For example, some people are running pygame applications > written 8 years ago that are still running today on the latest > versions of pygame. I don't think people in the python world > understand API, and ABI compatibility as much as those in the C world. > > However buildout is a solution to their problem, and allows them to > iterate quickly with many participants, on many different projects. > Many of these people work on maybe 20-100 different projects at once, > and some machines may be running that many applications at once too. > So using the system pythons packages is completely out of the question > for them. This is all great, but I don't care about solving this issue, this is a *developer* issue. I don't mean this is not an important issue, it is just totally out of scope. The developer issues I care about are much more fine-grained (corrent dependency handling between target, toolchain customization, etc...). Note however that hopefully, by simplifying the packaging tools, the problems you see with numpy on 2.6 would be less common. The whole distutils/setuptools/distribute stack is hopelessly intractable, given how messy the code is. > > It is very easy to include a dozen packages in a buildout, so that you > have all the packages required. I think there is a confusion - I mostly care about *end users*. People who may not have compilers, who want to be able to easily upgrade one package, etc... David

1 0

Re: [SciPy-dev] [matplotlib-devel] Announcing toydist, improving distribution and packaging situation
by Gael Varoquaux 29 Dec '09

29 Dec '09

On Mon, Dec 28, 2009 at 02:29:24PM -0500, Neal Becker wrote: > Perhaps this could be useful: > http://checkinstall.izto.org/ Yes, checkinstall is really cool. However, I tend to prefer things with no magic that I don't have to sandbox to know what they are doing. This is why I am also happy to hear about toydist. Gaël

2 1