mlabwrap scikit [Was: Re: scikits project]
[Second attempt to move this to scipy-dev, this time using the right email account; for those who have no idea what mlabwrap is about: it's a python to matlab bridge currently on the move from sourceforge to scipy's emerging scikits infrastructure] Hi, First, it's great that mlabwrap has found a new home at scipy.org, big thanks to everyone involved. I'm afraid that I'm still not quite up to speed on all infrastructural issues -- I've not used subversion before and I haven't completely grokked the scipy infrastructure yet (although I've been trying to read up, especially on svn) and I'm a bit puzzled by some things (see below), so please excuse if some of what follows sounds a bit clueless. In order to bring the mlabwrap project forward, there are three things I'd like to achieve in the immediate future (by end of next week): 1. Resolve remaining infrastructural questions 2. Bring out mlabwrap-1.0 final 3. Discuss a roadmap and feature requirements for mlabwrap 2.0 So let's start out with 1. : ---------------------------- "Jarrod Millman" <millman@berkeley.edu> writes:
Hey,
Jeff got everything setup and I tested it out. Everything looks good (Thanks Jeff and Robert!).
I checked in mlabwrap v1.0b: https://projects.scipy.org/scipy/scikits/browser/trunk/mlabwrap And then since Jeff installed the rest macro; I added [[ReST(/trunk/mlabwrap/README.txt)]] which Trac renders like this: https://projects.scipy.org/scipy/scikits/wiki/MlabWrap I also moved the discussion about proxy objects from the NIPY site: https://projects.scipy.org/scipy/scikits/wiki/MlabProxyObjects
Hopefully everyone has access to the scikits subversion repository now. If not, please try and get access ASAP.
Shouldn't having a svn account mean that I'd be able to succesfully log in to the trac interface at <https://projects.scipy.org/scipy/scikits/browser/trunk/mlabwrap> using the same account? I can't either use the AlexanderSchmolck login I created on scipy.org, nor the POP3/SVN login a.schmolck@gmail.com[1] I received an email from webmin@scipy.org on Friday[3]; however I *can* commit to the repository using svn (I've sent an email to webmin@scipy.org, but haven't heard anything back). Another thing I've been wondering about is whether it would be easy to import the existing CVS repository into svn? I've fiddled around a little bit with cvs2svn and on first sight it looks like it migh work OK. Loosing the project history wouln't be too tragic (and I can just try to get the sf staff to upload the old CVS repository on the sf page, so it's publically available somewhere), but if we can just import the existing repository without too much of a hassle, I think this might be the best option. Before doing anything with the CVS repository, I'd first manually adapt the directory structure, though, bringing us to 1.a. source code and svn organization ''''''''''''''''''''''''''''''''''''' In an email some time ago Robert Kern proposed the following directory structure for scikits: branches/ tags/ trunk/ mlabwrap/ setup.py scikits/ __init__.py mlabwrap/ __init__.py The current directory structure misses the scikits/ dir and the second mlabwrap/ dir; instead it has mlabwrap.py at top-level (as well as the utility libraries awmstools.py and awmsmeta.py and the c++-extension mlabraw.cpp). Independent of scikits conventions, it seems that making mlabwrap a package and stuffing everything it needs inside (i.e. the utility libs and mlabraw) rather than polluting the toplevel module-namespace seems desirable I assume this question reflects my svn-newbie status, but why doesn't the scikits structure look something like this, given that as I understand it scikits are meant to be essentially independent (and hence independely versioned) projects under a common namespace? mlabwrap/ branches/ tags/ trunk/ setup.py scikits/ __init__.py mlabwrap/ __init__.py some_other_scikit/ branches/ ... It somehow seems to me that tags and branches should apply to individual projects and that one could still do convenient releases and svn-checkouts of scikits/ as a whole by using e.g. svn:externals to the individual projects. As I said, I'm not really knowledgable about svn, but I'd like to understand the logic a bit better, because I'm also trying to work out what to do about the awmsmeta.py and awmstools.py stuff, which isn't really a part of mlabwrap as such. I see three ways of dealing with the dependency on the utility modules: D1. Drop it and copy-and-paste the needed bits into mlabwrap.py (certainly possible, only a few functions are needed; OTOH it seems a bit ugly) D2. Rely on cheeseshop (but I don't have the impression that it is equally acceptable and painless for a python module to download and install dependencies as it is for a CTAN module) D3. Stuff copies into mlabwrap/ and do a relative import D3. seems most attractive to me at the moment but one would still have to figure out where the VC of awmstools.py and awmsmeta.py would "live"; I thought it would be cleanest if they were included as svn:externals and not directly part of the mlabwrap svn tree, so I thought maybe something like this (ignoring the question of where branches/ trunk/ and tags/ go exactly): mlabwrap/ setup.py scikits/ __init__.py README.txt # etc. mlabraw.py # dummy importing mlabwrap/mlabraw, for backwards comp. mlabwrap/ __init__.py -> awmstools.py -> awmsmeta.py mlabwraw.cpp test/ test_mlabwrap.py # etc. I also assume we want a branch for the 1.0.x version, a tag for the 1.0 release (when it's ready) and that development of the next version will continue in the trunk. Comments? Also are there any strong feelings/established procedures concerning __init__.py? The generic name is somewhat inconvient in various contexts (e.g. buffer switching im emacs) so I thought about making __init__.py a dummy importing ./_mlabwrap.py or so. OK, sorry this is getting so long, on to 2. bringing out mlabwrap-1.0 final: ------------------------------------ Apart from the question what the directory structure should look like, the only issue that I'm currently aware of is that there appears to be a problem with `setup.py` automatically finding out about the matlab version under windows; if no one steps up to do some windows testing I'm also willing to apply some fix I assume will work and release as is (windows users might have to set MATLAB_VERSION etc. in setup.py by hand if it doesn't, which isn't *that* terrible). Actually, one more thing: distutils vs. scipy distutils vs. setuptools -- which one should mlabwrap-1.0 final use? I'm only really fully familiar with the first one. Finally point 3, the road-map: ------------------------------ I'd like to get some idea what is felt as needed for the next version of mlabwrap (let's say 2), by when (e.g. because it would be really handy for NIPY; how important is e.g. that 'proxy.a.b = c' works as expected?) and who is interested on working on specific parts (like the move to ctypes). Also what python and matlab versions should mlabwrap 2 have to support? How should changes affecting backwards compatibility best be handled (maybe allowing getting v1 default behavior with something like `from mlabwrap.v1 import mlab`?)? I've put up some notes concerning mlabwrap development and design issues at <http://www.scipy.org/MlabWrap>, corrections, additions and feedback (using either the wiki page itself or scipy-dev) are highly welcome. Sorry for the relative long silence -- apart from hacking on the next version, writing up my thoughts and wikifying took *far* longer than I would have thought, especially since it's only personal note-quality rather than something polished. I hope it's at least partly intelligible and useful. cheers, 'as Footnotes: [1] Are those two account logins meant to be different or is this just because I was too dumb to notify Jeff Strunk of my newly created scipy.org account in time? BTW unless there's some convention that the svn login should be an email address, I'd rather just have a.schmolck (sans gmail.com), if it's something that's easy to change.
On Thu, Mar 29, 2007 at 04:13:28AM +0100, Alexander Schmolck wrote:
In an email some time ago Robert Kern proposed the following directory structure for scikits:
branches/ tags/ trunk/ mlabwrap/ setup.py scikits/ __init__.py mlabwrap/ __init__.py
Shouldn't that be branches/ tags/ trunk/ scikits/ setup.py __init__.py mlabwrap/ __init__.py gpc/ __init__.py etc.? Cheers Stéfan
Stefan van der Walt wrote:
On Thu, Mar 29, 2007 at 04:13:28AM +0100, Alexander Schmolck wrote:
In an email some time ago Robert Kern proposed the following directory structure for scikits:
branches/ tags/ trunk/ mlabwrap/ setup.py scikits/ __init__.py mlabwrap/ __init__.py
Shouldn't that be
branches/ tags/ trunk/ scikits/ setup.py __init__.py mlabwrap/ __init__.py gpc/ __init__.py
etc.?
No. The point of scikits is to keep each subpackage independent of the others. It is very difficult to build each subpackage separately from the others if they are in that kind of structure. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
On Thu, Mar 29, 2007 at 08:36:01AM -0500, Robert Kern wrote:
Shouldn't that be
branches/ tags/ trunk/ scikits/ setup.py __init__.py mlabwrap/ __init__.py gpc/ __init__.py
etc.?
No. The point of scikits is to keep each subpackage independent of the others. It is very difficult to build each subpackage separately from the others if they are in that kind of structure.
So what is the use of having a scikits subdirectory in each module? I'm just trying to understand the structure you propose. Cheers Stéfan
Stefan van der Walt wrote:
On Thu, Mar 29, 2007 at 08:36:01AM -0500, Robert Kern wrote:
Shouldn't that be
branches/ tags/ trunk/ scikits/ setup.py __init__.py mlabwrap/ __init__.py gpc/ __init__.py
etc.? No. The point of scikits is to keep each subpackage independent of the others. It is very difficult to build each subpackage separately from the others if they are in that kind of structure.
So what is the use of having a scikits subdirectory in each module? I'm just trying to understand the structure you propose.
scikits is a namespace package. While each subpackage will be built separately, they will still be scikits.mlabwrap, scikits.gpc, etc. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
On Thu, Mar 29, 2007 at 09:17:05AM -0500, Robert Kern wrote:
So what is the use of having a scikits subdirectory in each module? I'm just trying to understand the structure you propose.
scikits is a namespace package. While each subpackage will be built separately, they will still be scikits.mlabwrap, scikits.gpc, etc.
OK, I see. I wrote a ctypes wrapper for Birchfield's KLT tracker, and I'd like to see Polygon (a gpc wrapper) maintained somewhere. I also have a numpy-ported NURBS wrapper. Then there's also some FFMpeg wrapper code to read/write avi files. Are any of these suitable candidates for scikits? Cheers Stéfan
Alexander Schmolck wrote:
1.a. source code and svn organization '''''''''''''''''''''''''''''''''''''
In an email some time ago Robert Kern proposed the following directory structure for scikits:
branches/ tags/ trunk/ mlabwrap/ setup.py scikits/ __init__.py mlabwrap/ __init__.py
The current directory structure misses the scikits/ dir and the second mlabwrap/ dir; instead it has mlabwrap.py at top-level (as well as the utility libraries awmstools.py and awmsmeta.py and the c++-extension mlabraw.cpp). Independent of scikits conventions, it seems that making mlabwrap a package and stuffing everything it needs inside (i.e. the utility libs and mlabraw) rather than polluting the toplevel module-namespace seems desirable
I assume this question reflects my svn-newbie status, but why doesn't the scikits structure look something like this, given that as I understand it scikits are meant to be essentially independent (and hence independely versioned) projects under a common namespace?
mlabwrap/ branches/ tags/ trunk/ setup.py scikits/ __init__.py mlabwrap/ __init__.py some_other_scikit/ branches/ ...
It somehow seems to me that tags and branches should apply to individual projects and that one could still do convenient releases and svn-checkouts of scikits/ as a whole by using e.g. svn:externals to the individual projects.
branches/ and tags/ directories can be shared between projects. There's nothing special about a branch or a tag in SVN; they are just copies. Just make sure you name your branches and tags appropriately (mlabwrap-1.0, etc.). I would like all of the scikits to be under one trunk, though, for conveniently grabbing the current source of all of the scikits without also grabbing every branch and tag. Branches and tags take little space on the server, but in checkouts, there would be a tremendous amount of duplication.
As I said, I'm not really knowledgable about svn, but I'd like to understand the logic a bit better, because I'm also trying to work out what to do about the awmsmeta.py and awmstools.py stuff, which isn't really a part of mlabwrap as such. I see three ways of dealing with the dependency on the utility modules:
D1. Drop it and copy-and-paste the needed bits into mlabwrap.py (certainly possible, only a few functions are needed; OTOH it seems a bit ugly) D2. Rely on cheeseshop (but I don't have the impression that it is equally acceptable and painless for a python module to download and install dependencies as it is for a CTAN module) D3. Stuff copies into mlabwrap/ and do a relative import
D3. seems most attractive to me at the moment but one would still have to figure out where the VC of awmstools.py and awmsmeta.py would "live"; I thought it would be cleanest if they were included as svn:externals and not directly part of the mlabwrap svn tree, so I thought maybe something like this (ignoring the question of where branches/ trunk/ and tags/ go exactly):
mlabwrap/ setup.py scikits/ __init__.py README.txt # etc.
This should be one level up.
mlabraw.py # dummy importing mlabwrap/mlabraw, for backwards comp.
This shouldn't be here. I'd just put it into the toplevel mlabwrap/ directory and not install it. Just have it there for people to use if they need backwards compatibility.
mlabwrap/ __init__.py -> awmstools.py -> awmsmeta.py
This seems appropriate.
mlabwraw.cpp test/ test_mlabwrap.py # etc.
This directory should move into mlabwrap/scikits/mlabwrap/ or even mlabwrap/.
I also assume we want a branch for the 1.0.x version, a tag for the 1.0 release (when it's ready) and that development of the next version will continue in the trunk.
Comments?
Also are there any strong feelings/established procedures concerning __init__.py? The generic name is somewhat inconvient in various contexts (e.g. buffer switching im emacs) so I thought about making __init__.py a dummy importing ./_mlabwrap.py or so.
Leave scikits/__init__.py empty. Do what you like with scikits/mlabwrap/__init__.py . For a package as small as yours, importing everything from mlabwrap.py is reasonable. Then people will only have to import scikits.mlabwrap instead of scikits.mlabwrap.mlabwrap .
OK, sorry this is getting so long, on to
2. bringing out mlabwrap-1.0 final: ------------------------------------
Apart from the question what the directory structure should look like, the only issue that I'm currently aware of is that there appears to be a problem with `setup.py` automatically finding out about the matlab version under windows; if no one steps up to do some windows testing I'm also willing to apply some fix I assume will work and release as is (windows users might have to set MATLAB_VERSION etc. in setup.py by hand if it doesn't, which isn't *that* terrible). Actually, one more thing: distutils vs. scipy distutils vs. setuptools -- which one should mlabwrap-1.0 final use? I'm only really fully familiar with the first one.
We need setuptools to handle the scikits namespace package. Does your extension module use numpy? If so it should also use numpy.distutils. Just make sure to import setuptools first. import setuptools from numpy.distutils.core import setup ... -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
Robert Kern <robert.kern@gmail.com> writes: Thanks for the feedback!
I assume this question reflects my svn-newbie status, but why doesn't the scikits structure look something like this, given that as I understand it scikits are meant to be essentially independent (and hence independely versioned) projects under a common namespace?
mlabwrap/ branches/ tags/ trunk/ setup.py scikits/ __init__.py mlabwrap/ __init__.py some_other_scikit/ branches/ ...
It somehow seems to me that tags and branches should apply to individual projects and that one could still do convenient releases and svn-checkouts of scikits/ as a whole by using e.g. svn:externals to the individual projects.
branches/ and tags/ directories can be shared between projects. There's nothing special about a branch or a tag in SVN; they are just copies. Just make sure you name your branches and tags appropriately (mlabwrap-1.0, etc.). I would like all of the scikits to be under one trunk, though, for conveniently grabbing the current source of all of the scikits without also grabbing every branch and tag.
This is actually exactly what I thought svn:externals would solve (but maybe I don't understand the problems associated with that solution correctly): mlabwrap/ branches/ tags/ trunk/ ... some_other_scikit/ branches/ ... scikits trunk scikits -> mlabwrap/trunk/ # -> = external -> some_other_scikit/trunk/ ... tags ... Thus, if you want all scikits, you just checkout above scikits sub-dir. (I presumably haven't got the directory-structure quite right, but I hope it doesn't matter for getting the point across) On the other hand, if the tags/ and branches/ directories are shared between all projects, you need to prefix all tags/branches with the project name (which would otherwise not be necessary) and have potentially quite a lot of dirs too look at that don't interest you in the least (e.g. when browsing tags/ or branches/ with trac), and, AFAICT, no particularly easy way to just check out everything related to your project. I assume this could also be fixed by 'symlinking' via externals, but you'd need many, many more (per project: one for the trunk and one for each branch and tag, vs. one for the trunk of each project -- and possibly the odd 'whole' scikits release tag). Maybe the structure I proposed would also make importing and exporting of projects slightly easier (because it mirrors the typical layout of an individual svn project). Speaking of which -- is there something I can do with the existing CVS so that it can be easily imported in the scikits svn (in which case we can get rid of what's already checked in), or would importing the CVS involve a hassle in any case, because then I'll just archive it on sourceforge. The other thing I've been wondering is if such a setup couldn't also be made to accomodate something like Stefan van der Walt's layout proposal, which as far as I can see would allow for the most convenient way possible to grab all scikits and build them: setup.py scikits/ __init__.py -> mlabwrap/ mlabwrap_setup.py __init__.py awmstools.py ... -> some_other_scikit/ some_other_scikit_setup.py ... Couldn't one have a toplevel setup.py that just runs all scikits/DIRNAME/DIRNAME_setup.py's it can find, passing through the command line options (or something along those lines[1])? I.e. the distribution of each scikit would contain the same ``scikits/setup.py`` which just looks for subdirs with *setup.py's which it then calls (the XXX_setup.py's could also move one dir up). To install any subset of scikits from svn one could then just do something like: svn co ...trunk/scikits/{setup.py,scikits{_mlabwrap,some_other_scikit,...}} cd scikits; python setup.py install Just an idea (possibly a bad one :).
mlabwrap/ setup.py scikits/ __init__.py README.txt # etc.
This should be one level up.
OK
mlabraw.py # dummy importing mlabwrap/mlabraw, for backwards comp.
This shouldn't be here. I'd just put it into the toplevel mlabwrap/ directory and not install it. Just have it there for people to use if they need backwards compatibility.
OK
mlabwrap/ __init__.py -> awmstools.py -> awmsmeta.py
This seems appropriate.
Good.
mlabwraw.cpp test/ test_mlabwrap.py # etc.
This directory should move into mlabwrap/scikits/mlabwrap/ or even mlabwrap/.
OK, mlabwrap/test it is then.
Leave scikits/__init__.py empty. Do what you like with scikits/mlabwrap/__init__.py . For a package as small as yours, importing everything from mlabwrap.py is reasonable. Then people will only have to import scikits.mlabwrap instead of scikits.mlabwrap.mlabwrap .
OK, I think I'll have __init__.py just do an import * from ./_mlabwrap.py then.
Actually, one more thing: distutils vs. scipy distutils vs. setuptools -- which one should mlabwrap-1.0 final use? I'm only really fully familiar with the first one.
We need setuptools to handle the scikits namespace package. Does your extension module use numpy?
Yes, but optionally -- it also still works with Numeric (before someone grumbles, Numeric support will be ripped out as soon as 1.0 is released).
If so it should also use numpy.distutils. Just make sure to import setuptools first.
import setuptools from numpy.distutils.core import setup ...
Is there a recipe/template for this somewhere? Googling "scipy setuptools" comes up with <http://projects.scipy.org/ipython/ipython/wiki/SetupTools> as the first hit, which seems to indicate that setuptools is still a bit alpha and the docs can't be trusted if one wants something that actually works. I've currently got a distutils setup.py that in common scenarios builds out of the box with ``python setup.py install`` (it automatically detects if numpy or Numeric is installed and which matlab version it needs to build for). What I'd hope setuptool will add (apart from scikits namespace support) is the ability to easy_install mlabwrap from PyPI, also downloading numpy if required. I'd assume the same applies to most other projects that will live under scikits, so it would be good to establish a standard way to do this and document it somewhere, if there isn't anything around already. Should people think that setuptools is still a bit problematic, I'll just release 1.0final on sourceforge and distutils-based (and outside the scikits hierachy), reserving the scikits treatment for subsequent versions. thanks, 'as Footnotes: [1] Obviously this would need to be fleshed out a bit and maybe it's more hassel than it's worth.
Alexander Schmolck wrote:
Robert Kern <robert.kern@gmail.com> writes:
Thanks for the feedback!
I assume this question reflects my svn-newbie status, but why doesn't the scikits structure look something like this, given that as I understand it scikits are meant to be essentially independent (and hence independely versioned) projects under a common namespace?
mlabwrap/ branches/ tags/ trunk/ setup.py scikits/ __init__.py mlabwrap/ __init__.py some_other_scikit/ branches/ ...
It somehow seems to me that tags and branches should apply to individual projects and that one could still do convenient releases and svn-checkouts of scikits/ as a whole by using e.g. svn:externals to the individual projects. branches/ and tags/ directories can be shared between projects. There's nothing special about a branch or a tag in SVN; they are just copies. Just make sure you name your branches and tags appropriately (mlabwrap-1.0, etc.). I would like all of the scikits to be under one trunk, though, for conveniently grabbing the current source of all of the scikits without also grabbing every branch and tag.
This is actually exactly what I thought svn:externals would solve (but maybe I don't understand the problems associated with that solution correctly):
mlabwrap/ branches/ tags/ trunk/ ... some_other_scikit/ branches/ ...
scikits trunk scikits -> mlabwrap/trunk/ # -> = external -> some_other_scikit/trunk/ ... tags ...
Thus, if you want all scikits, you just checkout above scikits sub-dir. (I presumably haven't got the directory-structure quite right, but I hope it doesn't matter for getting the point across)
On the other hand, if the tags/ and branches/ directories are shared between all projects, you need to prefix all tags/branches with the project name (which would otherwise not be necessary) and have potentially quite a lot of dirs too look at that don't interest you in the least (e.g. when browsing tags/ or branches/ with trac), and, AFAICT, no particularly easy way to just check out everything related to your project. I assume this could also be fixed by 'symlinking' via externals, but you'd need many, many more (per project: one for the trunk and one for each branch and tag, vs. one for the trunk of each project -- and possibly the odd 'whole' scikits release tag).
By and large, it simply doesn't matter to "get everything related to your project". Believe me, you don't want all of branches/ and tags/ in a checkout. On the other hand, "getting all of the packages in scikits" does matter. IMO, the inconvenience of prefixing your branches and tags is secondary to the performance problems of svn:external, which slows down all checkouts and updates for everyone (although I'll have to double-check that claim for svn:external links within the same repository). Also, it appears that Trac doesn't like svn:external to other repositories; I'm not sure if that extends to svn:external within the same repository. http://trac.edgewall.org/wiki/TracFaq#DoesTracsupportsvn:externalsubversionr...
Maybe the structure I proposed would also make importing and exporting of projects slightly easier (because it mirrors the typical layout of an individual svn project). Speaking of which -- is there something I can do with the existing CVS so that it can be easily imported in the scikits svn (in which case we can get rid of what's already checked in), or would importing the CVS involve a hassle in any case, because then I'll just archive it on sourceforge.
I don't know. You'll have to read the cvs2svn documentation.
The other thing I've been wondering is if such a setup couldn't also be made to accomodate something like Stefan van der Walt's layout proposal, which as far as I can see would allow for the most convenient way possible to grab all scikits and build them:
setup.py scikits/ __init__.py -> mlabwrap/ mlabwrap_setup.py __init__.py awmstools.py ... -> some_other_scikit/ some_other_scikit_setup.py
Having two ways to install something is just begging for trouble.
... Couldn't one have a toplevel setup.py that just runs all scikits/DIRNAME/DIRNAME_setup.py's it can find, passing through the command line options (or something along those lines[1])?
That's unworkable. I've tried.
If so it should also use numpy.distutils. Just make sure to import setuptools first.
import setuptools from numpy.distutils.core import setup ...
Is there a recipe/template for this somewhere? Googling "scipy setuptools" comes up with
<http://projects.scipy.org/ipython/ipython/wiki/SetupTools>
as the first hit, which seems to indicate that setuptools is still a bit alpha and the docs can't be trusted if one wants something that actually works.
Fernando's a curmudgeon, and that page is old. Ignore him. :-) Like I said, just import setuptools before you import numpy.distutils. Then use numpy.distutils as normal to handle all of the building and stuff. setuptools adds some keywords to setup() that you should also provide, namely namespace_packages=['scikits'], That's all that's necessary. There's no particular magic to combining setuptools and numpy.distutils. Read the setuptools documentation for other keyword arguments the setup() that you might want to use for the extra features it provides, like install_requires. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
On 3/29/07, Robert Kern <robert.kern@gmail.com> wrote:
Is there a recipe/template for this somewhere? Googling "scipy setuptools" comes up with
<http://projects.scipy.org/ipython/ipython/wiki/SetupTools>
as the first hit, which seems to indicate that setuptools is still a bit alpha and the docs can't be trusted if one wants something that actually works.
Fernando's a curmudgeon, and that page is old. Ignore him. :-)
I certainly won't dispute the curmudgeon part :) And that page /is/ indeed old: if the problems that led me to develop my personal flavor of setuptools-allergy have all been solved, then my strategy will have worked out like a charm: wait things out, let a few high-profile projects start using setuptools and run into all these problems, and let that force the technical reality of setuptools catch up to the marketing. I'm sure the system has lots of good features, and it's even likely we may end up using it a lot in ipython for 1.0. By now things are probably a lot better than 2 years ago. Cheers, f
Robert Kern <robert.kern@gmail.com> writes: [on the question whether the svn-layout {branch,tags,trunk}/subproject (Robert) or subproject/{branch,tags,trunk} (me) is preferable for scikits]
By and large, it simply doesn't matter to "get everything related to your project". Believe me, you don't want all of branches/ and tags/ in a checkout.
I have done so in order to get some overview over a project that I was unfamiliar with, but I agree that that's hardly an important use case. Migration convenience seems more relevant, but unless rearranging dir structure in svn repositories is really hard, that's not a big factor either.
On the other hand, "getting all of the packages in scikits" does matter.
Sure, but not that often either. Why would many people want to frequently update the svn versions of all of several unrelated and fairly special purpose scientific packages? I would assume the typical case is that someone hacks e.g. mlabwrap and updates that from time to time rather than a dozen other porjects.
IMO, the inconvenience of prefixing your branches and tags is secondary to the performance problems of svn:external, which slows down all checkouts and updates for everyone
Only dirs with svn:externals would be affected, right? Since the only such dir would be /scikits, which most people won't checkout, I don't think there would be much of an impact (see above).
(although I'll have to double-check that claim for svn:external links within the same repository).
I think you're right: I added an external 'ext' on mlabwrap to mlabwrap/test, and it appears to slow down checkout; I've got a very old svn client, but you can try yourself: svn co http://scipy.org/svn/scikits/trunk/mlabwrap/
Also, it appears that Trac doesn't like svn:external to other repositories; I'm not sure if that extends to svn:external within the same repository.
http://trac.edgewall.org/wiki/TracFaq#DoesTracsupportsvn:externalsubversionr...
For this purpose it seems to work fine: <https://projects.scipy.org/scipy/scikits/browser/trunk/mlabwrap> It just displays the svn:externals property, which is all you want here, I think. Anyway, the choice between either layout (i.e. subproject internal/external {tags,branches,trunk}) seems not terribly important. Each approach has advantages and disadvantages, and apparently both are common; the 'official' SVN book authors prefer internal, as do I, but write about the toplevel {tags,branches,trunk} approach: <http://svnbook.red-bean.com/nightly/en/svn-book.html#svn.reposadmin.projects...> There's nothing particularly incorrect about such a layout, but it may or may not seem as intuitive for your users. Especially in large, multi-project situations with many users, those users may tend to be familiar with only one or two of the projects in the repository. But the projects-as-branch-siblings tends to de-emphasize project individuality and focus on the entire set of projects as a single entity. That's a social issue though. We like our originally suggested arrangement for purely practical reasons: it's easier to ask about (or modify, or migrate elsewhere) the entire history of a single project when there's a single repository path that holds the history entirepast, present, tagged, branched andfor that project and that project alone.
Maybe the structure I proposed would also make importing and exporting of projects slightly easier (because it mirrors the typical layout of an individual svn project). Speaking of which -- is there something I can do with the existing CVS so that it can be easily imported in the scikits svn (in which case we can get rid of what's already checked in), or would importing the CVS involve a hassle in any case, because then I'll just archive it on sourceforge.
I don't know. You'll have to read the cvs2svn documentation.
Sorry, I should have phrased this better. I have already read the cvs2svn docs and converted the CVS repository (just a single trunk, no branches or tags). If I put a tar.bz2 of that repository on the web, can someone with admin rights easily install it in lieu of the existing repository? Is Berkeley db fine as backend? If it's not easy and quick to do, I'm also happy to loose the revision history and abandon conversion attempts.
The other thing I've been wondering is if such a setup couldn't also be made to accomodate something like Stefan van der Walt's layout proposal, which as far as I can see would allow for the most convenient way possible to grab all scikits and build them:
setup.py scikits/ __init_xg_.py -> mlabwrap/ mlabwrap_setup.py __init__.py awmstools.py ... -> some_other_scikit/ some_other_scikit_setup.py
Having two ways to install something is just begging for trouble.
I wasn't advocating two ways; you always call scikits/setup.py and it just installs different amounts of stuff depending on how many subdirs you've checked out.
Couldn't one have a toplevel setup.py that just runs all scikits/DIRNAME/DIRNAME_setup.py's it can find, passing through the command line options (or something along those lines[1])?
That's unworkable. I've tried.
I suspected that it might turn out to be. Pitty.
If so it should also use numpy.distutils. Just make sure to import setuptools first.
import setuptools from numpy.distutils.core import setup ...
Is there a recipe/template for this somewhere? Googling "scipy setuptools" comes up with
<http://projects.scipy.org/ipython/ipython/wiki/SetupTools>
as the first hit, which seems to indicate that setuptools is still a bit alpha and the docs can't be trusted if one wants something that actually works.
Fernando's a curmudgeon, and that page is old. Ignore him. :-)
OK, I'm going the``ez_setup.py`` way.
Like I said, just import setuptools before you import numpy.distutils. Then use numpy.distutils as normal to handle all of the building and stuff. setuptools adds some keywords to setup() that you should also provide, namely
namespace_packages=['scikits'],
That's all that's necessary. There's no particular magic to combining setuptools and numpy.distutils.
Well, it's not quite obvious how to fully take advantage of setuptools, though. One of the main reasons for using it is that it's meant to download and install depenedencies automatically, but that can't work if my setup.py import something from its sole dependency (numpy) to start with. Surely there must be some way to write packages that depend on numpy but can be installed automatically (and download numpy if required)? And why do I need to use numpy.distutils in the first place? I find <http://www.scipy.org/Documentation/numpy_distutils> rather unhelpful and I didn't find any other documentation. Another thing: should ``scikits/__init__.py`` be really empty? From <http://peak.telecommunity.com/DevCenter/setuptools#namespace-packages> it looks a bit to me like it should contain: __import__('pkg_resources').declare_namespace(__name__) ? Finally, what is the preferred download url for scikits projects? Should I continue to host the file-release on SF, or should they go somewhere else? Same for the webpage. So I think some kind of template for scikit authors would be useful and I'd suggest that once I've got setup.py etc. ironed out I put some info for other prospective scikit authors on a wikipage on scipy.org -- what would be a good place? Finally, just to double check, does this directory structure look good to you: mlabwrap/ setup.py README.txt tests/ # N.B. renamed from 'test' test_mlabwrap.py ... scikits/ __init__.py # empty mlabwrap/ __init__.py _mlabwrap.py mlabraw.py -> awmstools.py -> awmsmeta.py ? thanks, 'as
Alexander Schmolck wrote:
Robert Kern <robert.kern@gmail.com> writes:
Anyway, the choice between either layout (i.e. subproject internal/external {tags,branches,trunk}) seems not terribly important.
Fine. Please let's drop the issue, then.
Maybe the structure I proposed would also make importing and exporting of projects slightly easier (because it mirrors the typical layout of an individual svn project). Speaking of which -- is there something I can do with the existing CVS so that it can be easily imported in the scikits svn (in which case we can get rid of what's already checked in), or would importing the CVS involve a hassle in any case, because then I'll just archive it on sourceforge. I don't know. You'll have to read the cvs2svn documentation.
Sorry, I should have phrased this better. I have already read the cvs2svn docs and converted the CVS repository (just a single trunk, no branches or tags). If I put a tar.bz2 of that repository on the web, can someone with admin rights easily install it in lieu of the existing repository?
Is there no conversion tool that simply puts revisions into an existing directory of a repository instead of making a new repository?
Is Berkeley db fine as backend?
No. We use the fsfs.
If so it should also use numpy.distutils. Just make sure to import setuptools first.
import setuptools from numpy.distutils.core import setup ... Is there a recipe/template for this somewhere? Googling "scipy setuptools" comes up with
<http://projects.scipy.org/ipython/ipython/wiki/SetupTools>
as the first hit, which seems to indicate that setuptools is still a bit alpha and the docs can't be trusted if one wants something that actually works. Fernando's a curmudgeon, and that page is old. Ignore him. :-)
OK, I'm going the``ez_setup.py`` way.
No, ez_setup.py is deprecated. That part of the setuptools docs is out of date. When the Cheeseshop comes back up read this page with up-to-date information about how to get going with setuptools: http://cheeseshop.python.org/pypi/setuptools
Like I said, just import setuptools before you import numpy.distutils. Then use numpy.distutils as normal to handle all of the building and stuff. setuptools adds some keywords to setup() that you should also provide, namely
namespace_packages=['scikits'],
That's all that's necessary. There's no particular magic to combining setuptools and numpy.distutils.
Well, it's not quite obvious how to fully take advantage of setuptools, though. One of the main reasons for using it is that it's meant to download and install depenedencies automatically, but that can't work if my setup.py import something from its sole dependency (numpy) to start with. Surely there must be some way to write packages that depend on numpy but can be installed automatically (and download numpy if required)?
Not really. The dependencies that you can specify are requirements for using the package after it is installed, not requirements for building the package. The structure of distutils setup.py files pretty much enforces this. setuptools doesn't really get to do anything until setup() is called, i.e. after you've used your dependencies. You might be able to hack something together with pkg_resources.WorkingSet.resolve(), though. http://peak.telecommunity.com/DevCenter/PkgResources#workingset-methods-and-...
And why do I need to use numpy.distutils in the first place?
<shrug> You don't strictly have to. numpy.get_include() is probably sufficient for you, but it has the same problem.
Another thing: should ``scikits/__init__.py`` be really empty? From
<http://peak.telecommunity.com/DevCenter/setuptools#namespace-packages>
it looks a bit to me like it should contain:
__import__('pkg_resources').declare_namespace(__name__)
?
Yes, apologies.
Finally, what is the preferred download url for scikits projects? Should I continue to host the file-release on SF, or should they go somewhere else?
I recommend putting them on the Cheeseshop.
Same for the webpage.
scipy.org wiki if you can.
So I think some kind of template for scikit authors would be useful and I'd suggest that once I've got setup.py etc. ironed out I put some info for other prospective scikit authors on a wikipage on scipy.org -- what would be a good place?
http://projects.scipy.org/scipy/scikits
Finally, just to double check, does this directory structure look good to you:
mlabwrap/ setup.py README.txt tests/ # N.B. renamed from 'test' test_mlabwrap.py ... scikits/ __init__.py # empty mlabwrap/ __init__.py _mlabwrap.py mlabraw.py -> awmstools.py -> awmsmeta.py
?
Yes. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
On Apr 6, 2007, at 10:38 PM, Robert Kern wrote:
Alexander Schmolck wrote:
Sorry, I should have phrased this better. I have already read the cvs2svn docs and converted the CVS repository (just a single trunk, no branches or tags). If I put a tar.bz2 of that repository on the web, can someone with admin rights easily install it in lieu of the existing repository?
Is there no conversion tool that simply puts revisions into an existing directory of a repository instead of making a new repository?
cvs2svn can do this. http://cvs2svn.tigris.org/faq.html talks about an "options file method" that I haven't used, but the older dumpfile mechanism worked fine for us.
Robert Kern <robert.kern@gmail.com> writes:
If so it should also use numpy.distutils. Just make sure to import setuptools first.
import setuptools from numpy.distutils.core import setup ... Is there a recipe/template for this somewhere? Googling "scipy setuptools" comes up with
<http://projects.scipy.org/ipython/ipython/wiki/SetupTools>
as the first hit, which seems to indicate that setuptools is still a bit alpha and the docs can't be trusted if one wants something that actually works. Fernando's a curmudgeon, and that page is old. Ignore him. :-)
OK, I'm going the``ez_setup.py`` way.
No, ez_setup.py is deprecated. That part of the setuptools docs is out of date. When the Cheeseshop comes back up read this page with up-to-date information about how to get going with setuptools:
OK.
Surely there must be some way to write packages that depend on numpy but can be installed automatically (and download numpy if required)?
Not really. The dependencies that you can specify are requirements for using the package after it is installed, not requirements for building the package. The structure of distutils setup.py files pretty much enforces this. setuptools doesn't really get to do anything until setup() is called, i.e. after you've used your dependencies.
You might be able to hack something together with pkg_resources.WorkingSet.resolve(), though.
http://peak.telecommunity.com/DevCenter/PkgResources#workingset-methods-and-...
Sometimes
And why do I need to use numpy.distutils in the first place?
<shrug> You don't strictly have to. numpy.get_include() is probably sufficient for you, but it has the same problem.
Another thing: should ``scikits/__init__.py`` be really empty? From
<http://peak.telecommunity.com/DevCenter/setuptools#namespace-packages>
it looks a bit to me like it should contain:
__import__('pkg_resources').declare_namespace(__name__)
?
Yes, apologies.
OK, I think the upshot of all this is that I'll figure out how to do a robust and user-friendly setuptools-based package another time. I don't want to delay the release of 1.0 any further so I'll release mlabwrap-1.0final on SF as a distutils based install with the old (scikits-less) package structure. Since post-1.0 mlabwrap will represent a break anyway (Numeric support will be dropped, newer version of python and matlab might be required and the interface might change in some not backwards-compatible ways), the need to change the important statement is maybe not a bad thing.
Finally, what is the preferred download url for scikits projects? Should I continue to host the file-release on SF, or should they go somewhere else?
I recommend putting them on the Cheeseshop.
Thanks, will do.
Same for the webpage.
scipy.org wiki if you can.
I could make http://www.scipy.org/MlabWrap the project webpage and move its current (developer-only contents) into some subdir -- does that sound reasonable?
So I think some kind of template for scikit authors would be useful and I'd suggest that once I've got setup.py etc. ironed out I put some info for other prospective scikit authors on a wikipage on scipy.org -- what would be a good place?
I tried to create a trac account ('aschmolck') there but it didn't work; a new account seems to have been created (the name is taken) but I can't log into it. Do I need some other type of account first in order to be able to create the trac account? When I click on "create account" I get an authorization dialog, but the just created login and password don't work (I verified by going through the same process with login/passwd test_user). thanks, 'as
On Wednesday 28 March 2007 10:13 pm, Alexander Schmolck wrote:
Shouldn't having a svn account mean that I'd be able to succesfully log in to the trac interface at <https://projects.scipy.org/scipy/scikits/browser/trunk/mlabwrap> using the same account? I can't either use the AlexanderSchmolck login I created on scipy.org, nor the POP3/SVN login a.schmolck@gmail.com[1] I received an email from webmin@scipy.org on Friday[3]; however I *can* commit to the repository using svn (I've sent an email to webmin@scipy.org, but haven't heard anything back).
Trac accounts, moin accounts, and SVN accounts are all separate. You'll need to register and fill in your email address in the settings section for each Trac you use. If you had a Scipy Trac account, it was copied over to the Scikits trac when it was created. However, the settings section was not synchronized. If you want to get email about tickets, you must enter your email address in the settings section of each trac you use. I will start to receive email sent to webmin@scipy.org now. Thanks, Jeff Strunk IT Administrator Enthought, Inc.
Jeff Strunk <jstrunk@enthought.com> writes:
On Wednesday 28 March 2007 10:13 pm, Alexander Schmolck wrote:
Shouldn't having a svn account mean that I'd be able to succesfully log in to the trac interface at <https://projects.scipy.org/scipy/scikits/browser/trunk/mlabwrap> using the same account? I can't either use the AlexanderSchmolck login I created on scipy.org, nor the POP3/SVN login a.schmolck@gmail.com[1] I received an email from webmin@scipy.org on Friday[3]; however I *can* commit to the repository using svn (I've sent an email to webmin@scipy.org, but haven't heard anything back).
Trac accounts, moin accounts, and SVN accounts are all separate. You'll need to register and fill in your email address in the settings section for each Trac you use.
I see, thanks -- but is it customary/acceptable to use the same login (and possibly password, unless some systems are less secure than others) for all these accounts? Doing so would seem convenient...
If you had a Scipy Trac account, it was copied over to the Scikits trac when it was created.
No, I just had a moin account (AlexanderSchmolck)
If you want to get email about tickets, you must enter your email address in the settings section of each trac you use.
OK, I'll create a trac login as soon as I figured out what the login name ought to be :) cheers, 'as
On Thursday 29 March 2007 8:52 pm, Alexander Schmolck wrote:
Jeff Strunk <jstrunk@enthought.com> writes: ...
Trac accounts, moin accounts, and SVN accounts are all separate. You'll need to register and fill in your email address in the settings section for each Trac you use.
I see, thanks -- but is it customary/acceptable to use the same login (and possibly password, unless some systems are less secure than others) for all these accounts? Doing so would seem convenient...
The only account new users can't create for themselves is an svn account. SSL is not is wide use. You can use it to access svn and trac, but not moin. I don't know that very many people are using SSL for either. You can use whatever username and password you desire on any of them. Thanks, Jeff
participants (6)
-
Alexander Schmolck -
Fernando Perez -
Jeff Strunk -
Jonathan Guyer -
Robert Kern -
Stefan van der Walt