Build Process and Version Control
Hi, here is a summary of what we got so far with the Mac OS X build process: - We found it useful to put /Library/Frameworks/Python.framework/Versions/2.x under version control with git. This lets us get rid of the paver bootstrap thingy, since we just switch to a branch via ``git checkout numpy``. We can have a full-blown polluted Python and a minimial numpy Python in parallel, and we can revert to the built numpy libraries when we find some issue later. We will note the sha of the numpy system commit in the commits in the numpy build git repo. - We also modified the Pythons via a print statement in sitecustomize.py to tell what numpy is running on startup, this goes to the logfiles etc., it also tells the branch family of the system git repo (hardcoded). - We established a numpy git "build repo". It'll contain all the dmgs and so on, along with the log files of the build process. I will push soon. Its location is http://github.com/friedrichromstedt/numpy. We tried hard to make good design choices about the branches, but this is discussable, and might be easy to revert in this early stage where noone has fetched so far. - We also set up the "numpy build site", it's here: http://vincentdavis.info/, or for the hard-core people here :-) http://vincentdavis.info/numpy-build-doc. It's not that hard-core at all. Just a Sphinx site where we document what we're doing to our build system. Atm I'm not *that* satisfied with it, because I tend to document *too much* when new to some subject. I believe it will turn out as a well-readable detailed instruction for ourselves, usable to others too, and then we just crossref that instruction instead of writing down all the stuff again and again. Currently the numpy-build-docs are in another repo, http://github.com/friedrichromstedt/numpy-build-doc, I'm not sure if it is useful to merge this into my numpy repo. Pro: Users of the numpy repo will get all the stuff with little overhead. Con: Users of the numpy-build-doc repo will have *lots* of overhead. But I guess the latter are rather rare. Please comment. The process is open for others to chime in. The Sphinx site is ready to host also build logs of other systems. The sources are included in the git repo (numpy-build-docs), together with a current html build. Please use the Sphinx site online (http://vincentdavis.info/numpy-build-doc) and not the git repo, but I'll push soon. Documentation goes via Python functional programming, but this are details. In fact, Sphinx just documents Python functions. We are working towards an automatisation of build process steps using this Python functions. Yet another build system. But this is future. Please refrain from fetching anything until I did the authoritative push, so that you'll have a working system, which is at least beta and not rather alpha like the current things I put on github. Friedrich
Hi Friedrich, On Mon, Oct 25, 2010 at 6:08 PM, Friedrich Romstedt <friedrichromstedt@gmail.com> wrote:
Hi,
here is a summary of what we got so far with the Mac OS X build process:
Thanks for the detailed info on what you are doing. First maybe a general comment: the current release scripts work quite well, so it's probably not a very good idea to throw all that out. Also, I've tried to improve the docs on how the binaries are built and make them as complete as possible over the last half year, so I would appreciate you and Vincent pointing out any gaps or creating some patches to those docs.
- We found it useful to put /Library/Frameworks/Python.framework/Versions/2.x under version control with git. This lets us get rid of the paver bootstrap thingy, since we just switch to a branch via ``git checkout numpy``. We can have a full-blown polluted Python and a minimial numpy Python in parallel, and we can revert to the built numpy libraries when we find some issue later. We will note the sha of the numpy system commit in the commits in the numpy build git repo.
Not sure about the motivation here. Paver and virtualenv (the "bootstrap thingy") are widely used and relatively mature tools, why hack something together that does the same job?
- We also modified the Pythons via a print statement in sitecustomize.py to tell what numpy is running on startup, this goes to the logfiles etc., it also tells the branch family of the system git repo (hardcoded).
First time I heard about this sitecustomize.py, sounds like not the most robust thing I've ever heard of...
- We established a numpy git "build repo". It'll contain all the dmgs and so on, along with the log files of the build process.
I like the idea of recording all logs and having them accessible for later reference. Very large binaries under version control, less so.
I will push soon. Its location is http://github.com/friedrichromstedt/numpy. We tried hard to make good design choices about the branches, but this is discussable, and might be easy to revert in this early stage where noone has fetched so far.
- We also set up the "numpy build site", it's here: http://vincentdavis.info/, or for the hard-core people here :-) http://vincentdavis.info/numpy-build-doc. It's not that hard-core at all. Just a Sphinx site where we document what we're doing to our build system. Atm I'm not *that* satisfied with it, because I tend to document *too much* when new to some subject. I believe it will turn out as a well-readable detailed instruction for ourselves, usable to others too, and then we just crossref that instruction instead of writing down all the stuff again and again.
Currently the numpy-build-docs are in another repo, http://github.com/friedrichromstedt/numpy-build-doc, I'm not sure if it is useful to merge this into my numpy repo. Pro: Users of the numpy repo will get all the stuff with little overhead. Con: Users of the numpy-build-doc repo will have *lots* of overhead. But I guess the latter are rather rare. Please comment.
Why not add to http://projects.scipy.org/numpy/wiki/MakingReleases? Or otherwise a new plain reST file under doc/. Between the docs in the git repo, the Trac wiki and the wiki on scipy.org I think we have enough places to add docs. And maintain them... Again, I appreciate your work and am not trying to stop you from making improvements. But please try to improve what's already there, or at least first point out why that is difficult so we can discuss. Cheers, Ralf
The process is open for others to chime in. The Sphinx site is ready to host also build logs of other systems. The sources are included in the git repo (numpy-build-docs), together with a current html build. Please use the Sphinx site online (http://vincentdavis.info/numpy-build-doc) and not the git repo, but I'll push soon.
Documentation goes via Python functional programming, but this are details. In fact, Sphinx just documents Python functions. We are working towards an automatisation of build process steps using this Python functions. Yet another build system. But this is future.
Please refrain from fetching anything until I did the authoritative push, so that you'll have a working system, which is at least beta and not rather alpha like the current things I put on github.
Friedrich _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
participants (2)
-
Friedrich Romstedt -
Ralf Gommers