Hi all, We had a great Sprint at Berkeley over last weekend. Jarrod deserves a huge hand for organizing it and Fernando should be also congradulated for making the Sprint a productive communication session with a lot of different people. Going forward, there will be a relatively informal SciPy board whose purpose is to keep SciPy (and NumPy) moving forward. Currently, this board consists of (alphabetically) Eric Jones Robert Kern Jarrod Millman Travis Oliphant Our goal is to clean up SciPy and get it ready for 1.0 release over the next year or so (which will need lots of help from the community). If anybody else is interested in serving on this board, just send me email. As part of this goal, we will be having regular "sprints" as well virtual "bug-days" and "doc-days" where people who want to participate using IRC can join in and coordinate efforts. There will be at least one bug-day or doc-day every month over the next year (on the last Friday of the month). The first one is a "doc-day" which will be held Friday on December 28, 2007 (getting started on New Year's resolutions early). This doc-day will be virtual where anyone with an internet connection can join in on the scipy channel on irc.freenode.net. At least one board member will be available at each "doc-day" or "bug-day" (even if we have to recruit board members to make it happen :-) ) The recent Sprint was very helpful. Jarrod is putting together some material from the Sprint. But, I wanted to provide over-view information for those who may be interested in what happend. Summary: A lot of great discussion took place (and some fine actual coding by a few) which resulted in the following plans: Schedule ------------ * NumPy 1.0.5 in mid January * SciPy 0.7.0 in mid March to April * NumPy 1.1 by August 2008 (may slip a bit depending on what is wanted to be included) The plans below are for NumPy 1.0.5 and SciPy 0.7.0 unless otherwise noted. IO ---- * scipy.io will be gutted and what functionality remains will be placed in numpy.io. * scipy.io will be a place for file readers and writers for various data formats (data, audio, video, images, matlab, excel, etc.) * NumPy will get a standard binary file format (.npy/.npz) for arrays/groups_of_arrays. * NumPy will be trying to incorporate some of matplotlib's csv2rec and rec2csv functionality. * Pickling arrays will be discouraged (you will still be able to do it, we will just try to not make it seem that it is the "default" way to save arrays). Testing --------- * scipy unit-testing will be "nose-compliant" and therefore nose will be required to run the SciPy tests. * NumPy will still use the current testing framework but will support SciPy's desire to be nose-compliant. NumPy 1.1 tests may move to just being "nose-compliant" * The goal is to make tests easier for contributors to write. Weave --------- * weave will not move into NumPy yet, but possibly at NumPy 1.1, there could be a separate package containing all the "wrapping" support code for NumPy in a more unified fashion (if somebody is interested in this, it is a great time to jump in). Sandbox ----------- * the scipy sandbox is disappearing (except for user playgrounds) and useful code in it will be placed in other areas. Python versions -------------------- * SciPy 0.7.0 will require Python 2.4 (we can now use decorators for SciPy). * NumPy will still be useful with Python 2.3 until at least 1.1 Other discussions ---------------------- * numpy-scons will be a separate package for now for building extensions with scons (we need experience to figure out what to do with it). * fixes to repr for numpy float scalars were put in place * Thanks to Rob Falck scipy.optimize grew slsqp (sequential least-squares programming) method (allows for equality and inequality constraints). The code by Dieter Kraft was wrapped. * We will be working to coordinate efforts with William Stein (of SAGE fame) in the future. Sage developers will be coming to Austin at the end of February to do some cooperative sprinting. * Brian Granger is working on a parallel version of NumPy that is very interesting. Deprecation approaches ------------------------------- Functions in SciPy that are disappearing will be "deprecated" with appendages to the docstring to explain how to do it differently. The deprecation will issue a warning when the function is run. In the next version, the function will disappear. Once SciPy hits 1.0, the deprecation paradigm will be a bit more conservative. A lot of fabulous things are happening with SciPy. It is an exciting time to be a part of it. There are a lot of ways to jump in and participate so feel free. If there is something you think needs addressing, then please share it. We may have a simple PEP process in the future, but for now the rule is basically "working code." Best regards, -Travis O.
Thanks to all. This sounds great. Ryan On Dec 19, 2007 12:52 PM, Travis E. Oliphant <oliphant@enthought.com> wrote:
Hi all,
We had a great Sprint at Berkeley over last weekend. Jarrod deserves a huge hand for organizing it and Fernando should be also congradulated for making the Sprint a productive communication session with a lot of different people.
Going forward, there will be a relatively informal SciPy board whose purpose is to keep SciPy (and NumPy) moving forward. Currently, this board consists of (alphabetically)
Eric Jones Robert Kern Jarrod Millman Travis Oliphant
Our goal is to clean up SciPy and get it ready for 1.0 release over the next year or so (which will need lots of help from the community). If anybody else is interested in serving on this board, just send me email.
As part of this goal, we will be having regular "sprints" as well virtual "bug-days" and "doc-days" where people who want to participate using IRC can join in and coordinate efforts. There will be at least one bug-day or doc-day every month over the next year (on the last Friday of the month). The first one is a "doc-day" which will be held Friday on December 28, 2007 (getting started on New Year's resolutions early). This doc-day will be virtual where anyone with an internet connection can join in on the scipy channel on irc.freenode.net.
At least one board member will be available at each "doc-day" or "bug-day" (even if we have to recruit board members to make it happen :-) )
The recent Sprint was very helpful. Jarrod is putting together some material from the Sprint. But, I wanted to provide over-view information for those who may be interested in what happend.
Summary:
A lot of great discussion took place (and some fine actual coding by a few) which resulted in the following plans:
Schedule ------------ * NumPy 1.0.5 in mid January * SciPy 0.7.0 in mid March to April * NumPy 1.1 by August 2008 (may slip a bit depending on what is wanted to be included)
The plans below are for NumPy 1.0.5 and SciPy 0.7.0 unless otherwise noted.
IO ---- * scipy.io will be gutted and what functionality remains will be placed in numpy.io. * scipy.io will be a place for file readers and writers for various data formats (data, audio, video, images, matlab, excel, etc.) * NumPy will get a standard binary file format (.npy/.npz) for arrays/groups_of_arrays. * NumPy will be trying to incorporate some of matplotlib's csv2rec and rec2csv functionality. * Pickling arrays will be discouraged (you will still be able to do it, we will just try to not make it seem that it is the "default" way to save arrays).
Testing --------- * scipy unit-testing will be "nose-compliant" and therefore nose will be required to run the SciPy tests. * NumPy will still use the current testing framework but will support SciPy's desire to be nose-compliant. NumPy 1.1 tests may move to just being "nose-compliant" * The goal is to make tests easier for contributors to write.
Weave --------- * weave will not move into NumPy yet, but possibly at NumPy 1.1, there could be a separate package containing all the "wrapping" support code for NumPy in a more unified fashion (if somebody is interested in this, it is a great time to jump in).
Sandbox ----------- * the scipy sandbox is disappearing (except for user playgrounds) and useful code in it will be placed in other areas.
Python versions -------------------- * SciPy 0.7.0 will require Python 2.4 (we can now use decorators for SciPy). * NumPy will still be useful with Python 2.3 until at least 1.1
Other discussions ---------------------- * numpy-scons will be a separate package for now for building extensions with scons (we need experience to figure out what to do with it). * fixes to repr for numpy float scalars were put in place * Thanks to Rob Falck scipy.optimize grew slsqp (sequential least-squares programming) method (allows for equality and inequality constraints). The code by Dieter Kraft was wrapped. * We will be working to coordinate efforts with William Stein (of SAGE fame) in the future. Sage developers will be coming to Austin at the end of February to do some cooperative sprinting. * Brian Granger is working on a parallel version of NumPy that is very interesting.
Deprecation approaches -------------------------------
Functions in SciPy that are disappearing will be "deprecated" with appendages to the docstring to explain how to do it differently. The deprecation will issue a warning when the function is run. In the next version, the function will disappear. Once SciPy hits 1.0, the deprecation paradigm will be a bit more conservative.
A lot of fabulous things are happening with SciPy. It is an exciting time to be a part of it. There are a lot of ways to jump in and participate so feel free. If there is something you think needs addressing, then please share it. We may have a simple PEP process in the future, but for now the rule is basically "working code."
Best regards,
-Travis O.
_______________________________________________ Scipy-dev mailing list Scipy-dev@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-dev
On Dec 19, 2007 6:39 PM, Ryan Krauss <ryanlists@gmail.com> wrote:
Thanks to all. This sounds great.
Ryan
On Dec 19, 2007 12:52 PM, Travis E. Oliphant <oliphant@enthought.com> wrote:
Hi all,
We had a great Sprint at Berkeley over last weekend. Jarrod deserves a huge hand for organizing it and Fernando should be also congradulated for making the Sprint a productive communication session with a lot of different people.
<snip>
* NumPy will get a standard binary file format (.npy/.npz) for arrays/groups_of_arrays.
Will this new binary format contain endianess/type data? I am a bit concerned that we don't yet have a reliable way to distinguish extended precision floats from the coming quad precision as they both tend to use 128 bytes on 64 bit machines. Perhaps extended precision should just be dropped at some point, especially as it is not particularly portable between architectures/compilers. Chuck
Charles R Harris wrote:
On Dec 19, 2007 12:52 PM, Travis E. Oliphant <oliphant@enthought.com <mailto:oliphant@enthought.com>> wrote:
> * NumPy will get a standard binary file format (.npy/.npz) for > arrays/groups_of_arrays.
Will this new binary format contain endianess/type data? I am a bit concerned that we don't yet have a reliable way to distinguish extended precision floats from the coming quad precision as they both tend to use 128 bytes on 64 bit machines. Perhaps extended precision should just be dropped at some point, especially as it is not particularly portable between architectures/compilers.
It uses the dtype.descr to describe the dtype of the array. You can see the implementation here: http://svn.scipy.org/svn/numpy/branches/lib_for_io/format.py If it has holes, I would like to fix them. Can you point me to some documentation on the different quad precision formats? Doesn't IEEE-854 standardize this? There has been some discussion about whether to continue with this format or attempt to read and write a very tiny subset of HDF5, so don't get too attached to the format until it hits the trunk. I'll drop some warnings into the code to that effect. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
On Dec 19, 2007 11:28 PM, Robert Kern <robert.kern@gmail.com> wrote:
Charles R Harris wrote:
On Dec 19, 2007 12:52 PM, Travis E. Oliphant <oliphant@enthought.com <mailto:oliphant@enthought.com>> wrote:
> * NumPy will get a standard binary file format (.npy/.npz) for > arrays/groups_of_arrays.
Will this new binary format contain endianess/type data? I am a bit concerned that we don't yet have a reliable way to distinguish extended precision floats from the coming quad precision as they both tend to use 128 bytes on 64 bit machines. Perhaps extended precision should just be dropped at some point, especially as it is not particularly portable between architectures/compilers.
It uses the dtype.descr to describe the dtype of the array. You can see the implementation here:
http://svn.scipy.org/svn/numpy/branches/lib_for_io/format.py
If it has holes, I would like to fix them. Can you point me to some documentation on the different quad precision formats? Doesn't IEEE-854 standardize this?
There has been some discussion about whether to continue with this format or attempt to read and write a very tiny subset of HDF5, so don't get too attached to the format until it hits the trunk. I'll drop some warnings into the code to that effect.
Here is a PDF version of DRAFT Standard for Floating-Point Arithmetic IEEE P754 <http://www.validlab.com/754R/drafts/archive/2007-10-05.pdf> . Table 2 on page 16 gives a good summary of the proposed formats. I believe P754 is the latest step in the revision of the 754 standard as 754r concluded last year. Note that the proposed standard also adds decimal floats, with decimal digits packed 3 in 10 bits. Here is a bit more from Intel<http://www.intel.com/technology/itj/2007/v11i1/s2-decimal/1-sidebar.htm>, who are apparently working on implementations. Chuck
On Dec 21, 2007 1:20 AM, Charles R Harris <charlesr.harris@gmail.com> wrote:
On Dec 19, 2007 11:28 PM, Robert Kern <robert.kern@gmail.com> wrote:
Charles R Harris wrote:
On Dec 19, 2007 12:52 PM, Travis E. Oliphant <
oliphant@enthought.com
<mailto: oliphant@enthought.com>> wrote:
> * NumPy will get a standard binary file format (.npy/.npz) for > arrays/groups_of_arrays.
Will this new binary format contain endianess/type data? I am a bit concerned that we don't yet have a reliable way to distinguish
precision floats from the coming quad precision as they both tend to use 128 bytes on 64 bit machines. Perhaps extended precision should just be dropped at some point, especially as it is not particularly
extended portable
between architectures/compilers.
It uses the dtype.descr to describe the dtype of the array. You can see the implementation here:
http://svn.scipy.org/svn/numpy/branches/lib_for_io/format.py
If it has holes, I would like to fix them. Can you point me to some documentation on the different quad precision formats? Doesn't IEEE-854 standardize this?
There has been some discussion about whether to continue with this format or attempt to read and write a very tiny subset of HDF5, so don't get too attached to the format until it hits the trunk. I'll drop some warnings into the code to that effect.
Here is a PDF version of DRAFT Standard for Floating-Point Arithmetic IEEE P754 <http://www.validlab.com/754R/drafts/archive/2007-10-05.pdf> . Table 2 on page 16 gives a good summary of the proposed formats. I believe P754 is the latest step in the revision of the 754 standard as 754r concluded last year. Note that the proposed standard also adds decimal floats, with decimal digits packed 3 in 10 bits. Here is a bit more from Intel<http://www.intel.com/technology/itj/2007/v11i1/s2-decimal/1-sidebar.htm>, who are apparently working on implementations.
The actual _use_ of the floating types will depend on the C compilers. Quads will probably show up as long doubles, displacing extended precision doubles. What will happen to BLAS, LAPACK, and all those bits? Quad precision support will likely be added soon enough, there are already quad versions of BLAS out there. Here is a interesting presentation<http://www.csm.ornl.gov/workshops/SOS11/presentations/j_dongarra.pdf>somewhat related to such things. Hey, maybe we should rewrite numpy in FORTRAN ;) Anyway, the current identification of numbers by bit width works fine for stepping/slicing through data and is probably influenced by that implementation detail, but as far as numerics go I think we need to know what is actually *in* those bits and it would be nice to have that method in place early on. Maybe a short UTF-8 header with enough room to actually be descriptive would do the trick, or we could hash longer names to get a short identifier, although hashed values would have to be recognized, not decoded. I think something wordy and descriptive, i.e., "big endian IEEE binary128", would be a good starting point. Chuck
Hi Travis, During the sprint I also merged Pierre's MaskedArray code into the maskedarray branch. That is nearly done, with only a few unit tests still failing -- ones brought over from the old numpy.ma. This is mainly due to some changes in the API, for example put and putmask now behave like their ndarray counterparts. Pierre, would you remind us of any other such changes, and why they were made? What is the road forward with this code? We will probably only merge API changes into 1.4. Do we use svnmerge to keep the branch up to date until then? The branch can be found at http://svn.scipy.org/svn/numpy/branches/maskedarray Regards Stéfan On Wed, Dec 19, 2007 at 12:52:35PM -0600, Travis E. Oliphant wrote:
Hi all,
We had a great Sprint at Berkeley over last weekend. Jarrod deserves a huge hand for organizing it and Fernando should be also congradulated for making the Sprint a productive communication session with a lot of different people.
On Dec 19, 2007 11:52 AM, Travis E. Oliphant <oliphant@enthought.com> wrote:
Testing --------- * scipy unit-testing will be "nose-compliant" and therefore nose will be required to run the SciPy tests. * NumPy will still use the current testing framework but will support SciPy's desire to be nose-compliant. NumPy 1.1 tests may move to just being "nose-compliant" * The goal is to make tests easier for contributors to write.
This hadn't been committed yet, but last night I put it in, and Matthew Brett is planning today on updating the rest of the codebase to the nose tests, which do make a LOT of things vastly nicer and simpler to write, at the cost of a very small new dependency (only needed to run the actual test suite, not to use scipy). At some point in the next few days I imagine this will get merged back into trunk, though that merge probably needs to be done with a bit of coordination, since the changes will touch pretty much every scipy module (at least their test/ directory).
Weave --------- * weave will not move into NumPy yet, but possibly at NumPy 1.1, there could be a separate package containing all the "wrapping" support code for NumPy in a more unified fashion (if somebody is interested in this, it is a great time to jump in).
It's worth crediting Min (Benjamin Ragan-Kelley, ipython developer) for spending the whole day deep in the bowels of weave clenaing up tens of test failures that had been in for a long time. The weave tests aren't getting correctly picked up by scipy.test(), hence they hadn't been reported, but you can see them with weave.test(10). To make sure that this work doesn't fall too far out of sync with trunk, I just committed it all back into trunk as two separate changesets (one to update the branch itself in case anyone is going to work further on it, one to apply the changes to the trunk itself). Many thanks to Min for the spelunking work! Cheers, f
Weave --------- * weave will not move into NumPy yet, but possibly at NumPy 1.1, there could be a separate package containing all the "wrapping" support code for NumPy in a more unified fashion (if somebody is interested in this, it is a great time to jump in).
It's worth crediting Min (Benjamin Ragan-Kelley, ipython developer) for spending the whole day deep in the bowels of weave clenaing up tens of test failures that had been in for a long time. The weave tests aren't getting correctly picked up by scipy.test(), hence they hadn't been reported, but you can see them with weave.test(10).
This is good news :) Will other compilers be supported by the Blits converters, like ICL or Visual Studio ? Last time I tried ty use ICL, some headers were missing :| Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher
On Dec 27, 2007 11:13 PM, Matthieu Brucher <matthieu.brucher@gmail.com> wrote:
It's worth crediting Min (Benjamin Ragan-Kelley, ipython developer) for spending the whole day deep in the bowels of weave clenaing up tens of test failures that had been in for a long time. The weave tests aren't getting correctly picked up by scipy.test(), hence they hadn't been reported, but you can see them with weave.test(10).
This is good news :) Will other compilers be supported by the Blits converters, like ICL or Visual Studio ? Last time I tried ty use ICL, some headers were missing :|
Tom Waite, from Berkeley, spent a lot of time getting the intel compilers to work on win32, but I don't think that code is committed. You may want to ping the Berkeley guys tomorrow on irc for details on the status of that (I'm not there right now). Cheers, f
I got the Intel( Windows) compiler to work, but had problems with the link. The Intel linker (xilink6) has a problem (multi-file optimization error) and I noted with my Visual Studio running with Intel, the Intel linker actually is running the MS linker. This is not completed as of now as I still need to resolve the minor changes I am having to make to the inherited distutils scripts in my IntelW.py. Tom On 12/27/07, Fernando Perez <fperez.net@gmail.com> wrote:
On Dec 27, 2007 11:13 PM, Matthieu Brucher <matthieu.brucher@gmail.com> wrote:
It's worth crediting Min (Benjamin Ragan-Kelley, ipython developer) for spending the whole day deep in the bowels of weave clenaing up tens of test failures that had been in for a long time. The weave tests aren't getting correctly picked up by scipy.test(), hence they hadn't been reported, but you can see them with weave.test(10).
This is good news :) Will other compilers be supported by the Blits converters, like ICL or Visual Studio ? Last time I tried ty use ICL, some headers were missing :|
Tom Waite, from Berkeley, spent a lot of time getting the intel compilers to work on win32, but I don't think that code is committed. You may want to ping the Berkeley guys tomorrow on irc for details on the status of that (I'm not there right now).
Cheers,
f _______________________________________________ Scipy-dev mailing list Scipy-dev@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-dev
My worries are for the Intel compiler and Linux (this way, the compiler is free for my purposes). Last time I tried to use weave, the compilation failed because some headers were missing in Scipy. I don't know if this was fixed as not additional folder is present in the trunk. Matthieu 2007/12/28, Tom Waite <twaite@berkeley.edu>:
I got the Intel( Windows) compiler to work, but had problems with the link. The Intel linker (xilink6) has a problem (multi-file optimization error) and I noted with my Visual Studio running with Intel, the Intel linker actually is running the MS linker. This is not completed as of now as I still need to resolve the minor changes I am having to make to the inherited distutils scripts in my IntelW.py.
Tom
On 12/27/07, Fernando Perez <fperez.net@gmail.com> wrote:
On Dec 27, 2007 11:13 PM, Matthieu Brucher <matthieu.brucher@gmail.com > wrote:
It's worth crediting Min (Benjamin Ragan-Kelley, ipython developer) for spending the whole day deep in the bowels of weave clenaing up tens of test failures that had been in for a long time. The weave tests aren't getting correctly picked up by scipy.test(), hence they hadn't been reported, but you can see them with weave.test(10).
This is good news :) Will other compilers be supported by the Blits converters, like ICL or Visual Studio ? Last time I tried ty use ICL, some headers were missing :|
Tom Waite, from Berkeley, spent a lot of time getting the intel compilers to work on win32, but I don't think that code is committed. You may want to ping the Berkeley guys tomorrow on irc for details on the status of that (I'm not there right now).
Cheers,
f _______________________________________________ Scipy-dev mailing list Scipy-dev@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-dev
_______________________________________________ Scipy-dev mailing list Scipy-dev@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-dev
-- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher
On Dec 28, 2007 6:19 AM, Matthieu Brucher <matthieu.brucher@gmail.com> wrote:
My worries are for the Intel compiler and Linux (this way, the compiler is free for my purposes).
Are you sure it's free for your purposes? Class use as a student is OK, but research is not, unfortunately: http://www.intel.com/cd/software/products/asmo-na/eng/219771.htm Note that academic use of the products does not qualify for a non-commercial license. Intel offers heavily discounted licenses to academic developers through our Academic Developer Program. They have more details here: http://www.intel.com/cd/software/products/asmo-na/eng/219692.htm I'm pretty sure at some point intel changed this, because I'm almost certain that years ago academic research fell under their free license conditions. Not anymore, in a pretty explicit fashion (except for pure student-doing-unpaid-homework use). Cheers, f
Yes, I'm aware of this, but I can test it at home to see what warnings or errors it raises ;) (although I would like to use it in my research, but no budget for this...) Matthieu 2007/12/28, Fernando Perez <fperez.net@gmail.com>:
On Dec 28, 2007 6:19 AM, Matthieu Brucher <matthieu.brucher@gmail.com> wrote:
My worries are for the Intel compiler and Linux (this way, the compiler is free for my purposes).
Are you sure it's free for your purposes? Class use as a student is OK, but research is not, unfortunately:
http://www.intel.com/cd/software/products/asmo-na/eng/219771.htm
Note that academic use of the products does not qualify for a non-commercial license. Intel offers heavily discounted licenses to academic developers through our Academic Developer Program.
They have more details here:
http://www.intel.com/cd/software/products/asmo-na/eng/219692.htm
I'm pretty sure at some point intel changed this, because I'm almost certain that years ago academic research fell under their free license conditions. Not anymore, in a pretty explicit fashion (except for pure student-doing-unpaid-homework use).
Cheers,
f _______________________________________________ Scipy-dev mailing list Scipy-dev@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-dev
-- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher
On Dec 28, 2007 1:43 PM, Matthieu Brucher <matthieu.brucher@gmail.com> wrote:
Yes, I'm aware of this, but I can test it at home to see what warnings or errors it raises ;) (although I would like to use it in my research, but no budget for this...)
I just figured I'd mention it, because it caught me by surprise last time I went to update from an old version and saw their new faq with such explicit provisions against academic free use... cheers, f
participants (8)
-
Charles R Harris -
Fernando Perez -
Matthieu Brucher -
Robert Kern -
Ryan Krauss -
Stefan van der Walt -
Tom Waite -
Travis E. Oliphant