Introduction to Scott, Jason, and (possibly) others from Enthought
Hi everyone,
There has been some talk about re-factoring NumPy to separate out the Python C-API layer and make NumPy closer to a C-library. I know there are a few different ideas about what this means, and also that people are very busy. I also know there is a NumPy 2.0 release that is in the works.
I'm excited to let everyone know that we (at Enthought) have been able to find resources (about 3 man months) to work on this re-factoring project and Scott and Jason (both very experienced C and Python programmers) are actively pursuing it. My hope is that NumPy 2.0 will contain this re-factoring (which should be finished just after SciPy 2010 --- where I'm going to organize a Sprint on NumPy which will include at least date-time improvements and re-factoring work).
While we have specific goals for the re-factoring, we want this activity to be fully integrated with the NumPy community and Scott and Jason want to interact with the community as much as feasible as they suggest re-factoring changes (though they both have more experience with phone-conversations to resolve concerns than email chains and so some patience from everybody will be appreciated).
Because Jason and Scott are new to this mailing list (but not new to NumPy), I wanted to introduce them so they would feel more comfortable posting questions and people would have some context as to what they were trying to do.
Scott and Jason are both very proficient and skilled programmers and I have full confidence in their abilities. That said, we very much want the input of as many people as possible as we pursue the goal of grouping together more tightly the Python C-API interface layer to NumPy.
I will be involved in some of the discussions, but am currently on a different project which has tight schedules and so I will only be able to provide limited "mailing-list" visibility.
Best regards,
-Travis
On Tue, May 25, 2010 at 1:37 PM, Travis Oliphant oliphant@enthought.comwrote:
Hi everyone,
There has been some talk about re-factoring NumPy to separate out the Python C-API layer and make NumPy closer to a C-library. I know there are a few different ideas about what this means, and also that people are very busy. I also know there is a NumPy 2.0 release that is in the works.
I'm excited to let everyone know that we (at Enthought) have been able to find resources (about 3 man months) to work on this re-factoring project and Scott and Jason (both very experienced C and Python programmers) are actively pursuing it. My hope is that NumPy 2.0 will contain this re-factoring (which should be finished just after SciPy 2010 --- where I'm going to organize a Sprint on NumPy which will include at least date-time improvements and re-factoring work).
While we have specific goals for the re-factoring, we want this activity to be fully integrated with the NumPy community and Scott and Jason want to interact with the community as much as feasible as they suggest re-factoring changes (though they both have more experience with phone-conversations to resolve concerns than email chains and so some patience from everybody will be appreciated).
Because Jason and Scott are new to this mailing list (but not new to NumPy), I wanted to introduce them so they would feel more comfortable posting questions and people would have some context as to what they were trying to do.
Scott and Jason are both very proficient and skilled programmers and I have full confidence in their abilities. That said, we very much want the input of as many people as possible as we pursue the goal of grouping together more tightly the Python C-API interface layer to NumPy.
I will be involved in some of the discussions, but am currently on a different project which has tight schedules and so I will only be able to provide limited "mailing-list" visibility.
I think 2.0 would be a bit early for this. Is there any reason it couldn't be done in 2.1? What is the planned policy with regards to the visible interface for extensions? It would also be nice to have a rough idea of how the resulting code would be layered, i.e., what is the design for this re-factoring. Simply having a design would be a major step forward.
In any case, I think the primary goal for 2.0 should remain the python3k port.
Chuck
On May 25, 2010, at 2:50 PM, Charles R Harris wrote:
On Tue, May 25, 2010 at 1:37 PM, Travis Oliphant <oliphant@enthought.com
wrote:
Hi everyone,
There has been some talk about re-factoring NumPy to separate out the Python C-API layer and make NumPy closer to a C-library. I know there are a few different ideas about what this means, and also that people are very busy. I also know there is a NumPy 2.0 release that is in the works.
I'm excited to let everyone know that we (at Enthought) have been able to find resources (about 3 man months) to work on this re-factoring project and Scott and Jason (both very experienced C and Python programmers) are actively pursuing it. My hope is that NumPy 2.0 will contain this re-factoring (which should be finished just after SciPy 2010 --- where I'm going to organize a Sprint on NumPy which will include at least date-time improvements and re-factoring work).
While we have specific goals for the re-factoring, we want this activity to be fully integrated with the NumPy community and Scott and Jason want to interact with the community as much as feasible as they suggest re-factoring changes (though they both have more experience with phone-conversations to resolve concerns than email chains and so some patience from everybody will be appreciated).
Because Jason and Scott are new to this mailing list (but not new to NumPy), I wanted to introduce them so they would feel more comfortable posting questions and people would have some context as to what they were trying to do.
Scott and Jason are both very proficient and skilled programmers and I have full confidence in their abilities. That said, we very much want the input of as many people as possible as we pursue the goal of grouping together more tightly the Python C-API interface layer to NumPy.
I will be involved in some of the discussions, but am currently on a different project which has tight schedules and so I will only be able to provide limited "mailing-list" visibility.
I think 2.0 would be a bit early for this. Is there any reason it couldn't be done in 2.1? What is the planned policy with regards to the visible interface for extensions? It would also be nice to have a rough idea of how the resulting code would be layered, i.e., what is the design for this re-factoring. Simply having a design would be a major step forward.
The problem with doing it in 2.1 is that this re-factoring will require extensions to be re-built. The visible interface to extensions will not change, but there will likely be ABI incompatibility. It seems prudent to do this in NumPy 2.0. Perhaps we can also put in place the ABI-protecting indirection approaches that David C. was suggesting earlier.
Some aspects of the design are still being fleshed out, but the basic idea is to separate out a core library that is as independent of the Python C-API as possible. There will likely be at least some dependency on the Python C-API (reference counting and error handling and possibly others) which any interface would have to provide in a very simple Python.h -- equivalent, for example.
Our purpose is to allow NumPy to be integrated with other languages or other frameworks systems without explicitly relying on CPython. There are a lot of questions as to how this will work, and so much of that is being worked out. Part of the reason for this mail is to help ensure that as much of this discussion as possible takes place in public.
-Travis
In any case, I think the primary goal for 2.0 should remain the python3k port.
Chuck
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
-- Travis Oliphant Enthought Inc. 1-512-536-1057 http://www.enthought.com oliphant@enthought.com
On Tue, May 25, 2010 at 2:54 PM, Travis Oliphant oliphant@enthought.comwrote:
On May 25, 2010, at 2:50 PM, Charles R Harris wrote:
On Tue, May 25, 2010 at 1:37 PM, Travis Oliphant oliphant@enthought.comwrote:
Hi everyone,
There has been some talk about re-factoring NumPy to separate out the Python C-API layer and make NumPy closer to a C-library. I know there are a few different ideas about what this means, and also that people are very busy. I also know there is a NumPy 2.0 release that is in the works.
I'm excited to let everyone know that we (at Enthought) have been able to find resources (about 3 man months) to work on this re-factoring project and Scott and Jason (both very experienced C and Python programmers) are actively pursuing it. My hope is that NumPy 2.0 will contain this re-factoring (which should be finished just after SciPy 2010 --- where I'm going to organize a Sprint on NumPy which will include at least date-time improvements and re-factoring work).
While we have specific goals for the re-factoring, we want this activity to be fully integrated with the NumPy community and Scott and Jason want to interact with the community as much as feasible as they suggest re-factoring changes (though they both have more experience with phone-conversations to resolve concerns than email chains and so some patience from everybody will be appreciated).
Because Jason and Scott are new to this mailing list (but not new to NumPy), I wanted to introduce them so they would feel more comfortable posting questions and people would have some context as to what they were trying to do.
Scott and Jason are both very proficient and skilled programmers and I have full confidence in their abilities. That said, we very much want the input of as many people as possible as we pursue the goal of grouping together more tightly the Python C-API interface layer to NumPy.
I will be involved in some of the discussions, but am currently on a different project which has tight schedules and so I will only be able to provide limited "mailing-list" visibility.
I think 2.0 would be a bit early for this. Is there any reason it couldn't be done in 2.1? What is the planned policy with regards to the visible interface for extensions? It would also be nice to have a rough idea of how the resulting code would be layered, i.e., what is the design for this re-factoring. Simply having a design would be a major step forward.
The problem with doing it in 2.1 is that this re-factoring will require extensions to be re-built. The visible interface to extensions will not change, but there will likely be ABI incompatibility. It seems prudent to do this in NumPy 2.0. Perhaps we can also put in place the ABI-protecting indirection approaches that David C. was suggesting earlier.
Some aspects of the design are still being fleshed out, but the basic idea is to separate out a core library that is as independent of the Python C-API as possible. There will likely be at least some dependency on the Python C-API (reference counting and error handling and possibly others) which any interface would have to provide in a very simple Python.h -- equivalent, for example.
Our purpose is to allow NumPy to be integrated with other languages or other frameworks systems without explicitly relying on CPython. There are a lot of questions as to how this will work, and so much of that is being worked out. Part of the reason for this mail is to help ensure that as much of this discussion as possible takes place in public.
Sounds good, but what if it doesn't get finished in a few months? I think we should get 2.0.0 out pronto, ideally it would already have been released. I think a major refactoring like this proposal should get the 3.0.0 label. Admittedly that makes keeping a refactored branch current with fixes going into the trunk a hassle, but perhaps that can be worked around somewhat by clearly labeling what files will be touched in the refactoring and possibly rearranging the content of the existing files. This requires a game plan and a clear idea of the goal. Put simply, I think the proposed schedule is too ambitious and needs to be fleshed out. This refactoring isn't going to be as straight forward as the python3k port because a lot of design decisions need to be made along the way.
Chuck
Travis: do you already have a place on the NumPy Development Wikihttp://wiki.numpy.org/where you're (b)logging your design decisions? Seems like a good way for concerned parties to monitor your choices in more or less real time and thus provide comment in a timely fashion.
DG
On Tue, May 25, 2010 at 2:19 PM, Charles R Harris <charlesr.harris@gmail.com
wrote:
On Tue, May 25, 2010 at 2:54 PM, Travis Oliphant oliphant@enthought.comwrote:
On May 25, 2010, at 2:50 PM, Charles R Harris wrote:
On Tue, May 25, 2010 at 1:37 PM, Travis Oliphant oliphant@enthought.comwrote:
Hi everyone,
There has been some talk about re-factoring NumPy to separate out the Python C-API layer and make NumPy closer to a C-library. I know there are a few different ideas about what this means, and also that people are very busy. I also know there is a NumPy 2.0 release that is in the works.
I'm excited to let everyone know that we (at Enthought) have been able to find resources (about 3 man months) to work on this re-factoring project and Scott and Jason (both very experienced C and Python programmers) are actively pursuing it. My hope is that NumPy 2.0 will contain this re-factoring (which should be finished just after SciPy 2010 --- where I'm going to organize a Sprint on NumPy which will include at least date-time improvements and re-factoring work).
While we have specific goals for the re-factoring, we want this activity to be fully integrated with the NumPy community and Scott and Jason want to interact with the community as much as feasible as they suggest re-factoring changes (though they both have more experience with phone-conversations to resolve concerns than email chains and so some patience from everybody will be appreciated).
Because Jason and Scott are new to this mailing list (but not new to NumPy), I wanted to introduce them so they would feel more comfortable posting questions and people would have some context as to what they were trying to do.
Scott and Jason are both very proficient and skilled programmers and I have full confidence in their abilities. That said, we very much want the input of as many people as possible as we pursue the goal of grouping together more tightly the Python C-API interface layer to NumPy.
I will be involved in some of the discussions, but am currently on a different project which has tight schedules and so I will only be able to provide limited "mailing-list" visibility.
I think 2.0 would be a bit early for this. Is there any reason it couldn't be done in 2.1? What is the planned policy with regards to the visible interface for extensions? It would also be nice to have a rough idea of how the resulting code would be layered, i.e., what is the design for this re-factoring. Simply having a design would be a major step forward.
The problem with doing it in 2.1 is that this re-factoring will require extensions to be re-built. The visible interface to extensions will not change, but there will likely be ABI incompatibility. It seems prudent to do this in NumPy 2.0. Perhaps we can also put in place the ABI-protecting indirection approaches that David C. was suggesting earlier.
Some aspects of the design are still being fleshed out, but the basic idea is to separate out a core library that is as independent of the Python C-API as possible. There will likely be at least some dependency on the Python C-API (reference counting and error handling and possibly others) which any interface would have to provide in a very simple Python.h -- equivalent, for example.
Our purpose is to allow NumPy to be integrated with other languages or other frameworks systems without explicitly relying on CPython. There are a lot of questions as to how this will work, and so much of that is being worked out. Part of the reason for this mail is to help ensure that as much of this discussion as possible takes place in public.
Sounds good, but what if it doesn't get finished in a few months? I think we should get 2.0.0 out pronto, ideally it would already have been released. I think a major refactoring like this proposal should get the 3.0.0 label. Admittedly that makes keeping a refactored branch current with fixes going into the trunk a hassle, but perhaps that can be worked around somewhat by clearly labeling what files will be touched in the refactoring and possibly rearranging the content of the existing files. This requires a game plan and a clear idea of the goal. Put simply, I think the proposed schedule is too ambitious and needs to be fleshed out. This refactoring isn't going to be as straight forward as the python3k port because a lot of design decisions need to be made along the way.
Chuck
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On May 25, 2010, at 4:49 PM, David Goldsmith wrote:
Travis: do you already have a place on the NumPy Development Wiki where you're (b)logging your design decisions? Seems like a good way for concerned parties to monitor your choices in more or less real time and thus provide comment in a timely fashion.
This is a great idea of course and we will definitely post progess there. So far, the code has been reviewed and several functions identified for re-factoring. This is taking place in a github branch of numpy called numpy refactor.
-Travis
DG
On Tue, May 25, 2010 at 2:19 PM, Charles R Harris charlesr.harris@gmail.com wrote:
On Tue, May 25, 2010 at 2:54 PM, Travis Oliphant oliphant@enthought.com wrote:
On May 25, 2010, at 2:50 PM, Charles R Harris wrote:
On Tue, May 25, 2010 at 1:37 PM, Travis Oliphant oliphant@enthought.com wrote:
Hi everyone,
There has been some talk about re-factoring NumPy to separate out the Python C-API layer and make NumPy closer to a C-library. I know there are a few different ideas about what this means, and also that people are very busy. I also know there is a NumPy 2.0 release that is in the works.
I'm excited to let everyone know that we (at Enthought) have been able to find resources (about 3 man months) to work on this re-factoring project and Scott and Jason (both very experienced C and Python programmers) are actively pursuing it. My hope is that NumPy 2.0 will contain this re-factoring (which should be finished just after SciPy 2010 --- where I'm going to organize a Sprint on NumPy which will include at least date-time improvements and re-factoring work).
While we have specific goals for the re-factoring, we want this activity to be fully integrated with the NumPy community and Scott and Jason want to interact with the community as much as feasible as they suggest re-factoring changes (though they both have more experience with phone-conversations to resolve concerns than email chains and so some patience from everybody will be appreciated).
Because Jason and Scott are new to this mailing list (but not new to NumPy), I wanted to introduce them so they would feel more comfortable posting questions and people would have some context as to what they were trying to do.
Scott and Jason are both very proficient and skilled programmers and I have full confidence in their abilities. That said, we very much want the input of as many people as possible as we pursue the goal of grouping together more tightly the Python C-API interface layer to NumPy.
I will be involved in some of the discussions, but am currently on a different project which has tight schedules and so I will only be able to provide limited "mailing-list" visibility.
I think 2.0 would be a bit early for this. Is there any reason it couldn't be done in 2.1? What is the planned policy with regards to the visible interface for extensions? It would also be nice to have a rough idea of how the resulting code would be layered, i.e., what is the design for this re-factoring. Simply having a design would be a major step forward.
The problem with doing it in 2.1 is that this re-factoring will require extensions to be re-built. The visible interface to extensions will not change, but there will likely be ABI incompatibility. It seems prudent to do this in NumPy 2.0. Perhaps we can also put in place the ABI-protecting indirection approaches that David C. was suggesting earlier.
Some aspects of the design are still being fleshed out, but the basic idea is to separate out a core library that is as independent of the Python C-API as possible. There will likely be at least some dependency on the Python C-API (reference counting and error handling and possibly others) which any interface would have to provide in a very simple Python.h -- equivalent, for example.
Our purpose is to allow NumPy to be integrated with other languages or other frameworks systems without explicitly relying on CPython. There are a lot of questions as to how this will work, and so much of that is being worked out. Part of the reason for this mail is to help ensure that as much of this discussion as possible takes place in public.
Sounds good, but what if it doesn't get finished in a few months? I think we should get 2.0.0 out pronto, ideally it would already have been released. I think a major refactoring like this proposal should get the 3.0.0 label. Admittedly that makes keeping a refactored branch current with fixes going into the trunk a hassle, but perhaps that can be worked around somewhat by clearly labeling what files will be touched in the refactoring and possibly rearranging the content of the existing files. This requires a game plan and a clear idea of the goal. Put simply, I think the proposed schedule is too ambitious and needs to be fleshed out. This refactoring isn't going to be as straight forward as the python3k port because a lot of design decisions need to be made along the way.
Chuck
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
-- Mathematician: noun, someone who disavows certainty when their uncertainty set is non-empty, even if that set has measure zero.
Hope: noun, that delusive spirit which escaped Pandora's jar and, with her lies, prevents mankind from committing a general suicide. (As interpreted by Robert Graves) _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
--- Travis Oliphant Enthought, Inc. oliphant@enthought.com 1-512-536-1057 http://www.enthought.com
On 25 May 2010 21:22, Travis Oliphant oliphant@enthought.com wrote:
This is a great idea of course and we will definitely post progess there. So far, the code has been reviewed and several functions identified for re-factoring. This is taking place in a github branch of numpy called numpy refactor.
Awesome! Since github now supports SVN interaction, and all the core devs use Git, now might be a good time to move the entire numpy source tree? It will certainly make it easier to merge the refactor changes!
Regards Stéfan
2010/5/25 Stéfan van der Walt stefan@sun.ac.za:
Awesome! Since github now supports SVN interaction, and all the core devs use Git, now might be a good time to move the entire numpy source tree? It will certainly make it easier to merge the refactor changes!
I would love to move numpy to github as well. Almost everything I work on is there now and I am really enjoying using git and the github infrastructure is really nice. This is obviously a separate issue and one that shouldn't deflect the discussion on the proposed refactoring. But given how many of the developers are using git-svn and that you can use an svn client with github, it might be worth having a quick discussion about this in the near future. For instance, I wonder how many of the developer's prefer using git at this point. Also it would be interesting to hear from any of the developer's who would be opposed to git. A few year's ago this was a hot topic for discussion, but it may be that this isn't very controversial at this point.
Jarrod
On Wed, May 26, 2010 at 1:03 PM, Jarrod Millman millman@berkeley.eduwrote:
2010/5/25 Stéfan van der Walt stefan@sun.ac.za:
Awesome! Since github now supports SVN interaction, and all the core devs use Git, now might be a good time to move the entire numpy source tree? It will certainly make it easier to merge the refactor changes!
I would love to move numpy to github as well. Almost everything I work on is there now and I am really enjoying using git and the github infrastructure is really nice. This is obviously a separate issue and one that shouldn't deflect the discussion on the proposed refactoring. But given how many of the developers are using git-svn and that you can use an svn client with github, it might be worth having a quick discussion about this in the near future. For instance, I wonder how many of the developer's prefer using git at this point. Also it would be interesting to hear from any of the developer's who would be opposed to git. A few year's ago this was a hot topic for discussion, but it may be that this isn't very controversial at this point.
I think the main problem has been windows compatibility. Git is best from the command line whereas the windows command line is an afterthought. Another box that needs a check-mark is the buildbot. If svn clients are supported then it may be that neither of those are going to be a problem. However, It needs user testing.
Chuck
On Wed, May 26, 2010 at 1:54 PM, Charles R Harris <charlesr.harris@gmail.com
wrote:
On Wed, May 26, 2010 at 1:03 PM, Jarrod Millman millman@berkeley.eduwrote:
2010/5/25 Stéfan van der Walt stefan@sun.ac.za:
Awesome! Since github now supports SVN interaction, and all the core devs use Git, now might be a good time to move the entire numpy source tree? It will certainly make it easier to merge the refactor changes!
I would love to move numpy to github as well. Almost everything I work on is there now and I am really enjoying using git and the github infrastructure is really nice. This is obviously a separate issue and one that shouldn't deflect the discussion on the proposed refactoring. But given how many of the developers are using git-svn and that you can use an svn client with github, it might be worth having a quick discussion about this in the near future. For instance, I wonder how many of the developer's prefer using git at this point. Also it would be interesting to hear from any of the developer's who would be opposed to git. A few year's ago this was a hot topic for discussion, but it may be that this isn't very controversial at this point.
I think the main problem has been windows compatibility. Git is best from the command line whereas the windows command line is an afterthought. Another box that needs a check-mark is the buildbot. If svn clients are supported then it may be that neither of those are going to be a problem. However, It needs user testing.
A newish git windows client I hadn't heard of before is gitextensionshttp://code.google.com/p/gitextensions/ .
Chuck
On Thu, May 27, 2010 at 4:54 AM, Charles R Harris charlesr.harris@gmail.com wrote:
On Wed, May 26, 2010 at 1:03 PM, Jarrod Millman millman@berkeley.edu wrote:
2010/5/25 Stéfan van der Walt stefan@sun.ac.za:
Awesome! Since github now supports SVN interaction, and all the core devs use Git, now might be a good time to move the entire numpy source tree? It will certainly make it easier to merge the refactor changes!
I would love to move numpy to github as well. Almost everything I work on is there now and I am really enjoying using git and the github infrastructure is really nice. This is obviously a separate issue and one that shouldn't deflect the discussion on the proposed refactoring. But given how many of the developers are using git-svn and that you can use an svn client with github, it might be worth having a quick discussion about this in the near future. For instance, I wonder how many of the developer's prefer using git at this point. Also it would be interesting to hear from any of the developer's who would be opposed to git. A few year's ago this was a hot topic for discussion, but it may be that this isn't very controversial at this point.
I think the main problem has been windows compatibility. Git is best from the command line whereas the windows command line is an afterthought. Another box that needs a check-mark is the buildbot. If svn clients are supported then it may be that neither of those are going to be a problem. However, It needs user testing.
As I mentioned in a previous post, there is smartgit, which is free for personal use, and is a graphical UI (does *not* depend on the mingw port of git, uses the reimplementation of git jgit in java used in google for android I believe).
gitextensions is just a GUI around the mingw tools, and as such is less reliable.
github also supports smart http for people behind proxies (although I don't know about the authentification issues if any). Trac and buildbot could use a svn mirror as provided by github for the time being, although there seems to be an issue with the numpy repo ATM (maybe my fault: http://support.github.com/discussions/repos/3155-svn-checkout-error-200-ok-e...)
David
Hi,
I think the main problem has been windows compatibility. Git is best from the command line whereas the windows command line is an afterthought. Another box that needs a check-mark is the buildbot. If svn clients are supported then it may be that neither of those are going to be a problem. However, It needs user testing.
For windows - I think honestly this is now not a serious barrier to using git. I've installed msysgit on 4 or 5 machines recently, and it has been very smooth - as well as providing a nice bash shell.
I think it would be a huge reduction in the barrier to contributing to numpy if we could change to git.
Cheers,
Matthew
On Wed, May 26, 2010 at 7:44 PM, Matthew Brett matthew.brett@gmail.com wrote:
Hi,
I think the main problem has been windows compatibility. Git is best from the command line whereas the windows command line is an afterthought. Another box that needs a check-mark is the buildbot. If svn clients are supported then it may be that neither of those are going to be a problem. However, It needs user testing.
For windows - I think honestly this is now not a serious barrier to using git. I've installed msysgit on 4 or 5 machines recently, and it has been very smooth - as well as providing a nice bash shell.
there is no such thing as a "nice" bash shell for a windows user. I have no idea how to use one.
Josef
I think it would be a huge reduction in the barrier to contributing to numpy if we could change to git.
Cheers,
Matthew _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Hi,
there is no such thing as a "nice" bash shell for a windows user. I have no idea how to use one.
It is a nice bash shell. You may not want a nice bash shell ;)
I can't imagine you'd object to one though. It's just a useful place to type git commands, with file / directory path autocompletion, git branch autocompletion and so on.
See you,
Matthew
On Wed, May 26, 2010 at 7:34 PM, Matthew Brett matthew.brett@gmail.comwrote:
Hi,
there is no such thing as a "nice" bash shell for a windows user. I have no idea how to use one.
It is a nice bash shell. You may not want a nice bash shell ;)
I can't imagine you'd object to one though. It's just a useful place to type git commands, with file / directory path autocompletion, git branch autocompletion and so on.
Any shell on windows is a pain, if only because of the spaces in the filenames.
Chuck
On Wed, May 26, 2010 at 8:37 PM, Charles R Harris <charlesr.harris@gmail.com
wrote:
On Wed, May 26, 2010 at 7:34 PM, Matthew Brett matthew.brett@gmail.comwrote:
Hi,
there is no such thing as a "nice" bash shell for a windows user. I have no idea how to use one.
It is a nice bash shell. You may not want a nice bash shell ;)
I can't imagine you'd object to one though. It's just a useful place to type git commands, with file / directory path autocompletion, git branch autocompletion and so on.
Any shell on windows is a pain, if only because of the spaces in the filenames.
Why? Can't you just escape the spaces with backslashes... oh, nevermind...
Ben Root
Chuck
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Wed, May 26, 2010 at 7:47 PM, Benjamin Root ben.root@ou.edu wrote:
On Wed, May 26, 2010 at 8:37 PM, Charles R Harris < charlesr.harris@gmail.com> wrote:
On Wed, May 26, 2010 at 7:34 PM, Matthew Brett matthew.brett@gmail.comwrote:
Hi,
there is no such thing as a "nice" bash shell for a windows user. I have no idea how to use one.
It is a nice bash shell. You may not want a nice bash shell ;)
I can't imagine you'd object to one though. It's just a useful place to type git commands, with file / directory path autocompletion, git branch autocompletion and so on.
Any shell on windows is a pain, if only because of the spaces in the filenames.
Why? Can't you just escape the spaces with backslashes... oh, nevermind...
Sure, no doubt the experience would cleanse my soul. But flagellation would be less painful.
Chuck
On Wed, May 26, 2010 at 9:37 PM, Charles R Harris charlesr.harris@gmail.com wrote:
On Wed, May 26, 2010 at 7:34 PM, Matthew Brett matthew.brett@gmail.com wrote:
Hi,
there is no such thing as a "nice" bash shell for a windows user. I have no idea how to use one.
It is a nice bash shell. You may not want a nice bash shell ;)
I can't imagine you'd object to one though. It's just a useful place to type git commands, with file / directory path autocompletion, git branch autocompletion and so on.
Any shell on windows is a pain, if only because of the spaces in the filenames.
When I'm using git, or bzr or svn I use the windows shell which I'm very familiar with ,and allows "standard" copy-paste and has quotes.
But since, I think, there are no numpy developers on Windows, and I'm the only one for scipy, and occasional commits I can do with anything, I won't argue this time.
Josef
Chuck
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Hi,
Any shell on windows is a pain, if only because of the spaces in the filenames.
When I'm using git, or bzr or svn I use the windows shell which I'm very familiar with ,and allows "standard" copy-paste and has quotes.
But since, I think, there are no numpy developers on Windows, and I'm the only one for scipy, and occasional commits I can do with anything, I won't argue this time.
I've been testing quite a bit on windows recently, and I used to use windows all the time. I've found msysgit to be pretty good. I personally have always hated the windows shell, but you can use msysgit from the windows shell if you prefer...
If you're OK with bzr or svn from the windows command line, I am sure git will pose no major problems.
See you,
Matthew
On Wed, May 26, 2010 at 7:25 PM, josef.pktd@gmail.com wrote:
On Wed, May 26, 2010 at 7:44 PM, Matthew Brett matthew.brett@gmail.com wrote:
Hi,
I think the main problem has been windows compatibility. Git is best
from
the command line whereas the windows command line is an afterthought. Another box that needs a check-mark is the buildbot. If svn clients are supported then it may be that neither of those are going to be a
problem.
However, It needs user testing.
For windows - I think honestly this is now not a serious barrier to using git. I've installed msysgit on 4 or 5 machines recently, and it has been very smooth - as well as providing a nice bash shell.
there is no such thing as a "nice" bash shell for a windows user. I have no idea how to use one.
Heh. Can you to try the svn interface to github using your favorite svn ap. I suppose we need to set up a test account there. Is it possible to have a multiple user git account on github, or is it all push and merge?
Chuck
On Wed, May 26, 2010 at 6:35 PM, Charles R Harris charlesr.harris@gmail.com wrote:
Heh. Can you to try the svn interface to github using your favorite svn ap. I suppose we need to set up a test account there. Is it possible to have a multiple user git account on github, or is it all push and merge?
Yes, that is possible. Currently it is not officially allowed according to their terms of service, but they are planning to enable that and have granted exceptions to their current policy in a few instances (that I am familiar with). But this is a detail that we can easily address in a NEP, if there seems to be interest (which there currently seems to be). So for now just keep sending emails with issues you want addressed in any actual proposal for this transition.
Let's move this discussion to the git thread and use the thread for the discussion related directly to the subject line.
Thanks,
On Tue, May 25, 2010 at 9:22 PM, Travis Oliphant oliphant@enthought.comwrote:
On May 25, 2010, at 4:49 PM, David Goldsmith wrote:
Travis: do you already have a place on the NumPy Development Wikihttp://wiki.numpy.org/where you're (b)logging your design decisions? Seems like a good way for concerned parties to monitor your choices in more or less real time and thus provide comment in a timely fashion.
This is a great idea of course and we will definitely post progess there.
Thanks; specific URL please, when available; plus, prominently feature (a link to) the location on the Development Wiki home page, at the very least (i.e., if not also on the NumPy home page).
So far, the code has been reviewed,
I.e., the existing code, yes?
and several functions identified for re-factoring.
Please enumerate on the "Wiki Refactoring Log" (name tentative - I don't care what we call it, just so long as it exists, its purpose is clear, and we all know where it is).
This is taking place in a github branch of numpy called numpy refactor.
"This" = the actual creation/modification of code, yes?
DG
-Travis
DG
On Tue, May 25, 2010 at 2:19 PM, Charles R Harris < charlesr.harris@gmail.com> wrote:
On Tue, May 25, 2010 at 2:54 PM, Travis Oliphant oliphant@enthought.comwrote:
On May 25, 2010, at 2:50 PM, Charles R Harris wrote:
On Tue, May 25, 2010 at 1:37 PM, Travis Oliphant <oliphant@enthought.com
wrote:
Hi everyone,
There has been some talk about re-factoring NumPy to separate out the Python C-API layer and make NumPy closer to a C-library. I know there are a few different ideas about what this means, and also that people are very busy. I also know there is a NumPy 2.0 release that is in the works.
I'm excited to let everyone know that we (at Enthought) have been able to find resources (about 3 man months) to work on this re-factoring project and Scott and Jason (both very experienced C and Python programmers) are actively pursuing it. My hope is that NumPy 2.0 will contain this re-factoring (which should be finished just after SciPy 2010 --- where I'm going to organize a Sprint on NumPy which will include at least date-time improvements and re-factoring work).
While we have specific goals for the re-factoring, we want this activity to be fully integrated with the NumPy community and Scott and Jason want to interact with the community as much as feasible as they suggest re-factoring changes (though they both have more experience with phone-conversations to resolve concerns than email chains and so some patience from everybody will be appreciated).
Because Jason and Scott are new to this mailing list (but not new to NumPy), I wanted to introduce them so they would feel more comfortable posting questions and people would have some context as to what they were trying to do.
Scott and Jason are both very proficient and skilled programmers and I have full confidence in their abilities. That said, we very much want the input of as many people as possible as we pursue the goal of grouping together more tightly the Python C-API interface layer to NumPy.
I will be involved in some of the discussions, but am currently on a different project which has tight schedules and so I will only be able to provide limited "mailing-list" visibility.
I think 2.0 would be a bit early for this. Is there any reason it couldn't be done in 2.1? What is the planned policy with regards to the visible interface for extensions? It would also be nice to have a rough idea of how the resulting code would be layered, i.e., what is the design for this re-factoring. Simply having a design would be a major step forward.
The problem with doing it in 2.1 is that this re-factoring will require extensions to be re-built. The visible interface to extensions will not change, but there will likely be ABI incompatibility. It seems prudent to do this in NumPy 2.0. Perhaps we can also put in place the ABI-protecting indirection approaches that David C. was suggesting earlier.
Some aspects of the design are still being fleshed out, but the basic idea is to separate out a core library that is as independent of the Python C-API as possible. There will likely be at least some dependency on the Python C-API (reference counting and error handling and possibly others) which any interface would have to provide in a very simple Python.h -- equivalent, for example.
Our purpose is to allow NumPy to be integrated with other languages or other frameworks systems without explicitly relying on CPython. There are a lot of questions as to how this will work, and so much of that is being worked out. Part of the reason for this mail is to help ensure that as much of this discussion as possible takes place in public.
Sounds good, but what if it doesn't get finished in a few months? I think we should get 2.0.0 out pronto, ideally it would already have been released. I think a major refactoring like this proposal should get the 3.0.0 label. Admittedly that makes keeping a refactored branch current with fixes going into the trunk a hassle, but perhaps that can be worked around somewhat by clearly labeling what files will be touched in the refactoring and possibly rearranging the content of the existing files. This requires a game plan and a clear idea of the goal. Put simply, I think the proposed schedule is too ambitious and needs to be fleshed out. This refactoring isn't going to be as straight forward as the python3k port because a lot of design decisions need to be made along the way.
Chuck
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
-- Mathematician: noun, someone who disavows certainty when their uncertainty set is non-empty, even if that set has measure zero.
Hope: noun, that delusive spirit which escaped Pandora's jar and, with her lies, prevents mankind from committing a general suicide. (As interpreted by Robert Graves) _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Travis Oliphant Enthought, Inc. oliphant@enthought.com 1-512-536-1057 http://www.enthought.com
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
I'm a potential user of the C-API and therefore I'm very interested in the outcome. In the previous discussion (http://comments.gmane.org/gmane.comp.python.numeric.general/37409) many different views on what the new C-API "should" be were expressed.
Naturally, I wonder if the new C-API will be useful for my purposes. So, I'm not so excited about a "refactoring log" where only the progress is reported. I fear that some (potentially minor) design decisions would render the new C-API useless for me.
So my question is: Does this "refactoring log" also include something like a Numpy Enhancement Proposal? Something that can be discussed beforehand? I.e., will there be a detailed description (i.e. code examples) what the goal of the refactoring is?
If there is any interest, I could provide some simple test examples in C that would explain what I'd like to be able to do with the new C-API.
Sebastian
On Wed, May 26, 2010 at 8:21 AM, David Goldsmith d.l.goldsmith@gmail.com wrote:
On Tue, May 25, 2010 at 9:22 PM, Travis Oliphant oliphant@enthought.com wrote:
On May 25, 2010, at 4:49 PM, David Goldsmith wrote:
Travis: do you already have a place on the NumPy Development Wiki where you're (b)logging your design decisions? Seems like a good way for concerned parties to monitor your choices in more or less real time and thus provide comment in a timely fashion.
This is a great idea of course and we will definitely post progess there.
Thanks; specific URL please, when available; plus, prominently feature (a link to) the location on the Development Wiki home page, at the very least (i.e., if not also on the NumPy home page).
So far, the code has been reviewed,
I.e., the existing code, yes?
and several functions identified for re-factoring.
Please enumerate on the "Wiki Refactoring Log" (name tentative - I don't care what we call it, just so long as it exists, its purpose is clear, and we all know where it is).
This is taking place in a github branch of numpy called numpy refactor.
"This" = the actual creation/modification of code, yes?
DG
-Travis
DG
On Tue, May 25, 2010 at 2:19 PM, Charles R Harris charlesr.harris@gmail.com wrote:
On Tue, May 25, 2010 at 2:54 PM, Travis Oliphant oliphant@enthought.com wrote:
On May 25, 2010, at 2:50 PM, Charles R Harris wrote:
On Tue, May 25, 2010 at 1:37 PM, Travis Oliphant oliphant@enthought.com wrote:
Hi everyone,
There has been some talk about re-factoring NumPy to separate out the Python C-API layer and make NumPy closer to a C-library. I know there are a few different ideas about what this means, and also that people are very busy. I also know there is a NumPy 2.0 release that is in the works.
I'm excited to let everyone know that we (at Enthought) have been able to find resources (about 3 man months) to work on this re-factoring project and Scott and Jason (both very experienced C and Python programmers) are actively pursuing it. My hope is that NumPy 2.0 will contain this re-factoring (which should be finished just after SciPy 2010 --- where I'm going to organize a Sprint on NumPy which will include at least date-time improvements and re-factoring work).
While we have specific goals for the re-factoring, we want this activity to be fully integrated with the NumPy community and Scott and Jason want to interact with the community as much as feasible as they suggest re-factoring changes (though they both have more experience with phone-conversations to resolve concerns than email chains and so some patience from everybody will be appreciated).
Because Jason and Scott are new to this mailing list (but not new to NumPy), I wanted to introduce them so they would feel more comfortable posting questions and people would have some context as to what they were trying to do.
Scott and Jason are both very proficient and skilled programmers and I have full confidence in their abilities. That said, we very much want the input of as many people as possible as we pursue the goal of grouping together more tightly the Python C-API interface layer to NumPy.
I will be involved in some of the discussions, but am currently on a different project which has tight schedules and so I will only be able to provide limited "mailing-list" visibility.
I think 2.0 would be a bit early for this. Is there any reason it couldn't be done in 2.1? What is the planned policy with regards to the visible interface for extensions? It would also be nice to have a rough idea of how the resulting code would be layered, i.e., what is the design for this re-factoring. Simply having a design would be a major step forward.
The problem with doing it in 2.1 is that this re-factoring will require extensions to be re-built. The visible interface to extensions will not change, but there will likely be ABI incompatibility. It seems prudent to do this in NumPy 2.0. Perhaps we can also put in place the ABI-protecting indirection approaches that David C. was suggesting earlier. Some aspects of the design are still being fleshed out, but the basic idea is to separate out a core library that is as independent of the Python C-API as possible. There will likely be at least some dependency on the Python C-API (reference counting and error handling and possibly others) which any interface would have to provide in a very simple Python.h -- equivalent, for example. Our purpose is to allow NumPy to be integrated with other languages or other frameworks systems without explicitly relying on CPython. There are a lot of questions as to how this will work, and so much of that is being worked out. Part of the reason for this mail is to help ensure that as much of this discussion as possible takes place in public.
Sounds good, but what if it doesn't get finished in a few months? I think we should get 2.0.0 out pronto, ideally it would already have been released. I think a major refactoring like this proposal should get the 3.0.0 label. Admittedly that makes keeping a refactored branch current with fixes going into the trunk a hassle, but perhaps that can be worked around somewhat by clearly labeling what files will be touched in the refactoring and possibly rearranging the content of the existing files. This requires a game plan and a clear idea of the goal. Put simply, I think the proposed schedule is too ambitious and needs to be fleshed out. This refactoring isn't going to be as straight forward as the python3k port because a lot of design decisions need to be made along the way.
Chuck _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
-- Mathematician: noun, someone who disavows certainty when their uncertainty set is non-empty, even if that set has measure zero.
Hope: noun, that delusive spirit which escaped Pandora's jar and, with her lies, prevents mankind from committing a general suicide. (As interpreted by Robert Graves) _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Travis Oliphant Enthought, Inc. oliphant@enthought.com 1-512-536-1057 http://www.enthought.com
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
-- Mathematician: noun, someone who disavows certainty when their uncertainty set is non-empty, even if that set has measure zero.
Hope: noun, that delusive spirit which escaped Pandora's jar and, with her lies, prevents mankind from committing a general suicide. (As interpreted by Robert Graves)
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Wed, 26 May 2010 10:50:19 +0200, Sebastian Walter wrote:
I'm a potential user of the C-API and therefore I'm very interested in the outcome. In the previous discussion (http://comments.gmane.org/gmane.comp.python.numeric.general/37409) many different views on what the new C-API "should" be were expressed.
I believe the aim of the refactor is to *not* change the C-API at all, but separate it internally from the routines that do the heavy lifting. Externally, Numpy would still look the same, but be more easy to maintain.
The routines that do the heavy lifting could then be good for reuse and be more easy to maintain, but I think how and where they would be exposed hasn't been discussed so far...
On Wed, May 26, 2010 at 12:31 PM, Pauli Virtanen pav@iki.fi wrote:
Wed, 26 May 2010 10:50:19 +0200, Sebastian Walter wrote:
I'm a potential user of the C-API and therefore I'm very interested in the outcome. In the previous discussion (http://comments.gmane.org/gmane.comp.python.numeric.general/37409) many different views on what the new C-API "should" be were expressed.
I believe the aim of the refactor is to *not* change the C-API at all, but separate it internally from the routines that do the heavy lifting. Externally, Numpy would still look the same, but be more easy to maintain.
Sorry for the confusion. By C-API I meant a C-API that would be independent of the CPython API.
The routines that do the heavy lifting could then be good for reuse and be more easy to maintain, but I think how and where they would be exposed hasn't been discussed so far...
I had the impression that the goal is not only to have code that is easier to maintain but to give developers the possibility to use numpy functionality (broadcasting, ufuncs, ...) within C code without having to use CPython API (refcounts, construction of PyObjects etc.).
-- Pauli Virtanen
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On May 26, 2010, at 6:40 AM, Sebastian Walter wrote:
On Wed, May 26, 2010 at 12:31 PM, Pauli Virtanen pav@iki.fi wrote:
Wed, 26 May 2010 10:50:19 +0200, Sebastian Walter wrote:
I'm a potential user of the C-API and therefore I'm very interested in the outcome. In the previous discussion (http://comments.gmane.org/gmane.comp.python.numeric.general/ 37409) many different views on what the new C-API "should" be were expressed.
I believe the aim of the refactor is to *not* change the C-API at all, but separate it internally from the routines that do the heavy lifting. Externally, Numpy would still look the same, but be more easy to maintain.
Sorry for the confusion. By C-API I meant a C-API that would be independent of the CPython API.
The routines that do the heavy lifting could then be good for reuse and be more easy to maintain, but I think how and where they would be exposed hasn't been discussed so far...
I had the impression that the goal is not only to have code that is easier to maintain but to give developers the possibility to use numpy functionality (broadcasting, ufuncs, ...) within C code without having to use CPython API (refcounts, construction of PyObjects etc.).
This is partially correct. There may be a need to have some "stub" implementation of some aspects of the Python C-API (probably at least reference counting and exception handling for now).
We don't need to work all of this out initially. I think getting the separation done in the next several weeks will spawn conversations and ideas that may take several months to work out the new C-level-only API. That "interface" API is important in the short term, but also one that could change over the next several months.
-Travis
On May 26, 2010, at 5:31 AM, Pauli Virtanen wrote:
Wed, 26 May 2010 10:50:19 +0200, Sebastian Walter wrote:
I'm a potential user of the C-API and therefore I'm very interested in the outcome. In the previous discussion (http://comments.gmane.org/gmane.comp.python.numeric.general/37409) many different views on what the new C-API "should" be were expressed.
I believe the aim of the refactor is to *not* change the C-API at all, but separate it internally from the routines that do the heavy lifting. Externally, Numpy would still look the same, but be more easy to maintain.
The routines that do the heavy lifting could then be good for reuse and be more easy to maintain, but I think how and where they would be exposed hasn't been discussed so far...
This is correct. In our plans, the current NumPy C-API would not change.
Clearly there will be another potential API that could be used by other systems, but exactly what this is should be discussed and decided upon over time. I don't think our goals for the re-factoring depends on how this is done exactly.
-Travis
On May 26, 2010, at 3:50 AM, Sebastian Walter wrote:
I'm a potential user of the C-API and therefore I'm very interested in the outcome. In the previous discussion (http://comments.gmane.org/gmane.comp.python.numeric.general/37409) many different views on what the new C-API "should" be were expressed.
Naturally, I wonder if the new C-API will be useful for my purposes. So, I'm not so excited about a "refactoring log" where only the progress is reported. I fear that some (potentially minor) design decisions would render the new C-API useless for me.
So my question is: Does this "refactoring log" also include something like a Numpy Enhancement Proposal? Something that can be discussed beforehand? I.e., will there be a detailed description (i.e. code examples) what the goal of the refactoring is?
If there is any interest, I could provide some simple test examples in C that would explain what I'd like to be able to do with the new C-API.
Our team would love to see this if possible.
-Travis
Sebastian
On Wed, May 26, 2010 at 8:21 AM, David Goldsmith d.l.goldsmith@gmail.com wrote:
On Tue, May 25, 2010 at 9:22 PM, Travis Oliphant <oliphant@enthought.com
wrote:
On May 25, 2010, at 4:49 PM, David Goldsmith wrote:
Travis: do you already have a place on the NumPy Development Wiki where you're (b)logging your design decisions? Seems like a good way for concerned parties to monitor your choices in more or less real time and thus provide comment in a timely fashion.
This is a great idea of course and we will definitely post progess there.
Thanks; specific URL please, when available; plus, prominently feature (a link to) the location on the Development Wiki home page, at the very least (i.e., if not also on the NumPy home page).
So far, the code has been reviewed,
I.e., the existing code, yes?
and several functions identified for re-factoring.
Please enumerate on the "Wiki Refactoring Log" (name tentative - I don't care what we call it, just so long as it exists, its purpose is clear, and we all know where it is).
This is taking place in a github branch of numpy called numpy refactor.
"This" = the actual creation/modification of code, yes?
DG
-Travis
DG
On Tue, May 25, 2010 at 2:19 PM, Charles R Harris charlesr.harris@gmail.com wrote:
On Tue, May 25, 2010 at 2:54 PM, Travis Oliphant <oliphant@enthought.com
wrote:
On May 25, 2010, at 2:50 PM, Charles R Harris wrote:
On Tue, May 25, 2010 at 1:37 PM, Travis Oliphant oliphant@enthought.com wrote:
Hi everyone,
There has been some talk about re-factoring NumPy to separate out the Python C-API layer and make NumPy closer to a C-library. I know there are a few different ideas about what this means, and also that people are very busy. I also know there is a NumPy 2.0 release that is in the works.
I'm excited to let everyone know that we (at Enthought) have been able to find resources (about 3 man months) to work on this re- factoring project and Scott and Jason (both very experienced C and Python programmers) are actively pursuing it. My hope is that NumPy 2.0 will contain this re-factoring (which should be finished just after SciPy 2010 --- where I'm going to organize a Sprint on NumPy which will include at least date-time improvements and re-factoring work).
While we have specific goals for the re-factoring, we want this activity to be fully integrated with the NumPy community and Scott and Jason want to interact with the community as much as feasible as they suggest re-factoring changes (though they both have more experience with phone-conversations to resolve concerns than email chains and so some patience from everybody will be appreciated).
Because Jason and Scott are new to this mailing list (but not new to NumPy), I wanted to introduce them so they would feel more comfortable posting questions and people would have some context as to what they were trying to do.
Scott and Jason are both very proficient and skilled programmers and I have full confidence in their abilities. That said, we very much want the input of as many people as possible as we pursue the goal of grouping together more tightly the Python C-API interface layer to NumPy.
I will be involved in some of the discussions, but am currently on a different project which has tight schedules and so I will only be able to provide limited "mailing-list" visibility.
I think 2.0 would be a bit early for this. Is there any reason it couldn't be done in 2.1? What is the planned policy with regards to the visible interface for extensions? It would also be nice to have a rough idea of how the resulting code would be layered, i.e., what is the design for this re-factoring. Simply having a design would be a major step forward.
The problem with doing it in 2.1 is that this re-factoring will require extensions to be re-built. The visible interface to extensions will not change, but there will likely be ABI incompatibility. It seems prudent to do this in NumPy 2.0. Perhaps we can also put in place the ABI- protecting indirection approaches that David C. was suggesting earlier. Some aspects of the design are still being fleshed out, but the basic idea is to separate out a core library that is as independent of the Python C-API as possible. There will likely be at least some dependency on the Python C-API (reference counting and error handling and possibly others) which any interface would have to provide in a very simple Python.h -- equivalent, for example. Our purpose is to allow NumPy to be integrated with other languages or other frameworks systems without explicitly relying on CPython. There are a lot of questions as to how this will work, and so much of that is being worked out. Part of the reason for this mail is to help ensure that as much of this discussion as possible takes place in public.
Sounds good, but what if it doesn't get finished in a few months? I think we should get 2.0.0 out pronto, ideally it would already have been released. I think a major refactoring like this proposal should get the 3.0.0 label. Admittedly that makes keeping a refactored branch current with fixes going into the trunk a hassle, but perhaps that can be worked around somewhat by clearly labeling what files will be touched in the refactoring and possibly rearranging the content of the existing files. This requires a game plan and a clear idea of the goal. Put simply, I think the proposed schedule is too ambitious and needs to be fleshed out. This refactoring isn't going to be as straight forward as the python3k port because a lot of design decisions need to be made along the way.
Chuck _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
-- Mathematician: noun, someone who disavows certainty when their uncertainty set is non-empty, even if that set has measure zero.
Hope: noun, that delusive spirit which escaped Pandora's jar and, with her lies, prevents mankind from committing a general suicide. (As interpreted by Robert Graves) _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Travis Oliphant Enthought, Inc. oliphant@enthought.com 1-512-536-1057 http://www.enthought.com
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
-- Mathematician: noun, someone who disavows certainty when their uncertainty set is non-empty, even if that set has measure zero.
Hope: noun, that delusive spirit which escaped Pandora's jar and, with her lies, prevents mankind from committing a general suicide. (As interpreted by Robert Graves)
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
-- Travis Oliphant Enthought Inc. 1-512-536-1057 http://www.enthought.com oliphant@enthought.com
On Wed, May 26, 2010 at 6:19 AM, Charles R Harris charlesr.harris@gmail.com wrote:
Sounds good, but what if it doesn't get finished in a few months? I think we should get 2.0.0 out pronto, ideally it would already have been released. I think a major refactoring like this proposal should get the 3.0.0 label.
Naming it 3.0 or 2.1 does not matter much - I think we should avoid breaking things twice. I can see a few solutions: - postpone 2.0 "indefinitely", until this new work is done - backport py3k changes to 1.5 (which would be API and ABI compatible with 1.4.1), and 2.0 would contain all the breaking changes.
I am really worried about breaking things once now and once in a few months (or even a year).
David
On Tue, May 25, 2010 at 4:06 PM, David Cournapeau cournape@gmail.comwrote:
On Wed, May 26, 2010 at 6:19 AM, Charles R Harris charlesr.harris@gmail.com wrote:
Sounds good, but what if it doesn't get finished in a few months? I think
we
should get 2.0.0 out pronto, ideally it would already have been released.
I
think a major refactoring like this proposal should get the 3.0.0 label.
Naming it 3.0 or 2.1 does not matter much - I think we should avoid breaking things twice. I can see a few solutions:
- postpone 2.0 "indefinitely", until this new work is done
- backport py3k changes to 1.5 (which would be API and ABI
compatible with 1.4.1), and 2.0 would contain all the breaking changes.
If I had to choose between those, I would pick making a 1.5 release, that is, branch the current trunk and then excise datetime and all the related changes. Let me propose a schedule:
- Branch 1.5 in late June. The time until then to be spent closing tickets.
- Release 1.5 towards the end of July. That should be doable now that the release folks have had some practice.
- Release 2.0 next spring. I don't think 3 man months is enough time to redesign/refactor numpy, get it tested, and document the changes. If we hide stuff away it will be even longer before folks who have written extensions can make the needed changes.
Chuck
On 25 May 2010 15:06, David Cournapeau cournape@gmail.com wrote:
Naming it 3.0 or 2.1 does not matter much - I think we should avoid breaking things twice. I can see a few solutions: - postpone 2.0 "indefinitely", until this new work is done - backport py3k changes to 1.5 (which would be API and ABI compatible with 1.4.1), and 2.0 would contain all the breaking changes.
This is a good suggestion. Release 1.5 without ABI breakage and then leave enough time to discuss an optimal API, refactor the C code and include datetime functionality for 2.0. We don't stand anything to gain by rushing.
If I'm not mistaken, David did warn that this kind of situation may occur the last time around :)
Cheers Stéfan
2010/5/25 Stéfan van der Walt stefan@sun.ac.za
On 25 May 2010 15:06, David Cournapeau cournape@gmail.com wrote:
Naming it 3.0 or 2.1 does not matter much - I think we should avoid breaking things twice. I can see a few solutions:
- postpone 2.0 "indefinitely", until this new work is done
- backport py3k changes to 1.5 (which would be API and ABI
compatible with 1.4.1), and 2.0 would contain all the breaking changes.
This is a good suggestion. Release 1.5 without ABI breakage and then leave enough time to discuss an optimal API, refactor the C code and include datetime functionality for 2.0. We don't stand anything to gain by rushing.
If I'm not mistaken, David did warn that this kind of situation may occur the last time around :)
IIRC, David was seeing a refactor a year or two off, if ever. I'm concerned that the refactor will go on and on and on, not least because I haven't seen any plan or discussion as to what the precise goals of the refactor are, much less a plan for how to get there. It's hard to have a sprint when no one knows what they are trying to achieve.
Chuck
On May 25, 2010, at 5:06 PM, David Cournapeau wrote:
On Wed, May 26, 2010 at 6:19 AM, Charles R Harris charlesr.harris@gmail.com wrote:
Sounds good, but what if it doesn't get finished in a few months? I think we should get 2.0.0 out pronto, ideally it would already have been released. I think a major refactoring like this proposal should get the 3.0.0 label.
Naming it 3.0 or 2.1 does not matter much - I think we should avoid breaking things twice. I can see a few solutions:
- postpone 2.0 "indefinitely", until this new work is done
- backport py3k changes to 1.5 (which would be API and ABI
compatible with 1.4.1), and 2.0 would contain all the breaking changes.
This is an interesting idea and also workable.
I am really worried about breaking things once now and once in a few months (or even a year).
I am too. That's why this discussion. We will have the NumPy refactor done by end of July at the latest. Numpy 2.0 should be able to come out in August.
-Travis
David _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
--- Travis Oliphant Enthought, Inc. oliphant@enthought.com 1-512-536-1057 http://www.enthought.com
On Wed, May 26, 2010 at 12:23 PM, Travis Oliphant oliphant@enthought.comwrote:
On May 25, 2010, at 5:06 PM, David Cournapeau wrote:
On Wed, May 26, 2010 at 6:19 AM, Charles R Harris charlesr.harris@gmail.com wrote:
Sounds good, but what if it doesn't get finished in a few months? I
think we
should get 2.0.0 out pronto, ideally it would already have been
released. I
think a major refactoring like this proposal should get the 3.0.0 label.
Naming it 3.0 or 2.1 does not matter much - I think we should avoid breaking things twice. I can see a few solutions:
- postpone 2.0 "indefinitely", until this new work is done
- backport py3k changes to 1.5 (which would be API and ABI
compatible with 1.4.1), and 2.0 would contain all the breaking changes.
This is an interesting idea and also workable.
I am really worried about breaking things once now and once in a few months (or even a year).
I am too. That's why this discussion. We will have the NumPy refactor done by end of July at the latest. Numpy 2.0 should be able to come out in August.
This thread got a bit side-tracked with the move to git, but I don't see a
conclusion about what to release when.
Even if the refactoring is done in July, I think a 2.0 release with so many major changes will probably need a longer test/release cycle. So if we say September, do you still want a 1.5 release?
Cheers, Ralf
On Thu, May 27, 2010 at 8:51 PM, Ralf Gommers ralf.gommers@googlemail.com wrote:
On Wed, May 26, 2010 at 12:23 PM, Travis Oliphant oliphant@enthought.com wrote:
On May 25, 2010, at 5:06 PM, David Cournapeau wrote:
On Wed, May 26, 2010 at 6:19 AM, Charles R Harris charlesr.harris@gmail.com wrote:
Sounds good, but what if it doesn't get finished in a few months? I think we should get 2.0.0 out pronto, ideally it would already have been released. I think a major refactoring like this proposal should get the 3.0.0 label.
Naming it 3.0 or 2.1 does not matter much - I think we should avoid breaking things twice. I can see a few solutions: - postpone 2.0 "indefinitely", until this new work is done - backport py3k changes to 1.5 (which would be API and ABI compatible with 1.4.1), and 2.0 would contain all the breaking changes.
This is an interesting idea and also workable.
I am really worried about breaking things once now and once in a few months (or even a year).
I am too. That's why this discussion. We will have the NumPy refactor done by end of July at the latest. Numpy 2.0 should be able to come out in August.
This thread got a bit side-tracked with the move to git, but I don't see a conclusion about what to release when.
My understanding is that there was a general agreement on splitting the code that breaks the ABI/API (datetime, maybe future refactoring) from everything else (mostly py3k port): - 1.5: everything in the trunk minus API/ABI breaking stuff - 2.0: still in flux, depends on how the refactoring will happen
cheers,
David
On May 27, 2010, at 6:51 AM, Ralf Gommers wrote:
On Wed, May 26, 2010 at 12:23 PM, Travis Oliphant <oliphant@enthought.com
wrote:
On May 25, 2010, at 5:06 PM, David Cournapeau wrote:
On Wed, May 26, 2010 at 6:19 AM, Charles R Harris charlesr.harris@gmail.com wrote:
Sounds good, but what if it doesn't get finished in a few months?
I think we
should get 2.0.0 out pronto, ideally it would already have been
released. I
think a major refactoring like this proposal should get the 3.0.0
label.
Naming it 3.0 or 2.1 does not matter much - I think we should avoid breaking things twice. I can see a few solutions:
- postpone 2.0 "indefinitely", until this new work is done
- backport py3k changes to 1.5 (which would be API and ABI
compatible with 1.4.1), and 2.0 would contain all the breaking changes.
This is an interesting idea and also workable.
I am really worried about breaking things once now and once in a few months (or even a year).
I am too. That's why this discussion. We will have the NumPy refactor done by end of July at the latest. Numpy 2.0 should be able to come out in August.
This thread got a bit side-tracked with the move to git, but I don't see a conclusion about what to release when.
Even if the refactoring is done in July, I think a 2.0 release with so many major changes will probably need a longer test/release cycle. So if we say September, do you still want a 1.5 release?
I think this makes sense so that we can reschedule NumPy 2.0 for September and still provide a release with the Python 3k changes (I am assuming these can be done in an ABI-compatible way).
-Travis
Fri, 28 May 2010 12:13:57 -0500, Travis Oliphant wrote: [clip]
I think this makes sense so that we can reschedule NumPy 2.0 for September and still provide a release with the Python 3k changes (I am assuming these can be done in an ABI-compatible way).
As far as I remember, the Py3 changes did not require breaking ABI (on Py2).
On Sat, May 29, 2010 at 9:05 AM, Pauli Virtanen pav@iki.fi wrote:
Fri, 28 May 2010 12:13:57 -0500, Travis Oliphant wrote: [clip]
I think this makes sense so that we can reschedule NumPy 2.0 for September and still provide a release with the Python 3k changes (I am assuming these can be done in an ABI-compatible way).
As far as I remember, the Py3 changes did not require breaking ABI (on Py2).
Good. How much work is it to remove datetime, and who wants to do it?
Ralf
On Sun, May 30, 2010 at 9:52 AM, Ralf Gommers ralf.gommers@googlemail.comwrote:
On Sat, May 29, 2010 at 9:05 AM, Pauli Virtanen pav@iki.fi wrote:
Fri, 28 May 2010 12:13:57 -0500, Travis Oliphant wrote: [clip]
I think this makes sense so that we can reschedule NumPy 2.0 for September and still provide a release with the Python 3k changes (I am assuming these can be done in an ABI-compatible way).
As far as I remember, the Py3 changes did not require breaking ABI (on Py2).
Good. How much work is it to remove datetime, and who wants to do it?
Hey, I thought that was your job ;) Maybe the best thing is to start by making a branch for the removal and we can argue about who gets the short end of the stick later...
Chuck
On Mon, May 31, 2010 at 2:06 AM, Charles R Harris <charlesr.harris@gmail.com
wrote:
On Sun, May 30, 2010 at 9:52 AM, Ralf Gommers <ralf.gommers@googlemail.com
wrote:
On Sat, May 29, 2010 at 9:05 AM, Pauli Virtanen pav@iki.fi wrote:
Fri, 28 May 2010 12:13:57 -0500, Travis Oliphant wrote: [clip]
I think this makes sense so that we can reschedule NumPy 2.0 for September and still provide a release with the Python 3k changes (I am assuming these can be done in an ABI-compatible way).
As far as I remember, the Py3 changes did not require breaking ABI (on Py2).
Good. How much work is it to remove datetime, and who wants to do it?
Hey, I thought that was your job ;) Maybe the best thing is to start by making a branch for the removal and we can argue about who gets the short end of the stick later...
Manage the release yes, do the heavy lifting not necessarily:) OK, I'll
make the branch tonight.
If the removal just involves the same changes that were made for 1.4.1 then I can do it. But I'm not familiar with this code and if it's more work, I probably won't have time for it between the scipy 0.8.0 release and my 'real job', like you call it.
Cheers, Ralf
On Sun, May 30, 2010 at 5:53 PM, Ralf Gommers ralf.gommers@googlemail.comwrote:
On Mon, May 31, 2010 at 2:06 AM, Charles R Harris < charlesr.harris@gmail.com> wrote:
On Sun, May 30, 2010 at 9:52 AM, Ralf Gommers < ralf.gommers@googlemail.com> wrote:
On Sat, May 29, 2010 at 9:05 AM, Pauli Virtanen pav@iki.fi wrote:
Fri, 28 May 2010 12:13:57 -0500, Travis Oliphant wrote: [clip]
I think this makes sense so that we can reschedule NumPy 2.0 for September and still provide a release with the Python 3k changes (I am assuming these can be done in an ABI-compatible way).
As far as I remember, the Py3 changes did not require breaking ABI (on Py2).
Good. How much work is it to remove datetime, and who wants to do it?
Hey, I thought that was your job ;) Maybe the best thing is to start by making a branch for the removal and we can argue about who gets the short end of the stick later...
Manage the release yes, do the heavy lifting not necessarily:) OK, I'll
make the branch tonight.
If the removal just involves the same changes that were made for 1.4.1 then I can do it. But I'm not familiar with this code and if it's more work, I probably won't have time for it between the scipy 0.8.0 release and my 'real job', like you call it.
I think it may be a bit trickier because Travis made more changes. I think the relevant commits are r8113..r8115 and 8107..8108. After removing those we still need to remove the same stuff as we did for the 1.4.1 release. We will need some way of testing if the removal was successful. We probably want to make sure there is a documentation update also, maybe before making the branch.
Chuck
On Mon, May 31, 2010 at 8:23 AM, Charles R Harris <charlesr.harris@gmail.com
wrote:
On Sun, May 30, 2010 at 5:53 PM, Ralf Gommers <ralf.gommers@googlemail.com
wrote:
On Mon, May 31, 2010 at 2:06 AM, Charles R Harris < charlesr.harris@gmail.com> wrote:
Hey, I thought that was your job ;) Maybe the best thing is to start by
making a branch for the removal and we can argue about who gets the short end of the stick later...
Manage the release yes, do the heavy lifting not necessarily:) OK, I'll
make the branch tonight.
If the removal just involves the same changes that were made for 1.4.1 then I can do it. But I'm not familiar with this code and if it's more work, I probably won't have time for it between the scipy 0.8.0 release and my 'real job', like you call it.
I think it may be a bit trickier because Travis made more changes. I think the relevant commits are r8113..r8115 and 8107..8108. After removing those we still need to remove the same stuff as we did for the 1.4.1 release. We will need some way of testing if the removal was successful.
That still looks like it's not an insane amount of work.
We probably want to make sure there is a documentation update also, maybe
before making the branch.
I checked and there's not too much to merge, most of the changes (
http://docs.scipy.org/numpy/patch/) don't apply cleanly or at all. The latter because they're docs for constants, lists, etc.
The biggest chuck of recent changes is for the polynomial and chebyshev docs, can you give your opinion on those Charles? The OK to Apply is set to True for all of them, but I'm not sure who did that. The patch generation won't work for many of those docs, so if you could check if any docs should be merged manually right now that would be useful.
@ David G: there are some conflicts in docs you recently edited, http://docs.scipy.org/numpy/merge/. Would you mind resolving those?
Thanks, Ralf
On Mon, May 31, 2010 at 4:54 AM, Ralf Gommers ralf.gommers@googlemail.comwrote:
On Mon, May 31, 2010 at 8:23 AM, Charles R Harris < charlesr.harris@gmail.com> wrote:
On Sun, May 30, 2010 at 5:53 PM, Ralf Gommers < ralf.gommers@googlemail.com> wrote:
On Mon, May 31, 2010 at 2:06 AM, Charles R Harris < charlesr.harris@gmail.com> wrote:
Hey, I thought that was your job ;) Maybe the best thing is to start by
making a branch for the removal and we can argue about who gets the short end of the stick later...
Manage the release yes, do the heavy lifting not necessarily:) OK, I'll
make the branch tonight.
If the removal just involves the same changes that were made for 1.4.1 then I can do it. But I'm not familiar with this code and if it's more work, I probably won't have time for it between the scipy 0.8.0 release and my 'real job', like you call it.
I think it may be a bit trickier because Travis made more changes. I think the relevant commits are r8113..r8115 and 8107..8108. After removing those we still need to remove the same stuff as we did for the 1.4.1 release. We will need some way of testing if the removal was successful.
That still looks like it's not an insane amount of work.
We probably want to make sure there is a documentation update also, maybe
before making the branch.
I checked and there's not too much to merge, most of the changes (
http://docs.scipy.org/numpy/patch/) don't apply cleanly or at all. The latter because they're docs for constants, lists, etc.
The biggest chuck of recent changes is for the polynomial and chebyshev docs, can you give your opinion on those Charles? The OK to Apply is set to True for all of them, but I'm not sure who did that.
I'll go have a look. David (G) worked with me off-line so I expect they are generally OK. The markup seems a bit excessive to me, but that is probably a tradeoff between the appearance of the generated docs and the terminal help. I don't know what is standard practice there these days.
The patch generation won't work for many of those docs, so if you could check if any docs should be merged manually right now that would be useful.
The {Chebyshev, Polynomial} documentation that has conflicts can't be edited on the web, as they reference documents generated from a template at module load time. I'll fix that at some point, but at the moment it is easier to maintain one template than to maintain two separate versions. I'll hand merge anything there that looks appropriate.
@ David G: there are some conflicts in docs you recently edited,
http://docs.scipy.org/numpy/merge/. Would you mind resolving those?
Chuck
On Mon, May 31, 2010 at 3:54 AM, Ralf Gommers ralf.gommers@googlemail.comwrote:
On Mon, May 31, 2010 at 8:23 AM, Charles R Harris < charlesr.harris@gmail.com> wrote:
On Sun, May 30, 2010 at 5:53 PM, Ralf Gommers < ralf.gommers@googlemail.com> wrote:
On Mon, May 31, 2010 at 2:06 AM, Charles R Harris < charlesr.harris@gmail.com> wrote:
Hey, I thought that was your job ;) Maybe the best thing is to start by
making a branch for the removal and we can argue about who gets the short end of the stick later...
Manage the release yes, do the heavy lifting not necessarily:) OK, I'll
make the branch tonight.
If the removal just involves the same changes that were made for 1.4.1 then I can do it. But I'm not familiar with this code and if it's more work, I probably won't have time for it between the scipy 0.8.0 release and my 'real job', like you call it.
I think it may be a bit trickier because Travis made more changes. I think the relevant commits are r8113..r8115 and 8107..8108. After removing those we still need to remove the same stuff as we did for the 1.4.1 release. We will need some way of testing if the removal was successful.
That still looks like it's not an insane amount of work.
We probably want to make sure there is a documentation update also, maybe
before making the branch.
I checked and there's not too much to merge, most of the changes (
http://docs.scipy.org/numpy/patch/) don't apply cleanly or at all. The latter because they're docs for constants, lists, etc.
The biggest chuck of recent changes is for the polynomial and chebyshev docs, can you give your opinion on those Charles? The OK to Apply is set to True for all of them, but I'm not sure who did that. The patch generation won't work for many of those docs, so if you could check if any docs should be merged manually right now that would be useful.
@ David G: there are some conflicts in docs you recently edited, http://docs.scipy.org/numpy/merge/. Would you mind resolving those?
Nothing to resolve: those docstrings are generated via a template and should not be modified via the wiki at all - any differences between what's in the wiki and svn are inadvertent and can be ignored.
DG
Thanks, Ralf
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
I think 2.0 would be a bit early for this. Is there any reason it couldn't be done in 2.1? What is the planned policy with regards to the visible interface for extensions? It would also be nice to have a rough idea of how the resulting code would be layered, i.e., what is the design for this re-factoring. Simply having a design would be a major step forward.
The problem with doing it in 2.1 is that this re-factoring will require extensions to be re-built. The visible interface to extensions will not change, but there will likely be ABI incompatibility. It seems prudent to do this in NumPy 2.0. Perhaps we can also put in place the ABI-protecting indirection approaches that David C. was suggesting earlier.
Some aspects of the design are still being fleshed out, but the basic idea is to separate out a core library that is as independent of the Python C-API as possible. There will likely be at least some dependency on the Python C-API (reference counting and error handling and possibly others) which any interface would have to provide in a very simple Python.h -- equivalent, for example.
Our purpose is to allow NumPy to be integrated with other languages or other frameworks systems without explicitly relying on CPython. There are a lot of questions as to how this will work, and so much of that is being worked out. Part of the reason for this mail is to help ensure that as much of this discussion as possible takes place in public.
Sounds good, but what if it doesn't get finished in a few months? I think we should get 2.0.0 out pronto, ideally it would already have been released. I think a major refactoring like this proposal should get the 3.0.0 label. Admittedly that makes keeping a refactored branch current with fixes going into the trunk a hassle, but perhaps that can be worked around somewhat by clearly labeling what files will be touched in the refactoring and possibly rearranging the content of the existing files. This requires a game plan and a clear idea of the goal. Put simply, I think the proposed schedule is too ambitious and needs to be fleshed out. This refactoring isn't going to be as straight forward as the python3k port because a lot of design decisions need to be made along the way.
You are correct that there is not much time. However, our timeline is middle of July and we do have dedicated resources. I was also hoping to have discussions at SciPy to accelerate the process.
-Travis
On Wed, May 26, 2010 at 4:37 AM, Travis Oliphant oliphant@enthought.com wrote:
Hi everyone,
There has been some talk about re-factoring NumPy to separate out the Python C-API layer and make NumPy closer to a C-library. I know there are a few different ideas about what this means, and also that people are very busy. I also know there is a NumPy 2.0 release that is in the works.
I'm excited to let everyone know that we (at Enthought) have been able to find resources (about 3 man months) to work on this re-factoring project and Scott and Jason (both very experienced C and Python programmers) are actively pursuing it. My hope is that NumPy 2.0 will contain this re-factoring (which should be finished just after SciPy 2010 --- where I'm going to organize a Sprint on NumPy which will include at least date-time improvements and re-factoring work).
This sounds great. As for how this is to be done, what would be the numpy aspects to be worked on first ? The obvious candidates are broadcasting, indexing, and ufunc ?
Concerning the goal of making numpy available to other languages, is the strategy already decided ? For example, will the core C API be reference-counted though python, or will this be abstracted ? It may be not feasable in the timeframe that Enthought has in mind, but I have been wondering for some time if we could do something like LUA does. LUA has a stronger story than python in terms of embedding within other languages, and the stack may be one solution for abstracting the reference counting (only the API around the stack and its implementation would need to be aware of a particular implementation of memory management).
David
While we have specific goals for the re-factoring, we want this activity to be fully integrated with the NumPy community and Scott and Jason want to interact with the community as much as feasible as they suggest re-factoring changes (though they both have more experience with phone-conversations to resolve concerns than email chains and so some patience from everybody will be appreciated).
Because Jason and Scott are new to this mailing list (but not new to NumPy), I wanted to introduce them so they would feel more comfortable posting questions and people would have some context as to what they were trying to do.
Scott and Jason are both very proficient and skilled programmers and I have full confidence in their abilities. That said, we very much want the input of as many people as possible as we pursue the goal of grouping together more tightly the Python C-API interface layer to NumPy.
I will be involved in some of the discussions, but am currently on a different project which has tight schedules and so I will only be able to provide limited "mailing-list" visibility.
Best regards,
-Travis
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Tue, May 25, 2010 at 4:31 PM, David Cournapeau cournape@gmail.comwrote:
On Wed, May 26, 2010 at 4:37 AM, Travis Oliphant oliphant@enthought.com wrote:
Hi everyone,
There has been some talk about re-factoring NumPy to separate out the Python C-API layer and make NumPy closer to a C-library. I know there are a few different ideas about what this means, and also that people are very busy. I also know there is a NumPy 2.0 release that is in the works.
I'm excited to let everyone know that we (at Enthought) have been able to find resources (about 3 man months) to work on this re-factoring project and Scott and Jason (both very experienced C and Python programmers) are actively pursuing it. My hope is that NumPy 2.0 will contain this re-factoring (which should be finished just after SciPy 2010 --- where I'm going to organize a Sprint on NumPy which will include at least date-time improvements and re-factoring work).
This sounds great. As for how this is to be done, what would be the numpy aspects to be worked on first ? The obvious candidates are broadcasting, indexing, and ufunc ?
If it was ufuncs alone it could be broken out into a separate project where ufuncs operated on objects that exposed the buffer interface. This would keep it separate from from numpy and at some point we could drop it into numpy trunk. If things went that way it would also make sense to drop support for versions of python that don't support the new buffer protocol. Speaking of which, can the datetime types be supported with the buffer interface? If not, there is some design work to be done there as to how to use the buffer interface for new types.
<snip>
Chuck
participants (12)
-
Benjamin Root -
Charles R Harris -
David Cournapeau -
David Goldsmith -
Jarrod Millman -
josef.pktd＠gmail.com -
Matthew Brett -
Pauli Virtanen -
Ralf Gommers -
Sebastian Walter -
Stéfan van der Walt -
Travis Oliphant