The process I intend to follow for any proposed changes to NumPy
Hey all,
I just wanted to clarify, that I am very excited about a few ideas I have --- but I don't have time myself to engage in the community process to get these changes into NumPy. However, those are real processes --- I've been coaching a few people in those processes for the past several years already.
So, rather than do nothing, what I'm looking to do is to work with a few people who I can share my ideas with, get excited about the ideas, and then who will work with the community to get them implemented. That's what I was announcing and talking about yesterday --- looking for interested people who want to work on NumPy *with* the NumPy community.
In my enthusiasm, I realize that some may have mis-understood my intention. There is no 'imminent' fork, nor am I planning on doing some crazy amount of work that I then try to force on other developers of NumPy.
What I'm planning to do is find people to train on NumPy code base (people to increase the diversity of the developers would be ideal -- but hard to accomplish). I plan to train them on NumPy based on my experience, and on what I think should be done --- and then have *them* work through the community process and engage with others to get consensus (hopefully not losing too much in translation in the process --- but instead getting even better).
During that process I will engage as a member of the community and help write NEPs and other documents and help clarify where it makes sense as I can. I will be filtering for people that actually want to see NumPy get better. Until I identify the people and work with them, it will be hard to tell how this will best work. So, stay tuned.
If all goes well, what you should see in a few weeks time are specific proposals, a branch or two, and the beginnings of some pull requests. If you don't see that, then I will not have found the right people to help me, and we will all continue to go back to searching.
While I'm expecting the best, in the worst case, we get additional people who know the NumPy code base and can help squash bugs as well as implement changes that are desired. Three things are needed if you want to participate in this: 1) A willingness to work with the open source community, 2) a deep knowledge of C and in-particular CPython's brand of C, and 3) a willingness to engage with me, do a mind-meld and dump around the NumPy code base, and then improve on what is in my head with the rest of the community.
Thanks,
-Travis
Travis,
I'm sure you appreciate that this might all look a bit scary, given the recent discussion about numpy governance.
But it's an open-source project, and I, at least, fully understand that going through a big process is NOT the way to get a new idea tried out and implemented. So I think think this is a great development -- I know I want to see something like this dtype work done.
So, as someone who has been around this community for a long time, and dependent on Numeric, numarray, and numpy over the years, this looks like a great development.
And, in fact, with the new governance effort -- I think less scary -- people can go off and work on a branch or fork, do good stuff, and we, as a community, can be assured that API (or even ABI) changes won't be thrust upon us unawares :-)
As for the technical details -- I get a bit lost, not fully understanding the current dtype system either, but do your ideas take us in the direction of having dtypes independent of the container and ufunc machinery -- and thus easier to create new dtypes (even in Python?) 'cause that would be great.
I hope you find the partner you're looking for -- that's a challenge!
-Chris
Hey Chris (limiting to NumPy only),
I've had some great conversations with Nathaniel in the past few days and I'm glad he posted his thoughts so that there is no confusion about governance or what I was implying.
With respect to governance, I'm very supportive of what everyone is doing in organizing a governance document and approach and appreciate the effort of Nathaniel and others to move this forward. Nothing I said was meant to imply differently. I'm sorry if it made anyone nervous.
I'm a very enthusiastic person when I get an idea of what to do. I like to see things implemented. In this case, it also turns out that in terms of overall architecture, my ideas are actually very similar to Nathaniel's ideas. That's a good sign. We have different tactical approaches as to how to move forward, but I think it's a good thing to note that we see a very similar path forward. Nothing will be done in NumPy itself except via pull-request and review.
My approach for the ideas I'm pursuing will be to organize people around two new prototype packages I'm calling memtype and gufunc. The purpose of these is to allow playing with the design and ideas quickly before looking at how to put them into NumPy itself --- there will also be some training involved in getting people up to speed. There was a long discussion today at this BIDS data-structures for data-science summit part of which talked about how to improve NumPy's dtype system. I would love to these independent objects evolve into independent packages that could even go into Python standard library. Not everyone agrees that is the best idea, but regardless of whether this happens or not, the intent is to do work that could go into NumPy now.
I look forward to the activity.
-Travis
On Mon, Sep 14, 2015 at 10:46 AM, Chris Barker chris.barker@noaa.gov wrote:
Travis,
I'm sure you appreciate that this might all look a bit scary, given the recent discussion about numpy governance.
But it's an open-source project, and I, at least, fully understand that going through a big process is NOT the way to get a new idea tried out and implemented. So I think think this is a great development -- I know I want to see something like this dtype work done.
So, as someone who has been around this community for a long time, and dependent on Numeric, numarray, and numpy over the years, this looks like a great development.
And, in fact, with the new governance effort -- I think less scary -- people can go off and work on a branch or fork, do good stuff, and we, as a community, can be assured that API (or even ABI) changes won't be thrust upon us unawares :-)
As for the technical details -- I get a bit lost, not fully understanding the current dtype system either, but do your ideas take us in the direction of having dtypes independent of the container and ufunc machinery -- and thus easier to create new dtypes (even in Python?) 'cause that would be great.
I hope you find the partner you're looking for -- that's a challenge!
-Chris
--
Christopher Barker, Ph.D. Oceanographer
Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@noaa.gov
NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Hi Travis,
On Sun, Sep 13, 2015 at 4:51 PM, Travis Oliphant travis@continuum.io wrote:
Hey all,
I just wanted to clarify, that I am very excited about a few ideas I have --- but I don't have time myself to engage in the community process to get these changes into NumPy. However, those are real processes --- I've been coaching a few people in those processes for the past several years already.
So, rather than do nothing, what I'm looking to do is to work with a few people who I can share my ideas with, get excited about the ideas, and then who will work with the community to get them implemented. That's what I was announcing and talking about yesterday --- looking for interested people who want to work on NumPy *with* the NumPy community.
In my enthusiasm, I realize that some may have mis-understood my intention. There is no 'imminent' fork, nor am I planning on doing some crazy amount of work that I then try to force on other developers of NumPy.
What I'm planning to do is find people to train on NumPy code base (people to increase the diversity of the developers would be ideal -- but hard to accomplish). I plan to train them on NumPy based on my experience, and on what I think should be done --- and then have *them* work through the community process and engage with others to get consensus (hopefully not losing too much in translation in the process --- but instead getting even better).
During that process I will engage as a member of the community and help write NEPs and other documents and help clarify where it makes sense as I can. I will be filtering for people that actually want to see NumPy get better. Until I identify the people and work with them, it will be hard to tell how this will best work. So, stay tuned.
If all goes well, what you should see in a few weeks time are specific proposals, a branch or two, and the beginnings of some pull requests. If you don't see that, then I will not have found the right people to help me, and we will all continue to go back to searching.
While I'm expecting the best, in the worst case, we get additional people who know the NumPy code base and can help squash bugs as well as implement changes that are desired. Three things are needed if you want to participate in this: 1) A willingness to work with the open source community, 2) a deep knowledge of C and in-particular CPython's brand of C, and 3) a willingness to engage with me, do a mind-meld and dump around the NumPy code base, and then improve on what is in my head with the rest of the community.
I don't have time to do the work myself, but I'll be happy to help in terms of the community. Travis has paid me in the past to do the NumPy 1.7 release, and implement/fix some things that Travis had in mind. I had no prior involvement with the NumPy codebase / community, and within few months I had to get myself familiar with the code, fix release critical bugs, get my PRs accepted by the community, eventually get push access (from the core developers, not from Travis) so that I can merge other people's patches, and most importantly, make the community allow me to actually do the release, even though I came from outside. As far as I know, Travis didn't have to pull any strings for me, and as far as I know, there was no friction in the community either --- there was a job to be done, we got it done, and that was it.
So I'll be happy to share my experience and help the person (if coming from outside) how to do this. In this case a push access is not necessary, so it's just about sending high quality PRs and good communication.
Ondrej
Hi all,
Thanks, Travis, for the followup. I know some people were confused or concerned by some points in Travis’s recent emails (as was I, initially), but after checking in with Travis and the NumPy steering council, it sounds like the main points of possible confusion are actually things where we at least are actually on the same page. So to avoid any uncertainty or miscommunication, I just want to reiterate some hopefully-uncontroversial points here:
- There is currently no plan whatsoever for NumPy to drop Python 2 support; something like ~80% of our users are still using Python 2, and we anticipate that both Python 2 and Python 3 will continue to receive full support for the foreseeable future.
- While it is possible that there will eventually be a compatibility-breaking “NumPy 2.0” release, it won’t happen without a community consensus that this is a good idea, and that consensus has not yet been reached.
- As a clarification on how NumPy governance works: While we all continue to be grateful to Travis for his past contributions, in terms of formal authority he stepped down from his leadership role in the project several years ago, and the project has since switched to a community-driven governance model [1]. Under this model, all contributors participate as equal peers, and (outside of specific exceptional circumstances) all contributors should be assumed to be speaking only on behalf of themselves, not the project as a whole.
- Discussion about the best way to improve NumPy’s dtype system is ongoing, and several approaches are under consideration. No proposal will be accepted without review and consensus by the broader NumPy community. We welcome anyone who’s interested in these issues to join us on the mailing list -- the more input we have, the better the result will be :-).
Apologies for the wide distribution of this message; I suggest any followups be directed to numpy-discussion@scipy.org only.
Thanks, - Nathaniel
[1] http://thread.gmane.org/gmane.comp.python.numeric.general/61106 (we’ll move it into the repository soon, we promise!)
On Sun, Sep 13, 2015 at 3:51 PM, Travis Oliphant travis@continuum.io wrote:
Hey all,
I just wanted to clarify, that I am very excited about a few ideas I have --- but I don't have time myself to engage in the community process to get these changes into NumPy. However, those are real processes --- I've been coaching a few people in those processes for the past several years already.
So, rather than do nothing, what I'm looking to do is to work with a few people who I can share my ideas with, get excited about the ideas, and then who will work with the community to get them implemented. That's what I was announcing and talking about yesterday --- looking for interested people who want to work on NumPy *with* the NumPy community.
In my enthusiasm, I realize that some may have mis-understood my intention. There is no 'imminent' fork, nor am I planning on doing some crazy amount of work that I then try to force on other developers of NumPy.
What I'm planning to do is find people to train on NumPy code base (people to increase the diversity of the developers would be ideal -- but hard to accomplish). I plan to train them on NumPy based on my experience, and on what I think should be done --- and then have *them* work through the community process and engage with others to get consensus (hopefully not losing too much in translation in the process --- but instead getting even better).
During that process I will engage as a member of the community and help write NEPs and other documents and help clarify where it makes sense as I can. I will be filtering for people that actually want to see NumPy get better. Until I identify the people and work with them, it will be hard to tell how this will best work. So, stay tuned.
If all goes well, what you should see in a few weeks time are specific proposals, a branch or two, and the beginnings of some pull requests. If you don't see that, then I will not have found the right people to help me, and we will all continue to go back to searching.
While I'm expecting the best, in the worst case, we get additional people who know the NumPy code base and can help squash bugs as well as implement changes that are desired. Three things are needed if you want to participate in this: 1) A willingness to work with the open source community, 2) a deep knowledge of C and in-particular CPython's brand of C, and 3) a willingness to engage with me, do a mind-meld and dump around the NumPy code base, and then improve on what is in my head with the rest of the community.
Thanks,
-Travis
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
participants (4)
-
Chris Barker -
Nathaniel Smith -
Ondřej Čertík -
Travis Oliphant