Mailman 3 Deprecate chararray [was [SciPy-dev] Plea for help] - NumPy-Discussion

Deprecate chararray [was [SciPy-dev] Plea for help]

David Goldsmith

20 Aug 2009 20 Aug '09

9:57 p.m.

--- On Thu, 8/20/09, Christopher Hanley  wrote:

> I'd like to respectfully request that we move any
> discussion of what  
> to do with the numpy.char module to the numpy list.

NP, done.

> I'm a little concerned about some of the assumptions that
> are being  
> made about the number of users of the module.  I would
> also like to  
> better understand the reasons for wanting to dump it. 

I think Ralf did a pretty good job of synopsizing the reasons for deprecation, but since we're moving the thread, I'll reprint them here:

0) "it gets very little use" (an assumption you presumably dispute);

1) "is pretty much undocumented" (less true than a week ago, but still true for several of the attributes, with another handful or so falling into the category of "poorly documented");

2) "probably more buggy than most other parts of NumPy" ("probably" being a euphemism, IMO);

3) "there is not a really good use-case for it" (a conjecture, but one that has yet to be challenged by counter-example); 

4) it's not the first time its presence in NumPy has been questioned ("as Stefan pointed out when asking this same question last year")

5) NumPy already has a (perhaps superior) alternative ("object arrays would do nicely if one needs this functionality");

to which I'll add:

6) it is, on its face, "counter to the spirit" of NumPy.

So far, IIRC, the only reason in favor of its continued inclusion is inertia.

> Let me be  
> clear.  I'm not opposed to change.  However
> breaking other people's  
> code just for the sake of change seems like a poor reason

So, I don't think we're proposing this "just for the sake of change"

> and a mean  
> thing to do to our customers.

Apologies, but it is not proposed maliciously.

The only other things I would add by way of "review" from the scipy-dev thread:

a compromise proposal (made by Ralf):

"Put clearly in the docs that this module exists for backwards compatibility reasons, and is not recommended for new development"

and a clarification of deprecation process (provided by Robert):

"[asked by the present author] How has deprecation in Numpy worked in the past - by dictum, vote, or consensus?

[Robert's answer] Consensus or dictum without major objection. Voting is pointless except to inform one of those." 

Thanks for your time and consideration.

David Goldsmith

> Thank you for your time and help,
> Chris
> 
> 
> -- 
> Christopher Hanley
> Senior Systems Software Engineer
> Space Telescope Science Institute
> 3700 San Martin Drive
> Baltimore MD, 21218
> (410) 338-4338
> 
> On Aug 20, 2009, at 1:35 AM, Robert Kern wrote:
> 
> > On Wed, Aug 19, 2009 at 20:03, David  
> > Goldsmith
> wrote:
> >> I'm going to take it a step further: "breakage" is
> always the  
> >> deterrent to change, and yet "change we must"
> (i.e., "adapt or  
> >> die").  It's certainly not without precedent
> - even within Numpy, I  
> >> believe - for things (though perhaps not whole
> namespaces) to be  
> >> deemed "to-be-deprecated," have a warning to this
> effect  
> >> established in one x.[even #].0 release, and then
> be removed by the  
> >> x.[even # + 2 or + 4].0 release.  How has
> deprecation in Numpy  
> >> worked in the past - by dictum, vote, or
> consensus?
> >
> > Consensus or dictum without major objection. Voting is
> pointless
> > except to inform one of those.
> >
> > -- 
> > Robert Kern
> >
> > "I have come to believe that the whole world is an
> enigma, a harmless
> > enigma that is made terrible by our own mad attempt to
> interpret it as
> > though it had an underlying truth."
> >  -- Umberto Eco
> > _______________________________________________
> > Scipy-dev mailing list
> > Scipy-dev@scipy.org
> > http://mail.scipy.org/mailman/listinfo/scipy-dev
> 
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev@scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>

Show replies by date

Christopher Hanley

20 Aug 20 Aug

11:02 p.m.

New subject: Deprecate chararray [was [SciPy-dev] Plea for help]

Here is what I know about the chararray usage at STScI since first  
looking into it this morning.

It is used in PyFITS and within the COS instrument calibration code.   
I have not heard back from the other projects yet given most of our  
developers are away at this time.

It appears that the COS code can be changed easily.  I am waiting to  
hear back from PyFITs.

Also, I do not know how many people use this particular feature.   
However I would point out that many people who use numpy are not also  
on the mailing lists.  Most of the STScI do not follow the numpy  
list.  I serve as our point of contact to the numpy community.  I'm  
trying to gather a list of projects that use this feature and specific  
use cases for you.  As I do not use this module myself I cannot  
counter your arguments at this time.   If we decide to deprecate this  
module would we reverse this decision if we then find out that the  
assumptions that went into the decision were in error?

Another concern is that we told people coming from numarray to use  
this module.  It is my opinion that at this point in the numpy release  
cycle that an API change needs a very strong justification.  Anecdotes  
about the number of users, a "change or die" philosophy, and an un- 
articulated notion of  "the spirit of numpy"  do not in my  
consideration meet that high bar.  If you would like us to provide  
additional documentation and tests that would be possible.  I'll do it  
myself if that is the only think keeping the module from remaining in  
numpy.

This also raises the question of what else is going to go?  Will  
recarray be removed?  What about the numarray c-api compatibility layer?

Like I said earlier, I'm not opposed to change.  I am just of the  
opinion that this isn't a simple, cut and dry decision.

For those at SciPy 2009 feel free to come yell at me and beat me with  
sticks.  I'm the fat guy in jeans and a blue shirt sitting towards the  
back middle on the left.

Cheers,
Chris

-- 
Christopher Hanley
Senior Systems Software Engineer
Space Telescope Science Institute
3700 San Martin Drive
Baltimore MD, 21218
(410) 338-4338

On Aug 20, 2009, at 12:27 PM, David Goldsmith wrote:

> --- On Thu, 8/20/09, Christopher Hanley  wrote:
>
>> I'd like to respectfully request that we move any
>> discussion of what
>> to do with the numpy.char module to the numpy list.
>
> NP, done.
>
>> I'm a little concerned about some of the assumptions that
>> are being
>> made about the number of users of the module.  I would
>> also like to
>> better understand the reasons for wanting to dump it.
>
> I think Ralf did a pretty good job of synopsizing the reasons for  
> deprecation, but since we're moving the thread, I'll reprint them  
> here:
>
> 0) "it gets very little use" (an assumption you presumably dispute);
>
> 1) "is pretty much undocumented" (less true than a week ago, but  
> still true for several of the attributes, with another handful or so  
> falling into the category of "poorly documented");
>
> 2) "probably more buggy than most other parts of NumPy" ("probably"  
> being a euphemism, IMO);
>
> 3) "there is not a really good use-case for it" (a conjecture, but  
> one that has yet to be challenged by counter-example);
>
> 4) it's not the first time its presence in NumPy has been questioned  
> ("as Stefan pointed out when asking this same question last year")
>
> 5) NumPy already has a (perhaps superior) alternative ("object  
> arrays would do nicely if one needs this functionality");
>
> to which I'll add:
>
> 6) it is, on its face, "counter to the spirit" of NumPy.
>
> So far, IIRC, the only reason in favor of its continued inclusion is  
> inertia.
>
>> Let me be
>> clear.  I'm not opposed to change.  However
>> breaking other people's
>> code just for the sake of change seems like a poor reason
>
> So, I don't think we're proposing this "just for the sake of change"
>
>> and a mean
>> thing to do to our customers.
>
> Apologies, but it is not proposed maliciously.
>
> The only other things I would add by way of "review" from the scipy- 
> dev thread:
>
> a compromise proposal (made by Ralf):
>
> "Put clearly in the docs that this module exists for backwards  
> compatibility reasons, and is not recommended for new development"
>
> and a clarification of deprecation process (provided by Robert):
>
> "[asked by the present author] How has deprecation in Numpy worked  
> in the past - by dictum, vote, or consensus?
>
> [Robert's answer] Consensus or dictum without major objection.  
> Voting is pointless except to inform one of those."
>
> Thanks for your time and consideration.
>
> David Goldsmith
>
>> Thank you for your time and help,
>> Chris
>>
>>
>> -- 
>> Christopher Hanley
>> Senior Systems Software Engineer
>> Space Telescope Science Institute
>> 3700 San Martin Drive
>> Baltimore MD, 21218
>> (410) 338-4338
>>
>> On Aug 20, 2009, at 1:35 AM, Robert Kern wrote:
>>
>>> On Wed, Aug 19, 2009 at 20:03, David
>>> Goldsmith
>> wrote:
>>>> I'm going to take it a step further: "breakage" is
>> always the
>>>> deterrent to change, and yet "change we must"
>> (i.e., "adapt or
>>>> die").  It's certainly not without precedent
>> - even within Numpy, I
>>>> believe - for things (though perhaps not whole
>> namespaces) to be
>>>> deemed "to-be-deprecated," have a warning to this
>> effect
>>>> established in one x.[even #].0 release, and then
>> be removed by the
>>>> x.[even # + 2 or + 4].0 release.  How has
>> deprecation in Numpy
>>>> worked in the past - by dictum, vote, or
>> consensus?
>>>
>>> Consensus or dictum without major objection. Voting is
>> pointless
>>> except to inform one of those.
>>>
>>> -- 
>>> Robert Kern
>>>
>>> "I have come to believe that the whole world is an
>> enigma, a harmless
>>> enigma that is made terrible by our own mad attempt to
>> interpret it as
>>> though it had an underlying truth."
>>>   -- Umberto Eco
>>> _______________________________________________
>>> Scipy-dev mailing list
>>> Scipy-dev@scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-dev
>>
>> _______________________________________________
>> Scipy-dev mailing list
>> Scipy-dev@scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-dev
>>
>
>
>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion

David Cournapeau

21 Aug 21 Aug

12:34 a.m.

New subject: [SciPy-dev] Deprecate chararray [was Plea for help]

On Thu, Aug 20, 2009 at 10:32 AM, Christopher Hanley wrote:

...

Another concern is that we told people coming from numarray to use this module. It is my opinion that at this point in the numpy release cycle that an API change needs a very strong justification. Anecdotes about the number of users, a "change or die" philosophy, and an un- articulated notion of "the spirit of numpy" do not in my consideration meet that high bar.

I agree those are not strong reasons without more backing. What worries me the most, in both numpy and scipy, is code that nobody knows about, without any test or documentation. When it breaks, we can't fix it. That's unsustainable in the long term, because it takes a lot of time that people could spend somewhere else more useful. Especially when you have C code which does not work on some platforms, with new version of python (python 3k port, for example). I much prefer removing code to having code that barely works and cannot be maintained. Old code that people are ready to maintain, I have nothing against. cheers, David

Christopher Hanley

5:13 a.m.

New subject: Removing scipy.stsci was [Re: [SciPy-dev] Deprecate chararray [was Plea for help]]

I agree with David's comments. In that theme I have removed scipy.stsci from scipy. Users get it directly from us at STScI via STSCI_PYTHON. It doesn't have any documentation in the doc system. It isn't by default in the scipy namespace. And as a recent bug report indicates they can't import it anyway. That should clean some code up. If someday a generic image processing library is added to scipy we can consider incorporating our modules back into scipy. Until that time I would rather remove the redundancy. It also help scipy's maintainability and frees me from having to worry about a fork in the code developing. Cheers, Chris -- Christopher Hanley Senior Systems Software Engineer Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21218 (410) 338-4338 On Aug 20, 2009, at 3:04 PM, David Cournapeau wrote:

...

On Thu, Aug 20, 2009 at 10:32 AM, Christopher Hanley wrote:

I agree those are not strong reasons without more backing. What worries me the most, in both numpy and scipy, is code that nobody knows about, without any test or documentation. When it breaks, we can't fix it. That's unsustainable in the long term, because it takes a lot of time that people could spend somewhere else more useful. Especially when you have C code which does not work on some platforms, with new version of python (python 3k port, for example).

I much prefer removing code to having code that barely works and cannot be maintained. Old code that people are ready to maintain, I have nothing against.

cheers,

David _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

Stéfan van der Walt

5:18 a.m.

New subject: Removing scipy.stsci was [Re: [SciPy-dev] Deprecate chararray [was Plea for help]]

Hi Chris 2009/8/20 Christopher Hanley :

...

That should clean some code up. If someday a generic image processing library is added to scipy we can consider incorporating our modules back into scipy. Until that time I would rather remove the redundancy. It also help scipy's maintainability and frees me from having to worry about a fork in the code developing.

We'll be spriting on an Image Processing Scikit this weekend. If you have any functions you'd like to include, let me know. Regards Stéfan

Christopher Hanley

5:21 a.m.

New subject: Removing scipy.stsci was [Re: [SciPy-dev] Deprecate chararray [was Plea for help]]

On Aug 20, 2009, at 7:48 PM, Stéfan van der Walt wrote: Hi Stefan,

...

We'll be spriting on an Image Processing Scikit this weekend. If you have any functions you'd like to include, let me know.

Regards Stéfan

Will the Image Processing Scikit be dedicated to working with a single image or stacks of images? Cheers, Chris

Christopher Hanley

5:27 a.m.

New subject: Removing scipy.stsci was [Re: [SciPy-dev] Deprecate chararray [was Plea for help]]

Hi Stefan, Never mind. I just found the Sprint website and read the description. I'm sorry I hadn't found this sooner. I would have made plans to stay and help. My apologizes. Sorry, Chris -- Christopher Hanley Senior Systems Software Engineer Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21218 (410) 338-4338 On Aug 20, 2009, at 7:51 PM, Christopher Hanley wrote:

...

On Aug 20, 2009, at 7:48 PM, Stéfan van der Walt wrote:

Hi Stefan,

...
We'll be spriting on an Image Processing Scikit this weekend. If you have any functions you'd like to include, let me know.

Regards Stéfan

Will the Image Processing Scikit be dedicated to working with a single image or stacks of images?

Cheers, Chris

James Turner

5:34 a.m.

New subject: Removing scipy.stsci was [Re: [SciPy-dev] Deprecate chararray [was Plea for help]]

Hi Chris & Stefan, I will be around for most of the weekend (as I believe will Perry). I'm not sure I'll be able to contribute a lot to coding, but if there's any stuff you want to co-ordinate between STScI and Stefan's scikit, let me know if I can help. That's probably about the most useful thing I could do. Cheers, James.

...

Hi Stefan,

Never mind. I just found the Sprint website and read the description. I'm sorry I hadn't found this sooner. I would have made plans to stay and help. My apologizes.

Sorry, Chris

Vincent Schut

1:28 p.m.

New subject: Removing scipy.stsci was [Re: [SciPy-dev] Deprecate chararray [was Plea for help]]

Christopher Hanley wrote:

...

Hi Stefan,

Never mind. I just found the Sprint website and read the description. I'm sorry I hadn't found this sooner. I would have made plans to stay and help. My apologizes.

Hi list, I just saw this too and would like to misuse this thread to suggest another enhancement related to image processing you might consider: make ndimage maskedarray aware. Unfortunately we don't have any money to spend, otherwise I'd love to support this financially too. But it's a thing I'm running into (ndimage not working with missing data, that is) regularly, and for which it often is pretty hard to work out a workaround. E.g. any of the resampling (ndimage.zoom) or kernel filtering routines choke on array's with NaN's, and don't recognize masked arrays. Alas virtually all of the image data I process (satellite imagery) contains missing/bad data... I know it probably will be a pretty involved task, as ndimage comes from numarray and seems to be largely implemented in C. But I really wanted to raise the issue now the image processing subject turns up once again, and hope some folks with more/better programming skills than me might like the idea... Oh and I know of course ndimage is scipy, and this list is numpy. But as the image processing subject emerged here, well... Cheers, Vincent Schut.

...

Sorry, Chris

Stéfan van der Walt

11:03 p.m.

New subject: Removing scipy.stsci was [Re: [SciPy-dev] Deprecate chararray [was Plea for help]]

Hi Vincent 2009/8/21 Vincent Schut :

...

I know it probably will be a pretty involved task, as ndimage comes from numarray and seems to be largely implemented in C. But I really wanted to raise the issue now the image processing subject turns up once again, and hope some folks with more/better programming skills than me might like the idea...

What would you like the behaviour to be? For example, how should ndimage.zoom handle these missing values? Regards Stéfan

Vincent Schut

24 Aug 24 Aug

6:39 p.m.

New subject: Removing scipy.stsci was [Re: [SciPy-dev] Deprecate chararray [was Plea for help]]

Stéfan van der Walt wrote:

...

Hi Vincent

2009/8/21 Vincent Schut :

...
I know it probably will be a pretty involved task, as ndimage comes from numarray and seems to be largely implemented in C. But I really wanted to raise the issue now the image processing subject turns up once again, and hope some folks with more/better programming skills than me might like the idea...

What would you like the behaviour to be? For example, how should ndimage.zoom handle these missing values?

Good question :-) I see 2 possibilities, both of them can be usefull in their own situations. Note that I am really not into splines mathematically, so my suggestions and terminology might not apply at all... 1. for any output cell that depends on a missing value in the input, return a missing/masked/NaN value, but (and I think this differs from the current implementation), for any output cell which could be calculated, return a proper output. Currently any array that contains one or more NaNs will give an output array full of NaNs (except for order=0, which shows this exact behaviour already). But maybe that's inherent to splines interpolation? This would at least allow input arrays with missing values (or masked arrays) to be used; this behaviour could be extended to many of the ndimage functions, like the kernel based stuff. FFT based convolutions could be another story altogether... 2. In case of zoom&co: only use the non-missing values to calculate the splines, thus effectively inter/extrapolating missing/masked values in the process. This probably raises a lot of new questions about the implementation. It would however be highly usefull for me... I don't know if a irregular grid based splines interpolation implementation exists? What I currently do in a case like this (zooming an array with missing values) is first fill the missing values by using ndimage.generic_filter with a kernel function that averages the non-missing values in the moving window. This works as long as there are not too many missing values next to each other, however it is very slow... I think that, if an effort like this is to be made, a thorough discussion on the possible behaviours of ndimage functions with missing values should take place on one of the numpy/scipy related mailing lists. I'm sure I'm not the only one with ideas and/or use cases for this, and I'm certainly not someone with a lot of theoretical knowledge in this area. Regards, Vincent.

...

Regards Stéfan

Stéfan van der Walt

21 Aug 21 Aug

5:44 a.m.

New subject: Removing scipy.stsci was [Re: [SciPy-dev] Deprecate chararray [was Plea for help]]

2009/8/20 Christopher Hanley :

...

Will the Image Processing Scikit be dedicated to working with a single image or stacks of images?

Thanks for the reminder -- I have to add ImageCollection to the set of features. Fernando started working on something similar in 2006, and I've implemented a cached reader here: http://mentat.za.net/supreme/doc/supreme.misc.io.ImageCollection-class.html To get back to your comment: what kind of operations would you like to see supported on multiple images? Regards Stéfan

Travis Oliphant

5:36 a.m.

New subject: Deprecate chararray [was Plea for help]

On Aug 20, 2009, at 2:04 PM, David Cournapeau wrote:

...

On Thu, Aug 20, 2009 at 10:32 AM, Christopher Hanley wrote:

...
Another concern is that we told people coming from numarray to use this module. It is my opinion that at this point in the numpy release cycle that an API change needs a very strong justification. Anecdotes about the number of users, a "change or die" philosophy, and an un- articulated notion of "the spirit of numpy" do not in my consideration meet that high bar.

I agree those are not strong reasons without more backing. What worries me the most, in both numpy and scipy, is code that nobody knows about, without any test or documentation. When it breaks, we can't fix it. That's unsustainable in the long term, because it takes a lot of time that people could spend somewhere else more useful. Especially when you have C code which does not work on some platforms, with new version of python (python 3k port, for example).

The claim that "chararray" is not understood is not true. I know about the code. It's not that difficult of a piece of code. It's utility can be questioned, but it was and may still be an important part of Numarray compatibilty. I was asked a question that I didn't have time to answer properly (it would have taken more than 5 minutes). That lack of answer does not constitute "nobody knows about" the code. It's a great idea to add to the docstring that the code was created for compatibility with numarray. But, I'm not convinced by any of the given reasons to remove the code from NumPy. -Travis -- Travis Oliphant Enthought Inc. 1-512-536-1057 http://www.enthought.com oliphant@enthought.com

David Cournapeau

5:44 a.m.

New subject: Deprecate chararray [was Plea for help]

On Thu, Aug 20, 2009 at 5:06 PM, Travis Oliphant wrote:

...

On Aug 20, 2009, at 2:04 PM, David Cournapeau wrote:

...
On Thu, Aug 20, 2009 at 10:32 AM, Christopher Hanley wrote:

...
Another concern is that we told people coming from numarray to use this module. It is my opinion that at this point in the numpy release cycle that an API change needs a very strong justification. Anecdotes about the number of users, a "change or die" philosophy, and an un- articulated notion of "the spirit of numpy" do not in my consideration meet that high bar.

I agree those are not strong reasons without more backing. What worries me the most, in both numpy and scipy, is code that nobody knows about, without any test or documentation. When it breaks, we can't fix it. That's unsustainable in the long term, because it takes a lot of time that people could spend somewhere else more useful. Especially when you have C code which does not work on some platforms, with new version of python (python 3k port, for example).

The claim that "chararray" is not understood is not true. I know about the code. It's not that difficult of a piece of code. It's utility can be questioned, but it was and may still be an important part of Numarray compatibilty.

I did not want to imply that chararray is unknown, sorry for the confusion. I just wanted to say that old code is not an argument to remove something. Whether someone is willing to maintain it is a much better argument IMO. cheers, David

Ralf Gommers

10:30 a.m.

New subject: [SciPy-dev] Deprecate chararray [was Plea for help]

On Thu, Aug 20, 2009 at 1:32 PM, Christopher Hanley wrote:

...

Also, I do not know how many people use this particular feature. However I would point out that many people who use numpy are not also on the mailing lists. Most of the STScI do not follow the numpy list. I serve as our point of contact to the numpy community. I'm trying to gather a list of projects that use this feature and specific use cases for you.

Great. Even one good use case might change my opinion of chararray.

...

As I do not use this module myself I cannot counter your arguments at this time. If we decide to deprecate this module would we reverse this decision if we then find out that the assumptions that went into the decision were in error?

That would make sense. Another concern is that we told people coming from numarray to use

...

this module. It is my opinion that at this point in the numpy release cycle that an API change needs a very strong justification. Anecdotes about the number of users, a "change or die" philosophy, and an un- articulated notion of "the spirit of numpy" do not in my consideration meet that high bar.

That is not very fair. I gave you four reasons for assuming there are not many users, other arguments you leave out here are the state of the code, lack of docs, tests and (most importantly) a use case.

...

If you would like us to provide additional documentation and tests that would be possible. I'll do it myself if that is the only think keeping the module from remaining in numpy.

Thanks a lot, that would definitely help. How about for now we document the module as being there for numarray compatibility and not recommended for new development? Then if you turn up a good use case we add it to the docs, and if you don't we revisit the deprecation issue? Cheers, Ralf

Michael Droettboom

22 Sep 22 Sep

11:28 p.m.

New subject: [SciPy-dev] Deprecate chararray [was Plea for help]

Sorry to resurrect a long-dead thread, but I've been continuing Chris Hanley's investigation of chararray at Space Telescope Science Institute (and the broader astronomical community) for a while and have some findings to report back. What I've taken from this thread is that chararray is in need of a maintainer. I am able to spend some time to the cause, but first would like to clarify what it will take to make it's continued inclusion more comfortable. Let me start with the use case. chararrays are extensively returned from pyfits (a tool to handle the standard astronomy data format). pyfits is the basis of many applications, and it would be impossible to audit all of that code. Most authors of those tools do not track numpy-discussion closely, which is why we don't hear from them on this list, but there is a great deal of pyfits-using code. Doing some spot-checking on this code, a common thing I see is SQL-like queries on recarrays of objects. For instance, it is very common to a have a table of objects, with a "Target" column which is a string, and do something like (where c is a chararray of the 'Target' column): subset = array[np.where(c.startswith('NGC'))] Strictly speaking, this is a use case for "vectorized string operations", not necessarily for the chararray class as it presently stands. One could almost as easily do: subset = array[np.where([x.startswith('NGC') for x in c])] ...and the latter is even slightly faster, since chararray currently loops in Python anyway. Even better, though, I have some experimental code to perform the loop in C, and I get 5x speed up on a table with ~120,000 rows. If that were to be included in numpy, that's a strong argument against recommending list comprehensions in user code. The use case suggests the continued existence of vectorized string operations in numpy -- whether that continues to be chararray, or some newer/better interface + chararray for backward compatibility, is an open question. Personally I think a less object-oriented approach and just having a namespace full of vectorized string functions might be cleaner than the current situation of needing to create a view class around an ndarray. I'm suggesting something like the following, using the same example, where {STR} is some namespace we would fill with vectorized string operations: subset = array[np.where(np.{STR}.startswith(c, 'NGC'))] Now on to chararray as it now stands. I view chararray as really two separable pieces of functionality: 1) Convenience to perform vectorized string operations using '.method' syntax, or in some cases infix operators (+, *) 2) Implicit "rstrip"ping of values (Note that raw ndarray's truncate values at the first NULL character, like C strings, but chararrays will strip any and all whitespace characters from the end). Changing (2) just seems to be asking to be the source of subtle bugs. Unfortunately, there's an inconsistency between 1) and 2) in the present implementation. For example: In [9]: a = np.char.array(['a ']) In [10]: a Out[10]: chararray(['a'], dtype='|S3') In [11]: a[0] == 'a' Out[11]: True In [12]: a.endswith('a') Out[12]: array([False], dtype=bool) This is *the* design wart of chararray, IMHO, and one that's difficult to fix without breaking compatibility. It might be a worthwhile experiment to remove (2) and see how much we really break, but it would be impossible to know for sure. Now to address the concerns iterated in this thread. Unfortunately, I don't know where this thread began before it landed on the Numpy list, so I may be missing details which would help me address them.

...

0) "it gets very little use" (an assumption you presumably dispute);

Certainly not true from where I stand.

...

1) "is pretty much undocumented" (less true than a week ago, but still true for several of the attributes, with another handful or so falling into the category of "poorly documented");

I don't quite understand this one -- 99% of the methods are wrappers around standard Python string methods. I don't think we should redocument those. I agree it needs a better top level docstring about its purpose (see functionalities (1) and (2) above) and its status (for backward compatibility).

...

2) "probably more buggy than most other parts of NumPy" ("probably" being a euphemism, IMO);

Trac has these bugs. Any others? http://projects.scipy.org/numpy/ticket/1199 http://projects.scipy.org/numpy/ticket/1200 http://projects.scipy.org/numpy/ticket/856 http://projects.scipy.org/numpy/ticket/855 http://projects.scipy.org/numpy/ticket/1231

...

3) "there is not a really good use-case for it" (a conjecture, but one that has yet to be challenged by counter-example);

See above.

...

4) it's not the first time its presence in NumPy has been questioned ("as Stefan pointed out when asking this same question last year")

Hopefully we're addressing that now.

...

5) NumPy already has a (perhaps superior) alternative ("object arrays would do nicely if one needs this functionality");

No -- that gives the problem of even slower Python-looping to do vectorized string operations.

...

to which I'll add:

6) it is, on its face, "counter to the spirit" of NumPy.

I don't quite know what this means -- but I do find the fact that it's a view class with methods a little bit clumsy. Is that what you meant? So here's my TODO list related to all this: 1) Fix bugs in Trac 2) Improve documentation (though probably not in a method-by-method way) 3) Improve unit test coverage 4a) Create C-based vectorized string operations 4b) Refactor chararray in terms of those 4c) Design and create an interface to those methods that will be the "right way" going forward Anything else? Mike -- Michael Droettboom Science Software Branch Operations and Engineering Division Space Telescope Science Institute Operated by AURA for NASA

David Goldsmith

23 Sep 23 Sep

2:52 a.m.

New subject: [SciPy-dev] Deprecate chararray [was Plea for help]

Michael: First, thank you very much for your detailed and thorough analysis and recap of the situation - it sounds to me like chararray is now in good hands! :-) On Tue, Sep 22, 2009 at 10:58 AM, Michael Droettboom wrote:

...

Sorry to resurrect a long-dead thread, but I've been continuing Chris

IMO, no apology necessary!

...

Hanley's investigation of chararray at Space Telescope Science Institute (and the broader astronomical community) for a while and have some findings to report back.

What I've taken from this thread is that chararray is in need of a maintainer. I am able to spend some time to the cause, but first would

Yes, thank you!

...

like to clarify what it will take to make it's continued inclusion more comfortable.

Let me start with the use case. chararrays are extensively returned from pyfits (a tool to handle the standard astronomy data format). pyfits is the basis of many applications, and it would be impossible to audit all of that code. Most authors of those tools do not track numpy-discussion closely, which is why we don't hear from them on this list, but there is a great deal of pyfits-using code.

Doing some spot-checking on this code, a common thing I see is SQL-like queries on recarrays of objects. For instance, it is very common to a have a table of objects, with a "Target" column which is a string, and do something like (where c is a chararray of the 'Target' column):

subset = array[np.where(c.startswith('NGC'))]

Strictly speaking, this is a use case for "vectorized string operations", not necessarily for the chararray class as it presently stands. One could almost as easily do:

subset = array[np.where([x.startswith('NGC') for x in c])]

...and the latter is even slightly faster, since chararray currently loops in Python anyway.

Even better, though, I have some experimental code to perform the loop in C, and I get 5x speed up on a table with ~120,000 rows. If that were to be included in numpy, that's a strong argument against recommending list comprehensions in user code. The use case suggests the continued existence of vectorized string operations in numpy -- whether that continues to be chararray, or some newer/better interface + chararray for backward compatibility, is an open question. Personally I think a less object-oriented approach and just having a namespace full of vectorized string functions might be cleaner than the current situation of needing to create a view class around an ndarray. I'm suggesting something like the following, using the same example, where {STR} is some namespace we would fill with vectorized string operations:

subset = array[np.where(np.{STR}.startswith(c, 'NGC'))]

Now on to chararray as it now stands. I view chararray as really two separable pieces of functionality:

1) Convenience to perform vectorized string operations using '.method' syntax, or in some cases infix operators (+, *) 2) Implicit "rstrip"ping of values

(Note that raw ndarray's truncate values at the first NULL character, like C strings, but chararrays will strip any and all whitespace characters from the end).

Changing (2) just seems to be asking to be the source of subtle bugs. Unfortunately, there's an inconsistency between 1) and 2) in the present implementation. For example:

In [9]: a = np.char.array(['a '])

In [10]: a Out[10]: chararray(['a'], dtype='|S3')

In [11]: a[0] == 'a' Out[11]: True

In [12]: a.endswith('a') Out[12]: array([False], dtype=bool)

This is *the* design wart of chararray, IMHO, and one that's difficult to fix without breaking compatibility. It might be a worthwhile experiment to remove (2) and see how much we really break, but it would be impossible to know for sure.

Now to address the concerns iterated in this thread. Unfortunately, I don't know where this thread began before it landed on the Numpy list, so I may be missing details which would help me address them.

...
0) "it gets very little use" (an assumption you presumably dispute);

Certainly not true from where I stand.

I'm convinced.

...

...
1) "is pretty much undocumented" (less true than a week ago, but still true for several of the attributes, with another handful or so falling into the category of "poorly documented");

I don't quite understand this one -- 99% of the methods are wrappers

around standard Python string methods. I don't think we should

...

redocument those. I agree it needs a better top level docstring about

OK, that's what I needed to hear (that I don't believe anyone stated explicitly before - I'm sure I'll be corrected if I'm wrong): in that case, finishing these off is as simple as stating that in the functions' docstrings (albeit in a way compliant w/ the numpy docstring standard, of course; see below). <snip>

...

...
6) it is, on its face, "counter to the spirit" of NumPy.

I don't quite know what this means -- but I do find the fact that it's a view class with methods a little bit clumsy. Is that what you meant?

The rest of the arguments effectively become moot, but I will clarify what I meant by 6), which was simply that as I understood - and understand - it, the central purpose of numpy is to provide a fast (i.e., implemented in C), Python API for a _numerical_ multidimensional array object; it sounds like there is a need for a fast Python API for vectorized string operations, but IMO, numpy is not the place for it (maybe a sub-package in scipy? it could still use numpy "under the hood," of course); that said, my primary concern presently is getting everything that _is_ presently in numpy documented, and now, so it shall be.

...

So here's my TODO list related to all this:

1) Fix bugs in Trac 2) Improve documentation (though probably not in a method-by-method way)

So, you're volunteering to do this? Great, thanks! (Please be sure, of course, to conform to the numpy docstring standard: http://projects.scipy.org/numpy/wiki/CodingStyleGuidelines#docstring-standar... with clarification of referral practice, such as it is, at: http://docs.scipy.org/numpy/Questions+Answers/#documenting-equivalent-functi... )

...

3) Improve unit test coverage 4a) Create C-based vectorized string operations 4b) Refactor chararray in terms of those 4c) Design and create an interface to those methods that will be the "right way" going forward

Anything else?

Looks great to me! With much thanks again!!! DG

...

Mike

-- Michael Droettboom Science Software Branch Operations and Engineering Division Space Telescope Science Institute Operated by AURA for NASA

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

Ralf Gommers

4:32 a.m.

New subject: [SciPy-dev] Deprecate chararray [was Plea for help]

On Tue, Sep 22, 2009 at 1:58 PM, Michael Droettboom wrote:

...

Sorry to resurrect a long-dead thread, but I've been continuing Chris Hanley's investigation of chararray at Space Telescope Science Institute (and the broader astronomical community) for a while and have some findings to report back.

Thank you for the thorough investigation, it seems clear to me now that chararray does have a purpose and is in good hands.

...

Now to address the concerns iterated in this thread. Unfortunately, I don't know where this thread began before it landed on the Numpy list, so I may be missing details which would help me address them.

The discussion began on the scipy list, but I think you addressed most concerns in enough detail.

...

...
0) "it gets very little use" (an assumption you presumably dispute);

...
1) "is pretty much undocumented" (less true than a week ago, but still

Certainly not true from where I stand. true for several of the attributes, with another handful or so falling into the category of "poorly documented");

...
I don't quite understand this one -- 99% of the methods are wrappers around standard Python string methods. I don't think we should redocument those. I agree it needs a better top level docstring about its purpose (see functionalities (1) and (2) above) and its status (for backward compatibility).

Well, then the docstrings should say that. It can be a 4-line standard template that refers to the stdlib docs, but it would be nice to get something other than this in ipython:

...

...
...
charar = np.chararray(2) charar.ljust? <stripped> Docstring: <no docstring>

...

...
2) "probably more buggy than most other parts of NumPy" ("probably" being a euphemism, IMO);

Trac has these bugs. Any others?

http://projects.scipy.org/numpy/ticket/1199 http://projects.scipy.org/numpy/ticket/1200 http://projects.scipy.org/numpy/ticket/856 http://projects.scipy.org/numpy/ticket/855 http://projects.scipy.org/numpy/ticket/1231

This one: http://article.gmane.org/gmane.comp.python.numeric.general/23638/match=chara... Cheers, Ralf

David Goldsmith

8:21 a.m.

New subject: [SciPy-dev] Deprecate chararray [was Plea for help]

On Tue, Sep 22, 2009 at 4:02 PM, Ralf Gommers wrote:

...

On Tue, Sep 22, 2009 at 1:58 PM, Michael Droettboom wrote:

Trac has these bugs. Any others?

...
http://projects.scipy.org/numpy/ticket/1199 http://projects.scipy.org/numpy/ticket/1200 http://projects.scipy.org/numpy/ticket/856 http://projects.scipy.org/numpy/ticket/855 http://projects.scipy.org/numpy/ticket/1231

This one: http://article.gmane.org/gmane.comp.python.numeric.general/23638/match=chara...

Cheers, Ralf

That last one never got "promoted" to a ticket? DG

Michael Droettboom

25 Sep 25 Sep

6:37 p.m.

New subject: [SciPy-dev] Deprecate chararray [was Plea for help]

David Goldsmith wrote:

...

On Tue, Sep 22, 2009 at 4:02 PM, Ralf Gommers mailto:ralf.gommers@googlemail.com> wrote:

On Tue, Sep 22, 2009 at 1:58 PM, Michael Droettboom mailto:mdroe@stsci.edu> wrote:

Trac has these bugs. Any others?

http://projects.scipy.org/numpy/ticket/1199 http://projects.scipy.org/numpy/ticket/1200 http://projects.scipy.org/numpy/ticket/856 http://projects.scipy.org/numpy/ticket/855 http://projects.scipy.org/numpy/ticket/1231

This one: http://article.gmane.org/gmane.comp.python.numeric.general/23638/match=chara...

Cheers, Ralf

That last one never got "promoted" to a ticket? It's a symptom of this bug, that I created and produced a patch for yesterday:

http://projects.scipy.org/numpy/ticket/1235 Mike -- Michael Droettboom Science Software Branch Operations and Engineering Division Space Telescope Science Institute Operated by AURA for NASA

David Goldsmith

9:53 p.m.

New subject: [SciPy-dev] Deprecate chararray [was Plea for help]

Great, thanks! DG On Fri, Sep 25, 2009 at 6:07 AM, Michael Droettboom wrote:

...

David Goldsmith wrote:

...
On Tue, Sep 22, 2009 at 4:02 PM, Ralf Gommers mailto:ralf.gommers@googlemail.com> wrote:

On Tue, Sep 22, 2009 at 1:58 PM, Michael Droettboom mailto:mdroe@stsci.edu> wrote:

Trac has these bugs. Any others?

http://projects.scipy.org/numpy/ticket/1199 http://projects.scipy.org/numpy/ticket/1200 http://projects.scipy.org/numpy/ticket/856 http://projects.scipy.org/numpy/ticket/855 http://projects.scipy.org/numpy/ticket/1231

This one:

http://article.gmane.org/gmane.comp.python.numeric.general/23638/match=chara...

...
Cheers, Ralf

That last one never got "promoted" to a ticket?

It's a symptom of this bug, that I created and produced a patch for yesterday:

http://projects.scipy.org/numpy/ticket/1235

Mike

-- Michael Droettboom Science Software Branch Operations and Engineering Division Space Telescope Science Institute Operated by AURA for NASA

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

Michael Droettboom

29 Sep 29 Sep

11:25 p.m.

New subject: [SciPy-dev] Deprecate chararray [was Plea for help]

I now have a rather large patch ready which addresses the following issues with chararrays. Would it be possible to get SVN commit priviledges, or would you prefer a patch file? 1) Fix bugs in Trac http://projects.scipy.org/numpy/ticket/1199 (chararray.expandtabs broken) http://projects.scipy.org/numpy/ticket/856 (chararray __mod__ error) http://projects.scipy.org/numpy/ticket/855 (chararray __mul__ error) http://projects.scipy.org/numpy/ticket/1231 (chararray methods ignore all arguments following the first argument that evaluates to False) http://projects.scipy.org/numpy/ticket/1235 (Coercing object arrays to string arrays has surprising behaviour) http://projects.scipy.org/numpy/ticket/1240 (Casting from Unicode to String array ignores exception) http://projects.scipy.org/numpy/ticket/1241 (Array constructed with mixture of str and unicode objects fails length detection) I can provide small individual patches for some of these if necessary, but some are interrelated and can only be fixed by the "whole enchilada". 2) Improve documentation Every method now has a docstring, and a new page of routines has been added to the Sphinx tree. 3) Improve unit test coverage Full line-by-line coverage of defchararray.py, as well as lots of hairy Unicode side cases. 4a) Create C-based vectorized string operations This is benchmarking about 5x faster than the old Python-based looping on a large database of around 20k astronomical objects 4b) Refactor chararray class in terms of those 4c) Design and create an interface to those methods that will be the "right way" going forward All vectorized string operations are now available as regular functions in the numpy.char namespace. Usage of the chararray view class is only recommended for numarray backward compatibility. A few side notes: http://projects.scipy.org/numpy/ticket/1200 (chararray.rstrip inconsistency) This bug I believe should be marked as "won't fix". The inconsistent handling of trailing whitespace inconsistency is an unfortunate "feature" of the chararray class, and I am wary that fixing it may break backward compatibility. However, the new free functions in numpy.char do not have this inconsistency, so they should be recommended for new code. http://projects.scipy.org/numpy/ticket/1240 (Casting from Unicode to String array ignores exception) This bug probably needs review by someone deeply familiar with the low-level internals, as it affects more than just string and unicode arrays. It doesn't break any of the unit tests, for what it's worth ;) Cheers, Mike David Goldsmith wrote:

...

Great, thanks!

DG

On Fri, Sep 25, 2009 at 6:07 AM, Michael Droettboom mailto:mdroe@stsci.edu> wrote:

David Goldsmith wrote: > On Tue, Sep 22, 2009 at 4:02 PM, Ralf Gommers > mailto:ralf.gommers@googlemail.com mailto:ralf.gommers@googlemail.com>> wrote: > > > On Tue, Sep 22, 2009 at 1:58 PM, Michael Droettboom > mailto:mdroe@stsci.edu mailto:mdroe@stsci.edu>> wrote: > > Trac has these bugs. Any others? > > http://projects.scipy.org/numpy/ticket/1199 > http://projects.scipy.org/numpy/ticket/1200 > http://projects.scipy.org/numpy/ticket/856 > http://projects.scipy.org/numpy/ticket/855 > http://projects.scipy.org/numpy/ticket/1231 > > > This one: > http://article.gmane.org/gmane.comp.python.numeric.general/23638/match=chara... > > Cheers, > Ralf > > > That last one never got "promoted" to a ticket? It's a symptom of this bug, that I created and produced a patch for yesterday:

http://projects.scipy.org/numpy/ticket/1235

Mike

-- Michael Droettboom Science Software Branch Operations and Engineering Division Space Telescope Science Institute Operated by AURA for NASA

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org mailto:NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

------------------------------------------------------------------------

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

-- Michael Droettboom Science Software Branch Operations and Engineering Division Space Telescope Science Institute Operated by AURA for NASA

Charles R Harris

11:49 p.m.

New subject: [SciPy-dev] Deprecate chararray [was Plea for help]

On Tue, Sep 29, 2009 at 11:55 AM, Michael Droettboom wrote:

...

I now have a rather large patch ready which addresses the following issues with chararrays. Would it be possible to get SVN commit priviledges, or would you prefer a patch file?

If you are going to maintain this part of numpy, I think you should go for the privileges. <snip> Chuck

David Goldsmith

30 Sep 30 Sep

12:25 a.m.

New subject: [SciPy-dev] Deprecate chararray [was Plea for help]

Michael: Thanks so much, this is genuinely awesome! Don't forget to email Joe Harrington for your T-shirt - you more than deserve it! ;-) A few specific comments below On Tue, Sep 29, 2009 at 10:55 AM, Michael Droettboom wrote:

...

I now have a rather large patch ready which addresses the following issues with chararrays. Would it be possible to get SVN commit priviledges, or would you prefer a patch file?

I hope you're granted privileges (but I can't do that - don't even have 'em myself); I'm pretty sure everyone on scipy-dev monitors this list as well, but if you have to wait more than a few days for an affirmative reply one way or the other, you should try posting there.

...

2) Improve documentation

Every method now has a docstring, and a new page of routines has been added to the Sphinx tree.

Awesome, thanks!

...

4c) Design and create an interface to those methods that will be the "right way" going forward

All vectorized string operations are now available as regular functions in the numpy.char namespace. Usage of the chararray view class is only recommended for numarray backward compatibility.

Again, thanks for this. But I wonder aloud (asking long-time developers): do we have any systematic ways to steer people in such directions? (This should probably be a new thread...)

...

A few side notes:

http://projects.scipy.org/numpy/ticket/1200 (chararray.rstrip inconsistency)

This bug I believe should be marked as "won't fix". The inconsistent handling of trailing whitespace inconsistency is an unfortunate "feature" of the chararray class, and I am wary that fixing it may break backward compatibility. However, the new free functions in numpy.char do not have this inconsistency, so they should be recommended for new code.

OK, sounds "acceptable." And three more times: thank you, thank you, thank you!!! DG

...

Cheers, Mike

David Goldsmith wrote:

...
Great, thanks!

DG

On Fri, Sep 25, 2009 at 6:07 AM, Michael Droettboom mailto:mdroe@stsci.edu> wrote:

David Goldsmith wrote: > On Tue, Sep 22, 2009 at 4:02 PM, Ralf Gommers > mailto:ralf.gommers@googlemail.com mailto:ralf.gommers@googlemail.com>> wrote: > > > On Tue, Sep 22, 2009 at 1:58 PM, Michael Droettboom > mailto:mdroe@stsci.edu mailto:mdroe@stsci.edu>> wrote: > > Trac has these bugs. Any others? > > http://projects.scipy.org/numpy/ticket/1199 > http://projects.scipy.org/numpy/ticket/1200 > http://projects.scipy.org/numpy/ticket/856 > http://projects.scipy.org/numpy/ticket/855 > http://projects.scipy.org/numpy/ticket/1231 > > > This one: >

http://article.gmane.org/gmane.comp.python.numeric.general/23638/match=chara...

...
> > Cheers, > Ralf > > > That last one never got "promoted" to a ticket? It's a symptom of this bug, that I created and produced a patch for yesterday:

http://projects.scipy.org/numpy/ticket/1235

Mike

-- Michael Droettboom Science Software Branch Operations and Engineering Division Space Telescope Science Institute Operated by AURA for NASA

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org mailto:NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

------------------------------------------------------------------------

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

-- Michael Droettboom Science Software Branch Operations and Engineering Division Space Telescope Science Institute Operated by AURA for NASA

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

David Goldsmith

1:18 a.m.

New subject: [SciPy-dev] Deprecate chararray [was Plea for help]

On Tue, Sep 29, 2009 at 10:55 AM, Michael Droettboom wrote:

...

2) Improve documentation

Every method now has a docstring, and a new page of routines has been added to the Sphinx tree.

Um, where did you do this, 'cause it's not showing up in the doc wiki. DG

Michael Droettboom

6:33 p.m.

New subject: [SciPy-dev] Deprecate chararray [was Plea for help]

In the source in my working copy. Is that going to cause problems? I wasn't sure if it was possible to document methods that didn't yet exist in the code in the wiki. Mike David Goldsmith wrote:

...

On Tue, Sep 29, 2009 at 10:55 AM, Michael Droettboom mailto:mdroe@stsci.edu> wrote:

2) Improve documentation

Every method now has a docstring, and a new page of routines has been added to the Sphinx tree.

Um, where did you do this, 'cause it's not showing up in the doc wiki.

DG ------------------------------------------------------------------------

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

-- Michael Droettboom Science Software Branch Operations and Engineering Division Space Telescope Science Institute Operated by AURA for NASA

Ralf Gommers

6:56 p.m.

New subject: [SciPy-dev] Deprecate chararray [was Plea for help]

On Wed, Sep 30, 2009 at 9:03 AM, Michael Droettboom wrote:

...

In the source in my working copy. Is that going to cause problems? I wasn't sure if it was possible to document methods that didn't yet exist in the code in the wiki.

That is fine. New functions will automatically show up in the wiki. It would be helpful though if you could mark them ready for review in the wiki (if they are) after they show up. Could take up to 24 hours for svn changes to propagate.

Only if you moved functions around it would be useful if you pinged Pauli after you committed them. This is a temporary problem, right now the wiki creates a new page for a moved object, and the old content (if any) has to be copied over to the new page. Cheers, Ralf Mike

...

David Goldsmith wrote:

...
On Tue, Sep 29, 2009 at 10:55 AM, Michael Droettboom mailto:mdroe@stsci.edu> wrote:

2) Improve documentation

Every method now has a docstring, and a new page of routines has been added to the Sphinx tree.

Um, where did you do this, 'cause it's not showing up in the doc wiki.

DG ------------------------------------------------------------------------

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

-- Michael Droettboom Science Software Branch Operations and Engineering Division Space Telescope Science Institute Operated by AURA for NASA

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

Michael Droettboom

7:50 p.m.

New subject: [SciPy-dev] Deprecate chararray [was Plea for help]

Ralf Gommers wrote:

...

On Wed, Sep 30, 2009 at 9:03 AM, Michael Droettboom mailto:mdroe@stsci.edu> wrote:

In the source in my working copy. Is that going to cause problems? I wasn't sure if it was possible to document methods that didn't yet exist in the code in the wiki.

That is fine. New functions will automatically show up in the wiki. It would be helpful though if you could mark them ready for review in the wiki (if they are) after they show up. Could take up to 24 hours for svn changes to propagate.

Thanks. Will do.

...

Only if you moved functions around it would be useful if you pinged Pauli after you committed them. This is a temporary problem, right now the wiki creates a new page for a moved object, and the old content (if any) has to be copied over to the new page.

All of the functions that were moved were previously without docstrings in SVN, though some had docstrings (that I just now discovered) in the wiki. This may cause some hiccups, I suppose, so I'll be sure to announce when these things get committed to SVN so I know how to help straighten these things out. Mike

...

Cheers, Ralf

Mike

David Goldsmith wrote: > On Tue, Sep 29, 2009 at 10:55 AM, Michael Droettboom mailto:mdroe@stsci.edu > mailto:mdroe@stsci.edu>> wrote: > > 2) Improve documentation > > Every method now has a docstring, and a new page of routines has been > added to the Sphinx tree. > > > Um, where did you do this, 'cause it's not showing up in the doc wiki. > > DG > ------------------------------------------------------------------------ > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org mailto:NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion >

-- Michael Droettboom Science Software Branch Operations and Engineering Division Space Telescope Science Institute Operated by AURA for NASA

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org mailto:NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

------------------------------------------------------------------------

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

-- Michael Droettboom Science Software Branch Operations and Engineering Division Space Telescope Science Institute Operated by AURA for NASA

David Goldsmith

1 Oct 1 Oct

2:07 a.m.

New subject: [SciPy-dev] Deprecate chararray [was Plea for help]

So, Ralf (or anyone), how, if at all, should we modify the status of the existing chararray objects/methods in the wiki? Assuming you have no problem sharing them with me, Michael, I could add those docstrings you created for the existing methods, and we can promote them to "Ready for Review"; they can be "demoted" to "Unimportant" (by someone with the appropriate privileges); or I can promote them to "Being written" and add a comment that the docstring has been written, it just resides elsewhere for the time being (however, IMO they should not be left in "Needs editing" status). DG On Wed, Sep 30, 2009 at 7:20 AM, Michael Droettboom wrote:

...

Ralf Gommers wrote:

...
On Wed, Sep 30, 2009 at 9:03 AM, Michael Droettboom mailto:mdroe@stsci.edu> wrote:

In the source in my working copy. Is that going to cause problems?

I

...
wasn't sure if it was possible to document methods that didn't yet exist in the code in the wiki.

That is fine. New functions will automatically show up in the wiki. It would be helpful though if you could mark them ready for review in the wiki (if they are) after they show up. Could take up to 24 hours for svn changes to propagate.

Thanks. Will do.

...
Only if you moved functions around it would be useful if you pinged Pauli after you committed them. This is a temporary problem, right now the wiki creates a new page for a moved object, and the old content (if any) has to be copied over to the new page.

All of the functions that were moved were previously without docstrings in SVN, though some had docstrings (that I just now discovered) in the wiki. This may cause some hiccups, I suppose, so I'll be sure to announce when these things get committed to SVN so I know how to help straighten these things out.

Mike

...
Cheers, Ralf

Mike

David Goldsmith wrote: > On Tue, Sep 29, 2009 at 10:55 AM, Michael Droettboom mailto:mdroe@stsci.edu > mailto:mdroe@stsci.edu>> wrote: > > 2) Improve documentation > > Every method now has a docstring, and a new page of routines has been > added to the Sphinx tree. > > > Um, where did you do this, 'cause it's not showing up in the doc wiki. > > DG >

------------------------------------------------------------------------

...
> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org mailto:NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion >

-- Michael Droettboom Science Software Branch Operations and Engineering Division Space Telescope Science Institute Operated by AURA for NASA

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org mailto:NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

------------------------------------------------------------------------

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

-- Michael Droettboom Science Software Branch Operations and Engineering Division Space Telescope Science Institute Operated by AURA for NASA

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

Ralf Gommers

2:39 a.m.

New subject: [SciPy-dev] Deprecate chararray [was Plea for help]

On Wed, Sep 30, 2009 at 4:37 PM, David Goldsmith wrote:

...

So, Ralf (or anyone), how, if at all, should we modify the status of the existing chararray objects/methods in the wiki?

Nothing has to be done until *after* Mike has committed his changes to svn. Please see my previous email for what has to happen at that point. Since Mike wrote the new docstrings it would be best if he updated the status of the wiki pages then.

...

Assuming you have no problem sharing them with me, Michael, I could add those docstrings you created for the existing methods,

They will show up in the wiki when they get committed to svn (presumably within a few days), so this is needless effort for the most part. If there are different changes in the wiki and svn, that will show up in the "merge" page. The ony thing that requires manual effort is if there are changes in the wiki and the object got moved in svn. Cheers, Ralf

...

DG

On Wed, Sep 30, 2009 at 7:20 AM, Michael Droettboom wrote:

...
Ralf Gommers wrote:

...
On Wed, Sep 30, 2009 at 9:03 AM, Michael Droettboom mailto:mdroe@stsci.edu> wrote:

In the source in my working copy. Is that going to cause problems?

I

...
wasn't sure if it was possible to document methods that didn't yet exist in the code in the wiki.

That is fine. New functions will automatically show up in the wiki. It would be helpful though if you could mark them ready for review in the wiki (if they are) after they show up. Could take up to 24 hours for svn changes to propagate.

Thanks. Will do.

...
Only if you moved functions around it would be useful if you pinged Pauli after you committed them. This is a temporary problem, right now the wiki creates a new page for a moved object, and the old content (if any) has to be copied over to the new page.

All of the functions that were moved were previously without docstrings in SVN, though some had docstrings (that I just now discovered) in the wiki. This may cause some hiccups, I suppose, so I'll be sure to announce when these things get committed to SVN so I know how to help straighten these things out.

Mike

...
Cheers, Ralf

Mike

David Goldsmith wrote: > On Tue, Sep 29, 2009 at 10:55 AM, Michael Droettboom mailto:mdroe@stsci.edu > mailto:mdroe@stsci.edu>> wrote: > > 2) Improve documentation > > Every method now has a docstring, and a new page of routines has been > added to the Sphinx tree. > > > Um, where did you do this, 'cause it's not showing up in the doc wiki. > > DG >

------------------------------------------------------------------------

...
> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org mailto:NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion >

-- Michael Droettboom Science Software Branch Operations and Engineering Division Space Telescope Science Institute Operated by AURA for NASA

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org mailto:NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

------------------------------------------------------------------------

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

-- Michael Droettboom Science Software Branch Operations and Engineering Division Space Telescope Science Institute Operated by AURA for NASA

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

David Goldsmith

3:03 a.m.

New subject: [SciPy-dev] Deprecate chararray [was Plea for help]

On Wed, Sep 30, 2009 at 2:09 PM, Ralf Gommers wrote:

...

On Wed, Sep 30, 2009 at 4:37 PM, David Goldsmith wrote:

...
So, Ralf (or anyone), how, if at all, should we modify the status of the existing chararray objects/methods in the wiki?

Nothing has to be done until *after* Mike has committed his changes to svn. Please see my previous email for what has to happen at that point. Since Mike wrote the new docstrings it would be best if he updated the status of the wiki pages then.

OK; Mike: hopefully it will be clear what you have to do to update the status (it's pretty trivial) but of course don't hesitate to email (you can do so off-list if you prefer) w/ any questions; unfortunately, AFAIK, there's no way to update the status of many docstrings all at once - you'll have to do them each individually (if you like, let me know when you've committed them and I can help - it sounds like there will be a lot); the main "silly" thing to remember is that the option to change the "Review status" only appears if you're logged in. :-)

...

Assuming you have no problem sharing them with me, Michael, I could add

...
those docstrings you created for the existing methods,

They will show up in the wiki when they get committed to svn (presumably within a few days), so this is needless effort for the most part. If there are different changes in the wiki and svn, that will show up in the "merge" page.

The ony thing that requires manual effort is if there are changes in the wiki and the object got moved in svn.

And, as above, updating the status in the Wiki. :-) DG

...

Cheers, Ralf

...
DG

On Wed, Sep 30, 2009 at 7:20 AM, Michael Droettboom wrote:

...
Ralf Gommers wrote:

...
On Wed, Sep 30, 2009 at 9:03 AM, Michael Droettboom mailto:mdroe@stsci.edu> wrote:

In the source in my working copy. Is that going to cause problems?

I

...
wasn't sure if it was possible to document methods that didn't yet exist in the code in the wiki.

That is fine. New functions will automatically show up in the wiki. It would be helpful though if you could mark them ready for review in the wiki (if they are) after they show up. Could take up to 24 hours for svn changes to propagate.

Thanks. Will do.

...
Only if you moved functions around it would be useful if you pinged Pauli after you committed them. This is a temporary problem, right now the wiki creates a new page for a moved object, and the old content (if any) has to be copied over to the new page.

All of the functions that were moved were previously without docstrings in SVN, though some had docstrings (that I just now discovered) in the wiki. This may cause some hiccups, I suppose, so I'll be sure to announce when these things get committed to SVN so I know how to help straighten these things out.

Mike

...
Cheers, Ralf

Mike

David Goldsmith wrote: > On Tue, Sep 29, 2009 at 10:55 AM, Michael Droettboom mailto:mdroe@stsci.edu > mailto:mdroe@stsci.edu>> wrote: > > 2) Improve documentation > > Every method now has a docstring, and a new page of routines has been > added to the Sphinx tree. > > > Um, where did you do this, 'cause it's not showing up in the doc wiki. > > DG >

------------------------------------------------------------------------

...
> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org mailto:NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion >

-- Michael Droettboom Science Software Branch Operations and Engineering Division Space Telescope Science Institute Operated by AURA for NASA

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org mailto:NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

------------------------------------------------------------------------

...
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

-- Michael Droettboom Science Software Branch Operations and Engineering Division Space Telescope Science Institute Operated by AURA for NASA

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

Michael Droettboom

9:56 p.m.

New subject: [SciPy-dev] Deprecate chararray [was Plea for help]

Thanks. There's a bit of a snag getting my SVN permissions, and the doc editor permissions I assume are pending. Once those things are in place, I can move forward on this and hopefully it will be clear what needs to be done. Mike David Goldsmith wrote:

...

On Wed, Sep 30, 2009 at 2:09 PM, Ralf Gommers mailto:ralf.gommers@googlemail.com> wrote:

On Wed, Sep 30, 2009 at 4:37 PM, David Goldsmith mailto:d.l.goldsmith@gmail.com> wrote:

So, Ralf (or anyone), how, if at all, should we modify the status of the existing chararray objects/methods in the wiki?

Nothing has to be done until *after* Mike has committed his changes to svn. Please see my previous email for what has to happen at that point. Since Mike wrote the new docstrings it would be best if he updated the status of the wiki pages then.

OK; Mike: hopefully it will be clear what you have to do to update the status (it's pretty trivial) but of course don't hesitate to email (you can do so off-list if you prefer) w/ any questions; unfortunately, AFAIK, there's no way to update the status of many docstrings all at once - you'll have to do them each individually (if you like, let me know when you've committed them and I can help - it sounds like there will be a lot); the main "silly" thing to remember is that the option to change the "Review status" only appears if you're logged in. :-)

Assuming you have no problem sharing them with me, Michael, I could add those docstrings you created for the existing methods,

They will show up in the wiki when they get committed to svn (presumably within a few days), so this is needless effort for the most part. If there are different changes in the wiki and svn, that will show up in the "merge" page.

The ony thing that requires manual effort is if there are changes in the wiki and the object got moved in svn.

And, as above, updating the status in the Wiki. :-)

DG

Cheers, Ralf

DG

On Wed, Sep 30, 2009 at 7:20 AM, Michael Droettboom mailto:mdroe@stsci.edu> wrote:

Ralf Gommers wrote: > > > On Wed, Sep 30, 2009 at 9:03 AM, Michael Droettboom mailto:mdroe@stsci.edu > mailto:mdroe@stsci.edu>> wrote: > > In the source in my working copy. Is that going to cause problems? I > wasn't sure if it was possible to document methods that didn't yet > exist > in the code in the wiki. > > That is fine. New functions will automatically show up in the wiki. It > would be helpful though if you could mark them ready for review in the > wiki (if they are) after they show up. Could take up to 24 hours for > svn changes to propagate. Thanks. Will do. > > Only if you moved functions around it would be useful if you pinged > Pauli after you committed them. This is a temporary problem, right now > the wiki creates a new page for a moved object, and the old content > (if any) has to be copied over to the new page. All of the functions that were moved were previously without docstrings in SVN, though some had docstrings (that I just now discovered) in the wiki. This may cause some hiccups, I suppose, so I'll be sure to announce when these things get committed to SVN so I know how to help straighten these things out.

Mike > > Cheers, > Ralf > > > Mike > > David Goldsmith wrote: > > On Tue, Sep 29, 2009 at 10:55 AM, Michael Droettboom > mailto:mdroe@stsci.edu mailto:mdroe@stsci.edu> > > mailto:mdroe@stsci.edu mailto:mdroe@stsci.edu>>> wrote: > > > > 2) Improve documentation > > > > Every method now has a docstring, and a new page of routines > has been > > added to the Sphinx tree. > > > > > > Um, where did you do this, 'cause it's not showing up in the doc > wiki. > > > > DG > > > ------------------------------------------------------------------------ > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion@scipy.org mailto:NumPy-Discussion@scipy.org mailto:NumPy-Discussion@scipy.org> > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > -- > Michael Droettboom > Science Software Branch > Operations and Engineering Division > Space Telescope Science Institute > Operated by AURA for NASA > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org mailto:NumPy-Discussion@scipy.org mailto:NumPy-Discussion@scipy.org> > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > ------------------------------------------------------------------------ > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org mailto:NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion >

-- Michael Droettboom Science Software Branch Operations and Engineering Division Space Telescope Science Institute Operated by AURA for NASA

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org mailto:NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org mailto:NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org mailto:NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

------------------------------------------------------------------------

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

-- Michael Droettboom Science Software Branch Operations and Engineering Division Space Telescope Science Institute Operated by AURA for NASA

5320

Age (days ago)

5362

Last active (days ago)

List overview

Download

31 comments

11 participants

participants (11)

Charles R Harris
Christopher Hanley
David Cournapeau
David Goldsmith
David Goldsmith
James Turner
Michael Droettboom
Ralf Gommers
Stéfan van der Walt
Travis Oliphant
Vincent Schut

Deprecate chararray [was [SciPy-dev] Plea for help]

David Goldsmith

Christopher Hanley

Christopher Hanley

Christopher Hanley

Christopher Hanley

Vincent Schut

Vincent Schut

Michael Droettboom

David Goldsmith

David Goldsmith

Michael Droettboom

David Goldsmith

Michael Droettboom

David Goldsmith

David Goldsmith

Michael Droettboom

Michael Droettboom

David Goldsmith

David Goldsmith

Michael Droettboom

tags

participants (11)