On Fri, Jul 15, 2011 at 10:57 AM, Guido van Rossum <guido@python.org> wrote:

However, in many cases absence of the string is not an error -- you just need to do something else. So in cases where *if* it's found you need the position, and *if* it isn't found you need to do something else, you'd have to use a try/except block to catch the non-error that is absence. All in all I don't see enough reason to start deprecating find. But perhaps popular lint-like programs could flag likely abuses of find?

--Guido

It isn't necessarily an error if the substring is not in the string (though it sometimes is), but it is an exceptional case. Python uses exceptions pretty liberally most places -- it isn't necessarily an error if an iterator is exhausted or if float("4.2 bad user input") is called or if BdbQuit was raised. In these cases, an exception can be perfectly expected to indicate that what happened is different from the information used in a return value. Making a Python user write a try/except block when she wants to handle both the cases "substring is in s" and "substring isn't in s" seems perfectly fine to me and, really, preferable to the if statement required to handle these two cases. The base two cases really are about the same: try: i = s.index(sub) except IndexError: do_something() vs. i = s.find(sub) if i == -1: do_something() But what if I forgot to handle the special case? i = s.index(sub) # An exception is raised right here and I can fix my code vs. i = s.find(sub) # No exception is raised In this second case, I get the value of -1. Later I can use it as an index, use it in a slice, or perform arithmetic on it. This can introduce seemingly-unrelated values later on, making this especially hard to track down. If the failure return code was at least None it would behave more sanely, but at present the failure return code is a perfectly valid value for almost any use. If a programmer is sill averse to using try/except, we can still write if sub in s: i = s.index(sub) else: do_something() Now. we can dredge up some examples where -1 is the actual value someone wants to use. These cases are so rare and so subtle as to make their use so clever I don't really see their existence as an advantage. Additionally, it is unfortunate that we currently have two methods to do the same thing (which isn't even a super-common task) with different APIs. Nothing about the names "find" and "index" really makes clear which is which. This violates the "There should be one-- and preferably only one --obvious way to do it." principle and makes the Python user need to memorize an unnecessary, arbitrary distinction. I would also point out that it was not a contrived case that I mentioned where a beginner introduces a bug by trying "if s.find(sub):" instead of "if sub in s:"; I have really seen people try this several times. Obviously we cannot make many decisions based on new Python programmers' mistakes, it is worth recognizing them. I hope this additional discussion might be able to sway your opinion here. The only advantage to using str.find is that you do not have to use try/except blocks, but in fact you don't have to with str.index either. On the other hand, there are numerous disadvantages--practical, pedagogical, stylistic, and design--to having and using str.find. Mike

On Fri, Jul 15, 2011 at 12:31 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:

While this would be a very good argument to make if we were currently designing the str API, I don't think the benefits of suppressing str.find outweight the burden of converting old code to use a different idiom.

We could choose to write something about it in the documentation, though.

Regards

Antoine.

That could be just as suitable. To be honest, I can't really see too much of a difference between a stronger note about the problems I perceive with str.find in the documentation and a note that uses the word "deprecated". There certainly wouldn't be occasion in the foreseeable future for you actually to remove str.find. Mike

On Fri, Jul 15, 2011 at 1:06 PM, Bruce Leban <bruce@leapyear.org> wrote:

It seems to me that the solution is that you should never use find and use index instead. You could even modify pylint/etc. to flag any use of find as an error. That doesn't mean it should be deprecated so everyone else has to follow you. Deprecated now => removed in the future. While many can agree that warnings in the documentation about common mistakes in using APIs is a good thing, that doesn't translate into consensus that we should remove the API that might be misused. The documentation already says "The find() method should be used only if you need to know the position of sub. To check if sub is a substring or not, use the in operator." There is *no* general solution to the problem of people misusing APIs. Even with index someone can still write

if s.index(sub):

which probably doesn't do what they were thinking. Should we remove that too?

The case of "if s.index(sub):" as a misspelling of "if sub in s:" is not nearly as errorprone as the case of "if s.find(sub):". For one, "s.find(sub)" *sounds* like it might be true if you can find sub in s and not true if sub cannot be found. With str.index, the fact the return value is going to be an index is made clear. For two, str.find can fail silently in more cases than the str.index case can fail silently. For three, I've never see a learner write "if s.index(sub)" though I have seen several mistakenly write the "if s.find(sub)" version.

Note that one reason people might prefer find over index is that exceptions constrain how you write the code:

try:     i = s.index(sub)     do lots of stuff with s and i except ValueError:     result = 'not found'

In this case the try wraps way too much code so it could catch a ValueError in the middle of 'stuff'. Here's correct code:

try:     i = s.index(sub)     do_more = True except:     result = 'not found'     do_more = False if do_more:     do lots of stuff with s and i

Do you think that's better than:

i = s.find(sub) if i < 0:     result = 'not found' else:     do lots of stuff with s and i

Are you familiar with the "else" clause of a try/except suite? It already handles writing the "right" version nicely. try: i = s.index(sub) except IndexError: result = 'not found' else: do lots of stuff with s and i

--- Bruce

Mike

Am 15.07.2011 18:31, schrieb Antoine Pitrou:

On Fri, 15 Jul 2011 12:12:33 -0400 Mike Graham <mikegraham@gmail.com> wrote:

...

[...] Now. we can dredge up some examples where -1 is the actual value someone wants to use. These cases are so rare and so subtle as to make their use so clever I don't really see their existence as an advantage.

Additionally, it is unfortunate that we currently have two methods to do the same thing (which isn't even a super-common task) with different APIs. Nothing about the names "find" and "index" really makes clear which is which. This violates the "There should be one-- and preferably only one --obvious way to do it." principle and makes the Python user need to memorize an unnecessary, arbitrary distinction.

While this would be a very good argument to make if we were currently designing the str API, I don't think the benefits of suppressing str.find outweight the burden of converting old code to use a different idiom.

We could choose to write something about it in the documentation, though.

Patches are welcome at bugs.python.org ;) Georg

On Fri, Jul 15, 2011 at 12:38 PM, Guido van Rossum <guido@python.org> wrote:

On Fri, Jul 15, 2011 at 9:12 AM, Mike Graham <mikegraham@gmail.com> wrote:

...

It isn't necessarily an error if the substring is not in the string (though it sometimes is), but it is an exceptional case. Python uses exceptions pretty liberally most places -- it isn't necessarily an error if an iterator is exhausted or if float("4.2 bad user input") is called or if BdbQuit was raised. In these cases, an exception can be perfectly expected to indicate that what happened is different from the information used in a return value.

This smells a bit like uncalled-for religion. Remember that readline() returns an empty string at the end of the file instead of raising an exception, and IMO that makes it the better API.

-- --Guido van Rossum (python.org/~guido)

Thanks for this prompt, considerate reply (and the other). I really didn't mean to present an "always use exceptions" dogma here, so much as to claim "Having the programmer use a try/except block isn't a problem," and to say that the fact an exception was raised doesn't necessarily indicate that something went wrong. StopIteration is a strong example that exceptions can often be very expected and normal. Mike

Devin Jeanpierre wrote:

Strings don't even bother to be compatible with .find, if you pass in an index that was returned by .find() you might just get the wrong result. It's pretty much only ever useful in the if check, which is in essence identical to the try-except, except it adds the ability to forget to check and produce incorrect behaviour -- this is not a feature.

Hind-sight -- what a great thing! I do think None would be a better return value. I suspect the odds of that changing now are pretty slim, though. ~Ethan~

On Fri, 15 Jul 2011 20:26:42 +0200 Georg Brandl <g.brandl@gmx.net> wrote:

Sorry, I don't see the immense complication in

try: line = readline() except StopIteration: line = ''

Of course, if the tokenize API was designed from scratch nowadays, it would probably only accept iterables, and you'd have to make one yourself from a file object using

iter(fd.readline, '')

Or, since file objects are iterable out of the box, you could simply pass the file object. Regards Antoine.

On 15Jul2011 12:12, Mike Graham <mikegraham@gmail.com> wrote: | On Fri, Jul 15, 2011 at 10:57 AM, Guido van Rossum <guido@python.org> wrote: | > However, in many cases absence of the string is not an error -- you | > just need to do something else. [...] | | It isn't necessarily an error if the substring is not in the string | (though it sometimes is), but it is an exceptional case. No it isn't, IMO. It's simply the _other_ case. | Python uses | exceptions pretty liberally most places -- it isn't necessarily an | error if an iterator is exhausted or if float("4.2 bad user input") is | called or if BdbQuit was raised. In these cases, an exception can be | perfectly expected to indicate that what happened is different from | the information used in a return value. In all the cases you cite the exception indicates failure of the operation: .next() has nothing to "next" to, float is being handed garbage etc. str.find does not have a failure mode, it has string found and string not found. | Making a Python user write a try/except block when she wants to handle | both the cases "substring is in s" and "substring isn't in s" seems | perfectly fine to me and, really, preferable to the if statement | required to handle these two cases. You don't find try/except wordy and opaque? I find "if" more idiomatic most of the time. Not to mention vague: it can often be quite hard to be sure the raised exception came from just the operation you imagine it came from. With str.find there's little scope for vagueness I agree (unless you aren't really using a str, but a duck-type). But plenty of: try: x = foofunc(y) except IndexError, e: ... is subject to uncaught IndexError arbitrarily deep in foofunc's call stack. | The base two cases really are about the same: [... try ... excpt ...] | vs. | | i = s.find(sub) | if i == -1: | do_something() | | But what if I forgot to handle the special case? [...] | In this second case, I get the value of -1. Later I can use it as an | index, use it in a slice, or perform arithmetic on it. This can | introduce seemingly-unrelated values later on, making this especially | hard to track down. I agree it may be a pity that str.find doesn't return None on string not found, which would generally raise an exception on an attempt to use it as a number. Cheers, -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ Any company large enough to have a research lab is large enough not to listen to it. - Alan Kay

This is an absolute flight of fancy, and I'm sure it's already been rejected in the past (a quick search says Aug. 2009, http://mail.python.org/pipermail/python-ideas/2009-August/thread.html#5576 ), but what about some kind of try/except expression? i = s.index(substr) except ValueError is None I guess my main problem with that is the color of the bikeshed: it's hard to get a good idiomatic way of spelling the except expression. You could use a colon, as proposed in that thread, but it seems to me a colon indicates a new line follows. I would be for making this as simple as possible. No "as", no non-implicit "else", and no nesting of exception types. If you want something fancy, use a statement.

On Sat, Jul 16, 2011 at 6:47 PM, Georg Brandl <g.brandl@gmx.net> wrote:

By the way: str.index() raises ValueError, not IndexError, if the substring isn't found.  That's one argument against "the index() API is foolproof, while the find() API isn't".

Yeah, that bugs me too. Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

On Sat, Jul 16, 2011 at 7:11 PM, M.-A. Lemburg <mal@egenix.com> wrote:

Georg Brandl wrote:

...

By the way: str.index() raises ValueError, not IndexError, if the substring isn't found.  That's one argument against "the index() API is foolproof, while the find() API isn't".

That's because the *value* passed as argument to .index() isn't found. Which error class to use often depends on your view point, so in some cases it may seem natural to you, in others you have a different POV, and it feels wrong :-)

Yeah, but trying to catch IndexError instead of ValueError is a pretty easy (and understandable) mistake to make, given the correspondence in names and logical train of thought "index() tells me the index where a substring can be found, IndexError is used to denote that a given index doesn't exist, so if the requested index doesn't exist, then this function will throw IndexError". Easily detected by testing, but still unintuitive and annoying. If IndexError had instead been called IndexNotFound, then there would never have been the slightest question as to which exception should have been thrown. Too late to change it now, though. We just all have to learn that, from the str.index point of view, failing to find a substring means there's something *wrong* with one (or both) of the passed in strings rather than the less opinionated "the index you have requested doesn't actually exist". Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

Mike Graham wrote:

On Fri, Jul 15, 2011 at 6:50 PM, Cameron Simpson wrote:

...

You don't find try/except wordy and opaque? I find "if" more idiomatic most of the time.

Not really. If we don't like exceptionhandling, we're using the wrong language.

I think exception handling is awesome. I also think dictionaries are awesome, but I don't use them when what I need is a set. ~Ethan~

On 7/16/2011 2:46 PM, Terry Reedy wrote:

On 7/15/2011 6:50 PM, Cameron Simpson wrote:

...

On 15Jul2011 12:12, Mike Graham<mikegraham@gmail.com> wrote: age etc.

str.find does not have a failure mode, it has string found and string not found.

Not finding == failure to find == failure to match an implied re.

Indeed, negative values such as -1 are standard error/failure return codes for functions that normally return nonnegative ints and that written in statically typed languages without catchable exceptions. In C, for instance, EOF is defined as an implementation-defined negative int and I am sure -1 is used by some. -- Terry Jan Reedy

On 16Jul2011 22:21, Masklinn <masklinn@masklinn.net> wrote: | On 2011-07-16, at 21:52 , Terry Reedy wrote: | > On 7/16/2011 2:46 PM, Terry Reedy wrote: | >> On 7/15/2011 6:50 PM, Cameron Simpson wrote: | >>> str.find does not have a failure mode, it has string found and string | >>> not found. | >> | >> Not finding == failure to find == failure to match an implied re. | > Indeed, negative values such as -1 are standard error/failure | > return codes for functions that normally return nonnegative ints and | > that written in statically typed languages without catchable exceptions | | … or type systems worth using. And that's for those which are | 0-indexed of course, especially for finding sub-sequences. | | And interestingly, the function corresponding to `str.find` in libc | returns `NULL` in case of failure, not −1 (its return value is a | pointer to the first occurrence of the needle in the haysack). Though if you're thinking NULL is equivalent to None (which is often is conceptually), let's remember that in C NULL is just a pointer value; it is a sentinal, but not a different type. So NULL here is in some ways akin to -1 in Python's find return. You still need to test for it; not all platforms will (for example) segfault if NULL is dereferenced. Cheers, -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ There's no need to worry about death, it will not happen in your lifetime. - Raymond Smullyan

Nick Coghlan wrote:

So, I have a different suggestion (that actually paves the way for the eventual deprecation of find()):

Update str.index() to accept an optional sentinel value.

If the sentinel argument is not supplied, then a missing substring raises ValueError as it does now. If it is supplied, then a missing substring returns the sentinel value instead of throwing an exception. The above idiom could then be expressed cleanly as:

idx = s.index(sub, missing=None) if idx is not None: # do something that needs idx

+1 Excellent idea.

However, this seemingly simple suggestion is complicated by the fact that string methods do not currently accept keyword arguments and index() already accepts two optional positional arguments (for substring searching).

Does this require a PEP or just someone to start working on it? Getting str() to accept keyword arguments and adding missing to str.index() sounds like a nice project. ~Ethan~

On Sat, Jul 16, 2011 at 11:46 AM, Ethan Furman <ethan@stoneleaf.us> wrote:

Raymond Hettinger wrote:

...

Unless an API is flat out broken, deprecation is almost always a bad idea. This API has been around for a very long time, so deprecating it will break lots of people's code for almost zero benefit:

http://www.google.com/codesearch#search/&q=%5C.find%5C(%20lang:%5Epython$&type=cs <http://www.google.com/codesearch#search/&q=%5C.find%5C(%20lang:%5Epython$&type=cs>

How ironic that the fist hit seems to display the problem Mike is concerned with:

       position =  min(position, len(self.contents))        if hasattr(newChild, 'parent') and newChild.parent != None:            # We're 'inserting' an element that's already one            # of this object's children.            if newChild.parent == self:                index = self.find(newChild)                if index and index < position:                    # Furthermore we're moving it further down the                    # list of this object's children. That means that                    # when we extract this element, our target index                    # will jump down one.                    position = position - 1

I haven't read all the surrounding code to know if this will ever fail, but the whole 'index = ... .find(...); if index and ...' certainly doesn't lend confidence.  After all, if you *know* newChild is in self, why not use .index()?

~Ethan~

Note that this particular code is using BeautifulSoup.PageElement.find, not str.find. There are, however, in the first few pages similar snippets which use the result of str.find without a check for the special case. Mike

On Sat, Jul 16, 2011 at 11:26 AM, Raymond Hettinger <raymond.hettinger@gmail.com> wrote:

Unless an API is flat out broken, deprecation is almost always a bad idea. This API has been around for a very long time, so deprecating it will break lots of people's code for almost zero benefit: http://www.google.com/codesearch#search/&q=%5C.find%5C(%20lang:%5Epython$&type=cs

Raymond

I agree that breaking people's code is a bad thing and have not suggested removing str.find. This removal would require a transition like that from Python 2.x to Python 3.x, a move that is not planned and I personally do not ever expect. I appreciate your linking this search, which does indeed does show that str.find is in wide use. However, looking at the first give pages of results, this use seems largely unfortunate—literally the majority of the times str.find is used, I would have used "substring in s" or "s.startswith(substring)". I also see code like "pos = s.find(" ("); if pos + len(" (...)") > self._maxWidth:" which makes me very uncomfortable and which I would have to read in detail to figure out what's happening confidently if I was the maintaining the code. Thanks for the reply, Mike

On Sat, Jul 16, 2011 at 3:07 PM, Raymond Hettinger <raymond.hettinger@gmail.com> wrote:

Deprecation is a step towards removal and it always causes a certain amount of pain.

ISTM, the time for any move like this would have been the jump from Python 2 to Python 3 where significant breakage was expected and where transition tools were developed.

Because this would break backwards compatibility unnecessarily, I think anyone involved can agree that actual removal could not take place until a special jump akin to the Python 2->Python 3 jump. (It would surprise me if such a jump ever actually took place.)

...

I appreciate your linking this search, which does indeed does show that str.find is in wide use. However, looking at the first give pages of results, this use seems largely unfortunate—literally the majority of the times str.find is used, I would have used "substring in s" or "s.startswith(substring)". I also see code like "pos = s.find("  ("); if pos + len(" (...)") > self._maxWidth:" which makes me very uncomfortable and which I would have to read in detail to figure out what's happening confidently if I was the maintaining the code.

I am largely unsympathetic to arguments that are roughly equivalent to "I don't like the way other people write programs".

Reviewing five pages of results, over 2/3 of the uses of str.find could be replaced by str.__contains__. Using sub in s instead of s.find(sub) != -1 is *already* the advice in the official Python documentation, so I do not believe I am making especially personal judgments about the style of the code.

Something akin to the str.find() API has been present in many, many languages for a very long time.  For the most part, people seem to be able to use it reasonably well.  I find that beginning Python students never seem to have a problem with it.

I'm glad your experience with learners here has been more consistent than mine.

Also keep it mind that startswith() and endswith() were relatively recent additions to Python, so it is no surprise that lots of code uses find() instead of startswith().

Raymond

Mike

I just remembered one other thought on the subject. Usually, when Python introduces a method such as .index() that raises an exception for the not-found case, there are immediate requests for variants that don't raise exceptions: dict.pop(key, default) dict.get(key, defalut) next(iterable, default) getattr(obj, attr, default) re.match() --> None or matchobject People seem to hate wrapping try/except around simple calls. Expect those people to be agitated if you take away str.find(). Raymond

On Jul 16, 2011, at 2:04 PM, Mike Graham wrote:

Did you catch Nick's suggestion to enhance str.index to have the same basic API as dict.get/getattr/etc.?

Sorry, but I thought idea that was way off base. Things like dict.get and getattr are about returning values, so it is possible to provide a meaningful default. In the case of string.find, there is no meaningful default position in the string. So, a default would simply be a way to turn the -1 value into some other object which you would still need to test. We don't need to make the API worse by expanding it just for the sake of change. The str.find() method isn't broken or useless. If you want people to change the way they write code, it would be better to do it through education (blog posts, pylint, etc) rather than by breaking a venerable API. Raymond

On Jul 16, 2011, at 3:15 PM, Devin Jeanpierre wrote:

Slicing was brought up as a use-case where you can use a default without checking.

This thread has lost contact with reality. It started with the reasonable observation that many uses of str.find could be replaced with a simple test using the in-operator. Now, the thread is venturing into the typical python-ideas world of making-up random use cases and APIs. Something like str.find() is in many languages and it is definitely not the norm for them to have found a need to both be able to return -1 or to supply a default value. As a Python teacher, speaker, and consultant, I have the opportunity to see and review the code repositories for many companies. I'm pretty sure that I've never seen a utils module with the likes of: def myfind(fullstring, substring, default=None): i = fullstring.find(substring) if i == -1 and default is not None: return default return i When I start seeing people routinely using a helper function like this, I'll start to believe that a str.find default value isn't silly. Mike's initial post was well grounded in observations about code that could be improved by using "in" or str.index() instead of str.find(). Though I disagreed with the recommendation to deprecate, it would be even worse to exacerbate the usability issues by making the method signature even more complex (with a new optional argument and a new signature variant for the return value). That isn't progress. It's aspiring cruft that makes the language harder to learn and remember. Raymond

Raymond Hettinger wrote:

This thread has lost contact with reality. It started with the reasonable observation that many uses of str.find could be replaced with a simple test using the in-operator.

Now, the thread is venturing into the typical python-ideas world of making-up random use cases and APIs. Something like str.find() is in many languages and it is definitely not the norm for them to have found a need to both be able to return -1 or to supply a default value.

Nick's proposal was to enhance str.index(), not string.find(); Having str.index() accept a value to return on failure instead of raising an exception means it could do both jobs, and would also make it much less likely to wrongly use the failure return value of -1 from str.find() which is, unfortunately, a legitimate index value. ~Ethan~

On Jul 17, 2011, at 12:15 AM, Nick Coghlan wrote:

Indeed, the problem as I see it is that our general idiom for functions and methods that raise 'Not Found' exceptions is to accept an optional parameter that specifies a value to return in the Not Found case.

There's a difference between methods that return looked-up values (where a default might make sense) versus a method that returns an index (where it usually makes no sense at all).

For historical reasons, we currently break that idiom for index() methods: instead of supplying an extra parameter to str.index, one instead switches to a completely different method (.find()) with no control over the sentinel value returned (it's always -1). For other sequences (e.g. list), there's no find equivalent, so you *have* to write the exception handling out explicitly.

My proposal is to update the signature of index() (for all sequences, including the ABC) to follow the standard 'Not Found' idiom by accepting a 'missing' parameter that is returned for those cases where ValueError would otherwise be raised.

Code that uses str.find would continue to work, but the recommended alternative would be obj.index(x, missing=None) (or appropriate default value). I would advise against any actual deprecation of str,find (cf. the deliberate lack of optparse deprecation).

It's unfortunate that backwards compatibility means we can't use the more descriptive name, but that's life.

However, I already have too much on my plate to push this forward for Python 3.3. I'm able to offer advice if someone would like to try their hand at writing a PEP, though.

If someone takes this out of python-ideas land and into a serious PEP, they should be prepared to answer a number of tough questions: * Is this actually necessary? Is there something you currently can't code? If not, then it adds API complexity without adding any new capabilities. There is a high threshold for expanding the string API -- this would affect everyone learning python, every book written, every lint tool, every class seeking to be string-like, etc. So, it would need be a substantive improvement to be accepted. * Take a look at what other languages do. Practically every general purpose language has an API for doing substring searches. Since we're not blazing new territory here, there needs to be a good precedent for this change (no shooting from the hip when the problem has already been well solved many times over). * Use Google's code search to identify examples of real world code that would better with the new API. If the only use case is creating a new slicing one-liner, that likely is too rare and arcane to warrant a change. * Consider the effects of adding a second-way-to-do-it. Will it add to the learning curve, cause debates about the best way in a given situation, add more PEP 8 entries and pylint checks? Is it worth introducing version incompatibilities (i.e. runs on 3.3 but not earlier), etc. * What should the default value be? Is there any non-numerical result that ever makes sense; otherwise, you're just making a alias for the -1 currently returned by str.find(). If the default is some value that evaluates to False, will that create a common error where an if-test fails to disambiguate the default value from a substring found at position zero. If the new API is ambiguous or confusing in *any* way, then it will be a step backwards and make Python worse rather than better. * See if you can find examples where people have already found the need to write a helper function such as: def index_default(s, sub, default): try: return s.index(sub) except ValueError: return default If you find code like that in the wild, it may be an indication that people want this. If you don't, it may indicate otherwise. * Good API design requires some thinking about function/method signatures. Would making this a keyword-only argument soive the positional arguments problem? Since str.index() already takes arguments for the "start" and "end" index, is the full signature readable without keywords: mystr.index(possible_substr, 0, -1, default_value) Also look at the signature for the return value. Currently, it always returns a number, but if it can return a number or anything else, then all client code must be prepared to handle the alternatives with clean looking code that is self-evidently correct. * Perhaps talk to some people who write python code for a living to determine if they've ever needed this or whether it would end-up as cruft. (In my case, the answer is that I've not needed or wanted this in a over a decade of heavy Python use). Hopefully, this short and incomplete list will provide a good basis for thinking about whether the proposal is a good idea. Defending a PEP is no fun at all, so put in all your deep thinking up front. Cheers, Raymond

On Jul 17, 2011, at 2:13 AM, Masklinn wrote:

On 2011-07-17, at 10:09 , Raymond Hettinger wrote:

...

On Jul 17, 2011, at 12:15 AM, Nick Coghlan wrote:

...

Indeed, the problem as I see it is that our general idiom for functions and methods that raise 'Not Found' exceptions is to accept an optional parameter that specifies a value to return in the Not Found case.

There's a difference between methods that return looked-up values (where a default might make sense) versus a method that returns an index (where it usually makes no sense at all). Why not? An index is a looked-up value here (it's just a reverse lookup) and .find is returning a (non-configurable) default value is it not?

There is a fundamental difference between content (values in the list) and the list position. There are meaningful search defaults for the former but not the latter. It's important that to grasp this distinction before going further.

...

* Take a look at what other languages do. Practically every general purpose language has an API for doing substring searches. Since we're not blazing new territory here, there needs to be a good precedent for this change (no shooting from the hip when the problem has already been well solved many times over). Other languages do everything and their reverse. You have languages returning −1 or 0 (the latter for 1-indexed languages), languages returning differently-typed sentinels, etc…

SML even returns the length of the string.

See a listing at http://en.wikipedia.org/wiki/Comparison_of_programming_languages_(string_fun...

My reading is that not a single one of these entries has a signature with a user specifiable default value of arbitrary type. IOW, there is nothing resembling proposal on the table. AFAICT, all of them are equivalent to either our current str.find() or str.index(). Raymond

On Sun, Jul 17, 2011 at 8:56 PM, Steven D'Aprano <steve@pearwood.info> wrote:

I can't really see that they do "everything and their reverse". There are two basic strategies: return an out-of-bound value, and raise an exception, both of which Python already does.

In a language that accepts negative indices, -1 is not out of bounds, and hence is an objectively bad value to return. If str.find() returned None for missing substrings, we wouldn't be having this discussion. However, backwards compatibility requirements mean that option is not available to us. Is this a language wart where the correct answer is to tell everyone new to the language that complains about the current behaviour to "suck it up and learn to live with it"? Maybe. But claiming that str.find() is a *good* API in a language that accepts negative indices would be flat out wrong. Good *enough*, maybe, but not good in an absolute sense. As I see it, there are a few options. 1. Do nothing. Quite a reasonable option. str.find() is flawed and str.index() can be annoying to use, but fixing this may not be worth the hassle. However, this is not the same as claiming the str.find() behaviour is a good API - it's just acknowledging the wart and deciding not to address it. 2. Add an alternate behaviour to str.index() that allows the exception to optionally be converted into a sentinel value. Inspired by getattr, getitem, dict.get, next, etc. Made messy by the need for the sentinel to go after the existing arguments and the fact that the number of sane sentinel values is extremely limited (aside from None, the only remotely useful possibilities I can think of are 0 and len(seq), and even those only hypothetically). This solution requires changing a builtin method API, as well as adding keyword argument support to CPython string methods. Also raises the question of whether or not the Sequence ABC (and sequences in the stdlib) should be adjusted accordingly. 3. Search for a more general solution that simplifies the following common try/except pattern into a single expression (or simple statement): try: val = f() except RelevantException: val = default # further operations using val This has been attempted before, but never successfully as it isn't *that* common (so ugly solutions are never going to be accepted as better than the status quo) and it's extremely difficult to come up with a syntax that neatly captures the 3 necessary elements and can still be handled easily by the parser. Since I have no reason to believe 3 will get any further than it has in that past, making an explicit decision between 1 and 2 is the reason I'd like to see a PEP. PEPs aren't just about getting new features into the language - the Rejected ones are also about documenting the reasons we have chosen *not* to do certain things (and the Deferred ones point out that some things are just hard to do in a way that provides a net benefit to the language). Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

Raymond Hettinger wrote:

On Jul 17, 2011, at 12:15 AM, Nick Coghlan wrote:

...

Indeed, the problem as I see it is that our general idiom for functions and methods that raise 'Not Found' exceptions is to accept an optional parameter that specifies a value to return in the Not Found case.

There's a difference between methods that return looked-up values (where a default might make sense) versus a method that returns an index (where it usually makes no sense at all).

We are not talking about a default value to return -- the default will still be the behavior of raising a ValueError if the substring is not found. Consider the proposed signature: _sentinal = object() class str(): def index(substring, start, end, missing=_sentinal): # looks for string .... # string not found -- now what? if missing is _sentinal: raise ValueError('...') else: return missing The addition is that *if* the caller specifies an object for missing, return that value, *otherwise* raise ValueError just like we do now.

...

For historical reasons, we currently break that idiom for index() methods: instead of supplying an extra parameter to str.index, one instead switches to a completely different method (.find()) with no control over the sentinel value returned (it's always -1). For other sequences (e.g. list), there's no find equivalent, so you *have* to write the exception handling out explicitly.

My proposal is to update the signature of index() (for all sequences, including the ABC) to follow the standard 'Not Found' idiom by accepting a 'missing' parameter that is returned for those cases where ValueError would otherwise be raised.

Code that uses str.find would continue to work, but the recommended alternative would be obj.index(x, missing=None) (or appropriate default value).

Hmmm -- okay, perhaps we are... let me say, then, that I agree having a default return is not the way to go; this would break everything that expects .index() to exception out if the substring is not found -- in other words, everything that uses .index(). My take on the idea is to have the new 'missing' argument be optional, and if not specified then current behavior is unchanged, but if specified then that value is returned instead.

If someone takes this out of python-ideas land and into a serious PEP, they should be prepared to answer a number of tough questions:

* Is this actually necessary? Is there something you currently can't code? If not, then it adds API complexity without adding any new capabilities. There is a high threshold for expanding the string API -- this would affect everyone learning python, every book written, every lint tool, every class seeking to be string-like, etc.

Trying to be string-like at the moment is such a PITA I really don't see this tiny extra bit as a serious burden. Consider this nice simple code: class MyStr(str): def find(substr, start=None, end=None): # whatever extra I want to do before passing off to str # now pass off to str return str.find(substr, start, end) Too bad it doesn't work: --> test = MyStr('this is a test') --> test.find('is') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 5, in find TypeError: slice indices must be integers or None or have an __index__ method (Yes, this was fixed in 2.6, and as soon as I'm willing to drop support for earlier versions I can remove the following boilerplate: start = start or 0 end = end or len(self) and yes, I wouldn't be able to use the new .index(..., missing=_whatever) for a while, but that doesn't mean we should stop improving the language.)

* Take a look at what other languages do. Practically every general purpose language has an API for doing substring searches. Since we're not blazing new territory here, there needs to be a good precedent for this change (no shooting from the hip when the problem has already been well solved many times over).

Why not? 'Well solved' does not mean there is no room for improvement. And going through the whole PEP process does not feel like 'shooting from the hip'.

* Consider the effects of adding a second-way-to-do-it. Will it add to the learning curve, cause debates about the best way in a given situation, add more PEP 8 entries and pylint checks? Is it worth introducing version incompatibilities (i.e. runs on 3.3 but not earlier), etc.

You mean like 'runs on 2.6+ but not earlier'?

* What should the default value be?

There should be no default value, in my opinion.

Is there any non-numerical result that ever makes sense; otherwise, you're just making a alias for the -1 currently returned by str.find(). If the default is some value that evaluates to False, will that create a common error where an if-test fails to disambiguate the default value from a substring found at position zero.

The most effective argument by far, IMO, both for not having a default value, and for being very careful about what the caller chooses to use for the missing argument. I think a bomb would be appropriate here: class Bomb(): 'singleton object: blows up on any usage' def __bool__(self): raise OopsError('yell at the programmer!") etc then in usage it's a check for object identity, anything else reminds somebody they forgot to do something.

Hopefully, this short and incomplete list will provide a good basis for thinking about whether the proposal is a good idea. Defending a PEP is no fun at all, so put in all your deep thinking up front.

Many good points -- thank you for taking the time. ~Ethan~

Mike Graham wrote:

On Sat, Jul 16, 2011 at 11:26 AM, Raymond Hettinger <raymond.hettinger@gmail.com> wrote:

...

Unless an API is flat out broken, deprecation is almost always a bad idea. This API has been around for a very long time, so deprecating it will break lots of people's code for almost zero benefit: http://www.google.com/codesearch#search/&q=%5C.find%5C(%20lang:%5Epython$&type=cs

Raymond

I agree that breaking people's code is a bad thing and have not suggested removing str.find. This removal would require a transition like that from Python 2.x to Python 3.x, a move that is not planned and I personally do not ever expect.

I think that's a disingenuous thing to say. You started this thread with an explicit call to deprecate str.find -- see your chosen subject line, and your first paragraph in this thread states: "It should be explicitly deprecated ..." What is the point of deprecating something if you don't intend to eventually remove it?

I appreciate your linking this search, which does indeed does show that str.find is in wide use. However, looking at the first give pages of results, this use seems largely unfortunate—literally the majority of the times str.find is used, I would have used "substring in s" or

A lot of very old code predates startswith and endswith. They only appeared in 2.0. Surprisingly, as late as 2.2, we were limited to testing for a single character: [steve@sylar src]$ python2.2 Python 2.2.3 (#1, Aug 12 2010, 01:08:27) [GCC 4.1.2 20070925 (Red Hat 4.1.2-27)] on linux2 Type "help", "copyright", "credits" or "license" for more information.

...

...

"ab" in "abc" Traceback (most recent call last): File "<stdin>", line 1, in ? TypeError: 'in <string>' requires character as left operand

Some of the code on that first page supports Python 2.1.

"s.startswith(substring)". I also see code like "pos = s.find(" ("); if pos + len(" (...)") > self._maxWidth:" which makes me very uncomfortable and which I would have to read in detail to figure out what's happening confidently if I was the maintaining the code.

Whereas "pos = s.index(" ("); if pos + len(" (...)") > self._maxWidth:" is the height of readability, yes? People can write bad code no matter what tools you give them. -- Steven

On 7/16/2011 2:43 PM, Ronny Pfannschmidt wrote:

Since a DeprecationWarning seems rather intrusive, what would be a less intrusive way to cause code-enhancement in such places?

A PendingDeprecationWarning, which I believe would be off by default, and which could be left as such for several versions. -- Terry Jan Reedy

On Fri, Jul 15, 2011 at 10:57 AM, Guido van Rossum <guido@python.org> wrote:

However, in many cases absence of the string is not an error -- you just need to do something else. So in cases where *if* it's found you need the position, and *if* it isn't found you need to do something else, you'd have to use a try/except block to catch the non-error that is absence. All in all I don't see enough reason to start deprecating find. But perhaps popular lint-like programs could flag likely abuses of find?

--Guido

It isn't necessarily an error if the substring is not in the string (though it sometimes is), but it is an exceptional case. Python uses exceptions pretty liberally most places -- it isn't necessarily an error if an iterator is exhausted or if float("4.2 bad user input") is called or if BdbQuit was raised. In these cases, an exception can be perfectly expected to indicate that what happened is different from the information used in a return value. Making a Python user write a try/except block when she wants to handle both the cases "substring is in s" and "substring isn't in s" seems perfectly fine to me and, really, preferable to the if statement required to handle these two cases. The base two cases really are about the same: try: i = s.index(sub) except IndexError: do_something() vs. i = s.find(sub) if i == -1: do_something() But what if I forgot to handle the special case? i = s.index(sub) # An exception is raised right here and I can fix my code vs. i = s.find(sub) # No exception is raised In this second case, I get the value of -1. Later I can use it as an index, use it in a slice, or perform arithmetic on it. This can introduce seemingly-unrelated values later on, making this especially hard to track down. If the failure return code was at least None it would behave more sanely, but at present the failure return code is a perfectly valid value for almost any use. If a programmer is sill averse to using try/except, we can still write if sub in s: i = s.index(sub) else: do_something() Now. we can dredge up some examples where -1 is the actual value someone wants to use. These cases are so rare and so subtle as to make their use so clever I don't really see their existence as an advantage. Additionally, it is unfortunate that we currently have two methods to do the same thing (which isn't even a super-common task) with different APIs. Nothing about the names "find" and "index" really makes clear which is which. This violates the "There should be one-- and preferably only one --obvious way to do it." principle and makes the Python user need to memorize an unnecessary, arbitrary distinction. I would also point out that it was not a contrived case that I mentioned where a beginner introduces a bug by trying "if s.find(sub):" instead of "if sub in s:"; I have really seen people try this several times. Obviously we cannot make many decisions based on new Python programmers' mistakes, it is worth recognizing them. I hope this additional discussion might be able to sway your opinion here. The only advantage to using str.find is that you do not have to use try/except blocks, but in fact you don't have to with str.index either. On the other hand, there are numerous disadvantages--practical, pedagogical, stylistic, and design--to having and using str.find. Mike

On Fri, Jul 15, 2011 at 12:31 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:

While this would be a very good argument to make if we were currently designing the str API, I don't think the benefits of suppressing str.find outweight the burden of converting old code to use a different idiom.

We could choose to write something about it in the documentation, though.

Regards

Antoine.

That could be just as suitable. To be honest, I can't really see too much of a difference between a stronger note about the problems I perceive with str.find in the documentation and a note that uses the word "deprecated". There certainly wouldn't be occasion in the foreseeable future for you actually to remove str.find. Mike

On Fri, Jul 15, 2011 at 1:06 PM, Bruce Leban <bruce@leapyear.org> wrote:

It seems to me that the solution is that you should never use find and use index instead. You could even modify pylint/etc. to flag any use of find as an error. That doesn't mean it should be deprecated so everyone else has to follow you. Deprecated now => removed in the future. While many can agree that warnings in the documentation about common mistakes in using APIs is a good thing, that doesn't translate into consensus that we should remove the API that might be misused. The documentation already says "The find() method should be used only if you need to know the position of sub. To check if sub is a substring or not, use the in operator." There is *no* general solution to the problem of people misusing APIs. Even with index someone can still write

if s.index(sub):

which probably doesn't do what they were thinking. Should we remove that too?

The case of "if s.index(sub):" as a misspelling of "if sub in s:" is not nearly as errorprone as the case of "if s.find(sub):". For one, "s.find(sub)" *sounds* like it might be true if you can find sub in s and not true if sub cannot be found. With str.index, the fact the return value is going to be an index is made clear. For two, str.find can fail silently in more cases than the str.index case can fail silently. For three, I've never see a learner write "if s.index(sub)" though I have seen several mistakenly write the "if s.find(sub)" version.

Note that one reason people might prefer find over index is that exceptions constrain how you write the code:

try:     i = s.index(sub)     do lots of stuff with s and i except ValueError:     result = 'not found'

In this case the try wraps way too much code so it could catch a ValueError in the middle of 'stuff'. Here's correct code:

try:     i = s.index(sub)     do_more = True except:     result = 'not found'     do_more = False if do_more:     do lots of stuff with s and i

Do you think that's better than:

i = s.find(sub) if i < 0:     result = 'not found' else:     do lots of stuff with s and i

Are you familiar with the "else" clause of a try/except suite? It already handles writing the "right" version nicely. try: i = s.index(sub) except IndexError: result = 'not found' else: do lots of stuff with s and i

--- Bruce

Mike

Am 15.07.2011 18:31, schrieb Antoine Pitrou:

On Fri, 15 Jul 2011 12:12:33 -0400 Mike Graham <mikegraham@gmail.com> wrote:

...

[...] Now. we can dredge up some examples where -1 is the actual value someone wants to use. These cases are so rare and so subtle as to make their use so clever I don't really see their existence as an advantage.

Additionally, it is unfortunate that we currently have two methods to do the same thing (which isn't even a super-common task) with different APIs. Nothing about the names "find" and "index" really makes clear which is which. This violates the "There should be one-- and preferably only one --obvious way to do it." principle and makes the Python user need to memorize an unnecessary, arbitrary distinction.

While this would be a very good argument to make if we were currently designing the str API, I don't think the benefits of suppressing str.find outweight the burden of converting old code to use a different idiom.

We could choose to write something about it in the documentation, though.

Patches are welcome at bugs.python.org ;) Georg

On Fri, Jul 15, 2011 at 12:38 PM, Guido van Rossum <guido@python.org> wrote:

On Fri, Jul 15, 2011 at 9:12 AM, Mike Graham <mikegraham@gmail.com> wrote:

...

It isn't necessarily an error if the substring is not in the string (though it sometimes is), but it is an exceptional case. Python uses exceptions pretty liberally most places -- it isn't necessarily an error if an iterator is exhausted or if float("4.2 bad user input") is called or if BdbQuit was raised. In these cases, an exception can be perfectly expected to indicate that what happened is different from the information used in a return value.

This smells a bit like uncalled-for religion. Remember that readline() returns an empty string at the end of the file instead of raising an exception, and IMO that makes it the better API.

-- --Guido van Rossum (python.org/~guido)

Thanks for this prompt, considerate reply (and the other). I really didn't mean to present an "always use exceptions" dogma here, so much as to claim "Having the programmer use a try/except block isn't a problem," and to say that the fact an exception was raised doesn't necessarily indicate that something went wrong. StopIteration is a strong example that exceptions can often be very expected and normal. Mike

Devin Jeanpierre wrote:

Strings don't even bother to be compatible with .find, if you pass in an index that was returned by .find() you might just get the wrong result. It's pretty much only ever useful in the if check, which is in essence identical to the try-except, except it adds the ability to forget to check and produce incorrect behaviour -- this is not a feature.

Hind-sight -- what a great thing! I do think None would be a better return value. I suspect the odds of that changing now are pretty slim, though. ~Ethan~

On Fri, 15 Jul 2011 20:26:42 +0200 Georg Brandl <g.brandl@gmx.net> wrote:

Sorry, I don't see the immense complication in

try: line = readline() except StopIteration: line = ''

Of course, if the tokenize API was designed from scratch nowadays, it would probably only accept iterables, and you'd have to make one yourself from a file object using

iter(fd.readline, '')

Or, since file objects are iterable out of the box, you could simply pass the file object. Regards Antoine.

On 15Jul2011 12:12, Mike Graham <mikegraham@gmail.com> wrote: | On Fri, Jul 15, 2011 at 10:57 AM, Guido van Rossum <guido@python.org> wrote: | > However, in many cases absence of the string is not an error -- you | > just need to do something else. [...] | | It isn't necessarily an error if the substring is not in the string | (though it sometimes is), but it is an exceptional case. No it isn't, IMO. It's simply the _other_ case. | Python uses | exceptions pretty liberally most places -- it isn't necessarily an | error if an iterator is exhausted or if float("4.2 bad user input") is | called or if BdbQuit was raised. In these cases, an exception can be | perfectly expected to indicate that what happened is different from | the information used in a return value. In all the cases you cite the exception indicates failure of the operation: .next() has nothing to "next" to, float is being handed garbage etc. str.find does not have a failure mode, it has string found and string not found. | Making a Python user write a try/except block when she wants to handle | both the cases "substring is in s" and "substring isn't in s" seems | perfectly fine to me and, really, preferable to the if statement | required to handle these two cases. You don't find try/except wordy and opaque? I find "if" more idiomatic most of the time. Not to mention vague: it can often be quite hard to be sure the raised exception came from just the operation you imagine it came from. With str.find there's little scope for vagueness I agree (unless you aren't really using a str, but a duck-type). But plenty of: try: x = foofunc(y) except IndexError, e: ... is subject to uncaught IndexError arbitrarily deep in foofunc's call stack. | The base two cases really are about the same: [... try ... excpt ...] | vs. | | i = s.find(sub) | if i == -1: | do_something() | | But what if I forgot to handle the special case? [...] | In this second case, I get the value of -1. Later I can use it as an | index, use it in a slice, or perform arithmetic on it. This can | introduce seemingly-unrelated values later on, making this especially | hard to track down. I agree it may be a pity that str.find doesn't return None on string not found, which would generally raise an exception on an attempt to use it as a number. Cheers, -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ Any company large enough to have a research lab is large enough not to listen to it. - Alan Kay

This is an absolute flight of fancy, and I'm sure it's already been rejected in the past (a quick search says Aug. 2009, http://mail.python.org/pipermail/python-ideas/2009-August/thread.html#5576 ), but what about some kind of try/except expression? i = s.index(substr) except ValueError is None I guess my main problem with that is the color of the bikeshed: it's hard to get a good idiomatic way of spelling the except expression. You could use a colon, as proposed in that thread, but it seems to me a colon indicates a new line follows. I would be for making this as simple as possible. No "as", no non-implicit "else", and no nesting of exception types. If you want something fancy, use a statement.

On Sat, Jul 16, 2011 at 6:47 PM, Georg Brandl <g.brandl@gmx.net> wrote:

By the way: str.index() raises ValueError, not IndexError, if the substring isn't found.  That's one argument against "the index() API is foolproof, while the find() API isn't".

Yeah, that bugs me too. Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

On Sat, Jul 16, 2011 at 7:11 PM, M.-A. Lemburg <mal@egenix.com> wrote:

Georg Brandl wrote:

...

By the way: str.index() raises ValueError, not IndexError, if the substring isn't found.  That's one argument against "the index() API is foolproof, while the find() API isn't".

That's because the *value* passed as argument to .index() isn't found. Which error class to use often depends on your view point, so in some cases it may seem natural to you, in others you have a different POV, and it feels wrong :-)

Yeah, but trying to catch IndexError instead of ValueError is a pretty easy (and understandable) mistake to make, given the correspondence in names and logical train of thought "index() tells me the index where a substring can be found, IndexError is used to denote that a given index doesn't exist, so if the requested index doesn't exist, then this function will throw IndexError". Easily detected by testing, but still unintuitive and annoying. If IndexError had instead been called IndexNotFound, then there would never have been the slightest question as to which exception should have been thrown. Too late to change it now, though. We just all have to learn that, from the str.index point of view, failing to find a substring means there's something *wrong* with one (or both) of the passed in strings rather than the less opinionated "the index you have requested doesn't actually exist". Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

Mike Graham wrote:

On Fri, Jul 15, 2011 at 6:50 PM, Cameron Simpson wrote:

...

You don't find try/except wordy and opaque? I find "if" more idiomatic most of the time.

Not really. If we don't like exceptionhandling, we're using the wrong language.

I think exception handling is awesome. I also think dictionaries are awesome, but I don't use them when what I need is a set. ~Ethan~

On 7/16/2011 2:46 PM, Terry Reedy wrote:

On 7/15/2011 6:50 PM, Cameron Simpson wrote:

...

On 15Jul2011 12:12, Mike Graham<mikegraham@gmail.com> wrote: age etc.

str.find does not have a failure mode, it has string found and string not found.

Not finding == failure to find == failure to match an implied re.

Indeed, negative values such as -1 are standard error/failure return codes for functions that normally return nonnegative ints and that written in statically typed languages without catchable exceptions. In C, for instance, EOF is defined as an implementation-defined negative int and I am sure -1 is used by some. -- Terry Jan Reedy

On 16Jul2011 22:21, Masklinn <masklinn@masklinn.net> wrote: | On 2011-07-16, at 21:52 , Terry Reedy wrote: | > On 7/16/2011 2:46 PM, Terry Reedy wrote: | >> On 7/15/2011 6:50 PM, Cameron Simpson wrote: | >>> str.find does not have a failure mode, it has string found and string | >>> not found. | >> | >> Not finding == failure to find == failure to match an implied re. | > Indeed, negative values such as -1 are standard error/failure | > return codes for functions that normally return nonnegative ints and | > that written in statically typed languages without catchable exceptions | | … or type systems worth using. And that's for those which are | 0-indexed of course, especially for finding sub-sequences. | | And interestingly, the function corresponding to `str.find` in libc | returns `NULL` in case of failure, not −1 (its return value is a | pointer to the first occurrence of the needle in the haysack). Though if you're thinking NULL is equivalent to None (which is often is conceptually), let's remember that in C NULL is just a pointer value; it is a sentinal, but not a different type. So NULL here is in some ways akin to -1 in Python's find return. You still need to test for it; not all platforms will (for example) segfault if NULL is dereferenced. Cheers, -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ There's no need to worry about death, it will not happen in your lifetime. - Raymond Smullyan

Nick Coghlan wrote:

So, I have a different suggestion (that actually paves the way for the eventual deprecation of find()):

Update str.index() to accept an optional sentinel value.

If the sentinel argument is not supplied, then a missing substring raises ValueError as it does now. If it is supplied, then a missing substring returns the sentinel value instead of throwing an exception. The above idiom could then be expressed cleanly as:

idx = s.index(sub, missing=None) if idx is not None: # do something that needs idx

+1 Excellent idea.

However, this seemingly simple suggestion is complicated by the fact that string methods do not currently accept keyword arguments and index() already accepts two optional positional arguments (for substring searching).

Does this require a PEP or just someone to start working on it? Getting str() to accept keyword arguments and adding missing to str.index() sounds like a nice project. ~Ethan~

On Sat, Jul 16, 2011 at 11:46 AM, Ethan Furman <ethan@stoneleaf.us> wrote:

Raymond Hettinger wrote:

...

Unless an API is flat out broken, deprecation is almost always a bad idea. This API has been around for a very long time, so deprecating it will break lots of people's code for almost zero benefit:

http://www.google.com/codesearch#search/&q=%5C.find%5C(%20lang:%5Epython$&type=cs <http://www.google.com/codesearch#search/&q=%5C.find%5C(%20lang:%5Epython$&type=cs>

How ironic that the fist hit seems to display the problem Mike is concerned with:

       position =  min(position, len(self.contents))        if hasattr(newChild, 'parent') and newChild.parent != None:            # We're 'inserting' an element that's already one            # of this object's children.            if newChild.parent == self:                index = self.find(newChild)                if index and index < position:                    # Furthermore we're moving it further down the                    # list of this object's children. That means that                    # when we extract this element, our target index                    # will jump down one.                    position = position - 1

I haven't read all the surrounding code to know if this will ever fail, but the whole 'index = ... .find(...); if index and ...' certainly doesn't lend confidence.  After all, if you *know* newChild is in self, why not use .index()?

~Ethan~

Note that this particular code is using BeautifulSoup.PageElement.find, not str.find. There are, however, in the first few pages similar snippets which use the result of str.find without a check for the special case. Mike

On Sat, Jul 16, 2011 at 11:26 AM, Raymond Hettinger <raymond.hettinger@gmail.com> wrote:

Unless an API is flat out broken, deprecation is almost always a bad idea. This API has been around for a very long time, so deprecating it will break lots of people's code for almost zero benefit: http://www.google.com/codesearch#search/&q=%5C.find%5C(%20lang:%5Epython$&type=cs

Raymond

I agree that breaking people's code is a bad thing and have not suggested removing str.find. This removal would require a transition like that from Python 2.x to Python 3.x, a move that is not planned and I personally do not ever expect. I appreciate your linking this search, which does indeed does show that str.find is in wide use. However, looking at the first give pages of results, this use seems largely unfortunate—literally the majority of the times str.find is used, I would have used "substring in s" or "s.startswith(substring)". I also see code like "pos = s.find(" ("); if pos + len(" (...)") > self._maxWidth:" which makes me very uncomfortable and which I would have to read in detail to figure out what's happening confidently if I was the maintaining the code. Thanks for the reply, Mike

On Sat, Jul 16, 2011 at 3:07 PM, Raymond Hettinger <raymond.hettinger@gmail.com> wrote:

Deprecation is a step towards removal and it always causes a certain amount of pain.

ISTM, the time for any move like this would have been the jump from Python 2 to Python 3 where significant breakage was expected and where transition tools were developed.

Because this would break backwards compatibility unnecessarily, I think anyone involved can agree that actual removal could not take place until a special jump akin to the Python 2->Python 3 jump. (It would surprise me if such a jump ever actually took place.)

...

I appreciate your linking this search, which does indeed does show that str.find is in wide use. However, looking at the first give pages of results, this use seems largely unfortunate—literally the majority of the times str.find is used, I would have used "substring in s" or "s.startswith(substring)". I also see code like "pos = s.find("  ("); if pos + len(" (...)") > self._maxWidth:" which makes me very uncomfortable and which I would have to read in detail to figure out what's happening confidently if I was the maintaining the code.

I am largely unsympathetic to arguments that are roughly equivalent to "I don't like the way other people write programs".

Reviewing five pages of results, over 2/3 of the uses of str.find could be replaced by str.__contains__. Using sub in s instead of s.find(sub) != -1 is *already* the advice in the official Python documentation, so I do not believe I am making especially personal judgments about the style of the code.

Something akin to the str.find() API has been present in many, many languages for a very long time.  For the most part, people seem to be able to use it reasonably well.  I find that beginning Python students never seem to have a problem with it.

I'm glad your experience with learners here has been more consistent than mine.

Also keep it mind that startswith() and endswith() were relatively recent additions to Python, so it is no surprise that lots of code uses find() instead of startswith().

Raymond

Mike

I just remembered one other thought on the subject. Usually, when Python introduces a method such as .index() that raises an exception for the not-found case, there are immediate requests for variants that don't raise exceptions: dict.pop(key, default) dict.get(key, defalut) next(iterable, default) getattr(obj, attr, default) re.match() --> None or matchobject People seem to hate wrapping try/except around simple calls. Expect those people to be agitated if you take away str.find(). Raymond

On Jul 16, 2011, at 2:04 PM, Mike Graham wrote:

Did you catch Nick's suggestion to enhance str.index to have the same basic API as dict.get/getattr/etc.?

Sorry, but I thought idea that was way off base. Things like dict.get and getattr are about returning values, so it is possible to provide a meaningful default. In the case of string.find, there is no meaningful default position in the string. So, a default would simply be a way to turn the -1 value into some other object which you would still need to test. We don't need to make the API worse by expanding it just for the sake of change. The str.find() method isn't broken or useless. If you want people to change the way they write code, it would be better to do it through education (blog posts, pylint, etc) rather than by breaking a venerable API. Raymond

On Jul 16, 2011, at 3:15 PM, Devin Jeanpierre wrote:

Slicing was brought up as a use-case where you can use a default without checking.

This thread has lost contact with reality. It started with the reasonable observation that many uses of str.find could be replaced with a simple test using the in-operator. Now, the thread is venturing into the typical python-ideas world of making-up random use cases and APIs. Something like str.find() is in many languages and it is definitely not the norm for them to have found a need to both be able to return -1 or to supply a default value. As a Python teacher, speaker, and consultant, I have the opportunity to see and review the code repositories for many companies. I'm pretty sure that I've never seen a utils module with the likes of: def myfind(fullstring, substring, default=None): i = fullstring.find(substring) if i == -1 and default is not None: return default return i When I start seeing people routinely using a helper function like this, I'll start to believe that a str.find default value isn't silly. Mike's initial post was well grounded in observations about code that could be improved by using "in" or str.index() instead of str.find(). Though I disagreed with the recommendation to deprecate, it would be even worse to exacerbate the usability issues by making the method signature even more complex (with a new optional argument and a new signature variant for the return value). That isn't progress. It's aspiring cruft that makes the language harder to learn and remember. Raymond

Raymond Hettinger wrote:

This thread has lost contact with reality. It started with the reasonable observation that many uses of str.find could be replaced with a simple test using the in-operator.

Now, the thread is venturing into the typical python-ideas world of making-up random use cases and APIs. Something like str.find() is in many languages and it is definitely not the norm for them to have found a need to both be able to return -1 or to supply a default value.

Nick's proposal was to enhance str.index(), not string.find(); Having str.index() accept a value to return on failure instead of raising an exception means it could do both jobs, and would also make it much less likely to wrongly use the failure return value of -1 from str.find() which is, unfortunately, a legitimate index value. ~Ethan~

On Jul 17, 2011, at 12:15 AM, Nick Coghlan wrote:

Indeed, the problem as I see it is that our general idiom for functions and methods that raise 'Not Found' exceptions is to accept an optional parameter that specifies a value to return in the Not Found case.

There's a difference between methods that return looked-up values (where a default might make sense) versus a method that returns an index (where it usually makes no sense at all).

For historical reasons, we currently break that idiom for index() methods: instead of supplying an extra parameter to str.index, one instead switches to a completely different method (.find()) with no control over the sentinel value returned (it's always -1). For other sequences (e.g. list), there's no find equivalent, so you *have* to write the exception handling out explicitly.

My proposal is to update the signature of index() (for all sequences, including the ABC) to follow the standard 'Not Found' idiom by accepting a 'missing' parameter that is returned for those cases where ValueError would otherwise be raised.

Code that uses str.find would continue to work, but the recommended alternative would be obj.index(x, missing=None) (or appropriate default value). I would advise against any actual deprecation of str,find (cf. the deliberate lack of optparse deprecation).

It's unfortunate that backwards compatibility means we can't use the more descriptive name, but that's life.

However, I already have too much on my plate to push this forward for Python 3.3. I'm able to offer advice if someone would like to try their hand at writing a PEP, though.

If someone takes this out of python-ideas land and into a serious PEP, they should be prepared to answer a number of tough questions: * Is this actually necessary? Is there something you currently can't code? If not, then it adds API complexity without adding any new capabilities. There is a high threshold for expanding the string API -- this would affect everyone learning python, every book written, every lint tool, every class seeking to be string-like, etc. So, it would need be a substantive improvement to be accepted. * Take a look at what other languages do. Practically every general purpose language has an API for doing substring searches. Since we're not blazing new territory here, there needs to be a good precedent for this change (no shooting from the hip when the problem has already been well solved many times over). * Use Google's code search to identify examples of real world code that would better with the new API. If the only use case is creating a new slicing one-liner, that likely is too rare and arcane to warrant a change. * Consider the effects of adding a second-way-to-do-it. Will it add to the learning curve, cause debates about the best way in a given situation, add more PEP 8 entries and pylint checks? Is it worth introducing version incompatibilities (i.e. runs on 3.3 but not earlier), etc. * What should the default value be? Is there any non-numerical result that ever makes sense; otherwise, you're just making a alias for the -1 currently returned by str.find(). If the default is some value that evaluates to False, will that create a common error where an if-test fails to disambiguate the default value from a substring found at position zero. If the new API is ambiguous or confusing in *any* way, then it will be a step backwards and make Python worse rather than better. * See if you can find examples where people have already found the need to write a helper function such as: def index_default(s, sub, default): try: return s.index(sub) except ValueError: return default If you find code like that in the wild, it may be an indication that people want this. If you don't, it may indicate otherwise. * Good API design requires some thinking about function/method signatures. Would making this a keyword-only argument soive the positional arguments problem? Since str.index() already takes arguments for the "start" and "end" index, is the full signature readable without keywords: mystr.index(possible_substr, 0, -1, default_value) Also look at the signature for the return value. Currently, it always returns a number, but if it can return a number or anything else, then all client code must be prepared to handle the alternatives with clean looking code that is self-evidently correct. * Perhaps talk to some people who write python code for a living to determine if they've ever needed this or whether it would end-up as cruft. (In my case, the answer is that I've not needed or wanted this in a over a decade of heavy Python use). Hopefully, this short and incomplete list will provide a good basis for thinking about whether the proposal is a good idea. Defending a PEP is no fun at all, so put in all your deep thinking up front. Cheers, Raymond

On Jul 17, 2011, at 2:13 AM, Masklinn wrote:

On 2011-07-17, at 10:09 , Raymond Hettinger wrote:

...

On Jul 17, 2011, at 12:15 AM, Nick Coghlan wrote:

...

Indeed, the problem as I see it is that our general idiom for functions and methods that raise 'Not Found' exceptions is to accept an optional parameter that specifies a value to return in the Not Found case.

There's a difference between methods that return looked-up values (where a default might make sense) versus a method that returns an index (where it usually makes no sense at all). Why not? An index is a looked-up value here (it's just a reverse lookup) and .find is returning a (non-configurable) default value is it not?

There is a fundamental difference between content (values in the list) and the list position. There are meaningful search defaults for the former but not the latter. It's important that to grasp this distinction before going further.

...

* Take a look at what other languages do. Practically every general purpose language has an API for doing substring searches. Since we're not blazing new territory here, there needs to be a good precedent for this change (no shooting from the hip when the problem has already been well solved many times over). Other languages do everything and their reverse. You have languages returning −1 or 0 (the latter for 1-indexed languages), languages returning differently-typed sentinels, etc…

SML even returns the length of the string.

See a listing at http://en.wikipedia.org/wiki/Comparison_of_programming_languages_(string_fun...

My reading is that not a single one of these entries has a signature with a user specifiable default value of arbitrary type. IOW, there is nothing resembling proposal on the table. AFAICT, all of them are equivalent to either our current str.find() or str.index(). Raymond

On Sun, Jul 17, 2011 at 8:56 PM, Steven D'Aprano <steve@pearwood.info> wrote:

I can't really see that they do "everything and their reverse". There are two basic strategies: return an out-of-bound value, and raise an exception, both of which Python already does.

In a language that accepts negative indices, -1 is not out of bounds, and hence is an objectively bad value to return. If str.find() returned None for missing substrings, we wouldn't be having this discussion. However, backwards compatibility requirements mean that option is not available to us. Is this a language wart where the correct answer is to tell everyone new to the language that complains about the current behaviour to "suck it up and learn to live with it"? Maybe. But claiming that str.find() is a *good* API in a language that accepts negative indices would be flat out wrong. Good *enough*, maybe, but not good in an absolute sense. As I see it, there are a few options. 1. Do nothing. Quite a reasonable option. str.find() is flawed and str.index() can be annoying to use, but fixing this may not be worth the hassle. However, this is not the same as claiming the str.find() behaviour is a good API - it's just acknowledging the wart and deciding not to address it. 2. Add an alternate behaviour to str.index() that allows the exception to optionally be converted into a sentinel value. Inspired by getattr, getitem, dict.get, next, etc. Made messy by the need for the sentinel to go after the existing arguments and the fact that the number of sane sentinel values is extremely limited (aside from None, the only remotely useful possibilities I can think of are 0 and len(seq), and even those only hypothetically). This solution requires changing a builtin method API, as well as adding keyword argument support to CPython string methods. Also raises the question of whether or not the Sequence ABC (and sequences in the stdlib) should be adjusted accordingly. 3. Search for a more general solution that simplifies the following common try/except pattern into a single expression (or simple statement): try: val = f() except RelevantException: val = default # further operations using val This has been attempted before, but never successfully as it isn't *that* common (so ugly solutions are never going to be accepted as better than the status quo) and it's extremely difficult to come up with a syntax that neatly captures the 3 necessary elements and can still be handled easily by the parser. Since I have no reason to believe 3 will get any further than it has in that past, making an explicit decision between 1 and 2 is the reason I'd like to see a PEP. PEPs aren't just about getting new features into the language - the Rejected ones are also about documenting the reasons we have chosen *not* to do certain things (and the Deferred ones point out that some things are just hard to do in a way that provides a net benefit to the language). Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

Raymond Hettinger wrote:

On Jul 17, 2011, at 12:15 AM, Nick Coghlan wrote:

...

Indeed, the problem as I see it is that our general idiom for functions and methods that raise 'Not Found' exceptions is to accept an optional parameter that specifies a value to return in the Not Found case.

There's a difference between methods that return looked-up values (where a default might make sense) versus a method that returns an index (where it usually makes no sense at all).

We are not talking about a default value to return -- the default will still be the behavior of raising a ValueError if the substring is not found. Consider the proposed signature: _sentinal = object() class str(): def index(substring, start, end, missing=_sentinal): # looks for string .... # string not found -- now what? if missing is _sentinal: raise ValueError('...') else: return missing The addition is that *if* the caller specifies an object for missing, return that value, *otherwise* raise ValueError just like we do now.

...

For historical reasons, we currently break that idiom for index() methods: instead of supplying an extra parameter to str.index, one instead switches to a completely different method (.find()) with no control over the sentinel value returned (it's always -1). For other sequences (e.g. list), there's no find equivalent, so you *have* to write the exception handling out explicitly.

My proposal is to update the signature of index() (for all sequences, including the ABC) to follow the standard 'Not Found' idiom by accepting a 'missing' parameter that is returned for those cases where ValueError would otherwise be raised.

Code that uses str.find would continue to work, but the recommended alternative would be obj.index(x, missing=None) (or appropriate default value).

Hmmm -- okay, perhaps we are... let me say, then, that I agree having a default return is not the way to go; this would break everything that expects .index() to exception out if the substring is not found -- in other words, everything that uses .index(). My take on the idea is to have the new 'missing' argument be optional, and if not specified then current behavior is unchanged, but if specified then that value is returned instead.

If someone takes this out of python-ideas land and into a serious PEP, they should be prepared to answer a number of tough questions:

* Is this actually necessary? Is there something you currently can't code? If not, then it adds API complexity without adding any new capabilities. There is a high threshold for expanding the string API -- this would affect everyone learning python, every book written, every lint tool, every class seeking to be string-like, etc.

Trying to be string-like at the moment is such a PITA I really don't see this tiny extra bit as a serious burden. Consider this nice simple code: class MyStr(str): def find(substr, start=None, end=None): # whatever extra I want to do before passing off to str # now pass off to str return str.find(substr, start, end) Too bad it doesn't work: --> test = MyStr('this is a test') --> test.find('is') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 5, in find TypeError: slice indices must be integers or None or have an __index__ method (Yes, this was fixed in 2.6, and as soon as I'm willing to drop support for earlier versions I can remove the following boilerplate: start = start or 0 end = end or len(self) and yes, I wouldn't be able to use the new .index(..., missing=_whatever) for a while, but that doesn't mean we should stop improving the language.)

* Take a look at what other languages do. Practically every general purpose language has an API for doing substring searches. Since we're not blazing new territory here, there needs to be a good precedent for this change (no shooting from the hip when the problem has already been well solved many times over).

Why not? 'Well solved' does not mean there is no room for improvement. And going through the whole PEP process does not feel like 'shooting from the hip'.

* Consider the effects of adding a second-way-to-do-it. Will it add to the learning curve, cause debates about the best way in a given situation, add more PEP 8 entries and pylint checks? Is it worth introducing version incompatibilities (i.e. runs on 3.3 but not earlier), etc.

You mean like 'runs on 2.6+ but not earlier'?

* What should the default value be?

There should be no default value, in my opinion.

Is there any non-numerical result that ever makes sense; otherwise, you're just making a alias for the -1 currently returned by str.find(). If the default is some value that evaluates to False, will that create a common error where an if-test fails to disambiguate the default value from a substring found at position zero.

The most effective argument by far, IMO, both for not having a default value, and for being very careful about what the caller chooses to use for the missing argument. I think a bomb would be appropriate here: class Bomb(): 'singleton object: blows up on any usage' def __bool__(self): raise OopsError('yell at the programmer!") etc then in usage it's a check for object identity, anything else reminds somebody they forgot to do something.

Hopefully, this short and incomplete list will provide a good basis for thinking about whether the proposal is a good idea. Defending a PEP is no fun at all, so put in all your deep thinking up front.

Many good points -- thank you for taking the time. ~Ethan~

Mike Graham wrote:

On Sat, Jul 16, 2011 at 11:26 AM, Raymond Hettinger <raymond.hettinger@gmail.com> wrote:

...

Unless an API is flat out broken, deprecation is almost always a bad idea. This API has been around for a very long time, so deprecating it will break lots of people's code for almost zero benefit: http://www.google.com/codesearch#search/&q=%5C.find%5C(%20lang:%5Epython$&type=cs

Raymond

I agree that breaking people's code is a bad thing and have not suggested removing str.find. This removal would require a transition like that from Python 2.x to Python 3.x, a move that is not planned and I personally do not ever expect.

I think that's a disingenuous thing to say. You started this thread with an explicit call to deprecate str.find -- see your chosen subject line, and your first paragraph in this thread states: "It should be explicitly deprecated ..." What is the point of deprecating something if you don't intend to eventually remove it?

I appreciate your linking this search, which does indeed does show that str.find is in wide use. However, looking at the first give pages of results, this use seems largely unfortunate—literally the majority of the times str.find is used, I would have used "substring in s" or

A lot of very old code predates startswith and endswith. They only appeared in 2.0. Surprisingly, as late as 2.2, we were limited to testing for a single character: [steve@sylar src]$ python2.2 Python 2.2.3 (#1, Aug 12 2010, 01:08:27) [GCC 4.1.2 20070925 (Red Hat 4.1.2-27)] on linux2 Type "help", "copyright", "credits" or "license" for more information.

...

...

"ab" in "abc" Traceback (most recent call last): File "<stdin>", line 1, in ? TypeError: 'in <string>' requires character as left operand

Some of the code on that first page supports Python 2.1.

"s.startswith(substring)". I also see code like "pos = s.find(" ("); if pos + len(" (...)") > self._maxWidth:" which makes me very uncomfortable and which I would have to read in detail to figure out what's happening confidently if I was the maintaining the code.

Whereas "pos = s.index(" ("); if pos + len(" (...)") > self._maxWidth:" is the height of readability, yes? People can write bad code no matter what tools you give them. -- Steven

On 7/16/2011 2:43 PM, Ronny Pfannschmidt wrote:

Since a DeprecationWarning seems rather intrusive, what would be a less intrusive way to cause code-enhancement in such places?

A PendingDeprecationWarning, which I believe would be off by default, and which could be left as such for several versions. -- Terry Jan Reedy