Unicode and Python - how often do you index strings?
Dave Angel
davea at davea.name
Wed Jun 4 12:50:49 EDT 2014
Chris Angelico <rosuav at gmail.com> Wrote in message:
> On Wed, Jun 4, 2014 at 8:10 PM, Peter Otten <__peter__ at web.de> wrote:
>> The indices used for slicing typically don't come out of nowhere. A simple
>> example would be
>>
>> def strip_prefix(text, prefix):
>> if text.startswith(prefix):
>> text = text[len(prefix):]
>> return text
>>
>> If both prefix and text use UTF-8 internally the byte offset is already
>> known. The question is then how we can preserve that information.
>
> Almost completely useless. First off, it solves only the problem of
> operating on the string at exactly some point where you just got an
> index; and secondly, you don't always get that index from a string
> method. Suppose, for instance, that you iterate over a string thus:
>
> for i, ch in enumerate(string):
> if ch=='{': start = i
> elif ch=='}': return string[start:end+1]
>
> Okay, so this could be done by searching, but for something more
> complicated, I can imagine it being better to enumerate. (But "I can
> imagine" is much weaker than "Here's code that we use in production",
> which is why I asked the question.)
>
> Incidentally, the above code highlights the first problem too. With
> direct indexing, you can ask for inclusive or exclusive slicing by
> adding or subtracting one from the index. If you do that with a
> byte-position-retaining special integer, you lose the byte position.
>
> ChrisA
>
A string could have two extra fields in it that hold index and
offset for the most recent substring reference. Even though the
string is immutable, nothing prevents mutable elements that are
externally visible only by performance measurement.
So a loop using a subscript of a string would tend to be faster
even if written in a naive way.
It's also conceivable to build an array of such pairs in strings
over a threshold size. So if you had a megabyte string, there
might be 100 evenly spaced pairs, calculated when the string
object is first created.
And naturally there can be flags indicating that the particular
string is pure ASCII.
Clearly this breaks down if there are two alternating references
at different offsets, but I think this would be exceeding
rare.
--
DaveA
More information about the Python-list
mailing list