[python-ldap] Performance improvement for schema.tokenizer.split_tokens

[Christian Heimes]

On 2017-02-18 15:51, Michael Ströder wrote:
> Christian,
> 
> Christian Heimes wrote:
>> I was able to come up with a new implementation of split_tokens() which
>> is about 8 times faster on Python 2.
> 
> thanks for your code submission.
> 
> [1] http://www.openldap.org/devel/contributing.html#notice
> 
>> The new implementation uses a
>> regular expression to split the schema string into tokens. It is
>> successfully able to parse over 3,000 schema lines from 389-DS and
>> FreeIPA with same result as the curent split_tokens() function.
>> Personally I find it easier to read and understand, too.
> 
> Note that 389-DS schema is somewhat broken (re-use of same OIDs) but for testing this
> particular function it works. I've also successfully tested your implementation with my
> own monster schema collection for OpenLDAP.
> 
>> Please review my implementation and consider it for python-ldap.
> 
> I've prepared all to commit your improvement (slightly modified) but I have to insist on
> an IPR notice for your submission (I've been to careless in the past). Please have a look
> at [1] for some sample text and replace "OpenLDAP Software" with "python-ldap module".

Hi Michael,

I see no problem with the IPR notice. I added a note to the doc string
of fast_split_tokens.py.

In the mean time a user complained that my tokenizer silently ignores
invalid schema lines. The last version does some rudimentary checks and
refuses additional residue. It also handles one of the two special
Oracle cases from you test file.

Regards,
Christian