[python-ldap] Performance improvement for schema.tokenizer.split_tokens

[Michael Ströder]

Christian,

Christian Heimes wrote:
> I was able to come up with a new implementation of split_tokens() which
> is about 8 times faster on Python 2.

thanks for your code submission.

[1] http://www.openldap.org/devel/contributing.html#notice

> The new implementation uses a
> regular expression to split the schema string into tokens. It is
> successfully able to parse over 3,000 schema lines from 389-DS and
> FreeIPA with same result as the curent split_tokens() function.
> Personally I find it easier to read and understand, too.

Note that 389-DS schema is somewhat broken (re-use of same OIDs) but for testing this
particular function it works. I've also successfully tested your implementation with my
own monster schema collection for OpenLDAP.

> Please review my implementation and consider it for python-ldap.

I've prepared all to commit your improvement (slightly modified) but I have to insist on
an IPR notice for your submission (I've been to careless in the past). Please have a look
at [1] for some sample text and replace "OpenLDAP Software" with "python-ldap module".

Ciao, Michael.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3829 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://mail.python.org/pipermail/python-ldap/attachments/20170218/20854369/attachment.bin>