[ZODB] Multithreaded connection pooling support for ZODB databases?
Etienne Robillard
tkadm30 at yandex.com
Wed Apr 25 13:21:02 EDT 2018
Hi,
I would like to know if you could please share some input about this
ThreadConnectionPool class for libschevo/ZODB. :)
Thanks!
Etienne
Le 2018-04-20 à 04:49, Etienne Robillard a écrit :
> Heads up people!
>
> I've finally managed to make a working `ThreadedConnectionPool` class
> for supporting multiple ZODB databases connections on top of libschevo
> API! :)
>
> Technically, I decided to use gevent.Greenlet under the hood to spawn
> multiple cooperative ClientStorage client/server connections.
>
> I'm also making all my testing and development under PyPy 5.9.
>
> You can check out the code here:
> https://bitbucket.org/tkadm30/libschevo/commits/37feb029615d76f3d81233990e509b6eb6ffb5d7
>
> Comments are welcome! :)
>
> Etienne
>
>
> Le 2018-04-16 à 18:27, Etienne Robillard a écrit :
>> Hi Jason,
>>
>> I just made some more changes to my ThreadedConnectionPool class
>> here:
>> https://bitbucket.org/tkadm30/django-hotsauce/commits/81d2e8f30019840d9c8bbdf7f82df6de2be024fc
>>
>> In summary, the `ThreadedConnectionPool` class is now a subclass of
>> `threading.Thread` and is extending the `run` method to populate a
>> thread local dictionary with ClientStorage connections.
>>
>> Kind regards,
>>
>> Etienne
>> Le 2018-04-16 à 09:14, Jason Madden a écrit :
>>>>>> On Apr 15, 2018, at 8:17 PM, Etienne Robillard
>>>>>> <tkadm30 at yandex.com> wrote:
>>>>>>
>>>>>> I would like to define a `ThreadedConnectionPoll` class to allow
>>>>>> multithreaded caching of ZODB databases into memory.
>>>
>>> The ZODB.DB object is *already* thread safe for connection
>>> management. You shouldn't need to do anything other than use
>>> ZODB.DB().open() and conn.close() from all the threads you're using.
>>> (That is, create exactly one ZODB.DB object for each database you
>>> are using and share that object amongst your threads, using
>>> db.open()/conn.close() as needed. For storages like RelStorage that
>>> have caches at the ZODB.DB level---which are also thread safe---this
>>> is important.)
>>>
>>> If you are managing multiple databases and want to be able to make
>>> references between them (a multiple-database) it is critical that
>>> they share the same `databases` object. But that is an advanced usage.
>>>
>>>
>>>> Here's a updated version of my code so far using threading.local :
>>>>
>>>> from threading import local
>>>> from notmm.dbapi.orm import ClientStorageProxy
>>>>
>>>> class ThreadedConnectionPool(object):
>>>>
>>>> def __init__(self, d, debug=True):
>>>> if debug:
>>>> assert isinstance(d, dict) == True
>>>> local.pool = d
>>>> for key,value in local.pool.iteritems():
>>> Unless you left out part of the code, it appears that you're trying
>>> to set (and read) a class attribute on the threading.local class.
>>> This does not create thread-local state. It is also not portable and
>>> doesn't work when local is an extension type:
>>>
>>> py> from threading import local
>>> py> import sys
>>> py> sys.version_info # CPython 3.7; same result in 2.7
>>> sys.version_info(major=3, minor=7, micro=0, releaselevel='beta',
>>> serial=3)
>>> py> local
>>> <class '_thread._local'>
>>> py> local.foo = 1
>>> Traceback (most recent call last):
>>> File "<stdin>", line 1, in <module>
>>> TypeError: can't set attributes of built-in/extension type
>>> '_thread._local'
>>>
>>> pypy> sys.version_info # pypy 2
>>> (major=2, minor=7, micro=13, releaselevel='final', serial=42)
>>> pypy> from threading import local
>>> pypy> local.foo = 1
>>> Traceback (most recent call last):
>>> File "<stdin>", line 1, in <module>
>>> TypeError: can't set attributes on type object '_local'
>>>
>>> It generally also shouldn't be necessary as the parameters to
>>> __init__ are preserved and used again in each thread; you can use
>>> this to share information if those parameters are mutable:
>>>
>>> py> from threading import get_ident
>>> py> from threading import Thread
>>> py> class MyLocal(local):
>>> ... def __init__(self, l):
>>> ... print("Creating in thread", get_ident(), "param", l)
>>> ... l.append(get_ident())
>>> ...
>>> py> l = []
>>> py> mylocal = MyLocal(l)
>>> Creating in thread 140736147411840 param []
>>> py> l
>>> [140736147411840]
>>> py> def target():
>>> ... mylocal.foo = 1
>>> ...
>>> py> t = Thread(target=target)
>>> py> t.start()
>>> Creating in thread 123145538318336 param [140736147411840]
>>> None
>>> py> l
>>> [140736147411840, 123145538318336]
>>>
>>> Jason
>>>
>>
>
--
Etienne Robillard
tkadm30 at yandex.com
https://www.isotopesoftware.ca/
More information about the Python-list
mailing list