[ZODB] Multithreaded connection pooling support for ZODB databases?
Etienne Robillard
tkadm30 at yandex.com
Fri Apr 20 04:49:09 EDT 2018
Heads up people!
I've finally managed to make a working `ThreadedConnectionPool` class
for supporting multiple ZODB databases connections on top of libschevo
API! :)
Technically, I decided to use gevent.Greenlet under the hood to spawn
multiple cooperative ClientStorage client/server connections.
I'm also making all my testing and development under PyPy 5.9.
You can check out the code here:
https://bitbucket.org/tkadm30/libschevo/commits/37feb029615d76f3d81233990e509b6eb6ffb5d7
Comments are welcome! :)
Etienne
Le 2018-04-16 à 18:27, Etienne Robillard a écrit :
> Hi Jason,
>
> I just made some more changes to my ThreadedConnectionPool class here:
> https://bitbucket.org/tkadm30/django-hotsauce/commits/81d2e8f30019840d9c8bbdf7f82df6de2be024fc
>
> In summary, the `ThreadedConnectionPool` class is now a subclass of
> `threading.Thread` and is extending the `run` method to populate a
> thread local dictionary with ClientStorage connections.
>
> Kind regards,
>
> Etienne
> Le 2018-04-16 à 09:14, Jason Madden a écrit :
>>>>> On Apr 15, 2018, at 8:17 PM, Etienne Robillard
>>>>> <tkadm30 at yandex.com> wrote:
>>>>>
>>>>> I would like to define a `ThreadedConnectionPoll` class to allow
>>>>> multithreaded caching of ZODB databases into memory.
>>
>> The ZODB.DB object is *already* thread safe for connection
>> management. You shouldn't need to do anything other than use
>> ZODB.DB().open() and conn.close() from all the threads you're using.
>> (That is, create exactly one ZODB.DB object for each database you are
>> using and share that object amongst your threads, using
>> db.open()/conn.close() as needed. For storages like RelStorage that
>> have caches at the ZODB.DB level---which are also thread safe---this
>> is important.)
>>
>> If you are managing multiple databases and want to be able to make
>> references between them (a multiple-database) it is critical that
>> they share the same `databases` object. But that is an advanced usage.
>>
>>
>>> Here's a updated version of my code so far using threading.local :
>>>
>>> from threading import local
>>> from notmm.dbapi.orm import ClientStorageProxy
>>>
>>> class ThreadedConnectionPool(object):
>>>
>>> def __init__(self, d, debug=True):
>>> if debug:
>>> assert isinstance(d, dict) == True
>>> local.pool = d
>>> for key,value in local.pool.iteritems():
>> Unless you left out part of the code, it appears that you're trying
>> to set (and read) a class attribute on the threading.local class.
>> This does not create thread-local state. It is also not portable and
>> doesn't work when local is an extension type:
>>
>> py> from threading import local
>> py> import sys
>> py> sys.version_info # CPython 3.7; same result in 2.7
>> sys.version_info(major=3, minor=7, micro=0, releaselevel='beta',
>> serial=3)
>> py> local
>> <class '_thread._local'>
>> py> local.foo = 1
>> Traceback (most recent call last):
>> File "<stdin>", line 1, in <module>
>> TypeError: can't set attributes of built-in/extension type
>> '_thread._local'
>>
>> pypy> sys.version_info # pypy 2
>> (major=2, minor=7, micro=13, releaselevel='final', serial=42)
>> pypy> from threading import local
>> pypy> local.foo = 1
>> Traceback (most recent call last):
>> File "<stdin>", line 1, in <module>
>> TypeError: can't set attributes on type object '_local'
>>
>> It generally also shouldn't be necessary as the parameters to
>> __init__ are preserved and used again in each thread; you can use
>> this to share information if those parameters are mutable:
>>
>> py> from threading import get_ident
>> py> from threading import Thread
>> py> class MyLocal(local):
>> ... def __init__(self, l):
>> ... print("Creating in thread", get_ident(), "param", l)
>> ... l.append(get_ident())
>> ...
>> py> l = []
>> py> mylocal = MyLocal(l)
>> Creating in thread 140736147411840 param []
>> py> l
>> [140736147411840]
>> py> def target():
>> ... mylocal.foo = 1
>> ...
>> py> t = Thread(target=target)
>> py> t.start()
>> Creating in thread 123145538318336 param [140736147411840]
>> None
>> py> l
>> [140736147411840, 123145538318336]
>>
>> Jason
>>
>
--
Etienne Robillard
tkadm30 at yandex.com
https://www.isotopesoftware.ca/
More information about the Python-list
mailing list