Thanks. Are you aware of any resources that clarify which pymongo calls are lazy and which actually result in calls to the Mongo server (thus really benefit from derferToThread)? Clayton Daley clayton.daley@gmail.com 513.505.1236 On Tue, Dec 23, 2014 at 1:51 AM, Gelin Yan wrote:

On Mon, Dec 22, 2014 at 5:05 AM, Clayton Daley wrote:

...

Are there any major disadvantages of using pymongo with callInThread instead of txmongo? I'd like to take advantage of some newer features in pymongo (unfortunately not available in txmongo) and it's certainly easier to maintain feature parity using callInThread.

Clayton Daley

Hi Clayton

Txmongo has been inactive for a while so I suggest using pymongo & twisted thread pool. I have been using them for a couple of years and they work quite well.

Regards

gelin yan

_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

I'm still in "proof of concept" mode so take my comments with a grain of salt... but my question was motivated by datetime issues: - After switching to txmongo, timedelta seemed to break. Turns out it breaks autobahn as well so I may need to solve the problem sooner... but it's a material difference. - I also wanted to take advantage of some timezone stuff purportedly included in newer versions of pymongo (but I haven't fully implemented anything yet to verify). When it comes to interest in txmongo: - Insofar as it would address the question of lazy calls (and possible hassles like not-lazy iterators), it would be a welcome simplification to deferToThread. - At the same time, I don't see any intrinsic benefit to rebuild the client all-the-way-down. If it were possible to wrap PyMongo in a lazy-aware (and deferred returning) wrapper... that would be enough for me. - If a wrapper reduces the work to maintain the project at parity, I'd redouble my support for the approach. No need to commit volunteer supporters (current and/or future) to more work than absolutely necessary. Clayton Daley On Tue, Dec 30, 2014 at 4:50 AM, bret curtis wrote:

On Tue, Dec 23, 2014 at 7:51 AM, Gelin Yan wrote:

...

On Mon, Dec 22, 2014 at 5:05 AM, Clayton Daley wrote:

...

Are there any major disadvantages of using pymongo with callInThread instead of txmongo? I'd like to take advantage of some newer features in pymongo (unfortunately not available in txmongo) and it's certainly easier to maintain feature parity using callInThread.

Clayton Daley

Hi Clayton

Txmongo has been inactive for a while so I suggest using pymongo & twisted thread pool. I have been using them for a couple of years and they work quite well.

Regards

gelin yan

Hey guys,

just wanted to drop in and say that txmongo isn't dead. We (Amplidata) just recently sent a merge request to add SSL (authentication/encryption) support[1] that Alexandre will review after the holidays. While Alexandre isn't currently developing txmongo, he does review and accept pull requests. So while the community is small, it is still alive.

Amplidata (the company I work for) has also expressed an interest in getting txmongo up to speed with PyMongo, similar in how we got Ldaptor and its myriad forks to converge and development happening again.[2][3]

Is there any interest in seeing txmongo be developed further? What are some of the pain points that people would rather defer to thread PyMongo instead of using async txmongo?

Cheers, Bret

[1] https://github.com/fiorix/mongo-async-python-driver/pull/67 [2] https://twistedmatrix.com/pipermail/twisted-python/2014-July/028489.html [3] https://github.com/twisted/ldaptor

_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

On Sat, Jan 31, 2015 at 7:37 PM, Gelin Yan wrote:

On Tue, Dec 30, 2014 at 5:50 PM, bret curtis wrote:

...

On Tue, Dec 23, 2014 at 7:51 AM, Gelin Yan wrote:

...

On Mon, Dec 22, 2014 at 5:05 AM, Clayton Daley wrote:

...

Are there any major disadvantages of using pymongo with callInThread instead of txmongo? I'd like to take advantage of some newer features in pymongo (unfortunately not available in txmongo) and it's certainly easier to maintain feature parity using callInThread.

Clayton Daley

Hi Clayton

Txmongo has been inactive for a while so I suggest using pymongo & twisted thread pool. I have been using them for a couple of years and they work quite well.

Regards

gelin yan

Hey guys,

just wanted to drop in and say that txmongo isn't dead. We (Amplidata) just recently sent a merge request to add SSL (authentication/encryption) support[1] that Alexandre will review after the holidays. While Alexandre isn't currently developing txmongo, he does review and accept pull requests. So while the community is small, it is still alive.

Amplidata (the company I work for) has also expressed an interest in getting txmongo up to speed with PyMongo, similar in how we got Ldaptor and its myriad forks to converge and development happening again.[2][3]

Is there any interest in seeing txmongo be developed further? What are some of the pain points that people would rather defer to thread PyMongo instead of using async txmongo?

Cheers, Bret

Hi Bret

I am glad to know txmongo is still here. I used it several times a few years ago and was impressed by its speed (twice faster than pymongo & thread pool).

I hope txmongo can support the latest GEO index & replication.

Regards

gelin yan

Hello Gelin, You should check out our other thread about getting TxMongo moved into Twisted's github org. https://twistedmatrix.com/pipermail/twisted-python/2015-January/029144.html There is already some support for Geo and replication, but it could always use more work. What is it specifically you would like to see, aside from everything. ;) https://github.com/fiorix/txmongo/blob/master/txmongo/filter.py ^-- Support for GEO2D and GEOHAYSTACK spatial indices were added in 2012. When you specify your hosts in ConnectionPool, you can give it a full mongodb URI that includes all your replication servers, even with SSL support. It is smart enough to cycle through until it finds a master to continue operations. I'm still trying to a compile a list of pain points that people have that can be addressed. The more specific, the better. Cheers, Bret

We're not sure about what OP has done, but if there is ever anything lacking in TxMongo... please open an issue/ticket with us over at: https://github.com/twisted/txmongo Cheers, Bret On Wed, Oct 7, 2015 at 8:18 PM, Jonathan Vanasco wrote:

On Dec 21, 2014, at 4:05 PM, Clayton Daley wrote:

...

Are there any major disadvantages of using pymongo with callInThread instead of txmongo? I'd like to take advantage of some newer features in pymongo (unfortunately not available in txmongo) and it's certainly easier to maintain feature parity using callInThread.

I know thread is a bit old, but I'm wondering if the OP ended up using callInThread or not , and if there were any issues.

I started working on a txmongo solution to offload some storage, but then realized I was already in a deferred thread (to handle blocking operations) and i probably should be using pymongo.

_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

I (OP) was looking into these libraries/options as part of a side project that's been delayed by real work. I'm also relegating Mongo to a caching layer after stumbling upon Sarah Mei's "Why You Should Never Use MongoDB" and realizing I was going to have the exact same problems. Clayton Daley On Thu, Oct 8, 2015 at 12:23 PM, Jonathan Vanasco wrote:

On Oct 8, 2015, at 6:19 AM, bret curtis wrote:

We're not sure about what OP has done, but if there is ever anything lacking in TxMongo... please open an issue/ticket with us over at: https://github.com/twisted/txmongo

Thanks for the reply! I actually did open a few lowball tickets on TxMongo yesterday -- the package doesn't specify the minimum required versions for mongo or twisted.

I did some tests yesterday on PyMongo, and it seems like using it in deferToThread is safe. I also ran some simulations on staging under a heavy load; no exceptions, everything passes, data is stored correctly. The current pymongo "client" appears to be a threadsafe connection pool, and I can just grab a connection from it when needed in a thread. Woo Hoo!

If the next version of this tool stays in Python, we'll definitely need to use TxMongo. In the current design, we're already deferred to a thread and would have to restructure too much to use TxMongo.

_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

There is one thing that TxMongo does that PyMongo, Motor and the rest don't do: handle deadlines and timeouts. We're (HGST/WesternDigital) using MongoDB (with WiredTiger) in real-time applications and one of our requirements was to honour application set deadlines (or timeouts) on a per-call basis. So if you are interested in that, we have it. :) For those just tuning in... Timeout: how many seconds the call is allowed run before raising an error. Deadline: a time (unix timestamp) in the future that the call is allowed to run before raising an error. We take advantage of Twisted's native deferred cancellation mechanism in addition to validation checks that no modifications happen to MongoDB when a TimeExceeded is raised. We've been very satisfied with how MongoDB has performed (post 3.0/WT) along with TxMongo and Twisted. Cheers, Bret On Fri, Oct 9, 2015 at 7:52 PM, Jonathan Vanasco wrote:

On Oct 9, 2015, at 1:06 PM, Clayton Daley wrote:

I (OP) was looking into these libraries/options as part of a side project that's been delayed by real work. I'm also relegating Mongo to a caching layer after stumbling upon Sarah Mei's "Why You Should Never Use MongoDB" and realizing I was going to have the exact same problems

Thanks for the update!

FWIW, (sorry, non-twisted!) if you want to avoid mongodb:

1. We use redis for KV caching, and its really, really, really fast. 2. Riak can use LevelDB as a backing store -- leveldb supports snappy compression on documents 3. If you need the document search, i really like Elastic

I don't really like/trust mongodb from past experience myself. I would normally not use it for anything other than a read-through cache... except for my current need. We have a web spider and need to archive the snapshots. WireTiger, which was bought by mongodb and integrated into the 3.x branch, is the only storage backend I've found that will do "bucket level compression". I forget if it's applied on pages or blocks, but it will basically apply zlib or snappy compression onto sets of documents. I looked for weeks on something that could do that, it was the only option. We did a lot of storage tests on our typical data, and bucketed compression saves way more space than document compression or document compression using a seed.

_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

On Oct 12, 2015, at 1:23 AM, Gelin Yan wrote:

I have switched from pymongo to txmongo for about two months. I got at least 2x faster on query. Despite of some minor issues which have been fixed recently, I am very happy with it.

This is really cool to hear. I try not to say "twisted will make everything faster" because speed is a complex question and a component like Twisted is rarely ever the one answer you need... but it's so nice when it just does :-D. -glyph