[Twisted-Python] SQL ORM for Twisted & PostgreSQL?
I know that this has been asked before, but it's been a while and I'm hoping for some good news. Is there a SQL ORM that works well with Twisted and PostgreSQL? In particular I'm hoping to find something that works with txpostgres as that's the library I prefer to access the database with. -- Jeff Ollie
On Jul 17, 2013, at 5:04 PM, Jeffrey Ollie <jeff@ocjtech.us> wrote:
I know that this has been asked before, but it's been a while and I'm hoping for some good news. Is there a SQL ORM that works well with Twisted and PostgreSQL? In particular I'm hoping to find something that works with txpostgres as that's the library I prefer to access the database with.
There's bits of one in Calendar Server: <http://trac.calendarserver.org/browser/CalendarServer/trunk/twext/enterprise...> It doesn't use txpostgres, but that's because it relies on adbapi2 (also in calendar server, hopefully to be moved into Twisted at some point). -glyph
Hi,
I know that this has been asked before, but it's been a while and I'm hoping for some good news. Is there a SQL ORM that works well with Twisted and PostgreSQL? In particular I'm hoping to find something that works with txpostgres as that's the library I prefer to access the database with.
There's bits of one in Calendar Server: <http://trac.calendarserver.org/browser/CalendarServer/trunk/twext/enterprise...> It doesn't use txpostgres, but that's because it relies on adbapi2 (also in calendar server, hopefully to be moved into Twisted at some point).
How would you feel about packaging it up on PyPI so people can try it out effortlessly? What do Apple’s licenses say about that? Yes, I’m volunteering. Cheers, Hynek
On Fri, Jul 19, 2013 at 9:06 AM, Hynek Schlawack <hs@ox.cx> wrote:
How would you feel about packaging it up on PyPI so people can try it out effortlessly? What do Apple’s licenses say about that? Yes, I’m volunteering.
It seems it's released under the ASL2. I don't know if Apple prevents *its* employees from doing anything in particular, but it seems like third party contributors are free to do with it as they please (within the limits of the license, of course). cheers lvh
On Jul 19, 2013, at 1:21 AM, Laurens Van Houtven <_@lvh.io> wrote:
On Fri, Jul 19, 2013 at 9:06 AM, Hynek Schlawack <hs@ox.cx> wrote: How would you feel about packaging it up on PyPI so people can try it out effortlessly? What do Apple’s licenses say about that? Yes, I’m volunteering.
It seems it's released under the ASL2. I don't know if Apple prevents *its* employees from doing anything in particular, but it seems like third party contributors are free to do with it as they please (within the limits of the license, of course).
There are actually bits of this that I've been meaning to contribute back for a long time. If somebody would like to help me out with it then maybe I can eke out a little bit of time to do it - I've just been pretty busy :-). -glyph
Am 22.07.2013 um 23:37 schrieb Glyph <glyph@twistedmatrix.com>:
How would you feel about packaging it up on PyPI so people can try it out effortlessly? What do Apple’s licenses say about that? Yes, I’m volunteering.
It seems it's released under the ASL2. I don't know if Apple prevents *its* employees from doing anything in particular, but it seems like third party contributors are free to do with it as they please (within the limits of the license, of course).
There are actually bits of this that I've been meaning to contribute back for a long time. If somebody would like to help me out with it then maybe I can eke out a little bit of time to do it - I've just been pretty busy :-).
What exactly do you mean? Sounds like you’d like it to got straight into Twisted? Wouldn’t it make more sense to release it separately first? You know, kind of http://www.codinghorror.com/blog/2013/07/rule-of-three.html We could move it into the twisted namespace though.
On Jul 23, 2013, at 12:12 AM, Hynek Schlawack <hs@ox.cx> wrote:
Am 22.07.2013 um 23:37 schrieb Glyph <glyph@twistedmatrix.com>:
How would you feel about packaging it up on PyPI so people can try it out effortlessly? What do Apple’s licenses say about that? Yes, I’m volunteering.
It seems it's released under the ASL2. I don't know if Apple prevents *its* employees from doing anything in particular, but it seems like third party contributors are free to do with it as they please (within the limits of the license, of course).
There are actually bits of this that I've been meaning to contribute back for a long time. If somebody would like to help me out with it then maybe I can eke out a little bit of time to do it - I've just been pretty busy :-).
What exactly do you mean? Sounds like you’d like it to got straight into Twisted? Wouldn’t it make more sense to release it separately first? You know, kind of http://www.codinghorror.com/blog/2013/07/rule-of-three.html
That's also a possibility, but following that rule makes the assumption that Calendar Server is just one application, when it's at least two (calendar & contacts) ;-).
We could move it into the twisted namespace though.
The main motivation to do this, especially with adbapi2, is because Twisted already has an implementation of this exact thing with some deficiencies. -glyph
On 05:43 pm, glyph@twistedmatrix.com wrote:
On Jul 23, 2013, at 12:12 AM, Hynek Schlawack <hs@ox.cx> wrote:
What exactly do you mean? Sounds like you’d like it to got straight into Twisted? Wouldn’t it make more sense to release it separately first? You know, kind of http://www.codinghorror.com/blog/2013/07 /rule-of-three.html
That's also a possibility, but following that rule makes the assumption that Calendar Server is just one application, when it's at least two (calendar & contacts) ;-).
We could move it into the twisted namespace though.
The main motivation to do this, especially with adbapi2, is because Twisted already has an implementation of this exact thing with some deficiencies.
As far as I can tell, there's nothing stopping Hynek from grabbing all the code and releasing it as a separate project. It seems like this would get the ball rolling more quickly and I don't see the downside. If anything, having it as an independent project is more likely to get more people interested in it more quickly and so increase the pool of people who might be interested in addressing whatever concerns need to be addressed for its inclusion in Twisted. So, why block that work in favor of inclusion directly in Twisted? Jean-Paul
Am 24.07.2013 um 20:05 schrieb exarkun@twistedmatrix.com:
It seems like this would get the ball rolling more quickly and I don't see the downside. If anything, having it as an independent project is more likely to get more people interested in it more quickly and so increase the pool of people who might be interested in addressing whatever concerns need to be addressed for its inclusion in Twisted.
That’s exactly my line of thought here. adbapi2 has been pointed out several times now on this list but I doubt people really use it since it’s rather cumbersome to get. I’m not fighting/delaying its inclusion in Twisted, I just don’t see that happen effectively anytime soon even if the integration work started immediately. I would just add a smaller step in between that may provide additional insights about potential usage + add some momentum. Cheers, —h
It would be really beneficial if this were something that fully works with twisted, but is not dependent on it. For example, I have a "Project" that mostly uses SqlAlchemy. It started out in Pylons, new development is on Pyramid and there are additional tasks in Celery + some more in Twisted. Aside from a few manual db tasks in Twisted, they all share and re-use a common "Model" package. Having a great ORM for twisted is wonderful , but its way less exciting and attractive if it's only for twisted.
I am currently using SQLAlchemy with Twisted with deferToThread and it works rather well, have you tried it? So long as you create a new session for each thread you spawn (which you should also do without Twisted) it works without any modification required. Here's an example of using SQLAlchemy with Twisted— no guarantees on it's the best way to do things, but it works. def checkIfNodeExists(nodeId): def threadFunction(): s = Session() r = s.query(Node).filter(Node.NodeId == nodeId).count() if r is not 0: return True else: return False return threads.deferToThread(threadFunction) On Aug 21, 2013, at 9:05 PM, Jonathan Vanasco <twisted-python@2xlp.com> wrote:
It would be really beneficial if this were something that fully works with twisted, but is not dependent on it.
For example, I have a "Project" that mostly uses SqlAlchemy. It started out in Pylons, new development is on Pyramid and there are additional tasks in Celery + some more in Twisted. Aside from a few manual db tasks in Twisted, they all share and re-use a common "Model" package.
Having a great ORM for twisted is wonderful , but its way less exciting and attractive if it's only for twisted. _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Aug 21, 2013, at 11:05 AM, Jonathan Vanasco <twisted-python@2xlp.com> wrote:
Having a great ORM for twisted is wonderful , but its way less exciting and attractive if it's only for twisted.
Doing this is simple, although probably not easy: you just need to convince the SQLAlchemy folks to separate the process of generating SQL and executing SQL, and expose hooks for event-driven frameworks (which is an ever-expanding circle now, encompassing Twisted, Tornado, Pulsar, and Tulip) to schedule SQL execution with an event-driven backend rather than assuming it can block. When I faced this problem, I wanted to use SQLAlchemy rather than write my own ORM, but I was unable to due the lack of a public API for generating cross-database SQL. -glyph
This reminded me of something I read years ago from Doug Schmidt on a pattern half sync half async: http://www.cs.wustl.edu/~schmidt/PDF/PLoP-95.pdf I feel it is relevant but the picture may be upside down in relation to writing twisted wrappers which may want to wrap a blocking API. On Aug 21, 2013, at 11:05 AM, Jonathan Vanasco <twisted-python@2xlp.com<mailto:twisted-python@2xlp.com>> wrote: Having a great ORM for twisted is wonderful , but its way less exciting and attractive if it's only for twisted. Doing this is simple, although probably not easy: you just need to convince the SQLAlchemy folks to separate the process of generating SQL and executing SQL, and expose hooks for event-driven frameworks (which is an ever-expanding circle now, encompassing Twisted, Tornado, Pulsar, and Tulip) to schedule SQL execution with an event-driven backend rather than assuming it can block. When I faced this problem, I wanted to use SQLAlchemy rather than write my own ORM, but I was unable to due the lack of a public API for generating cross-database SQL. -glyph
FWIW, the separation of generating SQL and executing SQL was my intent in designing this: https://github.com/iffy/norm Currently, it *only* has asynchronous SQL execution, but it wouldn't be hard to add synchronous execution. Also, there's no subclassing of modeled classes. Matt Haggard On Wed, Aug 21, 2013 at 12:35 PM, Glyph <glyph@twistedmatrix.com> wrote:
On Aug 21, 2013, at 11:05 AM, Jonathan Vanasco <twisted-python@2xlp.com> wrote:
Having a great ORM for twisted is wonderful , but its way less exciting and attractive if it's only for twisted.
Doing this is simple, although probably not easy: you just need to convince the SQLAlchemy folks to separate the process of generating SQL and executing SQL, and expose hooks for event-driven frameworks (which is an ever-expanding circle now, encompassing Twisted, Tornado, Pulsar, and Tulip) to schedule SQL execution with an event-driven backend rather than assuming it can block.
When I faced this problem, I wanted to use SQLAlchemy rather than write my own ORM, but I was unable to due the lack of a public API for generating cross-database SQL.
-glyph
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Jul 24, 2013, at 11:05 AM, exarkun@twistedmatrix.com wrote:
On 05:43 pm, glyph@twistedmatrix.com wrote:
On Jul 23, 2013, at 12:12 AM, Hynek Schlawack <hs@ox.cx> wrote:
What exactly do you mean? Sounds like you’d like it to got straight into Twisted? Wouldn’t it make more sense to release it separately first? You know, kind of http://www.codinghorror.com/blog/2013/07 /rule-of-three.html
That's also a possibility, but following that rule makes the assumption that Calendar Server is just one application, when it's at least two (calendar & contacts) ;-).
We could move it into the twisted namespace though.
The main motivation to do this, especially with adbapi2, is because Twisted already has an implementation of this exact thing with some deficiencies.
As far as I can tell, there's nothing stopping Hynek from grabbing all the code and releasing it as a separate project.
It seems like this would get the ball rolling more quickly and I don't see the downside. If anything, having it as an independent project is more likely to get more people interested in it more quickly and so increase the pool of people who might be interested in addressing whatever concerns need to be addressed for its inclusion in Twisted.
So, why block that work in favor of inclusion directly in Twisted?
Jean-Paul
In principle I can see the appeal of spinning out small, lightweight projects to test enhancements to Twisted, but my enthusiasm for this idea has been tempered over time by the fact that it has never actually worked (that I can recall). How many projects on <https://launchpad.net/tx> have ever made it to be features within Twisted? Remember how we weren't going to work on our own process pooling implementation because <https://launchpad.net/ampoule> was going to solve that problem, and we'd just include it when it was done? As far as I know we haven't even managed to pull in small things like <https://github.com/dreid/treq/blob/master/treq/client.py#L57>. First of all, there's change-tracking overhead. People interested in ADBAPI would now have 3 places to potentially put code: Hynek's hypothetical project, calendarserver.org, and Twisted. Twisted code-reviewers are probably not going to look at Hynek's project (they hardly look at Twisted at it is), and that means that it might have a laxer approach to cool new features, which in turn means that "more interest" just means that it's going to be a bigger ball of code that is harder to get re-integrated. Calendar Server developers are going to keep maintaining the version in the Calendar Server repository, so it's going to diverge. Everybody's using a different source tree, so merging will all have to be done manually. Then there's the fact that provenance tracking is a pain. Let's say Hynek makes a new project on Github. The license clearly gives him the right to do that. Now, when people contribute to that repository, who are they contributing to? Hynek, or Twisted, or Calendar Server (or Github?)? What license are they providing those modifications under, Apache2 or MIT? This is all totally possible to keep track of and deal with, and indeed, if I don't have the time to be responsive to Hynek I'd recommend that he go ahead and do so; I'm sure we can work it out later if we need to. But preparing it for contribution to Twisted is just an engineering headache under a simple, established licensing and contribution process, and given the option between an engineering headache and a licensing headache PLUS an engineering headache PLUS a process headache, I'll take the engineering headache every time. ;-) If you just want to see broader testing first, a better solution would be for Hynek to just contribute to the Calendar Server project directly so that there are effectively 2 parties involved rather than 3; for starters, we could have a setup.py in the calendarserver.org repository that just submits twext.enterprise.* to PyPI as an independent source distribution, while remaining part of the same project. It can *appear* to be a separate project as long as it's developed in the same place. :-) -glyph
Am 25.07.2013 um 00:43 schrieb Glyph <glyph@twistedmatrix.com>:
If you just want to see broader testing first, a better solution would be for Hynek to just contribute to the Calendar Server project directly so that there are effectively 2 parties involved rather than 3; for starters, we could have a setup.py in the calendarserver.org repository that just submits twext.enterprise.* to PyPI as an independent source distribution, while remaining part of the same project. It can *appear* to be a separate project as long as it's developed in the same place. :-)
I don’t really care *how* it gets to PyPI – all I was suggesting to get it there *somehow* so people can get excited about it.
On Jul 25, 2013, at 1:08 AM, Hynek Schlawack <hs@ox.cx> wrote:
Am 25.07.2013 um 00:43 schrieb Glyph <glyph@twistedmatrix.com>:
If you just want to see broader testing first, a better solution would be for Hynek to just contribute to the Calendar Server project directly so that there are effectively 2 parties involved rather than 3; for starters, we could have a setup.py in the calendarserver.org repository that just submits twext.enterprise.* to PyPI as an independent source distribution, while remaining part of the same project. It can *appear* to be a separate project as long as it's developed in the same place. :-)
I don’t really care *how* it gets to PyPI – all I was suggesting to get it there *somehow* so people can get excited about it.
Maybe I just read too much into "grab all the source files", then :-). -glyph
On 24 Jul, 10:43 pm, glyph@twistedmatrix.com wrote:
If you just want to see broader testing first, a better solution would be for Hynek to just contribute to the Calendar Server project directly so that there are effectively 2 parties involved rather than 3; for starters, we could have a setup.py in the calendarserver.org repository that just submits twext.enterprise.* to PyPI as an independent source distribution, while remaining part of the same project. It can *appear* to be a separate project as long as it's developed in the same place. :-)
That sounds good to me. Jean-Paul
-glyph
participants (9)
-
Burak Nehbit -
exarkun＠twistedmatrix.com -
Glyph -
Hynek Schlawack -
Jeffrey Ollie -
Jonathan Vanasco -
Laurens Van Houtven -
Matt Haggard -
Mike Winter (miwinter)