CorePost 0.0.7 - all about content types
CorePost 0.0.7, the REST microframework built on top of twisted.web is out. http://pypi.python.org/pypi/CorePost/0.0.7 This is a big release with lots of enhancements all around content types: 1) automatic parsing of incoming content based on type @route("/post/json",(Http.POST,Http.PUT)) def test_json(self,request,**kwargs): return "%s" % json.dumps(request.json) @route("/post/xml",(Http.POST,Http.PUT)) def test_xml(self,request,**kwargs): return "%s" % ElementTree.tostring(request.xml) @route("/post/yaml",(Http.POST,Http.PUT)) def test_yaml(self,request,**kwargs): return "%s" % yaml.dump(request.yaml) 2) ability to route to different methods for the same URL by incoming content @route("/post/by/content",(Http.POST,Http.PUT),MediaType.APPLICATION_JSON) def test_content_app_json(self,request,**kwargs): return request.received_headers[HttpHeader.CONTENT_TYPE] @route("/post/by/content",(Http.POST,Http.PUT),(MediaType.TEXT_XML,MediaType.APPLICATION_XML)) def test_content_xml(self,request,**kwargs): return request.received_headers[HttpHeader.CONTENT_TYPE] @route("/post/by/content",(Http.POST,Http.PUT),MediaType.TEXT_YAML) def test_content_yaml(self,request,**kwargs): return request.received_headers[HttpHeader.CONTENT_TYPE] @route("/post/by/content",(Http.POST,Http.PUT)) def test_content_catch_all(self,request,**kwargs): return MediaType.WILDCARD 3) ability to return dict/list response and have them automatically convert to JSON or YAML, depending on what caller can Accept @route("/return/by/accept") def test_return_content_by_accepts(self,request,**kwargs): val = [{"test1":"Test1"},{"test2":"Test2"}] return val Calling this URL with "Accept: application/json" will return: [{"test1": "Test1"}, {"test2": "Test2"}] Calling it with "Accept: text/yaml" will return: - {test1: Test1} - {test2: Test2} 4) proper support for *defer.returnValue()* *in @defer.inlineCallbacks routers (which supports the auto-conversion shown above) @route("/",Http.GET) @defer.inlineCallbacks def root(self,request,**kwargs): val1 = yield db.query("SELECT ....") val2 = yield db.query("SELECT ....") defer.returnValue(val1 + val2) Cheers, Jacek
On Sep 29, 2011, at 5:49 PM, Jacek Furmankiewicz wrote:
CorePost 0.0.7, the REST microframework built on top of twisted.web is out.
Congrats on another quick release, Jacek! It seems like your first example doesn't really need the 'dumps' calls - your other features will automatically serialize those objects, right? I love the automatic MIME-type negotiation on output. Very cool. -glyph
Depends. pyaml is very forgiving in terms of what you feed to it, but json.dumps() only really seems to accept list/dict without complaining. I will keep refining this. One shortcoming is that it does not serialize Python classes (json.dumps() says they're not JSON serializable). I will probably need to write a custom marshaller that uses a class's __dict__ to convert it to a dict (including traversing the whole object graph). XML does not work either yet, could not find a library that does dict/list/class conversion to XML. Will need to write one myself too. But all the underlying plumbing in terms of content routing is there. Once I am done with all of this I will have a serious look at AMPoule to get that out-of-the-box multi-core support. So the next release 0.0.8 should finish all the content type stuff and then for 0.0.9 should start looking at AMPoule. Jacek On Thu, Sep 29, 2011 at 11:22 PM, Glyph Lefkowitz <glyph@twistedmatrix.com>wrote:
On Sep 29, 2011, at 5:49 PM, Jacek Furmankiewicz wrote:
CorePost 0.0.7, the REST microframework built on top of twisted.web is out.
Congrats on another quick release, Jacek!
It seems like your first example doesn't really need the 'dumps' calls - your other features will automatically serialize those objects, right?
I love the automatic MIME-type negotiation on output. Very cool.
-glyph
_______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
And once I reach 0.1 with all of these features, I will post some blog entries about this together with PyPy benchmarks...so that people don't think Javascript / Node.js is your only option when it comes to async I/O web frameworks :-) BTW, is there any progress on getting Twisted to install under PyPy in the near future? I know you guys have been looking into it.... Thanks! Jacek On Fri, Sep 30, 2011 at 9:08 AM, Jacek Furmankiewicz <jacek99@gmail.com>wrote:
Depends. pyaml is very forgiving in terms of what you feed to it, but json.dumps() only really seems to accept list/dict without complaining.
I will keep refining this. One shortcoming is that it does not serialize Python classes (json.dumps() says they're not JSON serializable). I will probably need to write a custom marshaller that uses a class's __dict__ to convert it to a dict (including traversing the whole object graph).
XML does not work either yet, could not find a library that does dict/list/class conversion to XML. Will need to write one myself too. But all the underlying plumbing in terms of content routing is there.
Once I am done with all of this I will have a serious look at AMPoule to get that out-of-the-box multi-core support.
So the next release 0.0.8 should finish all the content type stuff and then for 0.0.9 should start looking at AMPoule.
Jacek
On Thu, Sep 29, 2011 at 11:22 PM, Glyph Lefkowitz <glyph@twistedmatrix.com
wrote:
On Sep 29, 2011, at 5:49 PM, Jacek Furmankiewicz wrote:
CorePost 0.0.7, the REST microframework built on top of twisted.web is out.
Congrats on another quick release, Jacek!
It seems like your first example doesn't really need the 'dumps' calls - your other features will automatically serialize those objects, right?
I love the automatic MIME-type negotiation on output. Very cool.
-glyph
_______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
On 01:18 pm, jacek99@gmail.com wrote:
And once I reach 0.1 with all of these features, I will post some blog entries about this together with PyPy benchmarks...so that people don't think Javascript / Node.js is your only option when it comes to async I/O web frameworks :-)
BTW, is there any progress on getting Twisted to install under PyPy in the near future? I know you guys have been looking into it....
http://twistedmatrix.com/trac/ticket/5158 is resolved now. If there are still problems with Twisted trunk@HEAD on PyPy, please let us know. :) Jean-Paul
I wil check it out tonight and try installing with PyPy 1.6. Any ETA on an official 11.X.X build? Jacek On Fri, Sep 30, 2011 at 9:55 AM, <exarkun@twistedmatrix.com> wrote:
On 01:18 pm, jacek99@gmail.com wrote:
And once I reach 0.1 with all of these features, I will post some blog entries about this together with PyPy benchmarks...so that people don't think Javascript / Node.js is your only option when it comes to async I/O web frameworks :-)
BTW, is there any progress on getting Twisted to install under PyPy in the near future? I know you guys have been looking into it....
http://twistedmatrix.com/trac/ticket/5158 is resolved now. If there are still problems with Twisted trunk@HEAD on PyPy, please let us know. :)
Jean-Paul
_______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
On Sep 30, 2011, at 16:55 , exarkun@twistedmatrix.com wrote:
On 01:18 pm, jacek99@gmail.com wrote:
And once I reach 0.1 with all of these features, I will post some blog entries about this together with PyPy benchmarks...so that people don't think Javascript / Node.js is your only option when it comes to async I/O web frameworks :-)
BTW, is there any progress on getting Twisted to install under PyPy in the near future? I know you guys have been looking into it....
http://twistedmatrix.com/trac/ticket/5158 is resolved now. If there are still problems with Twisted trunk@HEAD on PyPy, please let us know. :)
There are in fact some problems. $ sudo pypy-c ./setup.py install running install running bdist_egg running egg_info writing top-level names to Twisted.egg-info/top_level.txt writing requirements to Twisted.egg-info/requires.txt writing dependency_links to Twisted.egg-info/dependency_links.txt writing Twisted.egg-info/PKG-INFO writing manifest file 'Twisted.egg-info/SOURCES.txt' installing library code to build/bdist.macosx-10.6-i386/egg running install_lib running build_py running build_ext cc -arch i386 -fPIC -Wimplicit -I/opt/local/lib/pypy/include -c conftest.c -o conftest.o building 'twisted.runner.portmap' extension cc -arch i386 -fPIC -Wimplicit -I/opt/local/lib/pypy/include -c twisted/runner/portmap.c -o build/temp.macosx-10.6-i386-2.7/twisted/runner/portmap.o twisted/runner/portmap.c:10:20: error: Python.h: No such file or directory twisted/runner/portmap.c:14: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘*’ token twisted/runner/portmap.c:31: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘*’ token twisted/runner/portmap.c:45: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘PortmapMethods’ twisted/runner/portmap.c: In function ‘initportmap’: twisted/runner/portmap.c:55: warning: implicit declaration of function ‘Py_InitModule’ twisted/runner/portmap.c:55: error: ‘PortmapMethods’ undeclared (first use in this function) twisted/runner/portmap.c:55: error: (Each undeclared identifier is reported only once twisted/runner/portmap.c:55: error: for each function it appears in.) I have PyPy 1.6 with cpyext. This patch seems to fix the C extensions errors: Index: twisted/topfiles/setup.py =================================================================== --- twisted/topfiles/setup.py (revision 32705) +++ twisted/topfiles/setup.py (working copy) @@ -37,10 +37,11 @@ condition=lambda _: _isCPython and sys.platform == "win32"), Extension("twisted.python._initgroups", - ["twisted/python/_initgroups.c"]), + ["twisted/python/_initgroups.c"], + condition=lambda _: _isCPython), Extension("twisted.internet._sigchld", ["twisted/internet/_sigchld.c"], - condition=lambda _: sys.platform != "win32"), + condition=lambda _: _isCPython and sys.platform != "win32"), ] # Figure out which plugins to include: all plugins except subproject ones Index: twisted/runner/topfiles/setup.py =================================================================== --- twisted/runner/topfiles/setup.py (revision 32705) +++ twisted/runner/topfiles/setup.py (working copy) @@ -1,6 +1,8 @@ # Copyright (c) Twisted Matrix Laboratories. # See LICENSE for details. +from twisted.python.dist import _isCPython + try: from twisted.python.dist import setup, ConditionalExtension as Extension except ImportError: @@ -11,7 +13,7 @@ extensions = [ Extension("twisted.runner.portmap", ["twisted/runner/portmap.c"], - condition=lambda builder: builder._check_header("rpc/rpc.h")), + condition=lambda builder: _isCPython and builder._check_header("rpc/rpc.h")), ] if __name__ == '__main__': however, setuptools trigger GC mis(?)behaviour in PyPy and setup fails: .... lots of logging snipped Extracting Twisted-11.0.0_r32705-py2.7-macosx-10.6-i386.egg to /opt/local/lib/pypy/site-packages error: /opt/local/lib/pypy/site-packages/Twisted-11.0.0_r32705-py2.7-macosx-10.6-i386.egg/twisted/enterprise/row.pyc: Too many open files This error is known to PyPy guys, they, however, insist that setuptools are wrong and not them. Which sounds totally wrong to me, but again, who am I to judge.
On 03:55 pm, jaroslaw.fedewicz@gmail.com wrote:
On Sep 30, 2011, at 16:55 , exarkun@twistedmatrix.com wrote:
On 01:18 pm, jacek99@gmail.com wrote:
And once I reach 0.1 with all of these features, I will post some blog entries about this together with PyPy benchmarks...so that people don't think Javascript / Node.js is your only option when it comes to async I/O web frameworks :-)
BTW, is there any progress on getting Twisted to install under PyPy in the near future? I know you guys have been looking into it....
http://twistedmatrix.com/trac/ticket/5158 is resolved now. If there are still problems with Twisted trunk@HEAD on PyPy, please let us know. :)
There are in fact some problems.
$ sudo pypy-c ./setup.py install [snip] twisted/runner/portmap.c:10:20: error: Python.h: No such file or directory [snip]
I have PyPy 1.6 with cpyext.
Notice the "Python.h" error message here. I think that some of the PyPy binary distributions are broken, missing all of the header files for cpyext. I filed a ticket about this for Windows in the PyPy issue tracker since I just observed the issue there. I didn't realize it affected other platforms as well. https://bugs.pypy.org/issue889
[snip]
Extracting Twisted-11.0.0_r32705-py2.7-macosx-10.6-i386.egg to /opt/local/lib/pypy/site-packages error: /opt/local/lib/pypy/site- packages/Twisted-11.0.0_r32705-py2.7-macosx-10.6-i386.egg/twisted/enterprise/row.pyc: Too many open files
This error is known to PyPy guys, they, however, insist that setuptools are wrong and not them. Which sounds totally wrong to me, but again, who am I to judge.
Another way to look at it is that they are not going to implement reference counting for file objects. So whether they are wrong or not, the problem has to be fixed somewhere else. Hopefully it can be fixed in the "distribute" project, which is supposed to be a drop-in replacement for setuptools, but with fewer bugs. Jean-Paul
Is there an option to separate the Twisted core (i.e. the part everyone runs on PyPy with the PYTHONPATH hack ) from the part with embedded C extensions? At least allow a trouble-free PyPy install for the core functionality? On Fri, Sep 30, 2011 at 12:25 PM, <exarkun@twistedmatrix.com> wrote:
On 03:55 pm, jaroslaw.fedewicz@gmail.com wrote:
On Sep 30, 2011, at 16:55 , exarkun@twistedmatrix.com wrote:
On 01:18 pm, jacek99@gmail.com wrote:
And once I reach 0.1 with all of these features, I will post some blog entries about this together with PyPy benchmarks...so that people don't think Javascript / Node.js is your only option when it comes to async I/O web frameworks :-)
BTW, is there any progress on getting Twisted to install under PyPy in the near future? I know you guys have been looking into it....
http://twistedmatrix.com/trac/ticket/5158 is resolved now. If there are still problems with Twisted trunk@HEAD on PyPy, please let us know. :)
There are in fact some problems.
$ sudo pypy-c ./setup.py install [snip] twisted/runner/portmap.c:10:20: error: Python.h: No such file or directory [snip]
I have PyPy 1.6 with cpyext.
Notice the "Python.h" error message here. I think that some of the PyPy binary distributions are broken, missing all of the header files for cpyext.
I filed a ticket about this for Windows in the PyPy issue tracker since I just observed the issue there. I didn't realize it affected other platforms as well.
https://bugs.pypy.org/issue889
[snip]
Extracting Twisted-11.0.0_r32705-py2.7-macosx-10.6-i386.egg to /opt/local/lib/pypy/site-packages error: /opt/local/lib/pypy/site-
packages/Twisted-11.0.0_r32705-py2.7-macosx-10.6-i386.egg/twisted/enterprise/row.pyc: Too many open files
This error is known to PyPy guys, they, however, insist that setuptools are wrong and not them. Which sounds totally wrong to me, but again, who am I to judge.
Another way to look at it is that they are not going to implement reference counting for file objects. So whether they are wrong or not, the problem has to be fixed somewhere else. Hopefully it can be fixed in the "distribute" project, which is supposed to be a drop-in replacement for setuptools, but with fewer bugs.
Jean-Paul
_______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
On 04:30 pm, jacek99@gmail.com wrote:
Is there an option to separate the Twisted core (i.e. the part everyone runs on PyPy with the PYTHONPATH hack ) from the part with embedded C extensions?
At least allow a trouble-free PyPy install for the core functionality?
There isn't now. Given what I know about distutils, it doesn't seem like an easy thing to implement. If anyone has ideas about how it could be done, we could discuss it (or feel free to just implement it ;). Jean-Paul
On Fri, Sep 30, 2011 at 12:25 PM, <exarkun@twistedmatrix.com> wrote:
On 03:55 pm, jaroslaw.fedewicz@gmail.com wrote:
On Sep 30, 2011, at 16:55 , exarkun@twistedmatrix.com wrote:
On 01:18 pm, jacek99@gmail.com wrote:
And once I reach 0.1 with all of these features, I will post some blog entries about this together with PyPy benchmarks...so that people don't think Javascript / Node.js is your only option when it comes to async I/O web frameworks :-)
BTW, is there any progress on getting Twisted to install under PyPy in the near future? I know you guys have been looking into it....
http://twistedmatrix.com/trac/ticket/5158 is resolved now. If there are still problems with Twisted trunk@HEAD on PyPy, please let us know.
:)
There are in fact some problems.
$ sudo pypy-c ./setup.py install [snip] twisted/runner/portmap.c:10:20: error: Python.h: No such file or directory [snip]
I have PyPy 1.6 with cpyext.
Notice the "Python.h" error message here. I think that some of the PyPy binary distributions are broken, missing all of the header files for cpyext.
I filed a ticket about this for Windows in the PyPy issue tracker since I just observed the issue there. I didn't realize it affected other platforms as well.
https://bugs.pypy.org/issue889
[snip]
Extracting Twisted-11.0.0_r32705-py2.7-macosx-10.6-i386.egg to /opt/local/lib/pypy/site-packages error: /opt/local/lib/pypy/site-
packages/Twisted-11.0.0_r32705-py2.7-macosx-10.6-i386.egg/twisted/enterprise/row.pyc: Too many open files
This error is known to PyPy guys, they, however, insist that setuptools are wrong and not them. Which sounds totally wrong to me, but again, who am I to judge.
Another way to look at it is that they are not going to implement reference counting for file objects. So whether they are wrong or not, the problem has to be fixed somewhere else. Hopefully it can be fixed in the "distribute" project, which is supposed to be a drop-in replacement for setuptools, but with fewer bugs.
Jean-Paul
_______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
could twisted itself be split into two separate packages? e.g. twisted and twisted-extensions, etc. I understand the frustration of dealing with pypy/distutils, etc. but from an end developer's perspective the hurdles one has to go through to get pypy+twisted working is a major roadblock when it comes to stealing some of the thunder Node.js is getting these days. Cheers, Jacek On Fri, Sep 30, 2011 at 12:52 PM, <exarkun@twistedmatrix.com> wrote:
On 04:30 pm, jacek99@gmail.com wrote:
Is there an option to separate the Twisted core (i.e. the part everyone runs on PyPy with the PYTHONPATH hack ) from the part with embedded C extensions?
At least allow a trouble-free PyPy install for the core functionality?
There isn't now. Given what I know about distutils, it doesn't seem like an easy thing to implement. If anyone has ideas about how it could be done, we could discuss it (or feel free to just implement it ;).
Jean-Paul
On Fri, Sep 30, 2011 at 12:25 PM, <exarkun@twistedmatrix.com> wrote:
On 03:55 pm, jaroslaw.fedewicz@gmail.com wrote:
On Sep 30, 2011, at 16:55 , exarkun@twistedmatrix.com wrote:
On 01:18 pm, jacek99@gmail.com wrote:
And once I reach 0.1 with all of these features, I will post some blog entries about this together with PyPy benchmarks...so that people don't think Javascript / Node.js is your only option when it comes to async I/O web frameworks :-)
BTW, is there any progress on getting Twisted to install under PyPy in the near future? I know you guys have been looking into it....
http://twistedmatrix.com/trac/ticket/5158 is resolved now. If there are still problems with Twisted trunk@HEAD on PyPy, please let us know.
:)
There are in fact some problems.
$ sudo pypy-c ./setup.py install [snip] twisted/runner/portmap.c:10:20: error: Python.h: No such file or directory [snip]
I have PyPy 1.6 with cpyext.
Notice the "Python.h" error message here. I think that some of the PyPy binary distributions are broken, missing all of the header files for cpyext.
I filed a ticket about this for Windows in the PyPy issue tracker since I just observed the issue there. I didn't realize it affected other platforms as well.
https://bugs.pypy.org/issue889
[snip]
Extracting Twisted-11.0.0_r32705-py2.7-macosx-10.6-i386.egg to /opt/local/lib/pypy/site-packages error: /opt/local/lib/pypy/site-
packages/Twisted-11.0.0_r32705-py2.7-macosx-10.6-i386.egg/twisted/enterprise/row.pyc:
Too many open files
This error is known to PyPy guys, they, however, insist that setuptools are wrong and not them. Which sounds totally wrong to me, but again, who am I to judge.
Another way to look at it is that they are not going to implement reference counting for file objects. So whether they are wrong or not, the problem has to be fixed somewhere else. Hopefully it can be fixed in the "distribute" project, which is supposed to be a drop-in replacement for setuptools, but with fewer bugs.
Jean-Paul
_______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
_______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
On 04:58 pm, jacek99@gmail.com wrote:
could twisted itself be split into two separate packages? e.g. twisted and twisted-extensions, etc.
I'm not sure what you're suggesting. All the potential solutions I can imagine have technical limitations that rule them out. "split into two separate packages" is too general for me to know what you have in mind. Apart from that, PyPy really just needs to fix their distribution so that cpyext actually works. Then Twisted will build on it as-is. Jean-Paul
I understand the frustration of dealing with pypy/distutils, etc. but from an end developer's perspective the hurdles one has to go through to get pypy+twisted working is a major roadblock when it comes to stealing some of the thunder Node.js is getting these days.
Cheers, Jacek
On Fri, Sep 30, 2011 at 12:52 PM, <exarkun@twistedmatrix.com> wrote:
On 04:30 pm, jacek99@gmail.com wrote:
Is there an option to separate the Twisted core (i.e. the part everyone runs on PyPy with the PYTHONPATH hack ) from the part with embedded C extensions?
At least allow a trouble-free PyPy install for the core functionality?
There isn't now. Given what I know about distutils, it doesn't seem like an easy thing to implement. If anyone has ideas about how it could be done, we could discuss it (or feel free to just implement it ;).
On Fri, Sep 30, 2011 at 12:25 PM, <exarkun@twistedmatrix.com> wrote:
On 03:55 pm, jaroslaw.fedewicz@gmail.com wrote:
On Sep 30, 2011, at 16:55 , exarkun@twistedmatrix.com wrote:
On 01:18 pm, jacek99@gmail.com wrote: >And once I reach 0.1 with all of these features, I will post
some
>blog >entries about this together with PyPy benchmarks...so that
>don't >think >Javascript / Node.js is your only option when it comes to async I/O >web >frameworks :-) > >BTW, is there any progress on getting Twisted to install under PyPy >in >the >near future? I know you guys have been looking into it....
http://twistedmatrix.com/trac/ticket/5158 is resolved now. If
Jean-Paul people there
are still problems with Twisted trunk@HEAD on PyPy, please let us know. :)
There are in fact some problems.
$ sudo pypy-c ./setup.py install [snip] twisted/runner/portmap.c:10:20: error: Python.h: No such file or directory [snip]
I have PyPy 1.6 with cpyext.
Notice the "Python.h" error message here. I think that some of the PyPy binary distributions are broken, missing all of the header files for cpyext.
I filed a ticket about this for Windows in the PyPy issue tracker since I just observed the issue there. I didn't realize it affected other platforms as well.
https://bugs.pypy.org/issue889
[snip]
Extracting Twisted-11.0.0_r32705-py2.7-macosx-10.6-i386.egg to /opt/local/lib/pypy/site-packages error: /opt/local/lib/pypy/site-
packages/Twisted-11.0.0_r32705-py2.7-macosx-10.6-i386.egg/twisted/enterprise/row.pyc:
Too many open files
This error is known to PyPy guys, they, however, insist that setuptools are wrong and not them. Which sounds totally wrong to me, but again, who am I to judge.
Another way to look at it is that they are not going to implement reference counting for file objects. So whether they are wrong or not, the problem has to be fixed somewhere else. Hopefully it can be fixed in the "distribute" project, which is supposed to be a drop-in replacement for setuptools, but with fewer bugs.
Jean-Paul
_______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
_______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
OK, I will try to revive the ticket I raised on the PyPy site, point it to this discussion and maybe get some feedback from them Thanks
On Sep 30, 2011, at 19:25 , exarkun@twistedmatrix.com wrote:
Notice the "Python.h" error message here. I think that some of the PyPy binary distributions are broken, missing all of the header files for cpyext.
I filed a ticket about this for Windows in the PyPy issue tracker since I just observed the issue there. I didn't realize it affected other platforms as well.
Ehrm. It's a PyPy from MacPorts. Which is in fact a source distribution, took me a whole day to build. I haven't tried to re-build PyPy's nightly, because I need the machine to work on, but can try to see if a hand-build will make the problem go away (and thus report it to the MacPorts guys).
On 09/30/11 14:11, Jarosław Fedewicz wrote:
I haven't tried to re-build PyPy's nightly, because I need the machine to work on, but can try to see if a hand-build will make the problem go away (and thus report it to the MacPorts guys).
Works for me, with Twisted trunk and PyPy built from the tip this morning. But that's on a Linux box, not a Mac with MacPorts, so I don't know if that helps you. -- David Ripton dripton@ripton.net
That's good news. Let's wait till PyPy 1.7 + Twisted 11.0.X (whatever the next release may be) to try it out. Thanks for everyone's work on this Jacek On Fri, Sep 30, 2011 at 5:41 PM, David Ripton <dripton@ripton.net> wrote:
On 09/30/11 14:11, Jarosław Fedewicz wrote:
I haven't tried to re-build PyPy's nightly, because I need the machine to work on, but can try to see if a hand-build will make the problem go away (and thus report it to the MacPorts guys).
Works for me, with Twisted trunk and PyPy built from the tip this morning. But that's on a Linux box, not a Mac with MacPorts, so I don't know if that helps you.
-- David Ripton dripton@ripton.net
_______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
On Sep 30, 2011, at 16:08 , Jacek Furmankiewicz wrote:
Once I am done with all of this I will have a serious look at AMPoule to get that out-of-the-box multi-core support.
So the next release 0.0.8 should finish all the content type stuff and then for 0.0.9 should start looking at AMPoule.
Can I request that looking at AMPoule doesn't result in it being a required dependency? AMPoule is just one implementation of multiprocessing + IPC, would be a pity to have to patch it heavily so that it works with other multiprocessing schemes (including homebrew ones) or with none at all.
Well, I plan to look at out-of-the-box support for: a) simple HTTP proxy/load balancer (instead of nginx) A simple runner that starts up "root" CorePost process and then starts up the real app on multiple local ports. Does simple load balancing between them. Not the most efficient (extra HTTP call), but simple to set up and *maybe* good enough for many apps b) Ampoule Mostly because the devs here recommended that approach. It seems a bit verbose (from the little docs I've seen), requires custom command classes, etc.). Not sure what the impact will be on ease-of-use and the API. It looks a bit daunting in terms of coding overhead. c) ZeroMQ (basically the Mongrel2 model) Single CorePost app posting messages to ZeroMQ, single receiver on each process picking them up and distributing to the appropriate router functions. Could be very fast, but requires external dependency (ZeroMQ). These are some of the idea I had for multicore support...any comments or suggestions for better approaches are welcome. Jacek 2011/10/1 Jarosław Fedewicz <jaroslaw.fedewicz@gmail.com>
On Sep 30, 2011, at 16:08 , Jacek Furmankiewicz wrote:
Once I am done with all of this I will have a serious look at AMPoule to get that out-of-the-box multi-core support.
So the next release 0.0.8 should finish all the content type stuff and then for 0.0.9 should start looking at AMPoule.
Can I request that looking at AMPoule doesn't result in it being a required dependency?
AMPoule is just one implementation of multiprocessing + IPC, would be a pity to have to patch it heavily so that it works with other multiprocessing schemes (including homebrew ones) or with none at all. _______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
On Oct 1, 2011, at 21:51 , Jacek Furmankiewicz wrote:
Well, I plan to look at out-of-the-box support for:
[snip] Jacek Again, there is such thing as concern separation. I would welcome all those as separate packages, but if CorePost made me do extra work to keep irrelevant bits out, it would be useless. It would be far more simple then for me to roll my own library which would just serve web requests and do no more than that.
Let me stress. A separate package that does web serving. A separate one for load balancing proxy. A separate one to work with AMPoule. A separate one to work with ZeroMQ. Not all-in-one blob which has all the shiny bits included there just in case which I'll never ever need. If I were you, I would even separate data formats support (XML/JSON/YAML/whatever else is there). Just provide an interface every data format adapter must implement. There might be people who would want ASN.1, like it or not... The reason is simple, your users might have their own libraries and solutions, there might be even actual reasons for them to have them, and it just doesn't work in the enterprise environment like “Hey, let's add a bunchload of these cool 3rd party libraries just in case”. They might have a load-balancing infrastructure of their own in place, Twisted or not. They might actually *prefer* nginx for some reason. They might have other libraries to do data formats handling (there's a big reason behind Jython's existence, namely, have all those funky Java libraries here and now). What they face would be supporting two different process management infrastuctures instead of one, or support two libraries which basically do one thing, instead of one library. This is going to be at least as twice as expensive, if not even more expensive if you ever try to glue those things together and get them to play nicely with one another.
Well, I see your point, but do not necessarily agree. As a mostly Java guy (that's what I do at work 90% of the time), I deal daily with the complexity of these dozens of tiny little libs that you can assemble...as a result my Maven pom.xml file is hundreds of lines long. There is something to be said about an "all batteries included" approach that handles 80% of the most common use cases out of the box. Handling the other 20% often entails so much more additional complexity, that maybe it's just better to leave it to a hand-rolled solution anyway. Django rolls a lot of things into one complete package...and that is the reason we are slowly moving all of our Java / .Net code to it (for web development). There is value to this level of integration, even if it doesn't do everything for everybody. Jacek 2011/10/1 Jarosław Fedewicz <jaroslaw.fedewicz@gmail.com>
On Oct 1, 2011, at 21:51 , Jacek Furmankiewicz wrote:
Well, I plan to look at out-of-the-box support for:
[snip] Jacek Again, there is such thing as concern separation. I would welcome all those as separate packages, but if CorePost made me do extra work to keep irrelevant bits out, it would be useless. It would be far more simple then for me to roll my own library which would just serve web requests and do no more than that.
Let me stress. A separate package that does web serving. A separate one for load balancing proxy. A separate one to work with AMPoule. A separate one to work with ZeroMQ. Not all-in-one blob which has all the shiny bits included there just in case which I'll never ever need.
If I were you, I would even separate data formats support (XML/JSON/YAML/whatever else is there). Just provide an interface every data format adapter must implement. There might be people who would want ASN.1, like it or not...
The reason is simple, your users might have their own libraries and solutions, there might be even actual reasons for them to have them, and it just doesn't work in the enterprise environment like "Hey, let's add a bunchload of these cool 3rd party libraries just in case". They might have a load-balancing infrastructure of their own in place, Twisted or not. They might actually *prefer* nginx for some reason. They might have other libraries to do data formats handling (there's a big reason behind Jython's existence, namely, have all those funky Java libraries here and now). What they face would be supporting two different process management infrastuctures instead of one, or support two libraries which basically do one thing, instead of one library. This is going to be at least as twice as expensive, if not even more expensive if you ever try to glue those things together and get them to play nicely with one another.
_______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
participants (5)
-
David Ripton -
exarkun＠twistedmatrix.com -
Glyph Lefkowitz -
Jacek Furmankiewicz -
Jarosław Fedewicz