[Twisted-Python] twisted.web with dynamic + static content
I'm sure I'm overlooking something obvious here but I just can't get my head around it. Here's the setup: twisted.web server that generates dynamic content. Child that serves up static content, e.g. css and favoicon. However, the static content isn't making it. Instead, any hit to localhost/static actually yields up a copy of / again. Here's the server code import sys from twisted.internet import reactor, endpoints from twisted.web import server from twisted.web.resource import Resource from twisted.web.static import File sys.path.append('lib') content = """ <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <link rel="stylesheet" href="/static/test.css" type="text/css" /> </head> <body> <span class='twistedTest'>This</span> is a test </body> </html> """ class tServer(Resource): isLeaf = True def render_GET(self, request): return bytes(content, "utf-8") if __name__ == "__main__": root = tServer() root.putChild(b"static", File("static")) site = server.Site(root) endpoint = endpoints.TCP4ServerEndpoint(reactor, 8080) endpoint.listen(site) reactor.run() print("Shutting down!") It's run with the command 'python tserver.py'. The expectation is that what is inside the custom <span> will be red. In the same dir as the script is a subdir 'static' with the css file inside it. If I replace 'root' with root = Resource() then / doesn't serve up anything, but /static is a directory listing of the static directory. The dynamic server is basically a copy of several tutorials cooked down to something that I could use to demonstrate the problem. What am I missing here? /headscratch Regards, Jeff
Try replacing: isLeaf = True with isLeaf = False On Tue, 30 Oct 2018 at 21:32, Jeff Grimmett <grimmtooth@gmail.com> wrote:
I'm sure I'm overlooking something obvious here but I just can't get my head around it.
Here's the setup: twisted.web server that generates dynamic content. Child that serves up static content, e.g. css and favoicon. However, the static content isn't making it. Instead, any hit to localhost/static actually yields up a copy of / again.
Here's the server code
import sys
from twisted.internet import reactor, endpoints from twisted.web import server from twisted.web.resource import Resource from twisted.web.static import File
sys.path.append('lib')
content = """ <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <link rel="stylesheet" href="/static/test.css" type="text/css" /> </head> <body> <span class='twistedTest'>This</span> is a test </body> </html> """
class tServer(Resource): isLeaf = True
def render_GET(self, request): return bytes(content, "utf-8")
if __name__ == "__main__": root = tServer() root.putChild(b"static", File("static"))
site = server.Site(root) endpoint = endpoints.TCP4ServerEndpoint(reactor, 8080) endpoint.listen(site)
reactor.run() print("Shutting down!")
It's run with the command 'python tserver.py'. The expectation is that what is inside the custom <span> will be red.
In the same dir as the script is a subdir 'static' with the css file inside it.
If I replace 'root' with root = Resource() then / doesn't serve up anything, but /static is a directory listing of the static directory.
The dynamic server is basically a copy of several tutorials cooked down to something that I could use to demonstrate the problem.
What am I missing here? /headscratch
Regards,
Jeff _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Tried that, I get a big No Such Resource No such child resource. back. Watching it in FF's development panel, I see a 404 come back for /. /static doesn't get served at all, of course. This, however, DID work. class tServer(Resource): isLeaf = False def getChild(self, path, request): print('You know what you doing.') if path == b'': print("Rendering /") return self return Resource.getChild(self, path, request) def render_GET(self, request): return bytes(content, "utf-8") (ignore my printf debugging plz) So, Thanks! :) Regards, Jeff On Tue, Oct 30, 2018 at 6:42 PM Donal McMullan <donal.mcmullan@gmail.com> wrote:
Try replacing: isLeaf = True with isLeaf = False
On Tue, 30 Oct 2018 at 21:32, Jeff Grimmett <grimmtooth@gmail.com> wrote:
I'm sure I'm overlooking something obvious here but I just can't get my head around it.
Here's the setup: twisted.web server that generates dynamic content. Child that serves up static content, e.g. css and favoicon. However, the static content isn't making it. Instead, any hit to localhost/static actually yields up a copy of / again.
Here's the server code
import sys
from twisted.internet import reactor, endpoints from twisted.web import server from twisted.web.resource import Resource from twisted.web.static import File
sys.path.append('lib')
content = """ <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <link rel="stylesheet" href="/static/test.css" type="text/css" /> </head> <body> <span class='twistedTest'>This</span> is a test </body> </html> """
class tServer(Resource): isLeaf = True
def render_GET(self, request): return bytes(content, "utf-8")
if __name__ == "__main__": root = tServer() root.putChild(b"static", File("static"))
site = server.Site(root) endpoint = endpoints.TCP4ServerEndpoint(reactor, 8080) endpoint.listen(site)
reactor.run() print("Shutting down!")
It's run with the command 'python tserver.py'. The expectation is that what is inside the custom <span> will be red.
In the same dir as the script is a subdir 'static' with the css file inside it.
If I replace 'root' with root = Resource() then / doesn't serve up anything, but /static is a directory listing of the static directory.
The dynamic server is basically a copy of several tutorials cooked down to something that I could use to demonstrate the problem.
What am I missing here? /headscratch
Regards,
Jeff _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Hi Jeff, Thanks for using Twisted. Here's a version with some small changes that works, and is self-contained. import sys from twisted.internet import reactor, endpoints from twisted.web import server from twisted.web.resource import Resource from twisted.web.static import Data sys.path.append('lib') content = """ <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <link rel="stylesheet" href="/static/test.css" type="text/css" /> </head> <body> <span class='twistedTest'>This</span> is a test </body> </html> """ class tServer(Resource): def render_GET(self, request): return bytes(content, "utf-8") if __name__ == "__main__": root = Resource() static_collection = Resource() static_collection.putChild(b"test.css", Data(b".twistedTest {color: red;}", "text/css")) root.putChild(b"static", static_collection) root.putChild(b"", tServer()) site = server.Site(root) endpoint = endpoints.TCP4ServerEndpoint(reactor, 8080) endpoint.listen(site) reactor.run() print("Shutting down!") The problem with your first version was 'isLeaf', as Donal suggested. However, the problem was not simply that the flag was set, but rather what the flag means, and why it works that way. The root resource in any web server is a collection. Which is to say, under normal circumstances, the root resource never has render_* invoked on it; you can't render it, because it's impossible, in the HTTP protocol, to spell a URL that doesn't start with "/". isLeaf changes this, and says "this resource is responsible for rendering all of its children; traversal stops here". That means that it starts invoking render_GET to render "/", but also to render every other path on the server, including (unfortunately for you) /static/test.css. The modified example above instead uses a Resource() as the collection, and inserts a '' child for the index, and a separate 'static' child for the static index. You can use a static.File for a directory here instead of a static resource, and anywhere you see putChild, you could also use a dynamic resource which overrides getChild to return the object rather than inserting it in advance. Of course, you might wonder what the point of 'isLeaf' is if it short circuits this stuff and makes it impossible to tell the difference between resources. Given that you have a directory, you want to use a static.File child resource and almost certainly don't want to set isLeaf; however, you might be wondering how one would even use isLeaf if it just cuts off the ability to tell the difference between resources. The documentation on this is not great - it doesn't even appear as an attribute in the API reference, just an oblique reference in the docstring for https://twistedmatrix.com/documents/current/api/twisted.web.resource.Resourc... <https://twistedmatrix.com/documents/current/api/twisted.web.resource.Resource.html#getChild>. But, the 'prepath' and 'postpath' attributes, lists of bytes, will tell you about where in the request traversal cycle you are, and allow you to distinguish which content to render directly within the body of render_*, rather than having to route to the right object using Twisted APIs. So here's a working version with isLeaf=True: import sys from twisted.internet import reactor, endpoints from twisted.web import server from twisted.web.resource import Resource content = """ <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <link rel="stylesheet" href="/static/test.css" type="text/css" /> </head> <body> <span class='twistedTest'>This</span> is a test </body> </html> """ css = """ .twistedTest { color: red; } """ class tServer(Resource): isLeaf = True def render_GET(self, request): if request.postpath == [b'']: request.setHeader("content-type", "text/html") return bytes(content, "utf-8") elif request.postpath == [b'static', b'test.css']: request.setHeader("content-type", "text/css") return bytes(css, 'utf-8') else: request.setResponseCode(404) return b'not found' if __name__ == "__main__": site = server.Site(tServer()) endpoint = endpoints.TCP4ServerEndpoint(reactor, 8080) endpoint.listen(site) reactor.run() print("Shutting down!") I hope this clears up the request traversal model a little bit. -glyph
On Oct 31, 2018, at 2:15 PM, Jeff Grimmett <grimmtooth@gmail.com> wrote:
Tried that, I get a big
No Such Resource
No such child resource.
back. Watching it in FF's development panel, I see a 404 come back for /. /static doesn't get served at all, of course.
This, however, DID work.
class tServer(Resource): isLeaf = False
def getChild(self, path, request): print('You know what you doing.')
if path == b'': print("Rendering /") return self
return Resource.getChild(self, path, request)
def render_GET(self, request): return bytes(content, "utf-8")
(ignore my printf debugging plz)
So, Thanks! :)
Regards,
Jeff
On Tue, Oct 30, 2018 at 6:42 PM Donal McMullan <donal.mcmullan@gmail.com <mailto:donal.mcmullan@gmail.com>> wrote: Try replacing: isLeaf = True with isLeaf = False
On Tue, 30 Oct 2018 at 21:32, Jeff Grimmett <grimmtooth@gmail.com <mailto:grimmtooth@gmail.com>> wrote: I'm sure I'm overlooking something obvious here but I just can't get my head around it.
Here's the setup: twisted.web server that generates dynamic content. Child that serves up static content, e.g. css and favoicon. However, the static content isn't making it. Instead, any hit to localhost/static actually yields up a copy of / again.
Here's the server code
import sys
from twisted.internet import reactor, endpoints from twisted.web import server from twisted.web.resource import Resource from twisted.web.static import File
sys.path.append('lib')
content = """ <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <link rel="stylesheet" href="/static/test.css" type="text/css" /> </head> <body> <span class='twistedTest'>This</span> is a test </body> </html> """
class tServer(Resource): isLeaf = True
def render_GET(self, request): return bytes(content, "utf-8")
if __name__ == "__main__": root = tServer() root.putChild(b"static", File("static"))
site = server.Site(root) endpoint = endpoints.TCP4ServerEndpoint(reactor, 8080) endpoint.listen(site)
reactor.run() print("Shutting down!")
It's run with the command 'python tserver.py'. The expectation is that what is inside the custom <span> will be red.
In the same dir as the script is a subdir 'static' with the css file inside it.
If I replace 'root' with root = Resource() then / doesn't serve up anything, but /static is a directory listing of the static directory.
The dynamic server is basically a copy of several tutorials cooked down to something that I could use to demonstrate the problem.
What am I missing here? /headscratch
Regards,
Jeff _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com <mailto:Twisted-Python@twistedmatrix.com> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python <https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python> _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com <mailto:Twisted-Python@twistedmatrix.com> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python <https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python> _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Glyph, Belated (I was in the thick of a few unrelated crises) thanks - it helps to understand a little bit of the background on the way it works. I have made adjustments and everything works the way I was hoping for now. I wish I understood it a little better, if for no other reason than to contribute some documentation tweaks. Just how would that work, anyway? The contribution model for Twisted is a little opaque from where I am sitting. I'd love to help in areas where I am actually capable of doing :) Regards, Jeff On Thu, Nov 1, 2018 at 3:44 AM Glyph <glyph@twistedmatrix.com> wrote:
Hi Jeff,
Thanks for using Twisted.
Here's a version with some small changes that works, and is self-contained.
import sys
from twisted.internet import reactor, endpoints from twisted.web import server from twisted.web.resource import Resource from twisted.web.static import Data
sys.path.append('lib')
content = """ <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <link rel="stylesheet" href="/static/test.css" type="text/css" /> </head> <body> <span class='twistedTest'>This</span> is a test </body> </html> """
class tServer(Resource): def render_GET(self, request): return bytes(content, "utf-8")
if __name__ == "__main__": root = Resource() static_collection = Resource() static_collection.putChild(b"test.css", Data(b".twistedTest {color: red;}", "text/css")) root.putChild(b"static", static_collection) root.putChild(b"", tServer())
site = server.Site(root) endpoint = endpoints.TCP4ServerEndpoint(reactor, 8080) endpoint.listen(site)
reactor.run() print("Shutting down!")
The problem with your first version was 'isLeaf', as Donal suggested. However, the problem was not simply that the flag was set, but rather what the flag means, and why it works that way.
The root resource in any web server is a *collection*. Which is to say, under normal circumstances, the root resource never has render_* invoked on it; you can't render it, because it's impossible, in the HTTP protocol, to spell a URL that doesn't start with "/".
isLeaf changes this, and says "this resource is responsible for rendering all of its children; traversal stops here". That means that it starts invoking render_GET to render "/", but also to render *every other path on the server*, including (unfortunately for you) /static/test.css.
The modified example above instead uses a Resource() as the collection, and inserts a '' child for the index, and a separate 'static' child for the static index. You can use a static.File for a directory here instead of a static resource, and anywhere you see putChild, you could also use a dynamic resource which overrides getChild to return the object rather than inserting it in advance.
Of course, you might wonder what the point of 'isLeaf' is if it short circuits this stuff and makes it impossible to tell the difference between resources.
Given that you have a directory, you want to use a static.File child resource and almost certainly *don't* want to set isLeaf; however, you might be wondering how one would even use isLeaf if it just cuts off the ability to tell the difference between resources. The documentation on this is not great - it doesn't even appear as an attribute in the API reference, just an oblique reference in the docstring for https://twistedmatrix.com/documents/current/api/twisted.web.resource.Resourc.... But, the 'prepath' and 'postpath' attributes, lists of bytes, will tell you about where in the request traversal cycle you are, and allow you to distinguish which content to render directly within the body of render_*, rather than having to route to the right object using Twisted APIs. So here's a working version with isLeaf=True:
import sys
from twisted.internet import reactor, endpoints from twisted.web import server from twisted.web.resource import Resource
content = """ <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <link rel="stylesheet" href="/static/test.css" type="text/css" /> </head> <body> <span class='twistedTest'>This</span> is a test </body> </html> """
css = """ .twistedTest { color: red; } """
class tServer(Resource): isLeaf = True def render_GET(self, request): if request.postpath == [b'']: request.setHeader("content-type", "text/html") return bytes(content, "utf-8") elif request.postpath == [b'static', b'test.css']: request.setHeader("content-type", "text/css") return bytes(css, 'utf-8') else: request.setResponseCode(404) return b'not found'
if __name__ == "__main__": site = server.Site(tServer()) endpoint = endpoints.TCP4ServerEndpoint(reactor, 8080) endpoint.listen(site)
reactor.run() print("Shutting down!")
I hope this clears up the request traversal model a little bit.
-glyph
On Oct 31, 2018, at 2:15 PM, Jeff Grimmett <grimmtooth@gmail.com> wrote:
Tried that, I get a big
No Such Resource
No such child resource. back. Watching it in FF's development panel, I see a 404 come back for /. /static doesn't get served at all, of course.
This, however, DID work.
class tServer(Resource): isLeaf = False
def getChild(self, path, request): print('You know what you doing.')
if path == b'': print("Rendering /") return self
return Resource.getChild(self, path, request)
def render_GET(self, request): return bytes(content, "utf-8")
(ignore my printf debugging plz)
So, Thanks! :)
Regards,
Jeff
On Tue, Oct 30, 2018 at 6:42 PM Donal McMullan <donal.mcmullan@gmail.com> wrote:
Try replacing: isLeaf = True with isLeaf = False
On Tue, 30 Oct 2018 at 21:32, Jeff Grimmett <grimmtooth@gmail.com> wrote:
I'm sure I'm overlooking something obvious here but I just can't get my head around it.
Here's the setup: twisted.web server that generates dynamic content. Child that serves up static content, e.g. css and favoicon. However, the static content isn't making it. Instead, any hit to localhost/static actually yields up a copy of / again.
Here's the server code
import sys
from twisted.internet import reactor, endpoints from twisted.web import server from twisted.web.resource import Resource from twisted.web.static import File
sys.path.append('lib')
content = """ <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <link rel="stylesheet" href="/static/test.css" type="text/css" /> </head> <body> <span class='twistedTest'>This</span> is a test </body> </html> """
class tServer(Resource): isLeaf = True
def render_GET(self, request): return bytes(content, "utf-8")
if __name__ == "__main__": root = tServer() root.putChild(b"static", File("static"))
site = server.Site(root) endpoint = endpoints.TCP4ServerEndpoint(reactor, 8080) endpoint.listen(site)
reactor.run() print("Shutting down!")
It's run with the command 'python tserver.py'. The expectation is that what is inside the custom <span> will be red.
In the same dir as the script is a subdir 'static' with the css file inside it.
If I replace 'root' with root = Resource() then / doesn't serve up anything, but /static is a directory listing of the static directory.
The dynamic server is basically a copy of several tutorials cooked down to something that I could use to demonstrate the problem.
What am I missing here? /headscratch
Regards,
Jeff _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Mon, Nov 19, 2018, at 3:06 PM, Jeff Grimmett wrote:
Glyph,
Belated (I was in the thick of a few unrelated crises) thanks - it helps to understand a little bit of the background on the way it works. I have made adjustments and everything works the way I was hoping for now.> I wish I understood it a little better, if for no other reason than to contribute some documentation tweaks. Just how would that work, anyway? The contribution model for Twisted is a little opaque from where I am sitting. I'd love to help in areas where I am actually capable of doing :)> Regards,
Jeff
Hi Jeff, The Twisted development process is documented here on the wiki[1]. It boils down to: 1. Create a ticket[2] (on Trac) 2. Create a pull request (on GitHub) 3. Wait for a review (and address any feedback the reviewer gives) There is more on the wiki. The same process applies to code and documentation. Documentation changes are really helpful, especially because it is difficult for the experienced to see the rough edges that catch new users. Your confusion is a bug, and particularly with respect to the Resource model, a not uncommon one. Let's fix it! ---Tom Links: 1. https://twistedmatrix.com/trac/wiki/TwistedDevelopment#GettingYourPatchAccep... 2. https://twistedmatrix.com/trac/newticket
Thanks, Tom. Is there a limit to the smallness of the proposed changes? For example, if I had a possible two-line change to https://twistedmatrix.com/documents/current/api/twisted.internet.wxreactor.h... (which looks like it would be done in the docstring), is it okay to just have a PR for that or does it need to be bundled with more changes? Thanks, Jeff On Wed, Nov 21, 2018 at 1:31 AM Tom Most <twm@freecog.net> wrote:
On Mon, Nov 19, 2018, at 3:06 PM, Jeff Grimmett wrote:
Glyph,
Belated (I was in the thick of a few unrelated crises) thanks - it helps to understand a little bit of the background on the way it works. I have made adjustments and everything works the way I was hoping for now.
I wish I understood it a little better, if for no other reason than to contribute some documentation tweaks. Just how would that work, anyway? The contribution model for Twisted is a little opaque from where I am sitting. I'd love to help in areas where I am actually capable of doing :)
Regards,
Jeff
Hi Jeff,
The Twisted development process is documented here on the wiki <https://twistedmatrix.com/trac/wiki/TwistedDevelopment#GettingYourPatchAccepted>. It boils down to:
1. Create a ticket <https://twistedmatrix.com/trac/newticket> (on Trac) 2. Create a pull request (on GitHub) 3. Wait for a review (and address any feedback the reviewer gives)
There is more on the wiki. The same process applies to code and documentation.
Documentation changes are really helpful, especially because it is difficult for the experienced to see the rough edges that catch new users. Your confusion is a bug, and particularly with respect to the Resource model, a not uncommon one. Let's fix it!
---Tom
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Nov 27, 2018, at 2:37 PM, Jeff Grimmett <grimmtooth@gmail.com> wrote:
Thanks, Tom.
Is there a limit to the smallness of the proposed changes? For example, if I had a possible two-line change to https://twistedmatrix.com/documents/current/api/twisted.internet.wxreactor.h... <https://twistedmatrix.com/documents/current/api/twisted.internet.wxreactor.html> (which looks like it would be done in the docstring), is it okay to just have a PR for that or does it need to be bundled with more changes?
Thanks,
Jeff
Smaller changes are always better! Sometimes changes need to be larger and don't make sense when split up, so it's OK to propose large things, but a one-liner is the easiest possible thing to review, and will generally get integrated faster. -g
participants (4)
-
Donal McMullan -
Glyph -
Jeff Grimmett -
Tom Most