[Twisted-Python] Reloading of the same `.rpy' file
Hi, Twisted people. Playing with resource files, I noticed that the outer-level code within `.rpy' files is executed again every time the page using them is fetched by the browser. If `test.py' contains: ----------------------------------------------------------------------> print "Hello!" ... resource = ... ----------------------------------------------------------------------< than each `Alt-R' within Netscape, once `http://localhost:8080/test.rpy' has been loaded, will have "Hello!" added to the Twisted log file. I would have expected the resource file to be loaded once, and reused after without being wholly re-initialised each time. I tried "import remanent" within the resource file, and checked that "remanent" is loaded and intialised only once (moreover, it gets compiled!), so I'm tempted to move all code from `test.rpy' into `remanent.py' and reduce `test.rpy' to a mere: ----------------------------------------------------------------------> import remanent resource = remanent.resource ----------------------------------------------------------------------< which is a bit ridiculous if it has to be a standard idiom. On the other hand, for debugging, I agree it is a bit handsome being able to modify `test.rpy' and trying the new version without restarting the server. This is a weak justification however. Whenever the `.tap' file needs to be rebuilt, the server needs to be restarted anyway. This soon warrants a Makefile, and adding another dependency over `test.rpy' is a simple matter. Maybe the Twisted design is trying, here, to spare memory by not keeping `.rpy' code in its memory, in case these would be very numerous or contain a lot of code? Or trying to protect itself against leaks from such, but how? I have the vague feeling that I'm missing something important in this area. Would some soul be kind enough to accept enlightening mine? :-) -- François Pinard http://www.iro.umontreal.ca/~pinard
[Philippe Lafoucrière]
The solution is often to put a reverse proxy before twisted.web.
I'm at lost here. What is a reverse proxy in the context of Twisted? (Could you give me some pointers?) And how does a reverse proxy (whatever it is) change the reloading behaviour of Twisted for `.rpy' files? -- François Pinard http://www.iro.umontreal.ca/~pinard
On 29 Apr 2003 14:07:25 -0400 pinard@iro.umontreal.ca (François Pinard) wrote:
The solution is often to put a reverse proxy before twisted.web.
Nope.
It doesn't affect .rpy's in any way whatsoever. RPYs are designed to be reloaded every time. If you don't want this behaviour, don't use RPYs - rather, create your Resource tree in memory, rather than out of the filesystem. -- Itamar Shtull-Trauring http://itamarst.org/ http://www.zoteca.com -- Python & Twisted consulting
[Itamar Shtull-Trauring]
Hello, Itamar. Thanks for replying. I'm not trying to challenge the design, I'm much too new to Twisted. :-) I'm just not understanding it. What are the reasons for reloading `.rpy' files every time? There ought to be some advantages, but I'm not seeing them. -- François Pinard http://www.iro.umontreal.ca/~pinard
On 29 Apr 2003, pinard@iro.umontreal.ca (=?iso-8859-1?q?Fran=E7ois_Pinard?=) wrote:
which is a bit ridiculous if it has to be a standard idiom.
The standard idiom is to use the Twisted Registry. Look at http://twistedmatrix.com/documents/howto/using-twistedweb.xhtml -- Moshe Zadka -- http://moshez.org/ Buffy: I don't like you hanging out with someone that... short. Riley: Yeah, a lot of young people nowadays are experimenting with shortness. Agile Programming Language -- http://www.python.org/
[Moshe Zadka]
The standard idiom is to use the Twisted Registry. Look at http://twistedmatrix.com/documents/howto/using-twistedweb.xhtml
Thanks for the pointer, yet I'm not ready to understand it yet. It seems I should know `Componentized' first. Later, hopefully. I wonder what is the extent and scope of a `registry'. Exactly one per `.rpy' file presumably? Does this explain why, or why not, `.rpy' files are reloaded many times? -- François Pinard http://www.iro.umontreal.ca/~pinard
On 29 Apr 2003, pinard@iro.umontreal.ca (=?iso-8859-1?q?Fran=E7ois_Pinard?=) wrote:
Does this explain why, or why not, `.rpy' files are reloaded many times?
No, it merely talks about how to persistize objects...such as your resource. See the example :) -- Moshe Zadka -- http://moshez.org/ Buffy: I don't like you hanging out with someone that... short. Riley: Yeah, a lot of young people nowadays are experimenting with shortness. Agile Programming Language -- http://www.python.org/
On 2003.04.29 14:05, François Pinard wrote:
In itself, it doesn't, but that caching mechanism I mentioned in my last email uses it (I just went and read the source in twisted/web/script.py) And the scope of the registry is one per top-level static.File instance; i.e., if you have a web server set up with `mktap web --path foo', then there will only be one Registry unless you create some static.Files explicitly elsewhere. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://twistedmatrix.com/users/radix.twistd/
On 2003.04.29 10:41, François Pinard wrote:
Why is it ridiculous? The semantics for .rpy are good because they're flexible; you can either recreate your Resource object every time, or you can cache it. Also, mind that .rpys are meant to be as tiny as possible; they're just supposed to be glue code between your web app code and the web server.
Nope, just trying to remain flexible. It gives you a choice between CPU usage (recreating your Resource every time) and memory usage (keeping your Resources around in memory between requests).
Also, IIRC there's some sort of caching mechanism that glyph implemented for rpys, but I don't remember how it works. IIRC, you just add "cache()" to the top of your .rpy, or something. Unfortunately, it's not documented as it is. Can someone comment on this? I'll add some docstrings to ResourceScript if someone explains it. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://twistedmatrix.com/users/radix.twistd/
[Christopher Armstrong]
Why is it ridiculous?
Sorry if the word offended anyone, it surely was not my intent to do so. I only meant that if there is really no advantage at reloading an RPY file, then there is an advantage at loading it only once (unless such RPY files are so numerous or so big that it would create a memory problem, but it might likely not be the case in practice), which is avoiding spurious or useless re-initialisation. Then, the proper way would be to use a two-liner for each such file, and having such small bootstraps is questionable.
If a resource was consuming a lot of memory, I would understand. But as you just explained, RPY files are meant to be tiny, and then unlikely to build tons of fresh bytes in Python structures. Yet, within a small RPY file, one might import from library big modules having Resource based classes, and resources are meant to be freed immediately after having been consumed. For example, such a library resource might define some big triple-quoted static string holding the template of an HTML page. But then, the module will stay imported after the RPY file disappears, the big string will not be freed, and reloading will not trigger significant memory savings. If the template string gets read from disk each time, it will likely be kept in a local variable in the `render' method, and will be freed as soon as we exit that method, so reloading does not in itself trigger memory savings in that case either. So, just to help me understand, could you illustrate by an example, a likely case demonstrating the flexibility gained by multiple reloading?
Also, IIRC there's some sort of caching mechanism that glyph implemented for rpys, but I don't remember how it works.
It might be those Registry instances which Moshe referred to in a recent message, and which are sketched a bit quickly in the book, page 23, (2.3.2 "Using Twisted.Web", section "Resource Scripts"). -- François Pinard http://www.iro.umontreal.ca/~pinard
[Philippe Lafoucrière]
The solution is often to put a reverse proxy before twisted.web.
I'm at lost here. What is a reverse proxy in the context of Twisted? (Could you give me some pointers?) And how does a reverse proxy (whatever it is) change the reloading behaviour of Twisted for `.rpy' files? -- François Pinard http://www.iro.umontreal.ca/~pinard
On 29 Apr 2003 14:07:25 -0400 pinard@iro.umontreal.ca (François Pinard) wrote:
The solution is often to put a reverse proxy before twisted.web.
Nope.
It doesn't affect .rpy's in any way whatsoever. RPYs are designed to be reloaded every time. If you don't want this behaviour, don't use RPYs - rather, create your Resource tree in memory, rather than out of the filesystem. -- Itamar Shtull-Trauring http://itamarst.org/ http://www.zoteca.com -- Python & Twisted consulting
[Itamar Shtull-Trauring]
Hello, Itamar. Thanks for replying. I'm not trying to challenge the design, I'm much too new to Twisted. :-) I'm just not understanding it. What are the reasons for reloading `.rpy' files every time? There ought to be some advantages, but I'm not seeing them. -- François Pinard http://www.iro.umontreal.ca/~pinard
On 29 Apr 2003, pinard@iro.umontreal.ca (=?iso-8859-1?q?Fran=E7ois_Pinard?=) wrote:
which is a bit ridiculous if it has to be a standard idiom.
The standard idiom is to use the Twisted Registry. Look at http://twistedmatrix.com/documents/howto/using-twistedweb.xhtml -- Moshe Zadka -- http://moshez.org/ Buffy: I don't like you hanging out with someone that... short. Riley: Yeah, a lot of young people nowadays are experimenting with shortness. Agile Programming Language -- http://www.python.org/
[Moshe Zadka]
The standard idiom is to use the Twisted Registry. Look at http://twistedmatrix.com/documents/howto/using-twistedweb.xhtml
Thanks for the pointer, yet I'm not ready to understand it yet. It seems I should know `Componentized' first. Later, hopefully. I wonder what is the extent and scope of a `registry'. Exactly one per `.rpy' file presumably? Does this explain why, or why not, `.rpy' files are reloaded many times? -- François Pinard http://www.iro.umontreal.ca/~pinard
On 29 Apr 2003, pinard@iro.umontreal.ca (=?iso-8859-1?q?Fran=E7ois_Pinard?=) wrote:
Does this explain why, or why not, `.rpy' files are reloaded many times?
No, it merely talks about how to persistize objects...such as your resource. See the example :) -- Moshe Zadka -- http://moshez.org/ Buffy: I don't like you hanging out with someone that... short. Riley: Yeah, a lot of young people nowadays are experimenting with shortness. Agile Programming Language -- http://www.python.org/
On 2003.04.29 14:05, François Pinard wrote:
In itself, it doesn't, but that caching mechanism I mentioned in my last email uses it (I just went and read the source in twisted/web/script.py) And the scope of the registry is one per top-level static.File instance; i.e., if you have a web server set up with `mktap web --path foo', then there will only be one Registry unless you create some static.Files explicitly elsewhere. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://twistedmatrix.com/users/radix.twistd/
On 2003.04.29 10:41, François Pinard wrote:
Why is it ridiculous? The semantics for .rpy are good because they're flexible; you can either recreate your Resource object every time, or you can cache it. Also, mind that .rpys are meant to be as tiny as possible; they're just supposed to be glue code between your web app code and the web server.
Nope, just trying to remain flexible. It gives you a choice between CPU usage (recreating your Resource every time) and memory usage (keeping your Resources around in memory between requests).
Also, IIRC there's some sort of caching mechanism that glyph implemented for rpys, but I don't remember how it works. IIRC, you just add "cache()" to the top of your .rpy, or something. Unfortunately, it's not documented as it is. Can someone comment on this? I'll add some docstrings to ResourceScript if someone explains it. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://twistedmatrix.com/users/radix.twistd/
[Christopher Armstrong]
Why is it ridiculous?
Sorry if the word offended anyone, it surely was not my intent to do so. I only meant that if there is really no advantage at reloading an RPY file, then there is an advantage at loading it only once (unless such RPY files are so numerous or so big that it would create a memory problem, but it might likely not be the case in practice), which is avoiding spurious or useless re-initialisation. Then, the proper way would be to use a two-liner for each such file, and having such small bootstraps is questionable.
If a resource was consuming a lot of memory, I would understand. But as you just explained, RPY files are meant to be tiny, and then unlikely to build tons of fresh bytes in Python structures. Yet, within a small RPY file, one might import from library big modules having Resource based classes, and resources are meant to be freed immediately after having been consumed. For example, such a library resource might define some big triple-quoted static string holding the template of an HTML page. But then, the module will stay imported after the RPY file disappears, the big string will not be freed, and reloading will not trigger significant memory savings. If the template string gets read from disk each time, it will likely be kept in a local variable in the `render' method, and will be freed as soon as we exit that method, so reloading does not in itself trigger memory savings in that case either. So, just to help me understand, could you illustrate by an example, a likely case demonstrating the flexibility gained by multiple reloading?
Also, IIRC there's some sort of caching mechanism that glyph implemented for rpys, but I don't remember how it works.
It might be those Registry instances which Moshe referred to in a recent message, and which are sketched a bit quickly in the book, page 23, (2.3.2 "Using Twisted.Web", section "Resource Scripts"). -- François Pinard http://www.iro.umontreal.ca/~pinard
participants (5)
-
Christopher Armstrong -
Itamar Shtull-Trauring -
Moshe Zadka -
Philippe Lafoucrière -
pinard＠iro.umontreal.ca