[Twisted-Python] Persisted protocol?

Hi again,
From http://twistedmatrix.com/documents/current/core/howto/udp.html
"""As you can see, the protocol is registered with the reactor. This means it may be persisted if it's added to an application...""" What does 'persisted` here mean? -- anatoly t.

On 2011-04-06, anatoly techtonik wrote:
From http://twistedmatrix.com/documents/current/core/howto/udp.html
"""As you can see, the protocol is registered with the reactor. This means it may be persisted if it's added to an application..."""
What does 'persisted` here mean?
It sounds like this is a rememant of when the reactor shutdown it used to serialise its state so it could be restarted later. -- Regards, Stephen Thorne Development Engineer Netbox Blue

On Wed, Apr 06, 2011 at 10:16:03AM +0300, anatoly techtonik wrote:
From http://twistedmatrix.com/documents/current/core/howto/udp.html
"""As you can see, the protocol is registered with the reactor. This means it may be persisted if it's added to an application..."""
What does 'persisted` here mean?
At some point in the past, somebody decided it would be a great idea to have Twisted's "twistd" start applications from Python pickles; you could start up a server, configure it interactively, and when you shut it down the entire server state would be persisted in a pickle-file; when you started it up it would resume exactly as it had been when you shut it down. The idea seems to have fallen from grace, possibly because restarting a process is often the simplest way to resolve a production problem. These days, new Twisted users are encouraged to use Python ".tac" files rather than ".tap" pickles, and I expect that sentence is just a relic of that bygone era.

On Wed, Apr 6, 2011 at 9:30 AM, Tim Allen <screwtape@froup.com> wrote:
On Wed, Apr 06, 2011 at 10:16:03AM +0300, anatoly techtonik wrote:
From http://twistedmatrix.com/documents/current/core/howto/udp.html
"""As you can see, the protocol is registered with the reactor. This means it may be persisted if it's added to an application..."""
What does 'persisted` here mean?
At some point in the past, somebody decided it would be a great idea to have Twisted's "twistd" start applications from Python pickles; you could start up a server, configure it interactively, and when you shut it down the entire server state would be persisted in a pickle-file; when you started it up it would resume exactly as it had been when you shut it down.
Whoa hang on. Without trying to hijack the thread, this is the entire premise of infobarb, the IRC bot I'm building for #python-*, except s/pickle/sqlite/, so if this is a horrible idea I'd like to know before I build it.
The idea seems to have fallen from grace, possibly because restarting a process is often the simplest way to resolve a production problem. These days, new Twisted users are encouraged to use Python ".tac" files rather than ".tap" pickles, and I expect that sentence is just a relic of that bygone era.
I suppose the problem you're suggesting is that the pickle is in a state where running the server is impossible: do you mean like the pickle is corrupted, or pickle has a bunch of settings that just can't work... ? Would this problem be fixed by implementing features like differential snapshots? Maybe a replayable or reverse-replayable event log? cheers lvh

On Wed, Apr 06, 2011 at 12:41:08PM +0200, Laurens Van Houtven wrote:
Whoa hang on. Without trying to hijack the thread, this is the entire premise of infobarb, the IRC bot I'm building for #python-*, except s/pickle/sqlite/, so if this is a horrible idea I'd like to know before I build it.
I'm not familiar with the reasons Twistd moved from .tap to .tac (it's rather before my time), but as a user I'd be worried about things like "what happens if, due to a bug, the server winds up choking on some unexpected input, or deadlocked, or just mis-files some piece of state?" Restarting the process in question is a sledgehammer approach, but that's often what you want if the alternative is unscheduled downtime. The difference between Pickle and SQLite is that a SQLite database has probably had some thought put into its schema, and is much less likely to accidentally scoop up random other objects by reference. Also, if your database *does* pick up some unwanted state, you have the option of tinkering with the database manually; something that's much more difficult to do with pickles. I'd be interested in hearing from Twisted greybeards why .tap was deprecated, but I don't think your IRC bot has a fatal design flaw.

On Wed, Apr 6, 2011 at 1:27 PM, Tim Allen <screwtape@froup.com> wrote:
On Wed, Apr 06, 2011 at 12:41:08PM +0200, Laurens Van Houtven wrote:
Whoa hang on. Without trying to hijack the thread, this is the entire premise of infobarb, the IRC bot I'm building for #python-*, except s/pickle/sqlite/, so if this is a horrible idea I'd like to know before I build it.
I'm not familiar with the reasons Twistd moved from .tap to .tac (it's rather before my time), but as a user I'd be worried about things like "what happens if, due to a bug, the server winds up choking on some unexpected input, or deadlocked, or just mis-files some piece of state?" Restarting the process in question is a sledgehammer approach, but that's often what you want if the alternative is unscheduled downtime.
Right -- I don't think this applies to my bot since if it does the argument can also be extended to "don't have any persistence at all, state is bad" ;-) (It is, but that's not the most pragmatic of answers.) The difference between Pickle and SQLite is that a SQLite database has
probably had some thought put into its schema, and is much less likely to accidentally scoop up random other objects by reference. Also, if your database *does* pick up some unwanted state, you have the option of tinkering with the database manually; something that's much more difficult to do with pickles.
Right -- I'm giving plugins to the bot a SQLite database, it's up to them if they use it. All state serialization is explicit. I'm hoping it's going to be more sensible as a result. I'd be interested in hearing from Twisted greybeards why .tap was
deprecated, but I don't think your IRC bot has a fatal design flaw.
Whew :) Thanks for confirming anyway :) -- cheers lvh

On Wed, Apr 6, 2011 at 1:32 PM, Laurens Van Houtven <_@lvh.cc> wrote:
On Wed, Apr 6, 2011 at 1:27 PM, Tim Allen <screwtape@froup.com> wrote:
The difference between Pickle and SQLite is that a SQLite database has probably had some thought put into its schema, and is much less likely to accidentally scoop up random other objects by reference. Also, if your database *does* pick up some unwanted state, you have the option of tinkering with the database manually; something that's much more difficult to do with pickles.
Right -- I'm giving plugins to the bot a SQLite database, it's up to them if they use it. All state serialization is explicit. I'm hoping it's going to be more sensible as a result.
For what it's worth, you might find this interesting: https://launchpad.net/eridanus It's an IRC bot implemented on Mantissa, which uses Axiom databases (SQLite under the hood) in order to store and persist service configuration as well as user data. -- mithrandi, i Ainil en-Balandor, a faer Ambar

Urgh, thanks gmail -- that was supposed to be to mithrandi, not to the entire mailing list. My apologies. cheers lvh

On 07:16 am, techtonik@gmail.com wrote:
Hi again,
From http://twistedmatrix.com/documents/current/core/howto/udp.html
"""As you can see, the protocol is registered with the reactor. This means it may be persisted if it's added to an application..."""
What does 'persisted` here mean?
As others have pointed out, this is referring to an old feature which is being phased out. It would be great if you could file a ticket for cleaning up the docs; they mostly shouldn't talk about application persistence (certainly not in the UDP section). Jean-Paul

On Wed, Apr 6, 2011 at 3:58 PM, <exarkun@twistedmatrix.com> wrote:
As others have pointed out, this is referring to an old feature which is being phased out. It would be great if you could file a ticket for cleaning up the docs; they mostly shouldn't talk about application persistence (certainly not in the UDP section).
Tried to do as clear as possible. http://twistedmatrix.com/trac/ticket/5031 -- anatoly t.
participants (6)
-
anatoly techtonik
-
exarkun@twistedmatrix.com
-
Laurens Van Houtven
-
Stephen Thorne
-
Tim Allen
-
Tristan Seligmann