I'm looking at various options for implementing a high throughput
database logger that will work with Twisted.
My requirements, listed by importance:
1) small memory footprint
2) high speed
3) low garbage generation
The application I'm working on runs continuously (24/7). I've
experimented a bit with pysqlite and Twisted to see which approach is
better suited (see attached example).
Question 1: I noticed that all of the Twisted based versions are very
slow compared to the plain sqlite3 test. This seems to be caused by
atomic transaction management, namely a commit after each insert.
Would be interested to know if there is a simple way to avoid this and
do my own transaction management (aka batch commit).
One other thing is the greatly varying amounts of garbage generated
(peak memory) and memory usage between the Twisted variants.
Question 2: I would have expected B (Twisted ADBAPI) to behave very
similar to C/E since I'm using a connection pool of size 1 and all
requests are queued and handled sequentially.
Could any of you please give me some pointers as to why this is happening?
Question 3: Even though objgraph lists the exact same reference count
once the code is ran, the amount of used memory greatly differs. Any
ideas what might be causing this?
Any suggestions and/or pointers on how to improve/do this are more than
Thank you for your time,
>From spread/jelly.py in _unjelly_method:
- if type(im_class) is not types.ClassType:
raise InsecureJelly("Method found with non-class class.")
Same problem in _unjelly_instance
What is a non-class class or rather what sort of types is meant to
Suggestions for a better error message instead of "non-class class"?
This should work with both old style and new style classes, right
now it works only for old style and fails for new style. Working
with Python3 gives bonus points.
Current source code does not have test cases for both _unjelly_instance
and _unjelly_method using new style classes. My patch for ticket 4935
has a such a test case for _unjelly_method which fails as expected.
If only user defined classes are accepted as secure, see also
Would it be acceptable to instead exclude an explicit list of
basic types as that stackoverflow answer suggests?
if im_class in (int, long, float, bool, list, set, frozenset, dict, type(None), str,
unicode, tuple, what else?):
does not cover this.
>>> class A(object):
>>> import types
>>> class B:
>>> class A:
I am little new to twisted.
I've a function(get_handle) which returns a handle.
But get_handle() may throw an exception(SystemDelayException) because of
some delay in system.
My requirement is : When I get SystemDelayException re-attempt
get_handle() and return only when handle is available. How do I use
deferred/reactor in this scenario?
handle = yield self.get_handle()
// code to re-attempt get_handle and return only when handle is
If I call spawnProcess with a bad executable, on Unix the process will
exit with exit code 1 while on Windows an OSError is raised.
I am working on a multi-OS software and to make my life easier I ended
up with this hack inspired by _BackRelay
Is there something already in Twisted doing this? Am I reinventing the wheel?
If not, do you think that it would help to update _BackRelay to
support timeout and cross-os error handling?
I have wrote a custom protocol(inherit from SMTPClient) and factory(inherit from ClientFactory) in order to send emails from multiple ips(multiple connections, factories).
Now I want to run this pice of code multiple times using spawnProcess to send emails simultaneous, each process having "n" connections from "n" ips.
How can i use spawnProcess? Or is another way to send emails in a parallel fashion with twisted? Maybe another approach?
while trying to port jelly to Python 3 (banana tests already pass with Python 3),
I found a bug in jelly. It has already been reported 3 years ago with ticket 4935.
And there is a fix attached to that ticket. This fix resolves the problem for me.
Why has this fix never been included? It is necessary for porting to Python 3.