multiple objects handling
… Hello! I have a stream of objects, each of dictionary type. Actually, they represent events and are emitted by an asynchronous thread. Sample event: { 'type': 'neigh', 'action': 'add', 'index': 2, 'dest': '10.1.0.1', 'lladdr': '0:2:b3:39:2e:4c', 'timestamp': 'Wed Aug 10 17:20:28 2011', 'probes': None, 'cacheinfo': None } I shoud take an action depending on event's 'type' (neigh[bor], link, address, route etc.) and 'action' (add, del) items. Now I see two ways: * build a tree of «if/elif» statements within one function * create a dictionary that points to different methods The issue is that a method call is slower that jump within one function. But a dictionary lookup is faster than «if/elif» statements (if I have a lot of variants and do not want to rebuild B-like tree of «if» statements by hands). Is there a way to combine these methods? Can I create a dictionary of jump instructions? So far I see that JUMP_ABSOLUTE and JUMP_FORWARD take an integer as a parameter. It is reasonable, 'cause it is faster for if/elif/for etc. statements. And thus I can not use a dictionary to store offsets and then use them for one JUMP instruction. But can we have another separate codeop, e.g. something like JUMP_BY_TOS, that takes offset (or absolute address) from the top of the stack? … Yes, I understand that it seems like `goto` statement. … Anyway, it can significantly speed up any stream parsing. … Thanks. -- Peter V. Saveliev
On Fri, Aug 12, 2011 at 1:49 PM, Peter V. Saveliev <peet@altlinux.ru> wrote:
Is the overhead of a method call the biggest problem in your code?
How would this be useful to someone writing a python program? Or put another way, if the Python interpreter had such a feature, what language feature would use it? Since there is no switch statement http://www.python.org/dev/peps/pep-3103/ this seems premature optimization. --- Bruce Follow me: http://www.twitter.com/Vroo http://www.vroospeak.com
On 8/12/2011 4:49 PM, Peter V. Saveliev wrote:
Really? Your runtime is dominated by a *single* dict lookup and a *single* method call? What do you do with these objects that takes so little time that such a micro-optimization will have a "significant speed up" to your program? Have you actually profiled the performance of your program? I would guess that the time spent dispatching through a dictionary is dwarfed by the time spent constructing those event objects and the ultimate processing of them.
How would you use such a opcode? There is no syntax available that would allow you to get the offset (or absolute address) of any given line in a function. To have a "goto" statement, you have to have labels, we don't have those either. Or, are you proposing that CPython optimize large if/elif blocks into dispatch tables? That sounds more interesting, but the performance win would have to worth the additional code complexity. -- Scott Dial scott@scottdial.com
On 13.08.2011 01:47, Scott Dial wrote: <skip />
Really? Your runtime is dominated by a *single* dict lookup and a *single* method call?
No, surely :))) But it is one of the problems.
You're right, but having really huge stream of packets I try to minimize any overhead — using C modules, ctypes library and so on. But I hope to save high-level logic in Python, and before to reject a chance to speed up parsing, I asked for some alternatives and contras. I believe that method calls here are unnecessary while the «if» statements tree makes the code hard to support.
'Cause there can be any expression used in «if» statement, I see no way to create pre-computed hash :( Maybe something like «switch» would work, as Bruce says, but it is rejected for now. -- Peter V. Saveliev
On Sat, Aug 13, 2011 at 8:18 AM, Peter V. Saveliev <peet@altlinux.ru> wrote:
This kind of logic micro-optimisation is best handled by a JIT compiler rather than messing with the language definition and asking people to do it by hand. I suggest running your application on PyPy and seeing what kind of speed up you get. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Fri, Aug 12, 2011 at 1:49 PM, Peter V. Saveliev <peet@altlinux.ru> wrote:
Is the overhead of a method call the biggest problem in your code?
How would this be useful to someone writing a python program? Or put another way, if the Python interpreter had such a feature, what language feature would use it? Since there is no switch statement http://www.python.org/dev/peps/pep-3103/ this seems premature optimization. --- Bruce Follow me: http://www.twitter.com/Vroo http://www.vroospeak.com
On 8/12/2011 4:49 PM, Peter V. Saveliev wrote:
Really? Your runtime is dominated by a *single* dict lookup and a *single* method call? What do you do with these objects that takes so little time that such a micro-optimization will have a "significant speed up" to your program? Have you actually profiled the performance of your program? I would guess that the time spent dispatching through a dictionary is dwarfed by the time spent constructing those event objects and the ultimate processing of them.
How would you use such a opcode? There is no syntax available that would allow you to get the offset (or absolute address) of any given line in a function. To have a "goto" statement, you have to have labels, we don't have those either. Or, are you proposing that CPython optimize large if/elif blocks into dispatch tables? That sounds more interesting, but the performance win would have to worth the additional code complexity. -- Scott Dial scott@scottdial.com
On 13.08.2011 01:47, Scott Dial wrote: <skip />
Really? Your runtime is dominated by a *single* dict lookup and a *single* method call?
No, surely :))) But it is one of the problems.
You're right, but having really huge stream of packets I try to minimize any overhead — using C modules, ctypes library and so on. But I hope to save high-level logic in Python, and before to reject a chance to speed up parsing, I asked for some alternatives and contras. I believe that method calls here are unnecessary while the «if» statements tree makes the code hard to support.
'Cause there can be any expression used in «if» statement, I see no way to create pre-computed hash :( Maybe something like «switch» would work, as Bruce says, but it is rejected for now. -- Peter V. Saveliev
On Sat, Aug 13, 2011 at 8:18 AM, Peter V. Saveliev <peet@altlinux.ru> wrote:
This kind of logic micro-optimisation is best handled by a JIT compiler rather than messing with the language definition and asking people to do it by hand. I suggest running your application on PyPy and seeing what kind of speed up you get. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (5)
-
Bruce Leban -
Nick Coghlan -
Peter V. Saveliev -
Scott Dial -
海韵