<div dir="ltr">Here are the collated results of running each query. For each run, I count how many of each of the pypy debug lines i get. I.e. there were 668 runs that printed 58 loglines that contain "{gc-minor" which was eventually followed by "gc-minor}". I have also counted if the query was slow; interestingly, not all the queries with many gc-minors were slow (but all slow queries had a gc-minor).<div>
<br></div><div>Please let me know if this is unclear :)<br><div><br></div><div><div><font face="courier new, monospace"> 668 gc-minor:58 gc-minor-walkroots:58</font></div><div><font face="courier new, monospace"> 10 gc-minor:58 gc-minor-walkroots:58 gc-collect-step:5</font></div>
<div><font face="courier new, monospace"> 140 gc-minor:59 gc-minor-walkroots:59</font></div><div><font face="courier new, monospace"> 1 gc-minor:8441 gc-minor-walkroots:8441 gc-collect-step:8403</font></div><div><font face="courier new, monospace"> 1 gc-minor:9300 gc-minor-walkroots:9300 gc-collect-step:9249</font></div>
<div><font face="courier new, monospace"> 9 gc-minor:9643 <b>slow</b>:1 gc-minor-walkroots:9643 gc-collect-step:9589</font></div><div><font face="courier new, monospace"> 1 gc-minor:9644 <b>slow</b>:1 gc-minor-walkroots:9644 gc-collect-step:9590</font></div>
<div><font face="courier new, monospace"> 10 gc-minor:9647 <b>slow</b>:1 gc-minor-walkroots:9647 gc-collect-step:9609</font></div><div><font face="courier new, monospace"> 1 gc-minor:9663 gc-minor-walkroots:9663 gc-collect-step:9614</font></div>
<div><font face="courier new, monospace"> 1 jit-backend-dump:5 gc-minor:58 gc-minor-walkroots:58</font></div><div><font face="courier new, monospace"> 1 jit-log-compiling-loop:1 gc-collect-step:8991 jit-backend-dump:78 jit-backend:3 jit-log-noopt-loop:6 jit-log-virtualstate:3 gc-minor:9030 jit-tracing:3 gc-minor-walkroots:9030 jit-optimize:6 jit-log-short-preamble:2 jit-backend-addr:3 jit-log-opt-loop:1 jit-mem-looptoken-alloc:3 jit-abort:3 jit-log-rewritten-bridge:2 jit-log-rewritten-loop:1 jit-log-opt-bridge:2 jit-log-compiling-bridge:2 jit-resume:84</font></div>
<div><font face="courier new, monospace"> 1 jit-log-compiling-loop:1 jit-backend-dump:13 jit-backend:1 jit-log-noopt-loop:2 gc-minor:60 jit-tracing:1 gc-minor-walkroots:60 jit-optimize:2 jit-log-short-preamble:1 jit-backend-addr:1 jit-log-opt-loop:1 jit-mem-looptoken-alloc:1 jit-log-rewritten-loop:1 jit-resume:14</font></div>
<div><font face="courier new, monospace"> 1 jit-log-compiling-loop:1 jit-backend-dump:73 jit-backend:3 jit-log-noopt-loop:6 jit-log-virtualstate:3 gc-minor:60 jit-tracing:3 gc-minor-walkroots:60 jit-optimize:6 jit-log-short-preamble:2 jit-backend-addr:3 jit-log-opt-loop:1 jit-mem-looptoken-alloc:3 jit-abort:3 jit-log-rewritten-bridge:2 jit-log-rewritten-loop:1 jit-log-opt-bridge:2 jit-log-compiling-bridge:2 jit-resume:84</font></div>
<div><font face="courier new, monospace"> 2 jit-log-compiling-loop:1 jit-backend-dump:78 jit-backend:3 jit-log-noopt-loop:6 jit-log-virtualstate:3 gc-minor:61 jit-tracing:3 gc-minor-walkroots:61 jit-optimize:6 jit-log-short-preamble:2 jit-backend-addr:3 jit-log-opt-loop:1 jit-mem-looptoken-alloc:3 jit-abort:3 jit-log-rewritten-bridge:2 jit-log-rewritten-loop:1 jit-log-opt-bridge:2 jit-log-compiling-bridge:2 jit-resume:84</font></div>
<div><font face="courier new, monospace"> 1 jit-log-short-preamble:2 jit-log-compiling-loop:2 jit-backend-dump:92 jit-log-noopt-loop:7 jit-log-virtualstate:3 gc-minor:61 jit-tracing:4 gc-minor-walkroots:61 jit-optimize:7 jit-backend:4 jit-backend-addr:4 jit-log-opt-loop:2 jit-mem-looptoken-alloc:4 jit-abort:3 jit-log-rewritten-bridge:2 jit-log-rewritten-loop:2 jit-log-opt-bridge:2 jit-log-compiling-bridge:2 jit-resume:104</font></div>
</div><div><br></div><div><br></div><div>Thanks, </div><div>/Martin</div><div><br></div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Mar 17, 2014 at 2:23 PM, Maciej Fijalkowski <span dir="ltr"><<a href="mailto:fijall@gmail.com" target="_blank">fijall@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="">On Mon, Mar 17, 2014 at 3:20 PM, Maciej Fijalkowski <<a href="mailto:fijall@gmail.com">fijall@gmail.com</a>> wrote:<br>
> are you *sure* it's the walkroots that take that long and not<br>
> something else (like gc-minor)? More of those mean that you allocate a<br>
> lot more surviving objects. Can you do two things:<br>
><br>
> a) take a max of gc-minor (and gc-minor-stackwalk), per request<br>
> b) take the sum of those<br>
><br>
> and plot them<br>
<br>
</div>^^^ or just paste the results actually<br>
<div class="HOEnZb"><div class="h5"><br>
><br>
> On Mon, Mar 17, 2014 at 3:18 PM, Martin Koch <<a href="mailto:mak@issuu.com">mak@issuu.com</a>> wrote:<br>
>> Well, then it works out to around 2.5GHz, which seems reasonable. But it<br>
>> doesn't alter the conclusion from the previous email: The slow queries then<br>
>> all have a duration around 34*10^9 units, 'normal' queries 1*10^9 units, or<br>
>> .4 seconds at this conversion. Also, the log shows that a slow query<br>
>> performs many more gc-minor operations than a 'normal' one: 9600<br>
>> gc-collect-step/gc-minor/gc-minor-walkroots operations vs 58.<br>
>><br>
>> So the question becomes: Why do we get this large spike in<br>
>> gc-minor-walkroots, and, in particular, is there any way to avoid it :) ?<br>
>><br>
>> Thanks,<br>
>> /Martin<br>
>><br>
>><br>
>> On Mon, Mar 17, 2014 at 1:53 PM, Maciej Fijalkowski <<a href="mailto:fijall@gmail.com">fijall@gmail.com</a>><br>
>> wrote:<br>
>>><br>
>>> I think it's the cycles of your CPU<br>
>>><br>
>>> On Mon, Mar 17, 2014 at 2:48 PM, Martin Koch <<a href="mailto:mak@issuu.com">mak@issuu.com</a>> wrote:<br>
>>> > What is the unit? Perhaps I'm being thick here, but I can't correlate it<br>
>>> > with seconds (which the program does print out). Slow runs are around 13<br>
>>> > seconds, but are around 34*10^9(dec), 0x800000000 timestamp units (e.g.<br>
>>> > from<br>
>>> > 0x2b994c9d31889c to 0x2b9944ab8c4f49).<br>
>>> ><br>
>>> ><br>
>>> ><br>
>>> > On Mon, Mar 17, 2014 at 12:09 PM, Maciej Fijalkowski <<a href="mailto:fijall@gmail.com">fijall@gmail.com</a>><br>
>>> > wrote:<br>
>>> >><br>
>>> >> The number of lines is nonsense. This is a timestamp in hex.<br>
>>> >><br>
>>> >> On Mon, Mar 17, 2014 at 12:46 PM, Martin Koch <<a href="mailto:mak@issuu.com">mak@issuu.com</a>> wrote:<br>
>>> >> > Based On Maciej's suggestion, I tried the following<br>
>>> >> ><br>
>>> >> > PYPYLOG=- pypy mem.py 10000000 > out<br>
>>> >> ><br>
>>> >> > This generates a logfile which looks something like this<br>
>>> >> ><br>
>>> >> > start--><br>
>>> >> > [2b99f1981b527e] {gc-minor<br>
>>> >> > [2b99f1981ba680] {gc-minor-walkroots<br>
>>> >> > [2b99f1981c2e02] gc-minor-walkroots}<br>
>>> >> > [2b99f19890d750] gc-minor}<br>
>>> >> > [snip]<br>
>>> >> > ...<br>
>>> >> > <--stop<br>
>>> >> ><br>
>>> >> ><br>
>>> >> > It turns out that the culprit is a lot of MINOR collections.<br>
>>> >> ><br>
>>> >> > I base this on the following observations:<br>
>>> >> ><br>
>>> >> > I can't understand the format of the timestamp on each logline (the<br>
>>> >> > "[2b99f1981b527e]"). From what I can see in the code, this should be<br>
>>> >> > output<br>
>>> >> > from time.clock(), but that doesn't return a number like that when I<br>
>>> >> > run<br>
>>> >> > pypy interactively<br>
>>> >> > Instead, I count the number of debug lines between start--> and the<br>
>>> >> > corresponding <--stop.<br>
>>> >> > Most runs have a few hundred lines of output between start/stop<br>
>>> >> > All slow runs have very close to 57800 lines out output between<br>
>>> >> > start/stop<br>
>>> >> > One such sample does 9609 gc-collect-step operations, 9647 gc-minor<br>
>>> >> > operations, and 9647 gc-minor-walkroots operations.<br>
>>> >> ><br>
>>> >> ><br>
>>> >> > Thanks,<br>
>>> >> > /Martin<br>
>>> >> ><br>
>>> >> ><br>
>>> >> > On Mon, Mar 17, 2014 at 8:21 AM, Maciej Fijalkowski<br>
>>> >> > <<a href="mailto:fijall@gmail.com">fijall@gmail.com</a>><br>
>>> >> > wrote:<br>
>>> >> >><br>
>>> >> >> there is an environment variable PYPYLOG=gc:- (where - is stdout)<br>
>>> >> >> which will do that for you btw.<br>
>>> >> >><br>
>>> >> >> maybe you can find out what's that using profiling or valgrind?<br>
>>> >> >><br>
>>> >> >> On Sun, Mar 16, 2014 at 11:34 PM, Martin Koch <<a href="mailto:mak@issuu.com">mak@issuu.com</a>> wrote:<br>
>>> >> >> > I have tried getting the pypy source and building my own version<br>
>>> >> >> > of<br>
>>> >> >> > pypy. I<br>
>>> >> >> > have modified<br>
>>> >> >> > rpython/memory/gc/incminimark.py:major_collection_step()<br>
>>> >> >> > to<br>
>>> >> >> > print out when it starts and when it stops. Apparently, the slow<br>
>>> >> >> > queries<br>
>>> >> >> > do<br>
>>> >> >> > NOT occur during major_collection_step; at least, I have not<br>
>>> >> >> > observed<br>
>>> >> >> > major<br>
>>> >> >> > step output during a query execution. So, apparently, something<br>
>>> >> >> > else<br>
>>> >> >> > is<br>
>>> >> >> > blocking. This could be another aspect of the GC, but it could<br>
>>> >> >> > also<br>
>>> >> >> > be<br>
>>> >> >> > anything else.<br>
>>> >> >> ><br>
>>> >> >> > Just to be sure, I have tried running the same application in<br>
>>> >> >> > python<br>
>>> >> >> > with<br>
>>> >> >> > garbage collection disabled. I don't see the problem there, so it<br>
>>> >> >> > is<br>
>>> >> >> > somehow<br>
>>> >> >> > related to either GC or the runtime somehow.<br>
>>> >> >> ><br>
>>> >> >> > Cheers,<br>
>>> >> >> > /Martin<br>
>>> >> >> ><br>
>>> >> >> ><br>
>>> >> >> > On Fri, Mar 14, 2014 at 4:19 PM, Martin Koch <<a href="mailto:mak@issuu.com">mak@issuu.com</a>><br>
>>> >> >> > wrote:<br>
>>> >> >> >><br>
>>> >> >> >> We have hacked up a small sample that seems to exhibit the same<br>
>>> >> >> >> issue.<br>
>>> >> >> >><br>
>>> >> >> >> We basically generate a linked list of objects. To increase<br>
>>> >> >> >> connectedness,<br>
>>> >> >> >> elements in the list hold references (dummy_links) to 10 randomly<br>
>>> >> >> >> chosen<br>
>>> >> >> >> previous elements in the list.<br>
>>> >> >> >><br>
>>> >> >> >> We then time a function that traverses 50000 elements from the<br>
>>> >> >> >> list<br>
>>> >> >> >> from a<br>
>>> >> >> >> random start point. If the traversal reaches the end of the list,<br>
>>> >> >> >> we<br>
>>> >> >> >> instead<br>
>>> >> >> >> traverse one of the dummy links. Thus, exactly 50K elements are<br>
>>> >> >> >> traversed<br>
>>> >> >> >> every time. To generate some garbage, we build a list holding the<br>
>>> >> >> >> traversed<br>
>>> >> >> >> elements and a dummy list of characters.<br>
>>> >> >> >><br>
>>> >> >> >> Timings for the last 100 runs are stored in a circular buffer. If<br>
>>> >> >> >> the<br>
>>> >> >> >> elapsed time for the last run is more than twice the average<br>
>>> >> >> >> time,<br>
>>> >> >> >> we<br>
>>> >> >> >> print<br>
>>> >> >> >> out a line with the elapsed time, the threshold, and the 90%<br>
>>> >> >> >> runtime<br>
>>> >> >> >> (we<br>
>>> >> >> >> would like to see that the mean runtime does not increase with<br>
>>> >> >> >> the<br>
>>> >> >> >> number of<br>
>>> >> >> >> elements in the list, but that the max time does increase<br>
>>> >> >> >> (linearly<br>
>>> >> >> >> with the<br>
>>> >> >> >> number of object, i guess); traversing 50K elements should be<br>
>>> >> >> >> independent of<br>
>>> >> >> >> the memory size).<br>
>>> >> >> >><br>
>>> >> >> >> We have tried monitoring memory consumption by external<br>
>>> >> >> >> inspection,<br>
>>> >> >> >> but<br>
>>> >> >> >> cannot consistently verify that memory is deallocated at the same<br>
>>> >> >> >> time<br>
>>> >> >> >> that<br>
>>> >> >> >> we see slow requests. Perhaps the pypy runtime doesn't always<br>
>>> >> >> >> return<br>
>>> >> >> >> freed<br>
>>> >> >> >> pages back to the OS?<br>
>>> >> >> >><br>
>>> >> >> >> Using top, we observe that 10M elements allocates around 17GB<br>
>>> >> >> >> after<br>
>>> >> >> >> building, 20M elements 26GB, 30M elements 28GB (and grows to 35GB<br>
>>> >> >> >> shortly<br>
>>> >> >> >> after building).<br>
>>> >> >> >><br>
>>> >> >> >> Here is output from a few runs with different number of elements:<br>
>>> >> >> >><br>
>>> >> >> >><br>
>>> >> >> >> pypy mem.py 10000000<br>
>>> >> >> >> start build<br>
>>> >> >> >> end build 84.142424<br>
>>> >> >> >> that took a long time elapsed: 13.230586 slow_threshold:<br>
>>> >> >> >> 1.495401<br>
>>> >> >> >> 90th_quantile_runtime: 0.421558<br>
>>> >> >> >> that took a long time elapsed: 13.016531 slow_threshold:<br>
>>> >> >> >> 1.488160<br>
>>> >> >> >> 90th_quantile_runtime: 0.423441<br>
>>> >> >> >> that took a long time elapsed: 13.032537 slow_threshold:<br>
>>> >> >> >> 1.474563<br>
>>> >> >> >> 90th_quantile_runtime: 0.419817<br>
>>> >> >> >><br>
>>> >> >> >> pypy mem.py 20000000<br>
>>> >> >> >> start build<br>
>>> >> >> >> end build 180.823105<br>
>>> >> >> >> that took a long time elapsed: 27.346064 slow_threshold:<br>
>>> >> >> >> 2.295146<br>
>>> >> >> >> 90th_quantile_runtime: 0.434726<br>
>>> >> >> >> that took a long time elapsed: 26.028852 slow_threshold:<br>
>>> >> >> >> 2.283927<br>
>>> >> >> >> 90th_quantile_runtime: 0.374190<br>
>>> >> >> >> that took a long time elapsed: 25.432279 slow_threshold:<br>
>>> >> >> >> 2.279631<br>
>>> >> >> >> 90th_quantile_runtime: 0.371502<br>
>>> >> >> >><br>
>>> >> >> >> pypy mem.py 30000000<br>
>>> >> >> >> start build<br>
>>> >> >> >> end build 276.217811<br>
>>> >> >> >> that took a long time elapsed: 40.993855 slow_threshold:<br>
>>> >> >> >> 3.188464<br>
>>> >> >> >> 90th_quantile_runtime: 0.459891<br>
>>> >> >> >> that took a long time elapsed: 41.693553 slow_threshold:<br>
>>> >> >> >> 3.183003<br>
>>> >> >> >> 90th_quantile_runtime: 0.393654<br>
>>> >> >> >> that took a long time elapsed: 39.679769 slow_threshold:<br>
>>> >> >> >> 3.190782<br>
>>> >> >> >> 90th_quantile_runtime: 0.393677<br>
>>> >> >> >> that took a long time elapsed: 43.573411 slow_threshold:<br>
>>> >> >> >> 3.239637<br>
>>> >> >> >> 90th_quantile_runtime: 0.393654<br>
>>> >> >> >><br>
>>> >> >> >> Code below<br>
>>> >> >> >> --------------------------------------------------------------<br>
>>> >> >> >> import time<br>
>>> >> >> >> from random import randint, choice<br>
>>> >> >> >> import sys<br>
>>> >> >> >><br>
>>> >> >> >><br>
>>> >> >> >> allElems = {}<br>
>>> >> >> >><br>
>>> >> >> >> class Node:<br>
>>> >> >> >> def __init__(self, v_):<br>
>>> >> >> >> self.v = v_<br>
>>> >> >> >> self.next = None<br>
>>> >> >> >> self.dummy_data = [randint(0,100)<br>
>>> >> >> >> for _ in xrange(randint(50,100))]<br>
>>> >> >> >> allElems[self.v] = self<br>
>>> >> >> >> if self.v > 0:<br>
>>> >> >> >> self.dummy_links = [allElems[randint(0, self.v-1)]<br>
>>> >> >> >> for _<br>
>>> >> >> >> in<br>
>>> >> >> >> xrange(10)]<br>
>>> >> >> >> else:<br>
>>> >> >> >> self.dummy_links = [self]<br>
>>> >> >> >><br>
>>> >> >> >> def set_next(self, l):<br>
>>> >> >> >> self.next = l<br>
>>> >> >> >><br>
>>> >> >> >><br>
>>> >> >> >> def follow(node):<br>
>>> >> >> >> acc = []<br>
>>> >> >> >> count = 0<br>
>>> >> >> >> cur = node<br>
>>> >> >> >> assert node.v is not None<br>
>>> >> >> >> assert cur is not None<br>
>>> >> >> >> while count < 50000:<br>
>>> >> >> >> # return a value; generate some garbage<br>
>>> >> >> >> acc.append((cur.v, [choice("abcdefghijklmnopqrstuvwxyz")<br>
>>> >> >> >> for<br>
>>> >> >> >> x<br>
>>> >> >> >> in<br>
>>> >> >> >> xrange(100)]))<br>
>>> >> >> >><br>
>>> >> >> >> # if we have reached the end, chose a random link<br>
>>> >> >> >> cur = choice(cur.dummy_links) if cur.next is None else<br>
>>> >> >> >> cur.next<br>
>>> >> >> >> count += 1<br>
>>> >> >> >><br>
>>> >> >> >> return acc<br>
>>> >> >> >><br>
>>> >> >> >><br>
>>> >> >> >> def build(num_elems):<br>
>>> >> >> >> start = time.time()<br>
>>> >> >> >> print "start build"<br>
>>> >> >> >> root = Node(0)<br>
>>> >> >> >> cur = root<br>
>>> >> >> >> for x in xrange(1, num_elems):<br>
>>> >> >> >> e = Node(x)<br>
>>> >> >> >> cur.next = e<br>
>>> >> >> >> cur = e<br>
>>> >> >> >> print "end build %f" % (time.time() - start)<br>
>>> >> >> >> return root<br>
>>> >> >> >><br>
>>> >> >> >><br>
>>> >> >> >> num_timings = 100<br>
>>> >> >> >> if __name__ == "__main__":<br>
>>> >> >> >> num_elems = int(sys.argv[1])<br>
>>> >> >> >> build(num_elems)<br>
>>> >> >> >> total = 0<br>
>>> >> >> >> timings = [0.0] * num_timings # run times for the last<br>
>>> >> >> >> num_timings<br>
>>> >> >> >> runs<br>
>>> >> >> >> i = 0<br>
>>> >> >> >> beginning = time.time()<br>
>>> >> >> >> while time.time() - beginning < 600:<br>
>>> >> >> >> start = time.time()<br>
>>> >> >> >> elem = allElems[randint(0, num_elems - 1)]<br>
>>> >> >> >> assert(elem is not None)<br>
>>> >> >> >><br>
>>> >> >> >> lst = follow(elem)<br>
>>> >> >> >><br>
>>> >> >> >> total += choice(lst)[0] # use the return value for<br>
>>> >> >> >> something<br>
>>> >> >> >><br>
>>> >> >> >> end = time.time()<br>
>>> >> >> >><br>
>>> >> >> >> elapsed = end-start<br>
>>> >> >> >> timings[i % num_timings] = elapsed<br>
>>> >> >> >> if (i > num_timings):<br>
>>> >> >> >> slow_time = 2 * sum(timings)/num_timings # slow<br>
>>> >> >> >> defined<br>
>>> >> >> >> as<br>
>>> >> >> >> ><br>
>>> >> >> >> 2*avg run time<br>
>>> >> >> >> if (elapsed > slow_time):<br>
>>> >> >> >> print "that took a long time elapsed: %f<br>
>>> >> >> >> slow_threshold:<br>
>>> >> >> >> %f 90th_quantile_runtime: %f" % \<br>
>>> >> >> >> (elapsed, slow_time,<br>
>>> >> >> >> sorted(timings)[int(num_timings*.9)])<br>
>>> >> >> >> i += 1<br>
>>> >> >> >> print total<br>
>>> >> >> >><br>
>>> >> >> >><br>
>>> >> >> >><br>
>>> >> >> >><br>
>>> >> >> >><br>
>>> >> >> >> On Thu, Mar 13, 2014 at 7:45 PM, Maciej Fijalkowski<br>
>>> >> >> >> <<a href="mailto:fijall@gmail.com">fijall@gmail.com</a>><br>
>>> >> >> >> wrote:<br>
>>> >> >> >>><br>
>>> >> >> >>> On Thu, Mar 13, 2014 at 1:45 PM, Martin Koch <<a href="mailto:mak@issuu.com">mak@issuu.com</a>><br>
>>> >> >> >>> wrote:<br>
>>> >> >> >>> > Hi Armin, Maciej<br>
>>> >> >> >>> ><br>
>>> >> >> >>> > Thanks for responding.<br>
>>> >> >> >>> ><br>
>>> >> >> >>> > I'm in the process of trying to determine what (if any) of the<br>
>>> >> >> >>> > code<br>
>>> >> >> >>> > I'm<br>
>>> >> >> >>> > in a<br>
>>> >> >> >>> > position to share, and I'll get back to you.<br>
>>> >> >> >>> ><br>
>>> >> >> >>> > Allowing hinting to the GC would be good. Even better would be<br>
>>> >> >> >>> > a<br>
>>> >> >> >>> > means<br>
>>> >> >> >>> > to<br>
>>> >> >> >>> > allow me to (transparently) allocate objects in unmanaged<br>
>>> >> >> >>> > memory,<br>
>>> >> >> >>> > but I<br>
>>> >> >> >>> > would expect that to be a tall order :)<br>
>>> >> >> >>> ><br>
>>> >> >> >>> > Thanks,<br>
>>> >> >> >>> > /Martin<br>
>>> >> >> >>><br>
>>> >> >> >>> Hi Martin.<br>
>>> >> >> >>><br>
>>> >> >> >>> Note that in case you want us to do the work of isolating the<br>
>>> >> >> >>> problem,<br>
>>> >> >> >>> we do offer paid support to do that (then we can sign NDAs and<br>
>>> >> >> >>> stuff).<br>
>>> >> >> >>> Otherwise we would be more than happy to fix bugs once you<br>
>>> >> >> >>> isolate<br>
>>> >> >> >>> a<br>
>>> >> >> >>> part you can share freely :)<br>
>>> >> >> >><br>
>>> >> >> >><br>
>>> >> >> ><br>
>>> >> ><br>
>>> >> ><br>
>>> ><br>
>>> ><br>
>><br>
>><br>
</div></div></blockquote></div><br></div>