[Numpy-discussion] Histograms of extremely large data sets

[Cameron Walsh]

Thanks very much, Eric.  That line fixed it for me, although I'm still
not sure why it broke with the last line.

Your weave_histogram works a charm and is around 16 times faster than
any of the other options I've tried.  On my laptop it took 30 seconds
to generate a histogram from 500 million numbers, which is fine.

Thanks and best regards,

Cameron.


On 14/12/06, eric jones <eric at enthought.com> wrote:
> Hmmm.
>
> ?  Not sure. ?
>
> Change that line to this instead which should work as well.
>
>         code = array_converter.declaration_code(self, templatize, inline)
>
> Both work for me.
>
> eric
>
> Cameron Walsh wrote:
> > On 13/12/06, Cameron Walsh <cameron.walsh at gmail.com> wrote:
> >
> >> On 13/12/06, eric jones <eric at enthought.com> wrote 290 lines of
> >> awesome code and a fantastic explanation:
> >>
> >>
> >>> Hey Cameron,
> >>>
> >>> I wrote a simple weave based histogram function that should work for
> >>> your problem.  It should work for any array input data type.  The needed
> >>> files (and a few tests and examples) are attached.
> >>>
> > [...]
> >
> > Hi Eric,
> >
> > I've ran test_weave_histogram.py and histogram_speed.py, but each one
> > seems to fail on calling typed_array_converter.declaration_code() as
> > follows:
> >
> > Traceback (most recent call last):
> >   File "histogram_speed.py", line 26, in <module>
> >     res2 = histogram(data, bins)
> >   File "/home/cameron/repos/wavesmaker/trunk/code/gui/process_modules/eric_histo/weave_histogram.py",
> > line 67, in histogram
> >     compiler='gcc')
> >   File "/usr/local/lib/python2.5/site-packages/scipy/weave/inline_tools.py",
> > line 339, in inline
> >     **kw)
> >   File "/usr/local/lib/python2.5/site-packages/scipy/weave/inline_tools.py",
> > line 447, in compile_function
> >     verbose=verbose, **kw)
> >   File "/usr/local/lib/python2.5/site-packages/scipy/weave/ext_tools.py",
> > line 353, in compile
> >     kw,file = self.build_kw_and_file(location,kw)
> >   File "/usr/local/lib/python2.5/site-packages/scipy/weave/ext_tools.py",
> > line 334, in build_kw_and_file
> >     file = self.generate_file(location=location)
> >   File "/usr/local/lib/python2.5/site-packages/scipy/weave/ext_tools.py",
> > line 295, in generate_file
> >     code = self.module_code()
> >   File "/usr/local/lib/python2.5/site-packages/scipy/weave/ext_tools.py",
> > line 203, in module_code
> >     self.function_code(),
> >   File "/usr/local/lib/python2.5/site-packages/scipy/weave/ext_tools.py",
> > line 269, in function_code
> >     all_function_code += func.function_code()
> >   File "/usr/local/lib/python2.5/site-packages/scipy/weave/inline_tools.py",
> > line 83, in function_code
> >     decl_code = indent(self.arg_declaration_code(),4)
> >   File "/usr/local/lib/python2.5/site-packages/scipy/weave/inline_tools.py",
> > line 62, in arg_declaration_code
> >     arg_strings.append(arg.declaration_code(inline=1))
> >   File "/home/cameron/repos/wavesmaker/trunk/code/gui/process_modules/eric_histo/typed_array_converter.py",
> > line 18, in declaration_code
> >     code = super(typed_array_converter, self).declaration_code(templatize,
> > TypeError: super() argument 1 must be type, not classobj
> >
> > I've tried this with Python2.4 and Python2.5 with the same results.
> >
> > What do I need to change, since it seems to have worked for you but not for me.
> >
> > Thanks and best regards,
> >
> > Cameron.
> > _______________________________________________
> > Numpy-discussion mailing list
> > Numpy-discussion at scipy.org
> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >
> >
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>