rgaddi at technologyhighland.invalid
Fri Sep 26 20:30:16 CEST 2014
On Tue, 23 Sep 2014 22:01:51 -0700 (PDT)
Miki Tebeka <miki.tebeka at gmail.com> wrote:
> On Tuesday, September 23, 2014 7:33:06 PM UTC+3, Rob Gaddi wrote:
> > While you're at it, think
> > long and hard about that definition of fuzziness. If you can make it
> > closer to the concept of histogram "bins" you'll get much better
> > performance.
> The problem for me here is that I can't determine the number of bins in advance. I'd like to get frequencies. I guess every "new" (don't have any previous equal item) can be a bin.
> > TL;DR you need to think very hard about your problem definition and
> > what you want to happen before you actually try to implement this.
> Always a good advice :) I'm actually implementing algorithm for someone else (in the bio world where I know very little about).
See, THERE's your problem. You've got a scientist trying to make
prescriptions for an engineering problem. He's given you a fuzzy
description of the sort of thing he's trying to do. Your job is to
turn that fuzzy description into a concrete, actual algorithm
before you even write a single line of code, which means understanding
what the data is, and what the desired result of that data is. Because
the thing you keep trying to do, with all of its order dependencies
fundamentally CANNOT be right, regardless of what the squishy scientist
The "histogram" bin solution that everyone keeps trying to steer you
towards is almost certainly what you really want. Epsilon is your
resolution. You cannot resolve any information below your resolution
limit. Yes, 1.49 and 1.51 wind up in different bins, whereas 1.51 and
2.49 are in the same one, but that's what it means to have a resolution
of 1; you can't say anything about whether any given count in the "2,
plus or minus a bit" bin is very nearly 1 or very nearly 3.
This doesn't require you to know the number of bins in advance, you can
just create and fill them as needed. That said, you're trying to solve
a physical problem, and so it has physical limits. Your biologist
should be able to give you an order of magnitude estimate of how many
"bins" you're expecting, and what the ultimate shape is expected to
look like. Normally distributed? Wildly bimodal? Is the overall span
of data going to span 10 epsilon or 10,000 epsilon? If there are going
to be a ton of bins, you may be better served by putting 1/3 of a count
into bins n-1, n, and n+1 rather than just in bin n; it's the
equivalent of squinting a bit when you look at the bins.
But you have to understand the problem to solve it.
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order. See above to fix.
More information about the Python-list