Hi all, I wonder if there is any method to do exact clustering (hierarchy cluster) on a huge dataset where it is impossible to use distance matrix. I am considering KD-tree but every time it needs to rebuild it, consuming lots time. Any ideas?
Have you had a look at BIRCH? http://scikit-learn.org/stable/modules/clustering.html#birch -- Olivier
Yes, it is an efficient method, still, we need to specify the number of clusters or the threshold. Is there another way to run hierarchy clustering on the big dataset? The main problem is the distance matrix. Thanks. On Tue, Jan 2, 2018 at 6:02 AM, Olivier Grisel <olivier.grisel@ensta.org> wrote:
Have you had a look at BIRCH?
http://scikit-learn.org/stable/modules/clustering.html#birch
-- Olivier
_______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
Can you use nearest neighbors with a KD tree to build a distance matrix that is sparse, in that distances to all but the nearest neighbors of a point are (near-)infinite? Yes, this again has an additional parameter (neighborhood size), just as BIRCH has its threshold. I suspect you will not be able to improve on having another, approximating, parameter. You do not need to set n_clusters to a fixed value for BIRCH. You only need to provide another clusterer, which has its own parameters, although you should be able to experiment with different "global clusterers". On 4 January 2018 at 11:04, Shiheng Duan <shiduan@ucdavis.edu> wrote:
Yes, it is an efficient method, still, we need to specify the number of clusters or the threshold. Is there another way to run hierarchy clustering on the big dataset? The main problem is the distance matrix. Thanks.
On Tue, Jan 2, 2018 at 6:02 AM, Olivier Grisel <olivier.grisel@ensta.org> wrote:
Have you had a look at BIRCH?
http://scikit-learn.org/stable/modules/clustering.html#birch
-- Olivier
_______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
_______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
Thanks, Joel, I am working on KD-tree to find the nearest neighbors. Basically, I find the nearest neighbors for each point and then merge a couple of points if they are both NN for each other. The problem is that after each iteration, we will have a new bunch of points, where new clusters are added. So the tree needs to be updated. Since I didn't find any dynamic way to update the tree, I just rebuild it after each iteration, costing lots of time. Any idea about it? Actually, it takes around 16 mins to build the tree in the first iteration, which is not slow I think. But it still runs slowly. I have a dataset of 12*872505 (features, samples). It takes several days to run the program. Is there any way to speed up the query process of NN? I doubt query may be too slow. Thanks for your time. On Thu, Jan 4, 2018 at 3:55 AM, Joel Nothman <joel.nothman@gmail.com> wrote:
Can you use nearest neighbors with a KD tree to build a distance matrix that is sparse, in that distances to all but the nearest neighbors of a point are (near-)infinite? Yes, this again has an additional parameter (neighborhood size), just as BIRCH has its threshold. I suspect you will not be able to improve on having another, approximating, parameter. You do not need to set n_clusters to a fixed value for BIRCH. You only need to provide another clusterer, which has its own parameters, although you should be able to experiment with different "global clusterers".
On 4 January 2018 at 11:04, Shiheng Duan <shiduan@ucdavis.edu> wrote:
Yes, it is an efficient method, still, we need to specify the number of clusters or the threshold. Is there another way to run hierarchy clustering on the big dataset? The main problem is the distance matrix. Thanks.
On Tue, Jan 2, 2018 at 6:02 AM, Olivier Grisel <olivier.grisel@ensta.org> wrote:
Have you had a look at BIRCH?
http://scikit-learn.org/stable/modules/clustering.html#birch
-- Olivier
_______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
_______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
_______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
Yes, use an approximate nearest neighbors approach. None is included in scikit-learn, but there are numerous implementations with Python interfaces. On 5 January 2018 at 12:51, Shiheng Duan <shiduan@ucdavis.edu> wrote:
Thanks, Joel, I am working on KD-tree to find the nearest neighbors. Basically, I find the nearest neighbors for each point and then merge a couple of points if they are both NN for each other. The problem is that after each iteration, we will have a new bunch of points, where new clusters are added. So the tree needs to be updated. Since I didn't find any dynamic way to update the tree, I just rebuild it after each iteration, costing lots of time. Any idea about it? Actually, it takes around 16 mins to build the tree in the first iteration, which is not slow I think. But it still runs slowly. I have a dataset of 12*872505 (features, samples). It takes several days to run the program. Is there any way to speed up the query process of NN? I doubt query may be too slow. Thanks for your time.
On Thu, Jan 4, 2018 at 3:55 AM, Joel Nothman <joel.nothman@gmail.com> wrote:
Can you use nearest neighbors with a KD tree to build a distance matrix that is sparse, in that distances to all but the nearest neighbors of a point are (near-)infinite? Yes, this again has an additional parameter (neighborhood size), just as BIRCH has its threshold. I suspect you will not be able to improve on having another, approximating, parameter. You do not need to set n_clusters to a fixed value for BIRCH. You only need to provide another clusterer, which has its own parameters, although you should be able to experiment with different "global clusterers".
On 4 January 2018 at 11:04, Shiheng Duan <shiduan@ucdavis.edu> wrote:
Yes, it is an efficient method, still, we need to specify the number of clusters or the threshold. Is there another way to run hierarchy clustering on the big dataset? The main problem is the distance matrix. Thanks.
On Tue, Jan 2, 2018 at 6:02 AM, Olivier Grisel <olivier.grisel@ensta.org
wrote:
Have you had a look at BIRCH?
http://scikit-learn.org/stable/modules/clustering.html#birch
-- Olivier
_______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
_______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
_______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
_______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
participants (3)
-
Joel Nothman -
Olivier Grisel -
Shiheng Duan