Will look into it.Although I have problem generating cluster as my data is 14000x14000 distance_matrix and it says "Memory Error". I have 6GB RAM. Any insight on this error is welcomed. Regards On Tue, Feb 13, 2018 at 3:19 AM, federico vaggi <vaggi.federico@gmail.com> wrote:

[image: Boxbe] <https://www.boxbe.com/overview> This message is eligible for Automatic Cleanup! (vaggi.federico@gmail.com) Add cleanup rule <https://www.boxbe.com/popup?url=https%3A%2F%2Fwww.boxbe.com%2Fcleanup%3Fkey%3DWmN8Mni1Bb%252FE8vwryDuPZWhWHXFeTNQTCcULsb9jMjU%253D%26token%3DYb1l0t6dUur3BzQcPWoAtfZoRneTcSjBc7Hz71Vlw9rgeYey9pcoZFeiA382Ppwp1hXBAD8avWAhOWnB8n69OcpSzLkqvY%252BMRFSiHXcSsJwtcw0QnW%252BWD%252BK4P9E88owiW5dp7GSNsBxS0EzTLeEwNQ%253D%253D&tc_serial=36655412130&tc_rand=323288304&utm_source=stf&utm_medium=email&utm_campaign=ANNO_CLEANUP_ADD&utm_content=001> | More info <http://blog.boxbe.com/general/boxbe-automatic-cleanup?tc_serial=36655412130&tc_rand=323288304&utm_source=stf&utm_medium=email&utm_campaign=ANNO_CLEANUP_ADD&utm_content=001>

As a caveat, a lot of clustering algorithms assume that the distance matrix is a proper metric. If your distance is not a proper metric then the results might be meaningless (the narrative docs do a good job of discussing this).

On Mon, 12 Feb 2018 at 13:30 prince gosavi <princegosavi12@gmail.com> wrote:

...

Hi, Thanks for those tips Sebastian.That just saved my day.

Regards, Rajkumar

On Tue, Feb 13, 2018 at 12:44 AM, Sebastian Raschka <se.raschka@gmail.com

...

wrote:

...

[image: Boxbe] <https://www.boxbe.com/overview> This message is eligible for Automatic Cleanup! (se.raschka@gmail.com) Add cleanup rule <https://www.boxbe.com/popup?url=https%3A%2F%2Fwww.boxbe.com%2Fcleanup%3Fkey%3D0a2mz6HiALxmseA8EtEa3hg8FtAfQyTwNzLAvbS3JOk%253D%26token%3D8qZlnKU2OJ%252BeTscNUfA9PjpDKa2%252FZO8i9dvKkAyr7bKz%252Bi2MdFTFnLILfmhv4s3s%252Bva0Dy7LpRz63wO18BlP48DNIu3aSb%252FmxAVjQq1fCD0tDxFcxxdH2mq9Otany%252FdER3CzXyokyLg%253D&tc_serial=36653890807&tc_rand=854549477&utm_source=stf&utm_medium=email&utm_campaign=ANNO_CLEANUP_ADD&utm_content=001> | More info <http://blog.boxbe.com/general/boxbe-automatic-cleanup?tc_serial=36653890807&tc_rand=854549477&utm_source=stf&utm_medium=email&utm_campaign=ANNO_CLEANUP_ADD&utm_content=001>

...

Hi,

by default, the clustering classes from sklearn, (e.g., DBSCAN), take an [num_examples, num_features] array as input, but you can also provide the distance matrix directly, e.g., by instantiating it with metric='precomputed'

my_dbscan = DBSCAN(..., metric='precomputed') my_dbscan.fit(my_distance_matrix)

Not sure if it helps in that particular case (depending on how many zero elements you have), you can also use a sparse matrix in CSR format ( https://docs.scipy.org/doc/scipy-1.0.0/reference/ generated/scipy.sparse.csr_matrix.html).

Also, you don't need to for-loop through the rows if you want to compute the pair-wise distances, you can simply do that on the complete array. E.g.,

from sklearn.metrics.pairwise import cosine_distances from scipy import sparse

distance_matrix = cosine_distances(sparse.csr_matrix(X), dense_output=False)

where X is your "[num_examples, num_features]" array.

Best, Sebastian

...

On Feb 12, 2018, at 1:10 PM, prince gosavi <princegosavi12@gmail.com> wrote:

...

np.shape(distance_matrix)==(14000,14000)

I would like to know which clustering suits better and is there any need to process the data further to get it in the form so that a model can be applied. Also any performance tip as the matrix takes around 3-4 hrs of

I have generated a cosine distance matrix and would like to apply clustering algorithm to the given matrix. processing.

...

You can find my code here https://github.com/ maxyodedara5/BE_Project/blob/master/main.ipynb Code for READ ONLY PURPOSE. -- Regards _______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn

_______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn

-- Regards _______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn

_______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn

-- Regards