[scikit-learn] Need Urgent help please in resolving JobLibMemoryError

[Debabrata Ghosh]

Hi All,

                      Greetings !



I am getting JoblibMemoryError while executing a scikit-learn
RandomForestClassifier code. Here is my algorithm in short:



from sklearn.ensemble import RandomForestClassifier

from sklearn.cross_validation import train_test_split

import pandas as pd

import numpy as np

clf = RandomForestClassifier(n_estimators=5000, n_jobs=1000)

clf.fit(p_input_features_train,p_input_labels_train)


The dataframe p_input_features contain 134 columns (features) and 5 million
rows (observations). The exact *error message* is given below:


Executing Random Forest Classifier
Traceback (most recent call last):
  File "/home/user/rf_fold.py", line 43, in <module>
    clf.fit(p_features_train,p_labels_train)
  File "/var/opt/ lib/python2.7/site-packages/sklearn/ensemble/forest.py",
line 290, in fit
    for i, t in enumerate(trees))
  File
"/var/opt/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.py",
line 810, in __call__
    self.retrieve()
  File "/var/opt/lib
/python2.7/site-packages/sklearn/externals/joblib/parallel.py", line 757,
in retrieve
    raise exception
sklearn.externals.joblib.my_exceptions.JoblibMemoryError: JoblibMemoryError
___________________________________________________________________________
Multiprocessing exception:
...........................................................................

/var/opt/lib/python2.7/site-packages/sklearn/ensemble/forest.py in
fit(self=RandomForestClassifier(bootstrap=True, class_wei...te=None,
verbose=0,
            warm_start=False), X=array([[ 0.        ,  0.        ,
0.        , ....        0.        ,  0.        ]], dtype=float32),
y=array([[ 0.],
       [ 0.],
       [ 0.],
       ...,
       [ 0.],
       [ 0.],
       [ 0.]]), sample_weight=None)
    285             trees = Parallel(n_jobs=self.n_jobs,
verbose=self.verbose,
    286                              backend="threading")(
    287                 delayed(_parallel_build_trees)(
    288                     t, self, X, y, sample_weight, i, len(trees),
    289                     verbose=self.verbose,
class_weight=self.class_weight)
--> 290                 for i, t in enumerate(trees))
        i = 4999
    291
    292             # Collect newly grown trees
    293             self.estimators_.extend(trees)
    294

...........................................................................



Please can you help me to identify a possible resolution to this.


Thanks,

Debu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scikit-learn/attachments/20161209/2664cc4a/attachment.html>