Mailman 3 March 2017 - scikit-learn

Anomaly/Outlier detection based on user access for a large application
by John Doe March 20, 2017

March 20, 2017

Hi All, I am trying to solve a problem of finding Anomalies/Outliers using application logs of a large KMS. Please find the details below: *Problem Statement*: Find Anomalies/outliers using application access logs in an un-supervised learning environment. Basic use case is to find any suspicious activity by user/group, that deviates from a trend that the algorithm has learned. *Input Data*: Data would be created from log file that are in the following format: "ts, src_ip, decrypt, user_a, group_b, kms_region, key" Where: *ts* : time of access in epoch Eg: 1489840335 *decrypt* : is one of the various possible actions. *user_a*, *group_a* : are the user and group that did the access *kms_region* : the region in which the key exists *key* : the key that was accessed *Train Set*: This comes under the un-supervised learning and hence we cant have a "normal" training set which the model can learn. *Example of anomalies*: 1. User A suddenly accessing from a different IP: xx.yy 2. No. of access for a given key going up suddenly for a given user,key pair 3. Increased access on a generally quite long weekend 4. Increased access on a Thu (compared to last Thursdays) 5. Unusual sequences of actions for a given user. Eg. read, decrypt, delete in quick succession for all keys for a given user ------------------------ >From our research, we have come up with below list of algorithms that are applied to similar problems: - ARIMA <https://en.wikipedia.org/wiki/Autoregressive_integrated_moving_average> : This might be good for timeseries predicting, but will it also learn to flag anomalies like #3, #4, sequences of actions(#5) etc? - scikit-learn's Novelty and Outlier Detection <http://scikit-learn.org/stable/modules/outlier_detection.html> : Not sure if these will address #3, #4 and #5 use cases above. - Neural Networks - k-nearest neighbors - Clustering-Based Anomaly Detection Techniques: k-Means Clustering etc - Parametric Techniques <https://www.vs.inf.ethz.ch/edu/HS2011/CPS/papers/chandola09_anomaly-detecti…> (See Section 7): This might work well on continuous variables, but will it work on discrete features like, is_weekday etc? Also will it cover cases like #4 and #5 above? Most of the research I did were on problems that had continuous features and did not consider discrete variables like "Holiday_today?" / succession of events etc. Any feedback on the algorithm / technique that can be used for above usecases would be highly appreciated. Thanks. Regards, John.

1 0

recommended feature selection method to train an MLPRegressor
by Thomas Evangelidis March 20, 2017

March 20, 2017

Which of the following methods would you recommend to select good features (<=50) from a set of 534 features in order to train a MLPregressor? Please take into account that the datasets I use for training are small. http://scikit-learn.org/stable/modules/feature_selection.html And please don't tell me to use a neural network that supports the dropout or any other algorithm for feature elimination. This is not applicable in my case because I want to know the best 50 features in order to append them to other types of feature that I am confident that are important. cheers Thomas -- ====================================================================== Thomas Evangelidis Research Specialist CEITEC - Central European Institute of Technology Masaryk University Kamenice 5/A35/1S081, 62500 Brno, Czech Republic email: tevang(a)pharm.uoa.gr tevang3(a)gmail.com website: https://sites.google.com/site/thomasevangelidishomepage/

3 2

Regarding Adaboost classifier
by Afzal Ansari March 19, 2017

March 19, 2017

Hello Developers! I am currently working on feature extraction method which is based on Haar features for image classification. I am unable to find pure implementation of adaboost classifier algorithm on the internet even on scikit learn web. I need to train the classifier using adaboost classifier to obtain Haar features from image dataset. Please help me regarding this code. Reply soon. Thanks in advance.

4 6

Intermediate results using gridsearchCV?
by Carlton Banks March 19, 2017

March 19, 2017

Is it possible to receive intermediate the intermediate result of a gridsearchcv? instead getting the final result?

3 3

(no subject)
by Carlton Banks March 18, 2017

March 18, 2017

I am currently struggling with getting good results with my CNN in which i decided to optimize parameter using grid search. I am currently trying to use scikit-learn GridSearchCV. def create_model(init_mode='uniform',activation_mode='linear',optimizer_mode="adam", activation_mode_conv = 'linear'): model = Sequential() model.add(ZeroPadding2D((6,4),input_shape=(6,3,3))) model.add(Convolution2D(32,3,3 , activation=activation_mode_conv)) print model.output_shape model.add(Convolution2D(32, 3,3, activation=activation_mode_conv)) print model.output_shape model.add(MaxPooling2D(pool_size=(2,2),strides=(2,1))) print model.output_shape model.add(Convolution2D(64, 3,3 , activation=activation_mode_conv)) print model.output_shape model.add(Convolution2D(64, 3,3 , activation=activation_mode_conv)) print model.output_shape model.add(MaxPooling2D(pool_size=(2,2),strides=(2,1))) model.add(Flatten()) print model.output_shape model.add(Dense(output_dim=32, input_dim=64, init=init_mode,activation=activation_mode)) model.add(Dense(output_dim=13, input_dim=50, init=init_mode,activation=activation_mode)) model.add(Dense(output_dim=1, input_dim=13, init=init_mode,activation=activation_mode)) model.add(Dense(output_dim=1, init=init_mode, activation=activation_mode)) #print model.summary() model.compile(loss='mean_squared_error',optimizer=optimizer_mode) return model #reduce_lr=ReduceLROnPlateau(monitor='val_loss', factor=0.01, patience=3, verbose=1, mode='auto', epsilon=0.1, cooldown=0, min_lr=0.000000000000000001) #stop = EarlyStopping(monitor='val_loss', min_delta=0, patience=5, verbose=1, mode='auto') #log=csv_logger = CSVLogger('training_'+str(i)+'.csv') #print "Model Train" #hist_current = model.fit(np.array(data_train_input), # np.array(data_train_output), # shuffle=False, # validation_data=(np.array(data_test_input),np.array(data_test_output)), # validation_split=0.1, # nb_epoch=150000, # verbose=1, # callbacks=[reduce_lr,log,stop]) #print() #print model.summary() #print "Model stored" #model.save(spectogram_path+"Model"+str(feature)+".h5") #model.save_weights(spectogram_path+"Model"+str(feature)+"_weights.h5") #del model ## Make it work for other feature ranges ## Add the CNN part and test it ## Try with gabor kernels as suggested by the other paper.. input_train, input_test, output_train, output_test = model(0,train_input_data_interweawed_normalized[:-(len(train_input_data_interweawed_normalized)-1000)],output_data_train[:-(len(output_data_train)-1000)],test_input_data_interweawed_normalized[:-(len(test_input_data_interweawed_normalized)-1000)],output_data_test[:-(len(output_data_test)-1000)]) del test_input_data del test_name del test_input_data_normalized del test_name_normalized del test_input_data_interweawed del test_name_interweawed del test_input_data_interweawed_normalized del test_name_interweawed_normalized del train_input_data del train_name del train_input_data_normalized del train_name_normalized del train_input_data_interweawed del train_name_interweawed del train_input_data_interweawed_normalized del train_name_interweawed_normalized seed = 7 np.random.seed(seed) print "Regressor" model = KerasRegressor(build_fn = create_model, verbose = 10) init_mode_list = ['uniform', 'lecun_uniform', 'normal', 'zero', 'glorot_normal', 'glorot_uniform', 'he_normal', 'he_uniform'] activation_mode_list = ['softmax', 'softplus', 'softsign', 'relu', 'tanh', 'sigmoid', 'hard_sigmoid', 'linear'] activation_mode_list_conv = ['softplus', 'softsign', 'relu', 'tanh', 'sigmoid', 'hard_sigmoid', 'linear'] optimizer_mode_list = ['SGD', 'RMSprop', 'Adagrad', 'Adadelta', 'Adam', 'Adamax', 'Nadam'] batch_size_list = [10, 20, 40, 60, 80, 100] epochs = [10, 50, 100] param_grid = dict(init_mode=init_mode_list, batch_size=batch_size_list, nb_epoch=epochs, activation_mode=activation_mode_list, optimizer_mode = optimizer_mode_list, activation_mode_conv = activation_mode_list_conv) grid = GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=1) print "Grid fit" grid_result = grid.fit(np.asarray(input_train), np.array(output_train)) # summarize results print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_)) means = grid_result.cv_results_['mean_test_score'] stds = grid_result.cv_results_['std_test_score'] params = grid_result.cv_results_['params'] for mean, stdev, param in zip(means, stds, params): print("%f (%f) with: %r" % (mean, stdev, param)) This runs.. but problems with it that it only provides a result at the end. I ran the code once but then it crashed with this error message: cannot allocate memory for thread-local data: ABORT I am not sure what could cause this problem?

1 0

Is something wrong with this gridsearchCV?
by Carlton Banks March 16, 2017

March 16, 2017

I am currently using grid search to optimize my keras model… Something seemed a bit off during the training? https://www.dropbox.com/s/da0ztv2kqtkrfpu/Screenshot%20from%202017-03-16%20… <https://www.dropbox.com/s/da0ztv2kqtkrfpu/Screenshot%20from%202017-03-16%20…> For some reason is the training for each epoch not done for all datapoints?… What could be wrong? Here is the code: http://pastebin.com/raw/itJFm5a6 <http://pastebin.com/raw/itJFm5a6> Anything that seems off?

3 9

[GSoC 2017] First Draft, request for suggestions - Improve Online Learning of Linear Models.
by Karan Desai March 16, 2017

March 16, 2017

Hello developers, I'm Karan Desai, an Electrical Engineering Undergraduate at IIT Roorkee. I was following the community since October and initially planned to work on Pytest Migration idea. But on meticulous discussions, it was concluded that the migration task might be short for a three month wide timeline. Besides work is in progress on that. I particularly found the first project idea appealing, and went about gathering ingredients to make the perfect recipe for summers. Finally I can outline it as stated below. The description was quite short, so I will be happy to include more in it if need be. 1. There's a gradient descent optimizer, but I could not find an optimizer for adaptive learning strategies (I saw a method for adam in MLP though). So adding that can be a part of my project. 2. I looked into benchmarks directory, and checked a comparison of SGD against coordinate descent and ridge regression. Similar type of benching should be done with this new Optimizer/s as well. 3. There's a lack of multinomial logloss as mentioned in description (categorical cross entropy for classification tasks). I can work on adding that as well. As an addition, I can work on KL divergence, poisson and cosine proximity losses, to name a few. In my opinion, these are pretty standard and can be a nice to have. They already exist as metrics, just need to be ported to Cython and used as an optimization objective for linear classifiers. 4. About a tool to anneal learning rate: I suggest a new approach to look at this - as a callback. I searched through the documentation and I could not find this way of handling tidbits during training of models. We should be able to provide a callback to the constructor of a linear model which can do any dedicated job after every epoch, be it learning rate annealing, saving model checkpoint, getting custom verbose output, or as creative as uploading data to server for real time plots on any website. If this gets working in place, we can generalize this to many classes of scikit-learn. As a part of my project, I am planning to enrich scikit-learn to be shipping some ready made callback helpers for easy plug and play. I am still not sure whether this is sufficient for a three months timeline, because I am assuming the review cycles might take slightly longer time because of scikit-learn being such a huge community. As far as the math is concerned, I have searched for some good references, some of which are listed below: 1. First two points will heavily rely on @mblondel's lightning package, and this blog post: http://sebastianruder.com/optimizing-gradient-descent/ 2. For the losses (third point), I have seen the way existing losses are written in cython, as well as in the metrics submodule. That should help a lot. 3. About the fourth point, first of all I would be happy to get some suggestions from the community. Once satisfied, I should implement a very basic prototype with some existing class, maybe convert verbose logging of some class to a callback structure. Will include that in the second draft of proposal which would be a preliminary version of what I shall submit on GSoC website. More about me: 1. Github Profile: https://www.github.com/karandesai-96 2. GSoC 2016 Project: https://goo.gl/mdFZ6m 3. Joblib Contributions: https://git.io/vyMSx 4. Scikit-learn Contributions: https://git.io/vyMSF I'll be eagerly waiting for feedback. Thanks. Regards, Karan Desai, Department of Electrical Engineering, IIT Roorkee, India.

2 3

best way to scale on the random forest for text w bag of words ...
by Sasha Kacanski March 16, 2017

March 16, 2017

Hi, As soon as number of trees and features goes higher, 70Gb of ram is gone and i am getting out of memory errors. file size is 700Mb. Dataframe quickly shrinks from 14 to 2 columns but there is ton of text ... with 10 estimators and 100 features per word I can't tackle ~900 k of records ... Training set, about 15% of data does perfectly fine but when test come that is it. i can split stuff and multiprocess it but I believe that will simply skew results... Any ideas? -- Aleksandar Kacanski

3 4

PyParis 2017
by Gael Varoquaux March 16, 2017

March 16, 2017

The PyParis conference will be held in ... Paris! June 12-13: http://pyparis.org/cfp.html There will be a data track, that will be very scikit-learn related. Scikit-learn users or developers are most welcomed to come and talk about what they are doing. Cheers, Gaël -- Gael Varoquaux Researcher, INRIA Parietal NeuroSpin/CEA Saclay , Bat 145, 91191 Gif-sur-Yvette France Phone: ++ 33-1-69-08-79-68 http://gael-varoquaux.info http://twitter.com/GaelVaroquaux

1 0

GridsearchCV
by Carlton Banks March 16, 2017

March 16, 2017

Hi… I currently trying to optimize my CNN model using gridsearchCV, but seem to have some problems feading my input data.. My training data is stored as a list of Np.ndarrays of shape (6,3,3) and my output is stored as a list of np.array with one entry. Why am I having problems parsing my data to it? best regards Carl B.

3 10