To maintain my SQL skills I regularly practice them. I use the following sites:
https://sqlbolt.com/
https://sqlzoo.net/
http://www.sql-ex.ru/
http://www.sql-tutorial.ru/ru/book_sql_tips_and_solutions_10.html
http://sql-tutorial.ru/ru/book_computers_database.html
https://www.pgexercises.com/
http://www.sqlcourse.com/
online books with exercises
http://rsdn.org/res/book/db/sqltraining.xml
not much explanation here, only forums:
https://www.hackerrank.com/domains/sql/select
Example of technical interview:
https://habr.com/ru/post/181033/
https://www.youth4work.com/onlinetalenttest/Test-SQL-Server
Thursday, October 17, 2019
Friday, September 20, 2019
Practicing with the book "Spark: the Definitive Guide"
In a technical book club at meetup "San Diego Technology Immersion Group" we are reading a book about Spark. Here is a complete reference:
"Spark: The definitive guide" by Bill Chambers, Matei Zaharia(O'Reilly), Copyright 2018 Databricks, Inc., 978-1-491-91221-8
I was especially interested in Machine Learning part of the book. Unfortunately I learned that code provided in the book github account has some missing lines comparatively with the book, and in addition some of it does not work. So I made my source files and notebooks for chapters 24, 25 and 26. They are here:
https://github.com/Mathemilda/SparkTheDefinitiveGuideBook/tree/master/Part_IV_AdvAn%26ML
"Spark: The definitive guide" by Bill Chambers, Matei Zaharia(O'Reilly), Copyright 2018 Databricks, Inc., 978-1-491-91221-8
I was especially interested in Machine Learning part of the book. Unfortunately I learned that code provided in the book github account has some missing lines comparatively with the book, and in addition some of it does not work. So I made my source files and notebooks for chapters 24, 25 and 26. They are here:
https://github.com/Mathemilda/SparkTheDefinitiveGuideBook/tree/master/Part_IV_AdvAn%26ML
Thursday, February 28, 2019
Online Ad selection: Upper Confidence Bound method and Thompson Sampling method
The post is devoted to selection of the most popular ad to display on a webpage to gather the most clicks, or A/B test. The rate at which the webpage visitors click on an ad is called a conversion rate for the ad. The considered methods allow to make such selection in real time, so the most popular ads are showed as early as possible.
Assume that we have several ads and a place on a webpage to show one of them. We can display them one by one, record all the clicks, analyze the results afterwards and figure out the most popular. But an ad display may be pricey. It would be more efficient to estimate rates in real time and to display the most popular one as soon as rates can be compared. Especially if an ad leads to a page for a visitor to buy something. There are couple of method for such estimations: Upper Confidence Bound method and Thompson Sampling method.
The first one is based on an confidence interval concept which is studied in a Statistics course and has a good intuitive explanation. Roughly speaking a confidence interval is a numeric interval were our value is supposed to lie with some probability, usually 95%. (The real statistical definition is more technical and means not quite this, but in practice the above explanation is close enough.) During our ad displaying we can compute average rates at each step with corresponding confidence intervals and pick up for next display an ad with a highest upper confidence bound. For the picked ad its mean an interval get recalculated and as result the confidence interval shrinks a little. You can see how it happens in the video below. The intervals are colored by ad legend, and their upper bounds are the right sides of corresponding rectangles. Calculated means are marked by vertical lines.
As you see since a random value has more probability to appear where our curve is higher, then it get closer and closer to its mean at each step. With each step an interval where a distribution curve appears above horizontal axis shrinks. You might view the interval as a confidence interval analogue.
When we have several ads we compare the drawn random values at every step and pick up an ad to display with the greatest value. Here how it works for a few ads (I dropped means to make picture more clear):
Plus it accommodates an additional ad in the middle of the process more easily, taking less time to recover.
Do not hesitate to ask questions and point our mistakes!
Assume that we have several ads and a place on a webpage to show one of them. We can display them one by one, record all the clicks, analyze the results afterwards and figure out the most popular. But an ad display may be pricey. It would be more efficient to estimate rates in real time and to display the most popular one as soon as rates can be compared. Especially if an ad leads to a page for a visitor to buy something. There are couple of method for such estimations: Upper Confidence Bound method and Thompson Sampling method.
The first one is based on an confidence interval concept which is studied in a Statistics course and has a good intuitive explanation. Roughly speaking a confidence interval is a numeric interval were our value is supposed to lie with some probability, usually 95%. (The real statistical definition is more technical and means not quite this, but in practice the above explanation is close enough.) During our ad displaying we can compute average rates at each step with corresponding confidence intervals and pick up for next display an ad with a highest upper confidence bound. For the picked ad its mean an interval get recalculated and as result the confidence interval shrinks a little. You can see how it happens in the video below. The intervals are colored by ad legend, and their upper bounds are the right sides of corresponding rectangles. Calculated means are marked by vertical lines.
The method has some drawbacks. It does not take into account that our rates must be between 0 and 1, so initial confidence intervals usually wider than the [0, 1] segment. It means that we waste some time on getting realistic values for our intervals. The worst thing is that if after a start of the displaying we throw in an additional ad then the process takes a lot of time to recover.
Here is another method which is more efficient, Thompson Sampling Method. We assume Beta distributions for each ad rate and instead of computing averages draw a random number in accordance with each distribution for every step. Beta distributions are restricted to [0, 1] segment, so we do not have to wait for our numbers to fall into it. With each data point the beta distribution curve appears closer to the conversion rate mean. There is a picture how it goes for one ad, with a blue vertical line marking a mean and the red line for a random value:
When we have several ads we compare the drawn random values at every step and pick up an ad to display with the greatest value. Here how it works for a few ads (I dropped means to make picture more clear):
Plus it accommodates an additional ad in the middle of the process more easily, taking less time to recover.
Do not hesitate to ask questions and point our mistakes!
Tuesday, November 6, 2018
San Diego Map for Average Water Bacteria Counts at Measurement Stations
I have been working with San Diego Water quality data project:
https://www.sandiegodata.org/2018/04/summer-water-quality-data-project/
Here are data sets:
https://data.sandiegodata.org/dataset?tags=water-project
Regretfully my complete works do not fit into the blog post (or even a few posts) because of a post size restriction (1 Mb). Here is a repository for it: https://github.com/san-diego-water-quality/MyaBakhova
In particular I created a number of maps. You can see below one of them, where sizes of station marks correlate to pollution amounts:
https://www.sandiegodata.org/2018/04/summer-water-quality-data-project/
Here are data sets:
https://data.sandiegodata.org/dataset?tags=water-project
Regretfully my complete works do not fit into the blog post (or even a few posts) because of a post size restriction (1 Mb). Here is a repository for it: https://github.com/san-diego-water-quality/MyaBakhova
In particular I created a number of maps. You can see below one of them, where sizes of station marks correlate to pollution amounts:
Sunday, June 25, 2017
Neural Networks as a Corporation Chain of Command
Neural Networks are considered as complicated and they are always explained using neurons and a brain function. But we do not need to learn how brain works to understand Neural Networks structure and how they operate.
Let us start with a logistic regression. Recall that for a logistic regression model we represent our data as point coordinates. The model allows us to split all the points into 2 classes. Visually the model divides 2 classes of data records by a line (or a plane, or a hyperplane if we have higher dimensions) .
Numerically a logistic regression yields values from 0 to 1 for each data point. For every record we compute a probability for it to belong to one class or another and we can consider the process as making an evaluation. A value close to 1 means one class, while close to 0 means another. In the process we load data and calculate our evaluation by a formula.
For example we may have the following assignment: to compute if we have enough goods in warehouses to last for a week of sales. This is quite a common problem, and say inventory clerks report amounts of each item to a manager. The manager collects information, processes it and makes an evaluation: do we have enough of every product, "Yes" or "No".
Here we can use a logistic regression model which produces probabilities for 2 classes: "Yes" and "No". If you feel that our manager should supply a bit more information with it, for example, how much should be restocked if any, then in Neural Networks it could passed, too.
Let us start with a logistic regression. Recall that for a logistic regression model we represent our data as point coordinates. The model allows us to split all the points into 2 classes. Visually the model divides 2 classes of data records by a line (or a plane, or a hyperplane if we have higher dimensions) .
Numerically a logistic regression yields values from 0 to 1 for each data point. For every record we compute a probability for it to belong to one class or another and we can consider the process as making an evaluation. A value close to 1 means one class, while close to 0 means another. In the process we load data and calculate our evaluation by a formula.
For example we may have the following assignment: to compute if we have enough goods in warehouses to last for a week of sales. This is quite a common problem, and say inventory clerks report amounts of each item to a manager. The manager collects information, processes it and makes an evaluation: do we have enough of every product, "Yes" or "No".
Here we can use a logistic regression model which produces probabilities for 2 classes: "Yes" and "No". If you feel that our manager should supply a bit more information with it, for example, how much should be restocked if any, then in Neural Networks it could passed, too.
Usually one evaluation is not the only required indicator for a chain of stores to operate. In addition we need to know, for example, if our warehouses are full to optimal capacity (75% - 85% or something like this). Therefore we need to estimate this as well. In our organization we will have another manager gathering data and assessing if we have enough reserve storage space.
Then let us say that these people should report to their supervisor who will make next evaluation about store preparedness level:
So we get a whole hierarchy of managers, submitting their assessments and at the end the company CEO obtains a summary report. The CEO looks at the whole picture, issues adjusted guidelines for further work and sends them back to middle management. We can compare it with a neural network structure:
Now we can observe a lot of in common with a corporation chain of command. As we see middle managers are corporation hidden layers which do the balk of the job, as it happens in real life. They got reports from their subordinates and guidelines form their bosses, and they optimize their work based on the provided information. In neural networks we have the similar information flow and processing which is analogous to forward propagation (from lowest workers to CEO) and backward propagation (instructions from CEO to middle management, adapted for each lower manager by her/his boss).
I discovered that a lot
of people express a desire for the NN results to be more explainable by
its inner functioning. I believe that my analogy is helpful for this,
too. As
we know, a company success is defined by its top management. They
select a company structure and define an inner company policy, which middle management
adapts and directs further by usual means. We know that an industry
leader could originate due to a chance, a lucky initial idea. But its
following performance is a learned behavior. People use experience of
other company managers combining it with their own trial & error
process.
Wednesday, January 11, 2017
Machine Prediction for Human Resources Analytics
Introduction
Here I work on kaggle data set "Human Resources Analytics", which can be found by urlhttps://www.kaggle.com/ludobenistant/hr-analytics.
We're supposed to predict which valuable employees will leave next. Fields in the data set include:
- Employee satisfaction level
- Last evaluation
- Number of projects
- Average monthly hours
- Time spent at the company
- Whether they have had a work accident
- Whether they have had a promotion in the last 5 years
- Department
- Salary
- Whether the employee has left
Data Load
I start with loading data set and looking at its properties: dimensions, column names and types of data.dt=read.csv("HR_comma_sep.csv", stringsAsFactors=F) str(dt)
## 'data.frame': 14999 obs. of 10 variables: ## $ satisfaction_level : num 0.38 0.8 0.11 0.72 0.37 0.41 0.1 0.92 0.89 0.42 ... ## $ last_evaluation : num 0.53 0.86 0.88 0.87 0.52 0.5 0.77 0.85 1 0.53 ... ## $ number_project : int 2 5 7 5 2 2 6 5 5 2 ... ## $ average_montly_hours : int 157 262 272 223 159 153 247 259 224 142 ... ## $ time_spend_company : int 3 6 4 5 3 3 4 5 5 3 ... ## $ Work_accident : int 0 0 0 0 0 0 0 0 0 0 ... ## $ left : int 1 1 1 1 1 1 1 1 1 1 ... ## $ promotion_last_5years: int 0 0 0 0 0 0 0 0 0 0 ... ## $ sales : chr "sales" "sales" "sales" "sales" ... ## $ salary : chr "low" "medium" "medium" "low" ...
sapply(dt, function(x) sum(is.na(x)))
## satisfaction_level last_evaluation number_project ## 0 0 0 ## average_montly_hours time_spend_company Work_accident ## 0 0 0 ## left promotion_last_5years sales ## 0 0 0 ## salary ## 0
We can check pairwise plots, correlations and densities.
library(ggplot2) library(GGally) ggpairs(data=dt, columns=1:8, mapping = aes(color = "navy"), axisLabels="show")
Now let us look at what the values in last 2 columns.unique(dt$salary)
## [1] "low" "medium" "high"
unique(dt$sales)
## [1] "sales" "accounting" "hr" "technical" "support" ## [6] "management" "IT" "product_mng" "marketing" "RandD"
table(dt$sales)
## ## accounting hr IT management marketing product_mng ## 767 739 1227 630 858 902 ## RandD sales support technical ## 787 4140 2229 2720
table(dt$salary)
## ## high low medium ## 1237 7316 6446
library(dummies) dumdt=dummy(dt$salary) dumdt=as.data.frame(dumdt) names(dumdt)
## [1] "MachinePredictionForNorthropGrumman.Rhtmlhigh" ## [2] "MachinePredictionForNorthropGrumman.Rhtmllow" ## [3] "MachinePredictionForNorthropGrumman.Rhtmlmedium"
dumdt=dumdt[,-2] names(dumdt)=c("high_salary", "medium_salary")
dt=cbind(dt,dumdt) dt$salary=NULL str(dt)
## 'data.frame': 14999 obs. of 11 variables: ## $ satisfaction_level : num 0.38 0.8 0.11 0.72 0.37 0.41 0.1 0.92 0.89 0.42 ... ## $ last_evaluation : num 0.53 0.86 0.88 0.87 0.52 0.5 0.77 0.85 1 0.53 ... ## $ number_project : int 2 5 7 5 2 2 6 5 5 2 ... ## $ average_montly_hours : int 157 262 272 223 159 153 247 259 224 142 ... ## $ time_spend_company : int 3 6 4 5 3 3 4 5 5 3 ... ## $ Work_accident : int 0 0 0 0 0 0 0 0 0 0 ... ## $ left : int 1 1 1 1 1 1 1 1 1 1 ... ## $ promotion_last_5years: int 0 0 0 0 0 0 0 0 0 0 ... ## $ sales : chr "sales" "sales" "sales" "sales" ... ## $ high_salary : int 0 0 0 0 0 0 0 0 0 0 ... ## $ medium_salary : int 0 1 1 0 0 0 0 0 0 0 ...
dumdt=dummy(dt$sales) dumdt=as.data.frame(dumdt) names(dumdt)
## [1] "MachinePredictionForNorthropGrumman.Rhtmlaccounting" ## [2] "MachinePredictionForNorthropGrumman.Rhtmlhr" ## [3] "MachinePredictionForNorthropGrumman.RhtmlIT" ## [4] "MachinePredictionForNorthropGrumman.Rhtmlmanagement" ## [5] "MachinePredictionForNorthropGrumman.Rhtmlmarketing" ## [6] "MachinePredictionForNorthropGrumman.Rhtmlproduct_mng" ## [7] "MachinePredictionForNorthropGrumman.RhtmlRandD" ## [8] "MachinePredictionForNorthropGrumman.Rhtmlsales" ## [9] "MachinePredictionForNorthropGrumman.Rhtmlsupport" ## [10] "MachinePredictionForNorthropGrumman.Rhtmltechnical"
dumdt=dumdt[,-10] names(dumdt)=c("accounting","hr","IT","management","marketing","product_mng","RandD","sales","support") dt=cbind(dt,dumdt) dt$sales=NULL # Look at the new data frame: str(dt)
## 'data.frame': 14999 obs. of 19 variables: ## $ satisfaction_level : num 0.38 0.8 0.11 0.72 0.37 0.41 0.1 0.92 0.89 0.42 ... ## $ last_evaluation : num 0.53 0.86 0.88 0.87 0.52 0.5 0.77 0.85 1 0.53 ... ## $ number_project : int 2 5 7 5 2 2 6 5 5 2 ... ## $ average_montly_hours : int 157 262 272 223 159 153 247 259 224 142 ... ## $ time_spend_company : int 3 6 4 5 3 3 4 5 5 3 ... ## $ Work_accident : int 0 0 0 0 0 0 0 0 0 0 ... ## $ left : int 1 1 1 1 1 1 1 1 1 1 ... ## $ promotion_last_5years: int 0 0 0 0 0 0 0 0 0 0 ... ## $ high_salary : int 0 0 0 0 0 0 0 0 0 0 ... ## $ medium_salary : int 0 1 1 0 0 0 0 0 0 0 ... ## $ accounting : int 0 0 0 0 0 0 0 0 0 0 ... ## $ hr : int 0 0 0 0 0 0 0 0 0 0 ... ## $ IT : int 0 0 0 0 0 0 0 0 0 0 ... ## $ management : int 0 0 0 0 0 0 0 0 0 0 ... ## $ marketing : int 0 0 0 0 0 0 0 0 0 0 ... ## $ product_mng : int 0 0 0 0 0 0 0 0 0 0 ... ## $ RandD : int 0 0 0 0 0 0 0 0 0 0 ... ## $ sales : int 1 1 1 1 1 1 1 1 1 1 ... ## $ support : int 0 0 0 0 0 0 0 0 0 0 ...
rm(dumdt)
Machine Learning
Now my data is ready for prediction. I split the data set into 2 sets: for training and for testing.indeces=sample(1:dim(dt)[1], round(.7*dim(dt)[1])) train=dt[indeces,] test=dt[-indeces,]
str(train)
## 'data.frame': 10499 obs. of 19 variables: ## $ satisfaction_level : num 0.71 0.73 0.74 0.41 0.1 0.88 0.71 0.76 0.85 0.93 ... ## $ last_evaluation : num 0.5 0.6 0.55 0.49 0.94 1 0.92 0.8 0.65 0.65 ... ## $ number_project : int 4 3 6 2 6 5 3 4 4 4 ... ## $ average_montly_hours : int 253 137 130 130 255 219 202 226 280 212 ... ## $ time_spend_company : int 3 3 2 3 4 6 4 5 3 4 ... ## $ Work_accident : int 0 0 0 0 0 1 0 0 1 0 ... ## $ left : int 0 0 0 1 1 1 0 0 0 0 ... ## $ promotion_last_5years: int 0 0 0 0 0 0 0 0 0 0 ... ## $ high_salary : int 0 0 0 0 0 0 0 0 0 0 ... ## $ medium_salary : int 1 1 1 0 0 0 0 0 1 1 ... ## $ accounting : int 0 0 0 0 0 0 0 0 0 0 ... ## $ hr : int 0 1 0 0 0 0 0 0 0 0 ... ## $ IT : int 0 0 0 0 0 0 0 0 1 1 ... ## $ management : int 0 0 0 0 0 0 0 0 0 0 ... ## $ marketing : int 0 0 0 1 0 0 0 0 0 0 ... ## $ product_mng : int 0 0 0 0 0 0 0 0 0 0 ... ## $ RandD : int 1 0 0 0 0 0 0 0 0 0 ... ## $ sales : int 0 0 0 0 0 0 0 0 0 0 ... ## $ support : int 0 0 1 0 0 0 0 1 0 0 ...
Logistic Regression
glm_mod=glm(left~., data=train, family=binomial) options(width=120) summary(glm_mod)
## ## Call: ## glm(formula = left ~ ., family = binomial, data = train) ## ## Deviance Residuals: ## Min 1Q Median 3Q Max ## -2.252360 -0.669234 -0.404160 -0.118092 3.005301 ## ## Coefficients: ## Estimate Std. Error z value Pr(>|z|) ## (Intercept) 0.513671571 0.152904509 3.35943 0.00078104 *** ## satisfaction_level -4.102569262 0.116375474 -35.25287 < 2.22e-16 *** ## last_evaluation 0.636893223 0.177888069 3.58030 0.00034320 *** ## number_project -0.315622781 0.025421761 -12.41546 < 2.22e-16 *** ## average_montly_hours 0.004809019 0.000613495 7.83872 4.5515e-15 *** ## time_spend_company 0.270089407 0.018619140 14.50601 < 2.22e-16 *** ## Work_accident -1.489204860 0.104868336 -14.20071 < 2.22e-16 *** ## promotion_last_5years -1.268547473 0.277706609 -4.56794 4.9254e-06 *** ## high_salary -1.903687581 0.153808902 -12.37697 < 2.22e-16 *** ## medium_salary -0.485601493 0.054435288 -8.92071 < 2.22e-16 *** ## accounting -0.216824268 0.130317943 -1.66381 0.09615045 . ## hr 0.177414039 0.129105528 1.37418 0.16938628 ## IT -0.271998822 0.110571123 -2.45994 0.01389585 * ## management -0.523601663 0.162647148 -3.21925 0.00128527 ** ## marketing -0.098831483 0.127918381 -0.77261 0.43975108 ## product_mng -0.272539608 0.124254440 -2.19340 0.02827862 * ## RandD -0.599234520 0.144357350 -4.15105 3.3095e-05 *** ## sales -0.129873866 0.078066780 -1.66363 0.09618735 . ## support -0.048540168 0.089959935 -0.53958 0.58948988 ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 ## ## (Dispersion parameter for binomial family taken to be 1) ## ## Null deviance: 11544.393 on 10498 degrees of freedom ## Residual deviance: 9035.048 on 10480 degrees of freedom ## AIC: 9073.048 ## ## Number of Fisher Scoring iterations: 5
fittedValues=fitted(glm_mod) # How well is predicted people who stayed hist(fittedValues[train$left==0])
# And the ones who left. hist(fittedValues[train$left==1])
glm_predictions=predict(glm_mod,test[,-match("left", names(test))]) # How well is predicted people who stayed, colored red hist(glm_predictions[test$left==0], breaks =15, col=2 ) # And the ones who stayed, colored blue hist(glm_predictions[test$left==1],breaks =15, col=rgb(0,0.7,0.8,1/2),add=T)
Wednesday, November 30, 2016
MNIST set with Neural Networks using H2O
I continue doing my ML work with MNIST set, currently presented at kaggle as Digit Recognizer competition, which I’ve started in one of my previous posts. This time I’ve decided to try neural network method. At first I had taken a look at “nnet” and “neuralnet” packages, but they could not handle such big set. Not only memory and timing had been a problem, but there are default restrictions, like only one hidden layer and a number of nodes. The number of nodes may be increased, but I got memory overload.
I googled if there is anything new for NN with R. Found two frameworks, MXNET and H2O. Decided to try H2O first, because it looked simpler.
Both packages cannot be installed using usual R command “install.packages”. For H2O you can find installation instructions on the company web site. You may get a message that JAVA installation is required. MXNET installation instructions are more complicated and you may need to adapt what you google. I posted my story with it in my blog post here.
Now let us start predicting! At first we load the data set, check dimenstions and prepare target variable.
setwd("/home/mya/Kaggle/DigitRecognizer")
dt=read.csv("train.csv", stringsAsFactors=F)
## For classification you need your output as a factor
dim(dt)## [1] 42000 785dt[,1] = as.factor(dt[,1]) # for classificationTo work with H2O we should not only load its library, but to initialize it as well and convert our data to H2O format.
library(h2o)
localH2O = h2o.init(max_mem_size = '16g', # use 16GB of RAM of 32GB available
nthreads = 7) # use 7 CPUs##
## H2O is not running yet, starting it now...
##
## Note: In case of errors look at the following log files:
## /tmp/Rtmp8Uek1a/h2o_mya_started_from_r.out
## /tmp/Rtmp8Uek1a/h2o_mya_started_from_r.err
##
##
## Starting H2O JVM and connecting: .. Connection successful!
##
## R is connected to the H2O cluster:
## H2O cluster uptime: 1 seconds 643 milliseconds
## H2O cluster version: 3.10.0.6
## H2O cluster version age: 3 months and 4 days
## H2O cluster name: H2O_started_from_R_mya_oon021
## H2O cluster total nodes: 1
## H2O cluster total memory: 14.22 GB
## H2O cluster total cores: 8
## H2O cluster allowed cores: 7
## H2O cluster healthy: TRUE
## H2O Connection ip: localhost
## H2O Connection port: 54321
## H2O Connection proxy: NA
## R Version: R version 3.3.2 (2016-10-31)train_h2o = as.h2o(dt)##
|
| | 0%
|
|=================================================================| 100%## Now we are ready to build a model.
s <- proc.time()
## train model
model =
h2o.deeplearning(x = 2:785, # column numbers for predictors
y = 1, # a column number for label
training_frame = train_h2o, # data in H2O format
activation = "RectifierWithDropout", # activation function
loss = "CrossEntropy", #loss function
input_dropout_ratio = 0.1, # % of inputs dropout
hidden_dropout_ratios = c(0.2,0.2), # % for nodes dropout
balance_classes = TRUE, # for classificaton
hidden = c(300,100), # two layers 300 x 100 nodes
quiet_mode=T, # to reduce printed output
nesterov_accelerated_gradient = T, # use it for speed
epochs = 200) # no. of forward and backward propagations##
|
| | 0%
|
|= | 1%
|
|= | 2%
|
|== | 3%
|
|== | 4%
|
|=== | 4%
|
|=== | 5%
|
|==== | 6%
|
|==== | 7%
|
|===== | 7%
|
|===== | 8%
|
|====== | 9%
|
|====== | 10%
|
|======= | 10%
|
|======= | 11%
|
|======== | 12%
|
|======== | 13%
|
|========= | 13%
|
|========= | 14%
|
|========== | 15%
|
|========== | 16%
|
|=========== | 17%
|
|=================================================================| 100%s - proc.time()## user system elapsed
## -4.176 -0.120 -558.636It took about 10 minutes.
h2o.confusionMatrix(model)## Confusion Matrix: vertical: actual; across: predicted
## 0 1 2 3 4 5 6 7 8 9 Error Rate
## 0 1077 0 0 0 0 1 2 6 0 0 0.0083 = 9 / 1,086
## 1 0 1021 4 1 0 0 0 5 1 0 0.0107 = 11 / 1,032
## 2 0 0 959 0 0 0 2 11 0 0 0.0134 = 13 / 972
## 3 0 0 6 998 0 2 0 14 0 2 0.0235 = 24 / 1,022
## 4 0 0 3 0 986 0 1 5 0 3 0.0120 = 12 / 998
## 5 0 0 0 6 0 997 3 9 1 1 0.0197 = 20 / 1,017
## 6 1 0 3 0 0 1 981 21 0 0 0.0258 = 26 / 1,007
## 7 0 1 3 1 1 0 0 995 0 0 0.0060 = 6 / 1,001
## 8 0 2 2 1 0 4 0 9 952 1 0.0196 = 19 / 971
## 9 0 0 0 2 2 0 0 10 1 976 0.0151 = 15 / 991
## Totals 1078 1024 980 1009 989 1005 989 1085 955 983 0.0154 = 155 / 10,097Because I’m working with a kaggle set I’m supposed to submit my prediction for test set on their site. For this I need to load it, convert it to H2O format as well, make predictions and convert results back to R format. Aftewards I will shut down H2O instance and write a submission file.
I’m making this markdown file with RStudio, and it means that at first I need to go back to the directory where all my data are stored.
setwd("/home/mya/Kaggle/DigitRecognizer")
test=read.csv("test.csv", stringsAsFactors=F)
test_h2o = as.h2o(test)##
|
| | 0%
|
|=================================================================| 100%## classify test set
h2o_y_test <- h2o.predict(model, test_h2o)##
|
| | 0%
|
|======================================== | 62%
|
|=================================================================| 100%## convert H2O format into data frame and save as csv
df_y_test = as.data.frame(h2o_y_test)
df_y_test = data.frame(ImageId = seq(1,length(df_y_test$predict)),
Label = df_y_test$predict)
## shut down virutal H2O cluster
h2o.shutdown(prompt = F)## [1] TRUEwrite.csv(df_y_test, file = "H20_submission.csv", row.names=F)My submission scored 0.95900. Not as good as Random Forests.
Subscribe to:
Posts (Atom)