Sunday, December 12, 2010

R Code Example for Neural Networks

See also NEURAL NETWORKS.

In this past June's issue of R journal, the 'neuralnet' package was introduced. I had recently been familiar with utilizing neural networks via the 'nnet' package (see my post on Data Mining in A Nutshell) but I find the neuralnet package more useful because it will allow you to actually plot the network nodes and connections. (it may be possible to do this with nnet, but I'm not aware of how).The neuralnet package was written primarily for multilayer perceptron architectures, which may be a limitation if you are interested in other architectures.

The data set used was a default data set found in the package 'datasets' and consisted of 248 observations and 8 variables:

"education"      "age"            "parity"         "induced"        "case"           "spontaneous"    "stratum"        "pooled.stratum"

The following code runs the network (with 2 hidden layers) classifying 'case'  (a binary variable) as a function of several independent varaibles. The neural network is estimated, and the results are stored in the data frame 'nn.'

nn <- neuralnet(
 case~age+parity+induced+spontaneous,
 data=infert, hidden=2, err.fct="ce",
 linear.output=FALSE)

The weight estimates can be obtained with the following command:

nn$result.matrix

And, the network can be plotted or visualized with the simple command:

plot(nn)

As can be seen below, the output weights correspond directly with the visualized network. While this tool may be more difficult to interpret with more complex network architectures, I find this simplified version very useful for demonstration/instructional purposes.

Example: The weight for the path from input 'age' to the first hidden layer is  -3.0689 (age.to.1layhid1) which can easily be found in the network diagram. After all inputs feed into hidden layer 1, the weight associated with the path from hidden layer 1(1layhid.1.to.case) to the output layer (which along with information from the other layers of the network will give us the classification of 'case') is -1001.15.


Estimated Network Weights

error
123.811
reached.threshold
0.009853
steps
16822
Intercept.to.1layhid1
0.535237
age.to.1layhid1
-3.0689
parity.to.1layhid1
2.262792
induced.to.1layhid1
30.50824
spontaneous.to.1layhid1
0.000961
Intercept.to.1layhid2
-5.52491
age.to.1layhid2
0.11538
parity.to.1layhid2
-1.98372
induced.to.1layhid2
2.550932
spontaneous.to.1layhid2
3.829613
Intercept.to.case
-1.7714
1layhid.1.to.case
-1001.15
1layhid.2.to.case
4.570324
   
Neural Network Visualization (click to enlarge)
There are many other options avaiable with the neural net package, I encourage you to read the article referenced below for more details.

Reference: neuralnet: Training of Neural Networks  by Frauke G¸nther and Stefan Fritsch The R Journal Vol. 2/1, June 2010

R-Code:

#  ------------------------------------------------------------------
#  |PROGRAM NAME: NEURALNET_PKG_R
#  |DATE: 12/3/10
#  |CREATED BY: MATT BOGARD 
#  |PROJECT FILE:   P:\R  Code References\Data Mining_R           
#  |----------------------------------------------------------------
#  | PURPOSE: DEMO OF THE 'neuralnet' PACKAGE AND OUTPUT INTERPRETATION              
#  | 
#  |  ADAPTED FROM:  neuralnet: Training of Neural Networks
#  |    by Frauke Günther and Stefan Fritsch The R Journal Vol. 2/1, June 2010 
#  |    ISSN 2073-4859  (LOCATED: P:\TOOLS AND REFERENCES (Copy)\R References\Neural Networks
#  | 
#  | 
#  |------------------------------------------------------------------
#  |DATA USED: 'infert' FROM THE 'datasets' LIBRARY              
#  |------------------------------------------------------------------
#  |CONTENTS:               
#  |
#  |  PART 1: get the data 
#  |  PART 2: train the network
#  |  PART 3: 
#  |  PART 4: 
#  |  PART 5: 
#  |------------------------------------------------------------------
#  |COMMENTS:               
#  |
#  |-----------------------------------------------------------------
#  |UPDATES:               
#  |
#  |
#  |------------------------------------------------------------------
 
#  *------------------------------------------------------------------*
#  | get the data
#  *------------------------------------------------------------------* 
 
library(datasets)
 
names(infert)
 
#  *------------------------------------------------------------------*
#  | train the network
#  *------------------------------------------------------------------* 
 
library(neuralnet)
 
nn <- neuralnet(
 case~age+parity+induced+spontaneous,
 data=infert, hidden=2, err.fct="ce",
 linear.output=FALSE)
 
#  *------------------------------------------------------------------*
#  | output training results
#  *------------------------------------------------------------------*  
 
# basic
nn
 
# reults options
names(nn)
 
 
# result matrix
 
nn$result.matrix
 
# The given data is saved in nn$covariate and
# nn$response as well as in nn$data for the whole data
# set inclusive non-used variables. The output of the
# neural network, i.e. the fitted values o(x), is provided
# by nn$net.result:
 
out <- cbind(nn$covariate,nn$net.result[[1]])
 
dimnames(out) <- list(NULL, c("age", "parity","induced","spontaneous","nn-output"))
 
head(out)
 
# generalized weights
 
# The generalized weight expresses the effect of each
# ovariate xi and thus has an analogous interpretation
# as the ith regression parameter in regression models.
# However, the generalized weight depends on all
# other covariates. Its distribution indicates whether
# the effect of the covariate is linear since a small variance
# suggests a linear effect
 
# The columns refer to the four covariates age (j =
# 1), parity (j = 2), induced (j = 3), and spontaneous (j=4)
 
head(nn$generalized.weights[[1]])
 
# visualization
 
plot(nn)

Created by Pretty R at inside-R.org

24 comments:

  1. I am a newbie. Would you mind writing all commands needed to run the example you gave? How to load the package and stuff? Many thanks

    ReplyDelete
  2. I will try to find the program and post the code and add some instructions. In the mean time I would refer you to http://journal.r-project.org/archive/2010-1/RJournal_2010-1_Guenther+Fritsch.pdf

    which is the link to the article I cited. It provides much more detail and instructions for loading the package etc.

    ReplyDelete
  3. I've now included the code I used, but I have to say, each time I run it I get different results. I'm not sure if there is an option for specifying a seed or some way to get consistent results. there is also a lot more I need to know about interpreting the output beyond what I've already posted. Probably time to give the journal article and documentation a second read.

    ReplyDelete
  4. Note- to read the code you may have to use the pretty R scroll bar, or you should be able to copy and paste into a text doc or editor.

    ReplyDelete
  5. Thanks Matt, this is very useful. Do you know how we can specify the maximum number of epochs or the target error rate?

    ReplyDelete
    Replies
    1. Hello Gayathri..
      I am starting to work with neuralnet.
      Did you discovered if it is possible specify the maximum number of epochs or the target error rate??

      Thank you..

      Delete
  6. @Gyathri- I'm not sure about this. I first learned about Neural nets via the Rattle Package in R, and the journal article I cited above. I would like to know more if I get time to dig into it. If you use twitter, you might send out a tweet with #rstats hash tag and ask about this. I might if I get a chance in the future. You could try the R help listserv, but I've never used it as I've heard so much about it not being user friendly or welcoming to anyone that is not an expert in R.

    ReplyDelete
  7. I write:

    >library(datasets)
    >names(infert)
    >nn <- nnet(case~age+parity+induced+spontaneous, data=infert, hidden=2, err.fct="ce", linear.output=FALSE)

    But error occurs:
    Erro em c(dim(x)[2L], size, dim(y)[2L]) : 'size' is missing

    How to solve? Thanks

    ReplyDelete
    Replies
    1. Use Size option, you will get the weights
      > nn<-nnet(case~age+parity+induced+spontaneous,data=infert, size=10, hidden=2,linear.output=FALSE)
      # weights: 61
      initial value 88.005015
      final value 83.000000
      converged

      Delete
  8. I have a question on how to predict future ts-values after a svm-regression:
    If I execute:

    library(e1071)
    library(dyn)
    y=c(0,2,1,3,2,4,3,5,4,6,5,7,6,8,7,9,8,10,9,11)
    y<-ts(y)
    y.svm <- dyn$svm(y ~ lag(y))
    yp <- predict(y.svm)
    ts.plot(y, yp, col = 1:2)

    I only get values for yp[1] .. yp[19]

    What do I have to do to get future values yp[20] .. yp[26] for example?

    ReplyDelete
  9. @ Anonymous & Anonymous- I'll have to look at these issues- I'm not sure I'll get the answer. I'm still just learning these techniques and the code too! But my purpose of this blog is to facilitate more learning. I can barely keep up with new posts, ( I have a lot more I want to write) not to mention addressing these issues.

    In the mean time, typically what I do when I have an issue like this, is I'll make a blog post and ask the question via twitter (with a link to the post like this one) using the #rstats hash tag. Stack overlfow is another web site that is useful to post questions about R-code, as well as the R-user group. If you have a linkedin profile, I notice that R-users are often using the linkedin R user interest group for these kind of discussions.

    If feel bad for not providing you with a direct answer, but hopefully one of these resources will be helpful.

    ReplyDelete
  10. Anonymous is not using the correct package ('neuralnet', rather than 'nnet' - I am guessing nnet doesn't have the dataset... After loading nnet I got the same error, but the post's neuralnet example ran fine.

    ReplyDelete
  11. How to test model using neural network in R..? Please help me..

    ReplyDelete
  12. You're right that people, notably Brian Ripley, can get snarky about poorly written questions, but I found a lot of people willing to help me when I was first learning R and most beginner questions already have answers. Try a google search like "R how to install package" etc. Also take a look at the posting guide before posting:

    http://www.r-project.org/posting-guide.html

    ReplyDelete
  13. i am getting below error, when i run above code

    > nn <- neuralnet(case~age+parity+induced+spontaneous, data=infert, hidden=2, err.fct="ce", linear.output=FALSE)
    Warning message:
    algorithm did not converge in 1 of 1 repetition(s) within the stepmax

    ReplyDelete
  14. Dear matt,

    I am trying to use Nnet for predicting continous data, wil it work? my dependent is a gamma distributed varible, so i used ur code but with little tweek for linout=true. but i am not getting results. please see the error below and advice me what other changes can be made

    > nn <- nnet(Low.formula~Type.of.property+Type.of.occupation+Job+Number.of.rooms, data=France1, size=1, skip=FALSE, maxit=1000,hidden=1, err.fct="ce", linout=TRUE)
    # weights: 12
    initial value 19575694.371960
    iter 10 value 1513776.253055
    iter 20 value 1205329.256042
    iter 30 value 1046068.800635
    iter 40 value 969627.109067
    iter 50 value 739564.881771
    iter 60 value 727653.907905
    iter 70 value 661323.506699
    final value 661205.493839
    converged
    > nn$result.matrix
    NULL
    > plot(nn)
    Error in xy.coords(x, y, xlabel, ylabel, log) :
    'x' is a list, but does not have components 'x' and 'y'
    > fix(nn)
    Warning message:
    In edit.default(get(subx, envir = parent), title = subx, ...) :
    deparse may be incomplete
    > weights(nn)
    NULL


    thank you

    ReplyDelete
  15. I have the same problems:

    > nn$result.matrix
    NULL

    or

    algorithm did not converge in 1 of 1 repetition(s) within the stepmax

    -------------------------------------
    Does this mean that algorithm didn't find any relationships between dependent variable and covariates?

    Thanks!

    ReplyDelete
  16. Can we use Neural net R packages for predicting Continuous data like prices,
    If yes what changes we need to make in the code u have given above.

    ReplyDelete
  17. From the perspective of R, how do we use the result of the neural network exercise? Do we come up with an equation? Does it work like a regression equation where, after the procedure, we can just plug in numbers to get a predicted value?

    ReplyDelete
  18. I am testing out your codes on a sample data that I have. Can you please share me some method of obtaining forecasts using the network thus obtained.

    ReplyDelete
  19. Anil, I really want to work on this when I get time. I've done it with SAS Enterprise Miner but have not demonstrated it in R. I will as soon as I can! Thanks.

    ReplyDelete
  20. Do you know how to interpret the error in the output? Thanks.

    ReplyDelete
  21. Are we sure that this package works properly? I am trying to figure it out with the simplest of data, and even though it gives the very known error: Warning message:
    algorithm did not converge in 1 of 1 repetition(s) within the stepmax

    or a huge Error in the network prediction that makes outputs have no sense. It doesn't even working with the simple data table when you have a first column a number and the second column is numbers' 2X and as output you want the sum of these two columns...

    ReplyDelete