CS 527A, Spring 2002, Homework 3

A signed cover sheet for Homework 3 must be submitted with your homework.

1. (Up to 100 points) Implement the delta(LMS) training rule and the perceptron training rule for a two input linear unit. Train both to fit the target concept x₁ + 2 x₂ > 2.
   You can select each training example (x₁,x₂) where x₁ and x₂ are selected uniformly form [-1000,1000] or pick a different distribution if you want. Do whichever you choose from among the following (for the given point value):
   • (40 points) For both the delta and perceptron training rule, plot the squared error E (using both the training data and separate test data generated in the same manner) as a function of the number of training iterations. Also for each, plot the decision surface after 5, 10, 50, 100, ..., iterations.

   • (20 points) Now add labeling errors by giving the wrong label 10% of the time. That is compute the correct label as before but now with probablity 0.1 put the wrong label in the training data. Repeat the first part using this noisy data and discusss your findings.

   • (20 points) For the delta rule with noisy data, try a few different constant values for eta (the learning rate) and also try using a decaying learning rate of eta₀/i for the ith iteration where eta₀=0.1. Report on your findings.

   • (20 points) Do one more experiment of your choice. However, it should be of comparable depth as the experiment in varying the learning rate. As just a one example idea you could compare the performance of stochastic versus pure gradient descent across a range of settings.

2. (100 points) Compare the DT algorithm (with pruning) and neural networks for learning a rectangle in the plane (i.e. positive points are those in the rectangle and the rest are negative). For the DT you can use the code provided with HW2 and would just need to create the ssv file. For the neural network you can either write the code or use the code provided with Problem 6 below. You will need to think about the best way to apply a neural network to this problem and may decide to try a few variations and compare them. For example, what output units will you introduce? Do you want to have hidden units or just have the input units directly connected with the output units. Also, think about what you want to use for an error function to make it differentiable. You could consider both applying the standard sigmoid squashing funtion or redefining the error term to be continuous (such as defining the label as the distance from the center of the box). Report on which of the approaches you tried for the neural net worked best and how this compared to the performance of the DT? You should include a discussion about how training time differs. You are expected either to try two different options for the neural net architecture or two different options for what to use as the error function for the gradient search.

3. (20 pts) Derive a gradient descent training rule for a single unit with output o, where
   o = w₀ + w₁( x₁ + x₁²) + ... + w_n (x_n + x_n²).
   What are the tradeoffs between using this non-linear unit versus that for the standard perceptron? You should give an in-depth discussion about the tradeoffs which should rely upon the resulting training rule. This discussion is worth half of the points.

4. (10 pts) Consider the alternate error function that includes a penalty for large weights as given in class (and also shown in Problem 4.10 of the text). Derive the gradient descent update rule for this error function. Show it can be implemented by multiplying each weight by some constant before performing the standard gradient descent update.

5. (60 pts) Apply neural nets/backpropogation to the task of face recognition. Code for doing back propagation, some images and an assignment/instructions document (in postscript) from Tom Mitchell is available to guide you and also documents the provided code. You should do the required portions of that homework.

   I have placed the code and the quarter-size images on the web page (zipped up) in faces.zip to make it easier for you to copy everything. Save this into your directory. Then to extract the quarter-size images use tar xvf faces_4.tar. Full size images, additional image sources, and the trainset directory mentioned in the above assignment/instructions document can all be found at http://www.cs.cmu.edu/afs/cs.cmu.edu/user/mitchell/ftp/faces.html

6. (40 pts) Now do two of the "extra credit" options from the handout for Problem 5. If you would like to work in a group that is fine but then you must do more involved option sor two options per group member. Note, that if you choose to work as a group here you must still all do Problem 5 individually.

7. (30 points) Read one of the following papers and write a paper critique follwing these guidelines. You will be required to have a conference with Dr. Goldman to discuss the paper and part of your grade will be based on this conference.
   • Thomas G. Dietterich and Ghulum Bakiri (1995).
     A comparison of ID3 and backpropagation for English text-to-speech mapping. Machine Learning, 18(1), 263-286.

   • Y. LeCun, et al. (1995).
     Learning algorithms for classification: A comparison on handwritten digit recognition. Neural Networks: The Statistical Mechanics Perspective, pages 261-276.

   • Y. LeCun, et al. (1990).
     Handwritten digit recognition with a back-propagation network. In NIPS '90.

8. CHOOSE YOUR OWN ADVENTURE. You can propose any additional homework options (or variations of those given above) to Dr. Goldman. If approved a point value will be given.