CS 527A: Homework 1

CS 527A Homework 1

You are expected to complete 40 points worth of homework problems. For those selecting 10 and 20 point problems, you must select problems from at least two of the chapters. Also, no more than one paper critique can be selected.
If you are doing a 20 or 40 point problem be sure to attach the appropriate cover sheet and review the guidelines given there and in the course information handout.
If you are interested in doing a group project talk to Dr. Goldman.
Due on Wednesday February 7th.

Cover Sheets:

(10 pts) Prove that the LMS weight update rule (described in Chapter 1 of the text) performs a gradient descent to minimize the squared error. See Exercise 1-3 in the text for more about this problem including a description of gradient descent.
(10 pts) Consider alternative strategies for the Experiment Generator module of Figure 1.1 of the text. In particular, consider strategies in which the Experiment Generator suggest new board positions by:
- Generating random legal board positions
- Generating a position by picking a board state from the previous games, then applying one of the moves that was not executed
- a strategy of your own design Discuss the tradeoffs among these strategies. Which do you feel would work best if the number of training examples was held constant, given the performance measure of winning the most games at the world championship?
(20 points) Implement a game similar to that discussed for the checkers problem, but use the simpler game of tic-tac-toe. Represent the learned function as a linear combination of board features of your choice. To train your program, play it repeatedly against a second copy of the program that uses a fixed evaluation function you create by hand. Plot the percent of games won by your system, versus the number of training games played.
(40 points) Implement a game similar to that discussed for the checkers problem, but use a more complex game such as Othello. Represent the learned function as a linear combination of board features of your choice. You should provide at least two different well-selected sets of board features. Try training each of your programs (one with each feature set) by playing it repeatedly against a second copy of the program that uses a fixed evaluation function you create by hand. Which feature set yielded the better program. Now try training your programs by having them play against each other.
(10 points) Answer both of the following questions.
- Explain why the size of the hypothesis space in the EnjoySport learning task is 973. How would the number of possible hypotheses increase with the addition of the attribute WaterCurrent, which can take on the values Light, Moderate, or Strong? More generally, how does the number of possible instances and hypothesis grow with the addition of the new attribute A that takes on k possible values?
- Give the sequence of S and G boundary sets computed by the CANDIDATE-ELIMINATION algorithm if it is given the sequence of training examples from Table 2.1 in reverse order. Although the final version space will be the same regardless of the sequence of examples (why?), the sets S and G computed at intermediate stages will, of course, depend on the sequence. Can you come up with some ideas for ordering the training examples to minimize the sum of the sizes of these intermediate S and G sets for the H used in the EnjoySports example?
(10 points) Exercise 2.4 of the text which considers applying the version space algorithm for an instance space of integer points in the plane and the set of hypotheses H consists of rectangles.
(10 points) Complete the proof of the version space representation theorem (Theorem 2.1 in the text.)
(10 points) Prove that given an unbiased hypothesis space (the power set of the instances), the learner would find that each unobserved instance would match exactly half the current members of the version space, regardless of which training examples had been observed. That is, prove that for any instance space X, any set of training examples D, and any instance x in X not present in D, that if H is the power set of X, then exactly half the hypotheses in the Version Space defined by H and D will classify x as positive and half will classify it as negative.
(20 points) Consider a learning problem where each instance is described by a conjunction of n boolean attributes a1,a2,...,an. Now consider a hypothesis space H in which each hypothesis is a disjunction of constraints over these attributes. Propose an algorithm that accepts a sequence of training examples and outputs a consistent hypothesis if one exists. Your algorithm should run in time that is polynomial in n and in the number of training examples. Be sure to clearly describe the algorithm, prove it is correct, and analyze its time complexity. (See Exercise 2.9 for a little more detailed statement.)
(40 points) Implement the FIND-S algorithm and verify that it successfully produces the trace in Section 2.4 of the text for the EnjoySport example. Now use this program to study the number of training examples required to exactly learn the target concept following the guidelines given in Exercise 2.10 of the text.
(20 points) Read one of the following papers and write a paper critique.
- Haym Hirsh, Nina Mishra, and Leonard Pitt (1997).
  Version Spaces Without Boundary Sets.
  Proceedings of the 14th National Conference on AI (AAAI97).
- Neil Lesh and Oren Etzioni (1995).
  A Sound and Fast Goal Recognizer.
  Proceedings of the 14th International Joint Conference on AI (IJCAI95).
- Haym Hirsh (1992).
  The Computational Complexity of the Candidate Elimination Algorithm.
  Rutgers University Technical Report ML-TR-36.
- Haym Hirsh (1992).
  Polynomial-Time Learning with Version Spaces.
  Proceedings of the 10th National Conference on AI (AAAI92).
(10 points) Given decision trees to represent the four boolean functions given in Exercise 3.1 of the text.
(20 points) Do Exercise 3.4 of the text which explores the relationship between decision trees and version spaces.
(20 points) Read at least one of the following papers and write a report. See the grading criteria for this type of homework for the expectations.
- Chris Drummond and Robert C. Holte (2000).
  Exploiting the Cost (In)sensitive of Decision Tree Splitting Criteria.
  Proceedings of 17th International Conference on Machine Learning (ICML 2000).
- Ross Quinlan (1996).
  Improved Use of Continuous Attributes in C4.5.
  JAIR vol 4, 77-90, March 1996.
(10, 20, or 40 points) This assignment gives you an opportunity to experiment with a decision tree learning program. You can use the PlayTennis learning data as well as a larger set of data describing the voting records of congressmen/women form the U.S. House of Representatives. The code is all contained in dt-code.zip. Just save it into your account and unzip it (use WinZip on the PCs, unzip in unix, and any standard unzip program on other platforms.) This code is designed for the gcc C++ compiler. On CEC for unix you can access this by using pkgadd sc_5.0 and for nt users, from the start menu go to "Programming, then to Microsoft Visual Studio 6.0, then to Microsoft Visual C++ 6.0.
If you have selected this option you should (even before decision trees are discussed in class) be sure that you can compile and run this program. To compile it use make or gmake. Once you have done this you should find an executable called dt. To run it type
```
dt [-s ]    
```
If you do not provide a random seed then the system clock will be used. Each of train %, prune %, and test % are real numbers between 0 and 1 with the specification that their sum is at most 1.0. The specify, respectively, the fraction of the data that will be used for training, pruning and testing. To be sure it is working correctly, try doing
```
dt 1.0 0.0 0.0 play-tennis.ssv
```
When you do this you should get the decision tree show in Figure 3.1 of the textbook. You can go this far on this HW option before decision trees are discussed in class.
There are three parts to this problem. If you do only the first part it is worth 10 points. If you do the first two parts, then you can submit it as a 20 point homework option. Finally, if you do all three parts then it is a 40 point homework option. If you just do the first part then no cover sheet is needed. If you pick the 20 point or 40 point option then the appropriate cover sheet and write-up is expected.
NOTE: If you are doing the 40 point option, you may submit your final report as late as Monday February 12th without a late penalty prior to then. If you are selecting this option you must submit something on February 7th that shows that you have completed the experiments for Parts I and II. This can be a handwritten rough draft of your thoughts on those that you will be using for your report. This will not be graded but rather used to be sure that you have completed those parts by the deadline.
- PART I.
  Try running the decision tree learner on a randomly chosen subset of half of the examples for training, and using half for testing. What are the training and test accuracies? What conclusion(s) can you draw from this? Determine if each of the following are true or false assuming that the target concept is that described by the decision tree in Figure 3.1, that all examples you add must be consistent with the target concept, and without using any post pruning. Explain how you reached your answers and convince us of them.
  - Is it possible to get ID3 to further elaborate the tree below the right leaf in Figure 3.1 (and make no other changes), by adding a single new (correct) training example?
  - Is it possible to get ID3 to learn an incorrect tree (i.e. a tree that is not equivalent to the target concept of Figure 3.1) by adding a new correct training example?
  - Is it possible to get ID3 to include the attribute Temperature in the learned tree, even though the true target concept is independent of Temperature
- PART II.
  Here you will use the voting data. The first attribute of each example describes the political party of the representative, and the remaining attributes indicate their yes/no/absent vote for each bill considered by congress. You will use this data to learn a decision tree that predicts the political party of the representative based on his/her vote.
  Use the voting data to build a decision tree to predict the political party. Use 25% of the members of congress for training and the rest for testing. (So no pruning.) Try this several times and study the impact of different random splits. Report the sizes and accuracies of these trees over 6 distinct runs.
  Now measure the impact of training set size on the accuracy and size of the learned tree (no pruning, and 30% of the data for testing). Consider training set sizes in the range of 0-40% (include at least the values .02, .1, .2, .3 and .4 for training fractions). Because of the high variance due to random splits you should repeat each experiment with at least 10 different random seeds. Include in your report two plots showing how accuracy varies with training set size, and how the tree size varies with training set size. Also report the maximum, minimum and average accuracies and tree sizes for each training set size.
  Next measure the impact of pruning on accuracy and size of the learned trees, under the same conditions as above. Use the same training set sizes, but this time use 30% of the data for pruning and 30% for testing. What is the impact, if any, of post-pruning the tree?
- Part III.
  The code as provided uses information gain to select attributes while growing the tree. Modify it to instead select attributes at random and study the effect of this change by repeating all of the experiments you have done so far. To make this change, look at the function MaxGainAttribute() in entropy.c. Replace this by a method that randomly selects an attribute. From your experiments what can you conclude about the impact of learning with randomly selected attributes.
  Finally, explore one more non-trivial topic of your own choosing. For example, develop a different method for select attributes that you think will work well and report on it. Or you could implement a rule post-pruning method and add it to the source code and report how its performance compares to that of reduced error pruning?
CHOOSE YOUR OWN ADVENTURE. You can propose any additional homework options (or variations of those given above) to Dr. Goldman. If approved a point value will be given.