Have You Tried Using a 'Nearest Neighbor Search'?

Roughly a year and a half ago, I had the privelage of taking a graduate "Introduction to Machine Learning" course under the tutelage of the fantastic Professor Leslie Kaelbling. While I learned a great deal over the course of the semester, there was one minor point that she made to the class which stuck with me more than I expected it to at the time: before using a really fancy or sophisticated or "in-vogue" machine learning algorithm to solve your problem, try a simple Nearest Neighbor Search first. Let's say I gave you a bunch of data points, each with a location in space and a value, and then asked you to predict the value of a new point in space. Perhaps the values of you data are binary (just s and -s) and you've heard of Support Vector Machines. Should you give that a shot?