Sorting Into Groups

Step 1: Choosing a category

In Session 3 the answer was a number. Now the answer is a category: spam or not spam, cat or dog, this flower or that flower. This task is called classification.

A simple and powerful idea is k-nearest neighbors (KNN): to classify something new, find the k training examples most similar to it, and let them vote. If 4 of its 5 closest neighbors are red, the new point is probably red too.

Step 2: The iris dataset

Scientists measured 150 iris flowers of three kinds and recorded four measurements for each. scikit-learn includes this famous dataset.

from sklearn.datasets import load_iris

iris = load_iris()
print(iris.feature_names)
print(iris.target_names)
print(iris.data[:3])
print(len(iris.data))

['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)']
['setosa' 'versicolor' 'virginica']
[[5.1 3.5 1.4 0.2]
 [4.9 3.  1.4 0.2]
 [4.7 3.2 1.3 0.2]]
150

The four measurements are the features. The kind of flower is the label.

Step 3: Train and test

from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import accuracy_score

X_train, X_test, y_train, y_test = train_test_split(
    iris.data, iris.target, test_size=0.3, random_state=7
)

model = KNeighborsClassifier(n_neighbors=5)
model.fit(X_train, y_train)

predictions = model.predict(X_test)
print("accuracy:", round(accuracy_score(y_test, predictions), 3))

accuracy: 0.911

On flowers it had never seen, the model got about 91% right.

Step 4: Where does it go wrong?

A single accuracy number doesn’t tell you which mistakes happen. A confusion matrix does.

from sklearn.metrics import confusion_matrix

print(confusion_matrix(y_test, predictions))

[[12  0  0]
 [ 0 15  1]
 [ 0  3 14]]

Each row is the true kind and each column is what the model predicted, in the order setosa, versicolor, virginica.

• The numbers on the diagonal (12, 15, 14) are correct predictions.
• Off the diagonal are mistakes: 1 versicolor was called virginica, and 3 virginica were called versicolor.

Setosa is perfectly separated. The model mixes up the other two, which look similar. That is useful to know.

Step 5: Choosing k

The number of neighbors, k, is a setting we choose. Try a few:

for k in [1, 3, 5, 15]:
    m = KNeighborsClassifier(n_neighbors=k).fit(X_train, y_train)
    acc = accuracy_score(y_test, m.predict(X_test))
    print(k, round(acc, 3))

1 0.933
3 0.933
5 0.911
15 0.933

• A very small k (like 1) copies one neighbor and can be thrown off by one odd example.
• A very large k averages over so many neighbors that it blurs the groups.

There is no perfect k for every dataset. We try several and keep the one that works best on the testing data.

Predicting for a new flower works just like before:

new_flower = [[5.1, 3.5, 1.4, 0.2]]
print(iris.target_names[model.predict(new_flower)])

['setosa']