KNN – Complete Process

1. What is KNN?

K-Nearest Neighbours (KNN) is a supervised machine learning algorithm used for classification and regression.

It makes predictions by finding the most similar examples in an existing labelled dataset.

Simple idea: Find the examples most similar to the new example, look at their answers, and use them to make a prediction.

What we will do

Prepare a labelled dataset.
Receive a new image.
Calculate distances.
Find the K nearest neighbours.
Use majority voting to predict its label.

2. The KNN process, step by step

Imagine we have images of cats and dogs and want to classify a new image.

Prepare the dataset

First, we need a dataset containing labelled images. We identify characteristics (features) of each image and represent them as numbers.

This process is called Feature Engineering when we manually define and prepare the features.

Image	Ear shape	Eye size	Label
Image 1	3	4	Cat
Image 2	2	3	Cat
Image 3	5	2	Dog
Image 4	4	1	Dog
Image 5	3	3	Cat

Important terminology:

Features: Numerical characteristics describing the image.
Labels: The correct categories, such as Cat or Dog.
Dataset: The collection of labelled examples.

The developer defines the features and measurement rules. There is no universal standard for manually chosen feature values.

Receive a new image

Now imagine we receive another image that we want to classify.

We apply exactly the same feature extraction process used for the existing dataset.

New image	Ear shape	Eye size	Label
Image 6	3	3.5	Unknown

Notice that we have not assigned a label because we want KNN to predict it.

Calculate Euclidean distance

KNN compares the new image against every record in the existing dataset. It calculates the distance between their numerical features.

For example: New image: (3, 3.5). Existing image: (3, 4).

KNN uses a mathematical formula called Euclidean Distance.

d = √[(x₁ − x₂)² + (y₁ − y₂)²]
Euclidean distance for two features

Applying our values:

d = √[(3 − 3)² + (3.5 − 4)²]
d = √[0 + 0.25]
d = 0.5

The distance between these two records is 0.5.

Repeat the distance calculation

Suppose the existing dataset contains 100 images.

KNN calculates the Euclidean distance between the new image and all 100 existing images.

We now have 100 calculated distances.

Select the K value

K is a hyperparameter that determines how many nearest records we consider.

K = 4

This means we want the four nearest records.

The computer:

Sorts all calculated distances from smallest to largest.
Selects the four smallest distances.
Retrieves the labels of those four records.

Perform majority voting

Suppose the four nearest records are:

Image	Distance	Label
Image 1	0.5	Cat
Image 2	0.7	Cat
Image 3	0.9	Dog
Image 4	1.2	Cat

Count the labels:

Cat = 3
Dog = 1

Final prediction

CAT

The new image is classified as a Cat because three of its four nearest neighbours are labelled Cat.

3. Complete KNN workflow

1. Prepare labelled datasetImages + numerical features + labels

↓

2. Receive new imageExtract the same features

↓

3. Calculate Euclidean distancesCompare against every dataset record

↓

4. Find K nearest neighboursSort distances and select the smallest K

↓

5. Majority votingCount the labels of the nearest neighbours

↓

6. Predict the new image's labelCat or Dog

4. How do we select the best K value?

We don't necessarily know which K value will give the best predictions.

We can test different values:

K = 1
K = 3
K = 5
K = 7
K = 9

We evaluate each value using separate validation data and compare their performance.

This process is called Hyperparameter Tuning.

In practice, Python can automate this process.

5. The five important terms to remember

Feature Engineering

Defining and preparing useful features

Dataset

Collection of labelled examples

Euclidean Distance

Mathematical measurement of distance

Number of nearest neighbours

Majority Voting

Choosing the most common label

This explanation covers KNN classification using the image example we discussed. KNN can also be used for numerical predictions (regression).