Classifying Unseen Images and Detecting Anomalies with Clustering Centroids

Introduction

In the previous post, Auto-Labeling Unlabeled Images with CLIP and K-Means, we constructed a pipeline to encode a pool of images, reduce dimensionality with PCA, cluster them with K-Means, and auto-label whole images in each cluster via manually labeling a few samples.

However, there are two remaining questions:

1. How do we classify new unseen images?

2. Since we already know 10 distinct labels, how does this classification compare to CLIP zero-shot?

In this post, we will build a pipeline to classify unseen images and address the out-of-distribution problem, specifically the question of how to classify an image of a fish in the pool of 10 animals. Then benchmark the accuracy of both approaches using the 500 hold-out images from the previous post.

Dataset

We continue using the Animals-10 dataset from Kaggle, but in this article, we only use 500 holdout images for classification and evaluation.

Each group contains 50 images of 10 animals; we apply the same random seed to restore the holdout images from the previous post.

1
Holdout (500 images)
2
├── 50 × dog
3
├── 50 × cat
4
├── 50 × horse
5
├── 50 × chicken
6
├── 50 × sheep
7
├── 50 × cow
8
├── 50 × spider
9
├── 50 × butterfly
10
├── 50 × elephant
11
└── 50 × squirrel

Nearest Centroid Classification

Reusing Centroids

We reuse the saved centroids found from the clustering algorithm in the previous post to construct a pipeline to classify a new incoming unseen image.

1. Encode the new image using CLIP and L2-normalize → 512D vector
2. PCA reduces dimension using the saved PCA transformer and L2-normalize → 20D vector
3. Calculate the cosine similarity between the projected point and all 10 centroids.
4. Assign the label of the nearest centroid (highest similarity score) to the image.

Classify Holdout Images

In this scenario, we already know the label of these animals, but we fit it into the classification pipeline to evaluate our algorithm.

The process involves scanning 500 images and computing the cosine similarity against the 10 centroids to find the closest one (highest similarity score). Then assign its label to the image.

Handling Out-of-Distribution

Confidence Threshold

The current pipeline has a problem when the input image does not belong to the 10 animals. What if a user uploads a fish image? It will run through 10 centroids to find the nearest one and then assign the label of that centroid to the image.

To resolve this problem, we set a threshold; if the cosine similarity to the nearest centroid is lower than the threshold, we classify the image as unknown. Then the out-of-distribution image will need a human to review.

In our 10-animals example, the dataset of 25,679 images computed a cosine similarity score min-max from 0.244 to 0.975. We set the 10th percentile of score distribution as a threshold, which is 0.623; any input image with a similarity score lower than 0.623 will be flagged as unknown.

As introduced in the previous post, the workload is reduced to 99.8%, but when we apply a threshold, the workload is reduced to 90.0% (needs review: 50 images).

Testing the threshold with a fish image

This is a photo that I took to simulate the user uploading an abnormal image into the classification system.

Surprisingly, it is assigned to the butterfly centroid with a similarity score of 0.488, but with the introduced threshold, it will be marked as unknown, a fraud image.

This finally establishes a complete classification pipeline.

1. Encode the new image using CLIP and L2-normalize → 512D vector
2. PCA reduces dimension using the saved PCA transformer and L2-normalize → 20D vector
3. Calculate the cosine similarity between the projected point and all 10 centroids.
4. Compare the similarity score with the threshold; if it’s lower than the threshold, flag it as unknown.
5. Otherwise, assign the label of the nearest centroid (highest similarity score) to the image.

Images Flagged in the Holdout dataset

Let’s run again the 500 images, but this time with a threshold of 10% to flag for human review.

It is surprising that an image of a tiger in the original dataset was labeled as a cat, and our prediction matched it to the elephant, which is totally wrong.

Now we discovered an anomaly image, a tiger image, in the cat dataset; we will explore how to detect anomaly images within the true dataset in the next post.

CLIP Zero-Shot Classification

We already know the 10 species names, instead of running through our centroids with a threshold, we can directly run it with CLIP zero-shot classification. It is simple; we just define 10 text prompts for our classes.

• “a photo of a dog”
• “a photo of a cat”

We then compare the cosine similarity between the image embedding and the text embeddings of these prompts to assign the label.

With the given 500-image dataset, it gives us 98% accuracy of this example.

But why not use CLIP Zero-Shot in the first place? Because it is a demonstration of labeling via centroids, and also the prompt classification may be fail in complex situations where centroids will perform better. For example:

• “a photo of a short-haired cat”
• “a photo of a long-haired cat”

Evaluation

In this balance across all classes, we use the accuracy metric to evaluate the model.

Total images: 500

Conclusion

This post demonstrates the efficiency of classifying unknown images and detecting anomalies using clustering centroids. This approach can be easily scaled to real-world use cases:

• Content moderation: Detect violating spam or NSFW images.
• Labeling audit: Find mistakes made by human annotators (e.g., the tiger labeled as a cat found in the Animals-10 dataset).
• Fraud detection: Flag uploaded images that do not belong to any known cluster.

CLIP zero-shot classification is powerful, but it relies on a proper prompt to classify the correct image, which is limited in this topic. Having the ability to use custom centroids shows us the control we need for production-grade anomaly detection.

References

• Corrado, A. (2019). Animals-10 (Version 2). Kaggle.