When the confusion matrix reveals the true performance of AI

Vincent NOBLET is a machine vision project manager at Psycle. Late in the morning, he launches the analysis of the batch of images from a production line. Here, the AI must automatically detect any defects that may be present on small chocolate-filled biscuits before packaging. Because we really don't want to risk spoiling children's snack time. So, to find out whether the algorithm lives up to its promises, Vincent opens a table: the famous confusion matrix. This simple table, organized as “predictions vs. reality”, will give its verdict on the reliability of Psycle's calculations and show us the adjustments, if any are needed. In short, it is a concrete way to turn a visual stream into usable data.

What is the confusion matrix for?

The confusion matrix is a fundamental tool for evaluating a supervised artificial intelligence model. It compares the AI's predictions with human annotations. Each image can then be classified as “good” or “defective”. This makes it possible to quantify four cases: true positives, false positives, true negatives and false negatives.

Let's take a concrete quality control project. Here, a matrix may reveal that out of 10,000 parts inspected, 9,991 are correctly identified. The diagnosis is then said to be 99% reliable. It also means that 9 false negatives get through. This is a crucial risk for the manufacturer. But thanks to this diagnosis, Psycle can adjust its algorithms before deployment in order to get closer to zero defects.

Understanding how it works: precision and recall

The matrix is organized as a table with as many rows and columns as there are types of defects.

• True Positive (TP): the AI detects a defect, the operator confirms it.
• False Positive (FP): the AI flags a defect, but the part is good.
• True Negative (TN): the AI does not identify a defect, the part is good.
• False Negative (FN): the AI fails to detect a real defect.

From these values, we derive precision (the proportion of correct alerts) and recall (the ability to catch all defects).

These metrics help judge whether a model is suitable for industrial use. For example, for 100% quality inspection, recall is favored, because it is better to generate a few false alarms than to let a defect through.

Thresholds, confidence and production: when the matrix alone is not enough

The confusion matrix is built with a confidence threshold. Depending on this threshold, the same predictions can be considered positive or rejected. A 90% threshold guarantees few false positives, but may miss defects: false negatives increase. Conversely, a low threshold (e.g. 1%) reduces false negatives but inflates false alarms.

Thus, the matrix may appear pessimistic compared with the actual performance observed in production, or on the contrary too optimistic if the thresholds are poorly chosen. The real test remains actual use, on production lines, with real volumes and conditions.

The confusion matrix is much more than a table: it is an indispensable mirror for evaluating and refining machine vision solutions. But it is not magic. It must be interpreted carefully, because you cannot rely on a single metric, nor forget the impact of the different thresholds. To do this, it is essential to combine precision, recall and field experience, and of course to test the models in real conditions. At Psycle, it is this rigor that turns images into confidence and data into industrial performance.