1. Introduction
Machine learning is a powerful tool for solving complex problems and making predictions. However, how do you know if your machine learning model is performing well? How do you measure the quality of your model and compare it with other models? How do you identify the strengths and weaknesses of your model and improve it?
One of the most common and useful ways to evaluate machine learning models is to use a confusion matrix. A confusion matrix is a table that summarizes the performance of a classification model on a set of test data for which the true values are known. It shows how many instances of each class were correctly or incorrectly predicted by the model.
A confusion matrix can help you to understand how your model is making predictions, what kind of errors it is making, and how accurate it is. It can also help you to calculate other important metrics, such as precision, recall, and F1-score, that can provide more insight into the quality of your model.
In this blog, you will learn how to use confusion matrix for classification problems. You will learn how to:
- Calculate confusion matrix from the predicted and true values of a classification model
- Interpret confusion matrix and understand its components
- Calculate accuracy, precision, recall, and F1-score from confusion matrix and understand their meaning and interpretation
- Use confusion matrix for model selection and improvement
By the end of this blog, you will have a solid understanding of how to use confusion matrix for evaluating and improving your machine learning models. You will also be able to apply the concepts and techniques you learned to your own classification problems.
Are you ready to master confusion matrix? Let’s get started!
2. What is Confusion Matrix?
A confusion matrix is a table that summarizes the performance of a classification model on a set of test data for which the true values are known. It shows how many instances of each class were correctly or incorrectly predicted by the model.
A confusion matrix is also known as an error matrix or a contingency table. It is one of the most widely used tools for evaluating and improving classification models.
A confusion matrix can have any number of rows and columns, depending on the number of classes in the classification problem. For example, if you have a binary classification problem with two classes, such as spam or not spam, you can use a 2×2 confusion matrix. If you have a multi-class classification problem with more than two classes, such as animal, plant, or mineral, you can use a larger confusion matrix.
The general format of a confusion matrix is shown below:
\begin{array}{|c|c|c|c|}
\hline
& \text{Predicted Class 1} & \text{Predicted Class 2} & \cdots & \text{Predicted Class N} \\
\hline
\text{Actual Class 1} & \text{True Positive (TP)} & \text{False Negative (FN)} & \cdots & \text{False Negative (FN)} \\
\hline
\text{Actual Class 2} & \text{False Positive (FP)} & \text{True Negative (TN)} & \cdots & \text{False Negative (FN)} \\
\hline
\vdots & \vdots & \vdots & \ddots & \vdots \\
\hline
\text{Actual Class N} & \text{False Positive (FP)} & \text{False Negative (FN)} & \cdots & \text{True Negative (TN)} \\
\hline
\end{array}
The rows of the confusion matrix represent the actual classes of the test data, and the columns represent the predicted classes of the model. Each cell of the matrix contains the number of instances that fall into that category. For example, the cell in the first row and the first column shows the number of instances that belong to class 1 and were correctly predicted by the model as class 1. This is called a true positive (TP). Similarly, the cell in the second row and the second column shows the number of instances that belong to class 2 and were correctly predicted by the model as class 2. This is called a true negative (TN).
The other cells of the matrix show the number of instances that were incorrectly predicted by the model. For example, the cell in the first row and the second column shows the number of instances that belong to class 1 but were wrongly predicted by the model as class 2. This is called a false negative (FN). Similarly, the cell in the second row and the first column shows the number of instances that belong to class 2 but were wrongly predicted by the model as class 1. This is called a false positive (FP).
A confusion matrix can help you to visualize and quantify the errors that your model is making. It can also help you to calculate other important metrics, such as accuracy, precision, recall, and F1-score, that can provide more insight into the quality of your model. We will discuss these metrics in the next section.
But first, let’s see how to calculate a confusion matrix from the predicted and true values of a classification model. This is what you will learn in the following section.
3. How to Calculate Confusion Matrix?
To calculate a confusion matrix, you need two sets of values: the predicted values and the true values of your classification model. The predicted values are the outputs of your model on a set of test data, and the true values are the actual labels of the test data.
You can obtain the predicted values by applying your model to the test data and recording the class that the model assigns to each instance. You can obtain the true values by checking the original source of the test data or by using a labeled dataset.
Once you have the predicted and true values, you can compare them and count how many instances fall into each category of the confusion matrix. For example, if you have a binary classification problem with two classes, such as spam or not spam, you can use the following formula to calculate the elements of a 2×2 confusion matrix:
- TP = the number of instances that are spam and predicted as spam
- TN = the number of instances that are not spam and predicted as not spam
- FP = the number of instances that are not spam but predicted as spam
- FN = the number of instances that are spam but predicted as not spam
You can use a similar formula for a multi-class classification problem, but you need to consider each class separately and compare it with the rest of the classes. For example, if you have a three-class classification problem, such as animal, plant, or mineral, you can use the following formula to calculate the elements of a 3×3 confusion matrix:
- TP1 = the number of instances that are animal and predicted as animal
- TN1 = the number of instances that are not animal and predicted as not animal
- FP1 = the number of instances that are not animal but predicted as animal
- FN1 = the number of instances that are animal but predicted as not animal
- TP2 = the number of instances that are plant and predicted as plant
- TN2 = the number of instances that are not plant and predicted as not plant
- FP2 = the number of instances that are not plant but predicted as plant
- FN2 = the number of instances that are plant but predicted as not plant
- TP3 = the number of instances that are mineral and predicted as mineral
- TN3 = the number of instances that are not mineral and predicted as not mineral
- FP3 = the number of instances that are not mineral but predicted as mineral
- FN3 = the number of instances that are mineral but predicted as not mineral
After you calculate the elements of the confusion matrix, you can arrange them in a table format and label the rows and columns accordingly. For example, the confusion matrix for the binary classification problem of spam or not spam might look like this:
\begin{array}{|c|c|c|}
\hline
& \text{Predicted Spam} & \text{Predicted Not Spam} \\
\hline
\text{Actual Spam} & \text{TP} & \text{FN} \\
\hline
\text{Actual Not Spam} & \text{FP} & \text{TN} \\
\hline
\end{array}
And the confusion matrix for the three-class classification problem of animal, plant, or mineral might look like this:
\begin{array}{|c|c|c|c|}
\hline
& \text{Predicted Animal} & \text{Predicted Plant} & \text{Predicted Mineral} \\
\hline
\text{Actual Animal} & \text{TP}_1 & \text{FN}_1 & \text{FN}_1 \\
\hline
\text{Actual Plant} & \text{FP}_2 & \text{TP}_2 & \text{FN}_2 \\
\hline
\text{Actual Mineral} & \text{FP}_3 & \text{FN}_3 & \text{TP}_3 \\
\hline
\end{array}
Calculating a confusion matrix is a simple and effective way to summarize the performance of your classification model. However, a confusion matrix alone does not tell you much about the quality of your model. You need to interpret the confusion matrix and derive some metrics from it that can provide more insight into the accuracy and precision of your model. This is what you will learn in the next section.
4. How to Interpret Confusion Matrix?
A confusion matrix is a useful tool for summarizing the performance of your classification model, but it does not tell you much about the quality of your model by itself. You need to interpret the confusion matrix and derive some metrics from it that can provide more insight into the accuracy and precision of your model.
There are many metrics that can be calculated from a confusion matrix, but in this section, we will focus on four of the most common and important ones: accuracy, precision, recall, and F1-score. These metrics can help you to measure how well your model is predicting the correct class for each instance, how often your model is making false positive or false negative errors, and how balanced your model is between precision and recall.
Let’s see how to calculate and interpret each of these metrics from a confusion matrix.
Accuracy
Accuracy is the simplest and most intuitive metric to measure the performance of your classification model. It is the ratio of the number of correct predictions to the total number of predictions. It tells you how often your model is predicting the correct class for each instance.
To calculate accuracy from a confusion matrix, you need to add up the diagonal elements of the matrix, which are the true positives and true negatives, and divide them by the sum of all the elements of the matrix. The formula for accuracy is:
\text{Accuracy} = \frac{\text{TP} + \text{TN}}{\text{TP} + \text{TN} + \text{FP} + \text{FN}}
Accuracy ranges from 0 to 1, where 0 means that your model is always wrong and 1 means that your model is always right. A higher accuracy indicates a better performance of your model.
For example, if you have a binary classification problem of spam or not spam, and your confusion matrix looks like this:
\begin{array}{|c|c|c|}
\hline
& \text{Predicted Spam} & \text{Predicted Not Spam} \\
\hline
\text{Actual Spam} & 80 & 20 \\
\hline
\text{Actual Not Spam} & 10 & 90 \\
\hline
\end{array}
Then your accuracy is:
\text{Accuracy} = \frac{80 + 90}{80 + 90 + 10 + 20} = \frac{170}{200} = 0.85
This means that your model is correct 85% of the time, which is a good performance.
However, accuracy is not always a reliable metric, especially when you have a skewed or imbalanced dataset, where some classes are more frequent than others. In such cases, accuracy can be misleading, as it does not reflect how well your model is predicting each class. For example, if you have a dataset where 90% of the instances are not spam and 10% are spam, and your model always predicts not spam, then your accuracy will be 90%, which seems high, but your model is actually very poor, as it is not detecting any spam at all. Therefore, you need to use other metrics, such as precision and recall, to evaluate your model more accurately.
5. How to Calculate Accuracy, Precision, Recall, and F1-score from Confusion Matrix?
In the previous section, you learned how to calculate and interpret a confusion matrix for your classification model. In this section, you will learn how to calculate and interpret four important metrics that can be derived from a confusion matrix: accuracy, precision, recall, and F1-score. These metrics can help you to measure how well your model is predicting the correct class for each instance, how often your model is making false positive or false negative errors, and how balanced your model is between precision and recall.
Let’s see how to calculate and interpret each of these metrics from a confusion matrix.
Accuracy
Accuracy is the simplest and most intuitive metric to measure the performance of your classification model. It is the ratio of the number of correct predictions to the total number of predictions. It tells you how often your model is predicting the correct class for each instance.
To calculate accuracy from a confusion matrix, you need to add up the diagonal elements of the matrix, which are the true positives and true negatives, and divide them by the sum of all the elements of the matrix. The formula for accuracy is:
\text{Accuracy} = \frac{\text{TP} + \text{TN}}{\text{TP} + \text{TN} + \text{FP} + \text{FN}}
Accuracy ranges from 0 to 1, where 0 means that your model is always wrong and 1 means that your model is always right. A higher accuracy indicates a better performance of your model.
For example, if you have a binary classification problem of spam or not spam, and your confusion matrix looks like this:
\begin{array}{|c|c|c|}
\hline
& \text{Predicted Spam} & \text{Predicted Not Spam} \\
\hline
\text{Actual Spam} & 80 & 20 \\
\hline
\text{Actual Not Spam} & 10 & 90 \\
\hline
\end{array}
Then your accuracy is:
\text{Accuracy} = \frac{80 + 90}{80 + 90 + 10 + 20} = \frac{170}{200} = 0.85
This means that your model is correct 85% of the time, which is a good performance.
However, accuracy is not always a reliable metric, especially when you have a skewed or imbalanced dataset, where some classes are more frequent than others. In such cases, accuracy can be misleading, as it does not reflect how well your model is predicting each class. For example, if you have a dataset where 90% of the instances are not spam and 10% are spam, and your model always predicts not spam, then your accuracy will be 90%, which seems high, but your model is actually very poor, as it is not detecting any spam at all. Therefore, you need to use other metrics, such as precision and recall, to evaluate your model more accurately.
Precision
Precision is a metric that measures how often your model is correct when it predicts a positive class. It is the ratio of the number of true positives to the number of predicted positives. It tells you how precise your model is when it identifies a positive instance.
To calculate precision from a confusion matrix, you need to divide the true positive element of the matrix by the sum of the true positive and false positive elements of the matrix. The formula for precision is:
\text{Precision} = \frac{\text{TP}}{\text{TP} + \text{FP}}
Precision ranges from 0 to 1, where 0 means that your model is never correct when it predicts a positive class and 1 means that your model is always correct when it predicts a positive class. A higher precision indicates a better performance of your model.
For example, if you have a binary classification problem of spam or not spam, and your confusion matrix looks like this:
\begin{array}{|c|c|c|}
\hline
& \text{Predicted Spam} & \text{Predicted Not Spam} \\
\hline
\text{Actual Spam} & 80 & 20 \\
\hline
\text{Actual Not Spam} & 10 & 90 \\
\hline
\end{array}
Then your precision is:
\text{Precision} = \frac{80}{80 + 10} = \frac{8}{9} \approx 0.89
This means that your model is correct 89% of the time when it predicts spam, which is a good performance.
Precision is a useful metric to evaluate your model when you want to minimize the false positives, or the instances that are wrongly predicted as positive. For example, if you have a model that predicts whether a person has a disease or not, you want to have a high precision, as you don’t want to tell a healthy person that they have a disease.
Recall
Recall is a metric that measures how often your model is correct when the actual class is positive. It is the ratio of the number of true positives to the number of actual positives. It tells you how sensitive your model is when it detects a positive instance.
To calculate recall from a confusion matrix, you need to divide the true positive element of the matrix by the sum of the true positive and false negative elements of the matrix. The formula for recall is:
\text{Recall} = \frac{\text{TP}}{\text{TP} + \text{FN}}
Recall ranges from 0 to 1, where 0 means that your model is never correct when the actual class is positive and 1 means that your model is always correct when the actual class is positive. A higher recall indicates a better performance of your model.
For example, if you have a binary classification problem of spam or not spam, and your confusion matrix looks like this:
\begin{array}{|c|c|c|}
\hline
& \text{Predicted Spam} & \text{Predicted Not Spam} \\
\hline
\text{Actual Spam} & 80 & 20 \\
\hline
\text{Actual Not Spam} & 10 & 90 \\
\hline
\end{array}
Then your recall is:
\text{Recall} = \frac{80}{80 + 20} = \frac{4}{5} = 0.8
This means that your model is correct 80% of the time when the actual class is spam, which is a good performance.
Recall is a useful metric to evaluate your model when you want to minimize the false negatives, or the instances that are wrongly predicted as negative. For example, if you have a model that predicts whether a person has a disease or not, you want to have a high recall, as you don’t want to miss a person who has a disease.
F1-score
F1-score is a metric that combines precision and recall into a single measure. It is the harmonic mean of precision and recall, which gives more weight to low values. It tells you how balanced your model is between precision and recall.
To calculate F1-score from a confusion matrix, you need to multiply precision and recall and divide them by their sum. The formula for F1-score is:
\text{F1-score} = \frac{2 \times \text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}
F1-score ranges from 0 to 1, where 0 means that your model has either zero precision or zero recall and 1 means that your model has perfect precision and recall. A higher F1-score indicates a better performance of your model.
For example, if you have a binary classification problem of spam or not spam, and your confusion matrix looks like this:
\begin{array}{|c|c|c|}
\hline
& \text{Predicted Spam} & \text{Predicted Not Spam} \\
\hline
\text{Actual Spam} & 80 & 20 \\
\hline
\text{Actual Not Spam} & 10 & 90 \\
\hline
\end{array}
Then your F1-score is:
\text{F1-score} = \frac{2 \times 0.89 \times 0.8}{0.89 + 0.8} \approx \frac{1.42}{1.69} \approx 0.84
This means that your model has
6. How to Use Confusion Matrix for Model Selection and Improvement?
Now that you have learned how to calculate and interpret a confusion matrix and its derived metrics, you might be wondering how to use them for model selection and improvement. In this section, you will learn how to compare different models using confusion matrix and its metrics, and how to identify and fix the common problems that your model might have using confusion matrix.
How to Compare Different Models Using Confusion Matrix and Its Metrics?
One of the main applications of confusion matrix and its metrics is to compare different models and select the best one for your classification problem. You can compare different models based on their accuracy, precision, recall, and F1-score, and choose the one that has the highest or most balanced values for these metrics.
However, there is no single best metric that can capture the performance of your model for every problem. Depending on your problem and your goal, you might want to prioritize different metrics over others. For example, if you have a problem where false positives are more costly than false negatives, such as predicting whether a person has a rare disease or not, you might want to choose a model that has a high precision, as it minimizes the false positives. On the other hand, if you have a problem where false negatives are more costly than false positives, such as predicting whether a person is a terrorist or not, you might want to choose a model that has a high recall, as it minimizes the false negatives.
Therefore, you need to consider the trade-off between precision and recall, and choose a model that balances them according to your problem and your goal. A common way to do this is to use the F1-score, which combines precision and recall into a single measure. A high F1-score indicates a model that has both high precision and high recall, and a low F1-score indicates a model that has either low precision or low recall. You can compare different models based on their F1-score and choose the one that has the highest value.
For example, if you have two models for a binary classification problem of spam or not spam, and their confusion matrices look like this:
\text{Model A:}
\begin{array}{|c|c|c|}
\hline
& \text{Predicted Spam} & \text{Predicted Not Spam} \\
\hline
\text{Actual Spam} & 80 & 20 \\
\hline
\text{Actual Not Spam} & 10 & 90 \\
\hline
\end{array}
\text{Model B:}
\begin{array}{|c|c|c|}
\hline
& \text{Predicted Spam} & \text{Predicted Not Spam} \\
\hline
\text{Actual Spam} & 70 & 30 \\
\hline
\text{Actual Not Spam} & 5 & 95 \\
\hline
\end{array}
Then their accuracy, precision, recall, and F1-score are:
\text{Model A:}
\text{Accuracy} = \frac{80 + 90}{80 + 90 + 10 + 20} = \frac{170}{200} = 0.85
\text{Precision} = \frac{80}{80 + 10} = \frac{8}{9} \approx 0.89
\text{Recall} = \frac{80}{80 + 20} = \frac{4}{5} = 0.8
\text{F1-score} = \frac{2 \times 0.89 \times 0.8}{0.89 + 0.8} \approx \frac{1.42}{1.69} \approx 0.84
\text{Model B:}
\text{Accuracy} = \frac{70 + 95}{70 + 95 + 5 + 30} = \frac{165}{200} = 0.825
\text{Precision} = \frac{70}{70 + 5} = \frac{14}{15} \approx 0.93
\text{Recall} = \frac{70}{70 + 30} = \frac{7}{10} = 0.7
\text{F1-score} = \frac{2 \times 0.93 \times 0.7}{0.93 + 0.7} \approx \frac{1.3}{1.63} \approx 0.8
As you can see, model A has a higher accuracy and recall than model B, but model B has a higher precision than model A. If you want to choose a model that balances precision and recall, you can use the F1-score, which shows that model A has a higher F1-score than model B. Therefore, you can choose model A as the better model for your problem.
How to Identify and Fix the Common Problems Using Confusion Matrix?
Another application of confusion matrix and its metrics is to identify and fix the common problems that your model might have. You can use the confusion matrix to analyze the errors that your model is making, and use the metrics to diagnose the type and severity of the problems. You can then use some techniques to improve your model and solve the problems.
There are two common problems that your model might have: underfitting and overfitting. Underfitting is when your model is too simple and cannot learn the complexity of the data. Overfitting is when your model is too complex and learns the noise and outliers of the data. Both of these problems can lead to poor performance and generalization of your model.
You can use the confusion matrix and its metrics to detect and measure the underfitting and overfitting of your model. For example, if your model has a low accuracy, precision, recall, and F1-score, it might be underfitting the data, as it is not able to capture the patterns and relationships of the data. On the other hand, if your model has a high accuracy, precision, recall, and F1-score on the training data, but a low accuracy, precision, recall, and F1-score on the test data, it might be overfitting the data, as it is memorizing the specific features and noise of the training data and not generalizing well to the unseen data.
Once you identify the problem of your model, you can use some techniques to improve your model and solve the problem. For example, if your model is underfitting the data, you can try to increase the complexity of your model, such as adding more layers, neurons, or features, or using a different algorithm. If your model is overfitting the data, you can try to reduce the complexity of your model, such as removing some layers, neurons, or features, or using regularization techniques. You can also try to collect more data, or use data augmentation techniques, to increase the diversity and quality of your data.
By using the confusion matrix and its metrics, you can evaluate and improve your model in a systematic and effective way. You can compare different models and select the best one for your problem, and you can identify and fix the common problems that your model might have.
7. Conclusion
In this blog, you have learned how to use confusion matrix for classification problems. You have learned how to:
- Calculate confusion matrix from the predicted and true values of a classification model
- Interpret confusion matrix and understand its components
- Calculate accuracy, precision, recall, and F1-score from confusion matrix and understand their meaning and interpretation
- Compare different models using confusion matrix and its metrics and select the best one for your problem
- Identify and fix the common problems of underfitting and overfitting using confusion matrix and its metrics
By using the confusion matrix and its metrics, you can evaluate and improve your machine learning models in a systematic and effective way. You can measure how well your model is predicting the correct class for each instance, how often your model is making false positive or false negative errors, and how balanced your model is between precision and recall. You can also use some techniques to increase the complexity or reduce the complexity of your model, or to collect more data or augment your data, to solve the problems of underfitting and overfitting.
Confusion matrix is one of the most widely used tools for evaluating and improving classification models. It can help you to understand how your model is making predictions, what kind of errors it is making, and how accurate it is. It can also help you to choose the best model for your problem and to improve your model performance and generalization.
We hope that this blog has helped you to master confusion matrix and its applications for classification problems. You can apply the concepts and techniques you learned to your own classification problems and see how they can help you to achieve better results. Thank you for reading this blog and happy learning!
