Module 8: Evaluation Metrics

Lesson - 9: Evaluation Metrics

Evaluating the performance of machine learning models is paramount to ensure their effectiveness and reliability in real-world applications. In this lesson, we embark on a journey to explore a plethora of evaluation metrics designed to assess the performance of both classification and regression models. From classic metrics like accuracy, precision, and recall to more advanced measures such as ROC curve and AUC, we'll delve into the nuances of each metric, equipping you with the knowledge and tools to evaluate your models effectively. Additionally, we'll demonstrate how to implement these evaluation metrics using Python, empowering you to gauge the performance of your machine learning models with confidence.

Understanding Evaluation Metrics

Evaluation metrics serve as yardsticks to measure the performance of machine learning models across different tasks, including classification and regression. Let's delve into the key evaluation metrics for each type of task:

Classification Evaluation Metrics:

- Accuracy: The proportion of correctly classified instances out of the total instances.

- Precision: The ratio of true positive predictions to the total positive predictions, measuring the model's ability to avoid false positives.

- Recall (Sensitivity): The ratio of true positive predictions to the total actual positives, indicating the model's ability to capture positive instances.

- F1-score: The harmonic mean of precision and recall, providing a balanced measure of a model's performance.

Regression Evaluation Metrics:

- Mean Squared Error (MSE): The average of the squared differences between predicted and actual values, measuring the model's accuracy.

- R-squared (Coefficient of Determination): The proportion of the variance in the dependent variable that is predictable from the independent variables, indicating the goodness of fit of the model.

Binary Classification Evaluation Metrics:

- ROC Curve (Receiver Operating Characteristic Curve): A graphical plot that illustrates the performance of a binary classification model across different threshold values.

- AUC (Area Under the ROC Curve): The area under the ROC curve, providing a single scalar value representing the model's performance.

Implementing Evaluation Metrics using Python

Let's dive into practical examples to implement these evaluation metrics using Python and popular libraries such as scikit-learn:

1. Classification Metrics:

```python

from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score

from sklearn.model_selection import train_test_split

from sklearn.linear_model import LogisticRegression

X, y = iris.data, iris.target

# Split dataset into train and test sets

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train Logistic Regression classifier

clf = LogisticRegression()

clf.fit(X_train, y_train)

# Predictions

y_pred = clf.predict(X_test)

# Calculate evaluation metrics

accuracy = accuracy_score(y_test, y_pred)

precision = precision_score(y_test, y_pred, average='weighted')

recall = recall_score(y_test, y_pred, average='weighted')

f1 = f1_score(y_test, y_pred, average='weighted')

print("Accuracy:", accuracy)

print("Precision:", precision)

print("Recall:", recall)

print("F1-score:", f1)

```

1. Regression Metrics:

```python

from sklearn.metrics import mean_squared_error, r2_score

from sklearn.model_selection import train_test_split

from sklearn.linear_model import LinearRegression

X, y = boston.data, boston.target

# Split dataset into train and test sets

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train Linear Regression model

reg = LinearRegression()

reg.fit(X_train, y_train)

# Predictions

y_pred = reg.predict(X_test)

# Calculate evaluation metrics

mse = mean_squared_error(y_test, y_pred)

r2 = r2_score(y_test, y_pred)

print("Mean Squared Error (MSE):", mse)

print("R-squared (R2):", r2)

```

1. Binary Classification Metrics:

```python

from sklearn.metrics import roc_curve, auc

import matplotlib.pyplot as plt

from sklearn.datasets import make_classification

from sklearn.model_selection import train_test_split

from sklearn.linear_model import LogisticRegression

# Generate synthetic dataset

X, y = make_classification(n_samples=1000, n_features=20, n_classes=2, random_state=42)

#Split dataset into train and test sets

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

#Train Logistic Regression classifier

clf = LogisticRegression()

clf.fit(X_train, y_train)

#Predict probability scores

y_scores = clf.predict_proba(X_test)[:, 1]

#Compute ROC curve and AUC

fpr, tpr, thresholds = roc_curve(y_test, y_scores)

roc_auc = auc(fpr, tpr)

# Plot ROC curve

plt.figure(figsize=(8, 6))

plt.plot(fpr, tpr, color='blue', lw=2, label='ROC curve (AUC = %0.2f)' % roc_auc)

plt.plot([0, 1], [0, 1], color='red', linestyle='--')

plt.xlim([0.0, 1.0])

plt.ylim([0.0, 1.05])

plt.xlabel('False Positive Rate')

plt.ylabel('True Positive Rate')