A Beginner’s Journey to Logistic Regression with Python: Code and Concepts

Introduction to Logistic Regression#

Logistic regression is a fundamental machine learning algorithm used for binary classification tasks. Unlike linear regression, which predicts continuous values, logistic regression predicts the probability that a given input belongs to a certain class (e.g., yes/no, true/false).

Differences Between Linear and Logistic Regression#

• Linear Regression predicts a continuous output. It fits a line to the data points.
• Logistic Regression predicts a binary outcome. It fits a logistic curve (S-curve) to the data points.

Factor Values and How Logistic Regression Works#

• Factor Values: These are the independent variables (features) used for prediction.
• How It Works: Logistic regression uses the logistic function (sigmoid) to model the probability of a binary outcome. The output is between 0 and 1, representing the probability of belonging to a certain class.

Step-by-Step Guide Using Python#

In this section, we’ll go through the steps of implementing logistic regression using Python. We’ll use the provided Jupyter notebook and CSV file. Here is my dataset and jupyter lab file if you want to look at.

Step 1. Loading the Data#

First, we load the CSV file to understand its structure and contents.

import pandas as pd
 
#Read Data
df = pd.read_csv('single_binary.csv')

2. Data Preprocessing: Handling Missing Values#

Data preprocessing is a crucial step in the data analysis and machine learning pipeline. It involves cleaning and transforming raw data into a format that can be easily understood and used by machine learning algorithms. One of the common preprocessing tasks is handling missing values.

In this section, we’ll discuss how to identify and handle missing values in your dataset using the provided code.

Identifying Missing Values#

The first step in handling missing values is to identify them. We can use the isnull() function combined with sum() to get a count of missing values in each column.

## Check for missing values in the dataset
missing_values = df.isnull().sum()
print(missing_values)

This code will output the number of missing values in each column of the DataFrame df. If there are missing values in the dataset, the next step is to handle them.

Handling Missing Values#

There are several strategies to handle missing values, including:

• Removing rows or columns with missing values.
• Filling missing values with a specific value (e.g., mean, median, mode, or a constant).
• Using algorithms that support missing values inherently.

In this example, we’ll focus on filling missing values in the married column with the median value of that column. The median is a robust measure of central tendency, especially when there are outliers in the data.

## Handle missing values in the 'married' column by filling with the median
median_value = df['married'].median()
df['married'] = df['married'].fillna(median_value)

Explanation of the Code#

1. Identifying Missing Values: This code identifies the missing values in each column of the DataFrame df and prints the count of missing values.
2. Filling Missing Values with Median:

• Calculate the Median: The median value of the married column is calculated using df['married'].median().
• Fill Missing Values: The fillna() function is used to replace the missing values in the married column with the calculated median value.

Step 3: Splitting Data and Selecting Features#

Once the data is preprocessed and cleaned, the next step in building a logistic regression model is to split the data into training and testing sets. This allows us to train the model on one subset of the data and evaluate its performance on another, ensuring that our model generalizes well to new, unseen data.

Selecting Features and Target Variable#

In this example, we will use the age column as the feature (independent variable) and the married column as the target variable (dependent variable). The goal is to predict whether an individual is married based on their age.

We split the data into training and testing sets to evaluate our model’s performance.

from sklearn.model_selection import train_test_split
 
## Select feature and target variable
X = df[['age']]  # Features
y = df['married']  # Target
 
## Split the data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

Explanation of the Code#

1. Selecting Features and Target Variables:

X = df[['age']]  # Features
y = df['married']  # Target

• X: The feature matrix, containing the age column.
• y: The target vector, containing the married column.

2. Splitting the Data:

xTrain, xTest, yTrain, yTest = train_test_split(X, y, test_size=0.3, random_state=42)

• train_test_split: This function from sklearn.model_selection is used to split the data into training and testing sets.
• test_size=0.3: Specifies that 30% of the data should be used for testing, and 70% for training.
• random_state=42: Ensures that the split is reproducible by setting a seed for the random number generator.

Why Split the Data?#

Splitting the data into training and testing sets is crucial for several reasons:

• Model Training: The training set is used to fit the logistic regression model.
• Model Evaluation: The testing set is used to evaluate the model’s performance on unseen data, providing an estimate of how well the model generalizes.

Spet 4: Training and Evaluating#

After preprocessing the data and splitting it into training and testing sets, the next step is to train the logistic regression model using the training data and evaluate its performance on the testing data.

1. Training the Logistic Regression Model#

We start by creating an instance of the LogisticRegression class and fitting it to the training data.

from sklearn.linear_model import LogisticRegression
 
## Create an instance of the LogisticRegression model
model = LogisticRegression()
 
## Fit the model to the training data
model.fit(xTrain, yTrain)

2. Making Predictions#

Once the model is trained, we use it to make predictions on the testing data.

## Make predictions on the testing data
model.predict(xTest)

3. Evaluating the Model#

We evaluate the model’s performance using the testing data. The score method returns the mean accuracy of the model on the given test data and labels.

## Evaluate the model's performance
accuracy = model.score(xTest, yTest)
print(f'Model Accuracy: {accuracy}')

Step 5: Visualizing Model Performance#

Visualizing the performance of a logistic regression model is crucial for understanding how well the model is performing. One common method is to use a confusion matrix, which provides a summary of the prediction results on a classification problem.

Confusion Matrix#

A confusion matrix is a table used to describe the performance of a classification model. It shows the counts of true positive, true negative, false positive, and false negative predictions. Here’s how to generate and plot a confusion matrix using scikit-learn, seaborn, and matplotlib.

from sklearn.metrics import confusion_matrix
import seaborn as sns
import matplotlib.pyplot as plt
 
## Generate the confusion matrix
conf_matrix = confusion_matrix(yTest, yPred)
 
## Plot the confusion matrix
plt.figure(figsize=(8, 6))
sns.heatmap(conf_matrix, annot=True, fmt='d', cmap='Blues', xticklabels=['Not Married', 'Married'], yticklabels=['Not Married', 'Married'])
plt.xlabel('Predicted')
plt.ylabel('Actual')
plt.title('Confusion Matrix')
plt.show()

Generate the Confusion Matrix:

conf_matrix = confusion_matrix(yTest, yPred)

• confusion_matrix(yTest, yPred): Computes the confusion matrix to evaluate the accuracy of the classification. yTest is the true labels, and yPred is the predicted labels by the model.

Plot the Confusion Matrix:

plt.figure(figsize=(8, 6))
sns.heatmap(conf_matrix, annot=True, fmt='d', cmap='Blues', xticklabels=['Not Married', 'Married'], yticklabels=['Not Married', 'Married'])
plt.xlabel('Predicted')
plt.ylabel('Actual')
plt.title('Confusion Matrix')
plt.show()

• plt.figure(figsize=(8, 6)): Sets the figure size for the plot.
• sns.heatmap(conf_matrix, annot=True, fmt=’d’, cmap=’Blues’): Creates a heatmap to visualize the confusion matrix. annot=True adds the count annotation to each cell, fmt='d' ensures the annotation is shown as integers, and cmap='Blues' sets the color map.
• xticklabels and yticklabels: Sets the labels for the x-axis and y-axis to ‘Not Married’ and ‘Married’.
• plt.xlabel(‘Predicted’): Labels the x-axis as ‘Predicted’.
• plt.ylabel(‘Actual’): Labels the y-axis as ‘Actual’.
• plt.title(‘Confusion Matrix’): Sets the title of the plot.
• plt.show(): Displays the plot.

Understanding the Confusion Matrix#

• True Positives (TP): The model correctly predicted the positive class.
• True Negatives (TN): The model correctly predicted the negative class.
• False Positives (FP): The model incorrectly predicted the positive class.
• False Negatives (FN): The model incorrectly predicted the negative class.

The confusion matrix helps in understanding the performance of the classification model by providing insights into the types of errors the model is making.

Conclusion#

Training and evaluating a logistic regression model involves several key steps: selecting features and target variables, splitting the data, training the model, making predictions, and evaluating the model’s performance. By following these steps and visualizing the results, we can gain a deeper understanding of how well the model performs and where it may need improvement.