Prev Next

Advanced Missing Value Handling

Missing values are a common challenge in data analysis. Pandas provides advanced techniques such as forward filling, backward filling, and interpolation to handle missing data effectively. This tutorial explores these methods in detail.

Key Topics

Forward Fill

Forward filling propagates the last valid value forward to fill missing data. Use the fillna(method="ffill") method for this. Here’s an example: 

import pandas as pd
import numpy as np

# Create a sample DataFrame
data = {
    "Name": ["Karthick", "Durai", "Praveen"],
    "Score": [85, np.nan, 78]
}

df = pd.DataFrame(data)

# Forward fill missing values
df["Score"] = df["Score"].fillna(method="ffill")
print(df)

Output:

Name Score
Karthick 85.0
Durai 85.0
Praveen 78.0

Explanation: The fillna(method="ffill") method fills the missing value in the Score column by propagating the last valid value forward.

Backward Fill

Backward filling propagates the next valid value backward to fill missing data. Use the fillna(method="bfill") method for this. Here’s an example: 

# Backward fill missing values
df["Score"] = pd.Series([85, np.nan, 78])
df["Score"] = df["Score"].fillna(method="bfill")
print(df)

Output:

Name Score
Karthick 85.0
Durai 78.0
Praveen 78.0

Explanation: The fillna(method="bfill") method fills the missing value in the Score column by propagating the next valid value backward.

Interpolation

Interpolation estimates missing values using various mathematical methods. Use interpolate() for this. Here’s an example: 

# Interpolate missing values
df["Score"] = pd.Series([85, np.nan, 78])
df["Score"] = df["Score"].interpolate()
print(df)

Output:

Name Score
Karthick 85.0
Durai 81.5
Praveen 78.0

Explanation: The interpolate() method estimates the missing value in the Score column based on adjacent values, using linear interpolation.

Key Takeaways

  • Forward Fill: Propagates the last valid value forward to fill gaps.
  • Backward Fill: Propagates the next valid value backward to fill gaps.
  • Interpolation: Estimates missing values using mathematical methods like linear interpolation.
  • Flexibility: Advanced missing value handling ensures clean and consistent datasets for analysis.
Prev Next