Prev Next

Optimizing Memory Usage

Working with large datasets in Pandas can consume significant memory, leading to performance issues. By optimizing memory usage, you can handle larger datasets efficiently. This tutorial demonstrates techniques such as data type conversion, using categorical data, and loading data in chunks.

Checking Memory Usage

Use the memory_usage() method to inspect the memory usage of a DataFrame. Here’s an example: 

import pandas as pd

# Create a sample DataFrame
data = {
    "ID": [1, 2, 3],
    "Name": ["Karthick", "Durai", "Praveen"],
    "Age": [25, 30, 22]
}

df = pd.DataFrame(data)

# Check memory usage
print(df.memory_usage(deep=True))

Output: Displays memory usage for each column.

Explanation: The memory_usage(deep=True) method provides a detailed breakdown of memory usage for each column, including object columns like strings.

Downcasting Numerical Columns

Converting numerical columns to more memory-efficient types (e.g., from float64 to float32) can significantly reduce memory usage. Here’s an example: 

# Downcast numerical columns
df["Age"] = pd.to_numeric(df["Age"], downcast="integer")
print(df.dtypes)

Output: Shows the updated data types of columns after downcasting.

Explanation: The pd.to_numeric() method converts the Age column to a smaller integer type, reducing memory usage while maintaining accuracy.

Using Categorical Data

For columns with repeated string values, converting them to the category type reduces memory usage significantly. Here’s an example: 

# Convert object columns to category
df["Name"] = df["Name"].astype("category")
print(df.dtypes)
print(df.memory_usage(deep=True))

Output: The column Name is now of type category, reducing memory usage.

Explanation: Converting string columns to the category type optimizes memory usage by storing unique values as categories instead of repeated strings.

Loading Data in Chunks

For large datasets, load data in chunks using the chunksize parameter to avoid memory overload. Here’s an example: 

# Process a large file in chunks
chunk_size = 1000
for chunk in pd.read_csv("large_file.csv", chunksize=chunk_size):
    print(chunk.head())  # Process each chunk

Output: Processes chunks of the file without loading the entire dataset into memory.

Explanation: The chunksize parameter loads a manageable portion of the dataset into memory, enabling efficient processing of large files.

Key Takeaways

  • Inspect Memory: Use memory_usage() to analyze the memory footprint of your DataFrame.
  • Downcasting: Convert numerical columns to smaller types using pd.to_numeric().
  • Categorical Data: Use the category type for columns with repeated string values.
  • Chunk Processing: Load large files in chunks using the chunksize parameter to prevent memory overload.
  • Efficiency: Optimizing memory usage ensures better performance when working with large datasets.
Prev Next