Python Memory Profiling with tracemalloc

What is tracemalloc?

Tracemalloc is a powerful debugging tool included in Python's standard library that allows developers to trace memory blocks allocated by Python programs. Unlike traditional memory profilers that only show overall memory usage, tracemalloc provides detailed insights into where and how memory is being allocated in your code.

With tracemalloc, you can:

• Track where objects are created in your code
• Analyze memory usage patterns
• Compare memory snapshots to identify leaks
• View statistics on memory blocks by file and line number

Installation & Setup

Since tracemalloc is part of the Python standard library, no additional installation is required. However, to use it effectively, you need to enable it at the start of your program.

import tracemalloc

# Start tracing Python memory allocations
tracemalloc.start()

# Your code here
# ...

# Get the current snapshot
snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics('lineno')

# Print statistics
for stat in top_stats[:10]:
    print(stat)

# Set environment variable before running your script
# Linux/macOS
export PYTHONTRACEMALLOC=1

# Windows
set PYTHONTRACEMALLOC=1

# Then run your Python program
python your_script.py

import tracemalloc

# Start tracemalloc with custom parameters
# The parameter 25 is the maximum number of frames 
# to store in a traceback
tracemalloc.start(25)

# Get current size and peak
current, peak = tracemalloc.get_traced_memory()
print(f"Current memory usage: {current / 1024}KB")
print(f"Peak memory usage: {peak / 1024}KB")

# Reset peak to current
tracemalloc.reset_peak()

# Stop tracking when done
tracemalloc.stop()

Basic Usage

The workflow for using tracemalloc typically follows these steps:

1. 1

   Start Tracing

   Enable the tracemalloc module at the beginning of your program.

2. 2

   Run Your Code

   Execute the code you want to analyze for memory usage.

3. 3

   Take Snapshots

   Capture the memory state at specific points in execution.

4. 4

   Analyze Results

   Review statistics or compare snapshots to identify issues.

Key Features

Working with Snapshots

Snapshots are a core concept in tracemalloc that allow you to capture the memory state at a specific point in time. By comparing snapshots taken at different times, you can identify memory leaks and understand memory usage patterns.

import tracemalloc

# Start tracing memory
tracemalloc.start()

# ... some initial code ...

# Take a first snapshot
snapshot1 = tracemalloc.take_snapshot()

# ... more code that might cause memory changes ...

# Take a second snapshot
snapshot2 = tracemalloc.take_snapshot()

# Compare snapshots
top_stats = snapshot2.compare_to(snapshot1, 'lineno')

print("Memory usage growth:")
for stat in top_stats[:10]:
    print(stat)

Memory Statistics

Tracemalloc provides detailed statistics about memory usage in your application, helping you understand where memory is being allocated and how it's being used.

Code Examples

import tracemalloc
import gc

def detect_leaks():
    tracemalloc.start()
    
    # Run a baseline collection
    gc.collect()
    
    # Record the starting state
    snapshot1 = tracemalloc.take_snapshot()
    
    # Run the code that might leak memory
    potentially_leaky_function()
    
    # Force garbage collection to remove objects
    # that are no longer referenced
    gc.collect()
    
    # Take a second snapshot
    snapshot2 = tracemalloc.take_snapshot()
    
    # Compare snapshots
    top_stats = snapshot2.compare_to(snapshot1, 'lineno')
    
    print("Potential memory leaks:")
    for stat in top_stats[:5]:
        print(stat)

def potentially_leaky_function():
    # This is just a placeholder for your actual code
    large_list = [0] * 1000000  # Allocate a large list
    
    # In a real leak, this object would be
    # accidentally kept alive somewhere

detect_leaks()

import tracemalloc
import time

# Start tracking memory
tracemalloc.start()
snapshots = []

# Function to record a memory snapshot with timestamp
def record_memory():
    snapshot = tracemalloc.take_snapshot()
    current, peak = tracemalloc.get_traced_memory()
    snapshots.append((time.time(), current, snapshot))
    return current

# Record initial state
record_memory()
print("Initial memory:", snapshots[0][1] / 1024, "KB")

# Record several points for analysis
for i in range(3):
    # Simulate work that uses memory
    data = [object() for _ in range(100000)]
    
    # Record the memory at this point
    current = record_memory()
    print(f"Memory after step {i+1}:", current / 1024, "KB")
    
    # For the last iteration, keep the reference
    # to prevent garbage collection
    if i < 2:
        del data  # Remove reference to allow cleanup

# Print memory growth timeline
print("\nMemory Timeline:")
for i, (timestamp, memory, _) in enumerate(snapshots):
    if i > 0:
        prev_time, prev_mem = snapshots[i-1][0], snapshots[i-1][1]
        time_diff = timestamp - prev_time
        mem_diff = memory - prev_mem
        print(f"Step {i}: {memory/1024:.2f} KB " + 
              f"({mem_diff/1024:+.2f} KB in {time_diff:.2f}s)")

Resources

Frequently Asked Questions