Python Memory Profiling with tracemalloc
Contents
What is tracemalloc?
Available since: Python 3.4 (PEP 454)
Tracemalloc is a powerful debugging tool included in Python's standard library that allows developers to trace memory blocks allocated by Python programs. Unlike traditional memory profilers that only show overall memory usage, tracemalloc provides detailed insights into where and how memory is being allocated in your code.
With tracemalloc, you can:
- Track where objects are created in your code
- Analyze memory usage patterns
- Compare memory snapshots to identify leaks
- View statistics on memory blocks by file and line number
Installation & Setup
Since tracemalloc is part of the Python standard library, no additional installation is required. However, to use it effectively, you need to enable it at the start of your program.
import tracemalloc
# Start tracing Python memory allocations
tracemalloc.start()
# Your code here
# ...
# Get the current snapshot
snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics('lineno')
# Print statistics
for stat in top_stats[:10]:
print(stat)
# Set environment variable before running your script
# Linux/macOS
export PYTHONTRACEMALLOC=1
# Windows
set PYTHONTRACEMALLOC=1
# Then run your Python program
python your_script.py
import tracemalloc
# Start tracemalloc with custom parameters
# The parameter 25 is the maximum number of frames
# to store in a traceback
tracemalloc.start(25)
# Get current size and peak
current, peak = tracemalloc.get_traced_memory()
print(f"Current memory usage: {current / 1024}KB")
print(f"Peak memory usage: {peak / 1024}KB")
# Reset peak to current
tracemalloc.reset_peak()
# Stop tracking when done
tracemalloc.stop()
Basic Usage
The workflow for using tracemalloc typically follows these steps:
-
1
Start Tracing
Enable the tracemalloc module at the beginning of your program.
-
2
Run Your Code
Execute the code you want to analyze for memory usage.
-
3
Take Snapshots
Capture the memory state at specific points in execution.
-
4
Analyze Results
Review statistics or compare snapshots to identify issues.
Key Features
Memory Allocation Tracking
Trace where objects are allocated with detailed tracebacks
- Pinpoint exact file and line numbers
- Track objects by size and count
- Filter by specific modules or files
Snapshot Capabilities
Capture memory states for analysis and comparison
- Take snapshots at different execution points
- Compare snapshots to detect memory changes
- Filter by specific criteria for focused analysis
Memory Usage Statistics
Get comprehensive memory usage data
- Monitor current and peak memory usage
- View statistics by file, line, or category
- Track memory blocks by size distributions
Debugging Utilities
Tools to help interpret memory data
- Format tracebacks similar to standard Python tracebacks
- Filter noise from system libraries
- Integrate with other debugging tools
Working with Snapshots
Snapshots are a core concept in tracemalloc that allow you to capture the memory state at a specific point in time. By comparing snapshots taken at different times, you can identify memory leaks and understand memory usage patterns.
import tracemalloc
# Start tracing memory
tracemalloc.start()
# ... some initial code ...
# Take a first snapshot
snapshot1 = tracemalloc.take_snapshot()
# ... more code that might cause memory changes ...
# Take a second snapshot
snapshot2 = tracemalloc.take_snapshot()
# Compare snapshots
top_stats = snapshot2.compare_to(snapshot1, 'lineno')
print("Memory usage growth:")
for stat in top_stats[:10]:
print(stat)
Pro Tip
When comparing snapshots, focus on the biggest differences first. These are often the most likely candidates for memory leaks or optimization opportunities.
Memory Statistics
Tracemalloc provides detailed statistics about memory usage in your application, helping you understand where memory is being allocated and how it's being used.
Code Examples
Finding Memory Leaks
import tracemalloc
import gc
def detect_leaks():
tracemalloc.start()
# Run a baseline collection
gc.collect()
# Record the starting state
snapshot1 = tracemalloc.take_snapshot()
# Run the code that might leak memory
potentially_leaky_function()
# Force garbage collection to remove objects
# that are no longer referenced
gc.collect()
# Take a second snapshot
snapshot2 = tracemalloc.take_snapshot()
# Compare snapshots
top_stats = snapshot2.compare_to(snapshot1, 'lineno')
print("Potential memory leaks:")
for stat in top_stats[:5]:
print(stat)
def potentially_leaky_function():
# This is just a placeholder for your actual code
large_list = [0] * 1000000 # Allocate a large list
# In a real leak, this object would be
# accidentally kept alive somewhere
detect_leaks()
Memory Usage Timeline
import tracemalloc
import time
# Start tracking memory
tracemalloc.start()
snapshots = []
# Function to record a memory snapshot with timestamp
def record_memory():
snapshot = tracemalloc.take_snapshot()
current, peak = tracemalloc.get_traced_memory()
snapshots.append((time.time(), current, snapshot))
return current
# Record initial state
record_memory()
print("Initial memory:", snapshots[0][1] / 1024, "KB")
# Record several points for analysis
for i in range(3):
# Simulate work that uses memory
data = [object() for _ in range(100000)]
# Record the memory at this point
current = record_memory()
print(f"Memory after step {i+1}:", current / 1024, "KB")
# For the last iteration, keep the reference
# to prevent garbage collection
if i < 2:
del data # Remove reference to allow cleanup
# Print memory growth timeline
print("\nMemory Timeline:")
for i, (timestamp, memory, _) in enumerate(snapshots):
if i > 0:
prev_time, prev_mem = snapshots[i-1][0], snapshots[i-1][1]
time_diff = timestamp - prev_time
mem_diff = memory - prev_mem
print(f"Step {i}: {memory/1024:.2f} KB " +
f"({mem_diff/1024:+.2f} KB in {time_diff:.2f}s)")
Resources
Python Documentation
The comprehensive official documentation for the tracemalloc module in Python 3.13.3, updated April 2025.
Read MoreMemory Profiling Guide
A practical guide to memory profiling in Python using the standard library's tracemalloc module.
Read MoreFixing Memory Leaks
An in-depth guide on using tracemalloc for memory profiling to diagnose and fix memory leaks in Python applications.
Read MoreMemory Profiling in Python
A comprehensive video tutorial on understanding and fixing memory issues in Python using tracemalloc and other tools.
WatchFrequently Asked Questions
How much overhead does tracemalloc add?
Tracemalloc adds some memory and CPU overhead to your application. The memory overhead depends on the number of frames tracked per allocation and the number of tracked allocations. In general, the overhead ranges from 5% to 20% depending on your application.
Use tracemalloc.get_tracemalloc_memory()
to see how much memory tracemalloc itself is using.
Can tracemalloc track C extension memory usage?
No, tracemalloc only tracks memory blocks allocated by Python itself. Memory allocated by C extensions (like NumPy arrays using their own allocators) is not tracked unless the extension explicitly uses Python's memory allocation APIs.
For tracking memory in C extensions, consider external tools like Valgrind or Memory Profiler.
How can I reduce the amount of traceback data collected?
You can control the amount of traceback data collected in two ways:
- Limit the number of frames when starting tracemalloc:
tracemalloc.start(nframe=1)
- Filter traces when taking a snapshot:
snapshot.filter_traces(filters)
Both approaches can significantly reduce memory overhead for large applications.