Python Pickle Module

In Python development, we often need to save runtime objects or restore previous states after a program restarts, for example:

• Caching calculation results to disk to avoid recomputation
• Saving user configurations and intermediate program states
• Passing complex objects between different Python processes

The pickle module is precisely the official built-in solution Python provides for these problems.

Python’s pickle module is a standard library module for serializing and deserializing Python objects.

Before using Pickle, you need to understand two core concepts:

• Serialization (Pickling): Converting a Python object into a byte sequence
• Deserialization (Unpickling): Converting a byte sequence back into a Python object

The pickle module can save almost any Python object (such as lists, dictionaries, class instances, etc.) to a file, or transmit it over a network, and then reload it when needed.

Why Use the Pickle Module?

1. Data Persistence: Save Python objects to files so that the data remains accessible after the program closes.
2. Data Transfer: Transmit Python objects over a network, such as passing data in distributed systems.
3. Fast Storage and Loading: The pickle module can efficiently handle complex data structures, making it suitable for scenarios requiring fast storage and loading.

Typical Use Cases for Pickle

pickle is well-suited for the following scenarios:

1. Local data persistence
2. Saving and restoring program runtime state
3. Caching intermediate computation results
4. Python inter-process communication (IPC)
5. Saving machine learning models and feature data

Scenarios where it is not suitable:

• Cross-language data exchange
• Frontend-backend API data transmission
• Deserialization from untrusted data sources

What Objects Does Pickle Support?

1. Supported Object Types

Type	Supported
int / float / bool / str	Yes
list / tuple / dict / set	Yes
None	Yes
Custom class instances	Yes
Nested structures	Yes

Unsupported or Not Recommended Objects

• Open file objects
• Sockets, database connections
• OS resources
• Objects dependent on runtime environment state

Importing the Module

Using the Pickle module is very simple, just import it:

import pickle

---

Basic Usage of the Pickle Module

1. Serializing Objects

Use the pickle.dump() method to serialize a Python object and save it to a file.

Example

import pickle

# Create a Python object
data = {
    'name': 'Alice',
    'age': 25,
    'hobbies': ['reading', 'traveling']
}

# Serialize the object and save it to a file
with open('data.pkl', 'wb') as file:
    pickle.dump(data, file)

• 'wb' means opening the file in binary write mode.
• pickle.dump() serializes the data object and writes it to the file.

2. Deserializing Objects

Use the pickle.load() method to load and deserialize a Python object from a file.

Example

import pickle

# Load and deserialize the object from the file
with open('data.pkl', 'rb') as file:
    loaded_data = pickle.load(file)

print(loaded_data)

• 'rb' means opening the file in binary read mode.
• pickle.load() reads the byte stream from the file and deserializes it into a Python object.

3. Serializing to a Byte String

If you don’t want to save to a file, you can use pickle.dumps() to serialize an object into a byte string:

Example

import pickle

data = [1, 2, 3, 4, 5]

# Serialize to a byte string
byte_data = pickle.dumps(data)
print(byte_data)
# Output similar to: b'\x80\x04\x95\x0f\x00\x00\x00...'

4. Deserializing from a Byte String

Use pickle.loads() to deserialize an object from a byte string:

Example

import pickle

byte_data = b'\x80\x04\x95\x0f\x00\x00\x00\x00\x00\x00\x00]\x94(K\x01K\x02K\x03K\x04K\x05e.'

# Deserialize from byte string
original_data = pickle.loads(byte_data)
print(original_data)
# Output: [1, 2, 3, 4, 5]

5. Serializable Object Types

Pickle can serialize most Python objects, including:

• Basic data types: integers, floats, strings, booleans, None
• Collection types: lists, tuples, dictionaries, sets
• Custom class instances
• Functions and classes (with some limitations)

Example

import pickle

# Serialize different types of data
numbers = [1, 2, 3]
text = "Hello, Pickle"
dictionary = {'key': 'value'}
tuple_data = (1, 2, 3)
set_data = {1, 2, 3}

# Put all data into a list
all_data = [numbers, text, dictionary, tuple_data, set_data]

# Serialize
with open('mixed_data.pkl', 'wb') as f:
    pickle.dump(all_data, f)

# Deserialize
with open('mixed_data.pkl', 'rb') as f:
    loaded_data = pickle.load(f)
    print(loaded_data)

6. Serializing Custom Classes

Pickle handles custom class instances very well:

Example

import pickle

class Student:
    def __init__(self, name, age, grade):
        self.name = name
        self.age = age
        self.grade = grade
    
    def __repr__(self):
        return f"Student(name={self.name}, age={self.age}, grade={self.grade})"

# Create an instance
student = Student("Li Si", 20, "Junior")

# Serialize
with open('student.pkl', 'wb') as f:
    pickle.dump(student, f)

# Deserialize
with open('student.pkl', 'rb') as f:
    loaded_student = pickle.load(f)
    print(loaded_student)
    # Output: Student(name=Li Si, age=20, grade=Junior)

7. Serializing Multiple Objects

You can call pickle.dump() multiple times to save multiple objects:

Example

import pickle

data1 = {'item': 'apple', 'count': 5}
data2 = ['banana', 'orange', 'grape']
data3 = 42

# Save multiple objects
with open('multiple.pkl', 'wb') as f:
    pickle.dump(data1, f)
    pickle.dump(data2, f)
    pickle.dump(data3, f)

# Read multiple objects (order must match)
with open('multiple.pkl', 'rb') as f:
    loaded_data1 = pickle.load(f)
    loaded_data2 = pickle.load(f)
    loaded_data3 = pickle.load(f)
    
print(loaded_data1)
print(loaded_data2)
print(loaded_data3)

8. Practical Application Examples

Example 1: Saving and loading machine learning model configuration

Example

import pickle

# Simulate a machine learning model configuration
model_config = {
    'model_type': 'RandomForest',
    'n_estimators': 100,
    'max_depth': 10,
    'trained_date': '2024-01-13',
    'accuracy': 0.95
}

# Save configuration
with open('model_config.pkl', 'wb') as f:
    pickle.dump(model_config, f)

# Load configuration
with open('model_config.pkl', 'rb') as f:
    config = pickle.load(f)
    print(f"Model type: {config['model_type']}")
    print(f"Accuracy: {config['accuracy']}")

Example 2: Caching computation results

Example

import pickle
import os

def expensive_computation(n):
    """Simulate a time-consuming computation"""
    result = sum(i ** 2 for i in range(n))
    return result

def compute_with_cache(n, cache_file='cache.pkl'):
    # Check if cache exists
    if os.path.exists(cache_file):
        with open(cache_file, 'rb') as f:
            cache = pickle.load(f)
            if n in cache:
                print("Reading result from cache")
                return cache[n]
    else:
        cache = {}
    
    # Perform computation
    print("Performing computation...")
    result = expensive_computation(n)
    
    # Save to cache
    cache[n] = result
    with open(cache_file, 'wb') as f:
        pickle.dump(cache, f)
    
    return result

# Usage example
print(compute_with_cache(1000000))  # First call computes
print(compute_with_cache(1000000))  # Second call reads from cache

---

Important Notes on the Pickle Module

1. Security: The pickle module can execute arbitrary code during deserialization. Therefore, do not load pickle data from untrusted sources to avoid malicious attacks.
2. Compatibility: The byte stream generated by pickle is Python-specific, and there may be compatibility issues between different versions of Python.
3. Performance: For large datasets, pickle serialization and deserialization can be relatively slow. Consider using more efficient serialization tools such as json or msgpack.

---

Advanced Usage: Custom Object Serialization

The pickle module supports serialization of custom classes. By default, pickle saves the object’s attributes and class name. If you need more complex serialization logic, you can implement the __getstate__() and __setstate__() methods in your class.

Example

import pickle

class Person:
    def __init__(self, name, age):
        self.name = name
        self.age = age

    def __getstate__(self):
        # Custom serialization logic
        return {'name': self.name, 'age': self.age}

    def __setstate__(self, state):
        # Custom deserialization logic
        self.name = state['name']
        self.age = state['age']

# Create object and serialize
person = Person('Bob', 30)
with open('person.pkl', 'wb') as file:
    pickle.dump(person, file)

# Deserialize object
with open('person.pkl', 'rb') as file:
    loaded_person = pickle.load(file)

print(loaded_person.name, loaded_person.age)

---

Common Methods of the pickle Module

Method	Description	Example
pickle.dump(obj, file)	Serialize object and write to file	pickle.dump(data, open('data.pkl', 'wb'))
pickle.load(file)	Read and deserialize object from file	data = pickle.load(open('data.pkl', 'rb'))
pickle.dumps(obj)	Serialize object to byte string	bytes_data = pickle.dumps([1, 2, 3])
pickle.loads(bytes)	Deserialize object from byte string	lst = pickle.loads(bytes_data)
pickle.HIGHEST_PROTOCOL	Highest available protocol version (attribute)	pickle.dump(..., protocol=pickle.HIGHEST_PROTOCOL)
pickle.DEFAULT_PROTOCOL	Default protocol version (attribute, usually 4)	pickle.dumps(obj, protocol=pickle.DEFAULT_PROTOCOL)

1. Serialize object to file

Example

import pickle
data = {'name': 'Alice', 'age': 25}
with open('data.pkl', 'wb') as f:
    pickle.dump(data, f, protocol=pickle.HIGHEST_PROTOCOL)

2. Deserialize from file

Example

with open('data.pkl', 'rb') as f:
    loaded_data = pickle.load(f)
print(loaded_data)  # Output: {'name': 'Alice', 'age': 25}

3. Serialize to byte string (network transfer/caching)

Example

bytes_data = pickle.dumps([1, 2, 3], protocol=4)
restored_list = pickle.loads(bytes_data)

---

pickle Module Protocol Versions

Protocol Version	Description
0	Human-readable ASCII format (compatible with older versions)
1	Binary format (compatible with older versions)
2	Python 2.3+ optimized support for class objects
3	Python 3.0+ default protocol (does not support Python 2)
4	Python 3.4+ supports larger objects and more data types
5	Python 3.8+ supports memory optimization and data sharing