Ahmed Dawoud

---

Ever wondered why your Python code seems sluggish when working with data? While Python is known for its readability and ease of use, certain operations can be surprisingly slow, impacting your data analysis workflow. This blog post delves into ten key lessons learned from “Python for Data Analysis” (Chapters 1-4), providing insights and code examples to help you optimize your code and streamline your data analysis tasks.

By understanding the performance implications of different operations, you can choose the most efficient techniques for your specific needs, making your Python code faster, cleaner, and more enjoyable to use. So, buckle up and get ready to unlock the full potential of Python for efficient data analysis!

1. Insertion vs. Appending in Lists:

Inserting elements at any position in a list is generally more expensive than appending to the end due to the need to shift existing elements. Here’s an example:

import time

# Create a large list
lst = [i for i in range(100000)]

# Measure insertion time
start_insert = time.time()
for i in range(1000):
    lst.insert(50000, i)  # Inserting in the middle
end_insert = time.time()

# Measure append time
start_append = time.time()
for i in range(1000):
    lst.append(i)
end_append = time.time()

print("Insertion time:", end_insert - start_insert)
print("Append time:", end_append - start_append)

2. List Concatenation with + vs. extend:

Concatenating lists using + creates a new list, copying the elements from the original lists. Using extend is more efficient as it modifies the original list in place, avoiding unnecessary copying.

list1 = [1, 2, 3]
list2 = [4, 5, 6]

# Concatenation with + (creates a new list)
new_list = list1 + list2
print(new_list)  # Output: [1, 2, 3, 4, 5, 6]

# Concatenation with extend (modifies the original list)
list1.extend(list2)
print(list1)  # Output: [1, 2, 3, 4, 5, 6]

3. Membership Checking (List vs. Tuple):

Checking if a number is present in a list is slower than checking in a tuple because lists are mutable and their elements may change dynamically, requiring more extensive search algorithms. Tuples, being immutable, have optimized lookup mechanisms.

list_num = [1, 2, 3, 4, 5]
tuple_num = (1, 2, 3, 4, 5)

# Membership check in list
start_list = time.time()
is_in_list = 3 in list_num
end_list = time.time()

# Membership check in tuple
start_tuple = time.time()
is_in_tuple = 3 in tuple_num
end_tuple = time.time()

print("List check time:", end_list - start_list)
print("Tuple check time:", end_tuple - start_tuple)

4. bisect Module and Unsorted Sequences:

Indeed, the bisect module does not check if the sequence is sorted. Using it with an unsorted sequence will not raise an error, but it might lead to incorrect results. Always ensure the sequence is sorted before using bisect.insort.

5. Dictionary Keys and Hashing:

Dictionary keys must be immutable (e.g., int, float, str, tuple) because they are used for efficient lookup using hashing. Hashing converts a key into an integer, allowing for fast retrieval of its corresponding value. You can check an object’s hashability using hash().

# Example of hashable and non-hashable objects
hashable = "key"
non_hashable = list([1, 2, 3])  # Lists are mutable and not hashable

try:
    hash(non_hashable)
except TypeError:
    print("Lists are not hashable")

6. Set Elements and Immutability:

Set elements must also be immutable to support efficient membership checks and operations. They use hashing similar to dictionaries.

7. File Handling with with open:

Proper resource management is crucial when working with files. Using with open guarantees the file gets closed automatically, even if an exception occurs, avoiding potential resource leaks.

# Using with open
with open("data.txt", "r") as f:
    content = f.read()
    print(content)

# Without with open (risky, might not close the file properly)
f = open("data.txt", "r")
content = f.read()
print(content)
f.close()  # Manually closing the file

8. Array Slices as Views: Modifying Them Modifies the Original Array

In Python, when you create a slice of an array, it doesn’t actually create a new copy of the data. Instead, it creates a view of the original array. This means that any changes you make to the elements within the slice will also be reflected in the original array.

import numpy as np

# Create a NumPy array
arr = np.array([1, 2, 3, 4, 5])

# Create a slice
slice_arr = arr[1:4]  # This creates a view of elements at indices 1, 2, and 3

# Modify the slice
slice_arr[:] = 10

# Print the original array
print(arr)  # Output: [1, 10, 10, 10, 5]

# As you can see, modifying the slice also modified the original array.

While modifying a slice modifies the original array, remember that the slice itself still refers to the same portion of the original array. Its size and starting position remain unchanged.

9. Boolean and Fancy Indexing: Always Creating Copies

Unlike array slicing, which creates a view, using boolean and fancy indexing in NumPy always results in creating a copy of the data. This means any modifications you make to the resulting array will not affect the original array.

• Boolean indexing: Selects elements based on a condition.
• Fancy indexing: Uses an array-like object to select desired elements.

import numpy as np

# Create a NumPy array
arr = np.array([1, 2, 3, 4, 5])

# Boolean indexing (creates a copy)
filtered_arr = arr[arr > 2]  # This will create a new array with values greater than 2

# Fancy indexing (creates a copy)
indices = [0, 2, 4]
fancy_arr = arr[indices]  # This will create a new array with elements at indices 0, 2, and 4

# Modify the new arrays
filtered_arr[:] = 0
fancy_arr[:] = 100

# Print the original array
print(arr)  # Output: [1, 2, 3, 4, 5] (original array remains unchanged)

Use slicing when you want to modify the original array and avoid unnecessary copying. Use boolean and fancy indexing when you need to create a new array based on specific selection criteria.

10. Transposing and Swapping Axes in NumPy Arrays: arr.transpose((1, 0, 2)) vs. arr.swapaxes(1, 2)

Both arr.transpose((1, 0, 2)) and arr.swapaxes(1, 2) are used to rearrange the dimensions of a NumPy array, but they have slightly different functionalities:

• arr.transpose((1, 0, 2)): This method takes a tuple specifying the new order of axes. In this example, it swaps the first and second dimensions, resulting in:

import numpy as np

arr = np.array([[1, 2, 3], [4, 5, 6]])

transposed_arr = arr.transpose((1, 0, 2))

print(transposed_arr)  # Output: [[1, 4], [2, 5], [3, 6]]

• arr.swapaxes(1, 2): This method takes two integers representing the axes to be swapped. In this example, it swaps the second and third dimensions:

arr = np.array([[1, 2, 3], [4, 5, 6]])

swapped_arr = arr.swapaxes(1, 2)

print(swapped_arr)  # Output: [[1, 3, 2], [4, 6, 5]]

Choosing the Right Method:

• Use arr.transpose((1, 0, 2)) when you need to specify a specific order for all dimensions.
• Use arr.swapaxes(1, 2) when you only need to swap two specific axes.