Ahmed Dawoud

Ready to elevate your data analysis game? Let’s explore how to apply custom functions to Pandas DataFrames, harness the power of vectorization, and optimize with Numba!

1. Creating a DataFrame:

import pandas as pd

# Construct a DataFrame with columns 'a' and 'b'
df = pd.DataFrame({'a': [10, 20, 30], 'b': [20, 30, 40]})

2. Defining a Custom Function:

def exp(x, e):
  """Raises each element of x to the power of e."""
  return x ** e

3. Applying Functions to DataFrames:

• Applying to a single column:

# Apply the exp function to column 'a' with e=3
df['a'].apply(exp, e=3)

4. Handling Functions with Multiple Arguments:

• Trick: Use partial application:

# Apply a function with multiple arguments
from functools import partial
df['a'].apply(partial(exp, e=2))

5. Vectorizing Functions for Efficiency:

• Problem: Applying element-wise operations directly to DataFrames can be slow.
• Solution: Vectorization with np.vectorize() or Numba:

import numpy as np

def avg_mod(x, y):
  """Calculates the average of x and y, returning NaN if x is 20."""
  if x == 20:
    return np.NaN
  else:
    return (x + y) / 2

# Vectorize with NumPy:
vect_avg_mod = np.vectorize(avg_mod)
vect_avg_mod(df['a'], df['b'])

# Vectorize with Numba for even faster execution:
import numba

@numba.vectorize
def v_avg_mod(x, y):
  """Numba-optimized version of avg_mod."""
  if x == 20:
    return np.NaN
  else:
    return (x + y) / 2

Key takeaways:

• Leverage custom functions to tailor data transformations.
• Understand how functions apply column-wise to DataFrames.
• Employ vectorization for efficient element-wise operations.
• Optimize performance with Numba for computationally intensive tasks.

Master these techniques and transform your data effortlessly!