Ahmed Dawoud

---

Pandas, the popular Python library for data manipulation, offers a powerful tool for data analysis: the groupby function. This function allows you to group data based on specific columns and perform various operations on each group. Let’s explore different ways to leverage groupby for effective data analysis.

1. Aggregating by a Custom Function:

Imagine you want to calculate the sum of squares for the age column within each name group. The standard aggregation methods might not suffice. Here’s where a custom function comes in:

def sqr_sum(x):
  return np.sum(x ** 2)

df.groupby('name')['age'].agg(sqr_sum)

This code defines a function sqr_sum that calculates the sum of squares and applies it to the age column grouped by name.

2. Multiple Custom Aggregations:

Suppose you want to find the sum, count, mean, and standard deviation of age and the mean of grade for each name group:

df.groupby('name').agg({'age': ['sum', 'count', 'mean', 'std'], 'grade': 'mean'})

Here, we pass a dictionary to the agg function. Each key represents a column, and the value is a list of aggregation methods to apply.

3. Preserving Group Labels as Columns:

By default, groupby creates a new DataFrame with group labels as the index. To keep them as columns, use as_index=False:

df.groupby(['name', 'gov']).agg({'age': ['sum', 'count', 'mean', 'std'], 'grade': 'mean'}, as_index=False)

This ensures the group labels (name and gov) become regular columns in the resulting DataFrame.

4. Identifying Top Performers:

Let’s say you want to find the top two students (highest grades) for each name group:

def top(df, n=2, column='grade'):
  return df.nlargest(n, column)

df.groupby('name').apply(top)

We define a function top that takes a DataFrame and returns the rows with the n highest values in the specified column. groupby then applies this function to each name group.

5. Descriptive Statistics by Group:

The describe method provides summary statistics for each group:

df.groupby('name')['age'].describe()

This code calculates statistics like mean, standard deviation, quartiles, etc., for the age column within each name group.

6. Filling Missing Values by Group:

To fill missing values (NaN) in the age column with the group’s mean age:

def fill(var):
  return var.fillna(var.mean())

df.groupby('name')['age'].apply(fill)

We define a function fill that replaces missing values with the mean of the column within each group, effectively filling missing values based on category.

7. Weighted Average based on another Variable:

Imagine calculating the weighted average of age where the weights are from the weight column:

def get_weighted_avg(df):
  return np.average(df['age'], weights=df['weight'])

df.groupby('name').apply(get_weighted_avg)

We create a function get_weighted_avg that calculates the weighted average using the weight column as weights. This function is then applied to each group using groupby.

By mastering these groupby techniques, you can unlock powerful data analysis capabilities within Pandas!