Exploring Data with the describe() Method Pandas

What You’ll Learn 📚

• Understand what the describe() method does
• Learn how to use it with different types of data
• Explore practical examples and common mistakes
• Get answers to frequently asked questions
• Troubleshoot common issues

Introduction to the describe() Method

The describe() method in Pandas is a powerful tool that provides a quick overview of the statistical properties of your data. It’s like a summary report card for your dataset, giving you insights into the distribution, central tendency, and variability of your data.

Key Terminology

• DataFrame: A two-dimensional, size-mutable, potentially heterogeneous tabular data structure with labeled axes (rows and columns).
• Statistical Properties: Characteristics of data that include measures like mean, median, standard deviation, etc.
• Central Tendency: A measure that represents the center or typical value of a dataset (e.g., mean, median).

Getting Started: The Simplest Example

import pandas as pd

# Create a simple DataFrame
data = {'Age': [23, 45, 12, 35, 37],
        'Height': [165, 180, 150, 175, 170]}
df = pd.DataFrame(data)

# Use the describe() method
summary = df.describe()
print(summary)

       Age      Height
count   5.000000   5.000000
mean   30.400000 168.000000
std    12.701706 11.180340
min    12.000000 150.000000
25%    23.000000 165.000000
50%    35.000000 170.000000
75%    37.000000 175.000000
max    45.000000 180.000000

Progressively Complex Examples

Example 1: Including Non-Numeric Data

import pandas as pd

# Create a DataFrame with mixed data types
data = {'Name': ['Alice', 'Bob', 'Charlie', 'David', 'Eva'],
        'Age': [23, 45, 12, 35, 37],
        'Height': [165, 180, 150, 175, 170]}
df = pd.DataFrame(data)

# Use the describe() method
summary = df.describe(include='all')
print(summary)

        Name        Age      Height
count      5   5.000000   5.000000
unique     5        NaN        NaN
top     Alice        NaN        NaN
freq       1        NaN        NaN
mean      NaN  30.400000 168.000000
std       NaN  12.701706 11.180340
min       NaN  12.000000 150.000000
25%       NaN  23.000000 165.000000
50%       NaN  35.000000 170.000000
75%       NaN  37.000000 175.000000
max       NaN  45.000000 180.000000

Example 2: Describing Specific Columns

import pandas as pd

# Create a DataFrame
data = {'Age': [23, 45, 12, 35, 37],
        'Height': [165, 180, 150, 175, 170],
        'Weight': [55, 85, 40, 70, 65]}
df = pd.DataFrame(data)

# Use the describe() method on specific columns
summary = df[['Age', 'Weight']].describe()
print(summary)

       Age      Weight
count   5.000000   5.000000
mean   30.400000 63.000000
std    12.701706 16.583124
min    12.000000 40.000000
25%    23.000000 55.000000
50%    35.000000 65.000000
75%    37.000000 70.000000
max    45.000000 85.000000

Example 3: Handling Missing Data

import pandas as pd
import numpy as np

# Create a DataFrame with missing values
data = {'Age': [23, np.nan, 12, 35, 37],
        'Height': [165, 180, np.nan, 175, 170]}
df = pd.DataFrame(data)

# Use the describe() method
summary = df.describe()
print(summary)

       Age      Height
count   4.000000   4.000000
mean   26.750000 172.500000
std    11.238199  6.454972
min    12.000000 165.000000
25%    20.250000 168.750000
50%    29.000000 172.500000
75%    35.500000 176.250000
max    37.000000 180.000000

Common Questions and Answers

1. What does the describe() method return?

   The describe() method returns a DataFrame that contains summary statistics of the data, such as count, mean, standard deviation, min, max, and percentiles.

2. Can I use describe() on non-numeric data?

   Yes, by using include='all', you can get a summary of non-numeric data, including the count of unique values and the most frequent value.

3. How do I handle missing data with describe()?

   The describe() method automatically ignores missing values in its calculations, so you don’t need to do anything extra!

4. Why is the describe() method useful?

   It provides a quick and easy way to understand the basic statistical properties of your data, which is essential for data exploration and analysis.

5. Can I customize the percentiles shown by describe()?

   Yes, you can pass a list of percentiles to the describe() method to customize which percentiles are displayed.

Troubleshooting Common Issues

If you encounter errors, make sure your data is in a DataFrame format. The describe() method is designed to work with Pandas DataFrames.

If your DataFrame contains mixed data types and you want to summarize only numeric data, use df.describe() without any parameters.

Practice Exercises

1. Create a DataFrame with at least three columns of numeric data and use the describe() method to summarize it.
2. Experiment with a DataFrame containing both numeric and non-numeric data. Use include='all' to see the full summary.
3. Try creating a DataFrame with missing values and observe how describe() handles them.

For more information, check out the official Pandas documentation.