Data6 min read

Pandas & Data Analysis

Analyze and transform data with pandas - DataFrames, groupby, merge, time series, and memory-efficient patterns for data science workflows.

PandasPythonData AnalysisDataFrameNumPy

skill: Pandas & Data Analysis category: Data tags: [Pandas, Python, Data Analysis, DataFrame, NumPy] readingTime: 6

Pandas is the spreadsheet of Python - flexible enough for exploration, powerful enough for production data pipelines. Avoid row-wise loops (apply on every row is slow); use vectorized operations. For > 10M rows, consider Polars or DuckDB instead.

Workflow

Load and inspect dataAlways inspect shape, dtypes, and nulls before analysis - assumptions about data types cause silent errors downstream.
Filter, select, and transformBoolean indexing and vectorized operations are orders of magnitude faster than Python loops on DataFrames.
Aggregate with groupbyGroupBy + agg is the pandas equivalent of SQL GROUP BY - handles 90% of analytical questions.
Join and merge datasetsMerge works like SQL JOIN - understanding how='left'/'inner'/'outer' prevents accidental row inflation or data loss.
Handle time series dataPandas has first-class time series support - resampling, rolling windows, and date arithmetic make time-based analysis concise.

Output Format

Pandas pipeline: typed dtypes on load, vectorized transforms, groupby agg, validated merges, and time series resampling.

df = pd.read_csv(f, parse_dates=['date'], dtype={'status': 'category'})
df['col'] = df['col'].transform()           # vectorized
summary = df.groupby('key').agg(...)        # aggregate
result = df.merge(lookup, on='key', how='left') # join
monthly = df.resample('ME').sum()           # time series

Examples

Input

Output

Edge Cases & Gotchas

SettingWithCopyWarning - df[df.x > 0]['y'] = 1 may modify a copy. Use df.loc[df.x > 0, 'y'] = 1 to modify in place.
Memory usage - A 1GB CSV can balloon to 5GB in a DataFrame. Use dtype optimization, chunked reading (chunksize=), or DuckDB for large files.
Timezone-naive vs aware - Mixing tz-naive and tz-aware datetimes raises errors. Normalize all timestamps to UTC at load time.
Duplicate index values - If the index has duplicates, df.loc['key'] returns a DataFrame not a Series. Use df.loc['key'] carefully or reset_index().

When to Break These Rules

Use Polars for DataFrames > 10M rows - it's 5-50x faster than pandas with true multi-core parallelism.
Use DuckDB for SQL-style analytics on Parquet/CSV files without loading into memory - much more memory-efficient.

Reference Files

https://pandas.pydata.org/docs/
https://pola.rs/