Pandas

In one line: labelled, heterogeneously-typed tables with an index, built for the messy real-world data that NumPy refuses to hold.

Version at time of writing: 3.0.x (released 21 Jan 2026) · import pandas as pd

The mental model

A DataFrame is a dict of columns, where each column is a Series — a 1-D array plus a shared Index. Two things follow, and they explain most pandas behaviour:

1. Alignment. Almost every binary operation aligns on the index first. s1 + s2 matches labels, not positions, and produces NaN where labels are missing. This is the single biggest difference from NumPy and the source of most "why is my result all NaN" questions.
2. Column-oriented storage. Operations along a column are fast; operations along a row are not. Adding a row is expensive; adding a column is cheap.

What changed in pandas 3.0 — read this first

Three breaking changes make pre-2026 tutorials wrong:

• Copy-on-Write is mandatory. Chained assignment (df["a"][mask] = x) silently does nothing now. Use .loc. SettingWithCopyWarning no longer exists because the ambiguity is gone.
• Strings default to str dtype, not object, backed by PyArrow when installed. Any dtypes == object check for text columns is now wrong.
• Datetimes default to microsecond resolution, not nanosecond — which incidentally removes the old 1678–2262 range limit.

Details in Versions and Compatibility.

Why bioinformatics cares

Every annotation table, sample sheet, variant call set, differential-expression result and QC report is a heterogeneous table with labels. Genomics file formats are also relentlessly untidy — VCF's INFO field and GFF's attribute column both pack many variables into one string — and pandas' String Accessor plus explode is how you untangle them.

The typical bio pipeline is: parse with a format-specific library → pandas for wrangling and joining → NumPy for the matrix math → seaborn for the figure.

Core notes

The objects Series · DataFrame · Index Objects · MultiIndex

Getting data in and out Reading and Writing Data · pandas dtypes · Categorical dtype · Missing Data in pandas

Selecting loc vs iloc · Filtering and query · Copy-on-Write

Transforming assign and pipe · String Accessor · Datetime Handling in pandas · Sorting and Ranking

Combining and reshaping groupby · Merging and Joining · Concat · Reshaping with pivot and melt · Window Functions

Making it fast pandas Performance

Concepts

Tidy Data · Split-Apply-Combine

Applied

Applied - Reading VCF with pandas · Applied - Annotating a Variant Table · Applied - GFF and Genomic Intervals · Applied - Differential Expression Volcano Plot

Install and check

pip install "pandas>=3.0,<4" pyarrow
python -c "import pandas as pd; print(pd.__version__); pd.show_versions()"

Install pyarrow. It is technically optional but the string dtype, several IO paths, and memory usage are all substantially better with it.

The 20% that gets 80% of the work done

import pandas as pd

df = pd.read_csv("variants.tsv", sep="\t")

df.head(); df.info(); df.describe()
df.loc[df["af"] < 0.01, ["gene", "consequence"]]
df.assign(log_af=lambda d: np.log10(d["af"]))
df.groupby("gene")["af"].agg(["size", "mean"])
df.merge(genes, on="gene", how="left", validate="many_to_one")
df.sort_values("af", ascending=False).head(20)
df["consequence"].value_counts(dropna=False)

value_counts(dropna=False) and validate= on merges are the two habits that catch the most silent data errors.

Gotchas that bite newcomers

• Silent index misalignment producing NaN. df["new"] = other_series aligns on index; use .to_numpy() or .reset_index(drop=True) if you meant positional.
• merge duplicating rows because the join key was not unique. Always pass validate=.
• inplace=True is not faster and is being discouraged; it copies anyway under CoW.
• Reading a genomic TSV without dtype= — chromosome "X" and "1" in one column forces object/str, and sample_01 sample IDs get mangled by inference. Be explicit.
• .apply() on rows (axis=1) is a Python loop wearing a costume. See pandas Performance.

See also

NumPy · Seaborn · Ecosystem Map · Learning Path