pythonforbio
WASM
Library Overview04/06
Starting sandbox…
Beginner

Pandas

Pandas

Core Principle: pandas provides labeled, heterogeneously-typed DataFrame and Series data structures designed for tabular manipulation, relational joins, and alignment across biological datasets.

Version at time of writing: 3.0.x (released 21 Jan 2026) · import pandas as pd

Architectural Model

A DataFrame is a dict of columns, where each column is a Series consisting of a 1D array associated with a shared Index. Two things follow, and they explain most pandas behaviour:

  1. Alignment. Almost every binary operation aligns on the index first. s1 + s2 matches labels, not positions, and produces NaN where labels are missing. This is the single biggest difference from NumPy and the source of most "why is my result all NaN" questions.
  2. Column-oriented storage. Operations along a column are fast; operations along a row are not. Adding a row is expensive; adding a column is cheap.

Key Architectural Changes in pandas 3.0

Three breaking changes make pre-2026 tutorials wrong:

  • Copy-on-Write is mandatory. Chained assignment (df["a"][mask] = x) silently does nothing now. Use .loc. SettingWithCopyWarning no longer exists because the ambiguity is gone.
  • Strings default to str dtype, not object, backed by PyArrow when installed. Any dtypes == object check for text columns is now wrong.
  • **Datetimes default to microsecond resolution rather than nanoseconds, removing the historical 1678–2262 timestamp boundary.

Details in Versions and Compatibility.

Computational Relevance in Biology

Every annotation table, sample sheet, variant call set, differential-expression result and QC report is a heterogeneous table with labels. Standard genomic formats frequently encapsulate multiple variables within individual columns (such as the VCF INFO field or the GFF attribute column); the pandas .str accessor and .explode() method allow systematic parsing of these nested representations.

The typical bio pipeline is: parse with a format-specific library → pandas for wrangling and joining → NumPy for the matrix math → seaborn for the figure.

Core Reference Modules

The objects Series · DataFrame · Index Objects · MultiIndex

Getting data in and out Reading and Writing Data · pandas dtypes · Categorical dtype · Missing Data in pandas

Selecting loc vs iloc · Filtering and query · Copy-on-Write

Transforming assign and pipe · String Accessor · Datetime Handling in pandas · Sorting and Ranking

Combining and reshaping groupby · Merging and Joining · Concat · Reshaping with pivot and melt · Window Functions

Making it fast pandas Performance

Concepts

Tidy Data · Split-Apply-Combine

Applied Workflows

Applied - Reading VCF with pandas · Applied - Annotating a Variant Table · Applied - GFF and Genomic Intervals · Applied - Differential Expression Volcano Plot

Installation and Verification

pip install "pandas>=3.0,<4" pyarrow
python -c "import pandas as pd; print(pd.__version__); pd.show_versions()"

Install pyarrow. It is technically optional but the string dtype, several IO paths, and memory usage are all substantially better with it.

Essential API Patterns

import pandas as pd

df = pd.read_csv("variants.tsv", sep="\t")

df.head(); df.info(); df.describe()
df.loc[df["af"] < 0.01, ["gene", "consequence"]]
df.assign(log_af=lambda d: np.log10(d["af"]))
df.groupby("gene")["af"].agg(["size", "mean"])
df.merge(genes, on="gene", how="left", validate="many_to_one")
df.sort_values("af", ascending=False).head(20)
df["consequence"].value_counts(dropna=False)

value_counts(dropna=False) and validate= on merges are the two habits that catch the most silent data errors.

Common Pitfalls & Behavioral Edge Cases

  • Silent index misalignment producing NaN. df["new"] = other_series aligns on index; use .to_numpy() or .reset_index(drop=True) if you meant positional.
  • merge duplicating rows because the join key was not unique. Always pass validate=.
  • inplace=True is not faster and is being discouraged; it copies anyway under CoW.
  • Reading a genomic TSV without explicit dtype specifications: chromosome "X" and "1" in one column forces object/str, and sample_01 sample IDs get mangled by inference. Be explicit.
  • .apply() on rows (axis=1) is a Python loop wearing a costume. See pandas Performance.

Related References

NumPy · Seaborn · Ecosystem Map · Learning Path