Split-Apply-Combine
In one line: the universal shape of grouped computation — partition the data, run something per partition, reassemble.
Named by Hadley Wickham (2011). Once you see it you cannot unsee it: it is groupby in pandas, GROUP BY in SQL, MapReduce, np.add.reduceat, and every "per-gene", "per-sample", "per-chromosome" sentence you have ever written.
The three steps
┌──── split ────┐ ┌── apply ──┐ ┌── combine ──┐
data ──▶│ group by key │──▶│ f(group) │──▶│ result │
└───────────────┘ └───────────┘ └─────────────┘
In pandas:
df.groupby("gene")["count"].mean()
# split ──────┘ └─ apply ─┘ combine is implicit
Three apply shapes, three different outputs
This is the distinction people get wrong. What you get back depends on what your function returns.
| Method | Function returns | Output shape |
|---|---|---|
.agg() |
one scalar per group | one row per group |
.transform() |
one value per original row | same shape as input |
.filter() |
a boolean per group | subset of original rows |
.apply() |
anything | pandas guesses; slowest |
# aggregate: 20,000 genes in → 20,000 rows out
df.groupby("gene")["count"].agg("mean")
# transform: 5,000,000 rows in → 5,000,000 rows out (broadcast back)
df["centred"] = df["count"] - df.groupby("gene")["count"].transform("mean")
# filter: keep only genes observed in >10 samples
df.groupby("gene").filter(lambda g: len(g) > 10)
transform is the one worth internalising — per-group normalization (z-scoring within a batch, TPM within a sample, centring within a gene) is exactly this and people reach for a merge instead.
The same idea in NumPy
Without labels, grouping is done positionally:
# mean per row (per gene) of an expression matrix
counts.mean(axis=1)
# grouped sums over sorted group boundaries
np.add.reduceat(values, group_start_indices)
# group counts / sums by integer code
np.bincount(codes)
np.bincount(codes, weights=values)
See Axes and Reductions. The axis= argument is split-apply-combine where the groups are "everything along this axis".
Why it matters in bioinformatics
Practically every question in the field is a grouped one:
- variants per gene, per sample, per consequence class
- mean coverage per exon
- differential expression: a test per gene
- allele frequency per population
- per-sample QC metrics from per-read data
The unit you group by is usually the observational unit from Tidy Data. If grouping feels awkward, your frame is usually not tidy yet.
Performance note
Prefer built-in aggregation strings ("mean", "sum", "size", "nunique") over lambdas. The strings dispatch to compiled Cython; a lambda runs Python once per group. On 20,000 gene groups that is a 50–100× difference. See pandas Performance.
See also
groupby · Tidy Data · Axes and Reductions · Window Functions · pandas Performance · Applied - Expression Matrices with NumPy