Split-Apply-Combine

Core Principle: The split-apply-combine strategy structures data processing by partitioning a dataset into subsets, applying a function to each partition independently, and combining the results into a unified output.

Named by Hadley Wickham (2011). Once you see it you cannot unsee it: it is groupby in pandas, GROUP BY in SQL, MapReduce, np.add.reduceat, and every "per-gene", "per-sample", "per-chromosome" sentence you have ever written.

The three steps

In pandas:

df.groupby("gene")["count"].mean()
#     split ──────┘        └─ apply ─┘  combine is implicit

Three apply shapes, three different outputs

This is the distinction people get wrong. What you get back depends on what your function returns.

Method	Function returns	Output shape
.agg()	one scalar per group	one row per group
.transform()	one value per original row	same shape as input
.filter()	a boolean per group	subset of original rows
.apply()	anything	pandas guesses; slowest

# aggregate: 20,000 genes in → 20,000 rows out
df.groupby("gene")["count"].agg("mean")

# transform: 5,000,000 rows in → 5,000,000 rows out (broadcast back)
df["centred"] = df["count"] - df.groupby("gene")["count"].transform("mean")

# filter: keep only genes observed in >10 samples
df.groupby("gene").filter(lambda g: len(g) > 10)

The transform operation is particularly valuable for per-group normalization, including batch z-score calculations, per-sample transcript abundance scaling, and gene-level mean centering, eliminating the need for intermediate table merges.

The same idea in NumPy

Without labels, grouping is done positionally:

# mean per row (per gene) of an expression matrix
counts.mean(axis=1)

# grouped sums over sorted group boundaries
np.add.reduceat(values, group_start_indices)

# group counts / sums by integer code
np.bincount(codes)
np.bincount(codes, weights=values)

See Axes and Reductions. The axis= argument is split-apply-combine where the groups are "everything along this axis".

Why it matters in bioinformatics

Practically every question in the field is a grouped one:

• variants per gene, per sample, per consequence class
• mean coverage per exon
• differential expression: a test per gene
• allele frequency per population
• per-sample QC metrics from per-read data

The unit you group by is usually the observational unit from Tidy Data. If grouping feels awkward, your frame is usually not tidy yet.

Performance note

Prefer built-in aggregation strings ("mean", "sum", "size", "nunique") over lambdas. The strings dispatch to compiled Cython; a lambda runs Python once per group. On 20,000 gene groups that is a 50–100× difference. See pandas Performance.

See also

groupby · Tidy Data · Axes and Reductions · Window Functions · pandas Performance · Applied - Expression Matrices with NumPy

Exercise

Write grouped(rows), where rows is a list of (gene, sample, count) triples in file order. Implement the pattern by hand with a dictionary rather than with pandas.groupby, so that the three steps stay visible: split the rows by gene, discard every group holding fewer than two rows, and then run both an aggregation and a transform over the groups that remain.

Return the output lines in this order. First, one aggregate line per surviving gene in alphabetical order, formatted <gene> n=<rows> mean=<mean> with the mean to three decimal places. Second, one transform line per row belonging to a surviving gene, in the original row order, formatted <gene> <sample> <centred> where centred is that row's count minus its own group's mean, to three decimal places. Finish with a single line groups=<total> kept=<surviving>. When every group holds one row, only that last line is printed.