MultiIndex

Core Principle: A MultiIndex provides hierarchical, multi-level labeling across rows or columns, enabling the representation and manipulation of higher-dimensional datasets within 2D DataFrames.

Creating one

df.set_index(["chrom", "pos"])
df.groupby(["gene", "sample"]).mean()          # grouping by 2+ keys produces one
df.pivot_table(index="gene", columns=["tissue", "condition"])

pd.MultiIndex.from_tuples([("chr1", 100), ("chr1", 200)], names=["chrom", "pos"])
pd.MultiIndex.from_product([["chr1", "chr2"], [100, 200]], names=["chrom", "pos"])
pd.MultiIndex.from_arrays([chroms, positions], names=["chrom", "pos"])

from_product is how you build a complete grid: every chromosome × every bin: which is useful for reindexing sparse results into a full matrix.

Selecting

df.loc["chr1"]                        # whole first level → drops that level
df.loc[("chr1", 7676154)]             # a full key
df.loc[("chr1", 7676154), "ref"]      # key + column
df.loc["chr1":"chr3"]                 # slice the outer level (needs sorting)

# partial selection on inner levels needs slice(None) or IndexSlice
idx = pd.IndexSlice
df.loc[idx[:, 7676154], :]            # all chromosomes, that position
df.loc[idx["chr1", 100:200], :]

df.xs("chr1", level="chrom")          # cross-section, drops the level
df.xs(7676154, level="pos")           # select on an inner level directly

.xs is the readable option for "give me one value of one level". IndexSlice is more general but harder to read.

Sort before you slice

df = df.sort_index()
df.index.is_monotonic_increasing      # must be True for range slicing

Slicing an unsorted MultiIndex raises UnsortedIndexError, or is quietly slow. Sort immediately after creating one. This is the single most common MultiIndex frustration.

Level manipulation

df.index.names                            # ['chrom', 'pos']
df.index.get_level_values("chrom")        # flat array of that level
df.index.levels                            # the unique values per level

df.reset_index(level="pos")               # one level → column
df.reset_index()                           # all levels → columns
df.swaplevel("chrom", "pos").sort_index()
df.droplevel("chrom")
df.reorder_levels(["pos", "chrom"])

Grouping and stacking

df.groupby(level="chrom").size()
df.groupby(level=[0, 1]).mean()
df.sum(level=...)                          # removed: use groupby(level=...)

df.stack()      # columns → index level   (wider → longer)
df.unstack()    # index level → columns   (longer → wider)
df.unstack(level="condition")
df.unstack(fill_value=0)

stack/unstack are the MultiIndex-aware cousins of melt/pivot. See Reshaping with pivot and melt.

MultiIndex on columns

Just as common, and often the more useful direction:

expr.columns = pd.MultiIndex.from_arrays(
    [tissues, conditions, replicates], names=["tissue", "condition", "rep"]
)

expr["liver"]                                        # all liver samples
expr.loc[:, idx[:, "treated", :]]                    # all treated, any tissue
expr.T.groupby(level=["tissue", "condition"]).mean().T   # collapse replicates

That last line: averaging technical replicates within condition: is a genuinely elegant use of a column MultiIndex, and awkward without one.

When it earns its complexity

Worth it:

• Genuinely hierarchical keys: (chromosome, position), (gene, transcript, exon), (patient, timepoint).
• The output of a multi-key groupby you will immediately unstack.
• Repeated selection on the outer level of a large frame: the sorted index makes it fast.
• Column hierarchies over a wide experimental design.

Not worth it:

• As a habit. Flat columns plus groupby handles most work and is far easier to read, merge and debug.
• When you will immediately merge with something else. merge on MultiIndexes is fiddly; reset_index() first.
• When collaborators will read your code. MultiIndex selection syntax has a real learning cost.

The honest advice: reach for it when the data is hierarchical and you will exploit that; otherwise reset_index() and move on.

Bioinformatics examples

# a long-format expression table indexed by gene and sample
long = counts.melt(id_vars="gene", var_name="sample", value_name="count")
mi = long.set_index(["gene", "sample"]).sort_index()

mi.loc["TP53"]                              # all samples for one gene
mi.loc[("TP53", "sample_3"), "count"]       # one value
mi.groupby(level="gene")["count"].mean()

# variants keyed by locus
v = variants.set_index(["chrom", "pos"]).sort_index()
v.loc["chr17"]                              # all chr17 variants
v.loc[("chr17", slice(7_670_000, 7_680_000))]   # a genomic window

# a summary table that reads well
summary = de.groupby(["tissue", "direction"])["gene"].count().unstack(fill_value=0)

Common mistakes

• Slicing without sorting → UnsortedIndexError.
• Forgetting the tuple. df.loc["chr1", 7676154] is interpreted as (row, column), not as a two-level key. You need df.loc[("chr1", 7676154)].
• Levels retaining unused categories after filtering: df.index.levels still lists them. Use df.index.remove_unused_levels().
• Merging on a MultiIndex without resetting first.
• to_csv round-trip: a MultiIndex writes fine but needs index_col=[0, 1] on read.
• Using it when flat columns would do. Complexity you pay for on every subsequent line.

See also

Index Objects · groupby · Reshaping with pivot and melt · loc vs iloc · DataFrame

Exercise

Write window_report(text). The first line of stdin is a query: a chromosome name, a start position and an end position separated by spaces. Every following line is a variant: chromosome, position and quality, also space separated. Return a single string.

Index the variants by (chrom, pos), sort the index, and select the requested closed window start <= pos <= end on that chromosome by slicing the MultiIndex: not by boolean masking the columns. Return <chrom> n=<count> mean=<mean qual to 3 dp> top=<position of the highest quality>. When the window contains no variants: including when the chromosome is absent from the data entirely: return <chrom> n=0 and nothing more. If two variants tie on quality, report the lower position.