Index Objects

Core Principle: An Index is an immutable, array-like object that holds coordinate labels for DataFrame axes, automatically driving label alignment, relational indexing, and lookup operations.

What an Index is

1

It is immutable. idx[0] = "X" raises. To change labels you build a new Index (df.index = [...], df.rename(...)).

Both axes have one: df.index for rows, df.columns for columns. They are the same class, which is why df.columns.str.startswith("sample_") works.

The three jobs an Index does

1. Alignment. Every binary operation between two pandas objects aligns on the index first. This is the source of both pandas' power and its most confusing behaviour. See Series.

2. Fast lookup. A unique Index builds a hash table, so df.loc["TP53"] is O(1) rather than a scan. On a 20,000-gene frame that is the difference between instant and noticeable.

3. Metadata. The index says what each row is. expr.loc["TP53", "sample_3"] is self-documenting in a way expr[4711, 2] never is.

Index types

Type	Use
Index	general (str, object)
RangeIndex	default 0..n-1; stores only start/stop/step, so it is free
DatetimeIndex	time series; enables .loc["2026-01"] partial-string slicing
CategoricalIndex	repeated labels, memory-efficient
IntervalIndex	ranges: genomic intervals! see below
MultiIndex	hierarchical: see MultiIndex

Setting and resetting

df.set_index("gene")                    # a column becomes the index
df.set_index(["chrom", "pos"])          # → MultiIndex
df.reset_index()                        # index becomes a column again
df.reset_index(drop=True)               # discard it entirely
df.rename_axis("gene_id")               # rename the index itself
df.index = new_labels                   # direct assignment

When to set an index: when you will repeatedly look up by that key, when you will align two frames on it, or when it genuinely identifies the row. A gene ID on an expression matrix, yes. An arbitrary row number, no: leave the RangeIndex.

reset_index(drop=True) after filtering is a habit worth forming. Otherwise your index has gaps (0, 3, 7, 11...), which is harmless until something downstream assumes positions and labels agree.

Uniqueness matters

df.index.is_unique
df.index.duplicated().sum()
df[df.index.duplicated(keep=False)]     # show all duplicate rows

A non-unique index makes .loc["TP53"] return a DataFrame instead of a Series, which then breaks the code expecting a scalar. Worse, joins on a non-unique key multiply rows. Check uniqueness after any operation that could introduce duplicates.

Set operations

a.index.intersection(b.index)      # genes in both
a.index.union(b.index)
a.index.difference(b.index)        # genes only in a
a.index.isin(panel)                # boolean mask

df.reindex(new_index)                     # conform to a new index; NaN for missing
df.reindex(new_index, fill_value=0)
a.align(b, join="inner", axis=0)          # align two frames explicitly

intersection is the clean way to find shared genes across datasets: much better than converting to sets and losing order.

reindex is subtly different from .loc[]: reindex tolerates missing labels (fills NaN), .loc with a list of missing labels raises. Use reindex when you want to force a specific ordering, e.g. matching a metadata table's sample order.

IntervalIndex for genomic ranges

bins = pd.IntervalIndex.from_breaks([0, 1e6, 2e6, 3e6], closed="left")
binned = pd.cut(variants["pos"], bins)
variants.groupby(binned, observed=True).size()

pd.cut plus an IntervalIndex gives you binning with readable labels. For real interval overlap queries (which genes does this variant fall in?), pandas is the wrong tool: use pyranges, bioframe, or an interval tree. See Applied - GFF and Genomic Intervals.

Alignment in action

expr = pd.read_csv("counts.tsv", index_col=0)      # genes × samples
meta = pd.read_csv("samples.tsv", index_col=0)     # samples × attributes

# align samples explicitly rather than assuming the order matches
common = expr.columns.intersection(meta.index)
expr = expr[common]
meta = meta.loc[common]
assert (expr.columns == meta.index).all()

That assertion is worth writing every time. Sample order mismatch between a count matrix and a metadata table is a classic, silent, result-inverting bug: you compare the wrong groups and get a beautiful, wrong figure.

Performance

df.index.is_monotonic_increasing         # sorted?
df.sort_index()                           # sort: enables faster slicing

.loc on a sorted index can binary-search; on an unsorted non-unique index it must scan. For large frames you slice repeatedly, sorting once pays for itself.

Common mistakes

• Duplicate index values silently changing .loc return types and multiplying join output.
• Assuming two frames' rows correspond because they have the same length. Align explicitly.
• Forgetting reset_index(drop=True) after filtering, leaving gaps.
• df.index = list(...) of the wrong length → ValueError, or worse, silently wrong if lengths coincidentally match.
• Setting an index you then need as a column. .reset_index() gets it back, but grouping and merging are often easier with it as a plain column.
• Expecting the index to survive to_csv without index=True, or getting a spurious Unnamed: 0 column when reading back without index_col=0.

See also

Series · DataFrame · MultiIndex · loc vs iloc · Merging and Joining · Applied - GFF and Genomic Intervals

Exercise

Write align_counts(a_labels, b_labels, b_values). The first argument is the row labels of a target frame, which may contain repeats; the second and third are the labels and integer values of a second table, whose labels are unique. Return a list of four report lines.

The lines are a_unique=<True|False> for the target labels as given, then shared=<n> counting how many distinct target labels also appear in the second table, then missing=<labels> listing the distinct target labels absent from the second table as a comma-separated string in target order, or missing=none when there are none, then total=<sum> for the second table's values reindexed onto the deduplicated target labels with absent labels filled with zero. Report uniqueness before you drop the repeats, and keep the first occurrence of each repeated label.