Categorical dtype

In one line: store repeated strings once and reference them by integer code — huge memory savings, plus a defined ordering when you need one.

The mechanism

s = pd.Series(["chr1", "chr2", "chr1", "chr1"], dtype="category")

s.cat.categories       # Index(['chr1', 'chr2'])  — stored once
s.cat.codes            # array([0, 1, 0, 0], dtype=int8)  — the actual storage

A 10-million-row chromosome column: ~600 MB as strings, ~10 MB as a category (one int8 per row plus 25 strings). That is the headline benefit.

Creating

df["chrom"] = df["chrom"].astype("category")
pd.read_csv(path, dtype={"chrom": "category"})     # best — never materialises the strings

pd.Categorical(values, categories=["low", "med", "high"], ordered=True)
pd.cut(df["af"], bins=[0, 0.01, 0.05, 1], labels=["rare", "low", "common"])
pd.qcut(df["expr"], q=4, labels=["Q1", "Q2", "Q3", "Q4"])

pd.cut (fixed bin edges) and pd.qcut (equal-count quantile bins) both return categoricals, which is why binning and categoricals usually arrive together.

Ordered categoricals

The second reason to use them: a meaningful non-alphabetical order.

sev = pd.Categorical(
    df["impact"],
    categories=["MODIFIER", "LOW", "MODERATE", "HIGH"],
    ordered=True,
)

df["impact"] = sev
df[df["impact"] >= "MODERATE"]          # comparison works!
df.sort_values("impact")                 # sorts by severity, not alphabetically
df["impact"].max()                       # 'HIGH'

Without this, sorting variant impacts gives you HIGH, LOW, MODERATE, MODIFIER — alphabetical and meaningless. The same applies to timepoints (day0, day7, day14 sorts wrong as strings), dose levels, and disease stages.

Ordered categoricals also fix plot ordering. seaborn respects category order, so setting it once fixes every subsequent figure. See seaborn Themes and Palettes.

Managing categories

s.cat.categories
s.cat.add_categories(["chrM"])
s.cat.remove_categories(["chrUn"])
s.cat.remove_unused_categories()          # important after filtering!
s.cat.rename_categories({"chr1": "1"})
s.cat.reorder_categories([...], ordered=True)
s.cat.set_categories(ALL_CHROMS)          # enforce a canonical set

remove_unused_categories() after filtering matters more than it sounds. Filter to chr17 only, and the column still lists all 25 categories. groupby will then produce 25 groups, 24 of them empty, and your plot gets 24 empty x-axis slots. (Modern groupby defaults to observed=True, which mitigates this, but plotting and value_counts still see the full list.)

Where it bites

1. Assigning a new value.

df["chrom"] = df["chrom"].astype("category")
df.loc[0, "chrom"] = "chrM"       # if chrM is not a category → error or NaN
df["chrom"] = df["chrom"].cat.add_categories(["chrM"])    # add it first

2. Concatenating frames with different categories.

pd.concat([df1, df2])      # differing categories → falls back to object dtype

Fix by setting the same categories on both first, or use union_categoricals.

3. Comparing two differently-categoried columns raises TypeError. Comparing a category column to a plain string is fine.

4. String operations. .str accessors work but operate on the categories, and the result may not stay categorical.

When to use it

Yes:

• Chromosome, strand, REF/ALT bases, FILTER, consequence, impact
• Sample group, condition, tissue, batch
• Any low-cardinality repeated string in a large frame
• Anything with a meaningful non-alphabetical order
• Anything you will groupby repeatedly (grouping on codes is faster)

No:

• Variant IDs, read names, gene IDs in a per-gene table (cardinality ≈ row count)
• Free-text fields
• Columns you will heavily mutate
• Small frames where the complexity is not worth it

The heuristic: use it when nunique() / len() is below roughly 0.5, and especially when it is below 0.05.

Bioinformatics example

CHROMS = [f"chr{i}" for i in range(1, 23)] + ["chrX", "chrY", "chrM"]
IMPACTS = ["MODIFIER", "LOW", "MODERATE", "HIGH"]

variants = pd.read_csv("variants.tsv", sep="\t", dtype={
    "chrom": "category", "ref": "category", "alt": "category",
    "consequence": "category", "filter": "category",
})

# canonical chromosome order — not alphabetical (chr10 < chr2 alphabetically!)
variants["chrom"] = variants["chrom"].cat.set_categories(CHROMS, ordered=True)
variants["impact"] = pd.Categorical(variants["impact"], categories=IMPACTS, ordered=True)

variants.sort_values(["chrom", "pos"])           # correct genomic order
variants[variants["impact"] >= "MODERATE"]
variants.groupby("chrom", observed=False).size() # all chroms, including empty ones

print(variants.memory_usage(deep=True).sum() / 1e6, "MB")

The chromosome ordering point deserves emphasis: alphabetically, chr10 comes before chr2. Every chromosome-ordered plot you have ever seen with chr1, chr10, chr11... instead of chr1, chr2, chr3 is this bug. An ordered categorical fixes it once, everywhere.

observed= in groupby

df.groupby("chrom", observed=True).size()     # only chromosomes present
df.groupby("chrom", observed=False).size()    # all categories, zeros for absent

observed=True is the modern default. Pass observed=False deliberately when you want the zeros — e.g. a per-chromosome count table where absent chromosomes should show 0 rather than vanish.

Common mistakes

• Assigning an unlisted category.
• concat silently degrading to object on mismatched categories.
• Forgetting remove_unused_categories() after filtering.
• Categorizing a high-cardinality column — costs memory instead of saving it.
• Assuming default alphabetical order is fine for chromosomes, timepoints, or severity.
• Not setting ordered=True and then wondering why >= fails.
• astype("category") after loading instead of dtype= at read time — you still pay the peak memory of the string column.

See also

pandas dtypes · groupby · Sorting and Ranking · String Accessor · seaborn Themes and Palettes · pandas Performance