Sorting and Ranking

Core Principle: Ordering operations in pandas utilize sort_values() for data sorting, sort_index() for label ordering, and rank() for computing empirical quantile positions with configurable tie-breaking methods.

Sorting

df.sort_values("padj")
df.sort_values("padj", ascending=False)
df.sort_values(["chrom", "pos"])                       # multi-key
df.sort_values(["chrom", "pos"], ascending=[True, False])
df.sort_values("padj", na_position="first")            # default is 'last'
df.sort_values("padj", kind="stable")                  # preserve original order for ties
df.sort_index()
df.sort_index(axis=1)                                   # sort the columns
df.sort_values("gene", key=lambda s: s.str.lower())     # sort by a transformation

na_position="last" is the default, which is usually right for p-values (missing is not significant) and usually wrong for anything where missing should be visible. Decide deliberately.

kind="stable" matters when you sort twice. An unstable sort can reorder tied rows arbitrarily, so results are not reproducible across pandas versions or platforms. If your output includes tied values and you care about determinism, use it: or add a tiebreaker column.

Ranking

s.rank()                             # average rank for ties (default)
s.rank(method="min")                  # competition ranking: 1, 2, 2, 4
s.rank(method="dense")                # no gaps: 1, 2, 2, 3
s.rank(method="first")                # break ties by original order
s.rank(ascending=False)
s.rank(pct=True)                      # percentile rank in [0, 1]
s.rank(na_option="keep")              # NaN stays NaN (default)

method="dense" is the one you usually want for a human-readable ranking. method="average" (the default) produces fractional ranks like 2.5, which are correct for rank-based statistics (Spearman, Wilcoxon) and odd-looking in a report.

Ranking within groups

df["rank_in_gene"] = df.groupby("gene")["af"].rank(ascending=False)
df["pct_in_sample"] = df.groupby("sample")["count"].rank(pct=True)

Per-group ranking is a groupby().rank(), not a transform: rank already returns one value per row. See groupby.

Top N without a full sort

df.nlargest(20, "log2fc")
df.nsmallest(20, "padj")
df.nlargest(20, ["log2fc", "baseMean"])      # tiebreak on the second column
df.nlargest(5, "x", keep="all")              # include all ties

nlargest uses a partial selection algorithm: meaningfully faster than sort_values().head(n) on large frames, same as np.argpartition in NumPy. See Fancy Indexing.

Genomic sort order

The recurring problem: chromosomes do not sort alphabetically.

['chr1', 'chr10', 'chr2', 'chrX']

The fix is an ordered categorical, set once:

CHROM_ORDER = [f"chr{i}" for i in range(1, 23)] + ["chrX", "chrY", "chrM"]
df["chrom"] = pd.Categorical(df["chrom"], categories=CHROM_ORDER, ordered=True)
df = df.sort_values(["chrom", "pos"])

This fixes sorting, groupby output order, and: importantly: the x-axis order of every seaborn plot, all at once. Do it at load time. See Categorical dtype.

Any value not in CHROM_ORDER becomes NaN, so decide what to do with scaffolds and alt contigs explicitly:

main = df[df["chrom"].isin(CHROM_ORDER)]
print(f"dropped {len(df) - len(main):,} rows on non-primary contigs")

Sorting is not free

df.sort_values("pos")                    # O(n log n), copies the whole frame
df.sort_values("pos", ignore_index=True) # skip index bookkeeping
df.set_index("pos").sort_index()          # pays once, then .loc is fast

If you slice a large frame by position repeatedly, sorting once into an index pays for itself. If you sort inside a loop, you have a problem.

Bioinformatics examples

# DE results, most significant first, ties broken deterministically
de.sort_values(["padj", "gene"], kind="stable")

# canonical genomic order
variants = variants.sort_values(["chrom", "pos", "ref", "alt"])

# top expressed genes per sample
long.sort_values("count", ascending=False).groupby("sample").head(10)

# best transcript per gene (lowest p, then longest)
best = (variants
        .sort_values(["pvalue", "transcript_length"], ascending=[True, False])
        .drop_duplicates("gene", keep="first"))

# percentile of each gene's expression within its sample
long["expr_pct"] = long.groupby("sample")["count"].rank(pct=True)

# rank genes for GSEA (signed, by significance)
de["gsea_rank"] = np.sign(de["log2FoldChange"]) * -np.log10(de["pvalue"])
ranked = de.sort_values("gsea_rank", ascending=False)

The sort_values().groupby().head(n) pattern in the third example is the idiomatic "top N per group" and is much faster than groupby().apply(lambda g: g.nlargest(n, ...)).

Common mistakes

• Alphabetical chromosome order. chr1, chr10, chr11... in a published figure.
• Not setting kind="stable" when tied rows must stay in a fixed order.
• NaN position surprises. Default is last.
• Sorting inside a loop.
• Forgetting reset_index(drop=True) after sorting, leaving the index scrambled relative to position.
• rank() default returning fractional ranks in a report.
• Sorting a categorical without ordered=True: you get the category order, which may not be what you set.
• Losing non-primary contigs silently when applying a categorical chromosome order.

See also

Categorical dtype · groupby · Filtering and query · Index Objects · Fancy Indexing

Exercise

Write canonical_order(text). Stdin holds one variant per line: chromosome, position and score, separated by spaces. Return a list of strings.

Drop every variant whose chromosome is not in CHROM_ORDER, then put the survivors in genomic order: chromosome in CHROM_ORDER order, not alphabetical order, then position ascending: by making the chromosome an ordered categorical and sorting on it. Rank the survivors by score, highest first, using dense ranking so that tied scores share a rank and the next rank follows immediately with no gap.

Return dropped=<n> first, counting the variants removed for being on a non-primary contig, then one chrom:pos<TAB>rank line per surviving variant in genomic order. When every variant is dropped, dropped=<n> is the only line.