Window Functions

In one line: rolling, expanding and shifted computations — for smoothing coverage, computing running totals, and comparing each row to its neighbours.

Rolling

s.rolling(window=100).mean()
s.rolling(window=100, center=True).mean()      # centred rather than trailing
s.rolling(window=100, min_periods=1).mean()    # emit values at the edges
s.rolling(window=100).agg(["mean", "std", "min", "max"])
s.rolling(window=100).quantile(0.5)
s.rolling(window=100).apply(custom_fn, raw=True)   # raw=True passes ndarray — faster

The edge behaviour is the thing to decide up front. By default the first window-1 positions are NaN, because there is not a full window yet. min_periods=1 fills them using whatever data exists, which is usually what you want for a coverage track and usually not what you want for a statistic you will interpret.

center=True matters for genomics: an uncentred rolling mean shifts every feature to the right by half a window. On a coverage plot that is a visible, wrong offset.

Window types

s.rolling(100).mean()                                  # flat/rectangular
s.rolling(100, win_type="gaussian").mean(std=20)       # weighted
s.rolling("10min").mean()                               # time-based (needs DatetimeIndex)
s.ewm(span=20).mean()                                   # exponentially weighted
s.ewm(halflife=10).mean()

ewm (exponentially weighted) gives smoothing with no edge NaNs and a smooth response — often nicer than a rectangular rolling mean for visualisation.

Expanding

Cumulative, from the start to each point:

s.expanding().mean()          # running mean
s.expanding().max()           # running max
s.cumsum(); s.cummax(); s.cumprod()

Shift and diff

s.shift(1)              # previous value
s.shift(-1)             # next value
s.diff()                # s - s.shift(1)
s.diff(2)
s.pct_change()

df["gap"] = df["pos"] - df["pos"].shift(1)         # distance to the previous variant
df["is_next_to"] = df["gap"] < 10

shift and diff are how you compare each row to its neighbours. In genomics that means inter-variant distances, exon gaps, and detecting runs of adjacent features.

Grouped windows — the crucial bit

Rolling and shifting across group boundaries is a silent bug. A rolling mean over a position-sorted variant table will happily average the end of chr1 with the start of chr2.

# WRONG — smooths across chromosome boundaries
df["smooth"] = df["depth"].rolling(100, center=True).mean()

# RIGHT
df = df.sort_values(["chrom", "pos"])
df["smooth"] = (
    df.groupby("chrom")["depth"]
      .transform(lambda s: s.rolling(100, center=True, min_periods=1).mean())
)

# same for shift/diff
df["gap"] = df.groupby("chrom")["pos"].diff()      # NaN at each chromosome start ✓

The groupby().transform() form is the pattern. See groupby.

Rolling on a MultiIndex or with a time index

df.groupby("sample").rolling(window=10).mean()      # produces a MultiIndex
df.set_index("time").groupby("sample").rolling("1h").mean()

Time-based windows ("1h", "7D") require a sorted DatetimeIndex and are genuinely different from count-based windows when the sampling is irregular. See Datetime Handling in pandas.

Bioinformatics examples

# smooth a per-base coverage track for plotting
cov = cov.sort_values(["chrom", "pos"])
cov["smoothed"] = (
    cov.groupby("chrom")["depth"]
       .transform(lambda s: s.rolling(1000, center=True, min_periods=1).mean())
)

# GC content in sliding windows
gc = is_gc.rolling(100, center=True).mean()

# detect coverage dropouts: sustained low regions
low = cov["smoothed"] < 0.2 * cov["smoothed"].median()
runs = (low != low.shift()).cumsum()               # ← run-length encoding idiom
dropouts = cov[low].groupby(runs).agg(
    start=("pos", "min"), end=("pos", "max"), n=("pos", "size")
).query("n >= 50")

# distance to the nearest previous variant, per chromosome
variants["dist_prev"] = variants.groupby("chrom")["pos"].diff()

# rolling allele-frequency deviation, for LOH detection
baf["rolling_dev"] = (
    baf.groupby("chrom")["baf"]
       .transform(lambda s: (s - 0.5).abs().rolling(50, center=True).mean())
)

# running total along a chromosome
cov["cumulative"] = cov.groupby("chrom")["depth"].cumsum()

The run-length encoding idiom in the dropout example is worth memorising:

runs = (flag != flag.shift()).cumsum()

Each time the flag changes, the cumulative sum increments, so consecutive identical values share a run id. Grouping by it gives you contiguous blocks — the basis of segment detection, CNV calling, and any "find stretches where X is true" problem.

Performance

s.rolling(100).mean()                      # compiled — fast
s.rolling(100).apply(np.mean)              # Python per window — very slow
s.rolling(100).apply(f, raw=True)          # raw ndarray instead of Series — faster
s.rolling(100, engine="numba").mean()      # for large data with a custom function

Prefer the built-in aggregations. If you need apply, always pass raw=True — it skips constructing a Series per window.

For very large arrays with a simple window, NumPy's sliding_window_view is faster still and allocates nothing. See Memory Layout and Strides.

Common mistakes

• Rolling across chromosome/sample boundaries. The defining error of this note.
• Not sorting before rolling. Window functions assume order and will not warn.
• Forgetting center=True, shifting every feature by half a window.
• Leading NaNs breaking a downstream step. Decide min_periods deliberately.
• apply without raw=True.
• shift inside a group without groupby — the first row of each group picks up the last row of the previous group.
• Interpreting a smoothed track as data. Smoothing changes the effective resolution; do not call peaks on a smoothed signal without accounting for it.

See also

groupby · Series · Datetime Handling in pandas · Split-Apply-Combine · Applied - Quality Control Plots · Vectorization