Window Functions

Core Principle: Window operations in pandas support rolling window statistics (.rolling()), cumulative expansions (.expanding()), and lag transformations (.shift()) across contiguous sequence observations.

Rolling

s.rolling(window=100).mean()
s.rolling(window=100, center=True).mean()      # centred rather than trailing
s.rolling(window=100, min_periods=1).mean()    # emit values at the edges
s.rolling(window=100).agg(["mean", "std", "min", "max"])
s.rolling(window=100).quantile(0.5)
s.rolling(window=100).apply(custom_fn, raw=True)   # raw=True passes ndarray: faster

The edge behaviour is the thing to decide up front. By default the first window-1 positions are NaN, because there is not a full window yet. min_periods=1 fills them using whatever data exists, which is usually what you want for a coverage track and usually not what you want for a statistic you will interpret.

center=True matters for genomics: an uncentred rolling mean shifts every feature to the right by half a window. On a coverage plot that is a visible, wrong offset.

Window types

s.rolling(100).mean()                                  # flat/rectangular
s.rolling(100, win_type="gaussian").mean(std=20)       # weighted
s.rolling("10min").mean()                               # time-based (needs DatetimeIndex)
s.ewm(span=20).mean()                                   # exponentially weighted
s.ewm(halflife=10).mean()

ewm (exponentially weighted) gives smoothing with no edge NaNs and a smooth response: often nicer than a rectangular rolling mean for visualisation.

Expanding

Cumulative, from the start to each point:

s.expanding().mean()          # running mean
s.expanding().max()           # running max
s.cumsum(); s.cummax(); s.cumprod()

Shift and diff

s.shift(1)              # previous value
s.shift(-1)             # next value
s.diff()                # s - s.shift(1)
s.diff(2)
s.pct_change()

df["gap"] = df["pos"] - df["pos"].shift(1)         # distance to the previous variant
df["is_next_to"] = df["gap"] < 10

shift and diff are how you compare each row to its neighbours. In genomics that means inter-variant distances, exon gaps, and detecting runs of adjacent features.

Grouped windows: the crucial bit

Rolling and shifting across group boundaries is a silent bug. A rolling mean over a position-sorted variant table will happily average the end of chr1 with the start of chr2.

# WRONG: smooths across chromosome boundaries
df["smooth"] = df["depth"].rolling(100, center=True).mean()

# RIGHT
df = df.sort_values(["chrom", "pos"])
df["smooth"] = (
    df.groupby("chrom")["depth"]
      .transform(lambda s: s.rolling(100, center=True, min_periods=1).mean())
)

# same for shift/diff
df["gap"] = df.groupby("chrom")["pos"].diff()      # NaN at each chromosome start ✓

The groupby().transform() form is the pattern. See groupby.

Rolling on a MultiIndex or with a time index

df.groupby("sample").rolling(window=10).mean()      # produces a MultiIndex
df.set_index("time").groupby("sample").rolling("1h").mean()

Time-based windows ("1h", "7D") require a sorted DatetimeIndex and are genuinely different from count-based windows when the sampling is irregular. See Datetime Handling in pandas.

Bioinformatics examples

# smooth a per-base coverage track for plotting
cov = cov.sort_values(["chrom", "pos"])
cov["smoothed"] = (
    cov.groupby("chrom")["depth"]
       .transform(lambda s: s.rolling(1000, center=True, min_periods=1).mean())
)

# GC content in sliding windows
gc = is_gc.rolling(100, center=True).mean()

# detect coverage dropouts: sustained low regions
low = cov["smoothed"] < 0.2 * cov["smoothed"].median()
runs = (low != low.shift()).cumsum()               # ← run-length encoding idiom
dropouts = cov[low].groupby(runs).agg(
    start=("pos", "min"), end=("pos", "max"), n=("pos", "size")
).query("n >= 50")

# distance to the nearest previous variant, per chromosome
variants["dist_prev"] = variants.groupby("chrom")["pos"].diff()

# rolling allele-frequency deviation, for LOH detection
baf["rolling_dev"] = (
    baf.groupby("chrom")["baf"]
       .transform(lambda s: (s - 0.5).abs().rolling(50, center=True).mean())
)

# running total along a chromosome
cov["cumulative"] = cov.groupby("chrom")["depth"].cumsum()

The run-length encoding idiom in the dropout example is worth memorising:

runs = (flag != flag.shift()).cumsum()

Each time the flag changes, the cumulative sum increments, so consecutive identical values share a run id. Grouping by it gives you contiguous blocks: the basis of segment detection, CNV calling, and any "find stretches where X is true" problem.

Performance

s.rolling(100).mean()                      # compiled: fast
s.rolling(100).apply(np.mean)              # Python per window: very slow
s.rolling(100).apply(f, raw=True)          # raw ndarray instead of Series: faster
s.rolling(100, engine="numba").mean()      # for large data with a custom function

Prefer the built-in aggregations. If you need apply, always pass raw=True: it skips constructing a Series per window.

For very large arrays with a simple window, NumPy's sliding_window_view is faster still and allocates nothing. See Memory Layout and Strides.

Common mistakes

• Rolling across chromosome/sample boundaries. The defining error of this note.
• Not sorting before rolling. Window functions assume order and will not warn.
• Forgetting center=True, shifting every feature by half a window.
• Leading NaNs breaking a downstream step. Decide min_periods deliberately.
• apply without raw=True.
• shift inside a group without groupby: the first row of each group picks up the last row of the previous group.
• Interpreting a smoothed track as data. Smoothing changes the effective resolution; do not call peaks on a smoothed signal without accounting for it.

See also

groupby · Series · Datetime Handling in pandas · Split-Apply-Combine · Applied - Quality Control Plots · Vectorization

Exercise

Write dropout_runs(text). The first line of stdin is a depth threshold. Every following line is a coverage record: chromosome, position and depth, separated by spaces. Return a list of strings.

Sort by chromosome then position, then smooth the depth with a centred rolling mean of window 3 and min_periods=1 so the edges still produce a value. The smoothing must happen within each chromosome: a rolling window that runs off the end of one chromosome and onto the next silently mixes them. Flag every position whose smoothed depth is below the threshold, and find the runs of consecutive flagged positions; a run must not span a chromosome boundary either.

Return one line per run of length two or more, in genomic order, formatted chrom<TAB>start<TAB>end<TAB>length, where start and end are the first and last positions of the run. When no run qualifies, return the single line none.