Missing Data in NumPy

Core Principle: Floating-point missing data in NumPy is represented via IEEE 754 NaN values, which propagate through standard calculations and require dedicated nan-prefixed reduction functions.

NaN behaves unlike anything else

array([False,  True, False])

NaN is defined by IEEE 754 to be unequal to everything including itself. This is deliberate (it lets you detect it with x != x) and it is why == never works.

Contagion

array([ 1.,  3., nan])

One NaN poisons the entire reduction. A single missing cell in a 20,000 × 500 matrix makes expr.mean() return nan. This is a feature: it prevents you silently ignoring missing data: but it means you must decide explicitly what to do.

The nan-aware functions

(<function nanargmax at 0x1063a47f0>, <function nanargmin at 0x1063a4630>, <function nancumsum at 0x1063a4b30>)

a.mean()            # nan
np.nanmean(a)       # 2.0
np.nanmean(expr, axis=1)     # per-gene mean, skipping missing

These change your denominator silently. np.nanmean of a row with 400 of 500 values missing returns the mean of 100 values with no indication. Always compute the count alongside:

n_valid = (~np.isnan(expr)).sum(axis=1)
means = np.nanmean(expr, axis=1)
means[n_valid < 3] = np.nan          # require a minimum

np.nanmean of an all-NaN slice returns NaN and emits a RuntimeWarning; np.nansum of an all-NaN slice returns 0, which is worse because it looks like data.

Detection and cleaning

np.isnan(a)                     # floats only
np.isinf(a); np.isfinite(a)     # inf is separate from nan
np.isnan(a).any()               # is there any missing data at all?
np.isnan(a).sum()               # how much?
np.isnan(expr).sum(axis=1)      # per row

a[~np.isnan(a)]                 # drop
np.nan_to_num(a, nan=0.0, posinf=1e10, neginf=-1e10)   # replace

np.isnan raises a TypeError on integer and string arrays: there is no NaN in those dtypes. Use np.isfinite if you also care about ±inf, which arises from log(0) and division by zero.

Where NaN comes from in bioinformatics

np.log2(0)                      # -inf, not nan
np.log2(-1)                     # nan + RuntimeWarning
0 / 0                            # nan
np.array([1.0]) / 0              # inf
np.sqrt(-1)                      # nan

Concretely: log-transforming zero counts, dividing by a zero library size, correlating a constant gene (zero variance → 0/0), or a statistical test on a gene with no variation.

with np.errstate(divide="ignore", invalid="ignore"):
    ratio = treated / control          # deliberate: I know some controls are 0
ratio[~np.isfinite(ratio)] = np.nan    # normalise inf to nan

np.errstate is the right way to suppress a warning you have understood. warnings.filterwarnings("ignore") at module level hides the ones you have not.

Integer arrays have no NaN

a = np.array([1, 2, 3])
a[1] = np.nan          # ValueError: cannot convert float NaN to integer

Options, in order of preference:

1. Use pandas' nullable Int64: a real integer type with a separate mask. See Missing Data in pandas.
2. Promote to float. Fine up to 2⁵³; genomic coordinates are safe, large counts are too.
3. Use a sentinel like -1 or -9 (the PLINK convention for genotypes). Fast and compact, but every downstream consumer must know about it: an undocumented sentinel that leaks into a mean is a silent data-corruption bug.

gt = np.full((n_samples, n_variants), -1, dtype=np.int8)   # -1 = missing
missing = gt == -1

Masked arrays

NumPy's built-in answer, and one that never quite caught on:

m = np.ma.masked_array(data, mask=np.isnan(data))
m.mean(axis=1)              # skips masked automatically
m.filled(0)                 # back to a plain array

Works for any dtype including integers, and reductions handle the mask correctly. But support outside NumPy is patchy: matplotlib understands masked arrays, most other libraries do not, and pandas ignores them. Use it for self-contained numeric work; do not build an API around it.

Handling strategies

Strategy	When
Propagate (do nothing)	You want to know loudly that data is missing
Drop rows/columns	Missingness is rare and plausibly random
nan-aware reductions	Missingness is scattered; record the counts
Impute (median, kNN)	Downstream method cannot accept NaN (PCA, clustering)
Model it	Missingness is informative: e.g. a proteomics value below detection limit is not missing at random

That last row matters in bio: a zero in single-cell RNA-seq is a real measurement (dropout) with a biological interpretation, not a missing value. Imputing it changes the biology, not just the statistics. Decide deliberately.

Common mistakes

• arr == np.nan. Always False. Use np.isnan.
• np.isnan on an int or string array → TypeError.
• np.nansum returning 0 for an all-NaN slice and being mistaken for a real zero.
• Not tracking how many values nanmean actually used.
• Confusing inf with nan. np.isnan(np.inf) is False. Use np.isfinite.
• Silencing RuntimeWarnings globally instead of scoping with np.errstate.
• Sorting and forgetting NaNs go last: np.sort(a)[0] is the min, but np.sort(a)[-1] may be NaN, not the max.
• Treating single-cell zeros as missing.

See also

NumPy dtypes · ufuncs · Axes and Reductions · Boolean Masking · Missing Data in pandas

Exercise

Write row_summary(matrix), which summarises a float matrix whose missing entries are NaN. Return ["rows=0"] when the matrix holds no elements.

Otherwise emit one line per row, row<i> missing=<n> mean=<value>, with the mean of that row's present values rounded to three decimals. A row whose values are all missing has no mean at all, so print mean=NA for it rather than a NaN and compute the row means without ever averaging an empty selection: an all-NaN slice is exactly what a careless call to np.nanmean turns into a runtime warning. Close with one overall missing=<n> mean=<value> line for the whole matrix, obeying the same rule. Count present entries with np.isnan and accumulate the sums with np.where, so no reduction ever sees an empty axis.