Missing Data in NumPy

In one line: NumPy has no real missing-value support — it has NaN, which only works for floats, is contagious, and is never equal to itself.

NaN behaves unlike anything else

np.nan == np.nan        # False  ← the defining weirdness
np.nan != np.nan        # True
np.nan > 1              # False
np.nan < 1              # False   (both comparisons are False!)

a = np.array([1.0, np.nan, 3.0])
a == np.nan             # array([False, False, False])   ← never use this
np.isnan(a)             # array([False,  True, False])   ← use this

NaN is defined by IEEE 754 to be unequal to everything including itself. This is deliberate (it lets you detect it with x != x) and it is why == never works.

Contagion

a.sum()          # nan
a.mean()         # nan
a.max()          # nan
np.sort(a)       # NaNs sort to the END

One NaN poisons the entire reduction. A single missing cell in a 20,000 × 500 matrix makes expr.mean() return nan. This is a feature — it prevents you silently ignoring missing data — but it means you must decide explicitly what to do.

The nan-aware functions

np.nansum, np.nanmean, np.nanstd, np.nanvar, np.nanmedian,
np.nanmin, np.nanmax, np.nanpercentile, np.nanquantile,
np.nanargmax, np.nanargmin, np.nancumsum

a.mean()            # nan
np.nanmean(a)       # 2.0
np.nanmean(expr, axis=1)     # per-gene mean, skipping missing

These change your denominator silently. np.nanmean of a row with 400 of 500 values missing returns the mean of 100 values with no indication. Always compute the count alongside:

n_valid = (~np.isnan(expr)).sum(axis=1)
means = np.nanmean(expr, axis=1)
means[n_valid < 3] = np.nan          # require a minimum

np.nanmean of an all-NaN slice returns NaN and emits a RuntimeWarning; np.nansum of an all-NaN slice returns 0, which is worse because it looks like data.

Detection and cleaning

np.isnan(a)                     # floats only
np.isinf(a); np.isfinite(a)     # inf is separate from nan
np.isnan(a).any()               # is there any missing data at all?
np.isnan(a).sum()               # how much?
np.isnan(expr).sum(axis=1)      # per row

a[~np.isnan(a)]                 # drop
np.nan_to_num(a, nan=0.0, posinf=1e10, neginf=-1e10)   # replace

np.isnan raises a TypeError on integer and string arrays — there is no NaN in those dtypes. Use np.isfinite if you also care about ±inf, which arises from log(0) and division by zero.

Where NaN comes from in bioinformatics

np.log2(0)                      # -inf, not nan
np.log2(-1)                     # nan + RuntimeWarning
0 / 0                            # nan
np.array([1.0]) / 0              # inf
np.sqrt(-1)                      # nan

Concretely: log-transforming zero counts, dividing by a zero library size, correlating a constant gene (zero variance → 0/0), or a statistical test on a gene with no variation.

with np.errstate(divide="ignore", invalid="ignore"):
    ratio = treated / control          # deliberate: I know some controls are 0
ratio[~np.isfinite(ratio)] = np.nan    # normalise inf to nan

np.errstate is the right way to suppress a warning you have understood. warnings.filterwarnings("ignore") at module level hides the ones you have not.

Integer arrays have no NaN

a = np.array([1, 2, 3])
a[1] = np.nan          # ValueError: cannot convert float NaN to integer

Options, in order of preference:

1. Use pandas' nullable Int64 — a real integer type with a separate mask. See Missing Data in pandas.
2. Promote to float. Fine up to 2⁵³; genomic coordinates are safe, large counts are too.
3. Use a sentinel like -1 or -9 (the PLINK convention for genotypes). Fast and compact, but every downstream consumer must know about it — an undocumented sentinel that leaks into a mean is a silent data-corruption bug.

gt = np.full((n_samples, n_variants), -1, dtype=np.int8)   # -1 = missing
missing = gt == -1

Masked arrays

NumPy's built-in answer, and one that never quite caught on:

m = np.ma.masked_array(data, mask=np.isnan(data))
m.mean(axis=1)              # skips masked automatically
m.filled(0)                 # back to a plain array

Works for any dtype including integers, and reductions handle the mask correctly. But support outside NumPy is patchy — matplotlib understands masked arrays, most other libraries do not, and pandas ignores them. Use it for self-contained numeric work; do not build an API around it.

Handling strategies

Strategy	When
Propagate (do nothing)	You want to know loudly that data is missing
Drop rows/columns	Missingness is rare and plausibly random
nan-aware reductions	Missingness is scattered; record the counts
Impute (median, kNN)	Downstream method cannot accept NaN (PCA, clustering)
Model it	Missingness is informative — e.g. a proteomics value below detection limit is not missing at random

That last row matters in bio: a zero in single-cell RNA-seq is a real measurement (dropout) with a biological interpretation, not a missing value. Imputing it changes the biology, not just the statistics. Decide deliberately.

Common mistakes

• arr == np.nan. Always False. Use np.isnan.
• np.isnan on an int or string array → TypeError.
• np.nansum returning 0 for an all-NaN slice and being mistaken for a real zero.
• Not tracking how many values nanmean actually used.
• Confusing inf with nan. np.isnan(np.inf) is False. Use np.isfinite.
• Silencing RuntimeWarnings globally instead of scoping with np.errstate.
• Sorting and forgetting NaNs go last — np.sort(a)[0] is the min, but np.sort(a)[-1] may be NaN, not the max.
• Treating single-cell zeros as missing.

See also

NumPy dtypes · ufuncs · Axes and Reductions · Boolean Masking · Missing Data in pandas