Missing Data in pandas

Core Principle: Missing data representation in pandas includes floating-point np.nan and nullable typed sentinel pd.NA, handled through .isna(), .fillna(), and .dropna() methods across series.

Detecting

df.isna(); df.notna()          # element-wise
df.isna().sum()                 # count per column
df.isna().sum(axis=1)           # count per row
df.isna().any().any()           # is there any at all?
df["af"].isna().mean()          # proportion missing

df.isna().sum().sort_values(ascending=False).head(10)   # worst columns first

isnull/notnull are exact aliases. Use isna/notna for consistency with fillna, dropna.

The two sentinels

	np.nan	pd.NA
Where	NumPy-backed float columns	nullable and Arrow-backed dtypes
NA == NA	False	pd.NA (propagates!)
bool(NA)	False	raises TypeError
In an int column	impossible: promotes to float	fine

0       1
1       2
2    <NA>
dtype: Int64

pd.NA propagating through comparisons is the "correct" three-valued logic (like SQL NULL), and it is why if x == pd.NA raises instead of quietly being False. In practice you rarely touch it directly: use .isna().

The integer promotion problem is the one you meet daily: a genomic position column with one missing value becomes float64, and to_csv writes 7676154.0. Int64 fixes it.

Dropping

df.dropna()                                  # any NA in any column
df.dropna(how="all")                         # only fully-empty rows
df.dropna(subset=["padj", "log2fc"])         # ← usually what you want
df.dropna(axis=1, thresh=len(df) * 0.8)      # drop columns >20% missing

Bare df.dropna() is almost always too aggressive. One sparse annotation column will delete most of your rows. Always name the columns you actually require.

Filling

df.fillna(0)
df.fillna({"af": 0, "consequence": "unknown"})   # per-column
df["x"].fillna(df["x"].median())
df.ffill(); df.bfill()                            # forward/backward fill
df["x"].interpolate(method="linear")

# per-group fill: the common real case
df["x"] = df["x"].fillna(df.groupby("gene")["x"].transform("mean"))

ffill on unsorted data propagates values across group boundaries. Sort first, or group:

df = df.sort_values(["sample", "pos"])
df["depth"] = df.groupby("sample")["depth"].ffill()

NA-aware behaviour

df["x"].sum()            # skipna=True by default → skips NA
df["x"].sum(skipna=False)
df["x"].mean()           # denominator is the non-NA count
df["x"].count()          # non-NA count (vs len(df) for all rows)

df.groupby("gene")["x"].mean()          # NA dropped within each group
df.groupby("gene", dropna=False).size() # keep NA as its own group key
df["gene"].value_counts(dropna=False)   # ← always pass this

pandas skipping NA by default is the opposite of NumPy, where one NaN poisons everything. Convenient, but it means your denominator changes silently. Always report the count alongside the mean:

df.groupby("gene")["af"].agg(["count", "mean"])

value_counts(dropna=False) deserves to be a reflex. The default hides missing values from your frequency table, which is exactly where you would want to see them.

Missing does not mean the same thing everywhere

In genomics, the reason for missingness is usually informative:

Situation	Meaning	Sensible handling
padj is NA in DESeq2	Filtered by independent filtering or flagged as an outlier	Do not fill with 1: exclude from the test set
Genotype ./.	No call: low coverage	Keep as missing; imputation is a modelling decision
Zero in scRNA-seq	A real measurement (dropout)	Not missing. Do not impute as NA
Proteomics below LOD	Left-censored, not random	Model it, or impute low, not with the mean
Empty VCF INFO field	Annotation absent	Depends on the annotator

df.fillna(0) applied blindly across all of these produces different kinds of wrong. Decide per column, and write down why.

The DESeq2 case is worth spelling out: padj = NA means "this gene was excluded from multiple-testing correction". Filling with 1 makes it look tested-and-not-significant, which changes your denominator and your FDR. Drop those rows instead.

Interaction with joins

merged = df.merge(annot, on="gene", how="left")
merged["annotation"].isna().sum()       # ← how many failed to match?

A left join produces NA for unmatched keys. Always check that count. A high number usually means an ID-format mismatch (Ensembl IDs with version suffixes ENSG00000141510.16 vs without) rather than genuinely absent annotation. See Merging and Joining.

Common mistakes

• df.dropna() with no subset=, deleting most of your data.
• fillna(0) on p-values or fold changes. Zero is a meaningful value there.
• Filling DESeq2 padj NAs with 1.
• Treating scRNA-seq zeros as missing.
• value_counts() without dropna=False.
• Not noticing an integer column became float because of one NA.
• ffill across group boundaries.
• Reporting a group mean without the count, so a mean over 2 of 50 samples looks like a mean over 50.
• Not checking post-join NA counts.

See also

Missing Data in NumPy · pandas dtypes · groupby · Merging and Joining · Reading and Writing Data

Exercise

Write missing_report(text). The argument is the whole of stdin: one record per line, a gene name and a read depth separated by a tab. The depth field may be a number, or one of the file's missing-value markers ., NA or the empty string. Return a list of strings.

Report one line per gene, in alphabetical order, formatted as gene<TAB>present/total<TAB>mean, where present is the number of non-missing depths for that gene, total is the number of records for it, and mean is the mean of the non-missing depths rounded to three decimal places. A gene whose depths are all missing must print 0/total and the literal NA in place of the mean: never a filled-in zero. Append a final line missing=<n> giving the total number of missing depths in the input. An input with no records produces the single line missing=0.