pandas dtypes

In one line: pandas has three dtype systems now — NumPy-backed, nullable, and Arrow-backed — and pandas 3.0 changed the default for strings.

What changed in pandas 3.0

pd.Series(["a", "b"]).dtype        # str        (was: object)

Strings are now a dedicated dtype, backed by PyArrow when it is installed. This is faster, uses far less memory, and gives proper missing-value semantics.

It breaks the old idiom for finding text columns:

df.dtypes == object                          # ← no longer finds string columns
df.select_dtypes(include="object")           # ← same problem

# correct in pandas 3.0
df.select_dtypes(include="str")
[c for c in df.columns if pd.api.types.is_string_dtype(df[c])]

Also changed: datetimes default to microsecond resolution rather than nanosecond, which removes the old 1678–2262 range limit.

The three systems

	Example	NA support	Notes
NumPy-backed	int64, float64, bool	float only (NaN)	the classic default
Nullable (masked)	Int64, Float64, boolean	yes, pd.NA	note the capital letter
Arrow-backed	int64[pyarrow], str	yes	fastest, best memory

pd.Series([1, 2, None])                          # float64 — the int was promoted!
pd.Series([1, 2, None], dtype="Int64")           # Int64 with <NA> — stays integer
pd.Series([1, 2, None], dtype="int64[pyarrow]")  # Arrow

The capital-I Int64 is the fix for the classic problem: a single missing value promoting your integer column to float, so genomic positions become 7676154.0 and to_csv writes them with a decimal point.

Inspecting

df.dtypes
df.info()                                # dtypes + non-null counts + memory
df.select_dtypes(include="number")
df.select_dtypes(exclude=["str", "category"])

pd.api.types.is_numeric_dtype(df["x"])
pd.api.types.is_string_dtype(df["gene"])
pd.api.types.is_integer_dtype(df["pos"])

Use the pd.api.types predicates rather than comparing .dtype to a literal — they handle all three backing systems.

Converting

df.astype({"pos": "int32", "chrom": "category"})
pd.to_numeric(df["af"], errors="coerce")        # unparseable → NaN
pd.to_numeric(df["af"], errors="raise")         # unparseable → exception
pd.to_datetime(df["date"], format="%Y-%m-%d")
df.convert_dtypes()                              # infer best nullable dtypes

errors="coerce" is convenient and dangerous: it turns "1.2e-5x" into NaN without telling you. In a pipeline, prefer errors="raise" and handle the exception, or count the NaNs it created:

before = df["af"].notna().sum()
df["af"] = pd.to_numeric(df["af"], errors="coerce")
lost = before - df["af"].notna().sum()
if lost:
    warnings.warn(f"{lost} unparseable AF values coerced to NaN")

Memory

df.memory_usage(deep=True).sum() / 1e6      # MB — deep=True counts string payloads

For a 10-million-row variant table, in rough order of impact:

df = df.astype({
    "chrom": "category",        # 22 distinct values × 10M rows → huge saving
    "consequence": "category",
    "ref": "category",
    "alt": "category",
    "pos": "int32",             # max ~249 Mb fits comfortably
    "qual": "float32",
    "af": "float32",
})

category on a low-cardinality string column is typically a 10–50× reduction. See Categorical dtype.

Do not category a high-cardinality column like variant ID or read name — you pay for the dictionary and save nothing.

Choosing an integer width

Column	dtype	Why
Genomic position	int32	max chromosome ~249 Mb < 2.1 × 10⁹
Cumulative genome offset	int64	~3.1 Gb exceeds int32
Read depth	int16 or int32	rarely > 32,767, but ultra-deep panels are
Raw counts	int32	int16 overflows on real RNA-seq
Genotype code (0/1/2)	int8	3 values
Quality score	int8 / float32	Phred ≤ 60 typically

Overflow is silent in NumPy-backed dtypes. See NumPy dtypes.

Object dtype is a warning sign

df.select_dtypes(include="object").columns

After pandas 3.0, a column that is still object means it holds mixed types or Python objects — usually a parsing failure. Investigate it; it is almost never intentional.

Bioinformatics schema pattern

Define the schema once and reuse it:

VARIANT_SCHEMA = {
    "CHROM": "category",
    "POS": "int32",
    "ID": "str",
    "REF": "category",
    "ALT": "category",
    "QUAL": "float32",
    "FILTER": "category",
    "INFO": "str",
}

df = pd.read_csv(path, sep="\t", comment="#", dtype=VARIANT_SCHEMA, na_values=["."])

This is documentation, validation and optimisation in one object. If the file does not match, you find out at load time rather than three steps later.

Common mistakes

• dtypes == object to find strings in pandas 3.0.
• An integer column silently becoming float because of one NA. Use Int64.
• errors="coerce" hiding bad data.
• Categorizing a high-cardinality column.
• int16 for RNA-seq counts. Highly expressed genes exceed 32,767 routinely.
• Forgetting astype returns a new frame.
• Comparing a category column to a value not in its categories — works, but always False, and == between two differently-categoried columns raises.
• Assuming Arrow is installed. It is optional; check if your code depends on Arrow-specific behaviour.

See also

NumPy dtypes · Categorical dtype · Missing Data in pandas · Reading and Writing Data · pandas Performance · Copy-on-Write