pandas dtypes

Core Principle: The pandas type system spans NumPy-backed types, nullable extension arrays, and PyArrow-backed storage engines, governing memory layout, missing-value propagation, and execution speed.

What changed in pandas 3.0

<StringDtype(na_value=nan)>

Strings are now a dedicated dtype, backed by PyArrow when it is installed. This is faster, uses far less memory, and gives proper missing-value semantics.

It breaks the old idiom for finding text columns:

df.dtypes == object                          # ← no longer finds string columns
df.select_dtypes(include="object")           # ← same problem

# correct in pandas 3.0
df.select_dtypes(include="str")
[c for c in df.columns if pd.api.types.is_string_dtype(df[c])]

Also changed: datetimes default to microsecond resolution rather than nanosecond, which removes the old 1678–2262 range limit.

The three systems

	Example	NA support	Notes
NumPy-backed	int64, float64, bool	float only (NaN)	the classic default
Nullable (masked)	Int64, Float64, boolean	yes, pd.NA	note the capital letter
Arrow-backed	int64[pyarrow], str	yes	fastest, best memory

pd.Series([1, 2, None])                          # float64: the int was promoted!
pd.Series([1, 2, None], dtype="Int64")           # Int64 with <NA>: stays integer
pd.Series([1, 2, None], dtype="int64[pyarrow]")  # Arrow

The capital-I Int64 is the fix for the classic problem: a single missing value promoting your integer column to float, so genomic positions become 7676154.0 and to_csv writes them with a decimal point.

Inspecting

df.dtypes
df.info()                                # dtypes + non-null counts + memory
df.select_dtypes(include="number")
df.select_dtypes(exclude=["str", "category"])

pd.api.types.is_numeric_dtype(df["x"])
pd.api.types.is_string_dtype(df["gene"])
pd.api.types.is_integer_dtype(df["pos"])

Use the pd.api.types predicates rather than comparing .dtype to a literal: they handle all three backing systems.

Converting

df.astype({"pos": "int32", "chrom": "category"})
pd.to_numeric(df["af"], errors="coerce")        # unparseable → NaN
pd.to_numeric(df["af"], errors="raise")         # unparseable → exception
pd.to_datetime(df["date"], format="%Y-%m-%d")
df.convert_dtypes()                              # infer best nullable dtypes

errors="coerce" is convenient and dangerous: it turns "1.2e-5x" into NaN without telling you. In a pipeline, prefer errors="raise" and handle the exception, or count the NaNs it created:

before = df["af"].notna().sum()
df["af"] = pd.to_numeric(df["af"], errors="coerce")
lost = before - df["af"].notna().sum()
if lost:
    warnings.warn(f"{lost} unparseable AF values coerced to NaN")

Memory

df.memory_usage(deep=True).sum() / 1e6      # MB: deep=True counts string payloads

For a 10-million-row variant table, in rough order of impact:

df = df.astype({
    "chrom": "category",        # 22 distinct values × 10M rows → huge saving
    "consequence": "category",
    "ref": "category",
    "alt": "category",
    "pos": "int32",             # max ~249 Mb fits comfortably
    "qual": "float32",
    "af": "float32",
})

category on a low-cardinality string column is typically a 10–50× reduction. See Categorical dtype.

Do not category a high-cardinality column like variant ID or read name: you pay for the dictionary and save nothing.

Choosing an integer width

Column	dtype	Why
Genomic position	int32	max chromosome ~249 Mb < 2.1 × 10⁹
Cumulative genome offset	int64	~3.1 Gb exceeds int32
Read depth	int16 or int32	rarely > 32,767, but ultra-deep panels are
Raw counts	int32	int16 overflows on real RNA-seq
Genotype code (0/1/2)	int8	3 values
Quality score	int8 / float32	Phred ≤ 60 typically

Overflow is silent in NumPy-backed dtypes. See NumPy dtypes.

Object dtype is a warning sign

df.select_dtypes(include="object").columns

After pandas 3.0, a column that is still object means it holds mixed types or Python objects: usually a parsing failure. Investigate it; it is almost never intentional.

Bioinformatics schema pattern

Define the schema once and reuse it:

VARIANT_SCHEMA = {
    "CHROM": "category",
    "POS": "int32",
    "ID": "str",
    "REF": "category",
    "ALT": "category",
    "QUAL": "float32",
    "FILTER": "category",
    "INFO": "str",
}

df = pd.read_csv(path, sep="\t", comment="#", dtype=VARIANT_SCHEMA, na_values=["."])

This is documentation, validation and optimisation in one object. If the file does not match, you find out at load time rather than three steps later.

Common mistakes

• dtypes == object to find strings in pandas 3.0.
• An integer column silently becoming float because of one NA. Use Int64.
• errors="coerce" hiding bad data.
• Categorizing a high-cardinality column.
• int16 for RNA-seq counts. Highly expressed genes exceed 32,767 routinely.
• Forgetting astype returns a new frame.
• Comparing a category column to a value not in its categories: works, but always False, and == between two differently-categoried columns raises.
• Assuming Arrow is installed. It is optional; check if your code depends on Arrow-specific behaviour.

See also

NumPy dtypes · Categorical dtype · Missing Data in pandas · Reading and Writing Data · pandas Performance · Copy-on-Write

Exercise

Write clean_positions(text). Stdin holds one gene<TAB>position record per line. The position field is text as it came out of the file: it may be a whole number, the missing marker . or an empty field, or something unparseable such as 7,676,154. Return a list of strings.

Coerce the positions to numbers, then store them in a nullable integer column so that a missing value does not promote the column to float64. Print one gene<TAB>position line per record in input order, with the position as a bare integer with no decimal point, or the literal NA when it is missing. Then print coerced=<n>, the number of records that carried a value but whose value could not be parsed, and total=<sum>, the sum of the parsed positions with missing values skipped. When every position is missing, total=0.