NumPy File IO

In one line: .npy for one array, .npz for several, text formats for interoperability, memmap for data bigger than RAM — and pandas for anything with headers or mixed types.

Binary: the fast path

np.save("expr.npy", expr)                 # single array
expr = np.load("expr.npy")

np.savez("data.npz", expr=expr, genes=genes, samples=samples)
np.savez_compressed("data.npz", ...)      # zlib; smaller, slower

d = np.load("data.npz")                   # lazy — arrays load on access
d["expr"]; d.files

.npy preserves dtype, shape and byte order exactly, loads at memory-copy speed, and round-trips perfectly. Use it for intermediate results in a pipeline. Re-parsing a 500 MB TSV every time you tweak a downstream step is an unnecessary cost.

np.savez_compressed on a sparse-ish count matrix often gets 5–10×; the CPU cost is usually worth it for archival.

The security note

np.load("untrusted.npy", allow_pickle=True)     # ← arbitrary code execution

allow_pickle defaults to False for exactly this reason. Object-dtype arrays require pickling to save. Never enable it for a file you did not create.

Text: portable and slow

np.savetxt("m.tsv", m, delimiter="\t", fmt="%.4f", header="s1\ts2\ts3")
m = np.loadtxt("m.tsv", delimiter="\t", skiprows=1)
m = np.loadtxt("m.tsv", delimiter="\t", usecols=range(1, 10))
m = np.genfromtxt("m.tsv", delimiter="\t", names=True, missing_values="NA",
                  filling_values=np.nan)

loadtxt is strict — it fails on missing values and non-numeric columns. genfromtxt is tolerant and correspondingly slow and fiddly.

For real files, use pandas:

df = pd.read_csv("counts.tsv", sep="\t", index_col=0)
expr = df.to_numpy()
genes = df.index.to_numpy()

pd.read_csv is roughly 10× faster than genfromtxt, handles headers, missing values, comment lines, compression and mixed dtypes, and gives you the labels alongside the matrix instead of forcing you to track them separately. There is essentially no case where genfromtxt is the right answer for a real-world genomics file. See Reading and Writing Data.

Memory mapping

arr = np.memmap("huge.dat", dtype=np.float32, mode="r", shape=(1_000_000, 100))
chunk = arr[0:1000]              # only these pages are read from disk
arr.mean(axis=0)                  # streams through, bounded memory

np.load("expr.npy", mmap_mode="r")     # memmap an existing .npy — easiest route

Modes: "r" read-only, "r+" read-write, "w+" create/overwrite, "c" copy-on-write (changes stay in memory).

This lets you work with a 40 GB array on a 16 GB machine, provided your access pattern is reasonably sequential. Random access across a memmap larger than RAM will thrash. Access order matters enormously here — see Memory Layout and Strides.

Raw binary

arr.tofile("raw.bin")                          # NO dtype or shape metadata
arr = np.fromfile("raw.bin", dtype=np.float32).reshape(1000, 100)

Only for interop with C programs or a documented binary format. The file records nothing about its own structure, so getting the dtype or shape wrong gives you plausible garbage rather than an error. Prefer .npy, which stores a header.

For fixed-layout binary records, combine with a structured dtype — see Structured Arrays:

dt = np.dtype([("start", "<u4"), ("end", "<u4"), ("score", "<f4")])
records = np.fromfile("intervals.bin", dtype=dt)

Note the < prefixes: little-endian. Endianness is not optional in binary formats.

Choosing a format

Need	Use
Fast reload of one array	.npy
Several related arrays	.npz
Archival, compressible	.npz compressed, or HDF5
Bigger than RAM	np.memmap, or zarr/HDF5 for chunked
Cross-language (R, MATLAB, Julia)	HDF5 (h5py) or Parquet
Sharing with a collaborator who will open it in Excel	TSV via pandas
A labelled table	Parquet via pandas — see Reading and Writing Data
Sparse matrix	scipy.sparse.save_npz
Single-cell data	.h5ad (AnnData) — the field standard

HDF5 and zarr are worth knowing once your arrays outgrow .npy: both support chunked, compressed, partially-loadable n-dimensional arrays with metadata. h5py for HDF5, zarr for cloud-friendly chunked storage.

A pipeline caching pattern

from pathlib import Path

def load_expression(tsv_path):
    cache = Path(tsv_path).with_suffix(".npz")
    if cache.exists():
        d = np.load(cache, allow_pickle=False)
        return d["expr"], d["genes"], d["samples"]

    df = pd.read_csv(tsv_path, sep="\t", index_col=0)
    expr = df.to_numpy(dtype=np.float32)
    genes = df.index.to_numpy().astype(str)
    samples = df.columns.to_numpy().astype(str)
    np.savez_compressed(cache, expr=expr, genes=genes, samples=samples)
    return expr, genes, samples

Turns a 30-second parse into a 0.3-second load. Add the source file's mtime to the cache check if the input can change.

Common mistakes

• allow_pickle=True on untrusted files.
• tofile/fromfile for storage — no metadata, silent corruption on a dtype mismatch.
• genfromtxt on a large file. Use pandas.
• Losing the labels. Saving expr.npy without genes.npy produces an anonymous matrix. Use .npz and save all three.
• Wrong endianness reading a binary format.
• Memmapping with a random access pattern on a file much larger than RAM.
• Not saving in float32 when the data does not warrant float64 — doubles every file and every load.

See also

Array Creation · Structured Arrays · Memory Layout and Strides · Reading and Writing Data · Applied - Reproducible Environment