NumPy File IO

Core Principle: NumPy provides binary serialization (.npy for single arrays, .npz for archived collections), memory-mapped array streaming (np.memmap), and text parsing routines for disk I/O operations.

Binary: the fast path

np.save("expr.npy", expr)                 # single array
expr = np.load("expr.npy")

np.savez("data.npz", expr=expr, genes=genes, samples=samples)
np.savez_compressed("data.npz", ...)      # zlib; smaller, slower

d = np.load("data.npz")                   # lazy loading: arrays are read upon explicit key access
d["expr"]; d.files

.npy preserves dtype, shape and byte order exactly, loads at memory-copy speed, and round-trips perfectly. Use it for intermediate results in a pipeline. Re-parsing a 500 MB TSV every time you tweak a downstream step is an unnecessary cost.

np.savez_compressed on a sparse-ish count matrix often gets 5–10×; the CPU cost is usually worth it for archival.

The security note

np.load("untrusted.npy", allow_pickle=True)     # ← arbitrary code execution

allow_pickle defaults to False for exactly this reason. Object-dtype arrays require pickling to save. Never enable it for a file you did not create.

Text: portable and slow

np.savetxt("m.tsv", m, delimiter="\t", fmt="%.4f", header="s1\ts2\ts3")
m = np.loadtxt("m.tsv", delimiter="\t", skiprows=1)
m = np.loadtxt("m.tsv", delimiter="\t", usecols=range(1, 10))
m = np.genfromtxt("m.tsv", delimiter="\t", names=True, missing_values="NA",
                  filling_values=np.nan)

np.loadtxt requires uniform numeric data and strict formatting, whereas np.genfromtxt accommodates missing values at the cost of higher parsing overhead.

For real files, use pandas:

pd.read_csv is roughly 10× faster than genfromtxt, handles headers, missing values, comment lines, compression and mixed dtypes, and gives you the labels alongside the matrix instead of forcing you to track them separately. There is essentially no case where genfromtxt is the right answer for a real-world genomics file. See Reading and Writing Data.

Memory mapping

arr = np.memmap("huge.dat", dtype=np.float32, mode="r", shape=(1_000_000, 100))
chunk = arr[0:1000]              # only these pages are read from disk
arr.mean(axis=0)                  # streams through, bounded memory

np.load("expr.npy", mmap_mode="r")     # memory-maps existing binary array for out-of-core access

Modes: "r" read-only, "r+" read-write, "w+" create/overwrite, "c" copy-on-write (changes stay in memory).

This lets you work with a 40 GB array on a 16 GB machine, provided your access pattern is reasonably sequential. Random access across a memmap larger than RAM will thrash. Access order matters enormously here: see Memory Layout and Strides.

Raw binary

arr.tofile("raw.bin")                          # NO dtype or shape metadata
arr = np.fromfile("raw.bin", dtype=np.float32).reshape(1000, 100)

Only for interop with C programs or a documented binary format. The file records nothing about its own structure, so getting the dtype or shape wrong gives you plausible garbage rather than an error. Prefer .npy, which stores a header.

Fixed-layout binary records can be mapped using structured dtypes (see Structured Arrays):

import numpy as np

dt = np.dtype([("start", "<u4"), ("end", "<u4"), ("score", "<f4")])
records = np.fromfile("intervals.bin", dtype=dt)

Note the < prefixes: little-endian. Endianness is not optional in binary formats.

Choosing a format

Need	Use
Fast reload of one array	.npy
Several related arrays	.npz
Archival, compressible	.npz compressed, or HDF5
Bigger than RAM	np.memmap, or zarr/HDF5 for chunked
Cross-language (R, MATLAB, Julia)	HDF5 (h5py) or Parquet
Sharing with a collaborator who will open it in Excel	TSV via pandas
Heterogeneous labeled table	Apache Parquet via pandas (see Reading and Writing Data)
Sparse matrix	scipy.sparse.save_npz
Single-cell multi-modal matrix	.h5ad (AnnData format standard)

HDF5 and zarr are worth knowing once your arrays outgrow .npy: both support chunked, compressed, partially-loadable n-dimensional arrays with metadata. h5py for HDF5, zarr for cloud-friendly chunked storage.

A pipeline caching pattern

Turns a 30-second parse into a 0.3-second load. Add the source file's mtime to the cache check if the input can change.

Common mistakes

• allow_pickle=True on untrusted files.
• Using tofile/fromfile for persistent storage: raw binary files omit array metadata, risking data misinterpretation if loaded with incorrect dtypes.
• genfromtxt on a large file. Use pandas.
• Losing the labels. Saving expr.npy without genes.npy produces an anonymous matrix. Use .npz and save all three.
• Wrong endianness reading a binary format.
• Memmapping with a random access pattern on a file much larger than RAM.
• Not saving in float32 when the data does not warrant float64: doubles every file and every load.

See also

Array Creation · Structured Arrays · Memory Layout and Strides · Reading and Writing Data · Applied - Reproducible Environment

Exercise

Write round_trip(values), which writes an int64 array to disk three ways and reads each one back. Create the directory with tempfile.mkdtemp() and build every path underneath it, so the exercise never depends on a file the sandbox happens to provide.

Return four lines: npy dtype=<name> equal=<bool> for a .npy round trip, which preserves the dtype exactly; txt dtype=<name> n=<int> equal=<bool> for a np.savetxt and np.loadtxt round trip, which loses the integer dtype even though the values survive, noting that loadtxt returns a zero-dimensional array for single-value inputs for a single-row file, so wrap the result in np.atleast_1d before asking for its length; npz keys=<names joined by commas, sorted> doubled=<bool> for an np.savez archive holding the array under raw and twice the array under doubled; and shares=<bool>, whether the array loaded from the .npy file aliases the array you saved.