Structured Arrays

In one line: an ndarray whose elements are C-struct-like records with named, typed fields — NumPy's answer to a table, and usually the wrong answer.

Defining one

dt = np.dtype([
    ("chrom", "U5"),
    ("pos", np.int64),
    ("ref", "U1"),
    ("alt", "U1"),
    ("qual", np.float32),
])

variants = np.array([
    ("chr17", 7676154, "G", "C", 99.0),
    ("chr13", 32340300, "A", "T", 45.5),
], dtype=dt)

variants["pos"]                  # field access → array([7676154, 32340300])
variants[0]                      # one record (a np.void)
variants[0]["chrom"]             # 'chr17'
variants[variants["qual"] > 50]  # boolean masking works normally
variants.dtype.names             # ('chrom', 'pos', 'ref', 'alt', 'qual')
np.sort(variants, order=["chrom", "pos"])

Memory layout is one packed record after another (an "array of structs"), unlike pandas which stores each column separately (a "struct of arrays").

Record arrays

rec = variants.view(np.recarray)
rec.pos          # attribute access instead of rec["pos"]

Slightly nicer syntax, measurably slower attribute lookup. Rarely worth it.

Why you probably want pandas instead

For essentially every tabular task, DataFrame is better:

	Structured array	DataFrame
Field access	a["pos"]	df["pos"]
Add a column	rebuild the whole dtype	df["new"] = ...
Missing values	no NaN for int/str fields	full NA support
Group by	manual	df.groupby(...)
Join	manual	df.merge(...)
Strings	fixed-width, truncates silently	proper str dtype
IO	genfromtxt (fragile)	read_csv (robust)
Ecosystem	almost none	everything

The fixed-width string problem alone disqualifies structured arrays for most genomics: a U5 chromosome field silently truncates chrUn_KI270742v1.

When they are genuinely right

1. Binary file formats with a fixed record layout.

dt = np.dtype([("start", "<u4"), ("end", "<u4"), ("value", "<f4")])
data = np.fromfile("index.bin", dtype=dt)        # one call, no parsing

This is the killer use case. If a format specification says "each record is a 4-byte little-endian unsigned int followed by...", a structured dtype reads it directly at memory speed. Several bioinformatics index formats (bigWig internals, custom binary indices) look exactly like this.

2. np.memmap over a large binary record file.

records = np.memmap("huge.bin", dtype=dt, mode="r")
records["value"].mean()      # only touches the pages it needs

3. Interop with C libraries expecting an array of structs.

4. Multi-field sorting where you want NumPy semantics — np.sort(a, order=[...]) is clean.

Converting to and from pandas

df = pd.DataFrame(variants)                    # structured array → DataFrame
back = df.to_records(index=False)              # DataFrame → recarray

The usual pattern: read the binary file into a structured array, immediately convert to a DataFrame, work there.

The unstructured helpers

import numpy.lib.recfunctions as rfn

rfn.structured_to_unstructured(variants[["pos", "qual"]])   # → plain 2-D array
rfn.append_fields(variants, "af", af_values)
rfn.merge_arrays([a, b], flatten=True)

structured_to_unstructured is the useful one: pull the numeric fields out into a normal matrix for computation.

Common mistakes

• Reaching for these instead of pandas for ordinary tabular data. They are harder in every dimension.
• Fixed-width string truncation. U5 silently cuts chrUn_KI270742v1 to chrUn. Size generously or use object (which forfeits the memory advantage).
• Assuming variants["pos"] is a copy. It is a strided view into the packed records — non-contiguous, so downstream compiled calls may copy.
• Forgetting endianness when reading binary files. <u4 (little) vs >u4 (big) matters, and the wrong one gives plausible-looking nonsense numbers.
• No missing-data story. There is no NaN for an integer or string field; you must pick and document a sentinel.

See also

NumPy dtypes · NumPy File IO · DataFrame · Memory Layout and Strides · Applied - Genomic File Formats