Structured Arrays

Core Principle: Structured arrays represent heterogeneous data within fixed-width C-style memory records with named fields, primarily utilized for low-level binary format parsing and memory-mapped binary protocols.

Defining one

array([('chr13', 32340300, 'A', 'T', 45.5),
       ('chr17',  7676154, 'G', 'C', 99. )],
      dtype=[('chrom', '<U5'), ('pos', '<i8'), ('ref', '<U1'), ('alt', '<U1'), ('qual', '<f4')])

Memory layout is one packed record after another (an "array of structs"), unlike pandas which stores each column separately (a "struct of arrays").

Record arrays

array([ 7676154, 32340300])

Slightly nicer syntax, measurably slower attribute lookup. Rarely worth it.

Why you probably want pandas instead

For essentially every tabular task, DataFrame is better:

	Structured array	DataFrame
Field access	a["pos"]	df["pos"]
Add a column	rebuild the whole dtype	df["new"] = ...
Missing values	no NaN for int/str fields	full NA support
Group by	manual	df.groupby(...)
Join	manual	df.merge(...)
Strings	fixed-width, truncates silently	proper str dtype
IO	genfromtxt (fragile)	read_csv (robust)
Ecosystem	almost none	everything

The fixed-width string problem alone disqualifies structured arrays for most genomics: a U5 chromosome field silently truncates chrUn_KI270742v1.

When they are genuinely right

1. Binary file formats with a fixed record layout.

dt = np.dtype([("start", "<u4"), ("end", "<u4"), ("value", "<f4")])
data = np.fromfile("index.bin", dtype=dt)        # one call, no parsing

This is the killer use case. If a format specification says "each record is a 4-byte little-endian unsigned int followed by...", a structured dtype reads it directly at memory speed. Several bioinformatics index formats (bigWig internals, custom binary indices) look exactly like this.

2. np.memmap over a large binary record file.

records = np.memmap("huge.bin", dtype=dt, mode="r")
records["value"].mean()      # only touches the pages it needs

3. Interop with C libraries expecting an array of structs.

4. Multi-field sorting where you want NumPy semantics: np.sort(a, order=[...]) is clean.

Converting to and from pandas

The usual pattern: read the binary file into a structured array, immediately convert to a DataFrame, work there.

The unstructured helpers

import numpy.lib.recfunctions as rfn

rfn.structured_to_unstructured(variants[["pos", "qual"]])   # → plain 2-D array
rfn.append_fields(variants, "af", af_values)
rfn.merge_arrays([a, b], flatten=True)

structured_to_unstructured is the useful one: pull the numeric fields out into a normal matrix for computation.

Common mistakes

• Reaching for these instead of pandas for ordinary tabular data. They are harder in every dimension.
• Fixed-width string truncation. U5 silently cuts chrUn_KI270742v1 to chrUn. Size generously or use object (which forfeits the memory advantage).
• Assuming variants["pos"] is a copy. It is a strided view into the packed records: non-contiguous, so downstream compiled calls may copy.
• Forgetting endianness when reading binary files. <u4 (little) vs >u4 (big) matters, and the wrong one gives plausible-looking nonsense numbers.
• No missing-data story. There is no NaN for an integer or string field; you must pick and document a sentinel.

See also

NumPy dtypes · NumPy File IO · DataFrame · Memory Layout and Strides · Applied - Genomic File Formats

Exercise

Write rank_variants(rows, min_dp), which loads variant records into a structured array with the dtype [("chrom", "U5"), ("pos", "i8"), ("qual", "f8"), ("dp", "i4")], drops every record whose depth is below min_dp, and ranks what is left by quality descending, then chromosome ascending, then position ascending.

The first line is always kept=<n>/<total> itemsize=<bytes>, the survivors against the original record count and the byte width of one record. Then emit one line per surviving record, <chrom>:<pos> qual=<value> dp=<n>, with the quality to one decimal. Close with mean_qual=<value> over the survivors, rounded to three decimals, or mean_qual=NA when nothing survived: a mean over zero records is undefined, not zero. Note that a field sort is always ascending, so the descending quality key has to be negated or the sort has to be built from np.lexsort, whose keys are applied last one first.