Tidy Data

In one line: one variable per column, one observation per row, one table per kind of observational unit.

The idea comes from Hadley Wickham's 2014 paper. It matters in Python because Seaborn, groupby and most statistical tooling assume it, and because bioinformatics file formats almost never provide it.

The three rules

1. Each variable forms a column.
2. Each observation forms a row.
3. Each observational unit type forms a table.

Wide vs long, concretely

An expression matrix is wide — the natural storage format:

gene	sample_1	sample_2	sample_3
TP53	120	340	90
BRCA1	44	51	60

The tidy (long) form:

gene	sample	count
TP53	sample_1	120
TP53	sample_2	340
TP53	sample_3	90
BRCA1	sample_1	44
…	…	…

long = wide.melt(id_vars="gene", var_name="sample", value_name="count")
wide_again = long.pivot(index="gene", columns="sample", values="count")

See Reshaping with pivot and melt.

When wide is right anyway

Tidy is a rule for analysis frames, not for all storage. Keep data wide when:

• You are doing matrix mathematics — PCA, correlation, distance. A 20,000 × 500 long frame is 10 million rows of mostly redundant labels. Use a NumPy array plus separate metadata frames.
• The format is dictated by a tool (DESeq2, limma, most matrix file formats).
• Memory matters. Long form repeats every label once per value.

The practical pattern: store wide, reshape to long at the moment of plotting, then throw the long frame away.

sns.boxplot(
    data=wide.melt(id_vars="gene", var_name="sample", value_name="count"),
    x="sample", y="count",
)

Why bioinformatics formats fight you

Almost every genomics format violates tidiness deliberately, to save space:

• VCF packs many variables into one INFO string, and one column per sample with :-delimited subfields. Two levels of nesting in a flat file. See Applied - Reading VCF with pandas.
• GFF/GTF packs arbitrary key–value attributes into column 9.
• FASTA headers carry structured metadata in a free-text line.
• SAM/BAM uses optional TAG:TYPE:VALUE fields.

Untangling these is most of what "data wrangling" means in this field, and it is mostly String Accessor work plus explode.

The tidiness checklist

Before you analyse, ask:

• Is any column name actually a value? (sample_1, 2024, treated) → melt.
• Does any cell contain multiple values? (A|B|C, AF=0.3;DP=44) → split and explode.
• Does one table mix two observational units? (variant-level and sample-level in one frame) → split into two tables joined by a key. See Merging and Joining.

See also

Reshaping with pivot and melt · Split-Apply-Combine · groupby · DataFrame · Applied - Reading VCF with pandas · Applied - GFF and Genomic Intervals