Tidy Data

Core Principle: The tidy data standard structures datasets such that each variable forms a column, each observation forms a row, and each type of observational unit forms a dedicated table.

Formulated by Hadley Wickham in 2014, the tidy data framework establishes tabular consistency. It is foundational across Python scientific computing because Seaborn, pandas groupby, and modern statistical modeling libraries expect tidy input structures, whereas raw bioinformatics formats frequently store data in non-tidy representations.

The three rules

1. Each variable forms a column.
2. Each observation forms a row.
3. Each observational unit type forms a table.

Wide vs long, concretely

An expression matrix is commonly stored in a wide format:

gene	sample_1	sample_2	sample_3
TP53	120	340	90
BRCA1	44	51	60

The tidy (long) form:

gene	sample	count
TP53	sample_1	120
TP53	sample_2	340
TP53	sample_3	90
BRCA1	sample_1	44
…	…	…

long = wide.melt(id_vars="gene", var_name="sample", value_name="count")
wide_again = long.pivot(index="gene", columns="sample", values="count")

See Reshaping with pivot and melt.

When wide is right anyway

Tidy is a rule for analysis frames, not for all storage. Keep data wide when:

• High-dimensional numerical computations such as principal component analysis (PCA), correlation matrices, and distance calculations require compact 2D matrices. In these contexts, storing a 20,000 × 500 dataset in long format would create 10 million redundant rows; a 2D NumPy array with auxiliary metadata frames is significantly more efficient.
• The format is dictated by a tool (DESeq2, limma, most matrix file formats).
• Memory matters. Long form repeats every label once per value.

The practical pattern: store wide, reshape to long at the moment of plotting, then throw the long frame away.

sns.boxplot(
    data=wide.melt(id_vars="gene", var_name="sample", value_name="count"),
    x="sample", y="count",
)

Why bioinformatics formats fight you

Almost every genomics format violates tidiness deliberately, to save space:

• VCF packs many variables into one INFO string, and one column per sample with :-delimited subfields. Two levels of nesting in a flat file. See Applied - Reading VCF with pandas.
• GFF/GTF packs arbitrary key–value attributes into column 9.
• FASTA headers carry structured metadata in a free-text line.
• SAM/BAM uses optional TAG:TYPE:VALUE fields.

Untangling these is most of what "data wrangling" means in this field, and it is mostly String Accessor work plus explode.

The tidiness checklist

Before you analyse, ask:

• Is any column name actually a value? (sample_1, 2024, treated) → melt.
• Does any cell contain multiple values? (A|B|C, AF=0.3;DP=44) → split and explode.
• Does one table mix two observational units? (variant-level and sample-level in one frame) → split into two tables joined by a key. See Merging and Joining.

See also

Reshaping with pivot and melt · Split-Apply-Combine · groupby · DataFrame · Applied - Reading VCF with pandas · Applied - GFF and Genomic Intervals

Exercise

Write reshape(text), where text is a tab-separated wide table whose first column identifies the gene and whose remaining column names are sample names — column names that are really values, which is the tidiness violation this lesson names. Melt it to long form with sample and value columns, drop every observation whose value is missing (NA or an empty cell), and sort the result by gene and then by sample. Return the list of output lines.

Emit one line per surviving observation, formatted <gene> <sample> <value> with the value to three decimal places. Then emit columns: <names>, listing in alphabetical order, comma separated, the sample columns you get back when you pivot the long frame to wide again, or columns: none when nothing survived. Finish with observations=<n> dropped=<n> roundtrip=<state>, where dropped counts the missing cells in the original grid and roundtrip is exact when the pivoted frame carries every original sample column and no missing cell, and lossy otherwise. A table with a header but no data rows is lossy unless it had no sample columns at all.