DataFrame

Core Principle: A DataFrame is a two-dimensional labeled data structure composed of an ordered collection of Series sharing a common row Index, supporting heterogeneous column dtypes.

DataFrame Structure & Memory Anatomy

           ┌──────── columns (an Index) ────────┐
     index │  gene    chrom   log2fc    padj    │
       ────┼────────────────────────────────────┤
        0  │  TP53    chr17    -2.3    0.001    │
        1  │  BRCA1   chr13     1.8    0.04     │
        2  │  EGFR    chr7      0.2    0.9      │

Each column is a Series with its own dtype. The row index is shared. Storage is column-oriented, which means:

• Column operations are fast; row operations are slow.
• Adding a column is cheap; adding a row means rebuilding every column.
• df["gene"] is a cheap Series view; df.loc[0] builds a new object-dtype Series from four different column dtypes.

That last point explains why df.apply(f, axis=1) is so slow: it constructs one mixed-dtype Series per row.

Construction

pd.DataFrame({"gene": [...], "log2fc": [...]})          # from dict of columns
pd.DataFrame(array, columns=[...], index=[...])          # from 2-D array
pd.DataFrame([{"gene": "TP53", ...}, ...])               # from list of records
pd.read_csv("f.tsv", sep="\t")                           # the usual route

First five things to run on any new frame

df.shape
df.head()
df.info()                     # dtypes + non-null counts + memory
df.describe()                 # numeric summary
df.isna().sum()               # missing per column

df.info() is the highest-value call. It shows you in one screen whether a numeric column got parsed as text, whether your integers are float64 because of a stray NA, and how much memory you are using.

Selecting

df["gene"]                    # one column → Series
df[["gene", "padj"]]          # several → DataFrame
df.loc[:, "gene":"padj"]      # label slice: INCLUSIVE of the endpoint
df.iloc[0:5, 0:2]             # positional: exclusive, like Python
df.loc[df["padj"] < 0.05]     # boolean mask
df.loc[df["padj"] < 0.05, "gene"]     # mask + column, one step
df.query("padj < 0.05 and abs(log2fc) > 1")

Full treatment in loc vs iloc and Filtering and query. The rule worth learning now: use .loc[rows, cols] in one call, not df[rows][cols] in two. The two-step version is chained indexing, and under Copy-on-Write assignments through it silently do nothing.

Adding and modifying columns

df["neglog10p"] = -np.log10(df["padj"])
df.loc[df["padj"] < 0.05, "sig"] = True

df = df.assign(
    neglog10p=lambda d: -np.log10(d["padj"]),
    direction=lambda d: np.where(d["log2fc"] > 0, "up", "down"),
)

assign returns a new frame and lets later expressions reference earlier ones via the lambda, which makes it chainable. See assign and pipe.

Dropping, renaming, reordering

df.drop(columns=["tmp"])
df.drop(index=[0, 1])
df.rename(columns={"padj": "fdr"})
df.rename(columns=str.lower)
df[["gene", "log2fc", "padj"]]        # reorder by selecting
df.set_index("gene")
df.reset_index()
df.reset_index(drop=True)              # discard the old index

The method surface you will actually use

df.sort_values(["chrom", "pos"])
df.sort_values("padj").head(20)
df.nlargest(20, "log2fc"); df.nsmallest(20, "padj")
df.drop_duplicates(subset=["chrom", "pos"], keep="first")
df.groupby("gene")["padj"].min()               # → [[groupby]]
df.merge(annot, on="gene", how="left")          # → [[Merging and Joining]]
pd.concat([df1, df2], ignore_index=True)        # → [[Concat]]
df.melt(id_vars="gene"); df.pivot(...)          # → [[Reshaping with pivot and melt]]
df.fillna(0); df.dropna(subset=["padj"])        # → [[Missing Data in pandas]]
df.astype({"pos": "int32"})
df.apply(f)                                      # column-wise by default
df.pipe(my_function)                             # → [[assign and pipe]]

Method chaining

The idiomatic modern style:

result = (
    pd.read_csv("de_results.tsv", sep="\t")
      .query("baseMean > 10")
      .assign(neglog10p=lambda d: -np.log10(d["padj"].clip(lower=1e-300)))
      .merge(gene_annot, on="gene_id", how="left", validate="many_to_one")
      .sort_values("padj")
      .reset_index(drop=True)
)

Every step is a named operation, there are no intermediate variables to get stale, and you can comment out any line to bisect a problem. The .clip(lower=1e-300) guards against -log10(0) = inf: a small habit that saves a broken plot.

Iteration: mostly don't

for col in df.columns: ...              # fine
for name, group in df.groupby("gene"): ...  # fine, and idiomatic
for idx, row in df.iterrows(): ...      # SLOW, and dtypes are lost per row
for row in df.itertuples(): ...         # faster, dtypes preserved

iterrows builds a Series per row and coerces everything to a common dtype: your int64 positions come back as float64. If you must iterate, use itertuples. Better: find the vectorised form. See pandas Performance.

Memory

df.memory_usage(deep=True).sum() / 1e6      # MB; deep=True counts string data

Big wins, in order: category for low-cardinality strings like chromosome or consequence (Categorical dtype), float32 instead of float64, int32 for coordinates that fit, and Arrow-backed strings (default in pandas 3.0).

Bioinformatics examples

# a differential expression result table
de = pd.read_csv("deseq2.tsv", sep="\t").set_index("gene_id")

sig = de.query("padj < 0.05 and abs(log2FoldChange) > 1")
print(f"{len(sig)} significant of {len(de)}")

# per-chromosome variant counts
variants.groupby("chrom").size().sort_values(ascending=False)

# join a variant table to a gene panel
variants.merge(panel, on="gene", how="inner", validate="many_to_one")

# wide expression matrix with proper labels
expr = pd.read_csv("counts.tsv", sep="\t", index_col=0)
expr.shape                        # (n_genes, n_samples)
mat = expr.to_numpy()             # drop to NumPy for the math

Common mistakes

• Chained indexing (df[mask]["col"] = x). Silent no-op under CoW. Use .loc[mask, "col"] = x.
• df.merge silently duplicating rows from a non-unique key. Pass validate="one_to_one" / "many_to_one".
• iterrows in a loop over a million rows.
• Not setting dtype= on read for genomic files: chromosome columns mix "1" and "X", sample IDs get coerced to numbers, leading zeros vanish.
• Forgetting most methods return a new frame. df.drop(columns=[...]) does not modify df.
• df.append: removed in pandas 2.0. Use pd.concat.
• inplace=True as a performance strategy. It is not faster under CoW and it breaks chaining.

See also

Series · Index Objects · loc vs iloc · groupby · Merging and Joining · Copy-on-Write · Tidy Data · pandas Performance

Exercise

Write summarise(df), which takes a count matrix read from a tab-separated table whose first column is gene and whose remaining columns are samples. Return a list of report lines. The first line is always rows=<n_rows> cols=<n_cols>, counting gene among the columns. If the frame has no rows, return that line alone.

Otherwise append one line per gene for the three genes with the highest mean across the sample columns, formatted <gene> <mean> with the mean to two decimal places, sorted by mean descending and by gene name ascending where two means tie. Missing sample values are skipped when averaging rather than treated as zero, and a gene whose sample values are all missing is left out of the ranking entirely.