DataFrame

In one line: an ordered dict of Series sharing one row index — column-oriented, labelled, and heterogeneously typed.

The mental model

           ┌──────── columns (an Index) ────────┐
     index │  gene    chrom   log2fc    padj    │
       ────┼────────────────────────────────────┤
        0  │  TP53    chr17    -2.3    0.001    │
        1  │  BRCA1   chr13     1.8    0.04     │
        2  │  EGFR    chr7      0.2    0.9      │

Each column is a Series with its own dtype. The row index is shared. Storage is column-oriented, which means:

• Column operations are fast; row operations are slow.
• Adding a column is cheap; adding a row means rebuilding every column.
• df["gene"] is a cheap Series view; df.loc[0] builds a new object-dtype Series from four different column dtypes.

That last point explains why df.apply(f, axis=1) is so slow: it constructs one mixed-dtype Series per row.

Construction

pd.DataFrame({"gene": [...], "log2fc": [...]})          # from dict of columns
pd.DataFrame(array, columns=[...], index=[...])          # from 2-D array
pd.DataFrame([{"gene": "TP53", ...}, ...])               # from list of records
pd.read_csv("f.tsv", sep="\t")                           # the usual route

First five things to run on any new frame

df.shape
df.head()
df.info()                     # dtypes + non-null counts + memory
df.describe()                 # numeric summary
df.isna().sum()               # missing per column

df.info() is the highest-value call. It shows you in one screen whether a numeric column got parsed as text, whether your integers are float64 because of a stray NA, and how much memory you are using.

Selecting

df["gene"]                    # one column → Series
df[["gene", "padj"]]          # several → DataFrame
df.loc[:, "gene":"padj"]      # label slice — INCLUSIVE of the endpoint
df.iloc[0:5, 0:2]             # positional — exclusive, like Python
df.loc[df["padj"] < 0.05]     # boolean mask
df.loc[df["padj"] < 0.05, "gene"]     # mask + column, one step
df.query("padj < 0.05 and abs(log2fc) > 1")

Full treatment in loc vs iloc and Filtering and query. The rule worth learning now: use .loc[rows, cols] in one call, not df[rows][cols] in two. The two-step version is chained indexing, and under Copy-on-Write assignments through it silently do nothing.

Adding and modifying columns

df["neglog10p"] = -np.log10(df["padj"])
df.loc[df["padj"] < 0.05, "sig"] = True

df = df.assign(
    neglog10p=lambda d: -np.log10(d["padj"]),
    direction=lambda d: np.where(d["log2fc"] > 0, "up", "down"),
)

assign returns a new frame and lets later expressions reference earlier ones via the lambda, which makes it chainable. See assign and pipe.

Dropping, renaming, reordering

df.drop(columns=["tmp"])
df.drop(index=[0, 1])
df.rename(columns={"padj": "fdr"})
df.rename(columns=str.lower)
df[["gene", "log2fc", "padj"]]        # reorder by selecting
df.set_index("gene")
df.reset_index()
df.reset_index(drop=True)              # discard the old index

The method surface you will actually use

df.sort_values(["chrom", "pos"])
df.sort_values("padj").head(20)
df.nlargest(20, "log2fc"); df.nsmallest(20, "padj")
df.drop_duplicates(subset=["chrom", "pos"], keep="first")
df.groupby("gene")["padj"].min()               # → [[groupby]]
df.merge(annot, on="gene", how="left")          # → [[Merging and Joining]]
pd.concat([df1, df2], ignore_index=True)        # → [[Concat]]
df.melt(id_vars="gene"); df.pivot(...)          # → [[Reshaping with pivot and melt]]
df.fillna(0); df.dropna(subset=["padj"])        # → [[Missing Data in pandas]]
df.astype({"pos": "int32"})
df.apply(f)                                      # column-wise by default
df.pipe(my_function)                             # → [[assign and pipe]]

Method chaining

The idiomatic modern style:

result = (
    pd.read_csv("de_results.tsv", sep="\t")
      .query("baseMean > 10")
      .assign(neglog10p=lambda d: -np.log10(d["padj"].clip(lower=1e-300)))
      .merge(gene_annot, on="gene_id", how="left", validate="many_to_one")
      .sort_values("padj")
      .reset_index(drop=True)
)

Every step is a named operation, there are no intermediate variables to get stale, and you can comment out any line to bisect a problem. The .clip(lower=1e-300) guards against -log10(0) = inf — a small habit that saves a broken plot.

Iteration — mostly don't

for col in df.columns: ...              # fine
for name, group in df.groupby("gene"): ...  # fine, and idiomatic
for idx, row in df.iterrows(): ...      # SLOW, and dtypes are lost per row
for row in df.itertuples(): ...         # faster, dtypes preserved

iterrows builds a Series per row and coerces everything to a common dtype — your int64 positions come back as float64. If you must iterate, use itertuples. Better: find the vectorised form. See pandas Performance.

Memory

df.memory_usage(deep=True).sum() / 1e6      # MB; deep=True counts string data

Big wins, in order: category for low-cardinality strings like chromosome or consequence (Categorical dtype), float32 instead of float64, int32 for coordinates that fit, and Arrow-backed strings (default in pandas 3.0).

Bioinformatics examples

# a differential expression result table
de = pd.read_csv("deseq2.tsv", sep="\t").set_index("gene_id")

sig = de.query("padj < 0.05 and abs(log2FoldChange) > 1")
print(f"{len(sig)} significant of {len(de)}")

# per-chromosome variant counts
variants.groupby("chrom").size().sort_values(ascending=False)

# join a variant table to a gene panel
variants.merge(panel, on="gene", how="inner", validate="many_to_one")

# wide expression matrix with proper labels
expr = pd.read_csv("counts.tsv", sep="\t", index_col=0)
expr.shape                        # (n_genes, n_samples)
mat = expr.to_numpy()             # drop to NumPy for the math

Common mistakes

• Chained indexing (df[mask]["col"] = x). Silent no-op under CoW. Use .loc[mask, "col"] = x.
• df.merge silently duplicating rows from a non-unique key. Pass validate="one_to_one" / "many_to_one".
• iterrows in a loop over a million rows.
• Not setting dtype= on read for genomic files — chromosome columns mix "1" and "X", sample IDs get coerced to numbers, leading zeros vanish.
• Forgetting most methods return a new frame. df.drop(columns=[...]) does not modify df.
• df.append — removed in pandas 2.0. Use pd.concat.
• inplace=True as a performance strategy. It is not faster under CoW and it breaks chaining.

See also

Series · Index Objects · loc vs iloc · groupby · Merging and Joining · Copy-on-Write · Tidy Data · pandas Performance