Concat

Core Principle: pd.concat binds DataFrames along a specified axis (vertically stacking rows along axis=0 or horizontally combining columns along axis=1) aligning on existing index labels without relational key matching.

Vertical: more rows

pd.concat([df1, df2], ignore_index=True)
pd.concat([df1, df2, df3], ignore_index=True)
pd.concat(list_of_frames, ignore_index=True)

ignore_index=True unless the index is meaningful. Without it you get duplicate index values (0,1,2,0,1,2), which then break .loc, silently change join behaviour, and make reset_index() necessary later anyway.

Columns are matched by name, not position:

Mismatched columns produce NaN:

join="inner" when you want strictness, but be aware it silently drops columns: check the result's shape.

Tracking the source

combined = pd.concat(
    [df_batch1, df_batch2, df_batch3],
    keys=["batch1", "batch2", "batch3"],
)                                            # → MultiIndex with the key as level 0

combined = pd.concat(
    {"batch1": df1, "batch2": df2},          # a dict works too
    names=["batch", "row"],
)
combined.reset_index(level="batch")           # key becomes a column

Or, more simply and often more usefully, add the label yourself:

frames = [pd.read_csv(p, sep="\t").assign(sample=p.stem) for p in paths]
combined = pd.concat(frames, ignore_index=True)

The assign version keeps you in flat-column land, which is easier to merge and group later. See MultiIndex for when the keys version is worth it.

Horizontal: more columns

pd.concat([df1, df2], axis=1)

This aligns on the index, which is the part that surprises people:

If you meant "glue these side by side positionally", reset both indices first: or, better, use merge on an explicit key so the intent is visible.

The performance rule

# BAD: quadratic; each concat copies everything so far
result = pd.DataFrame()
for path in paths:
    result = pd.concat([result, pd.read_csv(path)])

# GOOD: one allocation
frames = [pd.read_csv(p) for p in paths]
result = pd.concat(frames, ignore_index=True)

Same lesson as np.append in Array Creation. With 500 per-sample files, the bad version is minutes and the good version is seconds.

df.append() was removed in pandas 2.0 precisely because it encouraged the bad pattern.

Dtype preservation

Concatenation can silently change dtypes:

• Categoricals with different categories → falls back to object. Set the same categories on both first, or use pd.api.types.union_categoricals.
• int64 + a frame missing that column → NaN forces float64. Use Int64 if you need to keep integers.
• Mixed numeric and string in the same column name → object.

combined = pd.concat(frames, ignore_index=True)
combined.dtypes                                   # ← check after concatenating

Bioinformatics examples

from pathlib import Path

# combine per-sample variant calls
frames = []
for vcf in Path("calls").glob("*.tsv"):
    df = pd.read_csv(vcf, sep="\t", dtype=SCHEMA)
    df["sample"] = vcf.stem
    frames.append(df)
variants = pd.concat(frames, ignore_index=True)

# combine per-chromosome results
by_chrom = [pd.read_parquet(f"results/{c}.parquet") for c in CHROMS]
all_results = pd.concat(by_chrom, ignore_index=True)

# stack DE results from several contrasts for a faceted plot
de_all = pd.concat(
    {"treated_vs_control": de1, "drug_vs_control": de2},
    names=["contrast", "row"],
).reset_index(level="contrast")

sns.relplot(data=de_all, x="log2FoldChange", y="neglog10p", col="contrast")

# combine chunked processing results
chunks = [c.query("qual > 30") for c in pd.read_csv(big, sep="\t", chunksize=100_000)]
filtered = pd.concat(chunks, ignore_index=True)

The per-sample loading pattern in the first example is one of the most common things you will write in bioinformatics. Note the df["sample"] = vcf.stem: without it, once concatenated, you cannot tell which row came from which sample, and there is no recovering it.

concat vs merge: deciding

When in doubt use merge: the explicit key makes the intent readable and lets you pass validate=.

Common mistakes

• concat in a loop. Quadratic.
• Forgetting ignore_index=True → duplicate index values.
• axis=1 aligning on index when you meant positional.
• Categorical dtype degrading to object.
• Silently missing columns filled with NaN: check df.isna().sum() after.
• Not recording the source file/sample.
• Assuming column order matters. It does not; names do.
• Using concat(axis=1) where merge would document the intent.

See also

Merging and Joining · DataFrame · Reading and Writing Data · Categorical dtype · Reshaping and Stacking

Exercise

Write combine(blocks), where blocks is a list of (sample, df) pairs read from several per-sample tables. The tables share the column gene, most also have count, and the column order differs between them. Stack them into one frame with a fresh index, recording which sample each row came from in a sample column, since that information is unrecoverable once the frames are stacked.

Return the lines rows=<n> for the combined row count, cols=<names> for the combined column names as a comma-separated, alphabetically sorted string, then one <sample> rows=<n> line per block in the order the blocks were given, then missing_count=<n> for the number of combined rows whose count is missing, which includes every row from a block that had no count column at all: and finally top=<sample> for the block contributing the most rows, ties broken alphabetically, or top=none when every block is empty.