Concat

In one line: stack frames vertically (more rows) or horizontally (more columns) — no key matching, unlike Merging and Joining.

Vertical: more rows

pd.concat([df1, df2], ignore_index=True)
pd.concat([df1, df2, df3], ignore_index=True)
pd.concat(list_of_frames, ignore_index=True)

ignore_index=True unless the index is meaningful. Without it you get duplicate index values (0,1,2,0,1,2), which then break .loc, silently change join behaviour, and make reset_index() necessary later anyway.

Columns are matched by name, not position:

a = pd.DataFrame({"gene": ["TP53"], "af": [0.1]})
b = pd.DataFrame({"af": [0.2], "gene": ["BRCA1"]})       # different order
pd.concat([a, b])                                         # aligns correctly

Mismatched columns produce NaN:

c = pd.DataFrame({"gene": ["EGFR"]})       # no 'af' column
pd.concat([a, c])                           # af is NaN for the EGFR row

pd.concat([a, c], join="inner")             # keep only shared columns

join="inner" when you want strictness, but be aware it silently drops columns — check the result's shape.

Tracking the source

combined = pd.concat(
    [df_batch1, df_batch2, df_batch3],
    keys=["batch1", "batch2", "batch3"],
)                                            # → MultiIndex with the key as level 0

combined = pd.concat(
    {"batch1": df1, "batch2": df2},          # a dict works too
    names=["batch", "row"],
)
combined.reset_index(level="batch")           # key becomes a column

Or, more simply and often more usefully, add the label yourself:

frames = [pd.read_csv(p, sep="\t").assign(sample=p.stem) for p in paths]
combined = pd.concat(frames, ignore_index=True)

The assign version keeps you in flat-column land, which is easier to merge and group later. See MultiIndex for when the keys version is worth it.

Horizontal: more columns

pd.concat([df1, df2], axis=1)

This aligns on the index, which is the part that surprises people:

a = pd.DataFrame({"x": [1, 2]}, index=["g1", "g2"])
b = pd.DataFrame({"y": [3, 4]}, index=["g2", "g3"])
pd.concat([a, b], axis=1)
#      x    y
# g1  1.0  NaN
# g2  2.0  3.0
# g3  NaN  4.0

If you meant "glue these side by side positionally", reset both indices first — or, better, use merge on an explicit key so the intent is visible.

The performance rule

# BAD — quadratic; each concat copies everything so far
result = pd.DataFrame()
for path in paths:
    result = pd.concat([result, pd.read_csv(path)])

# GOOD — one allocation
frames = [pd.read_csv(p) for p in paths]
result = pd.concat(frames, ignore_index=True)

Same lesson as np.append in Array Creation. With 500 per-sample files, the bad version is minutes and the good version is seconds.

df.append() was removed in pandas 2.0 precisely because it encouraged the bad pattern.

Dtype preservation

Concatenation can silently change dtypes:

• Categoricals with different categories → falls back to object. Set the same categories on both first, or use pd.api.types.union_categoricals.
• int64 + a frame missing that column → NaN forces float64. Use Int64 if you need to keep integers.
• Mixed numeric and string in the same column name → object.

combined = pd.concat(frames, ignore_index=True)
combined.dtypes                                   # ← check after concatenating

Bioinformatics examples

from pathlib import Path

# combine per-sample variant calls
frames = []
for vcf in Path("calls").glob("*.tsv"):
    df = pd.read_csv(vcf, sep="\t", dtype=SCHEMA)
    df["sample"] = vcf.stem
    frames.append(df)
variants = pd.concat(frames, ignore_index=True)

# combine per-chromosome results
by_chrom = [pd.read_parquet(f"results/{c}.parquet") for c in CHROMS]
all_results = pd.concat(by_chrom, ignore_index=True)

# stack DE results from several contrasts for a faceted plot
de_all = pd.concat(
    {"treated_vs_control": de1, "drug_vs_control": de2},
    names=["contrast", "row"],
).reset_index(level="contrast")

sns.relplot(data=de_all, x="log2FoldChange", y="neglog10p", col="contrast")

# combine chunked processing results
chunks = [c.query("qual > 30") for c in pd.read_csv(big, sep="\t", chunksize=100_000)]
filtered = pd.concat(chunks, ignore_index=True)

The per-sample loading pattern in the first example is one of the most common things you will write in bioinformatics. Note the df["sample"] = vcf.stem — without it, once concatenated, you cannot tell which row came from which sample, and there is no recovering it.

concat vs merge — deciding

Question	Answer
Same columns, more rows?	concat(axis=0)
Different columns for the same entities, matched by a key?	merge
Different columns, already index-aligned?	concat(axis=1) — but merge is clearer
Combining per-sample or per-chromosome outputs?	concat(axis=0)
Adding annotation?	merge

When in doubt use merge — the explicit key makes the intent readable and lets you pass validate=.

Common mistakes

• concat in a loop. Quadratic.
• Forgetting ignore_index=True → duplicate index values.
• axis=1 aligning on index when you meant positional.
• Categorical dtype degrading to object.
• Silently missing columns filled with NaN — check df.isna().sum() after.
• Not recording the source file/sample.
• Assuming column order matters. It does not; names do.
• Using concat(axis=1) where merge would document the intent.

See also

Merging and Joining · DataFrame · Reading and Writing Data · Categorical dtype · Reshaping and Stacking