assign and pipe

In one line: the two methods that make method chaining work — assign adds columns without breaking the chain, pipe inserts your own function into it.

Why chain at all

# imperative — intermediate variables, mutation, hard to reorder
df = pd.read_csv("de.tsv", sep="\t")
df = df[df["baseMean"] > 10]
df["neglog10p"] = -np.log10(df["padj"])
df["direction"] = np.where(df["log2FoldChange"] > 0, "up", "down")
df = df.merge(annot, on="gene_id", how="left")
df = df.sort_values("padj")

# chained — one expression, no stale intermediates
result = (
    pd.read_csv("de.tsv", sep="\t")
      .query("baseMean > 10")
      .assign(
          neglog10p=lambda d: -np.log10(d["padj"].clip(lower=1e-300)),
          direction=lambda d: np.where(d["log2FoldChange"] > 0, "up", "down"),
      )
      .merge(annot, on="gene_id", how="left", validate="many_to_one")
      .sort_values("padj")
      .reset_index(drop=True)
)

The chained version has three concrete advantages beyond aesthetics: no intermediate variable can go stale, you can comment out any single line to bisect a problem, and nothing is mutated so re-running a cell is idempotent.

assign

df.assign(new_col=values)
df.assign(new_col=lambda d: d["a"] * 2)              # lambda receives the frame
df.assign(a=..., b=lambda d: d["a"] + 1)             # b can reference a — order matters
df.assign(**{"col with spaces": lambda d: ...})      # awkward names

The lambda is the point. Inside a chain the intermediate frame has no name, so df["a"] would reference the original frame, not the filtered one. lambda d: d["a"] refers to whatever is flowing through at that step.

Keyword arguments are evaluated in order (Python 3.7+), so later assignments can use earlier ones in the same assign call.

assign always returns a new frame, which fits Copy-on-Write semantics naturally.

pipe

pipe lets your own function participate in a chain:

def add_qc_flags(df, min_depth=10):
    return df.assign(
        low_depth=lambda d: d["depth"] < min_depth,
        high_af=lambda d: d["af"] > 0.5,
    )

result = (
    variants
      .query("filter == 'PASS'")
      .pipe(add_qc_flags, min_depth=20)
      .pipe(annotate_consequences)
      .pipe(lambda d: d.loc[~d["low_depth"]])
)

df.pipe(f, **kw) is exactly f(df, **kw) — the value is purely that it reads left-to-right instead of inside-out. Compare:

filter_rare(annotate(add_qc(df, 20)), 0.01)          # read inside-out
df.pipe(add_qc, 20).pipe(annotate).pipe(filter_rare, 0.01)   # read in order

If your function takes the frame as a later argument:

df.pipe((my_func, "data_arg"), other=1)

Reusable pipeline steps

The natural way to structure a bioinformatics pipeline:

def load(path):
    return pd.read_csv(path, sep="\t", dtype=SCHEMA, na_values=["."])

def filter_quality(df, min_qual=30):
    return df.loc[(df["qual"] >= min_qual) & (df["filter"] == "PASS")]

def add_frequencies(df, gnomad):
    return df.merge(gnomad, on=["chrom", "pos", "ref", "alt"],
                    how="left", validate="one_to_one")

def flag_rare(df, threshold=0.001):
    return df.assign(rare=lambda d: d["gnomad_af"].fillna(0) < threshold)

variants = (
    load("cohort.vcf.gz")
      .pipe(filter_quality, min_qual=30)
      .pipe(add_frequencies, gnomad=gnomad_table)
      .pipe(flag_rare, threshold=0.001)
)

Each step is independently testable, takes a frame and returns a frame, and mutates nothing. That is a pipeline you can actually maintain.

Logging inside a chain

The main objection to chaining is that you cannot inspect intermediate state. pipe solves it:

def log_shape(df, label=""):
    print(f"{label}: {df.shape[0]:,} rows × {df.shape[1]} cols")
    return df

result = (
    df.pipe(log_shape, "loaded")
      .query("qual > 30")
      .pipe(log_shape, "after quality")
      .merge(annot, on="gene", how="left")
      .pipe(log_shape, "after merge")          # ← catches join-induced row explosion
)

That third checkpoint is the one that earns its keep. A merge on a non-unique key silently multiplying your rows is a common and hard-to-spot error; a shape log makes it obvious. See Merging and Joining.

Related chainable methods

Everything below returns a new frame, so it composes:

.rename(columns=...)          .astype({...})
.drop(columns=[...])          .fillna(...)
.sort_values(...)             .reset_index(drop=True)
.set_index(...)               .head(n) / .sample(n)
.round(3)                     .replace({...})
.melt(...) / .pivot(...)      .clip(...)

.eval() is the assign equivalent of .query() if you prefer string expressions:

df.eval("neglog10p = -log10(padj)", inplace=False)

Less common than assign, and it cannot call arbitrary functions.

When not to chain

• Longer than ~10 steps. Break into named functions and pipe them.
• You need an intermediate twice. Assign it to a variable; do not recompute.
• Debugging. Split it temporarily.
• The team does not know the idiom. Readability is measured against your actual readers.

Common mistakes

• df.assign(x=df["a"] * 2) inside a chain — references the original frame, not the current one. Use the lambda.
• Expecting assign to mutate. It returns a new frame; you must bind the result.
• Assigning to a name with spaces or a keyword — needs the **{...} form.
• inplace=True in a chain. Returns None, so the chain dies with AttributeError: 'NoneType' object has no attribute ....
• Chains so long nobody can debug them. Insert pipe(log_shape) checkpoints.
• Forgetting the chain is lazy in appearance only — every step allocates. On very large frames, fewer steps is genuinely cheaper.

See also

DataFrame · Filtering and query · Copy-on-Write · Merging and Joining · pandas Performance