assign and pipe

Core Principle: Method chaining in pandas relies on .assign() to create or transform columns and .pipe() to apply custom DataFrame-level functions without intermediate variable assignment.

Why chain at all

# imperative: intermediate variables, mutation, hard to reorder
df = pd.read_csv("de.tsv", sep="\t")
df = df[df["baseMean"] > 10]
df["neglog10p"] = -np.log10(df["padj"])
df["direction"] = np.where(df["log2FoldChange"] > 0, "up", "down")
df = df.merge(annot, on="gene_id", how="left")
df = df.sort_values("padj")

# chained: one expression, no stale intermediates
result = (
    pd.read_csv("de.tsv", sep="\t")
      .query("baseMean > 10")
      .assign(
          neglog10p=lambda d: -np.log10(d["padj"].clip(lower=1e-300)),
          direction=lambda d: np.where(d["log2FoldChange"] > 0, "up", "down"),
      )
      .merge(annot, on="gene_id", how="left", validate="many_to_one")
      .sort_values("padj")
      .reset_index(drop=True)
)

The chained version has three concrete advantages beyond aesthetics: no intermediate variable can go stale, you can comment out any single line to bisect a problem, and nothing is mutated so re-running a cell is idempotent.

assign

df.assign(new_col=values)
df.assign(new_col=lambda d: d["a"] * 2)              # lambda receives the frame
df.assign(a=..., b=lambda d: d["a"] + 1)             # b can reference a: order matters
df.assign(**{"col with spaces": lambda d: ...})      # awkward names

The lambda is the point. Inside a chain the intermediate frame has no name, so df["a"] would reference the original frame, not the filtered one. lambda d: d["a"] refers to whatever is flowing through at that step.

Keyword arguments are evaluated in order (Python 3.7+), so later assignments can use earlier ones in the same assign call.

assign always returns a new frame, which fits Copy-on-Write semantics naturally.

pipe

pipe lets your own function participate in a chain:

def add_qc_flags(df, min_depth=10):
    return df.assign(
        low_depth=lambda d: d["depth"] < min_depth,
        high_af=lambda d: d["af"] > 0.5,
    )

result = (
    variants
      .query("filter == 'PASS'")
      .pipe(add_qc_flags, min_depth=20)
      .pipe(annotate_consequences)
      .pipe(lambda d: d.loc[~d["low_depth"]])
)

df.pipe(f, **kw) is exactly f(df, **kw): the value is purely that it reads left-to-right instead of inside-out. Compare:

filter_rare(annotate(add_qc(df, 20)), 0.01)          # read inside-out
df.pipe(add_qc, 20).pipe(annotate).pipe(filter_rare, 0.01)   # read in order

If your function takes the frame as a later argument:

df.pipe((my_func, "data_arg"), other=1)

Reusable pipeline steps

The natural way to structure a bioinformatics pipeline:

def load(path):
    return pd.read_csv(path, sep="\t", dtype=SCHEMA, na_values=["."])

def filter_quality(df, min_qual=30):
    return df.loc[(df["qual"] >= min_qual) & (df["filter"] == "PASS")]

def add_frequencies(df, gnomad):
    return df.merge(gnomad, on=["chrom", "pos", "ref", "alt"],
                    how="left", validate="one_to_one")

def flag_rare(df, threshold=0.001):
    return df.assign(rare=lambda d: d["gnomad_af"].fillna(0) < threshold)

variants = (
    load("cohort.vcf.gz")
      .pipe(filter_quality, min_qual=30)
      .pipe(add_frequencies, gnomad=gnomad_table)
      .pipe(flag_rare, threshold=0.001)
)

Each step is independently testable, takes a frame and returns a frame, and mutates nothing. That is a pipeline you can actually maintain.

Logging inside a chain

The main objection to chaining is that you cannot inspect intermediate state. pipe solves it:

def log_shape(df, label=""):
    print(f"{label}: {df.shape[0]:,} rows × {df.shape[1]} cols")
    return df

result = (
    df.pipe(log_shape, "loaded")
      .query("qual > 30")
      .pipe(log_shape, "after quality")
      .merge(annot, on="gene", how="left")
      .pipe(log_shape, "after merge")          # ← catches join-induced row explosion
)

That third checkpoint is the one that earns its keep. A merge on a non-unique key silently multiplying your rows is a common and hard-to-spot error; a shape log makes it obvious. See Merging and Joining.

Related chainable methods

Everything below returns a new frame, so it composes:

.rename(columns=...)          .astype({...})
.drop(columns=[...])          .fillna(...)
.sort_values(...)             .reset_index(drop=True)
.set_index(...)               .head(n) / .sample(n)
.round(3)                     .replace({...})
.melt(...) / .pivot(...)      .clip(...)

.eval() is the assign equivalent of .query() if you prefer string expressions:

df.eval("neglog10p = -log10(padj)", inplace=False)

Less common than assign, and it cannot call arbitrary functions.

When not to chain

• Longer than ~10 steps. Break into named functions and pipe them.
• You need an intermediate twice. Assign it to a variable; do not recompute.
• Debugging. Split it temporarily.
• The team does not know the idiom. Readability is measured against your actual readers.

Common mistakes

• df.assign(x=df["a"] * 2) inside a chain: references the original frame, not the current one. Use the lambda.
• Expecting assign to mutate. It returns a new frame; you must bind the result.
• Assigning to a name with spaces or a keyword: needs the **{...} form.
• inplace=True in a chain. Returns None, so the chain dies with AttributeError: 'NoneType' object has no attribute ....
• Chains so long nobody can debug them. Insert pipe(log_shape) checkpoints.
• Forgetting the chain is lazy in appearance only: every step allocates. On very large frames, fewer steps is genuinely cheaper.

See also

DataFrame · Filtering and query · Copy-on-Write · Merging and Joining · pandas Performance

Exercise

Write keep_expressed(df, min_base) and add_scores(df), then use them from run_pipeline(df, min_base) as steps in a single chain. keep_expressed keeps the rows whose baseMean is at least min_base. add_scores first drops the rows whose padj is missing, then returns a new frame with three columns added in one assign call: neglog10p, the negative base-10 logarithm of padj clipped at a lower bound of 1e-300 so that a padj of zero does not become infinite; direction, which is up where log2FoldChange is greater than zero and down otherwise; and score, the product of neglog10p and the absolute value of log2FoldChange, computed from the neglog10p assigned earlier in the same call rather than recomputed.

run_pipeline returns loaded=<n> for the incoming row count and expressed=<n> for the count after the filter: both produced by a logging function inserted into the chain with pipe: followed by one <gene> <direction> <score> line, score to two decimal places, for the three highest-scoring genes, ordered by score descending and by gene name ascending where scores tie. When no row survives the filter, only the two counting lines are returned. Each step in the chain must reference the frame flowing through it via a lambda, not the outer df.