String Accessor

In one line: .str gives you vectorised string methods on a Series — and it is how you untangle every packed field in every bioinformatics file format.

The basics

s.str.upper(); s.str.lower(); s.str.strip()
s.str.len()
s.str.startswith("chr"); s.str.endswith("_1")
s.str.contains("BRCA")                          # regex by default
s.str.contains("BRCA", regex=False)             # literal, faster
s.str.replace("chr", "", regex=False)
s.str.removeprefix("chr"); s.str.removesuffix(".bam")
s.str.split(":")                                # → Series of lists
s.str.split(":", expand=True)                   # → DataFrame of columns
s.str.split(":", n=1, expand=True)              # split only on the first occurrence
s.str.cat(sep=",")                              # join all values
s.str[0:3]                                      # slice each string
s.str.zfill(2)                                  # '1' → '01'
s.str.pad(10, side="left")

.str only works on string dtype. Calling it on a numeric column raises, which is a free type check.

Note that in pandas 3.0 string columns are str dtype (Arrow-backed) rather than object. .str works the same way but is considerably faster. See pandas dtypes.

Extraction with regex

s.str.extract(r"(\w+):(\d+)")                   # first match → DataFrame of groups
s.str.extract(r"(?P<chrom>\w+):(?P<pos>\d+)")   # named groups → named columns
s.str.extractall(r"(\d+)")                       # ALL matches → MultiIndex
s.str.findall(r"\d+")                            # all matches → Series of lists
s.str.count("A")
s.str.match(r"^chr\d+$")                         # anchored boolean test
s.str.fullmatch(r"chr\d+")

extract with named groups is the cleanest way to parse a structured identifier:

loc = variants["locus"].str.extract(r"(?P<chrom>chr[\dXYM]+):(?P<pos>\d+)")
loc["pos"] = loc["pos"].astype("int32")

A non-matching row gives NaN, not an error. Always check:

failed = loc["chrom"].isna()
if failed.any():
    print(f"{failed.sum()} unparseable: {variants.loc[failed, 'locus'].head().tolist()}")

split + expand: the workhorse

df[["chrom", "pos", "ref", "alt"]] = df["variant"].str.split("_", expand=True)

expand=True turns one column into several in one line. Two cautions:

• If rows have different numbers of fields, the short ones get None and the column count is set by the longest row.
• The results are all strings. Cast afterwards.

parts = df["variant"].str.split("_", expand=True)
assert parts.shape[1] == 4, f"expected 4 fields, got {parts.shape[1]}"
df[["chrom", "pos", "ref", "alt"]] = parts
df["pos"] = df["pos"].astype("int32")

explode: one row per element

df.assign(csq=df["consequence"].str.split("&")).explode("csq")

A variant annotated missense_variant&splice_region_variant becomes two rows. This is how you get to a countable, groupable form. See Reshaping with pivot and melt.

Bioinformatics parsing recipes

# strip Ensembl version suffixes so IDs will actually join
df["gene_id"] = df["gene_id"].str.split(".").str[0]

# normalise chromosome naming (UCSC ↔ Ensembl)
df["chrom"] = df["chrom"].str.removeprefix("chr")
df["chrom"] = "chr" + df["chrom"].astype(str)

# parse a locus string
df[["chrom", "pos"]] = df["locus"].str.split(":", expand=True)

# parse a GFF attributes column
gff["gene_id"] = gff["attributes"].str.extract(r'gene_id "([^"]+)"')
gff["gene_name"] = gff["attributes"].str.extract(r'gene_name "([^"]+)"')

# parse a VCF INFO field for one key
df["dp"] = df["INFO"].str.extract(r"(?:^|;)DP=(\d+)").astype("Int64")
df["af"] = df["INFO"].str.extract(r"(?:^|;)AF=([\d.eE+-]+)").astype("float32")

# parse a sample FORMAT field
df[["gt", "ad", "dp"]] = df["sample1"].str.split(":", n=2, expand=True)
df[["ref_count", "alt_count"]] = df["ad"].str.split(",", expand=True).astype("int32")

# HGVS-shaped strings — for FILTERING only, never for interpretation
df["accession"] = df["hgvs"].str.split(":").str[0]
df["is_coding"] = df["hgvs"].str.contains(":c.")

# clean sample names from filenames
df["sample"] = df["path"].str.extract(r"([^/]+)\.bam$")

# structured sample IDs
df[["patient", "tissue", "rep"]] = df["sample_id"].str.split("_", expand=True)

The (?:^|;) in the INFO extractions matters: without it, DP= also matches inside MQ_DP= or ExcessHet;DP=. Anchoring to a semicolon or the start of the string prevents that.

On the HGVS line: string manipulation of HGVS is acceptable for coarse filtering (which accession, is it coding). It is never acceptable for extracting positions, comparing variants, or deciding equivalence — a regex cannot know that c.76_78del and c.77_79del are the same event. Use hgvs Parser. See hgvs Pitfalls.

Missing values

s.str.upper()                        # NaN stays NaN, no error
s.fillna("").str.contains("BRCA")    # NaN → False
s.str.contains("BRCA", na=False)     # same, more direct

.str.contains returns NaN for missing input, and NaN in a boolean mask raises when you use it in .loc. Always pass na=False (or na=True) when the result feeds a filter.

Performance

.str methods are vectorised at the pandas level but still involve per-element Python string operations for many methods — they are much faster than .apply(lambda x: ...) but not as fast as numeric ufuncs. With Arrow-backed strings (the pandas 3.0 default) many operations are now compiled and substantially faster.

For very heavy parsing of very large files, consider parsing at read time with a proper format library (pysam, cyvcf2) rather than regex over a DataFrame.

s.str.contains("x", regex=False)     # skip the regex engine when you can

Common mistakes

• NaN propagating into a boolean mask. Pass na=False.
• Forgetting .str is regex by default. s.str.replace(".", "") deletes every character. Pass regex=False or escape it.
• split(expand=True) on ragged rows producing surprise column counts.
• Not casting after splitting. Everything comes back as a string, so pos sorts as "1", "10", "100", "2".
• Unanchored INFO regexes matching the wrong key.
• Silent NaN from a failed extract. Always count them.
• Parsing HGVS with regex for anything semantic.
• Using .apply(lambda x: x.upper()) instead of .str.upper().

See also

pandas dtypes · Reshaping with pivot and melt · Applied - Reading VCF with pandas · Applied - GFF and Genomic Intervals · hgvs Parser · Series